Regression Models Using Numerous Variables

Assessing Regression Models Using Numerous Variables

Regression model on the Amex, Iowa housing data set builds regression models for the house sale price with numerous variables.  Some of which are highly correlated, continuous variables along to the other side of the continuum by evaluating categorical, low correlated variables.  An assessment of each model will be conducted along with a review on the statistical and ODS output will be conducted and interpreted.  Additionally, linear regression models will be conducted based on two and three variables along with an evaluation of the impact of these new variables and how they add value to predicting the sales price of a house.  Finally, an assessment on the model being specified and the next steps will be provided.

Part A – Step 1

The variable I choose is was one that was created in first installment – Total Floor Square Footage (TotalFlrSF).

Equation

The normal equation is y = β₀ + β₁x + ε

Based on the above parameter estimates, the equation is:

• Sale Price = $11,406 + $113.30303 x TotalFlrSF

Thus, for each unit increase in the TotalFlrSF, an increase in sales price of $113.30 occurs.  This assumes all values are greater than zero but even at zero, a SalePrice of$11,406 would result.  As we are evaluating houses, this would be logical but could be if perhaps a house was not livable and is considered a ‘tear down’ where someone would spend the time and money required to build a new house.  However, this would be outside the norm and would require a different equation to determine these types of sale prices.

Model Adequacy

The automatic generated ODS output:

regression models adequacy

The various produced ODS output from SAS shows a cluster of points in the residual versus predicted value indicating there is an issue as it should be completely random without any type of pattern.  The Q-Q Plot titled “Quantile” is abnormal as it deviates from the line and has heavy tails which means there is a larger probability of getting very large values.  The Predicted Value plot again shows that there are issues as the points are not on the line.

Assess the Goodness-of-Fit of this model

Regression Models – TotalSqFt Fit Plot

Regression Models – TotalSqFt Fit Plot

The P value is small but not equal to zero which means we can reject the null hypothesis that it is equal to zero.  Thus, each variable is significant. Additionally, the statistical summaries show an R-Square of .5092 and an adjusted R-Square of .509 which is also very good as it represents the proportion of variability in the dependent variable that can be explained by the regression model.

• Fitted regression model over the scatter-plot: The fit Plot shows a plot of all of the data along with the 95% confidence and prediction limits.  We can clearly see the amount of data that exceeds the boundaries confidence limits.
• Assessment of the normality of the residuals using a Q-Q plot and/or histogram of the standardized residuals:  The normal quantile plot of the residuals and the residual histogram are consistent with the assumption of Gaussian errors and shows the narrow spread in the residuals but with a high peak. The Q-Q plot helps detect violations from normality if they are normal, the points will cluster tightly around a reference line.  As we can see, we have a deviation from the line and has heavy tails which means there is a larger probability of getting very large values.  Thus, normality is not true in this model.
• Assessment of homoscedasticity by plotting the predictor variable against the standardized residuals:  The quantile histogram helps diagnose violation of the normality and homoscedasticity assumptions.  The points on the Predicted Value versus Sales Price plot of the dependent variables versus the predicted values along the 45-degree line show a definite pattern which indicates that the model is not appropriate.  Homoscedasticity also referred to as the assumption of constant error variance looks for errors that aren’t independent causing the plot to be more linear looking or that flares out as x increases.  Thus, based on the above, this model does not appear to be homoscedastic.
• Check for potential outliers using Cook’s Distance:  Cook’s Distance measures the effect of each data point on the predicted value with the lowest value being zero and the highest point showing the influence of the point.  As we can see, there are only two points that exceed .2 but are less than one, the acceptable limit, an investigation into these two outliers should be conducted.  Thus, these two points could be outliers but having an influence on the results.  While the above does not reflect the actual Cook’s D score, the values show how the vector of fitted values move when the observation is deleted.

Model Adequacy Conclusions

The variable TotalSqFt has normality issues based on the above analysis.  The ODS plots reflect patterns of clusters, heavy tailed Q-Q plot, and the Predicted versus Sale Price are not on the line.  Additionally, the Residual plot does not appear to be complete randomness and the Fit Plot which we would expect to be random, has more of a sideways funnel pattern.  However, the P value is < .0001 which means each variable is significant and that we can reject the null hypotheses.  Additionally, the Adjusted R-Square is .509 which is significant.  The matrix of graphs and plots provides a high-level overview of the relationship between total square footage in a house and sale price.

Part A – Step 2

The Best X to Predict Y in the Regression Model

The ‘best’ simple linear regression model to predict Sales Price using the R-square option reflects the following output:

Ironically, Total Floor Square Footage has the highest R-square value and Adjusted R-square which was discussed in Part A Step 1.  So not to repeat the same outputs, I’ll use GrLivArea for the rest of this task.

Equation & Interpret each coefficient:

The equation for GrLivArea is SalePrice = $13,290 x 111.694 x GrLivArea.   Which reflects that for each unit increase in GrLivArea the Sale Price increases by $111.69.  This seems reasonable and logical.

R² measures the variability in y remaining after x has been considered and is often called the proportion of variance explained by the regressor x (Montgomery, D.  Introduction to Linear Regression Analysis. p. 36).  So, the variables with the closest R² equaling to one implies that they explain most of the variability in y.

In what sense is the model the ‘best’ model:

GrLivArea is the best model as it logical to the sales price of a house which people obviously plan to live in so they are more likely to spend additional money for a larger space.  It has an Adjusted R-Square of .5004.  The p value is small so we can reject the null hypothesis.

Anything funny about it from an interpretation standpoint?

The equation is ‘funny’ if you consider a house with zero GrLivArea which would result in a sale price of $13,290.  While one could perhaps justify, it based on the price of land only, it is illogical.  Additionally, the increase in value based on a one unit change of ~$111.70 is logical.  Overall, we would expect the model to be better with the increased Adjusted R-Square yet we still have the same issues and concerns based on the plots.

Goodness-of-Fit

Goodness-of-fit statistics on the GrLivArea are as follows:

The F Value is 2,922.59 is significantly greater than one.  The p-value is small reflecting that each variable is significant. Based on the above, there is a linear relationship between SalePrice and GrLivArea. The Adjusted R-Square of .4994 reflects the variability that is accounted for using SalesPrice and GrLivArea.  While there is a small difference between R-Square and Adjusted R-Square, Adjusted R-Square will not penalize us for adding variables and will only change if the variable reduces the residual mean square.  The P value is small so we can reject the null hypothesis.

The ODS output is as follows:

Regression Models – SaS ODS output

The Predicted Value to the Residual has a definite pattern and seems to fan out showing that the variable is increasing.  The ODS Quantile or Q-Q plot indicates that normality is not true as the points deviate from the line (see red area) and is heavy tailed. Again, similar to TotalSqFt Cook’s D we have three influential points with two points that exceed 0.2.  It would only be prudent to look into these three points to ensure they are valid.  Additionally, the Residuals for Sale Price are not random as we would expect and again has a bit of a pattern to it.  The Fit Plot also has a distinct pattern to it.  Thus, we have normality issues.

Regression Models – FitPlot GrLivArea

Regression Models – Residuals GrLivArea

The Residuals between SalePrice and GrLivArea are again, similar to TotalSqFt, where homoscedasticity does not appear to be linear within the plot but does flare out a bit as GrLivArea increases.  Thus, this model does not appear to be homoscedastic.

Comments on Adequacy

While GrLivArea is the highest Adjusted R-Square value out of all of the continuous variables accounting for .4994 of the variability leaving another .5006 unexplained.  Perhaps, one of the categorical variables will be assist in improving the unexplained portion.  Again, we have conflicting information between the adjusted R-Square the ODS plots.  Additionally, GrLivArea has a small p-value but we have normality issues based on the ODS plots.

The Best X to Predict Y Conclusions

The GrLivArea variable is a variable with a small p-value which means that the variable is significant.  Additionally, it has a good Adjusted R-Square value however, the ODS outputs reflect that we have normality issues.  We have patterns in the Predicted Value versus Residual value where there shouldn’t be any, the Q-Q plot deviates from the line, the Predicted Value versus Sales Price is also not on the line which reflect that is it not ideal.  The residual plot also has a pattern and the fit plot also has a fanning out pattern.  Thus, the GrLivArea is not normal.

Part A – Step 3

Categorical value in a Regression Model:

Based on week one Exterior1 is a category variable with the following histogram

Regression Models- Exterior Frequency

Equation

Evaluating a few of the variables we get the following result:

• SalePrice = $150,475 x $2,575.12 x Exterior1

Interpret Coefficient

While Siding is only one of sixteen variables within the Exterior1 variable, a one unit change will result in an increase in value by $2,575 not that we can have a house with more than one siding, this is illogical but it simply means that a house will siding on average has an increased sales price of $2,575.  The p-values is also greater than one which means we can reject the null hypothesis and conclude that there is a linear relationship between Exterior1 Siding and SalePrice.

Looking at the Adjusted R-Square, we see that SalePrice using Exterior1 only explains 1.74% of the variability which is low.

Anything funny about the coefficient interpretation

I can’t see there is anything funny about the coefficient other than its low value in explaining variability.   Variance inflation is also one.

Generate ODS output and assesses the Goodness-of-fit

Regression Models – Exterior ODS

Reviewing the ODS plots we don’t seem to have a pattern in the Predicted Valued and Residual plot unlike in the GrLivArea ODS plot but we do have another heavy-tailed Q-Q plot that deviates from the line, a cluster in the predicted value with a lot more spikes in Cook’s D reflecting that there are more observations that are having an influence on the results but is still less than one on the axis scale.  Additionally, we see skewed residual.

Model Adequacy

Regression Models – FitPlot Exterior

Regression Models – Residuals Exterior

Reviewing the plots and various statistics, the residuals plot can’t fan out due to the type of variable but there does not appear to be any linear relationship between the different exterior1 categories.  The fit plot resembles shows a slight positive correlation with a bit of a ‘u’ shape between the data.  It appears to be bimodal which is reflective of categorical model.  Overall, Exterior1 is not normally distributed or a good predictor in determining SalePrice which is reflected in the various plot, statistics and reasons provided.

Does the predicted model go through the mean value of Y for each category group?

The above Fit Plot line does appear to go through the mean value of each category but shows the distribution of values for each category based on the variability.  The confidence limits but as mentioned above, reflects a slight correlation which is reflected in the slight slope of the line.  The plot shows the distribution of values within each category with the largest amount of variability in the SalePrice of a house using the Exterior1 Category of category16 which is wood siding.  Having the line go through the mean doesn’t really provide any value as the categories are listed in order of their assigned value.

Is this good or bad / why or why not?

I don’t think having the line adds any value and in fact, causes confusion and possible incorrect conclusions that there is some type of linear relationship when in fact there isn’t one.  Additionally, it doesn’t provide any insight on which categories perform better than others or the volume of data in each.

Categorical Value Conclusion

While the Exterior1 category analysis is interesting to conduct, the results in being able to predict the SalePrice is low with the Adjusted R-Square of 0.0170.  Additionally, the plots and charts provided additional insight into Cook’s D observations with few influential peaks, a heavy tailed Q-Q plot.  No surprise, the does not appear to be a linear relationship between Exterior1 and SalePrice.  Finally, the Fit Plot would not be my preferred method of making categorical assessment of data as they add very little value and can be misleading.

Part A – Task 4 Analysis

Of the three models described above, the best model is TotalSqFt as it has the highest Adjusted R-Square at .509, the errors across the values appear to be independent of the variables and thus considered to be homoscedastic.   Cook’s D has only a few data points that are having an influential influence.  These points along with the outliers need some further research but do not expect them to change the outcomes of the above analysis.  Additionally, the F value is significantly greater than one.  Again, we have conflicting information based on the plots and the R-Square values which needs to be further investigated.

Part B – Task 5 Regression Model Using Two Variables

Variables:

The two variables are TotalFlrSF and GrLivArea.

Equation:

• Sale Price = $11,688 + $185.03078 x TotalFlrSF – $71.69170 x GrLivArea

For each unit increase in TotalFlrSF the SalePrice increases by $185 yet a decrease of $71.69 impacts SalePrice for each unit increase in GrLivArea.  The above equation is different from the simple linear regression models above with the negative value each unit of GrLivArea has on SalePrice.  This is not intuitive but because of the unusual choice in variables and the obvious overlap, it makes logical sense.  Additionally, the Adjusted R-Square is .5106 again reflected the amount of variability that is accounted for in using these two variables.  However, this Adjusted R-Square has only improved slighted from the simple linear regression model using TotalFlrSF which had an Adjusted R-Square of .5090.  Again, this is not unexpected due to the overlap in the variables.

By adding one unit of TotalFlrSF with all other factors remaining constant, we can see how the SalePrice changes.  However, one unit of TotalFlrSF and GrLivArea are not the same and need to ensure that this is not confused.

Goodness-of-Fit

Regression Models – ODS Multiple Variables

Regression Models – ODS Multiple Variables – TotalFlrArea and GrLivArea

Based on the above, we have a small p-value reflecting that each variable is significant and thus, we cannot reject the null hypothesis.  However, unlike the prior variables, we have a Variance inflation number of 119.15477 which is significantly outside of the acceptable range of 1 to 5.  Thus, we have multi-linearity issues.  We can see a lot of similarities with TotalFlSF with the fanning out pattern in the Predicted Value and Residual plot, the Q-Q plot again, deviates from the line and Cook’s D has only two influential points that exceeding 0.2 which is less than one.  Again, the Adjusted R-Square is the largest out of all of the models analyzed with a value of 0.5106.  Homoscedasticity looks for errors that aren’t independent causing the plot to be more linear looking or that flares out as x increases.  Again, there is a clear clustered pattern here.  Above, we can see compare TotalFlrSF and GrLivArea which, for the most part, appear to be identical.

Better Fit

As mentioned earlier, we have only accounted for .5106 of the variability leaving another .4894 unexplained.  While this has increased from the prior models it is only a small increase.  Thus, a significant amount of additional work is required to reduce this value.

Regression Model Using Two Variables Conclusion

Combining two continuous explanatory variables TotalFlSF and GrLivArea provided similar results to the simple linear regression model using TotalFlSF.  This is not surprising, as GrLivArea is encapsulated in TotalFlSF as such the equation has a negative component to it with a small increase to the Adjusted R-Square value.  The p-value reflects that each variable is significant and thus, we can reject the null hypothesis.  However, the variance influence exceeds our normal range reflecting that we have multi-linearity issues which isn’t a surprise.  Evaluating the plots based on the multiple variables is again, very similar to the analysis on TotalFlSF with the a fanning out in the Predicted Value and Residual plot. Thus, this we again have normality issues.

Part B – Step 6 Using Three Variables

Variables:

The three variables are TotalFlrSF, GrLivArea and MiscVal.

Equation:

• Sales Price = $11,105 + $186.4389 x TotalFlrSF – $72.39791 x GrLivArea – $9.14957 x MiscVal

For each unit increase in TotalFlrSF the SalePrice increases by $186.44, a small increase of $1.40 over the equation in part 5, and similarly a decrease of $72.40 (a further decline from $71.69 in part 5) but now there is another decrease which is $9.15 per unit of MiscVal.  The decrease in value in MiscVal on SalePrice is not as intuitive as TotalFlrSF and GrLivArea however due to the low volume and value of the MiscVal data points it is understandable there the variable would have little impact on SalePrice.

Regression Models – ODS Multiple Variables TotalFlrSF, GrLivArea and MiscVal.

Goodness-of-Fit

We can see a lot of similarities with part 5, the p-values are small reflecting that each variable is significant and thus, we reject the null hypothesis.  The Variance Inflation for TotalFlSF is 119.17355, GrLivArea is 119.159 which exceed the upper boundaries of our 1-5 range and the MiscValue is close to the bottom of our acceptable range with a value of 1.00470.  We again see the Predicted Value to the Residual having a fanning out pattern which reflects that the variance is increasing.  Cook’s D plot has changed in our axis measures but still has only a few influential points.  The Q-Q plot is again, deviating from the line and is heavy tailed.  The Predicted Value to Sales Price shows a cluster of values.  Thus, we still have the same normality issues.

Adjusted R-Square increased from .5106 to .5146 again reflected the amount of variability that is accounted for in using these two variables.  However, this is a very small increase – 0.0040 over the Adjusted R-Value in question 5 and the value of adding the MiscVal variable.  The Adjusted R-Square has only improved slighted from the simple linear regression model using TotalFlrSF and the multiple variables between TotalFlSF and GrLivArea.

Regression Models – Residuals Multiple Variables TotalFlrSF, GrLivArea and MiscVal.

The Residual plots for each variable provides more of a view on how they alter depending on the variables over the results in part 5 but still do not appear to be random with distinct patterns to each variable.  However, we can see how MiscVal errors do not provide any easy to see, linear relationship and has only a few values.  Thus, we still have normality issues.

Changes & Better Fit

Overall, the additional variable – MiscVal has not made any real impact in predicting SalePrice.  While this is no surprise, based on its low correlation value.   While our Adjusted R-Value did increase, it was not significant with a small .0040 increase thus showing us that adding additional values does not necessarily improve the results or assist us in creating a better fitting equation and explaining the variability.  While Adjusted R-Square plays an increased importance on the criteria in comparing the models primarily due to not wanting to make incorrect decisions by simply adding variables, the plots and charts are helpful in understand the relationships and how they impact the model.

Using Three Variables Conclusion

Adding additional variables do not necessarily improve results.  By adding a low correlated variable – MiscVal the other variables within the equation received minor adjustments.  While each of the variables has a small p-value we can reject the null hypothesis.  Similar to the prior analysis, there are not a lot of changes with the exception being the variance influence.  The review of the plots reflects similar patterns with the exception of the residual plot where the MiscValue has a different pattern but still not a pattern of randomness which we hope for.  However, the analysis remained relatively the same as to having only two variables as in part 5.  Additionally, adding the MiscVal variable had a minor effect in explaining the variability and did not have any significant impact on SalePrice.

Conclusion

By observing the various models and the effects of adding and reviewing various variables we can easily grasp the changes through the various plots, equations and summary statistics.  Continuous variables have a much different look and resulting plots over continuous variables.  Additionally, evaluating the Adjusted R-Square and F-Value provides insight into how the model is performing even when the impact is difficult to ascertain through the plots.

The model may not be appropriately specified as all variables seem to be equal which may not be the case.  Additionally, we could be adding new variables without considering their association with other variables as shown between TotalFlSF and GrLivArea – removing GrLivArea would have significant effects on TotalFlSF.

Next steps in the modeling process would be to investigate a few of the influential points that have been mentioned in several parts such as Cook’s D and focus on determining whether the Adjusted R-Score can be enhanced through the evaluation of different variables and try to account for more of the variability as we all of the models have not exceeded 51.46%.  Finally, transformation of the data so the conflicts between the R-squares and ODS outputs are decreased would be really helpful in creating a useful model we are confident in.