Automated Variable Selection

The Amex, Iowa housing data set build has been utilized to develop various iterative regression models to determine the mean sales price of a house based on numerous variables. The variables range correlated, continuous variables to categorical variables. In this installment, we continue building the model using raw categories and later, the dummy coding.  Automated variable selection procedures using various selection procedures will be conducted where we will view some of the categorical variables being incorporated into the model including the entire category full variable and not just a few of the automated selected categories.  Finally, a validation framework will be conducted whereby the automated variable selection will again be applied to compare the results of the training sample versus the test data.  Operational validation will also be carried out with the resulting models being compared to the full test data.  Finally, the best model will be provided along with the results and some potential next steps for consideration.

Part A: Dummy Coding of Categorical Variables

Part 1:  Categorical Automated Variable Selection

The categorical value, BldgType has been chosen for further analysis.  BldgType consists of five different types:

• 1Fam: Single-family Detached
• 2FmCon: Two-family Conversion; originally built as one-family dwelling
• Duplx: Duplex
• TwnhsE: Townhouse End Unit
• TwnhsI: Townhouse Inside Unit

The frequency of each BldgType is as follows:

As we can see from above, there are a number of observations in each category with the 2 Family Complex having the least with just 62 observations.  Due to this, all categories have been utilized.  If a category had few observations they could be combined into one but as described is not the case.  Based on the above, the means of each category are as follows:

Automated Variable Selection – Category Means

As shown above, for category 1 – 1Fam, the predicted value is $180,677 + (1 x $91.32) = $180,768.32 however, as shown earlier the mean value is $184,812.04 for a difference of -$4,043.72.  Therefore, we know the model is wrong.

Similarly, for a 2famCon the equation from is $180,677 + (2 x $182.63) = $180,859.63 but the mean value is $125,581.71.  The model is supposed to predict the mean values which clearly, it is not so again, the model is wrong.

Throughout this iteration of the regression models, transformation of values using log have not been utilized so the results can be easily understood by all potential home buyers.  Using log can assist in dealing with skewed data but can also be confusing to understand.

A simple linear regression model would be:

Automated Variable Selection – Linear Regression SalePrice

Categorical or qualitative variables between the response variable and the regressor variable donot go through the mean of Y in each category as shown above.   As we can see above with the Predicted Value plot, the observations do go through the intercept of $180,677 but do not go through the rest of the values as outlined in the variance chart above.

Categorical Variable Conclusion

We have chosen the BldgType categorical variable as the variable to predict SalePrice. BldgType has five different types. Based on the mean values, a simple linear regression model was provided to determine SalePrice based on BldType.  Based on a comparison between the mean value and the expected value and the resulting variances, we know the model is incorrect.  However, due to the type of variable – BlgType categorical, we need to assign dummy codes to the variables in order to be able to interpret the variables.

Step 2:  Dummy Code

As qualitative variables have no natural scale of measurement, we need to assign a set of levels to them to account for the effect that the variable may have on the response through the use of indicator variables or often referred to as dummy variables (text, p. 260).

Using SAS, several dummy codes are assigned to the categorical variables in BldgType as follows:

If BldgType = ‘1Fam’ then BldgCategory1 = 1; else BldgCategory1 = 0;

If BldgType = ‘2FmCon’ then BldgCategory2 = 2; else BldgCategory2 = 0;

If BldgType = ‘Duplex’ then BldgCategory3 = 3; else BldgCategory3 = 0;

If BldgType = ‘TwnhsE’ then BldgCategory4 = 4; else BldgCategory4 = 0;

If BldgType = ‘TwnhsI’ then BldgCategory = 5; else BldgCategory5 = 0;

 While there are five BldType’s one variable is used as the basis for interpretation.  Thus, while we five variables, we base our model on four variables and have used BldgCategory5 as the basis of our interpretation.

The summary table is as follows:

SalePrice = .  Therefore, the average sale price by BldgCategroy is as follows:

• BldgCategory1: SalePrice = $131,996 + ($52,816 x 1) for Catgory1 = $184,812
• BldgCategory2: SalePrice = $131,996 + $0 for Category2 = $131,996**
• BldgCategory3: SalePrice = $131,996 + ($2,604.19 x 3) for Category3 = $139,808.57
• BldgCategory4: SalePrice = $131,996 + ($15,079 x 4) for Category 4 = $192,312

**However, in BldgCategory2 SAS alerts us to an error that the combination of variables results in perfect multicollinearity where the variance is infinite and therefore, cannot solve for   As such, the variable is removed from the model which should have been done in the first place.  Based on this, BldgCategory2 will be eliminated in the analysis.

It should be noted, that a BldgCategory can only have one value and thus, we can determine the SalePrice and cannot have two BldgCategory for example, a house cannot be categorized as a 1Fam and a Duplex.

Evaluating the above model for each BldgCategory:

The above reflection of the model if say, the BldgCategory is 1 (1Fam) the equation and the predicted value which is then compared to the mean value as detailed in Step 1.  As the summary table reflects, the predicted and mean values match this, each of them matches unlike in step 1, the line goes through the mean sales price.

As the model is predicting the mean values, we know that the model is correct.

Looking at the ODS generated results, we also gain insight into how the model is performing.

Automated Variable Selection – Building Category

Reviewing the above plots, we can see that the residuals do not form a straight line but if a curve was drawn, would curve upwards to the third line of values.  The Q-Q Plot does not fit the line with heavy and light tails reflecting that there are more extreme values than would be expected if they were a normal distribution.  A few outliers are also seen.  The predicted values versus the sale price line do not fit the 45-degree line but Cook’s D scale looks good as the values are low.   However, we also have a low Adjusted R-Square value of 0.0333.

Task 3:  Hypothesis Test for each beta

Testing the null hypothesis that all of the slopes are zero, the null hypothesis is:

H_o:  ß1 =  ß3 =  ß4 = 0

Note that the above does not include as  ß2 it has been eliminated from the model as described above.

H₁:    ß1 ≠ 0 or  ß3≠ 0  0  or  ß≠ 0 at least one mean differs from zero.

The p value for ß1, ß3, ß4   are all <.0001

Rejection of H₀ implies that at least one of the regressors contributes significantly to the model.

As we can see in the summary table, the p value is <.0001 in each category except BuildingCategory1, BldgCategory3 which exceeds the .0001 threshold.

Task 4:  Dummy Coded Variables v2

A second categorical variable, ‘RoofStyle’ was selected.  Below are a list of the categories and frequencies of each:

With the frequencies of Flat, Gambre, Mansar and Shed being very low, only two categories were assigned values in SAS as follows:

• If RoofStyle = ‘Gable’ then RoofStyleCategory1 = 1; else RoofStyleCategory1 = 0;
• If RoofStyle = ‘Hip’ then RoofStyleCategory2 = 2; else RoofStyleCategory2 = 0;

Based on these two categories, the related statistical information is as follows:

Automated Variable Selection – Roof style category

Residuals – Roof style category

Fit Plt for Roof Style Category

Dummy Code Conclusion

In summary, an analysis on the categorical value, BldgType was conducted using the SAS Proc Sort and Means.  However, because the predicted values did not go through the mean values, we know that dummy variables are required.  As such, a transformation of the BldgType was dummy coded so that the statistical information could be interpreted.  Additionally, a comparison of the dummy coded variables was compared to the predicted mean reflecting that the model was correct.  Hypothesis tests were provided for each of the betas reflecting the threshold of when to accept or reject the null hypothesis.  Finally, a second categorical variable, RoofStyle was selected and also dummy coded with two main categories being utilized ‘Gable’ and ‘Hip’ as they accounted for 98% of the observation on the types of roofs.

Part B:  Automated Variable Selection Procedures

Part 5:  Automated Variable selection

Utilizing a multitude of continuous predictor categorical variables that have been dummy coded, several automated variable selection criteria were utilized to various models as follows:

proc reg data = temp outest = stepwise_summary;

• adjrsq: model SalePrice = BldgCategory1 BldgCategory2 BldgCategory3 BldgCategory4 BldgCategory5 RoofStyleCategory1 RoofStyleCategory2 LotFrontage LotArea YearBuilt YearRemodel BsmtFinSF1 BsmtFinSF2 BsmtUnfSF TotalBsmtSF FirstFlrSF SecondFlrSF GrLivArea MasVnrArea LowQualFinSF BedroomAbvGr TotRmsAbvGrd WoodDeckSF GarageArea TotalPorchSF TotalFullBath TotalBath TotalHalfBath PoolArea MiscVal MoSold YrSold TotalFlrSF HouseAge / selection = adjrsq slentry=0.01 slstay=0.01 AIC VIF BIC MSE; output out=stepwise_out pred=yhat residual = resid ucl=ucl lcl=lcl cookd=cook  covratio=cov dffits=dfits press=prss;
• cp: model SalePrice = BldgCategory1 …selection = cp slentry=0.01 slstay=0.01 …
• rsquare: model SalePrice = BldgCategory1 … selection = rsquare slentry=0.01 …
• Forward: model SalePrice = BldgCategory1 …selection = forward slentry=0.01 slstay=0.01 …
• Backward: model SalePrice = BldgCategory1 … selection = backward slentry=0.01 slstay=0.01 …
• stepwise: model SalePrice = BldgCategory1 … selection = stepwise slentry=0.01 slstay=0.01 …

2,930 observations were read, 2,420 were used and 510 had missing values.

Adjusted R-Square

The results for Adjusted R-Square:

Sale Price = $-861,952 + 26367 (BldgCategory1) – 5582.03 (BldgCategory2) + 3773.80 (BldgCategory3) -6051.65 (RoofStyleCategory1) + 4053.25 (RoofStyleCategory2) – 119.37 (LotFrontage) + .48446 (LotArea) + 445.58 (YearRemodel) + 13.18 (BsmtFinSF1) + 22.86 (BsmtFinSF1) + 68.42 (GrLivArea) + 46.83 (MasVnrArea) – 40.56 (LowQualFinSF) – 12010 (BedroomAbvGr) + 2881.62 (TotRmsAbvGrd) + 28.06 (WoodDeckSF) + 43.69 (GarageArea) + 27.03 (TotalPorchSF) + 7400.21 (TotalFullBath) – 80.30 (PoolArea) – 16.47 (MiscVal) – 458.08 (HouseAge

The summary tables is as follows:

Automated Variable Selection – Adjusted R Square

Mallow’s Cp

The results for Mallow’s Cp:

Sale Price = $-872421 + 26470 (BldgCategory1) – 5447.83 (BldgCategory2) + 3815.00 (BldgCategory3) -7020.33 (RoofStyleCategory2) – 117.70 (LotFrontage) + .48918 (LotArea) + 447.75 (YearRemodel) + 13.18 (BsmtFinSF1) + 22.80 (TotalBsmtSF) + 68.62 (GrLivArea) + 46.84 (MasVnrArea) – 40.96 (LowQualFinSF) – 12037 (BedroomAbvGr) + 2862.99 (TotRmsAbvGrd) + 28.01 (WoodDeckSF) + 43.37 (GarageArea) + 26.97 (TotalPorchSF) + 7379.01 (TotalFullBath) – 80.83 (PoolArea) – 16.49 (MiscVal) – 454.47 (HouseAge)

The summary tables is as follows:

Automated Variable Selection – Mallow’s Cp

R-Square

The results for R-Square:

Sale Price = $-219,912 + 26253 (BldgCategory1) – 6028.14 (BldgCategory2) + 3696.75 (BldgCategory3) -5911.87 (RoofStyleCategory1) + 4046.56 (RoofStyleCategory2) – 122.91 (LotFrontage) + .47268 (LotArea) + 445.50 (Year Built) + 447.66 (YearRemodel) + 34.39 (BsmtFinSF1) + 21.21 (BsmtFinSF2) + 21.72 (BsmtUnfSF) + 42.47 (FirstFlrSF) + 38.81 (SecondFlrSF) + 27.70 (GrLivArea) + 46.56 (MasVnrArea) – 11894 (BedroomAbvGr) + 2896.27 (TotRmsAbvGrd) + 27.68 (WoodDeckSF) + 43.69 (GarageArea) + 26.80 (TotalPorchSF) + 6282.23 (TotalFullBath) + 1715.20 (TotalBath) – 81.57 (PoolArea) – 16.54 (MiscVal) – 35.18 (MoSold) – 768.34 (YrSold)

The summary tables is as follows:

Automated Variable Selection – R Square

Forward

The results for forward:

SalePrice = $-1782301 + 26463 (BldgCategory1) – 3667.85 (BldgCategory3) + 2868 (BldgCategory4) + 7013.66 (RoofStyleCategory2) – 118.22 (LotFrontage) + .49 (LotArea) + 454.58 (YearBuilt) + 446.29 (YearRemodel) + 13.14 (BsmtFinSF1) + 22.83 (TotalBsmtSF) + 27.46 (GrLivArea) + 46.96 (MasVnrArea) – 12030 (BedroomAbvGr) + 2875.93 (TotRmsAbvGrd) + 28.07 (WoodDeckSF) + 43.41 (GarageArea) + 27.03 (TotalPorchSF) + 7347.02 (TotalFullBath) – 80 (PoolArea) – 16.52 (MiscVal) + 41.17 (TotalFlrSF)

The summary tables is as follows:

Automated Variable Selection – Forward

Backward

The results for backward:

SalePrice = $-898,361 + 30720 BldgCategory1 + 4968.2 BldgCategory3 + 7004.04 RoofStyleCategory2 – 136.21 LotFrontage + .48205 LotArea + 459.55 YearRemodel + 35.81 BsmtFinSF1 + 23.04218 BsmtFinSF2 + 22.49 BsmtUnfSF + 69.32 FirstFlrSF + 68.76 SecondFlrsf + 47.14 MasVnrArea -12245 BedroomAbvGr + 2839.95 TotRmsAbvGrd + 28.85 WoodDeckSF + 43.273 GarageArea + 28.72 TotalPorchSF + 7117.04 TotalFullBath – 78.44 PoolArea -16.58 MiscVal – 446.97 HouseAge.

The summary tables is as follows:

Automated Variable Selection – Backward

Stepwise

The results for Stepwise:

SalePrice = $-1,791,258 + 30,720 (BldgCategory1) + 4993.997 (BldgCategory3)+7,025.3 (RoofStyleCategory2) – 135.38 (Lot Frontage) + .48669 (LotArea) + 446.51498 (YearBuilt) + 457.74856 (YearRemodel) + 13.24029 (BsmtFinSF1) + 22.84988 (TotalBsmtSF) + 47.24907 (MasVnrArea) -12264 (BedroomAbvGr) + 2841.91110 (TotRmsAbvGrd) + 28.94598 (WoodDeckSF) + 43.34671 (GarageArea) + 28.748 (TotalPorchSF) + 7112.83 (TotalFullBath) -77.40687 (PoolArea) – 16.59398 (MiscVal) + 68.87176 (TotalFlrSF)

The summary tables is as follows:

Automated Variable Selection – Stepwise

An overall summary of each model using the automated variable selection is as follows:

Based on the above summary, each technique created a different model based on the variables selected.  As reflected above, the Adjusted R-Square and Mallow’s Cp are the most similar with an identical Adjusted R-Square and very similar Root MSE and Cp.  The Forward model is seemed to be in the middle of the Adjusted R-Square model and Mallow’s Cp with BIC AIC values within the range of the other two models.

The number of parameters used in each of the models also varied significantly with the minimum of 20 used in the Stepwise model and the maximum of 28 in the R-Squared model.  Choosing the best regression model often considers the number of regressions to be as accurate as possible but also needs to consider the costs of attaining the data and maintaining the model thus, balancing it with fewer regressors as possible due to the variance of the prediction  increases as regressors increase (p. 328).  Additionally, we can see in the above summary table that the F Value of the R-Squared model is lower than the Adjusted R-Square and Mallow’s Cp model.  The other consideration is that we know that R-Squared increases when additional predictors are added to a model thus, a model with more terms may appear to have a better fit because of the increased terms.

Based on the above while considering the number of parameters, I think the Adjusted R-Square is the best model.

Sale Price = $-861,952 + 26367 (BldgCategory1) – 5582.03 (BldgCategory2) + 3773.80 (BldgCategory3) -6051.65 (RoofStyleCategory1) + 4053.25 (RoofStyleCategory2) – 119.37 (LotFrontage) + .48446 (LotArea) + 445.58 (YearRemodel) + 13.18 (BsmtFinSF1) + 22.86 (BsmtFinSF1) + 68.42 (GrLivArea) + 46.83 (MasVnrArea) – 40.56 (LowQualFinSF) – 12010 (BedroomAbvGr) + 2881.62 (TotRmsAbvGrd) + 28.06 (WoodDeckSF) + 43.69 (GarageArea) + 27.03 (TotalPorchSF) + 7400.21 (TotalFullBath) – 80.30 (PoolArea) – 16.47 (MiscVal) – 458.08 (HouseAge)

Automated Variable Selection Conclusion – Full Data Set

Based on the above, each modelling technique created a different regression model all resulting in different resulting statistics.  While some of the Goodness of Fit statistics had minimal differences between the model’s others were more significant.  I believe the R-Square is the best model due to the highest R-Square adjusted value, R-Square, F Value and AIC despite having more variables over the other models.  This is based on the logic that the variables and able to be acquired easily with no a very low cost and no obstacles exist in attaining the data.

Step 6: Multicollinarity on Automated Variable Selection

Dummy coded variables were both selected in the automated selection process – BldgCategory# and RoofStyleCategory#.  While RoofStyleCategory only has two items and both have been included in the chosen model but in the BldgCategory only three of the variables have been automatically selected.  However, the other two BldgCategory4 and BldgCategory5.  This is due to perfect multicollinarity where the variance inflation factor is infinite and thus, cannot invert the matrix and solve for beta.  Thus, SAS informs us that the model is not full rank and that the estimate is biased which is reflected with a 0 or B in the Degrees of Freedom column.

Thus, despite adding this category to the model, the additional variables (BldgCategory4 and BldgCategory5) do not appear.

Evaluating the continuous variables to determine if they are statistically significant, we have several variables where the p-value is larger than 0.0001 and thus, cannot conclude that a significant difference exists. These variables are as follows:

• LotFrontage: p=.007
• LowQualFinSF: p = .0111

Below 0.0001, significant. Over 0.0001, not significant.

Sale Price = $283,643 + 26467 (BldgCategory1) – 6663.69 (BldgCategory2) + 4172.51 (BldgCategory3) +  .26521 (LotArea) + 438.12 (YearBuilt) + 477.79 (YearRemodel) + 38.52 (BsmtFinSF1) + 26.21 (BsmtFinSF2) + 24.25 (BsmtUnfSF) + 45.68 (SecondFlrSF) + 22.52 (GrLivArea) + 47.32 (MasVnrArea) – 12,206 (BedrromAbvGrd) + 3591.76 (TotRmsAbvGrd) + 22.29 (WoodDeckSF) + 40.60 (GarageArea) + 23.29 (TotalPorchSF) + 6528.68 (TotalFullBath) – 560.77 (TotalBath) – 56.20 (PoolArea) – 10.61 (MiscVal) + 95.39 (MoSold) – 1044.80 (YrSold)

The above is the final adjusted model with the three non-statistically significant variables removed but including all BldgCategories.   As shown, two of the BldgCategories are not reflected in the equation as they are perfectly multicollinarity. 

Part C:  Validation Framework

Cross validation of data is accomplished using a training/test split of the data.  Taking a step back, we will split up the data as follows:

• Training/In-Sample: 70% of the data is used to train the model which you will see below at just over 2,000 observations
• Test/Out-of-Sample: 30% of the data will be held back with 100% of the data being used for testing which you will see below at around 900 observations

We will then assess the differences in the goodness-of-fit statistics to see how they differ from prior models using 100% of the data and evaluate the differences.

Step 8:  Training Data

Using the automated variable selection, a summary of each of the techniques are provided below.  In each of the models, only the training data is used which consists of 1,672 observations instead of the full dataset which contained 2,420 observations but includes missing values.

Adjusted R-Square

Using the training observations on the data, the Adjusted R-Square model summary is as follows:

Mallow’s Cp

Using the training observations data, the Mallow’s Cp model summary is as follows:

Automated Variable Selection – Validation Mallow’s Cp

R-Square

Using the training observations data, the R-Square model summary is as follows:

Automated Variable Selection – Validation R Square

Forward

Using the training observations data, the Forward model summary is as follows:

Automated Variable Selection – Validation Forward

Backward

Using the training observations data, the Backward model summary is as follows:

Automated Variable Selection – Validation Backward

Stepwise

Using the training observations data, the Stepward model summary is as follows:

Automated Variable Selection – Validation Stepwise

 When comparing the training data set to the full data set, a number of Goodness-of-Fit statistics change including:

• Root MSE training data increases from an average of 37,000 to 39,053 over the full data which is a 5.26% increase. While this is relatively small increase, we prefer a smaller Root MSE, using the training data decreases the fit of our model.
• Cp also increased from an average value of 20.598 to 21.61 which is also a relatively small increase of 4.7% over the full data. Again, we prefer to have a smaller Cp value.
• Small difference in the Adjusted R-Square moving from .802 to .79 in the training vs. full data set respectively which is a decrease of 1.07%. Unlike the Root MSE and Cp, we prefer a higher Adjusted R-Square value.
• The F Value moved from an average of 458 in the full data to an average of 327 in the training set which is a 39.9% change. A larger F value is preferred and the change in the number of observations significantly changes the indications in the fit of our model.
• AIC range in the full model had an average of 50,936 but declined to an average of 35,377 in the training sample which is a decline of almost 44%. This is expected due to the number of observations (n) being included in the formula and thus, as the number declines from the full data set to the training set, it is natural for the AIC volume to decline.
• AIC equation
• BIC range in the full model had an average of 50,936.31 which also declined to an average of 35,380 in the training sample which is also a decline of 44% this is expected as the BIC formula is based on the number of observations (n) thus, as the number of observations decline from the training data from the full data, the BIC statistic would also decline.
• BIC equation:

• There was also a decrease in the number of parameters used with the average in the full model of 23.67 and only 21.33 in the training sample which is approximately a 10.9% decrease.

Overall, the training data versus the full dataset have some minor differences in the majority of the goodness-of-fit statistics.  This is expected as the training set data is not going to be as accurate as utilizing all of the all the data but as long as the jumps are not significantly different then we know the model is performing well.  As described above, the only significant jumps were in the AIC and BIC statistics which is a natural reflection of the number of observations.  Thus, our model is performing well.

Step 9:  Summary of Results and Comparisons

Assigning The average squared error (MSE) and the average absolute error (MAE) for the test sample and the validation sample are as follows:

In all of the cases above, the in-sample data performs worse than the out-of-sample data which is no surprise as we would expect the training set to be a bit worse than the test data.  For the average squared error (MSE) the in-sample data was on average approx. 75.7% of the full data MSE.  Additionally, the average absolute error (MAE) on the in-sample data was on average 96% of the out-of-sample data.

Additionally, evaluating the Average Mean Square error the lowest in-sample model was the Cp model whereas the best MSE out of sample was the Forward model.  With the Average Absolute Error, the best in-model was again the Cp model but the best out-of-sample was again the Forward model.

Step 10:  Operational Validation

To change the format of “Prediction_Grade” in SAS we add in a format command as follows:

if yhat >= (SalePrice *0.9) and yhat <= (SalePrice *1.1) then

                Prediction_Grade=”Grade 1″;

                FORMAT Prediction_Grade $7.;

We can verify that it works in the output data variable overview:

From a business perspective, the models from both the training and test set predict house sale price within various levels of frequency and error as follows:

Hence, based on the training set, 430 or 48.26% house sale price we are able to predict within plus/minus 10% error, 162 or 18.18% of the houses sale price we are able to predict between, but not equal to, 10% – 15% error and 299 houses or 33.56% of housing sale prices we are able to predict with more than 15% error.

Similarly, based on the test set, 1,000 or 49.04% house sale price we are able to predict within plus/minus 10% error, 335 or 16.43% of the houses sale price we are able to predict between 10%-15% error and 704 houses or 34.53% of housing sale prices we are able to predict with more than 15% error.

Thus, depending on the expected deliverables of the model, we will be able to determine if the model is complete or whether further reiterations are required.

Step 11: Final Model

Based on step 9, the Forward model is the best model.  However, we did incorporate categorical variables and thus, need to ensure all categorical variables including the RoofStyleCategory# are added back into the model and then applied to the test data.  The results are as follows:

Sale Price = $341,417 + 22,972 (BldgCategory1) – 3,971.02 (BldgCategory3) + 2,936.69 (BldgCategory4) – 1,623.80 (RoofStyleCategory1) + 5,975.39 (RoofStyleCategory2) +.27286 (LotArea) + 445.67 (YearBuilt) + 486.79 (YearRemodel) + 37.29 (BsmtFinSF1) + 24.84 (BsmtFinSF2) + 24.04 (BsmtUnfSF) + 40.82 (FirstFlrSF) + 42.25 (SecondFlrSF) + 25.12 (GrLivArea) + 41.28 (MasVnrArea) – 12,235 (BedroomAbvGr) + 3,105.88 (TotRmsAbvGrd) + 22.67 (WoodDeckSF) + 40.37 (GarageArea) + 23.57 (TotalPorchSF) + 6,582.18 (TotalFullBath) – 567.88 (TotalBath) – 54.71 (PoolArea) – 10.80 (MiscVal) + 36.08 (MoSold) – 1,087.09 (YrSold)

Step 12:  Reflection/Conclusion

In conclusion, the housing data from Ames, has been utilized to develop numerous regression models to determine the mean sales price of a house based on numerous variables. The variables range from highly correlated, continuous variables to categorical variables with low correlations. In this installment, we continued building the model by assigning dummy codes to categorical variables and the use of automated variable selection procedures.  A validation framework was conducted and the automated variable selected variables were compared using the training sample versus the test data.  Operational validation based on the values were compared and analyzed.  The final and best model was also provided along with the results.

Understanding the goals of the model are key in determining if they should be employed and to what degree. Additionally, the use of these techniques is more of an art than a science. If the model already meets the needs of its purpose than perhaps these methods are not required. However, if the model is not appropriate than utilizing these techniques to transform the model to become more linear is appropriate.

Employing a process to help determine which observations are outliers and requiring an evaluation can to see if it is valid or if it is an error. If an observation is an error, then it should be removed. However, removing legitimate observations will distort the model and should not be removed. If done correctly, the transformation and deletion of outliers will benefit the model by creating a better, more appropriate model that is more linear. However, it does require additional processes and a certain finesse to determine the appropriate thresholds values. As easily observed from the above discussion, the choice of thresholds can drastically impact the number of observations it impacts and needs to be carefully selected and employed.

As reflected in this iteration in the development of the regression model based on the Ames housing data, we came up with the best model however, additional steps can still be taken. Outliers still exist and need to be further investigated to determine whether the observations are legitimate or not and should be discarded.  This process while potentially time consuming, could provide a lot of insight that may or may not be very valuable.  This additional insight may assist in uncovering other variables that may reduce the 20% of variability that is still not accounted for.   Violation of the assumptions of constant variance and normality as seen in the plots still exists.  Additionally, transformation on the data can be utilized but for simplicity and ease of understanding of potential home buyers has been left in its original dollar values.  However, transforming variables will assist in an increase in the goodness of fit and potentially creating a better fitting model.  Thus, if the intended audience is more sophisticated and well versed in interpreting log values, than I would recommend transforming the variables.  If not, and the current model meets the desired audience and requirements than an outlier investigation as detailed above should be undertaken.