Principal Components Analysis

Utilizing a stock portfolio data set and the Principal Components Analysis as a method in reducing dimension and as a remedial measure for multicollinearity in Ordinary Least Squares regression.  Beginning with the data, we will transform the variables into log values to explain the variation in the log-returns of the stocks and market index.  Next, correlations between the individual stocks and the market index will be conducted followed by the transposing the data tables into a format and various plot to visualize the correlations will be shown.  An analysis on the principal components and which components to keep based on our requirements will be applied to our regression model and later, applied to the data which will be split into a training and test set.  An analysis on fitting a regression model to the individual stocks on a goodness-of-fit and determination on the existence of multicollinearity issues will be provided.  Similarly, using eight principal components, a fitted regression model and an analysis on the variance inflation factors will be discussed leading into final determination on the best model.

Part 1:  Data & Transformation of Data

Using the dataset, stock_portfolio we load the data into SAS where we can view the daily closing price for 20 stocks and indexed funds from Vanguard(VV) beginning Jan 3, 2012 to Dec 31, 2013.  New variables are created to normalize the data and calculate the daily return for each stock by using the log function for example, return_AA  = log(AA/lag1(AA)) with lag being the difference of one day between the closing prices. The response or dependent variable, ‘VV’ is the variation in the log-returns of the market index which allows us to explain the variation on the individual stock returns.  If we did transform the stock prices, the stocks would be at their closing dollar value which does not provide a basis for comparison on their daily return as the ‘measuring stick’ would be different for each stock.

Using log-returns of each stock we will explain the variation in the log-returns of the market index using both linear regression and principal components analysis.

The results show 502 observations, 22 variables.  A quick overview:

Part 2:  Correlation

The response variable (VV) is then used with all of the predictors to calculate the PearsonCorr correlation.  The output reflects that all correlations are highly significant with the lowest correlation value at .44350 on return_DPS and the highest correlation of .76838 on return_MMM.  It is testing if the correlation is equal to zero along with the p value reflecting a significance level of  <.0001 and therefore, not equal to zero.  Thus, at the .0001 level we know that the results are not accidental and that the results are not just by chance.

Part 3:  Transposing to a Long Table with Stock Ticker & Correlation

Similar to part 2 reflecting the correlation but in a different format referred to as ‘long’ format with the ticket added as a separate line in SAS:   tkr = substr(_NAME_,8,3); with the 8 is the position with three positions for the ticker symbol. While the correlation values do not change, it is easier to examine all of the values but with the understanding that all of the p values meet our required significance levels.

Part 4:  Adding Sector Information

Similar to Part 3, we are adding information to the table.  In this part, we are adding the industry sector.  SAS code is again, similar to part 3 with “input tkr $ 1-3 sector $ 4-35;”.  VV is the market index whereas all other items are sectors with ticket symbols.  Including the sector codes provides context and insight into the different stocks as we can now see that the lowest correlated stock is DPS but that it is in the soft drink sector but can also scan the list for other ticker symbols within the soft drink sector and see that the ticker symbol PEP also has a similar correlation of .50753 and KO at .59980.  Similarly, the highest correlated stock as noted above was MMM (correlation of .76085) which is in the Manufacturing sector and is similar to another manufacturing stock HON with a correlation of .76838.

While the above table with the sector provides us with a lot more context, visually viewing the results is even easier showing the ticker symbol and value.

Principal Components Analysis – Ticker Correlations

However, in the above vbar chart, matching the sectors to each stock ticker is a not very efficient.  Thus, it would be easier for the reader to view the information with the sectors being grouped together.

Principal Components Analysis – Grouped by Sector

In the above vbar chart, the reader can easily compare the different stock tickers within each industry and get a feel for the different correlations between the stock and the Vanguard index (VV).  For instance, manufacturing correlations are very similar whereas oil refining stocks have the largest range.  However, mentally analyzing the vbar chart, it is difficult to calculate the mean correlation for each sector.

Thus, a table on the different sectors in both a table and vbar are easy to visual and compare the different sectors and with the frequency understand how many stocks are within each sector.

Principal components analysis Mean Sector Correlations

As we can see the highest correlation is Manufacturing with an average of .764 and the lowest being Soft Drinks with a correlation of .516.

Part 5:  Principal Components Analysis

Creating a new dataset by excluding the dependent variable – market index we can calculate the principle components.  Ignoring the response variables and focusing on the 20 predictor values/return values we can reduce the number of dimensions of the 20 variables using the principal component.  Evaluating only the return data.  SAS allows us to quickly match all of our log variables.  We can then use these to calculate the eigenvectors. The number of eigenvectors will match the number of variables.

Note:  the SAS code:

proc princomp data=return_data out=pca_output outstat=eigenvectors plots=scree(unpackpanel);

‘out=’ will have all of the outputs from the Principal Components Analysis whereas ‘outstat’ will only contain specific statistics captured which in this case is eigenvectors. Prin1 needs to be carefully reviewed as the same term is used later with all observations

Prin1 is the weights of the first principal component with the size of the Eigenvalue will be the same number of variables.  The values are the variable weights of all of the variables reflecting a linear combination of the eigenvalues and the cumulative total.

Variance Explained plot shows the number of principle components to explain the and the number of principle components you want to keep to explain the desired level of variance.

Determining the ‘right’ number of components to keep depends on the desired threshold for explaining the variation.  As explained in red text, if the threshold was 80%, eight principal components would be required.  If we wanted to explain 70% of the variation, we would want to keep four principal components and if we wanted to explain 90% of the variation, we would keep 13 of the principal components.  Assuming 80% is our threshold, we will want to keep eight principal components.

Evaluating whether there are clusters of data, a review of the Principal value scores, reflects that they are completely independent of each other.  Thus, the principal component scores can be used as another set of variables without having any multicollinearity issues as there is no correlation.

Graphically, we can view the first two principle component vectors to evaluate whether there is any relationship within the data.  This is usually accomplished by reviewing the graph in a quadrant-by-quadrant analysis.

Similar to the analysis above, there is a cluster of stock tickets in the first quadrant that are from the soft drink industry which are heavily weighted in Prin2 but lightly weighted in Prin1.  Prin1 is mostly in the fourth quadrant is heavily weighted all of the other industries compared to the soft drinks.   No surprise that we see the manufacturing tickers close together.

Part 6:  Regression Model using Training and Test Data Sets

Building a regression model using the original variables and the principal components with the response variable being the index to predict which model is better – the principal component or the raw data.  Split the data into a test and training set with a 70/30 split between the training and test data.  Created a cross validation data.  The table on the right reflects the observations used in the training data set.

Part 7:  Build a Regression Model

Using the Training Data Set:

Using the training data and the test data to build the model. The regression model using the raw predictor variables, the diagnostic summary:

The data reflects 20 predictors are being used with an Adjusted R-Square of .8919 which is good.  The F value of 140.04 will be used to compare against the other models and our p value is still within our threshold at <.0001.

The parameter estimates:

Reviewing the table, it seems weird that our F value is significant yet, the t-values are all non-significant.  Thus, there is a chance that we have multicollinearity taking place.  As such, we evaluate the Variance Inflation Factor (VIF) which measures the dependencies among the regressors.  VIFs values that exceed 5 or 10 are an indication that the associated regression coefficients are poorly estimated due to multicollinearity. With our highest value of 3.36439 which is on the low end of multicollinearity, we do not have an issue.  Some of the p values are also not significant.

Principal Components Analysis – Training Data Set

As we can see above, the data fits the line well in the Q-Q Plot, Cook’s D has low values with a maximum of .15.  Residuals are scattered in a random matter.

Using ALL of the Data:

Using all of the data, the diagnostic information on the second model is as follows:

Similar to the first regression model we still have 20 regressors.  Comparing the goodness-of-fit statistics using all of the data, Adjusted R-Square is very similar with a value of .8768 (vs .8919) and an improvement in the F value to 178.96 (vs. 140.04).

As reflected in the table above, the maximum VIF value is 3.19781 which is a decrease from 3.36439. Also, some of the p values are not significant.  Thus, this regression model appears to be performing very similar to the training model.

Principal Components Analysis

Evaluating the diagnostic plots, everything is very similar to the analysis provided in the first regression model using the training data with a minor change in the scale of the Cook’s D plot with the scale changing from .15 to .125 and a more compressed scale on the Q-Q plot.

Overall, the diagnostic plots look good.  Based on the above, this model does not have multicollinearity issues.  For ease of comparison, the mean square error and mean absolute error will be discussed in part 8.

Part 8:  Build a Regression Model Using Principal Components

Principal component analysis is used to reduce the dimensionality of a data set consisting of a large number of interrelated variables while trying to retain the variation in the data set.  By transforming to a new set of variables, the uncorrelated principal components are ordered so that the first few retain most of the variation present in all of the original variables (Jolliffe, I.T., 2010.  Principal Component Analysis, 2nd edition). Using the top eight principal components to build the model based on the training and test data to determine which model fits better.

Training Dataset:

Fitting a regression model to eight selected principal components, the diagnostic information is as follows:

Comparing the above information to the prior models, we can see that the Adjusted R-Square is similar to the other models (.8919 and .8768) and an increase in the F Value to 337.13 (an increase from 140.04 and 178.96) in the prior two models using all 20 regressors.

From the above table, we see some of the p values/predictors are not significant.  This is especially true in the independent variables four through eight.

Principal Components Analysis – Training Data Eight Components Only

The diagnostic plots are very similar to the prior model.

Test/Full Dataset:

Similar to the above but now applying the eight principal components to the full data set:

In comparison to the test data, the first four p values/predictors are now significant.  The VIF values are all one’s because all the principal components are independent with a correlation of 0, therefore minimum VIF is 1.  A VIF of 1 is ideal as it reflects that the regressors are completely unrelated.

Principal Components Analysis -Full Data Set with Eight Components Only

The above plots are also very similar.  The residual plot has a few outliers, the Q-Q plot fits the line and the Cook’s D is small with a scale maximum of 0.15.

Overall, there are only a few minor differences between the principal components model and the full data.  Additionally, we don’t see any red flags in the diagnostics plots.  The Adjusted R Squares are very similar and the VIF is exactly one.  Based on the above diagnostical information and plots we have no reason to believe there are any multicollinearity issues.

Validation:  Prediction Grades

Model Summaries:

Evaluating the two models, model1 using all of the variables and model2 using only eight principal components, we can make several different comparisons.

Prediction grades:

The prediction grades have three levels with Grade 1 being plus/minus 10% error, Grade 2 being plus/minus 15% error and grade three having a zero error.  Utilizing the prediction grades as shown in the summary table, we can see model 1 based on the test data has a prediction grade of 12.2% of the time within plus/minus 15% error.  Also on model 1 based on the training data, a 14.2% within plus/minus 15% error using all of the variables and finally, using all of the data in model 1, 14.94% with a plus/minus 15% error rate .   Conversely, in model two, 14.63% of the time we are within +/- 15% of error in test set and in the training set, 13.02% of the time we are within +/15% error and 13.75% using all of the data plus/minus 15% error.

Based on our side-by-side comparison, the mean square error and mean average error are performing better in model 1 and using the test data the +/- 15% error is lower but on the training set model 2 is better.  However, neither of the differences are large (12.2% v. 14.63% and 14.2% vs. 13.2%).  Additionally, based on the full data, the mean square error and mean average error are better along with the +/- 15% error.

Thus, we can see that the models are comparable with only a small change of 1.19% (14.94% vs 13.75%) in the prediction grades using the full data.  Compare full test set and training set – are they all in the same ball park as they are very close in value.   Thus, depending on the business requirement if the +/-15% error in the model is acceptable then will be accept this model.

Whether the above is satisfactory is not depend on our business requirements.  If the requirement was 20% of the time to be within +/-15% then we have met the desired threshold.  However, if the threshold is 10% then we have not met the requirement and further models need to be develop.

Mean Square Error and Mean Absolute Error:

Evaluating the mean square error and mean absolute error for model 1 and model 2 is as follows:

As we can see, there is a very small difference in the mean square error in both the test and training data and similarly, in the mean average error.  Thus, the difference is accuracy is very small and not a concern. Also, referring to the summary table, there is a small change in model 1 versus model 2 but is not significant.

Thus, whether we are using the training, test or full data, all of the results are comparable with no large swings between any of the goodness-of-fit indicators.

Based on the above analysis, model 2 is the better model as the eight-principal component model is performing well and is the better choice over the full data model with 20 variables.  For the insignificant difference, the number of parameters and perhaps intertwined with related costs in attaining the data makes better business sense.  Additionally, model 2 is simpler which is preferred.

Conclusion

In conclusion, the stock portfolio data set contained the daily closing price for 20 stocks and indexed funds from Vanguard beginning Jan 3, 2012 to Dec 31, 2013.  These values were transformed into log values for the purpose of explaining the variation in the log-returns of the stocks and market index.  Correlations between the individual stocks and the market index were conducted with various formats of data tables and plots to visualize the correlations between the stocks and industry sectors. Principal components were analyzed to determine which components to keep and were used in a regression model.  Comparing the full data to the data set that was split into a training and test set was also analyzed and contrasted. Additionally, a fitted regression model was analyzed to determine its goodness-of-fit and used to determine if multicollinearity issues existed.  Finally, using eight principal components, a fitted regression model and an analysis on the variance inflation factors were discussed leading into final determination on the best model due to its simplicity which was model 2 the principal component model.

Taking a step back and looking at the forest instead of the trees, we used the stock portfolio data to create a fitted model using all of the regressors.  Later after we performed an analysis on the model to determine if multicollinearity existed and transformed the regressions into log values for ease of comparison, we applied the principal components to explain 80% of the variation by choosing only eight components.  Decreasing the number of components from 20 to 8, simplified the model with minimal loss in the goodness-of-fit statistics.   Whether eight components are the right number depends on the business requirements and the impact on the plus/minus 15% error rate threshold as desired by the company.  If the threshold is not met, further models will need to be completed and if the threshold is met, than no further work is required.