Cluster Analysis on Transformed Predictor Variables

Cluster analysis is grouping a set of objects in a way that objects in the same group are more similar in some sense to each other than those in other groups.  Clusters are identified by assessing the relative distances between points, the relative homogeneity of each cluster and the degree of separation (Everitt, B. S. (2010) Multivariable Modeling and Multivariate Analysis for the Behavioral Sciences.  p. 241).  In this analysis of European employment data, an initial correlation analysis will be conducted to determine the highest correlated variables along with a scatterplot matrix to determine if any clusters can be detected.  Next, a principal component analysis will be conducted on the dataset containing nine variables along with a threshold of 80% explained variability to determine if a reduction of dimensionality can be attained along with the resulting number of components.  In the last step, a cluster analysis will be conducted to determine the number of clusters within our European Union regions along with an analysis based on three statistical measures including the Cubic Clustering Criterion, Pseudo F and Psudeo T- Squared.  For comparison purposes, scatterplots of three to five clusters are provided which will be compared to a cluster analysis using the SAS cluster command.  Finally, an analysis based on the two different methods and different cluster sizes will be used to determine the preferred cluster method and size.

Step 1: Initial Correlation Analysis

Using the dataset, european_employment, the simple statistics and Pearson correlation coefficients provide us with some insight into our data. There are nine variables relating to the different types of industries. Additionally, all of the 30 countries belong to one of three EU groups – EU for the European Union, EFTA for the European Free Trade Association, Eastern for the Eastern European nations and other. Other consists of four countries – Cyprus, Gibraltar, Malta and Turkey which we could rename as the Mediterranean region. Alternatively, they could be assigned to an existing category as follows:

• Cyprus: EU
• Gibraltar: As a dependent territory of the UK which belongs to the EU
• Malta: EU
• Turkey: EFTA

As shown below, there are both positive and negative related correlations.

• AGR: Agriculture
• MIN: mining
• MAN: manufacturing
• PS: Power and water supply
• CON: construction
• SER: services
• FIN: finance
• SPS: Social and personal services
• TC: Transport and communications

The related industries to the abbreviations are:

The Pearson coefficients:

As shown above, the highest correlation with a p value of <.0001 is between AGR and SPS with a -.81148 correlation followed by -.67193 between MAN and MIN.  The related matrix of scatter plots are shown below where we are searching for clusters or groups of data points.  The only plot that seems to have two or three distinct groups of data is the SER and FIN plot.

cluster analysis scatter plot

The scatterplot for SPS and AGR is shown below (left) and for comparison purposes also for MAN and MIN (below right).  The differences are very apparent.  Albania is clearly an outlier but the plots also don’t seem to see any groupings based on the groupings of EU, EFTA, Eastern and Other.

Cluster Analysis Scatterplot – two variables

Cluster Analysis Scatterplot – two variables

Step 2:  Principal Components Analysis & Reduction of Dimensionality

The eigenvectors are the ‘axis’ along which a linear transformation is stretched, compressed or flipped the eigenvalues give you the factors of the compression.  The scoring coefficients as eigenvectors creates components that have variances equal to the eigenvalues as follows:

The principal component score can be calculated using the eigenvectors to create the formula.  For example, AGR formula would be:

(-.511492 x AGR) + (.023475 x MIN) + (-.278591 x MAN) + (.016492 x PS) + (-.024038 x CON) + (.042397 x SER) + (-.163574 x FIN) + (.540409 x SPS) + (.582036 x TC)

Reducing the number of dimensions through principal component analysis with a 80%variability threshold can be determined using the eigenvalues of the correlation matrix, specifically the cumulative column, and the scree plot.

When there is a high correlation you will be able to do significant dimension reduction with usually the first two values accounting for 80% of the variance.  However, the above correlation matrix reflects that there is not a strong principal component as the cumulative values within the first two components is only .5468 data correlation.  If we want to explain 80% of the variation, four components will be required.

The scree plot and variance explained plots:

Cluster Analysis Scree Plot

Based on a threshold of 80%, four principal components will explain over 80% of the variability.  Additionally, if the 80% threshold is to low, adding one more principal component for a total of five would result in explaining over 90% of the variability.  The scree plot would be another alternative but looking at the elbows in the line.  A third method of determining the number of principal components to use is by evaluating the number of eigenvalues that exceed 1.0 in the correlation matrix.  As shown above, the matrix has four values (3.11, 1.809, 1.49, 1.06) that exceed one and as such would also result in four principal components.

Cluster Analysis Component Patterns

While it is difficult to determine which of the components is the most horizontal if you look closely you can see that component 9 (a red line has been added for easy of viewing) has the minimum depth on the valleys and hills indicating that it is the component with a high correlation across all of the variables.

Cluster Analysis Component Scores

The principal component scores scatter plot of the second and third components displays density with color based on the first component and moving from blue with the minimum density to pink with medium density and red with maximum density.

The below reflects two components accounting for 54.68% of the variability (component 1 with 34.58% and component 2 with 20.1%).  A two-dimensional scatterplot allows us to view the nine dimensional clusters a lot easier and how than say a three-dimension plot along with the different clusters and their strength of relatedness.

Cluster Analysis Component Pattern

Step 3:  Cluster Analysis

The scatterplots of FIN and SER where there seems to be three clusters.

Cluster Analysis Grouping

Comparing to MAN and SER there appears to be four clusters.

Cluster Analysis Grouping

As we can see from the two scatterplots between finance (FIN) and services (SER) and manufacturing (MAN) and services (SER), different projections of the data produce different clustering that we need to keep in mind.

The cluster analysis output from SAS shows a hierarchical cluster with 30 countries starting from the bottom and moving upwards.  It clusters the countries by their distance with the semi partial R-Square reflecting the loss of homogeneity due to combing groups to form a cluster with the r-squared measuring the extent to which the clusters are different from each other.

Cluster Analysis Table CCC Pseudo F Psedo t squared

Graphically, the cubic clustering criteria, pseudo F and Pseudo t-squared with noted peaks:

Cluster Analysis Criteria for Number of Clusters

The Cubic Clustering Criterion (CCC) compares the R-Squared with a specific number of clusters versus the R-Squared attained by clustering a uniformly distributed set of points.  Thus, it can be interpreted just like the R-Squared statistic.  The larger number indicates that there are clusters The CCC above reflects a local peak at three and the maximum peak at five clusters

Pseudo F is similar to r-squared and provides a measure of how separated the clusters are with the ratio between cluster variance to within cluster variance.  The larger the number indicates the more significant and the better the cluster.   The graphic reflects the change in ratio moving from 40 at three clusters to 50 at five clusters.

The Pseudo T-Squared is an index that quantifies the difference in the ratio of between cluster variance to within cluster variances once the clusters are merged (Simmons, R.  (2015).  ResearchGate). In the table, the smaller the number the better.  In the plot, the optimal number of clusters is indicated by moving right to left at the upsurges and adding one to attain the optimal number of clusters.  In the above we have two surges thus, the optimal number of clusters would be at five (four plus one) or at three (two plus one).

Based on the above CCC, Pseudo F and Pseudo T-Squared, five is the most appropriate number of clusters.

The scatterplots and trees will provide some additional insight into the number of clusters starting with three clusters and moving to five clusters.

Cluster Analysis Average Distance between Cluster

Moving from left (root) to the right, R-Squared approaches one and the clusters of countries are progressively joined to form larger clusters at the various numerical levels.  For example, Germany, France, Norway, Ireland and Belgium form a cluster at approximately at the .2 level.

Viewed vertically:

Three Clusters

Viewing three clusters comprises of two clusters and Albania being its own cluster.

Cluster Analysis Three Clusters

Four Clusters

An improvement over the three clusters plus Albania again being its own distinct cluster.

Cluster Analysis Four Variables

Five Clusters

Finally, five clusters with Albania still remaining to be its own clusters.

Cluster Analysis Five Variables

As the tables above demonstrate, the number of countries in each European cluster remain the same their composition and resulting summation changes between the different clusters except for Albania.

Additionally, while the membership of each group changed as the volume increased, the three clusters scatterplot seemed to lack sufficient detail.  Five clusters seem to have the right amount of detail with the four clusters and a fifth one for Albania.

Step 4:  Cluster Analysis using Cluster Commands

Cluster analysis using the cluster command in SAS provides an additional scatterplots and related clusters to compare to.

Three Clusters

The three cluster has two outliers which is very different from the prior three cluster without using the SAS cluster command.

Cluster Analysis Scatterplot

Four Clusters

Two main clusters and two outlier clusters are reflected in the scatterplot.  However, the two main clusters seem a bit disjointed.

Cluster Analysis – Four Clusters

Five Clusters

The five clusters are not as disjointed as the four clusters.

Cluster Analysis – Five Cluster

Six Clusters

Finally, the six clusters seem to have a bit too many clusters with the splitting of CL5 from the five-cluster scatterplot into two different clusters.

Cluster Analysis – Six Clusters

Based on the cluster analysis using cluster commands the three and four clusters don’t seem to produce enough details as one cluster is Albania.  On the five clusters, we have a nice balance of detail between the different geographic areas and in the six clusters, we have a bit too much details with the separation of Malta and Yugoslavia into a new cluster which doesn’t add much value.  None of the clusters are aligned with the European Union geographical areas.

Overall, the SAS cluster command’s results were not very effective in determining the clusters with the clusters in step three being more effective and logical to both interpret and understand.

Conclusion

In conclusion, the European employment data was used to perform a cluster analysis beginning with an exploratory data analysis based on 30 different countries.  A scatterplot matrix was examined to determine if there were any obvious clusters along with a correlation analysis and scatterplots of two correlated variables, agriculture (AGR) and social and personal services industries (SPS) within areas in the European Union.  While the correlations were not very strong, a scatterplot provided an effective method to view the differences.

A principal component analysis based on eigenvectors and their resulting values along with a scree and variance explained plot were also utilized to determine the appropriate number of principal components.  Based on a variance explained threshold of 80%, five principal components were determined to be the number of factors required.  While we would prefer to have fewer factors to explain the variability, reducing the original nine factors to five factors is not a significant reduction and in reality, a decision to use all of the nine factors should be considered.

A cluster analysis based on the number of clusters within four European Union regions along with their associated statistical measures including the Cubic Clustering Criterion, Pseudo F and Pseudo T- Squared were also analyzed.  Scatterplots of clusters ranging from three to five in size were reviewed with a determination that five clusters were the most appropriate.

Finally, a similar comparison was conducted but ranging from three to six clusters and utilizing the SAS cluster command.  As the SAS cluster command resulted in having a unique cluster for Albania and another for Gibraltar, the SAS cluster command was not very effective in producing meaningful clusters.  As such, our preferred cluster method was the earlier method, not utilizing the SAS cluster command, and resulted in five clusters as it grouped the points close together with a meaningful way to reflect the data between the countries and the clusters.

References

Everitt, B. S. (2010) Multivariable Modeling and Multivariate Analysis for the Behavioral Sciences.  Boca Raton, FF:  CRC Press.

Simmons, R. (2015, June 28).  ResearchGate: “Could someone help me decide the ideal no. of clusters from the pseudo t squared graph in SAS?”.  Retrieved from Research Gate.

SAS Code

%let path=/sscc/home/c/ctb2523/410data/;
libname orion “&path”;

data temp; set orion.european_employment;
proc contents; run;

* Step 1: Produce the scatterplot matrix;
ods graphics on;
title Correlation Structure of the Raw Data;
proc corr data=temp plot=matrix(histogram nvar=all);
run; quit;
title ;
ods graphics off;

ods graphics on;
proc sgplot data=temp;
title ‘Scatterplot of Raw Data’;
scatter y=AGR x=SPS / datalabel=country group=group; run;
ods graphics off;

** Step 2;
ods graphics on;
title Principal Components Analysis using PROC PRINCOMP;
proc princomp data=temp out=pca_9components outstat=eigenvectors plots=all; run;
ods graphics off;

** Step 3;
ods graphics on;
proc cluster data=temp method=average outtree=tree1 pseudo ccc plots=all; var fin ser;
id country; run; quit;
ods graphics off;

ods graphics on;
proc sgplot data=temp;
title ‘Scatterplot of FIN and SER’;
scatter y=FIN x=SER / datalabel=country group=group; run;
ods graphics off;

ods graphics on;
proc sgplot data=temp;
title ‘Scatterplot of MAN and SER’;
scatter y=MAN x=SER / datalabel=country group=group; run;
ods graphics off;

**Step 4

ods graphics on;
title ”;
proc tree data=tree1 ncl=5 out=_5_clusters; copy man ser;
run; quit;
ods graphics off;
ods graphics on;
proc tree data=tree1 ncl=4 out=_4_clusters; copy man ser;
run; quit;
ods graphics off;
ods graphics on;
proc tree data=tree1 ncl=3 out=_3_clusters; copy man ser;
run; quit;
ods graphics off;

%macro makeTable(treeout,group,outdata); data tree_data;
set &treeout.(rename=(_name_=country));
run;
proc sort data=tree_data; by country; run; quit; data group_affiliation;
set &group.(keep=group country);
run;
proc sort data=group_affiliation; by country; run; quit; data &outdata.;
merge tree_data group_affiliation; by country;
run;
proc freq data=&outdata.;
table group*clusname / nopercent norow nocol; run;
%mend makeTable;

* Call macro function;
%makeTable(treeout=_3_clusters,group=temp,outdata=_3_clusters_with_labels);
* Plot the clusters for a visual display; ods graphics on;
proc sgplot data=_3_clusters_with_labels;
title ‘Scatterplot of Raw Data’;
scatter y=man x=ser / datalabel=country group=clusname; run; quit;
ods graphics off;
%makeTable(treeout=_4_clusters,group=temp,outdata=_4_clusters_with_labels);
* Plot the clusters for a visual display; ods graphics on;
proc sgplot data=_4_clusters_with_labels;
title ‘Scatterplot of Raw Data’;
scatter y=man x=ser / datalabel=country group=clusname; run; quit;
ods graphics off;
%makeTable(treeout=_5_clusters,group=temp,outdata=_5_clusters_with_labels);
* Plot the clusters for a visual display; ods graphics on;
proc sgplot data=_5_clusters_with_labels;
title ‘Scatterplot of Raw Data’;
scatter y=man x=ser / datalabel=country group=clusname; run; quit;
***Cluster with all varibles;
ods graphics on;
proc cluster data=temp method=average outtree=tree1 pseudo ccc plots=all;
var agr min man ps con ser fin sps tc;
id country; run; quit;
ods graphics off;

ods graphics on;
proc tree data=tree1 ncl=5 out=_5_clusters;
copy agr min man ps con ser fin sps tc;
run; quit;
ods graphics off;
ods graphics on;
proc tree data=tree1 ncl=4 out=_4_clusters;
copy agr min man ps con ser fin sps tc;
run; quit;
ods graphics off;
ods graphics on;
proc tree data=tree1 ncl=3 out=_3_clusters;
copy agr min man ps con ser fin sps tc;
run; quit;
ods graphics off;

%macro makeTable(treeout,group,outdata); data tree_data;
set &treeout.(rename=(_name_=country));
run;
proc sort data=tree_data; by country; run; quit; data group_affiliation;
set &group.(keep=group country);
run;
proc sort data=group_affiliation; by country; run; quit; data &outdata.;
merge tree_data group_affiliation; by country;
run;
proc freq data=&outdata.;
table group*clusname / nopercent norow nocol; run;
%mend makeTable;

* Call macro function;
%makeTable(treeout=_3_clusters,group=temp,outdata=_3_clusters_with_labels);
* Plot the clusters for a visual display; ods graphics on;
proc sgplot data=_3_clusters_with_labels;
title ‘Scatterplot of Raw Data’;
scatter y=fin x=ser / datalabel=country group=clusname; run; quit;
ods graphics off;
%makeTable(treeout=_4_clusters,group=temp,outdata=_4_clusters_with_labels);
* Plot the clusters for a visual display; ods graphics on;
proc sgplot data=_4_clusters_with_labels;
title ‘Scatterplot of Raw Data’;
scatter y=fin x=ser / datalabel=country group=clusname; run; quit;
ods graphics off;
%makeTable(treeout=_5_clusters,group=temp,outdata=_5_clusters_with_labels);
* Plot the clusters for a visual display; ods graphics on;
proc sgplot data=_5_clusters_with_labels;
title ‘Scatterplot of Raw Data’;
scatter y=fin x=ser / datalabel=country group=clusname; run; quit;
* Using the first 2 principal components;
ods graphics on;
proc cluster data=pca_9components method=average outtree=tree3 pseudo ccc plots=all;
var prin1 prin2; id country;
run; quit;
ods graphics off;
ods graphics on;
proc tree data=tree3 ncl=6 out=_6_clusters; copy prin1 prin2;
run; quit;
proc tree data=tree3 ncl=5 out=_5_clusters; copy prin1 prin2;
run; quit;
proc tree data=tree3 ncl=4 out=_4_clusters; copy prin1 prin2;
run; quit;
proc tree data=tree3 ncl=3 out=_3_clusters; copy prin1 prin2;
run; quit;
ods graphics off;
%makeTable(treeout=_3_clusters,group=temp,outdata=_3_clusters_with_labels);
%makeTable(treeout=_4_clusters,group=temp,outdata=_4_clusters_with_labels);
%makeTable(treeout=_5_clusters,group=temp,outdata=_5_clusters_with_labels);
%makeTable(treeout=_6_clusters,group=temp,outdata=_6_clusters_with_labels);
* Plot the clusters for a visual display; ods graphics on;

proc sgplot data=_3_clusters_with_labels;
title ‘Scatterplot of Prin Data’;
scatter y=prin2 x=prin1 / datalabel=country group=clusname; run; quit;
ods graphics off;
* Plot the clusters for a visual display; ods graphics on;
proc sgplot data=_4_clusters_with_labels;
title ‘Scatterplot of Prin Data’;
scatter y=prin2 x=prin1 / datalabel=country group=clusname; run; quit;
ods graphics off;
* Plot the clusters for a visual display; ods graphics on;
proc sgplot data=_5_clusters_with_labels;
title ‘Scatterplot of Prin Data’;
scatter y=prin2 x=prin1 / datalabel=country group=clusname; run; quit;
proc sgplot data=_6_clusters_with_labels;
title ‘Scatterplot of Prin Data’;
scatter y=prin2 x=prin1 / datalabel=country group=clusname; run; quit;