<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/performing-principal-components-regression-pcr-in-r/"></div></div><div><a data-url="http://www.r-bloggers.com/performing-principal-components-regression-pcr-in-r/" data-counturl="http://www.r-bloggers.com/performing-principal-components-regression-pcr-in-r/" data-count="horizontal" data-text="Performing Principal Components Regression (PCR) in R" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.milanor.net/blog/performing-principal-components-regression-pcr-in-r/"> MilanoR</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p><em>This article was originally posted on Quantide blog – <a target="_blank" href="http://www.quantide.com/website/performing-principal-components-regression-pcr-in-r/">see here</a>.</em></p>
<p>Principal components regression (<strong>PCR</strong>) is a regression technique based on principal component analysis (<strong>PCA</strong>).</p>
<p>The basic idea behind <strong>PCR</strong> is to calculate the <strong>principal components</strong> and then use some of these components as predictors in a linear regression model fitted using the typical least squares procedure.</p>
<p>As you can easily notice, the core idea of PCR is very closely related to the one underlying PCA and the “trick” is very similar. In some cases a small number of principal components are enough to explain the vast majority of the variability in the data. For instance, say you have a dataset of 50 variables that you would like to use to predict a single variable. By using PCR you might found out that 4 or 5 principal components are enough to explain 90% of the variance of your data. In this case, you might be better off running PCR on with these 5 components instead of running a linear model on all the 50 variables. This is a rough example but I hope it helped to get the point through.</p>
<p>A core assumption of <strong>PCR</strong> is that the directions in which the predictors show the most variation are the exact directions associated with the response variable. On one hand, this assumption is not guaranteed to hold 100% of the times, however, even though the assumption is not completely true it can be a good approximation and yield interesting results.</p>
<p>Some of the most notable advantages of performing PCR are the following:</p>
<ul>
<li>Dimensionality reduction</li>
<li>Avoidance of multicollinearity between predictors</li>
<li>Overfitting mitigation</li>
</ul>
<p>Let’s briefly walk through each one of them:</p>
<div>
<h2>Dimensionality reduction</h2>
<p>By using PCR you can easily perform dimensionality reduction on a high dimensional dataset and then fit a linear regression model to a smaller set of variables, while at the same time keep most of the variability of the original predictors. Since the use of only some of the principal components reduces the number of variables in the model, this can help in reducing the model complexity, which is always a plus. In case you need a lot of principal components to explain most of the variability in your data, say roughly as many principal components as the number of variables in your dataset, then PCR might not perform that well in that scenario, it might even be worse than plain vanilla linear regression.</p>
<p>PCR tends to perform well when the first principal components are enough to explain most of the variation in the predictors.</p>
</div>
<div>
<h2>Avoiding multicollinearity</h2>
<p>A significant benefit of PCR is that by using the principal components, if there is some degree of multicollinearity between the variables in your dataset, this procedure should be able to avoid this problem since performing PCA on the raw data produces linear combinations of the predictors that are uncorrelated.</p>
</div>
<div>
<h2>Overfitting mitigation</h2>
<p>If all the assumptions underlying PCR hold, then fitting a least squares model to the principal components will lead to better results than fitting a least squares model to the original data since most of the variation and information related to the dependent variable is condensend in the principal components and by estimating less coefficients you can reduce the risk of overfitting.</p>
</div>
<div>
<h2>Potential drawbacks and warnings</h2>
<p>As always with potential benefits come potential risks and drawbacks.</p>
<p>For instance, a typical mistake is to consider PCR a feature selection method. PCR is not a feature selection method because each of the calculated principal components is a linear combination of the original variables. Using principal components instead of the actual features can make it harder to explain what is affecting what.</p>
<p>Another major drawback of PCR is that the directions that best represent each predictor are obtained in an unsupervised way. The dependent variable is not used to identify each principal component direction. This essentially means that it is not certain that the directions found will be the optimal directions to use when making predictions on the dependent variable.</p>
</div>
<div>
<h2>Performing PCR on a test dataset</h2>
<p>There are a bunch of packages that perform PCR however, in my opinion, the <strong>pls package</strong> offers the easiest option. It is very user friendly and furthermore it can perform data standardization too. Let’s make a test.</p>
<p>Before performing PCR, it is preferable to standardize your data. This step is not necessary but strongly suggested since PCA is not scale invariant. You might ask why is it important that each predictor is on the same scale as the others. The scaling will prevent the algorithm to be skewed towards predictors that are dominant in absolute scale but perhaps not so relevant as others. In other words, variables with higher variance will influence more the calculation of the principal components and overall have a larger effect on the final results of the algorithm. Personally I would prefer to standardize the data most of the times.</p>
<p>Another thing to assess before running PCR is missing data: you should remove all the observations containing missing data, or approximate the missing observations with some technique before running the PCR function.</p>
<p>For this toy example, I am using the evergreen <em>iris</em> dataset.</p>
<pre>require(pls)
set.seed (1000)

pcr_model &lt;- pcr(Sepal.Length~., data = iris, scale = TRUE, validation = &quot;CV&quot;)</pre>
<p>By setting the parameter <em>scale</em> equal to <em>TRUE</em> the data is standardized before running the pcr algorithm on it. You can also perform validation by setting the argument <em>validation</em>. In this case I chose to perform 10 fold cross-validation and therefore set the <em>validation</em> argument to “CV”, however there other validation methods available just type <em>?pcr</em> in the R command window to gather some more information on the parameters of the <em>pcr</em> function.</p>
<p>In oder to print out the results, simply use the <em>summary</em> function as below</p>
<pre>summary(pcr_model)</pre>

<pre>## Data:    X dimension: 150 5 
##  Y dimension: 150 1
## Fit method: svdpc
## Number of components considered: 5
## 
## VALIDATION: RMSEP
## Cross-validated using 10 random segments.
##        (Intercept)  1 comps  2 comps  3 comps  4 comps  5 comps
## CV          0.8308   0.5141   0.5098   0.3947   0.3309   0.3164
## adjCV       0.8308   0.5136   0.5092   0.3941   0.3303   0.3156
## 
## TRAINING: % variance explained
##               1 comps  2 comps  3 comps  4 comps  5 comps
## X               56.20    88.62    99.07    99.73   100.00
## Sepal.Length    62.71    63.58    78.44    84.95    86.73</pre>
<p>As you can see, two main results are printed, namely the <strong>validation error</strong> and the <strong>cumulative percentage of variance explained</strong> using <span><span><span><span><span>n</span></span></span></span></span> components.</p>
<p>The <strong>cross validation</strong> results are computed for each number of components used so that you can easily check the score with a particular number of components without trying each combination on your own.</p>
<p>The pls package provides also a set of methods to plot the results of PCR. For example you can plot the results of cross validation using the <strong>validationplot</strong> function.</p>
<p>By default, the pcr function computes the root mean squared error and the <strong>validationplot</strong> function plots this statistic, however you can choose to plot the usual mean squared error or the <span><span><span><span><span><span>R</span><span>2</span></span></span></span></span></span> by setting the <strong>val.type</strong> argument equal to “MSEP” or “R2” respectively</p>
<pre># Plot the root mean squared error
validationplot(pcr_model)</pre>
<p><a target="_blank" rel="attachment wp-att-2381" href="http://i2.wp.com/www.milanor.net/blog/wp-content/uploads/2016/07/1_PCR.png"><img sizes="(max-width: 338px) 100vw, 338px" srcset="http://www.milanor.net/blog/wp-content/uploads/2016/07/1_PCR-300x244.png 300w, http://i2.wp.com/www.milanor.net/blog/wp-content/uploads/2016/07/1_PCR.png?resize=338%2C275 443w" alt="1_PCR" class="wp-image-2381" src="http://i2.wp.com/www.milanor.net/blog/wp-content/uploads/2016/07/1_PCR.png?resize=338%2C275" data-recalc-dims="1"></a></p>
<pre># Plot the cross validation MSE
validationplot(pcr_model, val.type=&quot;MSEP&quot;)</pre>
<p><a target="_blank" rel="attachment wp-att-2380" href="http://i0.wp.com/www.milanor.net/blog/wp-content/uploads/2016/07/2_PCR.png"><img sizes="(max-width: 297px) 100vw, 297px" srcset="http://www.milanor.net/blog/wp-content/uploads/2016/07/2_PCR-300x279.png 300w, http://i0.wp.com/www.milanor.net/blog/wp-content/uploads/2016/07/2_PCR.png?resize=297%2C276 424w" alt="2_PCR" class="wp-image-2380" src="http://i0.wp.com/www.milanor.net/blog/wp-content/uploads/2016/07/2_PCR.png?resize=297%2C276" data-recalc-dims="1"></a></p>
<pre># Plot the R2
validationplot(pcr_model, val.type = &quot;R2&quot;)</pre>
<p><a target="_blank" rel="attachment wp-att-2379" href="http://i0.wp.com/www.milanor.net/blog/wp-content/uploads/2016/07/3_PCR.png"><img alt="3_PCR" class="wp-image-2379" src="http://i0.wp.com/www.milanor.net/blog/wp-content/uploads/2016/07/3_PCR.png?resize=277%2C256" data-recalc-dims="1"></a></p>
<p>What you would like to see is a low <strong>cross validation error</strong> with a lower number of components than the number of variables in your dataset. If this is not the case or if the smalles cross validation error occurs with a number of components close to the number of variables in the original data, then no dimensionality reduction occurs. In the example above, it looks like 3 components are enough to explain more than 90% of the <strong>variability</strong> in the data although the CV score is a little higher than with 4 or 5 components. Finally, note that 6 components explain all the variability as expected.</p>
<p>You can plot the predicted vs measured values using the <em>predplot</em> function as below</p>
<pre>predplot(pcr_model)</pre>
<p><a target="_blank" rel="attachment wp-att-2378" href="http://i2.wp.com/www.milanor.net/blog/wp-content/uploads/2016/07/4_PCR.png"><img sizes="(max-width: 285px) 100vw, 285px" srcset="http://www.milanor.net/blog/wp-content/uploads/2016/07/4_PCR-300x296.png 300w, http://i2.wp.com/www.milanor.net/blog/wp-content/uploads/2016/07/4_PCR.png?resize=285%2C281 408w" alt="4_PCR" class="wp-image-2378" src="http://i2.wp.com/www.milanor.net/blog/wp-content/uploads/2016/07/4_PCR.png?resize=285%2C281" data-recalc-dims="1"></a></p>
<p>while the regression coefficients can be plotted using the <em>coefplot</em> function</p>
<pre>coefplot(pcr_model)</pre>
<p><a target="_blank" rel="attachment wp-att-2377" href="http://i2.wp.com/www.milanor.net/blog/wp-content/uploads/2016/07/5_PCR.png"><img alt="5_PCR" class="wp-image-2377" src="http://i2.wp.com/www.milanor.net/blog/wp-content/uploads/2016/07/5_PCR.png?resize=287%2C263" data-recalc-dims="1"></a></p>
<p>Now you can try to use PCR on a traning-test set and evaluate its performance using, for example, using only 3 components.</p>
<pre># Train-test split
train &lt;- iris[1:120,]
y_test &lt;- iris[120:150, 1]
test &lt;- iris[120:150, 2:5]
    
pcr_model &lt;- pcr(Sepal.Length~., data = train,scale =TRUE, validation = &quot;CV&quot;)

pcr_pred &lt;- predict(pcr_model, test, ncomp = 3)
mean((pcr_pred - y_test)^2)</pre>

<pre>## [1] 0.213731</pre>
<p>With the iris dataset there is probably no need to use PCR, in fact, it may even be worse using it. However, I hope this toy example was useful to introduce this model.</p>
<p>Thank you for reading this article, please feel free to leave a comment if you have any questions or suggestions and share the post with others if you find it useful.</p>
</div>
</div>
<p>The post <a target="_blank" rel="nofollow" href="http://www.milanor.net/blog/performing-principal-components-regression-pcr-in-r/">Performing Principal Components Regression (PCR) in R</a> appeared first on <a target="_blank" rel="nofollow" href="http://www.milanor.net/blog">MilanoR</a>.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/performing-principal-components-regression-pcr-in-r/"></div></div><div><a data-url="http://www.r-bloggers.com/performing-principal-components-regression-pcr-in-r/" data-counturl="http://www.r-bloggers.com/performing-principal-components-regression-pcr-in-r/" data-count="vertical" data-text="Performing Principal Components Regression (PCR) in R" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.milanor.net/blog/performing-principal-components-regression-pcr-in-r/"> MilanoR</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/92lQHyZUgUM"></a></div>