<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/principal-components-regression-in-r-part-3/"></div></div><div><a data-url="http://www.r-bloggers.com/principal-components-regression-in-r-part-3/" data-counturl="http://www.r-bloggers.com/principal-components-regression-in-r-part-3/" data-count="horizontal" data-text="Principal Components Regression in R: Part 3" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2016/05/principal-components-regression-in-r-part-3.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>by John Mount Ph. D.<br>Data Scientist at<a target="_blank" href="http://www.win-vector.com/site/staff/"> Win-Vector LLC</a></p>
<p>In her series on principal components analysis for regression in <a target="_blank" href="https://cran.r-project.org/">R</a>, <a target="_blank" href="http://www.win-vector.com/">Win-Vector LLC</a>'s <a target="_blank" href="http://www.win-vector.com/site/staff/nina-zumel/">Dr. Nina Zumel</a> broke the demonstration down into the following pieces:</p>
<ul>
<li><a target="_blank" href="http://www.win-vector.com/blog/2016/05/pcr_part1_xonly/">Part 1</a>: the proper preparation of data and use of principal components analysis (particularly for supervised learning or regression).</li>
<li><a target="_blank" href="http://www.win-vector.com/blog/2016/05/pcr_part2_yaware/">Part 2</a>: the introduction of <em>y</em>-aware scaling to direct the principal components analysis to preserve variation correlated with the outcome we are trying to predict.</li>
<li>And now <a target="_blank" href="http://www.win-vector.com/blog/2016/05/pcr_part3_pickk/">Part 3</a>: how to pick the number of components to retain for analysis.</li>
</ul>
<p>In the earlier parts Dr. Zumel demonstrates common poor practice versus best practice and quantifies the degree of available improvement. In part 3, she moves from the usual &quot;pick the number of components by eyeballing it&quot; non-advice and teaches decisive decision procedures. For picking the number of components to retain for analysis there are a number of standard techniques in the literature including:</p>
<ul>
<li>Pick 2, as that is all you can legibly graph.</li>
<li>Pick enough to cover some fixed fraction of the variation (say 95%).</li>
<li>(for variance scaled data only) Retain components with singular values at least 1.0.</li>
<li>Look for a &quot;knee in the curve&quot; (the curve being the plot of the singular value magnitudes).</li>
<li>Perform a statistical test to see which singular values are larger than we would expect from an appropriate null hypothesis or noise process.</li>
</ul>
<p>Dr. Zumel shows that the last method (designing a formal statistical test) is particularly easy to encode as a permutation test in the <em>y</em>-aware setting (there is also an obvious similarly good bootstrap test). This is well-founded and pretty much state of the art. It is also a great example of why to use a scriptable analysis platform (such as R) as it is easy to wrap arbitrarily complex methods into functions and then directly perform empirical tests on these methods. The following &quot;broken stick&quot; type test yields the following graph which identifies five principal components as being significant:</p>
<p><img title="replot-1.png" alt="Replot 1" src="http://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2016/05/replot-1.png?resize=456%2C456" border="0" data-recalc-dims="1"> However, Dr. Zumel goes on to show that in a supervised learning or regression setting we can further exploit the structure of the problem and replace the traditional component magnitude tests with simple model fit significance pruning. The significance method in this case gets the stronger result of finding the two principal components that encode the known even and odd loadings of the example problem:</p>
<p><img title="plotsig-1.png" alt="Plotsig 1" src="http://i1.wp.com/www.win-vector.com/blog/wp-content/uploads/2016/05/plotsig-1.png?resize=456%2C456" border="0" data-recalc-dims="1"> In fact that is sort of her point: significance pruning either on the original variables or on the derived latent components is enough to give us the right answer. In general, we get much better results when (in a supervised learning or regression situation) we use knowledge of the dependent variable (the &quot;<em>y</em>&quot; or outcome) and do <em>all</em> of the following:</p>
<ul>
<li>Fit model and significance prune incoming variables.</li>
<li>Convert incoming variables into consistent response units by <em>y</em>-aware scaling.</li>
<li>Fit model and significance prune resulting latent components.</li>
</ul>
<p>The above will become much clearer and much more specific if you <a target="_blank" href="http://www.win-vector.com/blog/2016/05/pcr_part3_pickk/">click here to read part 3</a>.</p>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/principal-components-regression-in-r-part-3/"></div></div><div><a data-url="http://www.r-bloggers.com/principal-components-regression-in-r-part-3/" data-counturl="http://www.r-bloggers.com/principal-components-regression-in-r-part-3/" data-count="vertical" data-text="Principal Components Regression in R: Part 3" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2016/05/principal-components-regression-in-r-part-3.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/dse9Fyp-Soo"></a></div>