<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/partial-least-squares-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://poissonisfish.wordpress.com/2017/06/17/partial-least-squares-in-r/"> poissonisfish</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<figure data-shortcode="caption" class="wp-caption"><img data-orig-file="https://poissonisfish.files.wordpress.com/2017/06/img_1847_ins_8287_600.jpg?w=456" data-orig-size="600,401" data-medium-file="https://poissonisfish.files.wordpress.com/2017/06/img_1847_ins_8287_600.jpg?w=456?w=300" data-attachment-id="3018" data-image-meta="{" data-image-description="" src="https://poissonisfish.files.wordpress.com/2017/06/img_1847_ins_8287_600.jpg?w=456" data-permalink="https://poissonisfish.wordpress.com/2017/06/17/partial-least-squares-in-r/img_1847_ins_8287_600/" data-large-file="https://poissonisfish.files.wordpress.com/2017/06/img_1847_ins_8287_600.jpg?w=456?w=600" alt="img_1847_ins_8287_600" data-image-title="img_1847_ins_8287_600" sizes="(max-width: 600px) 100vw, 600px" data-comments-opened="1" class="wp-image-3018" data-recalc-dims="1"><br>
<figcaption class="wp-caption-text">Blood serum samples. From: https://health.onehowto.com</figcaption>
</figure>
<p>My <a rel="nofollow" href="https://poissonisfish.wordpress.com/2017/01/23/principal-component-analysis-in-r/" target="_blank">last entry</a> introduces principal component analysis (PCA), one of many unsupervised learning tools. I concluded the post with a demonstration of principal component regression (PCR), which essentially is a ordinary least squares (OLS) fit using the first <img alt="k " title="k " src="https://s0.wp.com/latex.php?latex=k+&bg=f9f9f9&%23038;fg=555555&%23038;s=1" class="latex"> principal components (PCs) from the predictors. This brings about many advantages:</p>
<ol>
<li><strong>There is virtually no limit for the number of predictors.</strong> PCA will perform the decomposition no matter how many variables you handle. Simpler models such as OLS do not cope with more predictors than observations (<em>i.e.</em> <img alt="p &gt; n " title="p &gt; n " src="https://s0.wp.com/latex.php?latex=p+%3E+n+&bg=f9f9f9&%23038;fg=555555&%23038;s=1" class="latex">).</li>
<li><strong>Correlated or noisy predictors do not undermine the regression fit.</strong> Collinearity is a problem for OLS, by widening the solution space, <em>i.e.</em> destabilizing coefficient estimation. PCA will always produce few uncorrelated PCs from a set of variables, correlated or not. On the other hand, and to some extent, uninformative predictors will not affect performance.</li>
<li><strong>The PCs carry the maximum amount of variance possible. </strong>In any predictive model, predictors with zero or near-zero variance often constitute a problem and behave as second intercepts. In the process of compression, PCA will find the projections where your data points spread out the most, thus facilitating learning from the subsequent OLS.</li>
</ol>
<p>However, in many cases it is much wiser performing a decomposition similar to the PCA, yet constructing PCs that best explain the response, be it quantitative (regression) or qualitative (classification). This is the concept of partial least squares (PLS), whose PCs are more often designated latent variables (LVs), although in my understanding the two terms can be used interchangeably.</p>
<p>PLS safeguards advantages 1. and 2. and does not necessarily follow 3. In addition, it follows that if you use the same number of <img alt="k " title="k " src="https://s0.wp.com/latex.php?latex=k+&bg=f9f9f9&%23038;fg=555555&%23038;s=1" class="latex"> LVs and <img alt="p " title="p " src="https://s0.wp.com/latex.php?latex=p+&bg=f9f9f9&%23038;fg=555555&%23038;s=1" class="latex"> predictors, you are going to get exactly a OLS fit – each predictor gets its own LV, so it does not help much. PLS (regression) and PLS followed by discriminant analysis (PLS-DA, classification) are tremendously useful in predictive modelling. They are adequate in a wide variety of experimental designs and linear in their parameters, therefore more easily interpretable.</p>
<p>Today we will perform PLS-DA on the <a rel="nofollow" href="https://archive.ics.uci.edu/ml/datasets/Arcene" target="_blank">Arcene data set</a> hosted at the UCI Machine Learning Repository that comprises 100 observations and 10,000 explanatory variables (<img alt="p gg n " title="p gg n " src="https://s0.wp.com/latex.php?latex=p+%5Cgg+n+&bg=f9f9f9&%23038;fg=555555&%23038;s=1" class="latex">) in order to diagnose cancer from serum samples. From the 10,000 features, 7,000 comprise distinct mass spectrometry (MS) peaks, each determining the levels of a protein. For some reason the contributors added 3,000 random variables, probably to test robustness against noise (check more information in the link above).</p>
<h2>Let’s get started with R</h2>
<p>For predictive modelling I always use the <strong>caret</strong> package, which builds up on existing model packages and employs a single syntax. In addition, <strong>caret</strong> features countless functions that deal with pre-processing, imputation, summaries, plotting and much more we will see firsthand shortly.</p>
<p>For some reason there is an empty column sticking with the data set upon loading, so I have added the <strong>colClasses</strong> argument to get rid of it. Once you load the data set take a look and note that the levels of most proteins are right-skewed, which can be a problem. However, the authors of this study conducted an exhaustive pre-processing and looking into this would take far too long. The cancer / no-cancer labels (coded as -1 / 1) are stored in a different file, so we can directly append it to the full data set and later use the formula syntax for training the models.</p>
<pre>
# Load caret, install if necessary
library(caret)
arcene &lt;- read.table(&quot;http://archive.ics.uci.edu/ml/machine-learning-databases/arcene/ARCENE/arcene_train.data&quot;, sep = &quot; &quot;,
 colClasses = c(rep(&quot;numeric&quot;, 10000), &quot;NULL&quot;))

# Add the labels as an additional column
arcene$class &lt;- factor(scan(&quot;https://archive.ics.uci.edu/ml/machine-learning-databases/arcene/ARCENE/arcene_train.labels&quot;, sep = &quot;\t&quot;))
</pre>
<p>We can finally search missing values (<strong>NA’s</strong>). If these were present it would require some form of imputation – that is not the case. Calling <strong>any(is.na(arcene))</strong> should prompt a <strong>FALSE</strong>.</p>
<p>So now the big questions are:</p>
<ul>
<li><strong>How accurately can we predict whether a patient is sick based of the MS profile of his/her blood serum?</strong></li>
<li><strong>Which proteins / MS peaks best discriminate sick and healthy patients?</strong></li>
</ul>
<p>Time to get started with the function <strong>train</strong>. With respect to pre-processing we will remove zero-variance predictors and center and scale all those remaining, in this exact order using the <strong>preProc</strong> argument. Considering the size of the sample (<em>n</em> = 100), I will pick a 10x repeated 5-fold cross-validation (CV) – a large number of repetitions compensates for the high variance stemming from a reduced number of folds – rendering a total of 50 estimates of accuracy. Many experts advise in favor of simpler models when performance is undistinguishable, so we will apply the “one SE” rule that selects the least complex model with the average cross-validated accuracy within one standard error (SE) from that in the optimal model. Finally, we must define a range of values for the tuning parameter, the number of LVs. In this particular case we will consider all models with 1-20 LVs.</p>
<pre>
# Compile cross-validation settings
set.seed(100)
myfolds &lt;- createMultiFolds(arcene$class, k = 5, times = 10)
control &lt;- trainControl(&quot;repeatedcv&quot;, index = myfolds, selectionFunction = &quot;oneSE&quot;)

# Train PLS model
mod1 &lt;- train(class ~ ., data = arcene,
 method = &quot;pls&quot;,
 metric = &quot;Accuracy&quot;,
 tuneLength = 20,
 trControl = control,
 preProc = c(&quot;zv&quot;,&quot;center&quot;,&quot;scale&quot;))

# Check CV profile
plot(mod1)
</pre>
<p><img data-orig-file="https://poissonisfish.files.wordpress.com/2017/06/rplot.png?w=456" data-orig-size="474,382" data-medium-file="https://poissonisfish.files.wordpress.com/2017/06/rplot.png?w=456?w=300" data-attachment-id="2785" data-image-meta="{" data-image-description="" src="https://poissonisfish.files.wordpress.com/2017/06/rplot.png?w=456" data-permalink="https://poissonisfish.wordpress.com/2017/06/17/partial-least-squares-in-r/rplot-3/" data-large-file="https://poissonisfish.files.wordpress.com/2017/06/rplot.png?w=456?w=474" alt="Rplot" data-image-title="Rplot" sizes="(max-width: 474px) 100vw, 474px" data-comments-opened="1" class="wp-image-2785" data-recalc-dims="1"></p>
<p>This figure depicts the CV profile, where we can learn the average accuracy (<em>y</em>-axis, %) obtained from models trained with different numbers of LVs (<em>x</em>-axis). Note the swift change in accuracy among models with 1-5 LVs. Although the model with six LVs had the highest average accuracy, calling <strong>mod1</strong> in the console will show you that, because of the “one SE” rule, the selected model has five LVs (accuracy of 80.6%).</p>
<p>Now we will compare our PLS-DA to the classifier homolog of PCR  – linear discriminant analysis (LDA) following a PCA reductive-step (PCA-DA, if such thing exists). Note that the PCA pre-processing is also set in the <strong>preProc</strong> argument. We can also try some more complicated models, say, random forests (RF). I will not go into details about the workings and parameterisation of RF, as for today it will be purely illustrative. Note that RF will comparatively take very long (~15min in my 8Gb MacBook Pro from 2012). As usual, a simple object call will show you the summary of the corresponding CV.</p>
<pre>
# PCA-DA
mod2 &lt;- train(class ~ ., data = arcene,
method = &quot;lda&quot;,
metric = &quot;Accuracy&quot;,
trControl = control,
preProc = c(&quot;zv&quot;,&quot;center&quot;,&quot;scale&quot;,&quot;pca&quot;))

# RF
mod3 &lt;- train(class ~ ., data = arcene,
method = &quot;ranger&quot;,
metric = &quot;Accuracy&quot;,
trControl = control,
tuneGrid = data.frame(mtry = seq(10,.5*ncol(arcene),length.out = 6)),
preProc = c(&quot;zv&quot;,&quot;center&quot;,&quot;scale&quot;))
</pre>
<p>Finally, we can compare PLS-DA, PCA-DA and RF with respect to accuracy. We will compile the three models using <strong>caret::resamples</strong>, borrowing the plotting capabilities of <strong>ggplot2 </strong>to compare the 50 accuracy estimates from the optimal cross-validated model in the three cases.</p>
<pre>
# Compile models and compare performance
models &lt;- resamples(list(&quot;PLS-DA&quot; = mod1, &quot;PCA-DA&quot; = mod2, &quot;RF&quot; = mod3))
bwplot(models, metric = &quot;Accuracy&quot;)
</pre>
<p><img data-orig-file="https://poissonisfish.files.wordpress.com/2017/06/rplot1.png?w=456" data-orig-size="474,382" data-medium-file="https://poissonisfish.files.wordpress.com/2017/06/rplot1.png?w=456?w=300" data-attachment-id="2912" data-image-meta="{" data-image-description="" src="https://poissonisfish.files.wordpress.com/2017/06/rplot1.png?w=456" data-permalink="https://poissonisfish.wordpress.com/2017/06/17/partial-least-squares-in-r/rplot-4/" data-large-file="https://poissonisfish.files.wordpress.com/2017/06/rplot1.png?w=456?w=474" alt="Rplot" data-image-title="Rplot" sizes="(max-width: 474px) 100vw, 474px" data-comments-opened="1" class="wp-image-2912" data-recalc-dims="1"></p>
<p>It is clear that the long RF run did not translate into a excelling performance, quite the opposite. Although in average all three models have similar performances, the RF displays a much larger variance in accuracy, which is of course a concern if we seek a robust model. In this case PLS-DA and PCA-DA exhibit the best  performance (63-95% accuracy) and either model would do well in diagnosing cancer in new serum samples.</p>
<p>To conclude, we will determine the ten proteins that best diagnose cancer using the variable importance in the projection (ViP), from both the PLS-DA and PCA-DA. With <strong>varImp</strong>, this is given in relative levels (scaled to the range 0-100).</p>
<pre>
plot(varImp(mod1), 10, main = &quot;PLS-DA&quot;)
plot(varImp(mod2), 10, main = &quot;PCA-DA&quot;)
</pre>
<p><img data-orig-file="https://poissonisfish.files.wordpress.com/2017/06/rplot011.png?w=456" data-orig-size="474,382" data-medium-file="https://poissonisfish.files.wordpress.com/2017/06/rplot011.png?w=456?w=300" data-attachment-id="2914" data-image-meta="{" data-image-description="" src="https://poissonisfish.files.wordpress.com/2017/06/rplot011.png?w=456" data-permalink="https://poissonisfish.wordpress.com/2017/06/17/partial-least-squares-in-r/rplot01-2/" data-large-file="https://poissonisfish.files.wordpress.com/2017/06/rplot011.png?w=456?w=474" alt="Rplot01" data-image-title="Rplot01" sizes="(max-width: 474px) 100vw, 474px" data-comments-opened="1" class="wp-image-2914" data-recalc-dims="1"></p>
<p><img data-orig-file="https://poissonisfish.files.wordpress.com/2017/06/rplot021.png?w=456" data-orig-size="474,382" data-medium-file="https://poissonisfish.files.wordpress.com/2017/06/rplot021.png?w=456?w=300" data-attachment-id="2916" data-image-meta="{" data-image-description="" src="https://poissonisfish.files.wordpress.com/2017/06/rplot021.png?w=456" data-permalink="https://poissonisfish.wordpress.com/2017/06/17/partial-least-squares-in-r/rplot02-3/" data-large-file="https://poissonisfish.files.wordpress.com/2017/06/rplot021.png?w=456?w=474" alt="Rplot02" data-image-title="Rplot02" sizes="(max-width: 474px) 100vw, 474px" data-comments-opened="1" class="wp-image-2916" data-recalc-dims="1"></p>
<p>Very different figures, right? Idiosyncrasies aside, how many PCs did the PCA-DA use? By calling <strong>mod2$preProcess</strong>, we learn that “PCA needed 82 components to capture 95 percent of the variance”. This is because the PCA pre-processing functionality in <strong>caret</strong> uses, by default, as many PCs as it takes to cover 95% of the data variance. So, in one hand we have a PLS-DA with five LVs, on the other hand a PCA-DA with 82 PCs. This not only makes PCA-DA cheaply complicated, but arcane at the interpretation level.</p>
<p>The PLS-DA ViP plot above clearly distinguishes V1184 from all other proteins. This could be an interesting cancer biomarker. Of course, many other tests and models must be conducted in order to provide a reliable diagnostic tool. My aim here is only to introduce PLS and PLS-DA.</p>
<p>And that’s it for today, I hope you enjoyed. Please drop me a comment or message if you have any suggestion for the next post. Cheers!</p>
<p> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/poissonisfish.wordpress.com/1605/" target="_blank"><img border="0" alt="" src="http://feeds.wordpress.com/1.0/comments/poissonisfish.wordpress.com/1605/"></a> <img border="0" alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" data-recalc-dims="1"></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://poissonisfish.wordpress.com/2017/06/17/partial-least-squares-in-r/"> poissonisfish</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/P50aO7EL-4c" height="1"></a></div>