<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/comparing-partial-least-squares-to-johnsons-relative-weights/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://www.displayr.com/comparing-partial-least-squares-to-johnsons-relative-weights/"> R – Displayr</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img sizes="(max-width: 300px) 100vw, 300px" src="https://i2.wp.com/www.displayr.com/wp-content/uploads/2017/06/compare-300x185.png?resize=300%2C185&ssl=1" alt="comparing relative weights and partial least squares" class="webfeedsFeaturedVisual wp-post-image" data-recalc-dims="1">
<p>In this post I explore two different methods for computing the <em>relative importance </em>of predictors in regression: Johnson’s <em>Relative Weights </em>and <em>Partial Least Squares (PLS)</em> regression.<em> </em>Both techniques solve a problem with Multiple Linear Regression, which can perform poorly when there are correlations between predictor variables.</p>
<p>When there is a very high correlation between two predictor variables, Multiple Linear Regression can lead to one of the variables being found to be a strong predictor, while the other is found to have a relatively small effect. Relative Weights computes importance scores that <a rel="nofollow" href="https://link.springer.com/article/10.1007/s10869-014-9351-z" target="_blank">factor in the correlation between the predictors</a>. The goal of PLS is slightly different. It is designed to work in situations where it is impossible to get stable results from multiple regression. Instability may be because of extremely high correlations between variables (aka <em>multicollinearity</em>), or where there are more predictor variables than observations.</p>
<p>Relative Weights is my “go to” method in situations where there are non-trivial correlations between predictor variables (<a rel="nofollow" href="https://www.displayr.com/reasons-to-use-relative-weights/" target="_blank">click here</a> to read more about some of its strengths). However, in some fields, such as <em>chemometrics </em>and <em>sensory research, </em>PLS is the “go to” method because they have more predictor variables than observations. In this post I seek to understand how similar the techniques are when applied to a data set with moderate correlations.</p>
<hr>
<h1><span>The case study</span></h1>
<p>The data set I am using for the case study contains 34 predictor variables and 1,893 observations. I use it to look at the relationship between 327 consumer’s perceptions of six different cola brands. <a rel="nofollow" href="https://www.displayr.com/the-problem-with-using-multiple-linear-regression-for-key-driver-analysis-a-case-study-of-the-cola-market/" target="_blank">Click here</a> to read more about this case study. The correlations between the variables are shown below.</p>
<p><iframe width="450" src="https://embedded.azureedge.net/75668/210761/bee9aca8-ac48-4c11-9eb6-755e793d072a.html?v=35214-844483783"></iframe></p>
<hr>
<h1><span>Comparing the methods</span></h1>
<p>The chart below shows the relative importance computed using <a rel="nofollow" href="https://www.displayr.com/reasons-to-use-relative-weights/" target="_blank">Johnson’s <em>Relative Weights</em></a>, <em>PLS,  </em>and Multiple Linear Regression. (<a rel="nofollow" href="https://www.displayr.com/using-partial-least-squares-to-conduct-relative-importance-analysis-in-r" target="_blank">Click here</a> to see how to estimate PLS in R and <a rel="nofollow" href="https://www.displayr.com/using-partial-least-squares-to-conduct-relative-importance-analysis-in-displayr" target="_blank">here </a>to see how to do this in <a rel="nofollow" href="http://www.displayr.com/" target="_blank">Displayr</a>.) In contrast to this <a rel="nofollow" href="https://www.displayr.com/the-problem-with-using-multiple-linear-regression-for-key-driver-analysis-a-case-study-of-the-cola-market/" target="_blank">earlier case study using this data</a>, I am showing negative effects for Multiple Linear Regression. I use the same signs for Relative Weights because Relative Weights does not distinguish between whether the effect is positive or negative.</p>
<p>The chart shows us that there is a strong correlation between each of the methods. However, that should not be a great surprise, as they all use the same data and have the same functional form (i.e., linear with additive effects).</p>
<p>The <em>R<sup>2</sup></em> statistic computed predicting the PLS coefficients by the Multiple Regression coefficients is 0.98. I fitted this model without an intercept. This demonstrates that the Multiple Linear Regression and PLS results are very similar.</p>
<p>The <em>R<sup>2</sup></em> between the PLS and the Relative Weights is 0.85, which tells us that PLS is much more similar to Multiple Linear Regression than to Relative Weights. The <em>R<sup>2 </sup></em>between Multiple Linear Regression and the Relative Weights is 0.81, which means that PLS is somewhere in between the two other techniques, but is much more similar to Multiple Linear Regression.</p>
<p>If we look at the chart we can see some striking differences between Relative Weights and the other two methods:</p>
<ul>
<li>Health-conscious is much less important for Relative Weights than in the other analyses.</li>
<li>All the negative importance scores are much closer to 0 for Relative Weights than for the other analyses.</li>
<li>The small positive importance scores are greater for Relative Weights than for the other analyses.</li>
</ul>
<p><iframe width="450" src="https://embedded.azureedge.net/75668/210761/d4c6eb99-6374-4b37-b032-0f89ef15d29f.html?v=35214928463485"></iframe></p>
<hr>
<h1><span>Rescaling and removing the negative scores</span></h1>
<p>The above analysis shows that we get large and meaningful differences between Relative Weights and PLS. Why? One explanation relates to standardization. Relative weights automatically standardizes the predictor variables, presenting the importance scores in terms of their ability to explain <em>R<sup>2</sup>. </em>PLS does not standardize the predictors. A second explanation relates to negative coefficients. Relative Weights ignores the sign of coefficients, with the signs for the Relative Weights in the above analyses derived from a separate Multiple Regression. To appreciate the effect of these two things, I:</p>
<ul>
<li>Standardized all the variables (i.e., divide each variable’s raw data by the variable’s standard deviation).</li>
<li>Re-estimated all of the models, but only used variables with positive coefficients.</li>
</ul>
<p>The coefficients are plotted below. They are a lot more similar than in the previous analysis. The PLS and Multiple Linear Regression,  are particularly close, with an <em>R<sup>2</sup> </em>of 0.998. The <em>R<sup>2</sup></em>  for the PLS versus the Relative Weights is 0.95. For the Relative Weights versus Multiple Linear Regression it is 0.94. Given each analysis is for the same data set and the same core assumptions, 0.94 is still quite a difference.</p>
<p>Looking at the chart, we can see that the conclusions from the models are substantively different. In particular:</p>
<ul>
<li>Relative Weights tends to conclude that the relatively unimportant variables are more important than is the case with either PLS and Multiple Linear Regression. This can be seen from the bars at the very bottom of the chart.</li>
<li>The conclusions of the models regarding Health-conscious remain striking. The Multiple Regression and PLS both conclude that is the 10th most important variables. By contrast, Relative Weights has it as the least important variable. This earlier <a rel="nofollow" href="https://www.displayr.com/the-problem-with-using-multiple-linear-regression-for-key-driver-analysis-a-case-study-of-the-cola-market/" target="_blank">case study</a> discusses the reasons for the difference.</li>
</ul>
<p><iframe width="450" src="https://embedded.azureedge.net/75668/210761/4e505a4e-11db-4213-a98d-5ec85cafaa9a.html?v=352141263370616"></iframe></p>
<hr>
<h1><span>Conclusion</span></h1>
<p>Although PLS and Johnson’s Relative Weights are both techniques for dealing with correlations between predictors, they give fundamentally different results. In this data set, where the correlations are moderate, PLS is little different from Multiple Linear Regression. By contrast, Relative Weights gives substantially different conclusions.</p>
<p> </p>
<hr>
<p><span>TRY IT OUT</span></p>
<p>All the analysis in this post was conducted using R in Displayr. <a rel="nofollow" href="https://app.displayr.com/Try/PLS%20Comparison" target="_blank">Click here</a> to review the underlying data and code, or to run your own analyses.</p>
<hr>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.displayr.com/comparing-partial-least-squares-to-johnsons-relative-weights/"> R – Displayr</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/69PzysSWX_4" height="1"></a></div>