<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/ridge-regression-and-the-lasso/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://realdataweb.wordpress.com/2017/05/23/ridge-regression-and-the-lasso/"> R – Real Data</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img src="https://i0.wp.com/www.bonappetit.com/wp-content/uploads/2015/05/double-rl-cowboy-lasso-1024x682.jpg"></p>
<p>In my last post <a rel="nofollow" href="https://realdataweb.wordpress.com/2017/05/23/ridge-regression-and-the-lasso/link" target="_blank">Which linear model is best?</a> I wrote about using<br>
stepwise selection as a method for selecting linear models, which turns<br>
out to have some issues (see <a rel="nofollow" href="https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=10&cad=rja&uact=8&ved=0ahUKEwi-68Tu8vbTAhWB7SYKHfkvAWkQFghUMAk&url=http%3A%2F%2Fwww.stata.com%2Fsupport%2Ffaqs%2Fstatistics%2Fstepwise-regression-problems%2F&usg=AFQjCNFQ7xu4ksmQ_LVDGZBDD1U8LSvaTw&sig2=3i--KOG61dhEMOpF0r1j3g" target="_blank">this article</a>, <a rel="nofollow" href="https://en.wikipedia.org/wiki/Stepwise_regression#Criticism" target="_blank">and Wikipedia</a>).<br>
This post will be about two methods that slightly modify ordinary least<br>
squares (OLS) regression – ridge regression and the lasso.</p>
<p>Ridge regression and the lasso are closely related, but only the Lasso<br>
has the ability to select predictors. Like OLS, ridge attempts to<br>
minimize residual sum of squares of predictors in a given model.<br>
However, ridge regression includes an additional ‘shrinkage’ term – the<br>
square of the coefficient estimate – which shrinks the estimate of the<br>
coefficients towards zero. The impact of this term is controlled by<br>
another term, lambda (determined seperately). Two interesting<br>
implications of this design are the facts that when λ = 0 the OLS<br>
coefficients are returned and when λ = ∞, coefficients will approach<br>
zero.</p>
<p>To take a look at this, setup a model matrix (removing the intercept<br>
column), store the independent variable as <code>y</code>, and create a vector of<br>
lambda values.</p>
<pre>
swiss &lt;- datasets::swiss
x &lt;- model.matrix(Fertility~., swiss)[,-1]
y &lt;- swiss$Fertility
lambda &lt;- 10^seq(10, -2, length = 100)
</pre>
<p>First, let's prove the fact that when λ = 0 we get the same coefficients<br>
as the OLS model.</p>
<pre>
#create test and training sets
library(glmnet)
</pre>
<pre><code>## Loading required package: Matrix

## Loading required package: foreach

## Loaded glmnet 2.0-10
</code></pre>
<pre>
set.seed(489)
train = sample(1:nrow(x), nrow(x)/2)
test = (-train)
ytest = y[test]
</pre>
<p>Fit your models.</p>
<pre>
#OLS
swisslm &lt;- lm(Fertility~., data = swiss)
coef(swisslm)
</pre>
<pre><code>##      (Intercept)      Agriculture      Examination        Education 
##       66.9151817       -0.1721140       -0.2580082       -0.8709401 
##         Catholic Infant.Mortality 
##        0.1041153        1.0770481
</code></pre>
<pre>
#ridge
ridge.mod &lt;- glmnet(x, y, alpha = 0, lambda = lambda)
predict(ridge.mod, s = 0, exact = T, type = 'coefficients')[1:6,]
</pre>
<pre><code>##      (Intercept)      Agriculture      Examination        Education 
##       66.9365901       -0.1721983       -0.2590771       -0.8705300 
##         Catholic Infant.Mortality 
##        0.1040307        1.0770215
</code></pre>
<p>The differences here are nominal. Let's see if we can use ridge to<br>
improve on the OLS estimate.</p>
<pre>
swisslm &lt;- lm(Fertility~., data = swiss, subset = train)
ridge.mod &lt;- glmnet(x[train,], y[train], alpha = 0, lambda = lambda)
#find the best lambda from our list via cross-validation
cv.out &lt;- cv.glmnet(x[train,], y[train], alpha = 0)
</pre>
<pre><code>## Warning: Option grouped=FALSE enforced in cv.glmnet, since &lt; 3 observations
## per fold
</code></pre>
<pre>
bestlam &lt;- cv.out$lambda.min
</pre>
<pre>
#make predictions
ridge.pred &lt;- predict(ridge.mod, s = bestlam, newx = x[test,])
s.pred &lt;- predict(swisslm, newdata = swiss[test,])
#check MSE
mean((s.pred-ytest)^2)
</pre>
<pre><code>## [1] 106.0087
</code></pre>
<pre>
mean((ridge.pred-ytest)^2)
</pre>
<pre><code>## [1] 93.02157
</code></pre>
<p>Ridge performs better for this data according to the MSE.</p>
<pre>
#a look at the coefficients
out = glmnet(x[train,],y[train],alpha = 0)
predict(ridge.mod, type = &quot;coefficients&quot;, s = bestlam)[1:6,]
</pre>
<pre><code>##      (Intercept)      Agriculture      Examination        Education 
##      64.90631178      -0.16557837      -0.59425090      -0.35814759 
##         Catholic Infant.Mortality 
##       0.06545382       1.30375306
</code></pre>
<p>As expected, most of the coefficient estimates are more conservative.</p>
<p>Let's have a look at the lasso. The big difference here is in the<br>
shrinkage term – the lasso takes the absolute value of the coefficient<br>
estimates.</p>
<pre>
lasso.mod &lt;- glmnet(x[train,], y[train], alpha = 1, lambda = lambda)
lasso.pred &lt;- predict(lasso.mod, s = bestlam, newx = x[test,])
mean((lasso.pred-ytest)^2)
</pre>
<pre><code>## [1] 124.1039
</code></pre>
<p>The MSE is a bit higher for the lasso estimate. Let's check out the<br>
coefficients.</p>
<pre>
lasso.coef  &lt;- predict(lasso.mod, type = 'coefficients', s = bestlam)[1:6,]
</pre>
<p>Looks like the lasso places high importance on <code>Education</code>,<br>
<code>Examination</code>, and <code>Infant.Mortality</code>. From this we also gain some<br>
evidence that <code>Catholic</code> and <code>Agriculture</code> are not useful predictors for<br>
this model. It is likely that <code>Catholic</code> and <code>Agriculture</code> do have some<br>
effect on <code>Fertility</code>, though, since pushing those coefficients to zero<br>
hurt the model.</p>
<p>There is plenty more to delve into here, but I'll leave the details to<br>
the experts. I am always happy to have your take on the topics I write<br>
about, so please feel free to leave a comment or contact me. Oftentimes<br>
I learn just as much from you all as I do in researching the topic I<br>
write about.</p>
<p>I think the next post will be about more GIS stuff – maybe on rasters or<br>
point pattern analysis.</p>
<p>Thank you for reading!</p>
<p>Kiefer</p>
<p> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/realdataweb.wordpress.com/610/" target="_blank"><img border="0" alt="" src="http://feeds.wordpress.com/1.0/comments/realdataweb.wordpress.com/610/"></a> <img border="0" alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" data-recalc-dims="1"></p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://realdataweb.wordpress.com/2017/05/23/ridge-regression-and-the-lasso/"> R – Real Data</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/a078GVHEUK0" height="1"></a></div>