<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/ridge-regression-in-r-exercises/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.r-exercises.com/2017/06/19/ridge-regression-in-r-exercises/?utm_source=rss&utm_medium=rss&utm_campaign=ridge-regression-in-r-exercises"> R-exercises</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a rel="nofollow" href="https://i1.wp.com/www.r-exercises.com/wp-content/uploads/2017/06/Rplotex6_2.jpeg" target="_blank"><img alt="" src="https://i1.wp.com/www.r-exercises.com/wp-content/uploads/2017/06/Rplotex6_2.jpeg?resize=150%2C150" class="wp-image-5754" data-recalc-dims="1"></a></p>
<p>Bias vs Variance tradeoff is always encountered in applying supervised learning algorithms. Least squares regression provides a good fit for the training set but can suffer from high variance which lowers predictive ability. To counter this problem, we can regularize the beta coefficients by employing a penalization term. Ridge regression applies l2 penalty to the residual sum of squares. In contrast, LASSO regression, which was covered <a rel="nofollow" href="http://www.r-exercises.com/2017/06/12/lasso-regression-in-r-exercises/" target="_blank">here</a> previously, applies l1 penalty.<br>
Using ridge regression, we can shrink the beta coefficients towards zero which would reduce variance at the cost of higher bias which can result in better predictive ability than least squares regression. In this exercise set we will use the <code>glmnet</code> package (package description: <a rel="nofollow" href="https://cran.r-project.org/web/packages/glmnet/glmnet.pdf" target="_blank">here</a>) to implement ridge regression in R.</p>
<p>Answers to the exercises are available <a rel="nofollow" href="http://r-exercises.com/2017/06/19/ridge-regression-in-r-solutions/" target="_blank">here</a>.</p>
<p><strong>Exercise 1</strong><br>
Load the <code>lars</code> package and the <code>diabetes</code> dataset (Efron, Hastie, Johnstone and Tibshirani (2003) “Least Angle Regression” (with discussion) Annals of Statistics). This is the same dataset from the LASSO exercise set and has patient level data on the progression of diabetes. Next, load the <code>glmnet</code> package that will that we will now use to implement ridge regression.<br>
The dataset has three matrices <code>x</code>, <code>x2</code> and <code>y</code>. <code>x</code> has a smaller set of independent variables while <code>x2</code> contains the full set with quadratic and interaction terms. <code>y</code> is the dependent variable which is a quantitative measure of the progression of diabetes.<br>
Generate separate scatterplots with the line of best fit for all the predictors in <code>x</code> with <code>y</code> on the vertical axis.<br>
Regress <code>y</code> on the predictors in <code>x</code> using OLS. We will use this result as benchmark for comparison.</p>
<p><strong>Exercise 2</strong><br>
Fit the ridge regression model using the <code>glmnet</code> function and plot the trace of the estimated coefficients against lambdas. Note that coefficients are shrunk closer to zero for higher values of lambda.</p>
<p><strong>Exercise 3</strong><br>
Use the <code>cv.glmnet</code> function to get the cross validation curve and the value of lambda that minimizes the mean cross validation error.</p>
<p><strong>Exercise 4</strong><br>
Using the minimum value of lambda from the previous exercise, get the estimated beta matrix. Note that coefficients are lower than least squares estimates.</p>
<p><strong>Exercise 5</strong><br>
To get a more parsimonious model we can use a higher value of lambda that is within one standard error of the minimum. Use this value of lambda to get the beta coefficients. Note the shrinkage effect on the estimates.</p>
<div>
<aside><img src="http://www.r-exercises.com/2017/06/19/ridge-regression-in-r-exercises/?utm_source=rss&utm_medium=rss&utm_campaign=ridge-regression-in-r-exercises"></aside>
<div><strong>Learn more</strong> about Model Evaluation in the online course <a rel="nofollow" href="http://www.r-exercises.com/product/regression-machine-learning-with-r/" target="_blank">Regression Machine Learning with R</a>. In this course you will learn how to:
<ul>
<li>Avoid model over-fitting using cross-validation for optimal parameter selection</li>
<li>Explore maximum margin methods such as best penalty of error term support vector machines with linear and non-linear kernels.</li>
<li>And much more</li>
</ul>
</div>
</div>
<p><strong>Exercise 6</strong><br>
Split the data randomly between a training set (80%) and test set (20%). We will use these to get the prediction standard error for least squares and ridge regression models.</p>
<p><strong>Exercise 7</strong><br>
Fit the ridge regression model on the training and get the estimated beta coefficients for both the minimum lambda and the higher lambda within 1-standard error of the minimum.</p>
<p><strong>Exercise 8</strong><br>
Get predictions from the ridge regression model for the test set and calculate the prediction standard error. Do this for both the minimum lambda and the higher lambda within 1-standard error of the minimum.</p>
<p><strong>Exercise 9</strong><br>
Fit the least squares model on the training set.</p>
<p><strong>Exercise 10</strong><br>
Get predictions from the least squares model for the test set and calculate the prediction standard error.</p>
<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="nofollow" href="http://www.r-exercises.com/2017/06/12/lasso-regression-in-r-exercises/" title="LASSO regression in R exercises" target="_blank">LASSO regression in R exercises </a></li>
<li><a rel="nofollow" href="http://www.r-exercises.com/2017/05/31/evaluate-your-model-with-r-exercises/" title="Evaluate your model with R Exercises" target="_blank">Evaluate your model with R Exercises </a></li>
<li><a rel="nofollow" href="http://www.r-exercises.com/2017/06/05/quantile-regression-in-r-exercises/" title="Quantile Regression in R exercises" target="_blank">Quantile Regression in R exercises </a></li>
<li><a rel="nofollow" href="http://www.r-exercises.com/start-here-to-learn-r/" target="_blank">Explore all our (>1000) R exercises</a></li>
<li><a rel="nofollow" href="http://www.r-exercises.com/r-courses/" target="_blank">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.r-exercises.com/2017/06/19/ridge-regression-in-r-exercises/?utm_source=rss&utm_medium=rss&utm_campaign=ridge-regression-in-r-exercises"> R-exercises</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/fuPNK8HhPHg" height="1"></a></div>