<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/what-hyper-parameters-are-and-what-to-do-with-them-an-illustration-with-ridge-regression/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.brodrigues.co/blog/2018-12-02-hyper-parameters/"> Econometrics and Free Software</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p><a rel="nofollow" href="https://www.youtube.com/watch?v=13Gd5kpLzsw" target="_blank"><br>
<img width="400" title="Gameboy ridge" src="https://i1.wp.com/www.brodrigues.co/blog/2018-12-02-hyper-parameters/img/ridge.jpg?w=456" data-recalc-dims="1"></a></p>
</div>
<p></p>
<p>This blog post is an excerpt of my ebook <em>Modern R with the tidyverse</em> that you can read for<br>
free <a rel="nofollow" href="https://b-rodrigues.github.io/modern_R/" target="_blank">here</a>. This is taken from Chapter 7, which deals<br>
with statistical models. In the text below, I explain what hyper-parameters are, and as an example<br>
I run a ridge regression using the <code>{glmnet}</code> package. The book is still being written, so<br>
comments are more than welcome!</p>
<div>
<h2>Hyper-parameters</h2>
<p>Hyper-parameters are parameters of the model that cannot be directly learned from the data.<br>
A linear regression does not have any hyper-parameters, but a random forest for instance has several.<br>
You might have heard of ridge regression, lasso and elasticnet. These are<br>
extensions to linear models that avoid over-fitting by penalizing <em>large</em> models. These<br>
extensions of the linear regression have hyper-parameters that the practitioner has to tune. There<br>
are several ways one can tune these parameters, for example, by doing a grid-search, or a random<br>
search over the grid or using more elaborate methods. To introduce hyper-parameters, let’s get<br>
to know ridge regression, also called Tikhonov regularization.</p>
<div>
<h3>Ridge regression</h3>
<p>Ridge regression is used when the data you are working with has a lot of explanatory variables,<br>
or when there is a risk that a simple linear regression might overfit to the training data, because,<br>
for example, your explanatory variables are collinear.<br>
If you are training a linear model and then you notice that it generalizes very badly to new,<br>
unseen data, it is very likely that the linear model you trained overfits the data.<br>
In this case, ridge regression might prove useful. The way ridge regression works might seem<br>
counter-intuititive; it boils down to fitting a <em>worse</em> model to the training data, but in return,<br>
this worse model will generalize better to new data.</p>
<p>The closed form solution of the ordinary least squares estimator is defined as:</p>
<p><span>\[<br>
\widehat{\beta} = (X'X)^{-1}X'Y<br>
\]</span></p>
<p>where <span>\(X\)</span> is the design matrix (the matrix made up of the explanatory variables) and <span>\(Y\)</span> is the<br>
dependent variable. For ridge regression, this closed form solution changes a little bit:</p>
<p><span>\[<br>
\widehat{\beta} = (X'X + \lambda I_p)^{-1}X'Y<br>
\]</span></p>
<p>where <span>\(\lambda \in \mathbb{R}\)</span> is an hyper-parameter and <span>\(I_p\)</span> is the identity matrix of dimension <span>\(p\)</span><br>
(<span>\(p\)</span> is the number of explanatory variables).<br>
This formula above is the closed form solution to the following optimisation program:</p>
<p><span>\[<br>
\sum_{i=1}^n \left(y_i – \sum_{j=1}^px_{ij}\beta_j\right)^2<br>
\]</span></p>
<p>such that:</p>
<p><span>\[<br>
\sum_{j=1}^p(\beta_j)^2 &lt; c<br>
\]</span></p>
<p>for any strictly positive <span>\(c\)</span>.</p>
<p>The <code>glmnet()</code> function from the <code>{glmnet}</code> package can be used for ridge regression, by setting<br>
the <code>alpha</code> argument to 0 (setting it to 1 would do LASSO, and setting it to a number between<br>
0 and 1 would do elasticnet). But in order to compare linear regression and ridge regression,<br>
let me first divide the data into a training set and a testing set. I will be using the <code>Housing</code><br>
data from the <code>{Ecdat}</code> package:</p>
<pre><code>library(tidyverse)
library(Ecdat)
library(glmnet)</code></pre>
<pre><code>index &lt;- 1:nrow(Housing)

set.seed(12345)
train_index &lt;- sample(index, round(0.90*nrow(Housing)), replace = FALSE)

test_index &lt;- setdiff(index, train_index)

train_x &lt;- Housing[train_index, ] %>% 
    select(-price)

train_y &lt;- Housing[train_index, ] %>% 
    pull(price)

test_x &lt;- Housing[test_index, ] %>% 
    select(-price)

test_y &lt;- Housing[test_index, ] %>% 
    pull(price)</code></pre>
<p>I do the train/test split this way, because <code>glmnet()</code> requires a design matrix as input, and not<br>
a formula. Design matrices can be created using the <code>model.matrix()</code> function:</p>
<pre><code>train_matrix &lt;- model.matrix(train_y ~ ., data = train_x)

test_matrix &lt;- model.matrix(test_y ~ ., data = test_x)</code></pre>
<p>To run an unpenalized linear regression, we can set the penalty to 0:</p>
<pre><code>model_lm_ridge &lt;- glmnet(y = train_y, x = train_matrix, alpha = 0, lambda = 0)</code></pre>
<p>The model above provides the same result as a linear regression. Let’s compare the coefficients between the two:</p>
<pre><code>coef(model_lm_ridge)</code></pre>
<pre><code>## 13 x 1 sparse Matrix of class &quot;dgCMatrix&quot;
##                       s0
## (Intercept) -3247.030393
## (Intercept)     .       
## lotsize         3.520283
## bedrooms     1745.211187
## bathrms     14337.551325
## stories      6736.679470
## drivewayyes  5687.132236
## recroomyes   5701.831289
## fullbaseyes  5708.978557
## gashwyes    12508.524241
## aircoyes    12592.435621
## garagepl     4438.918373
## prefareayes  9085.172469</code></pre>
<p>and now the coefficients of the linear regression (because I provide a design matrix, I have to use<br>
<code>lm.fit()</code> instead of <code>lm()</code> which requires a formula, not a matrix.)</p>
<pre><code>coef(lm.fit(x = train_matrix, y = train_y))</code></pre>
<pre><code>##  (Intercept)      lotsize     bedrooms      bathrms      stories 
## -3245.146665     3.520357  1744.983863 14336.336858  6737.000410 
##  drivewayyes   recroomyes  fullbaseyes     gashwyes     aircoyes 
##  5686.394123  5700.210775  5709.493884 12509.005265 12592.367268 
##     garagepl  prefareayes 
##  4439.029607  9085.409155</code></pre>
<p>as you can see, the coefficients are the same. Let’s compute the RMSE for the unpenalized linear<br>
regression:</p>
<pre><code>preds_lm &lt;- predict(model_lm_ridge, test_matrix)

rmse_lm &lt;- sqrt(mean(preds_lm - test_y)^2)</code></pre>
<p>The RMSE for the linear unpenalized regression is equal to 2077.4197343.</p>
<p>Let’s now run a ridge regression, with <code>lambda</code> equal to 100, and see if the RMSE is smaller:</p>
<pre><code>model_ridge &lt;- glmnet(y = train_y, x = train_matrix, alpha = 0, lambda = 100)</code></pre>
<p>and let’s compute the RMSE again:</p>
<pre><code>preds &lt;- predict(model_ridge, test_matrix)

rmse &lt;- sqrt(mean(preds - test_y)^2)</code></pre>
<p>The RMSE for the linear penalized regression is equal to 2072.6117757, which is smaller than before.<br>
But which value of <code>lambda</code> gives smallest RMSE? To find out, one must run model over a grid of<br>
<code>lambda</code> values and pick the model with lowest RMSE. This procedure is available in the <code>cv.glmnet()</code><br>
function, which picks the best value for <code>lambda</code>:</p>
<pre><code>best_model &lt;- cv.glmnet(train_matrix, train_y)
# lambda that minimises the MSE
best_model$lambda.min</code></pre>
<pre><code>## [1] 66.07936</code></pre>
<p>According to <code>cv.glmnet()</code> the best value for <code>lambda</code> is 66.0793576. In the<br>
next section, we will implement cross validation ourselves, in order to find the hyper-parameters<br>
of a random forest.</p>
<p>Hope you enjoyed! If you found this blog post useful, you might want to follow<br>
me on <a rel="nofollow" href="https://www.twitter.com/brodriguesco" target="_blank">twitter</a> for blog post updates and<br>
<a rel="nofollow" href="https://www.buymeacoffee.com/brodriguesco" target="_blank">buy me an espresso</a> or <a rel="nofollow" href="http://www.brodrigues.co/blog/2018-12-02-hyper-parameters/paypal.me/brodriguesco" target="_blank">paypal.me</a>.</p>

<a rel="nofollow" href="https://www.buymeacoffee.com/brodriguesco" target="_blank"><img alt="Buy me an Espresso" src="https://www.buymeacoffee.com/assets/img/BMC-btn-logo.svg"><span>Buy me an Espresso</span></a>
</div>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.brodrigues.co/blog/2018-12-02-hyper-parameters/"> Econometrics and Free Software</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/8bQu99AOeoI" height="1"></a></div>