<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/variable-selection-with-elastic-net/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://statcompute.wordpress.com/2017/09/03/variable-selection-with-elastic-net/"> S+/R – Yet Another Blog in Statistical Computing</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>LASSO has been a popular algorithm for the variable selection and extremely effective with high-dimension data. However, it often tends to “over-regularize” a model that might be overly compact and therefore under-predictive. </p>
<p>The Elastic Net addresses the aforementioned “over-regularization” by balancing between LASSO and ridge penalties. In particular, a hyper-parameter, namely Alpha, would be used to regularize the model such that the model would become a LASSO in case of Alpha = 1 and a ridge in case of Alpha = 0. In practice, Alpha can be tuned easily by the cross-validation. Below is a demonstration of Elastic Net with R glmnet package and its comparison with LASSO and ridge models. </p>
<pre>
pkgs &lt;- list(&quot;glmnet&quot;, &quot;doParallel&quot;, &quot;foreach&quot;, &quot;pROC&quot;)
lapply(pkgs, require, character.only = T)
registerDoParallel(cores = 4)
 
df1 &lt;- read.csv(&quot;Downloads/credit_count.txt&quot;)
df2 &lt;- df1[df1$CARDHLDR == 1, ]
set.seed(2017)
n &lt;- nrow(df2)
sample &lt;- sample(seq(n), size = n * 0.5, replace = FALSE)
train &lt;- df2[sample, -1]
test &lt;- df2[-sample, -1]
mdlY &lt;- as.factor(as.matrix(train[&quot;DEFAULT&quot;]))
mdlX &lt;- as.matrix(train[setdiff(colnames(df1), c(&quot;CARDHLDR&quot;, &quot;DEFAULT&quot;))])
newY &lt;- as.factor(as.matrix(test[&quot;DEFAULT&quot;]))
newX &lt;- as.matrix(test[setdiff(colnames(df1), c(&quot;CARDHLDR&quot;, &quot;DEFAULT&quot;))])
</pre>
<p>First of all, we estimates a LASSO model with Alpha = 1. The function cv.glmnet() is used to search for a regularization parameter, namely Lambda, that controls the penalty strength. As shown below, the model only identifies 2 attributes out of total 12. </p>
<pre>
# LASSO WITH ALPHA = 1
cv1 &lt;- cv.glmnet(mdlX, mdlY, family = &quot;binomial&quot;, nfold = 10, type.measure = &quot;deviance&quot;, paralle = TRUE, alpha = 1)
md1 &lt;- glmnet(mdlX, mdlY, family = &quot;binomial&quot;, lambda = cv1$lambda.1se, alpha = 1)
coef(md1)
#(Intercept) -1.963030e+00
#AGE          .           
#ACADMOS      .           
#ADEPCNT      .           
#MAJORDRG     .           
#MINORDRG     .           
#OWNRENT      .           
#INCOME      -5.845981e-05
#SELFEMPL     .           
#INCPER       .           
#EXP_INC      .           
#SPENDING     .           
#LOGSPEND    -4.015902e-02
roc(newY, as.numeric(predict(md1, newX, type = &quot;response&quot;)))
#Area under the curve: 0.636
</pre>
<p>We next estimates a ridge model as below by setting Alpha = 0. Similarly, Lambda is searched by the cross-validation. Since the ridge penalty would only regularize the magnitude of each coefficient, we end up with a “full” model with all model attributes. The model performance is slightly better with 10 more variables, which is a debatable outcome. </p>
<pre>
# RIDGE WITH ALPHA = 0
cv2 &lt;- cv.glmnet(mdlX, mdlY, family = &quot;binomial&quot;, nfold = 10, type.measure = &quot;deviance&quot;, paralle = TRUE, alpha = 0)
md2 &lt;- glmnet(mdlX, mdlY, family = &quot;binomial&quot;, lambda = cv2$lambda.1se, alpha = 0)
coef(md2)
#(Intercept) -2.221016e+00
#AGE         -4.184422e-04
#ACADMOS     -3.085096e-05
#ADEPCNT      1.485114e-04
#MAJORDRG     6.684849e-03
#MINORDRG     1.006660e-03
#OWNRENT     -9.082750e-03
#INCOME      -6.960253e-06
#SELFEMPL     3.610381e-03
#INCPER      -3.881890e-07
#EXP_INC     -1.416971e-02
#SPENDING    -1.638184e-05
#LOGSPEND    -6.213884e-03
roc(newY, as.numeric(predict(md2, newX, type = &quot;response&quot;)))
#Area under the curve: 0.6435
</pre>
<p>At last, we use the Elastic Net by tuning the value of Alpha through a line search with the parallelism. In this particular case, Alpha = 0.3 is chosen through the cross-validation. As shown below, 6 variables are used in the model that even performs better than the ridge model with all 12 attributes. </p>
<pre>
# ELASTIC NET WITH 0 &lt; ALPHA &lt; 1
a &lt;- seq(0.1, 0.9, 0.05)
search &lt;- foreach(i = a, .combine = rbind) %dopar% {
  cv &lt;- cv.glmnet(mdlX, mdlY, family = &quot;binomial&quot;, nfold = 10, type.measure = &quot;deviance&quot;, paralle = TRUE, alpha = i)
  data.frame(cvm = cv$cvm[cv$lambda == cv$lambda.1se], lambda.1se = cv$lambda.1se, alpha = i)
}
cv3 &lt;- search[search$cvm == min(search$cvm), ]
md3 &lt;- glmnet(mdlX, mdlY, family = &quot;binomial&quot;, lambda = cv3$lambda.1se, alpha = cv3$alpha)
coef(md3)
#(Intercept) -1.434700e+00
#AGE         -8.426525e-04
#ACADMOS      .           
#ADEPCNT      .           
#MAJORDRG     6.276924e-02
#MINORDRG     .           
#OWNRENT     -2.780958e-02
#INCOME      -1.305118e-04
#SELFEMPL     .           
#INCPER      -2.085349e-06
#EXP_INC      .           
#SPENDING     .           
#LOGSPEND    -9.992808e-02
roc(newY, as.numeric(predict(md3, newX, type = &quot;response&quot;)))
#Area under the curve: 0.6449
</pre>
<p> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/statcompute.wordpress.com/2985/" target="_blank"><img border="0" alt="" src="http://feeds.wordpress.com/1.0/comments/statcompute.wordpress.com/2985/"></a> <img border="0" alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" data-recalc-dims="1"></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://statcompute.wordpress.com/2017/09/03/variable-selection-with-elastic-net/"> S+/R – Yet Another Blog in Statistical Computing</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/J-sXEIwMpOw" height="1"></a></div>