<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/benchmarking-feature-selection-algorithms-with-xy/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://www.statworx.com/de/blog/benchmarking-feature-selection-algorithms-with-xy/"> r-bloggers – STATWORX</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h2> Feature Selection </h2>
<p>Feature Selection is one of the most interesting fields in machine learning in my opinion. It is a boundary point of two different perspectives on machine learning – performance and inference. From a performance point of view, feature selection is typically used to increase the model performance or to reduce the complexity of the problem in order to optimize computational efficiency. From an inference perspective, it is important to extract variable importance to identify key drivers of a problem.</p>
<p>Many people argue that in the era of deep learning feature selection is not important anymore. As a method of representation learning, deep learning models can find important features of the input data on their own. Those features are basically nonlinear transformations of the input data space. However, not every problem is suited to be approached with neural nets (actually, many problems). In many practical ML applications feature selection plays a key role on the road to success.</p>
<p>At <a rel="nofollow" href="https://statworx.com/" target="_blank">STATWORX</a> we work in a lot of dynamic pricing projects. One key piece in the puzzle of dynamic pricing is <a rel="nofollow" href="https://www.statworx.com/blog/food-for-regression-using-sales-data-to-identify-price-elasticity/%3Eprice%20elasticity%20estimation%3C/a%3E.%20Thereby,%20it%20is%20beneficial%20if%20you%20can%20rely%20on%20a%20white-box%20algorithm%20that%20lets%20you%20interpret%20its%20estimates%20while%20performing%20variable%20selection.%20This%20is%20where%20classic%20machine%20learning%20algorithm%20shine.%20For%20instance,%20%3Ca%20href=" target="_blank">component-wise Gradient Boosting</a> is an excellent example of such a white-box machine.</p>
<h2> Introducing <code>Xy()</code> </h2>
<p><img alt="xy hexagon logo" src="https://i2.wp.com/www.statworx.com/wp-content/uploads/Xy.png?w=450&ssl=1" class="wp-image-14540" data-recalc-dims="1"></p>
<p>In this blog post I will try to challenge feature selection algorithms with my simulation function <code>Xy()</code>. The function is designed to simulate regression and classification problems with a maximum degree of freedom for the user. If you want to get a quick introduction to the function I recommend reading my previous <a rel="nofollow" href="https://www.statworx.com/blog/pushing-ordinary-least-squares-to-the-limit-with-xy/" target="_blank">blog post</a>, where I have analyzed Ordinary Least Squares coefficients under different signal strengths. </p>
<p>To challenge the feature selection algorithms, we first need to simulate data. By default, <code>Xy()</code> simulates regression learning data with 1000 observations, two linear and two nonlinear features as well as five randomly generated variables, which are not used to generate the target. We will use these random variables as pitfalls for the feature selection algorithms. To make things a little spicier let’s see how we can generate ten linear, ten non-linear and three categorical variables along with 50 random variables. </p>
<pre><code># Simulating regression data 

# install the package
devtools::install_github(“andrebleier/Xy”)

# load the library
library(Xy)

fs_sim = Xy(n = obs,
            numvars = c(10,10),
            catvars = c(3, 2),
            noisevars = 50,
            task = Xy_task(),
            nlfun = function(x) {x^2},
            interactions = 1,
            sig = c(1,4), 
            cor = c(0),
            weights = c(-10,10),
            intercept = TRUE,
            stn = 4)

print(fs_sim)
</code></pre>
<pre><code lang="r"> Xy Simulation 
 	 | 
 	 | + task regression
 	 | + observations 1000
 	 | + interactions 1D
 	 | + signal to noise ratio 4
 	 | + effects 
 	   | - linear 10
 	   | - nonlinear 10
 	   | - categorical 3
 	   | - noise 50
 	 | + intervals 
 	   | - correlation 0
 	   | - weights [-10,10]
 	   | - sd [1,4]


Target generating process: 
y = -0.218 + 5.4NLIN_01 - 9.48NLIN_02 + 7NLIN_03 - 7.59NLIN_04 - 7.43NLIN_05 - 0.39NLIN_06 - 7.72NLIN_07 + 0.02NLIN_08 + 9.05NLIN_09 + 7.75NLIN_10 + 8.76LIN_01 + 7.7LIN_02 - 0.77LIN_03 + 8.16LIN_04 + 1.06LIN_05 + 4.46LIN_06 - 4.01LIN_07 + 3.4LIN_08 - 7.94LIN_09 - 8.65LIN_10 - 3.51DUMMY_01__2 - 10.04DUMMY_02__2 - 1.63DUMMY_03__2 + e ~ N(0,99.28)
</code></pre>
<p>You can extract the feature importance with the <code>varimp()</code> function. Furthermore, there is a build-in plotting argument, that allows for visualization. I decided to use Boxplots rather than a plain barchart as it visualizes a local and global view on the importance.</p>
<pre><code># Feature Importance
fs_varimp &lt;- varimp(fs_sim, plot = TRUE)
</code></pre>
<p><img alt="xy-variable-importance" src="https://i1.wp.com/www.statworx.com/wp-content/uploads/xy-variable-importance.png?w=450&ssl=1" class="wp-image-14772" data-recalc-dims="1"></p>
<h2> Benchmarking Framework </h2>
<p>Ok, we saw how we can generate data, but how about the algorithms? We are going to use four different classical machine learning algorithms. </p>
<ol>
<li><a rel="nofollow" href="https://xgboost.readthedocs.io/en/latest/model.html" target="_blank">Gradient Boosting</a> with <code>library(xgboost)</code>
</li>
<li><a rel="nofollow" href="https://cran.r-project.org/web/packages/mboost/mboost.pdf" target="_blank">Component-wise Gradient Boosting</a> with <code>library(mboost)</code>
</li>
<li><a rel="nofollow" href="https://cran.r-project.org/web/packages/foba/foba.pdf" target="_blank">Ridge Regression</a> with <code>library(foba)</code>
</li>
<li><a rel="nofollow" href="https://cran.r-project.org/web/packages/rpart/rpart.pdf" target="_blank">Random Forest</a> with <code>library(rpart)</code>
</li>
</ol>
<p>Two gradient boosting implementations, a greedy ridge regression and a random forest. If you want to read up on the mechanics of the algorithms I strongly recommend the package documentations.<br>
In the end we want to elect a winner of course, thus we have to define the rules. There is not much literature on this matter, so I created three measures which made sense to me. Why three though? I think there are several angles to this problem. Sure a mean absolute error (<img src="https://i1.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-453371d4360092fdef8eafc512db153b_l3.png?resize=27%2C13&ssl=1" alt="M1" width="27" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="13">) between the real and estimated importance could be the first approach. </p>
<p><span>   </span><span>   </span><img src="https://i1.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-e7264a12da792f024ba2cdca8c607796_l3.png?resize=195%2C50&ssl=1" width="195" alt="[M1 = frac{1}{n} sum_{i=1}^{n} | widehat{Imp} - Imp |]" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="50"></p>
<p>This gives us a feeling of how close an algorithm gets to the real importance. However, one could argue, that the deviation itself is not as interesting as the plain number of features it got right (<img src="https://i2.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-de8d466f415eef42e29c141c05b065bd_l3.png?resize=27%2C12&ssl=1" alt="M2" width="27" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="12">).</p>
<p><span>   </span><span>   </span><img src="https://i0.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-024e86b1b7fc8eba2bdbed4f1f5cb829_l3.png?resize=166%2C18&ssl=1" width="166" alt="[M2 = p_{est} / p_{real} * 100]" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="18"></p>
<p>where <img src="https://i2.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-23264d145f5c50e4bc8e3b94bd818cf2_l3.png?resize=27%2C12&ssl=1" alt="p_{est}" width="27" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="12"> are the number of true features selected by the algorithm, whereas <img src="https://i2.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-e5464f43011edb25b6126fcb2cf10531_l3.png?resize=34%2C12&ssl=1" alt="p_{real}" width="34" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="12"> is the total number of features.<br>
Another angle to this problem are falsely selected features. Remember, we simulated fifty noise variables in the example above. The last measure aims to clarify the ratio of true to noise features, which have been selected by an algorithm.</p>
<p><span>   </span><span>   </span><img src="https://i1.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-1ffb7cfc890826927c74e783e0ea0786_l3.png?resize=314%2C40&ssl=1" width="314" alt="[M3 = frac{p_{est} * imp_{est}}{(p_{true} * imp_{est} + p_{noise} * imp_{noise})}]" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="40"></p>
<p>where <img src="https://i2.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-f1215fbd56e24d306127afb22a56ec10_l3.png?resize=48%2C16&ssl=1" alt="imp_{est}" width="48" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="16"> is the aggregated importance for all estimated true features and <img src="https://i0.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-4c6eeff18ad522696210f72eeca293be_l3.png?resize=42%2C12&ssl=1" alt="p_{noise}" width="42" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="12"> are the noise variables selected by the algorithm and their summarized importance <img src="https://i0.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-982a98b5f953e5b49589f3da5084b5b4_l3.png?resize=63%2C16&ssl=1" alt="imp_{noise}" width="63" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="16">.</p>
<p>Each algorithm gets a total of 100 randomly drawn hyperparameter sets. The hyperparameter configuration with the lowest cross-validation error will represent the feature importance of that particular algorithm. Furthermore, we will repeat the data simulation 20 times to create a little more variation. Each time the number of features, noise variables, polynomial degree of nonlinearity will be drawn randomly. You can find the simulation code on my <a rel="nofollow" href="https://github.com/andrebleier/Xy" target="_blank">GitHub</a>. The whole simulation took three days on eight cores, so you might want to change the parameters at the beginning of the script.</p>
<h2> Evaluation </h2>
<p>A further point to note is, that the tree algorithms gradient boosting (xgboost) and the random forest (rpart) can capture nonlinearities while the ridge regression and the component-wise gradient boosting apply ordinary least squares approaches. Since, we are simulating nonlinear and linear features, this might be an advantage for the tree algorithms. The results are visualized in the following figure. </p>
<p><img alt="xy-simulation-results" src="https://i2.wp.com/www.statworx.com/wp-content/uploads/xy-simulation-results.png?w=450&ssl=1" class="wp-image-14769" data-recalc-dims="1"></p>
<p>While the metrics do not speak in favor of one particular algorithm overall, there are several takeaways.</p>
<ul>
<li>The greedy ridge regression (foba) tends to select all important features (M2), however, at the cost of selecting a relatively large amount of irrelevant variables (M3).
</li>
<li>The component-wise gradient boosting algorithm (glmboost) selects over 78% of the true features on average (M2), while it can reassemble the true feature importance with the lowest mean absolute error (M1). Unlike the ridge regression it does not select a large amount of noise variables (M3).
</li>
<li>The random forest (rpart) tends to be more restrictive on the amount of features it selects (M2). However, those which get selected seem to be true features (M3).
</li>
<li> The gradient boosted trees (xgbDART), seem to be the winner for most research questions, since it has the best M2 and M3 scores. On average xgboost finds over 93% of the true features (M2), while selecting a relatively low amount of irrelevant features (M3). However, compared to the other algorithms it cannot really reassemble the true feature importance (M1).
</li>
</ul>
<p>Of course, real-world problems are by far more complex than these simulations. However, I think that at least you can extract some tendencies about real-world problems from this example. If you have any ideas about other metrics or angles I did not cover in this example or you have an idea to improve my function, please feel free to give me some <a rel="nofollow" href="https://github.com/andrebleier" target="_blank">feedback</a>.</p>
<div>
<h6>Über den Autor</h6>
<p><img src="https://i2.wp.com/www.statworx.com/wp-content/uploads/andre-1-300x300.jpg?resize=180%2C180&ssl=1" width="180" alt="André Bleier" class="wp-user-avatar wp-user-avatar-180" data-recalc-dims="1" height="180"></p>
<div>
<h4>André Bleier</h4>
<p>André ist im Data Science Team und organisiert intern unsere Afterwork Veranstaltungen. In seiner Freizeit treibt er Sport und programmiert gerne kleine und große R Codes.</p>
</div>
</div>
<p>Der Beitrag <a rel="nofollow" href="https://www.statworx.com/de/blog/benchmarking-feature-selection-algorithms-with-xy/" target="_blank">Benchmarking Feature Selection Algorithms with Xy()</a> erschien zuerst auf <a rel="nofollow" href="https://www.statworx.com/de/" target="_blank">STATWORX</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.statworx.com/de/blog/benchmarking-feature-selection-algorithms-with-xy/"> r-bloggers – STATWORX</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/UQN_7FfEKuI" height="1"></a></div>