<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/venn-diagram-comparison-of-boruta-fselectorrcpp-and-glmnet-algorithms-2/"></div></div><div><a data-url="http://www.r-bloggers.com/venn-diagram-comparison-of-boruta-fselectorrcpp-and-glmnet-algorithms-2/" data-counturl="http://www.r-bloggers.com/venn-diagram-comparison-of-boruta-fselectorrcpp-and-glmnet-algorithms-2/" data-count="horizontal" data-text="Venn Diagram Comparison of Boruta, FSelectorRcpp and GLMnet Algorithms" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://r-addict.com/2016/06/18/Venn-Diagram-RTCGA-Feature-Selection.html"> http://r-addict.com</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img src="http://i0.wp.com/r-addict.com/images/fulls/venn.png?w=456" data-recalc-dims="1"> Feature selection is a process of extracting valuable features that have significant influence on <a target="_blank" href="https://nces.ed.gov/nceskids/help/user_guide/graph/variables.asp">dependent</a> variable. This is still an active field of research and machine wandering. In this post I compare few feature selection algorithms: traditional <a target="_blank" href="https://cran.r-project.org/web/packages/glmnet/vignettes/glmnet_beta.html">GLM with regularization</a>, computationally demanding <a target="_blank" href="https://m2.icm.edu.pl/boruta/">Boruta</a> and entropy based filter from <a target="_blank" href="http://github.com/mi2-warsaw/FSelectorRcpp">FSelectorRcpp</a> (free of Java/Weka) package. Check out the comparison on <a target="_blank" href="https://cran.r-project.org/web/packages/VennDiagram/">Venn Diagram</a> carried out on data from the <a target="_blank" href="http://r-addict.com/2016/05/04/RTCGA-Quick-Guide.html">RTCGA</a> factory of R data packages.</p>
<p>I would like to thank <a target="_blank" href="https://github.com/Magda13">Magda Sobiczewska</a> and <a target="_blank" href="http://github.com/pbiecek/"> pbiecek</a> for inspiration for this comparison. I have a chance to use Boruta nad FSelectorRcpp in action. GLMnet is here only to improve Venn Diagram.</p>
<h1>RTCGA data</h1>
<p>Data used for this comparison come from RTCGA (<a target="_blank" href="http://rtcga.github.io/RTCGA/">http://rtcga.github.io/RTCGA/</a>) and present genes’ expressions (RNASeq) from human sequenced genome. Datasets with RNASeq are available via <a target="_blank" href="https://bioconductor.org/packages/RTCGA.rnaseq/">RTCGA.rnaseq</a> data package and originally were provided by <a target="_blank" href="http://gdac.broadinstitute.org/">The Cancer Genome Atlas</a>. It’s a great set of over 20 thousand of features (1 gene expression = 1 continuous feature) that might have influence on various aspects of human survival. Let’s use data for Breast Cancer (<strong>Breast invasive carcinoma</strong> / BRCA) where we will try to find valuable genes that have impact on dependent variable denoting whether a sample of the collected readings came from tumor or normal, healthy tissue.</p>
<figure>
<pre><code data-lang="r"><span>## try http:// if https:// URLs are not supported
</span><span>source</span><span>(</span><span>&quot;https://bioconductor.org/biocLite.R&quot;</span><span>)</span>
<span>biocLite</span><span>(</span><span>&quot;RTCGA.rnaseq&quot;</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>RTCGA.rnaseq</span><span>)</span>
<span>BRCA.rnaseq</span><span>$</span><span>bcr_patient_barcode</span> <span>&lt;-</span> 
   <span>substr</span><span>(</span><span>BRCA.rnaseq</span><span>$</span><span>bcr_patient_barcode</span><span>,</span> <span>14</span><span>,</span> <span>14</span><span>)</span></code></pre>
</figure>
<p>The dependent variable, <code>bcr_patient_barcode</code>, is the <a target="_blank" href="https://wiki.nci.nih.gov/display/TCGA/TCGA+barcode">TCGA barcode</a> from which we receive information whether a sample of the collected readings came from tumor or normal, healthy tissue (14th character in the code).</p>
<p>Check another RTCGA use case: <a target="_blank" href="http://rtcga.github.io/RTCGA/Usecases.html">TCGA and The Curse of BigData</a>.</p>
<h1>GLMnet</h1>
<p>Logistic Regression, a model from generalized linear models (GLM) family, a first attempt model for class prediction, can be extended with regularization net to provide prediction and variables selection at the same time. We can assume that not valuable features will appear with equal to zero coefficient in the final model with best regularization parameter. Broader explanation can be found in the <a target="_blank" href="https://cran.r-project.org/web/packages/glmnet/vignettes/glmnet_beta.html#log">vignette of the glmnet package</a>. Below is the code I use to extract valuable features with the extra help of cross-validation and parallel computing.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>doMC</span><span>)</span>
<span>registerDoMC</span><span>(</span><span>cores</span><span>=</span><span>6</span><span>)</span>
<span>library</span><span>(</span><span>glmnet</span><span>)</span>
<span># fit the model
</span><span>cv.glmnet</span><span>(</span><span>x</span> <span>=</span> <span>as.matrix</span><span>(</span><span>BRCA.rnaseq</span><span>[,</span> <span>-1</span><span>]),</span>
          <span>y</span> <span>=</span> <span>factor</span><span>(</span><span>BRCA.rnaseq</span><span>[,</span> <span>1</span><span>]),</span>
          <span>family</span> <span>=</span> <span>&quot;binomial&quot;</span><span>,</span> 
          <span>type.measure</span> <span>=</span> <span>&quot;class&quot;</span><span>,</span> 
          <span>parallel</span> <span>=</span> <span>TRUE</span><span>)</span> <span>-></span> <span>cvfit</span>
<span># extract feature names that have # non zero coefficiant
</span><span>names</span><span>(</span><span>which</span><span>(</span>
   <span>coef</span><span>(</span><span>cvfit</span><span>,</span> <span>s</span> <span>=</span> <span>&quot;lambda.min&quot;</span><span>)[,</span> <span>1</span><span>]</span> <span>!=</span> <span>0</span><span>)</span>
   <span>)[</span><span>-1</span><span>]</span> <span>-></span> <span>glmnet.features</span>
<span>#</span> <span>first</span> <span>name</span> <span>is</span> <span>intercept</span></code></pre>
</figure>
<p>Function <code>coef</code> extracts coefficients for fitted model. Argument <code>s</code> specifies for which regularization parameter we would like to extract them – <code>lamba.min</code> is the parameter for which miss-classification error is minimal. You may also try to use <code>lambda.1se</code>.</p>
<figure>
<pre><code data-lang="r"><span>plot</span><span>(</span><span>cvfit</span><span>)</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-5" src="http://i1.wp.com/r-addict.com/figure/source/2016-06-18-Venn-Diagram-RTCGA-Feature-Selection/unnamed-chunk-5-1.png?w=456" data-recalc-dims="1"></p>
<p><a target="_blank" href="http://stats.stackexchange.com/questions/86434/is-standardisation-before-lasso-really-necessary">Discussion about standardization for LASSO can be found here</a>. I normally don’t do this, since I work with streaming data, for which checking assumptions, model diagnostics and standardization is problematic and is still a rapid field of research.</p>
<h1>Boruta</h1>
<p>The second feature selection algorithm, Boruta, is based on <a target="_blank" href="https://cran.r-project.org/web/packages/ranger/index.html">ranger</a> – a fast implementation of random forests.<br>
At start this algorithm<br>
creates duplicated variables for each attribute in the model’s formula. Duplicates<br>
then have randomly ordered values. Algorithm takes into consideration the Z-statistic of each variable in each tree and checks<br>
whether the boxplot of Z-statistics for a variable is higher than the highest boxplot for<br>
randomly ordered duplicate. Boruta is more sophisticated and its 9 page explanation can be found in <a target="_blank" href="https://www.jstatsoft.org/article/view/v036i11/v36i11.pdf">Journal of Statistical Software</a>. Below I present code that extracts valuable and tentative attributes</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>Boruta</span><span>)</span>
<span># fit features selection algorithm
</span><span>system.time</span><span>({</span>
<span>Boruta</span><span>(</span><span>as.factor</span><span>(</span><span>bcr_patient_barcode</span><span>)</span> <span>~</span><span>.</span><span>,</span>
       <span>data</span> <span>=</span> <span>BRCA.rnaseq</span><span>)</span> <span>-></span> <span>Boruta.model</span>
<span>})</span> <span># 20425 seconds ~ 35.5 min
# archive result
</span><span>library</span><span>(</span><span>archivist</span><span>)</span>
<span>saveToRepo</span><span>(</span><span>Boruta.model</span><span>,</span> 
           <span>repoDir</span> <span>=</span> <span>&quot;Museum&quot;</span><span>)</span>

<span># correct names
</span><span>gsub</span><span>(</span><span>x</span> <span>=</span> <span>getSelectedAttributes</span><span>(</span><span>Boruta.model</span><span>),</span>
     <span>pattern</span> <span>=</span> <span>&quot;`&quot;</span><span>,</span>
     <span>replacement</span> <span>=</span> <span>&quot;&quot;</span><span>,</span>
     <span>fixed</span> <span>=</span> <span>T</span><span>)</span> <span>-></span> <span>Boruta.features</span></code></pre>
</figure>
<p>There exist <code>plot()</code> overloaded function for <code>Boruta</code> class object which enables plotting boxplots of Z-statistics for each variable. It’s a good way to visualize the selection process. In this case there are over 20 thousand of attributes so I am using <code>getSelectedAttributes</code> function to extract valuable features. You could also extract <code>tentative</code> variables with <code>withTentative = TRUE</code>.</p>
<p>In case you run <code>Boruta</code> code yourself and it takes too long or you get such an error</p>
<figure>
<pre><code data-lang="r"><span>Error</span><span>:</span> <span>protect</span> <span>()</span> <span>:</span> <span>protection</span> <span>stack</span> <span>overflow</span> </code></pre>
</figure>
<p>I have archived my model on GitHub, and you can load it to R global environment with <a target="_blank" href="http://r-addict.com/2016/06/13/RHero-Saves-Backup-City-With-archivist-github.html">archivist</a> package</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>archivist</span><span>)</span>
<span>aread</span><span>(</span><span>&quot;MarcinKosinski/Museum/d3732742b989ed49666b0472ba52d705&quot;</span><span>)</span> <span>-></span> <span>Boruta.model</span></code></pre>
</figure>
<p>The history of Boruta decisions of rejecting and accepting features can be seen on such a graph</p>
<figure>
<pre><code data-lang="r"><span>plotImpHistory</span><span>(</span><span>Boruta.model</span><span>)</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-9" src="http://i2.wp.com/r-addict.com/figure/source/2016-06-18-Venn-Diagram-RTCGA-Feature-Selection/unnamed-chunk-9-1.png?w=456" data-recalc-dims="1"></p>
<h1>FSelectorRcpp::information_gain</h1>
<p>We have realized that there are many sophisticated and computationally absorbent feature selection algorithms. In many cases the time is a huge blocker and we can’t wait for Boruta to finish its computing. For such cases easier algorithms are demanded. One of them is entropy based filter implemented in <a target="_blank" href="https://github.com/larskotthoff/fselector">FSelector</a> package and its faster Rcpp (free of Java/Weka) reimplementation in <a target="_blank" href="https://github.com/mi2-warsaw/FSelectorRcpp">FSelectorRcpp</a> package (by <a target="_blank" href="https://github.com/zzawadz"> zzawadz</a>) that also has support for sparse matrixes. If you would like to contribute to this package, please check our <a target="_blank" href="https://github.com/mi2-warsaw/FSelectorRcpp/issues">issues list on GitHub</a>.</p>
<p><em>Information gain</em> is a simple, linear algorithm that computes the entropy of dependent and explanatory variables, and the conditional entropy of dependent variable with respect to each explanatory variable separately. This simple statistic enables to calculate the belief of the distribution of dependent variable when we only know the distribution of explanatory variable. Below code shows how easy it is to use</p>
<figure>
<pre><code data-lang="r"><span>devtools</span><span>::</span><span>install_github</span><span>(</span><span>'mi2-warsaw/FSelectorRcpp'</span><span>)</span>
<span>library</span><span>(</span><span>FSelectorRcpp</span><span>)</span>
<span>information_gain</span><span>(</span><span>y</span> <span>=</span> <span>BRCA.rnaseq</span><span>[,</span> <span>1</span><span>],</span>
  <span>x</span> <span>=</span> <span>BRCA.rnaseq</span><span>[,</span> <span>-1</span><span>])</span> <span>-></span> <span>FSelectorRcpp.weights</span>
<span>library</span><span>(</span><span>FSelector</span><span>)</span>
<span>cutoff.k.percent</span><span>(</span><span>FSelectorRcpp.weights</span><span>,</span> 
   <span>k</span> <span>=</span> <span>0.01</span><span>)</span> <span>-></span> <span>FSelectorRcpp.features</span></code></pre>
</figure>
<p>Information gain algorithm returns scores for attributes and we will cut top 1 % features. For that we will use simple <code>cutoff.k.percent</code> from <code>FSelector</code>.</p>
<h1>Venn Diagram</h1>
<p>Finally we can visualize differences in decisions of algorithms. One way is a <a target="_blank" href="https://en.wikipedia.org/wiki/Venn_diagram">Venn Diagram</a> that (after Wikipedia)</p>
<blockquote>
<p>shows all possible logical relations between a finite collection of different sets.</p>
</blockquote>
<figure>
<pre><code data-lang="r"><span>named.list.for.venn</span> <span>&lt;-</span>
   <span>list</span><span>(</span><span>` Boruta`</span> <span>=</span> <span>Boruta.features</span><span>,</span>
             <span>`GLMnet `</span> <span>=</span> <span>glmnet.features</span><span>,</span>
      <span>FSelectorRcpp</span> <span>=</span> <span>FSelectorRcpp.features</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>VennDiagram</span><span>)</span>
<span>venn.diagram</span><span>(</span>
    <span>x</span> <span>=</span> <span>named.list.for.venn</span><span>,</span>
    <span>filename</span> <span>=</span> <span>&quot;venn.png&quot;</span><span>,</span>
    <span>imagetype</span> <span>=</span> <span>&quot;png&quot;</span><span>,</span>
    <span>#col = &quot;transparent&quot;,,
</span>    <span>col</span> <span>=</span> <span>&quot;black&quot;</span><span>,</span>
    <span>lty</span> <span>=</span> <span>&quot;dotted&quot;</span><span>,</span>
    <span>lwd</span> <span>=</span> <span>4</span><span>,</span>
    <span>fill</span> <span>=</span> <span>c</span><span>(</span><span>&quot;cornflowerblue&quot;</span><span>,</span> <span>&quot;green&quot;</span><span>,</span> 
             <span>&quot;yellow&quot;</span><span>),</span>
    <span>alpha</span> <span>=</span> <span>0.50</span><span>,</span>
    <span>label.col</span> <span>=</span> <span>c</span><span>(</span><span>&quot;orange&quot;</span><span>,</span> <span>&quot;white&quot;</span><span>,</span>  
                  <span>&quot;orange&quot;</span><span>,</span> <span>&quot;white&quot;</span><span>,</span> 
      <span>&quot;darkblue&quot;</span><span>,</span> <span>&quot;white&quot;</span><span>,</span> <span>&quot;darkgreen&quot;</span><span>),</span>
    <span>cex</span> <span>=</span> <span>2.5</span><span>,</span>
    <span>fontfamily</span> <span>=</span> <span>&quot;serif&quot;</span><span>,</span>
    <span>fontface</span> <span>=</span> <span>&quot;bold&quot;</span><span>,</span>
    <span>cat.col</span> <span>=</span> <span>c</span><span>(</span><span>&quot;darkblue&quot;</span><span>,</span> <span>&quot;darkgreen&quot;</span><span>,</span> 
                <span>&quot;orange&quot;</span><span>),</span>
    <span>cat.cex</span> <span>=</span> <span>2.5</span><span>,</span>
    <span>cat.fontfamily</span> <span>=</span> <span>&quot;serif&quot;</span>
    <span>)</span></code></pre>
</figure>
<p>The effect of this code can be seen on the welcome graph of this post.<br>
We can see that only 1 variable was chosen with all 3 algorithms. VennDiagram package allows to find out the names of intersections.</p>
<figure>
<pre><code data-lang="r"><span>get.venn.partitions</span><span>(</span><span>named.list.for.venn</span><span>)</span> <span>-></span> 
   <span>partitions</span>
<span>apply</span><span>(</span><span>partitions</span><span>[,</span> <span>c</span><span>(</span><span>1</span><span>,</span><span>2</span><span>,</span><span>3</span><span>)],</span>
      <span>MARGIN</span> <span>=</span> <span>1</span><span>,</span>
      <span>function</span><span>(</span><span>configuration</span><span>)</span> <span>{</span>
         <span>all</span><span>(</span><span>configuration</span> <span>==</span> <span>rep</span><span>(</span><span>T</span><span>,</span><span>3</span><span>))</span>
      <span>})</span> <span>-></span> <span>all.algorithms</span>
<span>partitions</span><span>[</span><span>all.algorithms</span><span>,</span> <span>]</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">   Boruta GLMnet      FSelectorRcpp                           ..set..   ..values.. ..count..
1    TRUE        TRUE          TRUE  Boruta∩GLMnet     ∩FSelectorRcpp ABCA10|10349         1</code></pre>
</figure>
<p>Do you have any comments about the diagram or the unbelievable result that only 1 variable overlaps all 3 algorithms? Maybe some algorithms can be improved? Is this the case of random nature of <code>cv.glmnet</code> or <code>Boruta</code>? Feel free to leave your comments on the Disqus panel below.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/venn-diagram-comparison-of-boruta-fselectorrcpp-and-glmnet-algorithms-2/"></div></div><div><a data-url="http://www.r-bloggers.com/venn-diagram-comparison-of-boruta-fselectorrcpp-and-glmnet-algorithms-2/" data-counturl="http://www.r-bloggers.com/venn-diagram-comparison-of-boruta-fselectorrcpp-and-glmnet-algorithms-2/" data-count="vertical" data-text="Venn Diagram Comparison of Boruta, FSelectorRcpp and GLMnet Algorithms" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://r-addict.com/2016/06/18/Venn-Diagram-RTCGA-Feature-Selection.html"> http://r-addict.com</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/XQPoAMYMRbo"></a></div>