<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/on-ranger-respect-unordered-factors/"></div></div><div><a data-url="http://www.r-bloggers.com/on-ranger-respect-unordered-factors/" data-counturl="http://www.r-bloggers.com/on-ranger-respect-unordered-factors/" data-count="horizontal" data-text="On ranger respect.unordered.factors" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2016/05/on-ranger-respect-unordered-factors/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>It is often said that “<a target="_blank" href="https://cran.r-project.org/">R</a> it its packages.”</p>
<p>One package of interest is <a target="_blank" href="https://cran.r-project.org/package=ranger">ranger</a> a fast parallel C++ implementation of random forest machine learning. Ranger is great package and at first glance <em>appears</em> to remove the “only 63 levels allowed for string/categorical variables” limit found in the Fortran <a target="_blank" href="https://cran.r-project.org/package=randomForest">randomForest</a> package. Actually this appearance is due to the strange choice of default value <code>respect.unordered.factors=FALSE</code> in <code>ranger::ranger()</code> which we <em>strongly</em> advise overriding to <code>respect.unordered.factors=TRUE</code> in applications.<span></span></p>
<p>
<p>To illustrate the issue we build a simple data set (split into training and evaluation) where the dependent (or outcome) variable <code>y</code> is given as the sum of how many input level codes end in an odd digit minus how many input level codes end in an even digit.</p>
<p>Some example data is given below</p>
<p><code>
<pre>
print(head(dTrain))
##          x1      x2      x3      x4 y
## 77  lev_008 lev_004 lev_007 lev_011 0
## 41  lev_016 lev_015 lev_019 lev_012 0
## 158 lev_007 lev_019 lev_001 lev_015 4
## 69  lev_010 lev_017 lev_018 lev_009 0
## 6   lev_003 lev_014 lev_016 lev_017 0
## 18  lev_004 lev_015 lev_014 lev_007 0
</pre>
<p></p>
<p>Given enough data this relation is easily learnable. In our example we have only 100 training rows and 20 possible levels for each input variable- so we at best get a noisy impression of how each independent (or input) variable affects <code>y</code>.</p>
<p>What the default ranger default training setting <code>respect.unordered.factors=FALSE</code> does is decide that string-valued variables (such as we have here) are to be treated as “ordered”. This allows ranger to skip any of the expensive re-encoding of such variables as contrasts, dummies or indicators. This is achieved in ranger by only using ordered cuts in its underlying trees and is equivalent to re-encoding the categorical variable as the numeric order codes. These variables are thus essentially treated as numeric, and ranger appears to run faster over fairly complicated variables.</p>
<p>The above is good if all of your categorical variables were in fact known to have ordered relations with the outcome. We must emphasize that this is very rarely the case in practice as one of the main reasons for using categorical variables is that we may not a-priori know the relation between the variable levels and outcome and would like the downstream machine learning to estimate the relation. The default <code>respect.unordered.factors=FALSE</code> in fact weakens the expressiveness of the ranger model (which is why it is faster).</p>
<p>This is simpler to see with an example. Consider fitting a ranger model on our example data (all code/data shared <a target="_blank" href="https://github.com/WinVector/Examples/tree/master/ranger">here</a>).</p>
<p>If we try to build a ranger model on the data using the default settings we get the following:</p>
<p><code>
<pre>
# default ranger model, treat categoricals as ordered (a very limiting treatment)
m1 &lt;- ranger(y~x1+x2+x3+x4,data=dTrain, write.forest=TRUE)
</pre>
<p></p>
<p><img title="NewImage.png" alt="NewImage" src="http://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2016/05/NewImage-2.png?w=660" border="0" data-recalc-dims="1"></p>
<p>Keep in mind the 0.24 <a target="_blank" href="http://www.win-vector.com/blog/2011/11/correlation-and-r-squared/">R-squared</a> on test.</p>
<p>If we set <code>respect.unordered.factors=TRUE</code> ranger takes a lot longer to run (as it is doing more work in actually respecting the individual levels of our categorical variables) but gets a much better result (test R-squared 0.54).</p>
<p><code>
<pre>
m2 &lt;- ranger(y~x1+x2+x3+x4,data=dTrain,  write.forest=TRUE,
             respect.unordered.factors=TRUE)
</pre>
<p></p>
<p><img title="NewImage.png" alt="NewImage" src="http://i0.wp.com/www.win-vector.com/blog/wp-content/uploads/2016/05/NewImage-3.png?w=660" border="0" data-recalc-dims="1"></p>
<p>The loss of modeling power seen with the default <code>respect.unordered.factors=FALSE</code> is similar to the undesirable loss of modeling power seen if one <a target="_blank" href="http://www.win-vector.com/blog/2014/12/a-comment-on-preparing-data-for-classifiers/">hash-encodes categorical levels</a>. Everyone claims they would never do such a thing, but we strongly suggest inspecting your team’s work for these bad but tempting shortcuts.</p>
<p>If even one of the variables had 64 or more levels ranger would throw an exception and not complete training (as the randomForest library also does).</p>
<p>The correct way to feed large categoricals to a random forest model remains to explicitly introduce the dummy/indicators yourself or <a target="_blank" href="http://www.win-vector.com/blog/2012/07/modeling-trick-impact-coding-of-categorical-variables-with-many-levels/">re-encode them as impact/effect sub models</a>. Both of these are services supplied by the <a target="_blank" href="https://cran.r-project.org/package=vtreat">vtreat package</a> so we demonstrate the technique here.</p>
<p><code>
<pre>
# vtreat re-encoded model
ct &lt;- vtreat::mkCrossFrameNExperiment(dTrain,c('x1','x2','x3','x4'),'y')
newvars &lt;- ct$treatments$scoreFrame$varName[(ct$treatments$scoreFrame$code=='catN') &amp;
                                            (ct$treatments$scoreFrame$sig&lt;1)]
m3 &lt;- ranger(paste('y',paste(newvars,collapse=' + '),sep=' ~ '),data=ct$crossFrame,
              write.forest=TRUE)
dTestTreated &lt;- vtreat::prepare(ct$treatments,dTest,
                                pruneSig=c(),varRestriction=newvars)
dTest$rangerNestedPred &lt;- predict(m3,data=dTestTreated)$predictions
WVPlots::ScatterHist(dTest,'rangerNestedPred','y',
                     'ranger vtreat nested prediction on test',
                     smoothmethod='identity',annot_size=3)
</pre>
<p></p>
<p><img title="NewImage.png" alt="NewImage" src="http://i1.wp.com/www.win-vector.com/blog/wp-content/uploads/2016/05/NewImage-4.png?w=660" border="0" data-recalc-dims="1"></p>
<p>The point is a test R-squared of 0.6 or 0.54 is a <em>lot</em> better than an R-squared of 0.24. You do not want to achieve 0.24 if 0.6 is within easy reach. So at the very least when using ranger set <code>respect.unordered.factors=TRUE</code>; for unordered factors (the most common kind) the default is making things easy for ranger at the expense of model quality.</p>
<p>Instructions explaining the use of vtreat can be found <a target="_blank" href="http://www.win-vector.com/blog/2016/05/vtreat-cross-frames/">here</a>.</p>

<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/on-ranger-respect-unordered-factors/"></div></div><div><a data-url="http://www.r-bloggers.com/on-ranger-respect-unordered-factors/" data-counturl="http://www.r-bloggers.com/on-ranger-respect-unordered-factors/" data-count="vertical" data-text="On ranger respect.unordered.factors" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>

<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2016/05/on-ranger-respect-unordered-factors/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/WQvLPlRS5cI"></a></div></code></code></code></code>