<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/classification-from-scratch-bagging-and-forests-10-8/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://freakonometrics.hypotheses.org/52777"> R-english – Freakonometrics</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Tenth post of our series on <a rel="nofollow" href="https://freakonometrics.hypotheses.org/52731" target="_blank">classification from scratch</a>. Today, we’ll see the heuristics of the algorithm inside bagging techniques.</p>
<p>Often, bagging is associated with trees, to generate forests. But actually, it is possible using bagging for any kind of model. Recall that <a rel="nofollow" href="https://en.wikipedia.org/wiki/Bootstrap_aggregating" target="_blank">bagging</a> means “boostrap aggregation”. So, consider a model <img alt="m:\mathcal{X}\rightarrow \mathcal{Y}" title="m:\mathcal{X}\rightarrow \mathcal{Y}" src="https://s0.wp.com/latex.php?latex=m%3A%5Cmathcal%7BX%7D%5Crightarrow+%5Cmathcal%7BY%7D&bg=ffffff&fg=000&s=0" class="latex">. Let <img alt="\widehat{m}_{S}" title="\widehat{m}_{S}" src="https://s0.wp.com/latex.php?latex=%5Cwidehat%7Bm%7D_%7BS%7D&bg=ffffff&fg=000&s=0" class="latex"> denote the estimator of <img alt="m" title="m" src="https://s0.wp.com/latex.php?latex=m&bg=ffffff&fg=000&s=0" class="latex"> obtained from sample <img alt="S=\{y_i,\mathbf{x}_i\}" title="S=\{y_i,\mathbf{x}_i\}" src="https://s0.wp.com/latex.php?latex=S%3D%5C%7By_i%2C%5Cmathbf%7Bx%7D_i%5C%7D&bg=ffffff&fg=000&s=0" class="latex"> with <img alt="i=\{1,\cdots,n\}" title="i=\{1,\cdots,n\}" src="https://s0.wp.com/latex.php?latex=i%3D%5C%7B1%2C%5Ccdots%2Cn%5C%7D&bg=ffffff&fg=000&s=0" class="latex">.</p>
<p>Consider now some boostrap sample, <img alt="S_b=\{y_i,\mathbf{x}_i\}" title="S_b=\{y_i,\mathbf{x}_i\}" src="https://s0.wp.com/latex.php?latex=S_b%3D%5C%7By_i%2C%5Cmathbf%7Bx%7D_i%5C%7D&bg=ffffff&fg=000&s=0" class="latex"> with <img alt="i" title="i" src="https://s0.wp.com/latex.php?latex=i&bg=ffffff&fg=000&s=0" class="latex"> is randomly drawn from <img alt="\{1,\cdots,n\}" title="\{1,\cdots,n\}" src="https://s0.wp.com/latex.php?latex=%5C%7B1%2C%5Ccdots%2Cn%5C%7D&bg=ffffff&fg=000&s=0" class="latex"> (with replacement). Based on that sample, estimate <img alt="\widehat{m}_{S_b}" title="\widehat{m}_{S_b}" src="https://s0.wp.com/latex.php?latex=%5Cwidehat%7Bm%7D_%7BS_b%7D&bg=ffffff&fg=000&s=0" class="latex">. Then draw many samples, and consider the agregation of the estimators obtained, using either a majority rule, or using the average of probabilities (if a probabilist model was considered). Hence<img alt="\widehat{m}^{bag}(\mathbf{x})=\frac{1}{B}\sum_{b=1}^B \widehat{m}_{S_b}(\mathbf{x})" title="\widehat{m}^{bag}(\mathbf{x})=\frac{1}{B}\sum_{b=1}^B \widehat{m}_{S_b}(\mathbf{x})" src="https://s0.wp.com/latex.php?latex=%5Cwidehat%7Bm%7D%5E%7Bbag%7D%28%5Cmathbf%7Bx%7D%29%3D%5Cfrac%7B1%7D%7BB%7D%5Csum_%7Bb%3D1%7D%5EB+%5Cwidehat%7Bm%7D_%7BS_b%7D%28%5Cmathbf%7Bx%7D%29&bg=ffffff&fg=000&s=0" class="latex"></p>
<h2>Bagging logistic regression #1</h2>
<p>Consider the case of the logistic regression. To generate a bootstrap sample, it is natural to use the technique describe above. I.e. draw pairs <img alt="(y_i,\mathbf{x}_i)" title="(y_i,\mathbf{x}_i)" src="https://s0.wp.com/latex.php?latex=%28y_i%2C%5Cmathbf%7Bx%7D_i%29&bg=ffffff&fg=000&s=0" class="latex"> randomly, uniformly (with probability <img alt="1/n" title="1/n" src="https://s0.wp.com/latex.php?latex=1%2Fn&bg=ffffff&fg=000&s=0" class="latex">) with replacement. Consider here the small dataset, just to visualize. For the <strong>b</strong> part of <strong>b</strong>agging, use the following code</p>
<div><table><tr><td><pre>1
2
3
4
5
</pre></td><td><pre>L_logit <span>=</span> <span>list</span><span>(</span><span>)</span>
n <span>=</span> <span>nrow</span><span>(</span><span>df</span><span>)</span>
<span>for</span><span>(</span>s <span>in</span> <span>1</span><span>:</span><span>1000</span><span>)</span><span>{</span>
  df_s <span>=</span> <span>df</span><span>[</span><span>sample</span><span>(</span><span>1</span><span>:</span>n,size<span>=</span>n,<span>replace</span><span>=</span>TRUE<span>)</span>,<span>]</span>
  L_logit<span>[</span><span>[</span>s<span>]</span><span>]</span> <span>=</span> <span>glm</span><span>(</span>y~., df_s, <span>family</span><span>=</span><span>binomial</span><span>)</span><span>}</span></pre></td></tr></table></div>
<p>Then we should aggregate over the 1000 models, to get the <strong>agg</strong> part of b<strong>agg</strong>ing,</p>
<div><table><tr><td><pre>1
2
3
</pre></td><td><pre>p <span>=</span> <span>function</span><span>(</span>x<span>)</span><span>{</span>
  nd<span>=</span><span>data.<span>frame</span></span><span>(</span>x1<span>=</span>x<span>[</span><span>1</span><span>]</span>, x2<span>=</span>x<span>[</span><span>2</span><span>]</span><span>)</span> 
  <span>unlist</span><span>(</span><span>lapply</span><span>(</span><span>1</span><span>:</span><span>1000</span>,<span>function</span><span>(</span>z<span>)</span> <span>predict</span><span>(</span>L_logit<span>[</span><span>[</span>z<span>]</span><span>]</span>,newdata<span>=</span>nd,type<span>=</span><span>&quot;response&quot;</span><span>)</span><span>)</span><span>)</span><span>}</span></pre></td></tr></table></div>
<p>We now have a prediction for any new observation</p>
<div><table><tr><td><pre>1
2
3
4
5
6
</pre></td><td><pre>vu <span>=</span> <span>seq</span><span>(</span><span>0</span>,<span>1</span>,<span>length</span><span>=</span><span>101</span><span>)</span>
vv <span>=</span> <span>outer</span><span>(</span>vu,vu,<span>Vectorize</span><span>(</span><span>function</span><span>(</span>x,y<span>)</span> <span>mean</span><span>(</span>p<span>(</span><span>c</span><span>(</span>x,y<span>)</span><span>)</span><span>)</span><span>)</span><span>)</span>
<span>image</span><span>(</span>vu,vu,vv,xlab<span>=</span><span>&quot;Variable 1&quot;</span>,ylab<span>=</span><span>&quot;Variable 2&quot;</span>,<span>col</span><span>=</span>clr10,breaks<span>=</span><span>(</span><span>0</span><span>:</span><span>10</span><span>)</span><span>/</span><span>10</span><span>)</span>
<span>points</span><span>(</span><span>df</span>$x1,<span>df</span>$x2,pch<span>=</span><span>19</span>,cex<span>=</span><span>1.5</span>,<span>col</span><span>=</span><span>&quot;white&quot;</span><span>)</span>
<span>points</span><span>(</span><span>df</span>$x1,<span>df</span>$x2,pch<span>=</span><span>c</span><span>(</span><span>1</span>,<span>19</span><span>)</span><span>[</span><span>1</span><span>+</span><span>(</span><span>df</span>$y<span>==</span><span>&quot;1&quot;</span><span>)</span><span>]</span>,cex<span>=</span><span>1.5</span><span>)</span>
<span>contour</span><span>(</span>vu,vu,vv,<span>levels</span> <span>=</span> .5,add<span>=</span>TRUE<span>)</span></pre></td></tr></table></div>
<p><img alt="" src="https://i1.wp.com/freakonometrics.hypotheses.org/files/2018/06/Capture-d%E2%80%99e%CC%81cran-2018-06-08-a%CC%80-14.43.50.png?w=456&ssl=1" data-recalc-dims="1"></p>
<h2>Bagging logistic regression #2</h2>
<p>Another technique that can be used to generate a bootstrap sample is to keep all <img alt="\mathbf{x}_i" title="\mathbf{x}_i" src="https://s0.wp.com/latex.php?latex=%5Cmathbf%7Bx%7D_i&bg=ffffff&fg=000&s=0" class="latex">‘s, but for each of them, to draw (randomly) a value for <img alt="y" title="y" src="https://s0.wp.com/latex.php?latex=y&bg=ffffff&fg=000&s=0" class="latex">, with<img alt="Y_{i,b}\sim\mathcal{B}(\widehat{m}_{S}(\mathbf{x}_i))" title="Y_{i,b}\sim\mathcal{B}(\widehat{m}_{S}(\mathbf{x}_i))" src="https://s0.wp.com/latex.php?latex=Y_%7Bi%2Cb%7D%5Csim%5Cmathcal%7BB%7D%28%5Cwidehat%7Bm%7D_%7BS%7D%28%5Cmathbf%7Bx%7D_i%29%29&bg=ffffff&fg=000&s=0" class="latex">since<img alt="\widehat{m}(\mathbf{x})=\mathbb{P}[Y=1|\mathbf{X}=\mathbf{x}]." title="\widehat{m}(\mathbf{x})=\mathbb{P}[Y=1|\mathbf{X}=\mathbf{x}]." src="https://s0.wp.com/latex.php?latex=%5Cwidehat%7Bm%7D%28%5Cmathbf%7Bx%7D%29%3D%5Cmathbb%7BP%7D%5BY%3D1%7C%5Cmathbf%7BX%7D%3D%5Cmathbf%7Bx%7D%5D.&bg=ffffff&fg=000&s=0" class="latex">Thus, the code for the <strong>b</strong> part of <strong>b</strong>agging algorithm is now</p>
<div><table><tr><td><pre>1
2
3
4
5
6
7
8
</pre></td><td><pre>L_logit <span>=</span> <span>list</span><span>(</span><span>)</span>
n <span>=</span> <span>nrow</span><span>(</span><span>df</span><span>)</span>
reg <span>=</span> <span>glm</span><span>(</span>y~x1<span>+</span>x2, <span>df</span>, <span>family</span><span>=</span><span>binomial</span><span>)</span>
<span>for</span><span>(</span>s <span>in</span> <span>1</span><span>:</span><span>100</span><span>)</span><span>{</span>
  df_s <span>=</span> <span>df</span>
  df_s$y <span>=</span> <span>factor</span><span>(</span><span>rbinom</span><span>(</span>n,size<span>=</span><span>1</span>,prob<span>=</span><span>predict</span><span>(</span>reg,type<span>=</span><span>&quot;response&quot;</span><span>)</span><span>)</span>,<span>labels</span><span>=</span><span>0</span><span>:</span><span>1</span><span>)</span>
  L_logit<span>[</span><span>[</span>s<span>]</span><span>]</span> <span>=</span> <span>glm</span><span>(</span>y~., df_s, <span>family</span><span>=</span><span>binomial</span><span>)</span>
<span>}</span></pre></td></tr></table></div>
<p>The <strong>agg</strong> part of b<strong>agg</strong>ing algorithm remains unchanged. Here we obtain</p>
<div><table><tr><td><pre>1
2
3
4
5
6
</pre></td><td><pre>vu <span>=</span> <span>seq</span><span>(</span><span>0</span>,<span>1</span>,<span>length</span><span>=</span><span>101</span><span>)</span>
vv <span>=</span> <span>outer</span><span>(</span>vu,vu,<span>Vectorize</span><span>(</span><span>function</span><span>(</span>x,y<span>)</span> <span>mean</span><span>(</span>p<span>(</span><span>c</span><span>(</span>x,y<span>)</span><span>)</span><span>)</span><span>)</span><span>)</span>
<span>image</span><span>(</span>vu,vu,vv,xlab<span>=</span><span>&quot;Variable 1&quot;</span>,ylab<span>=</span><span>&quot;Variable 2&quot;</span>,<span>col</span><span>=</span>clr10,breaks<span>=</span><span>(</span><span>0</span><span>:</span><span>10</span><span>)</span><span>/</span><span>10</span><span>)</span>
<span>points</span><span>(</span><span>df</span>$x1,<span>df</span>$x2,pch<span>=</span><span>19</span>,cex<span>=</span><span>1.5</span>,<span>col</span><span>=</span><span>&quot;white&quot;</span><span>)</span>
<span>points</span><span>(</span><span>df</span>$x1,<span>df</span>$x2,pch<span>=</span><span>c</span><span>(</span><span>1</span>,<span>19</span><span>)</span><span>[</span><span>1</span><span>+</span><span>(</span><span>df</span>$y<span>==</span><span>&quot;1&quot;</span><span>)</span><span>]</span>,cex<span>=</span><span>1.5</span><span>)</span>
<span>contour</span><span>(</span>vu,vu,vv,<span>levels</span> <span>=</span> .5,add<span>=</span>TRUE<span>)</span></pre></td></tr></table></div>
<p><img alt="" src="https://i0.wp.com/freakonometrics.hypotheses.org/files/2018/06/Capture-d%E2%80%99e%CC%81cran-2018-06-08-a%CC%80-06.51.02.png?w=456&ssl=1" data-recalc-dims="1"><br>
Of course, we can use that code we check the prediction obtain on the observations we have in our sample. Just to change, consider here the myocarde data. The entiere code is here</p>
<div><table><tr><td><pre>1
2
3
4
5
6
7
8
9
10
11
</pre></td><td><pre>L_logit <span>=</span> <span>list</span><span>(</span><span>)</span>
reg <span>=</span> <span>glm</span><span>(</span><span>as.<span>factor</span></span><span>(</span>PRONO<span>)</span>~., myocarde, <span>family</span><span>=</span><span>binomial</span><span>)</span>
<span>for</span><span>(</span>s <span>in</span> <span>1</span><span>:</span><span>1000</span><span>)</span><span>{</span>
  myocarde_s <span>=</span> myocarde
  myocarde_s$PRONO <span>=</span> <span>1</span><span>*</span><span>rbinom</span><span>(</span>n,size<span>=</span><span>1</span>,prob<span>=</span><span>predict</span><span>(</span>reg,type<span>=</span><span>&quot;response&quot;</span><span>)</span><span>)</span>
  L_logit<span>[</span><span>[</span>s<span>]</span><span>]</span> <span>=</span> <span>glm</span><span>(</span><span>as.<span>factor</span></span><span>(</span>PRONO<span>)</span>~., myocarde_s, <span>family</span><span>=</span><span>binomial</span><span>)</span>
<span>}</span>
p <span>=</span> <span>function</span><span>(</span>x<span>)</span><span>{</span>
  nd<span>=</span><span>data.<span>frame</span></span><span>(</span>FRCAR<span>=</span>x<span>[</span><span>1</span><span>]</span>, INCAR<span>=</span>x<span>[</span><span>2</span><span>]</span>, INSYS<span>=</span>x<span>[</span><span>3</span><span>]</span>, PRDIA<span>=</span>x<span>[</span><span>4</span><span>]</span>, 
                PAPUL<span>=</span>x<span>[</span><span>4</span><span>]</span>, PVENT<span>=</span>x<span>[</span><span>5</span><span>]</span>, REPUL<span>=</span>x<span>[</span><span>6</span><span>]</span><span>)</span> 
  <span>unlist</span><span>(</span><span>lapply</span><span>(</span><span>1</span><span>:</span><span>1000</span>,<span>function</span><span>(</span>z<span>)</span> <span>predict</span><span>(</span>L_logit<span>[</span><span>[</span>z<span>]</span><span>]</span>,newdata<span>=</span>nd,type<span>=</span><span>&quot;response&quot;</span><span>)</span><span>)</span><span>)</span><span>}</span></pre></td></tr></table></div>
<p>For the first observation, with our 1000 simulated datasets, and our 1000 models, we obtained the following estimation for the probability to die.</p>
<div><table><tr><td><pre>1
2
3
4
5
6
7
8
9
10
11
</pre></td><td><pre>histo <span>=</span> <span>function</span><span>(</span>i<span>)</span><span>{</span>
x <span>=</span> <span>as.<span>numeric</span></span><span>(</span>myocarde<span>[</span>i,<span>1</span><span>:</span><span>7</span><span>]</span><span>)</span>
v_x <span>=</span> p<span>(</span>x<span>)</span>
<span>hist</span><span>(</span>v_x,proba<span>=</span>TRUE,breaks<span>=</span><span>seq</span><span>(</span><span>0</span>,<span>1</span>,<span>by</span><span>=</span>.05<span>)</span>,xlab<span>=</span><span>&quot;&quot;</span>,main<span>=</span><span>&quot;&quot;</span>,
<span>col</span><span>=</span><span>rep</span><span>(</span><span>c</span><span>(</span><span>rgb</span><span>(</span><span>0</span>,<span>0</span>,<span>1</span>,.4<span>)</span>,<span>rgb</span><span>(</span><span>1</span>,<span>0</span>,<span>0</span>,.4<span>)</span><span>)</span>,each<span>=</span><span>10</span><span>)</span>,ylim<span>=</span><span>c</span><span>(</span><span>0</span>,<span>5</span><span>)</span><span>)</span>
<span>segments</span><span>(</span><span>mean</span><span>(</span>v_x<span>)</span>,<span>0</span>,<span>mean</span><span>(</span>v_x<span>)</span>,<span>5</span>,<span>col</span><span>=</span><span>&quot;red&quot;</span>,lty<span>=</span><span>2</span><span>)</span>
<span>points</span><span>(</span>myocarde$PRONO<span>[</span>i<span>]</span>,<span>0</span>,pch<span>=</span><span>19</span>,cex<span>=</span><span>2</span><span>)</span>
xi <span>=</span> <span>round</span><span>(</span><span>mean</span><span>(</span>v_x.5<span>)</span><span>*</span><span>1000</span><span>)</span><span>/</span><span>10</span>
<span>text</span><span>(</span>.75,<span>-</span>.1,<span>paste</span><span>(</span>xi,<span>&quot;%&quot;</span>,sep<span>=</span><span>&quot;&quot;</span><span>)</span>,<span>col</span><span>=</span><span>rgb</span><span>(</span><span>1</span>,<span>0</span>,<span>0</span>,.6<span>)</span><span>)</span><span>}</span>
histo<span>(</span><span>1</span><span>)</span>
histo<span>(</span><span>4</span><span>)</span></pre></td></tr></table></div>
<p>Hence, for the first observation, in 77.8% of the models, the predicted probability was higher than 50%, and the average probability was actually close to 75%.<br>
<img alt="" src="https://i0.wp.com/freakonometrics.hypotheses.org/files/2018/06/Capture-d%E2%80%99e%CC%81cran-2018-06-08-a%CC%80-14.45.55.png?w=456&ssl=1" data-recalc-dims="1"><br>
or, for observation 22, predictions very close to the first one (except that the first one died, while the 22nd survived)</p>
<div><table><tr><td><pre>1
2
</pre></td><td><pre>histo<span>(</span><span>23</span><span>)</span>
histo<span>(</span><span>11</span><span>)</span></pre></td></tr></table></div>
<p>and, we observe here<br>
<img alt="" src="https://i2.wp.com/freakonometrics.hypotheses.org/files/2018/06/Capture-d%E2%80%99e%CC%81cran-2018-06-08-a%CC%80-14.47.20.png?w=456&ssl=1" data-recalc-dims="1"></p>
<h2>Bagging trees</h2>
<p>Let’s now get back on our trees, mentioned in the <a rel="nofollow" href="https://freakonometrics.hypotheses.org/52776" target="_blank">previous post</a>. Bagging was introduced in 1994 by Leo Breiman in <a rel="nofollow" href="https://www.stat.berkeley.edu/~breiman/bagging.pdf" target="_blank">Bagging Predictors</a>. If the first section describes the procedure, the second one introduces “Bagging Classification Trees”. Trees are nice for interpretation, but most of the time, they are rather poor predictors. The idea of bagging was to improve the accuracy of classification trees.</p>
<p>The idea of <strong>b</strong>agging to to generate a lot of trees</p>
<div><table><tr><td><pre>1
2
3
4
5
6
7
8
9
</pre></td><td><pre>clr12 <span>=</span> <span>c</span><span>(</span><span>&quot;#8dd3c7&quot;</span>,<span>&quot;#ffffb3&quot;</span>,<span>&quot;#bebada&quot;</span>,<span>&quot;#fb8072&quot;</span>,<span>&quot;#80b1d3&quot;</span>,<span>&quot;#fdb462&quot;</span>,<span>&quot;#b3de69&quot;</span>,<span>&quot;#fccde5&quot;</span>,<span>&quot;#d9d9d9&quot;</span>,<span>&quot;#bc80bd&quot;</span>,<span>&quot;#ccebc5&quot;</span>,<span>&quot;#ffed6f&quot;</span><span>)</span>
n <span>=</span> <span>nrow</span><span>(</span>myocarde<span>)</span>
<span>par</span><span>(</span>mfrow<span>=</span><span>c</span><span>(</span><span>4</span>,<span>3</span><span>)</span><span>)</span>
sed<span>=</span><span>c</span><span>(</span><span>1</span>,<span>2</span>,<span>4</span>,<span>5</span>,<span>6</span>,<span>10</span>,<span>11</span>,<span>21</span>,<span>22</span>,<span>24</span>,<span>27</span>,<span>28</span>,<span>30</span><span>)</span>
<span>for</span><span>(</span>i <span>in</span> <span>1</span><span>:</span><span>12</span><span>)</span><span>{</span>
  <span>set.<span>seed</span></span><span>(</span>sed<span>[</span>i<span>]</span><span>)</span>
idx <span>=</span> <span>sample</span><span>(</span><span>1</span><span>:</span>n, size<span>=</span>n, <span>replace</span><span>=</span>TRUE<span>)</span>
cart <span>=</span>  rpart<span>(</span>PRONO~., myocarde<span>[</span>idx,<span>]</span><span>)</span>
prp<span>(</span>cart,type<span>=</span><span>2</span>,extra<span>=</span><span>1</span>,box.<span>col</span><span>=</span>clr12<span>[</span>i<span>]</span><span>)</span><span>}</span></pre></td></tr></table></div>
<p><img src="https://i0.wp.com/freakonometrics.hypotheses.org/files/2018/06/Capture-d%E2%80%99e%CC%81cran-2018-06-07-a%CC%80-15.37.53.png?w=456&ssl=1" data-recalc-dims="1"><br>
The strategie is actually the same as before. For the <strong>b</strong>ootstrap part, store the tree in a list</p>
<div><table><tr><td><pre>1
2
3
4
5
</pre></td><td><pre>L_tree <span>=</span> <span>list</span><span>(</span><span>)</span>
<span>for</span><span>(</span>s <span>in</span> <span>1</span><span>:</span><span>1000</span><span>)</span><span>{</span>
  idx <span>=</span> <span>sample</span><span>(</span><span>1</span><span>:</span>n, size<span>=</span>n, <span>replace</span><span>=</span>TRUE<span>)</span>
  L_tree<span>[</span><span>[</span>s<span>]</span><span>]</span> <span>=</span> rpart<span>(</span><span>as.<span>factor</span></span><span>(</span>PRONO<span>)</span>~., myocarde<span>[</span>idx,<span>]</span><span>)</span>
<span>}</span></pre></td></tr></table></div>
<p>and for the <strong>agg</strong>regation part, just take the average of predicted probabilities</p>
<div><table><tr><td><pre>1
2
3
4
</pre></td><td><pre>p <span>=</span> <span>function</span><span>(</span>x<span>)</span><span>{</span>
  nd<span>=</span><span>data.<span>frame</span></span><span>(</span>FRCAR<span>=</span>x<span>[</span><span>1</span><span>]</span>, INCAR<span>=</span>x<span>[</span><span>2</span><span>]</span>, INSYS<span>=</span>x<span>[</span><span>3</span><span>]</span>, PRDIA<span>=</span>x<span>[</span><span>4</span><span>]</span>, 
                PAPUL<span>=</span>x<span>[</span><span>4</span><span>]</span>, PVENT<span>=</span>x<span>[</span><span>5</span><span>]</span>, REPUL<span>=</span>x<span>[</span><span>6</span><span>]</span><span>)</span> 
  <span>unlist</span><span>(</span><span>lapply</span><span>(</span><span>1</span><span>:</span><span>1000</span>,<span>function</span><span>(</span>z<span>)</span> <span>predict</span><span>(</span>L_tree<span>[</span><span>[</span>z<span>]</span><span>]</span>,newdata<span>=</span>nd,type<span>=</span><span>&quot;prob&quot;</span><span>)</span><span>[</span>,<span>2</span><span>]</span><span>)</span><span>)</span><span>}</span></pre></td></tr></table></div>
<p>Because with this example, we cannot visualize predictions, let us run the same code on the smaller dataset</p>
<div><table><tr><td><pre>1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
</pre></td><td><pre>L_tree <span>=</span> <span>list</span><span>(</span><span>)</span>
n <span>=</span> <span>nrow</span><span>(</span><span>df</span><span>)</span>
<span>for</span><span>(</span>s <span>in</span> <span>1</span><span>:</span><span>1000</span><span>)</span><span>{</span>
  idx <span>=</span> <span>sample</span><span>(</span><span>1</span><span>:</span>n, size<span>=</span>n, <span>replace</span><span>=</span>TRUE<span>)</span>
  L_tree<span>[</span><span>[</span>s<span>]</span><span>]</span> <span>=</span> rpart<span>(</span>y~x1<span>+</span>x2, <span>df</span><span>[</span>idx,<span>]</span>,control <span>=</span> rpart.<span>control</span><span>(</span>cp <span>=</span> <span>0.25</span>,
minsplit <span>=</span> <span>2</span><span>)</span><span>)</span>
<span>}</span>
p <span>=</span> <span>function</span><span>(</span>x<span>)</span><span>{</span>
  nd<span>=</span><span>data.<span>frame</span></span><span>(</span>x1<span>=</span>x<span>[</span><span>1</span><span>]</span>, x2<span>=</span>x<span>[</span><span>2</span><span>]</span><span>)</span> 
  <span>unlist</span><span>(</span><span>lapply</span><span>(</span><span>1</span><span>:</span><span>1000</span>,<span>function</span><span>(</span>z<span>)</span> <span>predict</span><span>(</span>L_tree<span>[</span><span>[</span>z<span>]</span><span>]</span>,newdata<span>=</span>nd,type<span>=</span><span>&quot;prob&quot;</span><span>)</span><span>[</span>,<span>2</span><span>]</span><span>)</span><span>)</span><span>}</span>
vu<span>=</span><span>seq</span><span>(</span><span>0</span>,<span>1</span>,<span>length</span><span>=</span><span>101</span><span>)</span>
vv<span>=</span><span>outer</span><span>(</span>vu,vu,<span>Vectorize</span><span>(</span><span>function</span><span>(</span>x,y<span>)</span> <span>mean</span><span>(</span>p<span>(</span><span>c</span><span>(</span>x,y<span>)</span><span>)</span><span>)</span><span>)</span><span>)</span>
<span>image</span><span>(</span>vu,vu,vv,xlab<span>=</span><span>&quot;Variable 1&quot;</span>,ylab<span>=</span><span>&quot;Variable 2&quot;</span>,<span>col</span><span>=</span>clr10,breaks<span>=</span><span>(</span><span>0</span><span>:</span><span>10</span><span>)</span><span>/</span><span>10</span><span>)</span>
<span>points</span><span>(</span><span>df</span>$x1,<span>df</span>$x2,pch<span>=</span><span>19</span>,cex<span>=</span><span>1.5</span>,<span>col</span><span>=</span><span>&quot;white&quot;</span><span>)</span>
<span>points</span><span>(</span><span>df</span>$x1,<span>df</span>$x2,pch<span>=</span><span>c</span><span>(</span><span>1</span>,<span>19</span><span>)</span><span>[</span><span>1</span><span>+</span><span>(</span><span>df</span>$y<span>==</span><span>&quot;1&quot;</span><span>)</span><span>]</span>,cex<span>=</span><span>1.5</span><span>)</span>
<span>contour</span><span>(</span>vu,vu,vv,<span>levels</span> <span>=</span> .5,add<span>=</span>TRUE<span>)</span></pre></td></tr></table></div>
<p><img src="https://i1.wp.com/freakonometrics.hypotheses.org/files/2018/06/Capture-d%E2%80%99e%CC%81cran-2018-06-08-a%CC%80-11.42.21.png?w=456&ssl=1" data-recalc-dims="1"> </p>
<h2>Fronm bags to forest</h2>
<p>Here, we grew a lot of trees, but it is not <em>stricto sensus</em> a <a rel="nofollow" href="https://en.wikipedia.org/wiki/Random_forest" target="_blank">random forest</a> algorithm, as introduced in 1995, in <a rel="nofollow" href="https://dl.acm.org/citation.cfm?id=844379.844681" target="_blank">Random decision forests</a>. Actually, the difference is in the creation of decision trees. To understand what happens, get back to the <a rel="nofollow" href="https://freakonometrics.hypotheses.org/52776" target="_blank">previous post</a> on classification trees. As we’ve seen, when we have a node, we look at possible splits : we consider all possible variable, and all possible threshold. The startegy here will be to draw randomly <img alt="k" title="k" src="https://s0.wp.com/latex.php?latex=k&bg=ffffff&fg=000&s=0" class="latex"> variables out of <img alt="p" title="p" src="https://s0.wp.com/latex.php?latex=p&bg=ffffff&fg=000&s=0" class="latex"> (with of course <img alt="" title="" src="https://s0.wp.com/latex.php?latex=&bg=ffffff&fg=000&s=0" class="latex">k&lt;p[/latex], for instance [latex]k=\sqrt{p}[/latex]). That's interesting in high dimension, because at each split, we should look for all variables, all cutoffs, and that can take quite some time (especially with the bootstrap procedure, where the goal will be to grow 1000 trees).</p>
<p><a rel="nofollow" href="https://freakonometrics.hypotheses.org/52782" target="_blank">To be continued…</a></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://freakonometrics.hypotheses.org/52777"> R-english – Freakonometrics</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/2RywyzBNKx8" height="1"></a></div>