<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/boost-boosting-smooth-trees-a-new-machine-learning-model-for-partial-effect-estimation-in-nonlinear-regressions/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://insightr.wordpress.com/2018/08/14/boost-boosting-smooth-trees-a-new-machine-learning-model-for-partial-effect-estimation-in-nonlinear-regressions/"> R – insightR</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<div><strong>By Gabriel Vasconcelos and Yuri Fonseca</strong></div>
</div>
<p> </p>
<p>We are happy to introduce our new machine learning method called Boosting Smooth Trees (BooST) (<a rel="nofollow" href="https://arxiv.org/abs/1808.03698" target="_blank">full article here</a>). This model was a joint work with professors Marcelo Medeiros and Álvaro Veiga. The BooST uses a different type of regression tree that allows us to estimate the derivatives of very general nonlinear models. In other words, the model is differentiable and it has an analytical solution. The consequence is that now we can estimate partial effects of a characteristic on the response variable, which provide us much more interpretation than traditional importance measures.</p>
<p><span></span></p>
<p>The idea behind the BooST is to replace traditional Classification and Regression Trees (CART), which are not differentiable, by Smooth logistic trees. We show that with this adaptation the BooST is a consistent estimator of the model’s derivatives under some assumptions.</p>
<h2>Example</h2>
<p>The example below shows that the BooST is very good to recover the derivatives of nonlinear functions. The data was generated from the following model:</p>
<p><img alt="displaystyle y_i = cos( pi [x_{i,1}+x_{i,2}])+ varepsilon_i" title="displaystyle y_i = cos( pi [x_{i,1}+x_{i,2}])+ varepsilon_i" src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle+y_i+%3D+%5Ccos%28+%5Cpi+%5Bx_%7Bi%2C1%7D%2Bx_%7Bi%2C2%7D%5D%29%2B+%5Cvarepsilon_i&bg=ffffff&%23038;fg=333333&%23038;s=0" class="latex"></p>
<p>where <img alt="x_{i,1} sim N(0,1)" title="x_{i,1} sim N(0,1)" src="https://s0.wp.com/latex.php?latex=x_%7Bi%2C1%7D+%5Csim+N%280%2C1%29&bg=ffffff&%23038;fg=333333&%23038;s=0" class="latex">, <img alt="x_{i,2}" title="x_{i,2}" src="https://s0.wp.com/latex.php?latex=x_%7Bi%2C2%7D&bg=ffffff&%23038;fg=333333&%23038;s=0" class="latex"> is a Bernoulli with <img alt="p = 0.5" title="p = 0.5" src="https://s0.wp.com/latex.php?latex=p+%3D+0.5&bg=ffffff&%23038;fg=333333&%23038;s=0" class="latex"> and <img alt="varepsilon_i sim N(0,1)" title="varepsilon_i sim N(0,1)" src="https://s0.wp.com/latex.php?latex=%5Cvarepsilon_i+%5Csim+N%280%2C1%29&bg=ffffff&%23038;fg=333333&%23038;s=0" class="latex">. Note that this is not an easy problem. The function that generates the data is not monotonic, very smooth and nonlinear. The Figure below shows how the BooST estimates the model and its derivatives with respect to <img alt="x_{i,1}" title="x_{i,1}" src="https://s0.wp.com/latex.php?latex=x_%7Bi%2C1%7D&bg=ffffff&%23038;fg=333333&%23038;s=0" class="latex">. We simulated 1000 data points and forced it to have an R2 of 0.5 (half of the variation in the data is noise).</p>
<p><img data-orig-file="https://insightr.files.wordpress.com/2018/08/examplecosine05.png?w=456" data-orig-size="829,624" data-medium-file="https://insightr.files.wordpress.com/2018/08/examplecosine05.png?w=456?w=300" data-attachment-id="851" data-image-meta="{" data-image-description="" src="https://insightr.files.wordpress.com/2018/08/examplecosine05.png?w=456" data-permalink="https://insightr.wordpress.com/examplecosine05/" data-large-file="https://insightr.files.wordpress.com/2018/08/examplecosine05.png?w=456?w=640" alt="examplecosine05" data-image-title="examplecosine05" sizes="(max-width: 640px) 100vw, 640px" data-comments-opened="1" class="wp-image-851" data-recalc-dims="1"></p>
<h2>Real data example</h2>
<p>To give a more practical example, suppose we want to estimate the effects of price changes on sales without any prior knowledge of the demand function. It is natural to think that these effects will depend on the current price level and possible some characteristics of the product. The BooST will estimate the partial effects of price on sales conditional to the current (or any) price and any other controls we choose. We will write a post specifically on this example in the future.</p>
<p>Consider yet an other example where we want to estimate how the price of a house changes as we move on the latitude or longitude. This is precisely what we did in the figure below using data from house sales in Melbourne (Dataset scrapped from the web by <a rel="nofollow" href="https://www.kaggle.com/anthonypino/melbourne-housing-market/home" target="_blank">Tony Pino</a>). The figure shows that if we are south of the CBD and move north the prices increase a lot. However, if we keep moving north prices start to decrease as we move away from the CBD.</p>
<p><img data-orig-file="https://insightr.files.wordpress.com/2018/08/delat.png?w=456" data-orig-size="682,626" data-medium-file="https://insightr.files.wordpress.com/2018/08/delat.png?w=456?w=300" data-attachment-id="852" data-image-meta="{" data-image-description="" src="https://insightr.files.wordpress.com/2018/08/delat.png?w=456" data-permalink="https://insightr.wordpress.com/delat/" data-large-file="https://insightr.files.wordpress.com/2018/08/delat.png?w=456?w=640" alt="delat" data-image-title="delat" sizes="(max-width: 640px) 100vw, 640px" data-comments-opened="1" class="wp-image-852" data-recalc-dims="1"></p>
<h2>Other considerations</h2>
<p>The BooST is also very good for forecasting. We will come back to this topic with examples on future posts. Just keep in mind that if the number of characteristics is to big it may improve forecasting accuracy but you will probably loose some interpretation of the partial effects not because of the model itself but because of some theoretical problems you may have.</p>
<p>For now we have an implementation of the BooST in R, which is fully documented and straightforward to use. It can be downloaded and installed from:</p>
<pre>library(devtools)
install_github(&quot;gabrielrvsc/BooST&quot;)
</pre>
<p>Note that this implementation is not very fast and it is not adequate for very big problems with to many variables. We already have a faster Julia implementation that  will be published soon. The even faster C++ version will take some time but it is also coming. In the next posts we will explore some empirical applications with codes to replicate our examples.</p>
<h2>References</h2>
<p>Fonseca, Y.; Medeiros, M.; Vasconcelos; G.; Veiga, A. “BooST: Boosting Smooth Trees for Partial Effect Estimation in Nonlinear Regressions” arXiv preprint available at <a rel="nofollow" href="https://arxiv.org/abs/1808.03698" target="_blank">https://arxiv.org/abs/1808.03698</a> (2018).</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://insightr.wordpress.com/2018/08/14/boost-boosting-smooth-trees-a-new-machine-learning-model-for-partial-effect-estimation-in-nonlinear-regressions/"> R – insightR</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/HoR9sPmjd5U" height="1"></a></div>