<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/ensemble-learning-for-time-series-forecasting-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://petolau.github.io/Ensemble-of-trees-for-forecasting-time-series/"> Peter Laurinec</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Ensemble learning methods are widely used nowadays for its predictive performance improvement. Ensemble learning combines multiple predictions (forecasts) from one or multiple methods to overcome accuracy of simple prediction and to avoid possible overfit. In the domain of time series forecasting, we have somehow obstructed situation because of dynamic changes in coming data. However, when a single regression model is used for forecasting, time dependency is not the obstacle, we can tune it at current time of a sliding window. For this reason, in this post, I will describe you two simple ensemble learning methods – Bagging and Random Forest. Bagging will be used with combination of two simple regression trees methods used in the previous post (RPART and CTREE). I will not repeat most of the things mentioned there so check it first if you didn’t make it already: <a rel="nofollow" href="https://petolau.github.io/Regression-trees-for-forecasting-time-series-in-R/" target="_blank">Using regression trees for forecasting double-seasonal time series with trend</a>.</p>
<p>You will learn in this post how to:</p>
<ul>
<li>tune simple regression trees methods by Bagging</li>
<li>set important hyperparameters of decision tree methods and randomized them in Bagging</li>
<li>use Random Forest method for forecasting time series</li>
<li>set hyperparameters of Random Forest and find it optimally by grid search method</li>
</ul>
<h2>Time series data of electricity consumption</h2>
<p>As in previous posts, I will use smart meter data of electricity consumption for demonstrating forecasting of seasonal time series. The dataset of aggregated electricity load of consumers from an anonymous area is used. Time series data have the length of 17 weeks.</p>
<p>Firstly, let’s scan all of the needed packages for data analysis, modeling and visualizing.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>feather</span><span>)</span><span> </span><span># data import</span><span>
</span><span>library</span><span>(</span><span>data.table</span><span>)</span><span> </span><span># data handle</span><span>
</span><span>library</span><span>(</span><span>rpart</span><span>)</span><span> </span><span># decision tree method</span><span>
</span><span>library</span><span>(</span><span>party</span><span>)</span><span> </span><span># decision tree method</span><span>
</span><span>library</span><span>(</span><span>forecast</span><span>)</span><span> </span><span># forecasting methods</span><span>
</span><span>library</span><span>(</span><span>randomForest</span><span>)</span><span> </span><span># ensemble learning method</span><span>
</span><span>library</span><span>(</span><span>ggplot2</span><span>)</span><span> </span><span># visualizations</span></code></pre>
</figure>
<p>Now read the mentioned time series data by <code>read_feather</code> to one <code>data.table</code>. The dataset can be found on <a rel="nofollow" href="https://github.com/PetoLau/petolau.github.io/tree/master/_rmd" target="_blank">my github repo</a>, the name of the file is <em>DT_load_17weeks</em>.</p>
<figure>
<pre><code data-lang="r"><span>DT</span><span> </span><span>&lt;-</span><span> </span><span>as.data.table</span><span>(</span><span>read_feather</span><span>(</span><span>&quot;DT_load_17weeks&quot;</span><span>))</span></code></pre>
</figure>
<p>And store information of the date and period of time series that is 48.</p>
<figure>
<pre><code data-lang="r"><span>n_date</span><span> </span><span>&lt;-</span><span> </span><span>unique</span><span>(</span><span>DT</span><span>[,</span><span> </span><span>date</span><span>])</span><span>
</span><span>period</span><span> </span><span>&lt;-</span><span> </span><span>48</span></code></pre>
</figure>
<p>For data visualization needs, store my favorite ggplot theme settings by function <code>theme</code>.</p>
<figure>
<pre><code data-lang="r"><span>theme_ts</span><span> </span><span>&lt;-</span><span> </span><span>theme</span><span>(</span><span>panel.border</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>NA</span><span>,</span><span> </span><span>colour</span><span> </span><span>=</span><span> </span><span>&quot;grey10&quot;</span><span>),</span><span>
</span><span>panel.background</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>panel.grid.minor</span><span> </span><span>=</span><span> </span><span>element_line</span><span>(</span><span>colour</span><span> </span><span>=</span><span> </span><span>&quot;grey85&quot;</span><span>),</span><span>
</span><span>panel.grid.major</span><span> </span><span>=</span><span> </span><span>element_line</span><span>(</span><span>colour</span><span> </span><span>=</span><span> </span><span>&quot;grey85&quot;</span><span>),</span><span>
</span><span>panel.grid.major.x</span><span> </span><span>=</span><span> </span><span>element_line</span><span>(</span><span>colour</span><span> </span><span>=</span><span> </span><span>&quot;grey85&quot;</span><span>),</span><span>
</span><span>axis.text</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>13</span><span>,</span><span> </span><span>face</span><span> </span><span>=</span><span> </span><span>&quot;bold&quot;</span><span>),</span><span>
</span><span>axis.title</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>15</span><span>,</span><span> </span><span>face</span><span> </span><span>=</span><span> </span><span>&quot;bold&quot;</span><span>),</span><span>
</span><span>plot.title</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>16</span><span>,</span><span> </span><span>face</span><span> </span><span>=</span><span> </span><span>&quot;bold&quot;</span><span>),</span><span>
</span><span>strip.text</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>16</span><span>,</span><span> </span><span>face</span><span> </span><span>=</span><span> </span><span>&quot;bold&quot;</span><span>),</span><span>
</span><span>strip.background</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>colour</span><span> </span><span>=</span><span> </span><span>&quot;black&quot;</span><span>),</span><span>
</span><span>legend.text</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>15</span><span>),</span><span>
</span><span>legend.title</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>16</span><span>,</span><span> </span><span>face</span><span> </span><span>=</span><span> </span><span>&quot;bold&quot;</span><span>),</span><span>
</span><span>legend.background</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;white&quot;</span><span>),</span><span>
</span><span>legend.key</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;white&quot;</span><span>))</span></code></pre>
</figure>
<p>I will use three weeks of data of electricity consumption for training regression trees methods. Forecasts will be performed to one day ahead. Let’s extract train and test set from the dataset.</p>
<figure>
<pre><code data-lang="r"><span>data_train</span><span> </span><span>&lt;-</span><span> </span><span>DT</span><span>[</span><span>date</span><span> </span><span>%in%</span><span> </span><span>n_date</span><span>[</span><span>1</span><span>:</span><span>21</span><span>]]</span><span>
</span><span>data_test</span><span> </span><span>&lt;-</span><span> </span><span>DT</span><span>[</span><span>date</span><span> </span><span>%in%</span><span> </span><span>n_date</span><span>[</span><span>22</span><span>]]</span></code></pre>
</figure>
<p>And visualize the train part:</p>
<figure>
<pre><code data-lang="r"><span>ggplot</span><span>(</span><span>data_train</span><span>,</span><span> </span><span>aes</span><span>(</span><span>date_time</span><span>,</span><span> </span><span>value</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>()</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Date&quot;</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Load (kW)&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_ts</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-7" src="https://i1.wp.com/petolau.github.io/images/post_6/unnamed-chunk-7-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<h2>Bagging</h2>
<p><a rel="nofollow" href="https://en.wikipedia.org/wiki/Bootstrap_aggregating" target="_blank"><strong>Bagging</strong></a> or bootstrap aggregating, is ensemble learning meta-algorithm used to improve prediction accuracy and to overcome (avoid) overfitting. The algorithm is very simple. The first step is sampling a training dataset with replacement with some defined sample ratio (e.g. 0.7). Then a model is trained on a new train set. This procedure is repeated <em>N_boot</em> times (e.g. 100). Final ensemble prediction is just average of <em>N_boot</em> predictions. For aggregating predictions, the median can be also used and will be used in this post.</p>
<h3>Bagging + RPART</h3>
<p>The first “bagged” method is RPART (CART) tree. Training set consists of lagged electricity load by one day and double-seasonal Fourier terms (daily and weekly seasonality). Electricity consumption is firstly detrended by STL decomposition and trend part is forecasted (modeled) by ARIMA (<code>auto.arima</code> function). Seasonal and remainder part is then forecasted by regression tree model. More detailed description and explanations are in my <a rel="nofollow" href="https://petolau.github.io/Regression-trees-for-forecasting-time-series-in-R/" target="_blank">previous post about regression trees methods</a>. Let’s define train and test data (by <code>matrix_train</code> and <code>matrix_test</code>).</p>
<figure>
<pre><code data-lang="r"><span>data_ts</span><span> </span><span>&lt;-</span><span> </span><span>ts</span><span>(</span><span>data_train</span><span>$</span><span>value</span><span>,</span><span> </span><span>freq</span><span> </span><span>=</span><span> </span><span>period</span><span> </span><span>*</span><span> </span><span>7</span><span>)</span><span>
</span><span>decomp_ts</span><span> </span><span>&lt;-</span><span> </span><span>stl</span><span>(</span><span>data_ts</span><span>,</span><span> </span><span>s.window</span><span> </span><span>=</span><span> </span><span>&quot;periodic&quot;</span><span>,</span><span> </span><span>robust</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>$</span><span>time.series</span><span>
</span><span>trend_part</span><span> </span><span>&lt;-</span><span> </span><span>ts</span><span>(</span><span>decomp_ts</span><span>[,</span><span>2</span><span>])</span><span>
</span><span>trend_fit</span><span> </span><span>&lt;-</span><span> </span><span>auto.arima</span><span>(</span><span>trend_part</span><span>)</span><span> </span><span># ARIMA</span><span>
</span><span>trend_for</span><span> </span><span>&lt;-</span><span> </span><span>as.vector</span><span>(</span><span>forecast</span><span>(</span><span>trend_fit</span><span>,</span><span> </span><span>period</span><span>)</span><span>$</span><span>mean</span><span>)</span><span> </span><span># trend forecast</span><span>
</span><span>data_msts</span><span> </span><span>&lt;-</span><span> </span><span>msts</span><span>(</span><span>data_train</span><span>$</span><span>value</span><span>,</span><span> </span><span>seasonal.periods</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>period</span><span>,</span><span> </span><span>period</span><span>*</span><span>7</span><span>))</span><span>
</span><span>K</span><span> </span><span>&lt;-</span><span> </span><span>2</span><span>
</span><span>fuur</span><span> </span><span>&lt;-</span><span> </span><span>fourier</span><span>(</span><span>data_msts</span><span>,</span><span> </span><span>K</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>K</span><span>,</span><span> </span><span>K</span><span>))</span><span> </span><span># Fourier features to model (Daily and Weekly)</span><span>
</span><span>N</span><span> </span><span>&lt;-</span><span> </span><span>nrow</span><span>(</span><span>data_train</span><span>)</span><span>
</span><span>window</span><span> </span><span>&lt;-</span><span> </span><span>(</span><span>N</span><span> </span><span>/</span><span> </span><span>period</span><span>)</span><span> </span><span>-</span><span> </span><span>1</span><span>
</span><span>new_load</span><span> </span><span>&lt;-</span><span> </span><span>rowSums</span><span>(</span><span>decomp_ts</span><span>[,</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span>3</span><span>)])</span><span> </span><span># detrended original time series</span><span>
</span><span>lag_seas</span><span> </span><span>&lt;-</span><span> </span><span>decomp_ts</span><span>[</span><span>1</span><span>:</span><span>(</span><span>period</span><span>*</span><span>window</span><span>),</span><span> </span><span>1</span><span>]</span><span> </span><span># lag feature to model</span><span>
</span><span>matrix_train</span><span> </span><span>&lt;-</span><span> </span><span>data.table</span><span>(</span><span>Load</span><span> </span><span>=</span><span> </span><span>tail</span><span>(</span><span>new_load</span><span>,</span><span> </span><span>window</span><span>*</span><span>period</span><span>),</span><span>
</span><span>fuur</span><span>[(</span><span>period</span><span> </span><span>+</span><span> </span><span>1</span><span>)</span><span>:</span><span>N</span><span>,],</span><span>
</span><span>Lag</span><span> </span><span>=</span><span> </span><span>lag_seas</span><span>)</span><span>
</span><span># create testing data matrix</span><span>
</span><span>test_lag</span><span> </span><span>&lt;-</span><span> </span><span>decomp_ts</span><span>[((</span><span>period</span><span>*</span><span>window</span><span>)</span><span>+1</span><span>)</span><span>:</span><span>N</span><span>,</span><span> </span><span>1</span><span>]</span><span>
</span><span>fuur_test</span><span> </span><span>&lt;-</span><span> </span><span>fourier</span><span>(</span><span>data_msts</span><span>,</span><span> </span><span>K</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>K</span><span>,</span><span> </span><span>K</span><span>),</span><span> </span><span>h</span><span> </span><span>=</span><span> </span><span>period</span><span>)</span><span>
</span><span>matrix_test</span><span> </span><span>&lt;-</span><span> </span><span>data.table</span><span>(</span><span>fuur_test</span><span>,</span><span>
</span><span>Lag</span><span> </span><span>=</span><span> </span><span>test_lag</span><span>)</span></code></pre>
</figure>
<p>I will perform 100 bootstrapped forecasts by RPART, that will be stored in the matrix of size \( 100\times 48 \) (<code>pred_mat</code>). Additional four parameters will be also sampled (randomized) to avoid overfitting. Sample ratio for sampling train set is sampled in the range from 0.7 to 0.9. Three hyperparameters of RPART are sampled also: <code>minsplit</code>, <code>maxdepth</code> and complexity parameter <code>cp</code>. Hyperparameters are sampled around values set on my previous post modeling.</p>
<figure>
<pre><code data-lang="r"><span>N_boot</span><span> </span><span>&lt;-</span><span> </span><span>100</span><span> </span><span># number of bootstraps</span><span>
</span><span>pred_mat</span><span> </span><span>&lt;-</span><span> </span><span>matrix</span><span>(</span><span>0</span><span>,</span><span> </span><span>nrow</span><span> </span><span>=</span><span> </span><span>N_boot</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>period</span><span>)</span><span>
</span><span>for</span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>N_boot</span><span>)</span><span> </span><span>{</span><span>
</span><span>matrixSam</span><span> </span><span>&lt;-</span><span> </span><span>matrix_train</span><span>[</span><span>sample</span><span>(</span><span>1</span><span>:</span><span>(</span><span>N</span><span>-</span><span>period</span><span>),</span><span>
</span><span>floor</span><span>((</span><span>N</span><span>-</span><span>period</span><span>)</span><span> </span><span>*</span><span> </span><span>sample</span><span>(</span><span>seq</span><span>(</span><span>0.7</span><span>,</span><span> </span><span>0.9</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>0.01</span><span>),</span><span> </span><span>1</span><span>)),</span><span>
</span><span>replace</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)]</span><span> </span><span># sampling with sampled ratio from 0.7 to 0.9</span><span>
</span><span>tree_bag</span><span> </span><span>&lt;-</span><span> </span><span>rpart</span><span>(</span><span>Load</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>matrixSam</span><span>,</span><span>
</span><span>control</span><span> </span><span>=</span><span> </span><span>rpart.control</span><span>(</span><span>minsplit</span><span> </span><span>=</span><span> </span><span>sample</span><span>(</span><span>2</span><span>:</span><span>3</span><span>,</span><span> </span><span>1</span><span>),</span><span>
</span><span>maxdepth</span><span> </span><span>=</span><span> </span><span>sample</span><span>(</span><span>26</span><span>:</span><span>30</span><span>,</span><span> </span><span>1</span><span>),</span><span>
</span><span>cp</span><span> </span><span>=</span><span> </span><span>sample</span><span>(</span><span>seq</span><span>(</span><span>0.0000009</span><span>,</span><span> </span><span>0.00001</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>0.0000001</span><span>),</span><span> </span><span>1</span><span>)))</span><span>
</span><span># new data and prediction</span><span>
</span><span>pred_mat</span><span>[</span><span>i</span><span>,]</span><span> </span><span>&lt;-</span><span> </span><span>predict</span><span>(</span><span>tree_bag</span><span>,</span><span> </span><span>matrix_test</span><span>)</span><span> </span><span>+</span><span> </span><span>mean</span><span>(</span><span>trend_for</span><span>)</span><span>
</span><span>}</span></code></pre>
</figure>
<p>Let’s compute median of forecasts and visualize all created forecasts. For <code>ggplot</code> visualization needs we must use <code>melt</code> function to the matrix of forecasts <code>pred_mat</code>.</p>
<figure>
<pre><code data-lang="r"><span>pred_melt_rpart</span><span> </span><span>&lt;-</span><span> </span><span>data.table</span><span>(</span><span>melt</span><span>(</span><span>pred_mat</span><span>))</span><span>
</span><span>pred_ave_rpart</span><span> </span><span>&lt;-</span><span> </span><span>pred_melt_rpart</span><span>[,</span><span> </span><span>.</span><span>(</span><span>value</span><span> </span><span>=</span><span> </span><span>median</span><span>(</span><span>value</span><span>)),</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>.</span><span>(</span><span>Var2</span><span>)]</span><span>
</span><span>pred_ave_rpart</span><span>[,</span><span> </span><span>Var1</span><span> </span><span>:=</span><span> </span><span>&quot;RPART_Bagg&quot;</span><span>]</span><span>
</span><span>ggplot</span><span>(</span><span>pred_melt_rpart</span><span>,</span><span> </span><span>aes</span><span>(</span><span>Var2</span><span>,</span><span> </span><span>value</span><span>,</span><span> </span><span>group</span><span> </span><span>=</span><span> </span><span>Var1</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.75</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>pred_ave_rpart</span><span>,</span><span> </span><span>aes</span><span>(</span><span>Var2</span><span>,</span><span> </span><span>value</span><span>),</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;firebrick2&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.9</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Time&quot;</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Load (kW)&quot;</span><span>,</span><span> </span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Bagging with RPART&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_ts</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-11" src="https://i1.wp.com/petolau.github.io/images/post_6/unnamed-chunk-11-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>The red line is median of forecasts. We can see that created forecasts by RPART have typical rectangular shape, but final ensemble forecasts has nice smooth behaviour. For comparison, compute forecast by RPART on whole train set and compare it with previous results. I am using function <code>RpartTrend</code> from my previous post.</p>
<figure>
<pre><code data-lang="r"><span>simple_rpart</span><span> </span><span>&lt;-</span><span> </span><span>RpartTrend</span><span>(</span><span>DT</span><span>,</span><span> </span><span>n_date</span><span>[</span><span>1</span><span>:</span><span>21</span><span>],</span><span> </span><span>K</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span><span>pred_rpart</span><span> </span><span>&lt;-</span><span> </span><span>data.table</span><span>(</span><span>value</span><span> </span><span>=</span><span> </span><span>simple_rpart</span><span>,</span><span> </span><span>Var2</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>48</span><span>,</span><span> </span><span>Var1</span><span> </span><span>=</span><span> </span><span>&quot;RPART&quot;</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>pred_melt_rpart</span><span>,</span><span> </span><span>aes</span><span>(</span><span>Var2</span><span>,</span><span> </span><span>value</span><span>,</span><span> </span><span>group</span><span> </span><span>=</span><span> </span><span>Var1</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.75</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>pred_ave_rpart</span><span>,</span><span> </span><span>aes</span><span>(</span><span>Var2</span><span>,</span><span> </span><span>value</span><span>),</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;firebrick2&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1.8</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>pred_rpart</span><span>,</span><span> </span><span>aes</span><span>(</span><span>Var2</span><span>,</span><span> </span><span>value</span><span>),</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;dodgerblue2&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1.8</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Time&quot;</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Load (kW)&quot;</span><span>,</span><span> </span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Bagging with RPART and comparison with simple RPART&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_ts</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-12" src="https://i0.wp.com/petolau.github.io/images/post_6/unnamed-chunk-12-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>The blue line is simple RPART forecast. The difference between ensemble forecast and simple one is not very evident in this scenario.</p>
<h2>Bagging with CTREE</h2>
<p>The second “bagged” regression tree method is CTREE. I will randomize only <code>mincriterion</code> hyperparameter of CTREE method that decisions about splitting a node. The sample ratio is of course sampled (randomized) as well as in the previous case with RPART.</p>
<figure>
<pre><code data-lang="r"><span>pred_mat</span><span> </span><span>&lt;-</span><span> </span><span>matrix</span><span>(</span><span>0</span><span>,</span><span> </span><span>nrow</span><span> </span><span>=</span><span> </span><span>N_boot</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>period</span><span>)</span><span>
</span><span>for</span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>N_boot</span><span>)</span><span> </span><span>{</span><span>
</span><span>matrixSam</span><span> </span><span>&lt;-</span><span> </span><span>matrix_train</span><span>[</span><span>sample</span><span>(</span><span>1</span><span>:</span><span>(</span><span>N</span><span>-</span><span>period</span><span>),</span><span>
</span><span>floor</span><span>((</span><span>N</span><span>-</span><span>period</span><span>)</span><span> </span><span>*</span><span> </span><span>sample</span><span>(</span><span>seq</span><span>(</span><span>0.7</span><span>,</span><span> </span><span>0.9</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>0.01</span><span>),</span><span> </span><span>1</span><span>)),</span><span>
</span><span>replace</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)]</span><span> </span><span># sampling with sampled ratio from 0.7 to 0.9</span><span>
</span><span>tree_bag</span><span> </span><span>&lt;-</span><span> </span><span>party</span><span>::</span><span>ctree</span><span>(</span><span>Load</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>matrixSam</span><span>,</span><span>
</span><span>controls</span><span> </span><span>=</span><span> </span><span>party</span><span>::</span><span>ctree_control</span><span>(</span><span>teststat</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;quad&quot;</span><span>),</span><span>
</span><span>testtype</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Teststatistic&quot;</span><span>),</span><span>
</span><span>mincriterion</span><span> </span><span>=</span><span> </span><span>sample</span><span>(</span><span>seq</span><span>(</span><span>0.88</span><span>,</span><span> </span><span>0.97</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>0.005</span><span>),</span><span> </span><span>1</span><span>),</span><span>
</span><span>minsplit</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span>
</span><span>minbucket</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span>
</span><span>mtry</span><span> </span><span>=</span><span> </span><span>0</span><span>,</span><span> </span><span>maxdepth</span><span> </span><span>=</span><span> </span><span>0</span><span>))</span><span>
</span><span># new data and prediction</span><span>
</span><span>pred_mat</span><span>[</span><span>i</span><span>,]</span><span> </span><span>&lt;-</span><span> </span><span>predict</span><span>(</span><span>tree_bag</span><span>,</span><span> </span><span>matrix_test</span><span>)</span><span> </span><span>+</span><span> </span><span>mean</span><span>(</span><span>trend_for</span><span>)</span><span>
</span><span>}</span></code></pre>
</figure>
<p>Let’s visualize results:</p>
<figure>
<pre><code data-lang="r"><span>pred_melt_ctree</span><span> </span><span>&lt;-</span><span> </span><span>data.table</span><span>(</span><span>melt</span><span>(</span><span>pred_mat</span><span>))</span><span>
</span><span>pred_ave_ctree</span><span> </span><span>&lt;-</span><span> </span><span>pred_melt_ctree</span><span>[,</span><span> </span><span>.</span><span>(</span><span>value</span><span> </span><span>=</span><span> </span><span>median</span><span>(</span><span>value</span><span>)),</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>.</span><span>(</span><span>Var2</span><span>)]</span><span>
</span><span>pred_ave_ctree</span><span>[,</span><span> </span><span>Var1</span><span> </span><span>:=</span><span> </span><span>&quot;CTREE_Bagg&quot;</span><span>]</span><span>
</span><span>ggplot</span><span>(</span><span>pred_melt_ctree</span><span>,</span><span> </span><span>aes</span><span>(</span><span>Var2</span><span>,</span><span> </span><span>value</span><span>,</span><span> </span><span>group</span><span> </span><span>=</span><span> </span><span>Var1</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.75</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>pred_ave_ctree</span><span>,</span><span> </span><span>aes</span><span>(</span><span>Var2</span><span>,</span><span> </span><span>value</span><span>),</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;firebrick2&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.9</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Time&quot;</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Load (kW)&quot;</span><span>,</span><span> </span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Bagging with CTREE&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_ts</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-14" src="https://i0.wp.com/petolau.github.io/images/post_6/unnamed-chunk-14-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Again, behaviour of bootstrapped forecasts is very rectangular and the final ensemble forecast is much more smoother. Compare it also with forecast performed on whole train set. I am using function <code>CtreeTrend</code> from my previous post.</p>
<figure>
<pre><code data-lang="r"><span>simple_ctree</span><span> </span><span>&lt;-</span><span> </span><span>CtreeTrend</span><span>(</span><span>DT</span><span>,</span><span> </span><span>n_date</span><span>[</span><span>1</span><span>:</span><span>21</span><span>],</span><span> </span><span>K</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span><span>pred_ctree</span><span> </span><span>&lt;-</span><span> </span><span>data.table</span><span>(</span><span>value</span><span> </span><span>=</span><span> </span><span>simple_ctree</span><span>,</span><span> </span><span>Var2</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>48</span><span>,</span><span> </span><span>Var1</span><span> </span><span>=</span><span> </span><span>&quot;CTREE&quot;</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>pred_melt_ctree</span><span>,</span><span> </span><span>aes</span><span>(</span><span>Var2</span><span>,</span><span> </span><span>value</span><span>,</span><span> </span><span>group</span><span> </span><span>=</span><span> </span><span>Var1</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.75</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>pred_ave_ctree</span><span>,</span><span> </span><span>aes</span><span>(</span><span>Var2</span><span>,</span><span> </span><span>value</span><span>),</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;firebrick2&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1.8</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>pred_ctree</span><span>,</span><span> </span><span>aes</span><span>(</span><span>Var2</span><span>,</span><span> </span><span>value</span><span>),</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;dodgerblue2&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1.8</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Time&quot;</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Load (kW)&quot;</span><span>,</span><span> </span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Bagging with CTREE and comparison with simple CTREE&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_ts</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-15" src="https://i0.wp.com/petolau.github.io/images/post_6/unnamed-chunk-15-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>The simple forecast is a little bit more rectangular than ensemble one.</p>
<h2>Random Forest</h2>
<p><a rel="nofollow" href="https://en.wikipedia.org/wiki/Random_forest" target="_blank"><strong>Random Forest</strong></a> is an improvement of Bagging ensemble learning method. It uses a modified tree learning algorithm that selects, at each candidate split in the learning process, a random subset of the features. This process is sometimes called “feature bagging”. The classical Bagging is also used in the method of course. Hyperparameters of Random Forest is similar to those in RPART method. Additional ones are: number of trees (<code>ntree</code>); number of variables sampled in each candidate split (<code>mtry</code>). Importance of variables can be also incorporated to learning process in order to enhance the performance. Random Forest is implemented in <strong>R</strong> by package <code>randomForest</code> and same-named function. Default settings for hyperparameters are: <code>mtry</code> = p/3 (where p is number of features), so 9/3 = 3 in our case, and <code>nodesize</code> = 5. I will set a number of trees to 1000.</p>
<figure>
<pre><code data-lang="r"><span>rf_model</span><span> </span><span>&lt;-</span><span> </span><span>randomForest</span><span>(</span><span>Load</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>data.frame</span><span>(</span><span>matrix_train</span><span>),</span><span>
</span><span>ntree</span><span> </span><span>=</span><span> </span><span>1000</span><span>,</span><span> </span><span>mtry</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span> </span><span>nodesize</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>importance</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span></code></pre>
</figure>
<p>Whereas we use ensemble of CART trees in Random Forest, we can compute variable importance. Checking variable importance can be important for extracting just most valuable features and to analyze dependencies and correlations in data. Let’s check variable importance by <code>varImpPlot</code>.</p>
<figure>
<pre><code data-lang="r"><span>varImpPlot</span><span>(</span><span>rf_model</span><span>,</span><span> </span><span>main</span><span> </span><span>=</span><span> </span><span>&quot;Variable importance&quot;</span><span>)</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-17" src="https://i2.wp.com/petolau.github.io/images/post_6/unnamed-chunk-17-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>We can see that Lag and S1.336 (weekly seasonal feature in sinus form) features have best scores in both measures. It implies that weekly seasonality and and lagged values of load are most important for our training data.</p>
<p>Let’s compute also forecast.</p>
<figure>
<pre><code data-lang="r"><span>pred_rf</span><span> </span><span>&lt;-</span><span> </span><span>predict</span><span>(</span><span>rf_model</span><span>,</span><span> </span><span>data.frame</span><span>(</span><span>matrix_test</span><span>))</span><span> </span><span>+</span><span> </span><span>mean</span><span>(</span><span>trend_for</span><span>)</span></code></pre>
</figure>
<p>And compare forecasts from all ensemble learning approaches so far and real electricity load.</p>
<figure>
<pre><code data-lang="r"><span>pred_rf</span><span> </span><span>&lt;-</span><span> </span><span>data.table</span><span>(</span><span>value</span><span> </span><span>=</span><span> </span><span>pred_rf</span><span>,</span><span> </span><span>Var2</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>48</span><span>,</span><span> </span><span>Var1</span><span> </span><span>=</span><span> </span><span>&quot;RF&quot;</span><span>)</span><span>
</span><span>pred_true</span><span> </span><span>&lt;-</span><span> </span><span>data.table</span><span>(</span><span>value</span><span> </span><span>=</span><span> </span><span>data_test</span><span>$</span><span>value</span><span>,</span><span> </span><span>Var2</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>48</span><span>,</span><span> </span><span>Var1</span><span> </span><span>=</span><span> </span><span>&quot;Real&quot;</span><span>)</span><span>
</span><span>preds_all</span><span> </span><span>&lt;-</span><span> </span><span>rbindlist</span><span>(</span><span>list</span><span>(</span><span>pred_ave_rpart</span><span>,</span><span> </span><span>pred_ave_ctree</span><span>,</span><span> </span><span>pred_rf</span><span>,</span><span> </span><span>pred_true</span><span>),</span><span> </span><span>use.names</span><span> </span><span>=</span><span> </span><span>T</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>preds_all</span><span>,</span><span> </span><span>aes</span><span>(</span><span>Var2</span><span>,</span><span> </span><span>value</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>as.factor</span><span>(</span><span>Var1</span><span>)))</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.7</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1.2</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Time&quot;</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Load (kW)&quot;</span><span>,</span><span> </span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Comparison of Ensemble Learning forecasts&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>guides</span><span>(</span><span>color</span><span>=</span><span>guide_legend</span><span>(</span><span>title</span><span>=</span><span>&quot;Method&quot;</span><span>))</span><span> </span><span>+</span><span>
</span><span>theme_ts</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-19" src="https://i1.wp.com/petolau.github.io/images/post_6/unnamed-chunk-19-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>There are just small differences.</p>
<h2>Hyperparameters tuning – grid search</h2>
<p>It is always highly recommended to tune hyperparameters of our used method. By hyperparameters tuning, we can significantly improve predictive performance. I will try to tune two hyperparameters of <strong>Random Forest</strong>, <code>mtry</code> and <code>nodesize</code>, by <a rel="nofollow" href="https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)#Grid_search" target="_blank"><strong>grid search</strong></a> method.<br>
For specificity of time series forecasting, I proposed my own function of grid search (<code>gridSearch</code>). The evaluation is based on sliding window forecasting, so forecasting error is average of errors in some range of time. I also created my own Random Forest function (<code>RFgrid</code>) that is compatible with grid search function. The output of <code>gridSearch</code> function is matrix of average MAPE (Mean Absolute Percentage Error) values corresponding to hyperparameters used.</p>
<figure>
<pre><code data-lang="r"><span>RFgrid</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>data_train</span><span>,</span><span> </span><span>param1</span><span>,</span><span> </span><span>param2</span><span>,</span><span> </span><span>K</span><span>,</span><span> </span><span>period</span><span> </span><span>=</span><span> </span><span>48</span><span>)</span><span> </span><span>{</span><span>
</span><span>N</span><span> </span><span>&lt;-</span><span> </span><span>length</span><span>(</span><span>data_train</span><span>)</span><span>
</span><span>window</span><span> </span><span>&lt;-</span><span> </span><span>(</span><span>N</span><span> </span><span>/</span><span> </span><span>period</span><span>)</span><span> </span><span>-</span><span> </span><span>1</span><span>
</span><span>data_ts</span><span> </span><span>&lt;-</span><span> </span><span>msts</span><span>(</span><span>data_train</span><span>,</span><span> </span><span>seasonal.periods</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>period</span><span>,</span><span> </span><span>period</span><span>*</span><span>7</span><span>))</span><span>
</span><span>fuur</span><span> </span><span>&lt;-</span><span> </span><span>fourier</span><span>(</span><span>data_ts</span><span>,</span><span> </span><span>K</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>K</span><span>,</span><span> </span><span>K</span><span>))</span><span>
</span><span>fuur_test</span><span> </span><span>&lt;-</span><span> </span><span>as.data.frame</span><span>(</span><span>fourier</span><span>(</span><span>data_ts</span><span>,</span><span> </span><span>K</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>K</span><span>,</span><span> </span><span>K</span><span>),</span><span> </span><span>h</span><span> </span><span>=</span><span> </span><span>period</span><span>))</span><span>
</span><span>data_ts</span><span> </span><span>&lt;-</span><span> </span><span>ts</span><span>(</span><span>data_train</span><span>,</span><span> </span><span>freq</span><span> </span><span>=</span><span> </span><span>period</span><span>*</span><span>7</span><span>)</span><span>
</span><span>decomp_ts</span><span> </span><span>&lt;-</span><span> </span><span>stl</span><span>(</span><span>data_ts</span><span>,</span><span> </span><span>s.window</span><span> </span><span>=</span><span> </span><span>&quot;periodic&quot;</span><span>,</span><span> </span><span>robust</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>new_load</span><span> </span><span>&lt;-</span><span> </span><span>rowSums</span><span>(</span><span>decomp_ts</span><span>$</span><span>time.series</span><span>[,</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span>3</span><span>)])</span><span>
</span><span>trend_part</span><span> </span><span>&lt;-</span><span> </span><span>ts</span><span>(</span><span>decomp_ts</span><span>$</span><span>time.series</span><span>[,</span><span>2</span><span>])</span><span>
</span><span>trend_fit</span><span> </span><span>&lt;-</span><span> </span><span>auto.arima</span><span>(</span><span>trend_part</span><span>)</span><span>
</span><span>trend_for</span><span> </span><span>&lt;-</span><span> </span><span>as.vector</span><span>(</span><span>forecast</span><span>(</span><span>trend_fit</span><span>,</span><span> </span><span>period</span><span>)</span><span>$</span><span>mean</span><span>)</span><span>
</span><span>lag_seas</span><span> </span><span>&lt;-</span><span> </span><span>decomp_ts</span><span>$</span><span>time.series</span><span>[</span><span>1</span><span>:</span><span>(</span><span>period</span><span>*</span><span>window</span><span>),</span><span> </span><span>1</span><span>]</span><span>
</span><span>matrix_train</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>Load</span><span> </span><span>=</span><span> </span><span>tail</span><span>(</span><span>new_load</span><span>,</span><span> </span><span>window</span><span>*</span><span>period</span><span>),</span><span>
</span><span>fuur</span><span>[(</span><span>period</span><span>+1</span><span>)</span><span>:</span><span>N</span><span>,],</span><span>
</span><span>Lag</span><span> </span><span>=</span><span> </span><span>lag_seas</span><span>)</span><span>
</span><span>tree_2</span><span> </span><span>&lt;-</span><span> </span><span>randomForest</span><span>(</span><span>Load</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>matrix_train</span><span>,</span><span>
</span><span>ntree</span><span> </span><span>=</span><span> </span><span>1000</span><span>,</span><span> </span><span>mtry</span><span> </span><span>=</span><span> </span><span>param1</span><span>,</span><span> </span><span>nodesize</span><span> </span><span>=</span><span> </span><span>param2</span><span>,</span><span> </span><span>importance</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>test_lag</span><span> </span><span>&lt;-</span><span> </span><span>decomp_ts</span><span>$</span><span>time.series</span><span>[((</span><span>period</span><span>*</span><span>(</span><span>window</span><span>))</span><span>+1</span><span>)</span><span>:</span><span>N</span><span>,</span><span> </span><span>1</span><span>]</span><span>
</span><span>matrix_test</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>fuur_test</span><span>,</span><span>
</span><span>Lag</span><span> </span><span>=</span><span> </span><span>test_lag</span><span>)</span><span>
</span><span>pred_tree</span><span> </span><span>&lt;-</span><span> </span><span>predict</span><span>(</span><span>tree_2</span><span>,</span><span> </span><span>matrix_test</span><span>)</span><span> </span><span>+</span><span> </span><span>mean</span><span>(</span><span>trend_for</span><span>)</span><span>
</span><span>return</span><span>(</span><span>as.vector</span><span>(</span><span>pred_tree</span><span>))</span><span>
</span><span>}</span><span>
</span><span>gridSearch</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>Y</span><span>,</span><span> </span><span>train.win</span><span> </span><span>=</span><span> </span><span>21</span><span>,</span><span> </span><span>FUN</span><span>,</span><span> </span><span>param1</span><span>,</span><span> </span><span>param2</span><span>,</span><span> </span><span>period</span><span> </span><span>=</span><span> </span><span>48</span><span>)</span><span> </span><span>{</span><span>
</span><span>days</span><span> </span><span>&lt;-</span><span> </span><span>length</span><span>(</span><span>Y</span><span>)</span><span>/</span><span>period</span><span>
</span><span>test.days</span><span> </span><span>&lt;-</span><span> </span><span>days</span><span> </span><span>-</span><span> </span><span>train.win</span><span>
</span><span>mape.matrix</span><span> </span><span>&lt;-</span><span> </span><span>matrix</span><span>(</span><span>0</span><span>,</span><span> </span><span>nrow</span><span> </span><span>=</span><span> </span><span>length</span><span>(</span><span>param1</span><span>),</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>length</span><span>(</span><span>param2</span><span>))</span><span>
</span><span>row.names</span><span>(</span><span>mape.matrix</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>param1</span><span>
</span><span>colnames</span><span>(</span><span>mape.matrix</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>param2</span><span>
</span><span>forecast.rf</span><span> </span><span>&lt;-</span><span> </span><span>vector</span><span>(</span><span>length</span><span> </span><span>=</span><span> </span><span>test.days</span><span>*</span><span>period</span><span>)</span><span>
</span><span>for</span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>seq_along</span><span>(</span><span>param1</span><span>)){</span><span>
</span><span>for</span><span>(</span><span>j</span><span> </span><span>in</span><span> </span><span>seq_along</span><span>(</span><span>param2</span><span>)){</span><span>
</span><span>for</span><span>(</span><span>k</span><span> </span><span>in</span><span> </span><span>0</span><span>:</span><span>(</span><span>test.days</span><span>-1</span><span>)){</span><span>
</span><span>train.set</span><span> </span><span>&lt;-</span><span> </span><span>Y</span><span>[((</span><span>k</span><span>*</span><span>period</span><span>)</span><span>+1</span><span>)</span><span>:</span><span>((</span><span>period</span><span>*</span><span>k</span><span>)</span><span>+</span><span>(</span><span>train.win</span><span>*</span><span>period</span><span>))]</span><span>
</span><span>forecast.rf</span><span>[((</span><span>k</span><span>*</span><span>period</span><span>)</span><span>+1</span><span>)</span><span>:</span><span>((</span><span>k</span><span>+1</span><span>)</span><span>*</span><span>period</span><span>)]</span><span> </span><span>&lt;-</span><span> </span><span>FUN</span><span>(</span><span>train.set</span><span>,</span><span> </span><span>param1</span><span> </span><span>=</span><span> </span><span>param1</span><span>[</span><span>i</span><span>],</span><span> </span><span>param2</span><span> </span><span>=</span><span> </span><span>param2</span><span>[</span><span>j</span><span>],</span><span> </span><span>K</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span><span>}</span><span>
</span><span>mape.matrix</span><span>[</span><span>i</span><span>,</span><span>j</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>mape</span><span>(</span><span>Y</span><span>[</span><span>-</span><span>(</span><span>1</span><span>:</span><span>(</span><span>train.win</span><span>*</span><span>period</span><span>))],</span><span> </span><span>forecast.rf</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span><span>return</span><span>(</span><span>mape.matrix</span><span>)</span><span>
</span><span>}</span></code></pre>
</figure>
<p>For our purpose, I will make 20 forecasts for each hyperparameter combination. I will test <code>nodesize</code> and <code>mtry</code> parameters for values \( 2,3,4,5,6 \). It is taking some time to compute, so be aware.</p>
<figure>
<pre><code data-lang="r"><span>all_data</span><span> </span><span>&lt;-</span><span> </span><span>DT</span><span>$</span><span>value</span><span>[</span><span>1</span><span>:</span><span>(</span><span>period</span><span>*</span><span>41</span><span>)]</span><span>
</span><span>res_1</span><span> </span><span>&lt;-</span><span> </span><span>gridSearch</span><span>(</span><span>all_data</span><span>,</span><span> </span><span>FUN</span><span> </span><span>=</span><span> </span><span>RFgrid</span><span>,</span><span> </span><span>param1</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>2</span><span>,</span><span>3</span><span>,</span><span>4</span><span>,</span><span>5</span><span>,</span><span>6</span><span>),</span><span> </span><span>param2</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>2</span><span>,</span><span>3</span><span>,</span><span>4</span><span>,</span><span>5</span><span>,</span><span>6</span><span>))</span></code></pre>
</figure>
<p>Let’s check the results.</p>
<figure>
<pre><code data-lang="r"><span>res_1</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##          2        3        4        5        6
## 2 2.937151 2.931990 2.952967 2.970451 2.972355
## 3 2.888450 2.896477 2.897801 2.912977 2.914294
## 4 2.874241 2.878923 2.880959 2.882959 2.896501
## 5 2.876503 2.872179 2.873983 2.875953 2.882657
## 6 2.868838 2.872933 2.874093 2.867840 2.881881</code></pre>
</figure>
<p>And find the minimum.</p>
<figure>
<pre><code data-lang="r"><span>c</span><span>(</span><span>mtry</span><span> </span><span>=</span><span> </span><span>row.names</span><span>(</span><span>res_1</span><span>)[</span><span>which</span><span>(</span><span>res_1</span><span> </span><span>==</span><span> </span><span>min</span><span>(</span><span>res_1</span><span>),</span><span> </span><span>arr.ind</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)[</span><span>1</span><span>]],</span><span>
</span><span>nodesize</span><span> </span><span>=</span><span> </span><span>colnames</span><span>(</span><span>res_1</span><span>)[</span><span>which</span><span>(</span><span>res_1</span><span> </span><span>==</span><span> </span><span>min</span><span>(</span><span>res_1</span><span>),</span><span> </span><span>arr.ind</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)[</span><span>2</span><span>]])</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##     mtry nodesize 
##      &quot;6&quot;      &quot;5&quot;</code></pre>
</figure>
<p>So best (optimal) setting is <code>mtry</code> = 6 and <code>nodesize</code> = 5.</p>
<p>For better imagination and analysis of results, let’s visualize the computed grid of MAPE values.</p>
<figure>
<pre><code data-lang="r"><span>data_grid</span><span> </span><span>&lt;-</span><span> </span><span>data.table</span><span>(</span><span>melt</span><span>(</span><span>res_1</span><span>))</span><span>
</span><span>colnames</span><span>(</span><span>data_grid</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;mtry&quot;</span><span>,</span><span> </span><span>&quot;nodesize&quot;</span><span>,</span><span> </span><span>&quot;MAPE&quot;</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>data_grid</span><span>,</span><span> </span><span>aes</span><span>(</span><span>mtry</span><span>,</span><span> </span><span>nodesize</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>MAPE</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>MAPE</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_color_distiller</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Reds&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_ts</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-24" src="https://i1.wp.com/petolau.github.io/images/post_6/unnamed-chunk-24-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>We can see that lowest average MAPEs are for <code>mtry</code> = 6 and <code>nodesize</code> = 2 or 5.</p>
<h2>Comparison</h2>
<p>For more exact evaluation, I made comparison of five methods – simple RPART, simple CTREE, Bagging + RPART, Bagging + CTREE and Random Forest on whole available dataset (so 98 forecasts were performed).<br>
I created <code>plotly</code> boxplots graph of MAPE values from these 5 methods. Whole evaluation can be seen in the script that is stored in <a rel="nofollow" href="https://github.com/PetoLau/petolau.github.io/tree/master/_Rscripts" target="_blank">my GitHub repository</a>.</p>
<p><iframe width="450" src="https://plot.ly/~PetoLau2/68.embed"></iframe></p>
<p>We can see that Bagging really helps decrease forecasting error for RPART and CTREE. The Random Forest has best results among all tested methods, so I proposed suitable approach for seasonal time series forecasting.</p>
<h2>Conclusion</h2>
<p>In this post, I showed you how to use basic ensemble learning methods to improve forecasting accuracy. I used classical Bagging in combination with regression tree methods – RPART and CTREE. The extension of Bagging, Random Forest, was also used and evaluated. I showed you how to set important hyperparameters and how to tune them by grid search method. The Random Forest method comes most accurate and I highly recommend it for time series forecasting. But, it must be said that feature engineering is very important part also of regression modeling of time series. So, I don’t generalize results for every possible task of time series forecasting.</p>
<p>In the future post, I will write about other bootstrapping techniques for time series or Boosting.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://petolau.github.io/Ensemble-of-trees-for-forecasting-time-series/"> Peter Laurinec</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/paQfc-nZXa8" height="1"></a></div>