<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/cross-validation-for-time-series/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://robjhyndman.com/hyndsight/tscv/"> R – Hyndsight</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I’ve added a couple of new functions to the <a target="_blank" rel="nofollow" href="https://github.com/robjhyndman/forecast">forecast package for R</a> which implement two types of cross-validation for time series.<span></span></p>
<h3>K-fold cross-validation for autoregression</h3>
<p>The first is regular k-fold cross-validation for autoregressive models. Although cross-validation is sometimes not valid for time series models, it does work for autoregressions, which includes many machine learning approaches to time series. The theoretical background is provided in <a target="_blank" rel="nofollow" href="http://robjhyndman.com/working-papers/cv-time-series">Bergmeir, Hyndman and Koo (2015)</a>. So cross-validation can be applied to any model where the predictors are lagged values of the response variable.</p>
<p>This is implemented for NNAR models (neural network autoregressions) in R as follows:</p>
<div>
<table>
<tr>
<td>
<pre>modelcv <span>&lt;-</span> CVar<span>(</span><span>lynx</span>, k<span>=</span><span>5</span>, lambda<span>=</span><span>0.15</span><span>)</span>
<span>print</span><span>(</span>modelcv<span>)</span></pre>
</td>
</tr>
</table>
</div>
<p>The output is a summary of the accuracy across folds:</p>
<pre>5-fold cross-validation
                  Mean          SD
ME        -32.88142801  98.0725227
RMSE      931.90966858 352.8705338
MAE       608.99488205 272.1244879
MPE       -17.84710226  15.2700638
MAPE       53.99760978  12.7264054
ACF1        0.04842174   0.1480883
Theil's U   0.82984737   0.1487229
</pre>
<p>The <code>CVar</code> function is rather limited at this stage, and will only handle cross-validation for models computed using <code>nnetar</code>. If there is enough interest, I might add more functionality at a later stage.</p>
<h3>Time series cross-validation</h3>
<p>In this procedure, there is a series of test sets, each consisting of a single observation. The corresponding training set consists only of observations that occurred <em>prior</em> to the observation that forms the test set. Thus, no future observations can be used in constructing the forecast. The following diagram illustrates the series of training and test sets, where the blue observations form the training sets, and the red observations form the test sets.</p>
<p><img sizes="(max-width: 816px) 100vw, 816px" alt="cv1-1" class="wp-image-3934" src="https://i1.wp.com/robjhyndman.com/hyndsight/files/2016/12/cv1-1.png?resize=456%2C215" data-recalc-dims="1"></p>
<p>The forecast accuracy is computed by averaging over the test sets. This procedure is sometimes known as “evaluation on a rolling forecasting origin” because the “origin” at which the forecast is based rolls forward in time.</p>
<p>With time series forecasting, one-step forecasts may not be as relevant as multi-step forecasts. In this case, the cross-validation procedure based on a rolling forecasting origin can be modified to allow multi-step errors to be used. Suppose that we are interested in models that produce good 4-step-ahead forecasts. Then the corresponding diagram is shown below.</p>
<p><img sizes="(max-width: 816px) 100vw, 816px" alt="cv4-1" class="wp-image-3935" src="https://i1.wp.com/robjhyndman.com/hyndsight/files/2016/12/cv4-1.png?resize=456%2C215" data-recalc-dims="1"></p>
<p>Time series cross validation is implemented with the <code>tsCV</code> function. In the following example, we compare the residual RMSE with the RMSE obtained via time series cross-validation.</p>
<div>
<table>
<tr>
<td>
<pre><span>library</span><span>(</span>fpp<span>)</span>
e <span>&lt;-</span> tsCV<span>(</span>dj, rwf, drift<span>=</span>TRUE, h<span>=</span><span>1</span><span>)</span>
<span>sqrt</span><span>(</span><span>mean</span><span>(</span>e<span>^</span><span>2</span>, na.<span>rm</span><span>=</span>TRUE<span>)</span><span>)</span>
<span>## [1] 22.68249</span>
<span>sqrt</span><span>(</span><span>mean</span><span>(</span><span>residuals</span><span>(</span>rwf<span>(</span>dj, drift<span>=</span>TRUE<span>)</span><span>)</span><span>^</span><span>2</span>, na.<span>rm</span><span>=</span>TRUE<span>)</span><span>)</span>
<span>## [1] 22.49681</span></pre>
</td>
</tr>
</table>
</div>
<p>Here I apply a random walk with drift to the Dow-Jones index time series <code>dj</code>. The first calculation implements a one-step time series cross-validation where the drift parameter is re-estimated at every forecast origin. The second calculation estimates the drift parameter once for the whole data set, and then computes the RMSE from the one-step forecasts. As expected, the RMSE from the residuals is smaller, as the corresponding “forecasts” are based on a model fitted to the entire data set, rather than being true forecasts.</p>
<p>The <code>tsCV</code> function is very general, and will work for any forecasting function that returns an object of class <code>forecast</code>. You don’t even have to specify the minimum sample size for model fitting, as it will silently fit models beginning with a single observation, and return a missing value whenever the model cannot be estimated.</p>
<p>These functions are only on the <a target="_blank" rel="nofollow" href="https://github.com/robjhyndman/forecast">github version</a> for now, but will migrate to <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=forecast">CRAN</a> next time I release a new stable version.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://robjhyndman.com/hyndsight/tscv/"> R – Hyndsight</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/EB62lkpBryA"></a></div>