<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/time-series-deep-learning-forecasting-sunspots-with-keras-stateful-lstm-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.business-science.io/timeseries-analysis/2018/04/18/keras-lstm-sunspots-time-series-prediction.html"> business-science.io - Articles</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><strong>Time series prediction (forecasting) has experienced dramatic improvements in predictive accuracy as a result of the data science machine learning and deep learning evolution</strong>. As these ML/DL tools have evolved, businesses and financial institutions are now able to forecast better by applying these new technologies to solve old problems. In this article, we showcase the use of a special type of <strong>Deep Learning model called an <a rel="nofollow" href="https://en.wikipedia.org/wiki/Long_short-term_memory" target="_blank">LSTM (Long Short-Term Memory)</a></strong>, which is useful for problems involving <a rel="nofollow" href="https://en.wikipedia.org/wiki/Autocorrelation" target="_blank">sequences with autocorrelation</a>. We analyze a famous historical data set called <a rel="nofollow" href="https://stat.ethz.ch/R-manual/R-devel/library/datasets/html/sunspot.month.html" target="_blank">“sunspots”</a> (a <a rel="nofollow" href="https://en.wikipedia.org/wiki/Solar_cycle" target="_blank">sunspot</a> is a solar phenomenon wherein a dark spot forms on the surface of the sun). We’ll show you how you can use an LSTM model to predict sunspots ten years into the future with an LSTM model.</p>
<h2>Tutorial Overview</h2>
<p>This code tutorial goes along with a <a rel="nofollow" href="https://github.com/business-science/presentations/tree/master/2018_04_19_SP_Global_Time_Series_Deep_Learning" target="_blank">presentation on Time Series Deep Learning</a> given to <a rel="nofollow" href="https://www.spglobal.com/" target="_blank">SP Global</a> on Thursday, April 19, 2018. The slide deck that complements this article is available for download.</p>
<p>This is an <strong>advanced tutorial</strong> implementing <strong>deep learning for time series</strong> and several other complex machine learning topics such as backtesting cross validation. For those seeking an introduction to <strong>Keras in R</strong>, please check out <a rel="nofollow" href="http://www.business-science.io/business/2017/11/28/customer_churn_analysis_keras.html" target="_blank">Customer Analytics: Using Deep Learning With Keras To Predict Customer Churn</a>.</p>
<p>In this tutorial, you will learn how to:</p>
<ul>
<li>Develop a <strong>Stateful LSTM Model</strong> with the <code>keras</code> package, which connects to the <a rel="nofollow" href="https://tensorflow.rstudio.com/" target="_blank">R TensorFlow backend</a>.</li>
<li>Apply a Keras Stateful LSTM Model to a <strong>famous time series, Sunspots</strong>.</li>
<li>Perform <strong>Time Series Cross Validation using Backtesting</strong> with the <code>rsample</code> package <a rel="nofollow" href="https://topepo.github.io/rsample/articles/Applications/Time_Series.html" target="_blank">rolling forecast origin resampling</a>.</li>
<li><strong>Visualize Backtest Sampling Plans and Prediction Results</strong> with <code>ggplot2</code> and <code>cowplot</code>.</li>
<li>Evaluate whether or not a time series may be a good candidate for an LSTM model by reviewing the <strong>Autocorrelation Function (ACF) plot</strong>.</li>
</ul>
<p>The end result is a <strong>high performance deep learning algorithm</strong> that does an excellent job at predicting ten years of sunspots! Here’s the plot of the <strong>Backtested Keras Stateful LSTM Model</strong>.</p>
<p><img alt="Stateful Keras LSTM Predictions" src="https://i0.wp.com/www.business-science.io/assets/2018-04-18-keras-lstm-sunspots/keras_stateful_lstm.png?w=456" data-recalc-dims="1"></p>
<h2>Applications in Business</h2>
<p>Time series prediction (forecasting) has a dramatic effect on the top and bottom line. In business, we could be interested in predicting which day of the month, quarter, or year that large expenditures are going to occur or we could be interested in understanding how the consumer price index (CPI) will change over the course of the next six months. These are common questions that impact organizations both on a microeconomic and macroeconomic level. While the data set used in this tutorial is not a “business” data set, it shows the power of the <strong>tool-problem fit</strong>, meaning that using the right tool for the job can offer tremendous improvements in accuracy. <strong>The net result is increased prediction accuracy ultimately leads to quantifiable improvements to the top and bottom line</strong>.</p>
<h2>Long Short-Term Memory (LSTM) Models</h2>
<p>A <strong>Long Short-Term Memory (LSTM) model</strong> is a powerful type of recurrent neural network (RNN). The blog article, <a rel="nofollow" href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/" target="_blank">“Understanding LSTM Networks”</a>, does an excellent job at explaining the underlying complexity in an easy to understand way. Here’s an image depicting the LSTM internal cell architecture that enables it to persist for long term states (in addition to short term), which traditional RNN’s have difficulty with:</p>
<p><img src="https://i0.wp.com/www.business-science.io/assets/2018-04-18-keras-lstm-sunspots/LSTM3-chain.png?w=456" data-recalc-dims="1"></p>
<p>Source: <a rel="nofollow" href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/" target="_blank">Understanding LSTM Networks</a></p>
<p>LSTMs are quite useful in time series prediction tasks involving <strong>autocorrelation</strong>, the presence of correlation between the time series and lagged versions of itself, because of their ability to maintain state and recognize patterns over the length of the time series. The recurrent architecture enables the states to persist, or communicate between updates of the weights as each epoch progresses. Further, the LSTM cell architecture enhances the RNN by enabling long term persistence in addition to short term, which is fascinating!</p>
<p>In Keras, LSTM’s can be operated in a “stateful” mode, which according to the Keras documentation:</p>
<blockquote>
<p>The last state for each sample at index <em>i</em> in a batch will be used as initial state for the sample of index <em>i</em> in the following batch</p>
</blockquote>
<p>In normal (or “stateless”) mode, Keras shuffles the samples, and the dependencies between the time series and the lagged version of itself are lost. However, <strong>when run in “stateful” mode, we can often get high accuracy results by leveraging the autocorrelations present in the time series</strong>.</p>
<p>We’ll explain more as we go through this tutorial. For now, just understand that LSTM’s can be really useful for time series problems involving autocorrelation and Keras has the capability to create stateful LSTMs that are perfect for time series modeling.</p>
<h2>Sunspots Data Set</h2>
<p><a rel="nofollow" href="https://stat.ethz.ch/R-manual/R-devel/library/datasets/html/sunspot.month.html" target="_blank">Sunspots</a> is a famous data set that ships with R (refer to the <code>datasets</code> library). The dataset tracks sunspots, which are the occurrence of a dark spot on the sun. Here’s an image from NASA showing the solar phenomenon. Pretty cool!</p>
<p><img src="https://i1.wp.com/www.business-science.io/assets/2018-04-18-keras-lstm-sunspots/sunspot_nasa.jpg?w=456" data-recalc-dims="1"></p>
<p>Source: <a rel="nofollow" href="https://www.nasa.gov/content/goddard/largest-sunspot-of-solar-cycle" target="_blank">NASA</a></p>
<p>The dataset we use in this tutorial is called <code>sunspots.month</code>, and it contains 265 years (from 1749 through 2013) of monthly data on number of sunspots per month.</p>
<p><img alt="plot of chunk unnamed-chunk-1" src="https://i1.wp.com/www.business-science.io/figure/source/2018-04-18-keras-lstm-sunspots-time-series-prediction/unnamed-chunk-1-1.png?w=456" data-recalc-dims="1"></p>
<h2>Implementing An LSTM To Predict Sunspots</h2>
<p>Time to get to business. Let’s predict sunspots. Here’s our objective:</p>
<p><strong>Objective</strong>:<br>
Use an LSTM model to generate a forecast of sunspots that <strong>spans 10-years into the future</strong>.</p>
<h3>1.0 Libraries</h3>
<p>Here are the libraries needed for this tutorial. All are available on CRAN. You can install with <code>install.packages()</code> if you do not have them installed already. <strong>NOTE: Before proceeding with this code tutorial, make sure to update all packages as previous versions of these packages may not be compatible with the code used. One issue we are aware of is upgrading to <code>lubridate</code> version 1.7.4.</strong></p>
<figure>
<pre><code data-lang="r"><span># Core Tidyverse</span><span>
</span><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>library</span><span>(</span><span>glue</span><span>)</span><span>
</span><span>library</span><span>(</span><span>forcats</span><span>)</span><span>
</span><span># Time Series</span><span>
</span><span>library</span><span>(</span><span>timetk</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tidyquant</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tibbletime</span><span>)</span><span>
</span><span># Visualization</span><span>
</span><span>library</span><span>(</span><span>cowplot</span><span>)</span><span>
</span><span># Preprocessing</span><span>
</span><span>library</span><span>(</span><span>recipes</span><span>)</span><span>
</span><span># Sampling / Accuracy</span><span>
</span><span>library</span><span>(</span><span>rsample</span><span>)</span><span>
</span><span>library</span><span>(</span><span>yardstick</span><span>)</span><span> </span><span># Modeling</span><span>
</span><span>library</span><span>(</span><span>keras</span><span>)</span></code></pre>
</figure>
<p>If you have not previously run Keras in R, you will need to install Keras using the <code>install_keras()</code> function.</p>
<figure>
<pre><code data-lang="r"><span># Install Keras if you have not installed before</span><span>
</span><span>install_keras</span><span>()</span></code></pre>
</figure>
<h3>2.0 Data</h3>
<p>The dataset, <code>sunspot.month</code>, is available for all of us (it ships with base R). It’s a <code>ts</code> class (not tidy), so we’ll convert to a tidy data set using the <code>tk_tbl()</code> function from <code>timetk</code>. We use this instead of <code>as.tibble()</code> from <code>tibble</code> to automatically preserve the time series index as a <code>zoo</code> <code>yearmon</code> index. Last, we’ll convert the <code>zoo</code> index to date using <code>lubridate::as_date()</code> (loaded with <code>tidyquant</code>) and then change to a <code>tbl_time</code> object to make time series operations easier.</p>
<figure>
<pre><code data-lang="r"><span>sun_spots</span><span> </span><span>&lt;-</span><span> </span><span>datasets</span><span>::</span><span>sunspot.month</span><span> </span><span>%>%</span><span>
</span><span>tk_tbl</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>index</span><span> </span><span>=</span><span> </span><span>as_date</span><span>(</span><span>index</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>as_tbl_time</span><span>(</span><span>index</span><span> </span><span>=</span><span> </span><span>index</span><span>)</span><span>
</span><span>sun_spots</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A time tibble: 3,177 x 2
## # Index: index
##    index      value
##    </code></pre></figure>