<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/time-series-deep-learning-part-2-predicting-sunspot-frequency-with-keras-lstm-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.business-science.io/timeseries-analysis/2018/07/01/keras-lstm-sunspots-part2.html"> business-science.io - Articles</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>One of the ways <strong>Deep Learning can be used in business</strong> is to improve the accuracy of time series forecasts (prediction). We recently showed how a <a rel="nofollow" href="http://www.business-science.io/timeseries-analysis/2018/04/18/keras-lstm-sunspots-time-series-prediction.html" target="_blank">Long Short Term Memory (LSTM) Models developed with the Keras library in R</a> could be used to take advantage of <em>autocorrelation</em> to predict the next 10 years of monthly Sunspots (a solar phenomenon that’s tracked by NASA). In this article, we teamed up with RStudio to take another look at the Sunspots data set, this time implementing some really advanced Deep Learning functionality available with <a rel="nofollow" href="https://tensorflow.rstudio.com/" target="_blank">TensorFlow for R</a>. Sigrid Keydana, TF Developer Advocate at RStudio, put together an amazing Deep Learning tutorial using <code>keras</code> for <a rel="nofollow" href="https://tensorflow.rstudio.com/keras/" target="_blank">implementing Keras in R</a> and <code>tfruns</code>, a <a rel="nofollow" href="https://tensorflow.rstudio.com/blog/tfruns.html" target="_blank">suite of tools</a> for trackingtracking, visualizing, and managing TensorFlow training runs and experiments from R. Sounds amazing, right? It is! <strong>Let’s get started with this Deep Learning Tutorial!</strong></p>
<h2>Related Articles In This Series</h2>
<ul>
<li>
<p><a rel="nofollow" href="http://www.business-science.io/timeseries-analysis/2018/04/18/keras-lstm-sunspots-time-series-prediction.html" target="_blank">Time Series Deep Learning: Forecasting Sunspots With Keras Stateful LSTM In R</a></p>
</li>
<li>
<p>You can also find this article on <a rel="nofollow" href="https://tensorflow.rstudio.com/blog/sunspots-lstm.html" target="_blank">RStudio’s TensorFlow Blog</a>.</p>
</li>
</ul>
<h2>Learning Trajectory</h2>
<p>In this <strong>DEEP LEARNING TUTORIAL</strong>_, you will learn:</p>
<ul>
<li>
<p><a rel="nofollow" href="http://www.business-science.io/timeseries-analysis/2018/07/01/keras-lstm-sunspots-part2.html#dl-in-business" target="_blank">How Time Series Deep Learning can be used in business</a></p>
</li>
<li>
<p><a rel="nofollow" href="http://www.business-science.io/timeseries-analysis/2018/07/01/keras-lstm-sunspots-part2.html#dl-sunspots" target="_blank">Forecasting Sunspots With Deep Learning</a></p>
</li>
<li>
<p><a rel="nofollow" href="http://www.business-science.io/timeseries-analysis/2018/07/01/keras-lstm-sunspots-part2.html#lstm" target="_blank">How To Build The LSTM</a></p>
</li>
<li>
<p><a rel="nofollow" href="http://www.business-science.io/timeseries-analysis/2018/07/01/keras-lstm-sunspots-part2.html#backtesting" target="_blank">Backtesting The LSTM Model</a></p>
</li>
</ul>
<p>In fact, one of the coolest things you’ll develop is this plot of backtested LSTM forecasts.</p>
<p><img alt="Backtested LSTM Forecasts" src="https://i0.wp.com/www.business-science.io/assets/2018-07-01-keras-lstm-sunspots2/backtested_test.png?w=456" data-recalc-dims="1"></p>
<p>Backtested LSTM Forecasts</p>
<h2>Time Series Deep Learning In Business <a></a></h2>
<blockquote>
<p>Introduction by Matt Dancho, Founder of Business Science</p>
</blockquote>
<p>Time Series Forecasting is a key area that can lead to Return On Investment (ROI) in a business. Think about this: <strong>A 10% improvement in forecast accuracy can save an organization millions of dollars</strong>. How is this possible? Let’s find out.</p>
<p>We’ll take <a rel="nofollow" href="http://www.nvidia.com/page/home.html" target="_blank">NVIDIA</a>, a semiconductor manufacturer that manufactures state-of-the-art chips for <em>Artificial Intelligence (AI)</em> and <em>Deep Learning (DL)</em>, as an example. <em>NVIDIA</em> builds <a rel="nofollow" href="https://en.wikipedia.org/wiki/Graphics_processing_unit" target="_blank">Graphics Processing Unitis or GPUs</a>, which are necessary for the computational intensitity resulting from the massive number of numerical calculations required in high-performance Deep Learning. The chips look like this.</p>
<p><img alt="NVIDIA" src="https://i0.wp.com/www.business-science.io/assets/2018-07-01-keras-lstm-sunspots2/nvidia.png?w=456" data-recalc-dims="1"></p>
<p>Source: <a rel="nofollow" href="http://www.nvidia.com/page/home.html" target="_blank">NVIDIA USA</a></p>
<p>Like all manufacturers, <em>NVIDIA</em> needs to <strong><em>forecast demand</em></strong> for their products. Why? So they can build the right amount of chips to supply their customers. This forecast is critical and it takes a lot of skill and some luck to get this right.</p>
<p>What we are talking about is the <strong><em>Sales Forecast</em></strong>, which drives every manufacturing decision that NVIDIA makes. This includes how much raw material to purchase, how many people to have on staff to build the chips, and how much machining and assembly operations to budget. The more error in the sales forecast, the more unnecessary cost NVIDIA experiences because all of these activities (supply chain, inventory management, financial planning, etc) get thrown off!</p>
<h4>Time Series Deep Learning For Business</h4>
<p><strong><em>Time Series Deep Learning</em></strong> is amazingly accurate on data that has a high presence of autocorrelation because algorithms including <a rel="nofollow" href="https://en.wikipedia.org/wiki/Long_short-term_memory" target="_blank">LSTMs</a> and <a rel="nofollow" href="https://en.wikipedia.org/wiki/Gated_recurrent_unit" target="_blank">GRUs</a> can learn information from sequences regardless of when the pattern occurred. These special <a rel="nofollow" href="https://en.wikipedia.org/wiki/Recurrent_neural_network" target="_blank">RNNs</a> are designed to have a long memory meaning that they excel at learning patterns from both recent observations and observeration that occurred long ago. This makes them perfect for time series! But are they great for sales data? Maybe. Let’s discuss.</p>
<p>Sales data comes in all flavors, but often it has <strong><em>seasonal patterns</em></strong> and <strong><em>trend</em></strong>. The <strong><em>trend</em></strong> can be flat, linear, exponential, etc. This is typically not where the LSTM will have success, but other traditional forecasting methods can detect trend. However, seasonality is different. <strong><em>Seasonality</em></strong> in sales data is a pattern that can happen at multiple frequencies (annual, quarterly, monthly, weekly, and even daily). The LSTM is great at detecting seasonality because it often has autocorrelation. Therefore, LSTM’s and GRU’s can be great options to help improve seasonality detection, and therefore reduce overall forecast error in the <strong><em>Sales Forecast</em></strong>.</p>
<h2>About The Authors</h2>
<p>This <strong><em>DEEP LEARNING TUTORIAL</em></strong> was a combined effort: <a rel="nofollow" href="https://www.linkedin.com/in/sigrid-keydana-9a16b410/" target="_blank">Sigrid Keydana</a> and <a rel="nofollow" href="https://www.linkedin.com/in/mattdancho/" target="_blank">Matt Dancho</a>.</p>
<p>Sigrid is the <em>TensorFlow Developer Advocate at RStudio</em>, where she develops amazing deep learning using the <a rel="nofollow" href="https://tensorflow.rstudio.com/" target="_blank">R TensorFlow API</a>. She’s passionate about deep learning, machine learning and statistics, R, Linux and functional programming. In a short period of time, I’ve developed a tremendous respect for Sigrid. You’ll see why when you walk through the R TensorFlow code in this tutorial</p>
<p>You know me (Matt). I’m the Founder of <a rel="nofollow" href="http://www.business-science.io/" target="_blank">Business Science</a> where I strive for one mission: To empower Data scientists interested in Business and Finance.</p>
<h2>Deep Learning for Time Series Forecasting: Predicting Sunspot Frequency with Keras</h2>
<blockquote>
<p>By Sigrid Keydana, TensorFlow Developer Advocate at RStudio,<br>
And Matt Dancho, Founder of Business Science</p>
</blockquote>
<h2>Forecasting sunspots with deep learning <a></a></h2>
<p>In this post we will examine making time series predictions using the <a rel="nofollow" href="https://stat.ethz.ch/R-manual/R-devel/library/datasets/html/sunspot.month.html" target="_blank">sunspots</a> dataset that ships with base R. Sunspots are dark spots on the sun, associated with lower temperature. Here’s an image from NASA showing the solar phenomenon.</p>
<p><img src="https://i1.wp.com/www.business-science.io/assets/2018-04-18-keras-lstm-sunspots/sunspot_nasa.jpg?w=456" data-recalc-dims="1"></p>
<p>Source: <a rel="nofollow" href="https://www.nasa.gov/content/goddard/largest-sunspot-of-solar-cycle" target="_blank">NASA</a></p>
<p>We’re using the monthly version of the dataset, <code>sunspots.month</code> (there is a yearly version, too).<br>
It contains 265 years worth of data (from 1749 through 2013) on the number of sunspots per month.</p>
<p><img alt="Sunspots Full" src="https://i0.wp.com/www.business-science.io/assets/2018-07-01-keras-lstm-sunspots2/sunspots_full.png?w=456" data-recalc-dims="1"></p>
<p>Forecasting this dataset is challenging because of high short term variability as well as long-term irregularities evident in the cycles. For example, maximum amplitudes reached by the low frequency cycle differ a lot, as does the number of high frequency cycle steps needed to reach that maximum low frequency cycle height.</p>
<p>Our post will focus on two dominant aspects: how to apply deep learning to time series forecasting, and how to properly apply cross validation in this domain.<br>
For the latter, we will use the <a rel="nofollow" href="https://cran.r-project.org/package=rsample" target="_blank">rsample</a> package that allows to do resampling on time series data.<br>
As to the former, our goal is not to reach utmost performance but to show the general course of action when using recurrent neural networks to model this kind of data.</p>
<h2>Recurrent neural networks</h2>
<p>When our data has a sequential structure, it is recurrent neural networks (RNNs) we use to model it.</p>
<p>As of today, among RNNs, the best established architectures are the GRU (Gated Recurrent Unit) and the LSTM (Long Short Term Memory). For today, let’s not zoom in on what makes them special, but on what they have in common with the most stripped-down RNN: the basic recurrence structure.</p>
<p>In contrast to the prototype of a neural network, often called Multilayer Perceptron (MLP), the RNN has a state that is carried on over time. This is nicely seen in this diagram from <a rel="nofollow" href="http://www.deeplearningbook.org/" target="_blank">Goodfellow et al.</a>, a.k.a. the “bible of deep learning”:</p>
<p><img alt="Source: http://www.deeplearningbook.org" src="https://i2.wp.com/www.business-science.io/assets/2018-07-01-keras-lstm-sunspots2/rnn.png?w=456" data-recalc-dims="1"></p>
<p>At each time, the state is a combination of the current input and the previous hidden state. This is reminiscent of autoregressive models, but with neural networks, there has to be some point where we halt the dependence.</p>
<p>That’s because in order to determine the weights, we keep calculating how our loss changes as the input changes.<br>
Now if the input we have to consider, at an arbitrary timestep, ranges back indefinitely – then we will not be able to calculate all those gradients.<br>
In practice, then, our hidden state will, at every iteration, be carried forward through a fixed number of steps.</p>
<p>We’ll come back to that as soon as we’ve loaded and pre-processed the data.</p>
<h2>Setup, pre-processing, and exploration</h2>
<h3>Libraries</h3>
<p>Here, first, are the libraries needed for this tutorial.</p>
<figure>
<pre><code data-lang="r"><span># Core Tidyverse</span><span>
</span><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>library</span><span>(</span><span>glue</span><span>)</span><span>
</span><span>library</span><span>(</span><span>forcats</span><span>)</span><span>
</span><span># Time Series</span><span>
</span><span>library</span><span>(</span><span>timetk</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tidyquant</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tibbletime</span><span>)</span><span>
</span><span># Visualization</span><span>
</span><span>library</span><span>(</span><span>cowplot</span><span>)</span><span>
</span><span># Preprocessing</span><span>
</span><span>library</span><span>(</span><span>recipes</span><span>)</span><span>
</span><span># Sampling / Accuracy</span><span>
</span><span>library</span><span>(</span><span>rsample</span><span>)</span><span>
</span><span>library</span><span>(</span><span>yardstick</span><span>)</span><span> </span><span># Modeling</span><span>
</span><span>library</span><span>(</span><span>keras</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tfruns</span><span>)</span></code></pre>
</figure>
<p>If you have not previously run Keras in R, you will need to install Keras using the <code>install_keras()</code> function.</p>
<figure>
<pre><code data-lang="r"><span># Install Keras if you have not installed before</span><span>
</span><span>install_keras</span><span>()</span></code></pre>
</figure>
<h3>Data</h3>
<p><code>sunspot.month</code> is a <code>ts</code> class (not tidy), so we’ll convert to a tidy data set using the <code>tk_tbl()</code> function from <code>timetk</code>. We use this instead of <code>as.tibble()</code> from <code>tibble</code> to automatically preserve the time series index as a <code>zoo</code> <code>yearmon</code> index. Last, we’ll convert the <code>zoo</code> index to date using <code>lubridate::as_date()</code> (loaded with <code>tidyquant</code>) and then change to a <code>tbl_time</code> object to make time series operations easier.</p>
<figure>
<pre><code data-lang="r"><span>sun_spots</span><span> </span><span>&lt;-</span><span> </span><span>datasets</span><span>::</span><span>sunspot.month</span><span> </span><span>%>%</span><span>
</span><span>tk_tbl</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>index</span><span> </span><span>=</span><span> </span><span>as_date</span><span>(</span><span>index</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>as_tbl_time</span><span>(</span><span>index</span><span> </span><span>=</span><span> </span><span>index</span><span>)</span><span>
</span><span>sun_spots</span></code></pre>
</figure>
<div>
<div>
<pre><code># A time tibble: 3,177 x 2
# Index: index
   index      value
   </code></pre></div></div>