<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/time-series-analysis-in-r-part-2-time-series-transformations/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://datascienceplus.com/time-series-analysis-in-r-part-2-time-series-transformations/"> R Programming – DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In <a rel="nofollow" href="https://datascienceplus.com/time-series-analysis-in-r-part-1-the-time-series-object/" target="_blank">Part 1</a> of this series, we got started by looking at the <code>ts</code> object in R and how it represents time series data. In Part 2, I’ll discuss some of the many time series transformation functions that are available in R. This is by no means an exhaustive catalog. If you feel I left out anything important, please let me know. I compile these posts as a guide in RMarkdown which I plan to make available on the web soon. </p>
<p>Often in time series analysis and modeling, we will want to transform data. There are a number of different functions that can be used to transform time series data such as the difference, log, moving average, percent change, lag, or cumulative sum. These type of function are useful for both visualizing time series data and for modeling time series. For example, the moving average function can be used to more easily visualize a high-variance time series and is also a critical part the ARIMA family of models. Functions such as the difference, percent change, and log difference are helpful for making non-stationary data stationary.</p>
<p>Let’s start with a very common operation. To take a lag we use the <code>lag()</code> function from the <code>stats</code> package. It takes a <code>ts</code> object and a value for the <code>lag</code> argument.</p>
<pre>tseries_lag1 &lt;- lag(tseries, 1)
head(cbind(tseries, tseries_lag1))
<em>tseries tseries_lag1
1999 Q4 NA 7.07667
2000 Q1 7.07667 10.38876
2000 Q2 10.38876 23.91096
2000 Q3 23.91096 14.49356
2000 Q4 14.49356 15.90501
2001 Q1 15.90501 28.00545</em></pre>
<p>Notice that when we do this, <code>tseries_lag1</code> is equivalent to <code>tseries</code> offset by 1, such that the value for 2000q1 in <code>tseries</code> become the value for 1999q4 in <code>tseries_lag1</code> and so on. We can do this for any lag of our choosing. Let’s try 3 instead:</p>
<pre>tseries_lag3 &lt;- lag(tseries, 3)
head(cbind(tseries, tseries_lag3))
<em> tseries tseries_lag3
1999 Q2 NA 7.07667
1999 Q3 NA 10.38876
1999 Q4 NA 23.91096
2000 Q1 7.07667 14.49356
2000 Q2 10.38876 15.90501
2000 Q3 23.91096 28.00545</em></pre>
<p>We can also do this in the opposite direction using a negative value for the <code>lag</code> argument, effectively creating a lead series instead:</p>
<pre>tseries_lead1 &lt;- lag(tseries, -1)
head(cbind(tseries, tseries_lead1))
<em> tseries tseries_lead1
2000 Q1 7.07667 NA
2000 Q2 10.38876 7.07667
2000 Q3 23.91096 10.38876
2000 Q4 14.49356 23.91096
2001 Q1 15.90501 14.49356
2001 Q2 28.00545 15.90501</em></pre>
<p>Notice that <code>tseries_lead1</code> now leads the original <code>tseries</code> by one time period.<br>
We can take differences of a time series as well. This is equivalent to taking the difference between each value and its previous value:</p>
<pre>tseries_diff1 &lt;- diff(tseries, lag = 1)
tm &lt;- cbind(tseries, tseries_diff1)
head(tm)
<em>tseries tseries_diff1
[1,] 7.07667 NA
[2,] 10.38876 3.312087
[3,] 23.91096 13.522201
[4,] 14.49356 -9.417399
[5,] 15.90501 1.411455
[6,] 28.00545 12.100441</em></pre>
<pre>plot.ts(tm)</pre>
<p>Give this plot:<br>
<a rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_1.png?ssl=1" target="_blank"><img sizes="(max-width: 1344px) 100vw, 1344px" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_1.png?w=450&ssl=1" alt="" class="wp-image-8558" data-recalc-dims="1"></a></p>
<p>As you can see, taking a difference is an effective way to remove a trend and make a time series stationary.<br>
The difference can be calculated over more than one lag. To do so, simply alter the value of the <code>lag</code> argument.</p>
<pre>tseries_diff2 &lt;- diff(tseries, lag = 2)
tm &lt;- cbind(tseries, tseries_diff2)
head(tm)
<em> tseries tseries_diff2
2000 Q1 7.07667 NA
2000 Q2 10.38876 NA
2000 Q3 23.91096 16.834288
2000 Q4 14.49356 4.104801
2001 Q1 15.90501 -8.005944
2001 Q2 28.00545 13.511896</em></pre>
<pre>plot(tm)</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_2.png?ssl=1" target="_blank"><img sizes="(max-width: 1344px) 100vw, 1344px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_2.png?w=450&ssl=1" alt="" class="wp-image-8560" data-recalc-dims="1"></a></p>
<p>Some time series transformation functions are useful for series in which the variance gets larger over time. These range from the basic logarithm function to the Box-Cox group of transformations (of which the natural logarithm is a special case). We can take the log of a time series using the log function in the same way that we would take the log of a vector. Let’s generate a time series that has increasing variance:</p>
<pre>trend &lt;- ts(seq(from = 10, to = 110))
cycle &lt;- ts(sin(trend)) * 0.2 * trend
tseries_h &lt;- trend + cycle
plot.ts(tseries_h)</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_3.png?ssl=1" target="_blank"><img sizes="(max-width: 1344px) 100vw, 1344px" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_3.png?w=450&ssl=1" alt="" class="wp-image-8562" data-recalc-dims="1"></a></p>
<p>As we’ll discuss in more detail, the prevalence of increasing, or more generally non-constant, variance is called heteroskedasticity and can cause problems in linear regression. Often, it will need to be corrected before modeling. One way to do this is taking a log:</p>
<pre>tseries_log &lt;- log(tseries_h)
tm &lt;-cbind(tseries_h, tseries_log)
plot.ts(tm)</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_4.png?ssl=1" target="_blank"><img sizes="(max-width: 1344px) 100vw, 1344px" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_4.png?w=450&ssl=1" alt="" class="wp-image-8563" data-recalc-dims="1"></a></p>
<p>There are other more advanced ways of eliminating non-constant variance, one of which is the Box-Cox transformation, which allows us a bit more control over the transformation. The Box-Cox takes the form (Hyndman and Athanasopoulos, 2013):</p>
<p>$$w_t = \begin{cases} \log{ y_t }, \text{ if } \lambda = 0; \\ \frac{ ({y_t}^{\lambda} – 1) }{ \lambda },\text{ otherwise } \\ \end{cases} $$</p>
<p>In order to demonstrate the Box-Cox transformation, I’ll introduce Rob Hyndman’s `forecast` package:</p>
<pre>library(forecast)</pre>
<p>It has two functions that are of use here. The primary function is <code>BoxCox()</code>, which will return a transformed time series given a time series and a value for the parameter <code>lambda</code>:</p>
<pre>plot.ts(BoxCox(tseries_h, lambda = 0.5))</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_5.png?ssl=1" target="_blank"><img sizes="(max-width: 1344px) 100vw, 1344px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_5.png?w=450&ssl=1" alt="" class="wp-image-8564" data-recalc-dims="1"></a></p>
<p>Notice that this value of <code>lambda</code> here does not entirely take care of the heteroskedasticity problem. We can experiment with different values of <code>lambda</code>, or we can use the <code>BoxCox.lambda()</code> function, which will provide us an optimal value for parameter <code>lambda</code>:</p>
<pre>lambda &lt;- BoxCox.lambda(tseries_h)
print(lambda)
<em>[1] 0.05527724</em></pre>
<pre>plot.ts(BoxCox(tseries_h, lambda = lambda))</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_6.png?ssl=1" target="_blank"><img sizes="(max-width: 1344px) 100vw, 1344px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_6.png?w=450&ssl=1" alt="" class="wp-image-8565" data-recalc-dims="1"></a></p>
<p>The <code>BoxCox.lambda()</code> function has chosen the value 0.055. If we then use this value in our <code>BoxCox()</code> function, it returns a time series that appears to have constant variance.</p>
<p>Another common calculation that we may want to perform on time series is the percent change from one period to another. Because it seems that R does not have a function for performing this calculation, we can do it ourselves.</p>
<pre>tseries_pch &lt;- tseries / lag(tseries, -1) - 1
head(cbind(tseries, tseries_pch))
<em> tseries tseries_pch
2000 Q1 7.07667 NA
2000 Q2 10.38876 0.46802901
2000 Q3 23.91096 1.30161865
2000 Q4 14.49356 -0.39385287
2001 Q1 15.90501 0.09738501
2001 Q2 28.00545 0.76079409</em></pre>
<p>We can turn this into a function so that we can easily calculate percent change in the future. We’ll add an argument to specify the number of periods over which we want to calculate the change and set it to 1 by default. Additionally, we’ll add some argument verification.</p>
<pre>pch &lt;- function(data, lag = 1) {
    # argument verification
    if (!is.ts(data)) stop(&quot;data must be of type ts&quot;)
    if (!is.numeric(lag)) stop(&quot;lag must be of type numeric&quot;)
    # return percent change
    data / lag(data, -lag) - 1
}
head(cbind(tseries, pch(tseries)))
<em> tseries pch(tseries)
2000 Q1 7.07667 NA
2000 Q2 10.38876 0.46802901
2000 Q3 23.91096 1.30161865
2000 Q4 14.49356 -0.39385287
2001 Q1 15.90501 0.09738501
2001 Q2 28.00545 0.76079409</em></pre>
<p>Now if we wished to calculate the percentage change over more than one period we can do so. Let’s say we wanted to calculate the year over year percent change. We can call our <code>pch()</code> function with a lag of 4.</p>
<pre>tseries_pchya &lt;- pch(tseries, lag=4)</pre>
<p>Two of the functions that we have discussed so far, the difference and the log, are often combined in time series analysis. The log difference function is useful for making non-stationary data stationary and has some other useful properties. We can calculate the log difference in R by simply combining the <code>log()</code> and <code>diff()</code> functions.</p>
<pre>tseries_dlog &lt;- ts(diff(log(tseries)), start = c(2000, 1), frequency = 4)
plot.ts(cbind(tseries, tseries_dlog))</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_7.png?ssl=1" target="_blank"><img sizes="(max-width: 1344px) 100vw, 1344px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_7.png?w=450&ssl=1" alt="" class="wp-image-8566" data-recalc-dims="1"></a></p>
<p>Notice that after taking the log return, <code>tseries</code> appears to be stationary. We see some higher than normal volatility in the beginning of the series. This is due largely to the fact that the series levels start off so small. This leads into a nice property of the log return function, which is that it is a close approximation to the percent change:</p>
<pre>plot.ts(cbind(pch(tseries), tseries_dlog))</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_8.png?ssl=1" target="_blank"><img sizes="(max-width: 1344px) 100vw, 1344px" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_8.png?w=450&ssl=1" alt="" class="wp-image-8567" data-recalc-dims="1"></a></p>
<p>This similarity is only approximate. The relationship does break down somewhat when the percent change from one period to the next is particularly large. You can read a good discussion of this topic <a rel="nofollow" href="https://stats.stackexchange.com/questions/244199/why-is-it-that-natural-log-changes-are-percentage-changes-what-is-about-logs-th" target="_blank">here</a>.</p>
<p>A moving average is another essential function for working with time series. For series with particularly high volatility, a moving average can help us to more clearly visualize its trend. Unfortunately, base R does not (to my knowledge) have a convenient function for calculating the moving average of a time series directly. However, we can use base R’s <code>filter()</code> function, which allows us to perform general linear filtering. We can set up the parameters of this function to be a moving average (Shumway and Stoffer, 2011). Here we apply the <code>filter()</code> function to <code>tseries</code> to create a 5 period moving average. The <code>filter</code> argument lets up specify the filter, which in this case is just a weighted average of 5 observations. The <code>sides</code> argument allows us to specify whether we want to apply the filter over past values (<code>sides</code> = 1), or to both past and future values (<code>sides</code> = 2). Here I set <code>sides</code> to 1 indicating that I want a trailing moving average:</p>
<pre>tseries_lf5 &lt;- filter(tseries, filter = rep(1/5, 5), sides = 1)
plot.ts(cbind(tseries, tseries_lf5), plot.type = 'single', col = c('black', 'red'))</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_9.png?ssl=1" target="_blank"><img sizes="(max-width: 1344px) 100vw, 1344px" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/09/tsguide2_9.png?w=450&ssl=1" alt="" class="wp-image-8568" data-recalc-dims="1"></a></p>
<p>The fact that we have to define the linear filter each time we use this function makes it a little cumbersome to use. If we don’t mind introducing a dependency to our code, we could use the <code>SMA()</code> function from the <code>TTR</code> package or the <code>ma()</code> function from the <code>forecast</code> package. The <code>SMA()</code> function takes a <code>ts</code> object and a value for <code>n</code> – the window over which we want to calculate the moving average.</p>
<pre>library(TTR)
tseries_ma5 &lt;- SMA(tseries, n = 5)</pre>
<p>The <code>ma()</code> function from the <code>forecast</code> package also performs moving average calculations. We supply the time series and a value for the <code>order</code> argument.</p>
<pre>tseries_ma5fore &lt;- ma(tseries, order = 5)</pre>
<p>Let’s compare the results. The <code>SMA()</code> function returns a trailing moving average where each value is the mean of the n most recent trailing values. This is equivalent to the results we get from using the <code>filter()</code> function. The <code>ma()</code> function from the <code>forecast</code> package returns a centered moving average. In this case <code>tseries_ma5for</code> is equal to the average of the current observation, the previous two observation, and the next two observations. Which one you use would depend on your application.</p>
<pre>
<em> tseries tseries_lf5 tseries_ma5 tseries_ma5fore
2000 Q1 7.076670 NA NA NA
2000 Q2 10.388758 NA NA NA
2000 Q3 23.910958 NA NA 14.35499
2000 Q4 14.493559 NA NA 18.54075
2001 Q1 15.905014 14.35499 14.35499 20.50828
2001 Q2 28.005455 18.54075 18.54075 17.55500
2001 Q3 20.226413 20.50828 20.50828 17.49470
2001 Q4 9.144571 17.55500 17.55500 17.68977
2002 Q1 14.192030 17.49470 17.49470 18.00239
2002 Q2 16.880366 17.68977 17.68977 18.86085</em></pre>
<p>That’s it for Part 2. I don’t know about you, but I am sick of working with generated data. I want to work with some real world data. In Part 3 we’ll do just that using <a rel="nofollow" href="https://www.quandl.com/" target="_blank">Quandl</a>. See you then.</p>
<h3>References</h3>
<ul>Hyndman, Rob J. and George Athanasopoulos (2013) Forecasting: Principles and Practice. <a rel="nofollow" href="https://www.otexts.org/fpp" target="_blank">https://www.otexts.org/fpp</a><br>
Shumway, Robert H. and David S. Stoffer (2011) Time Series Analysis and Its Applications With R Examples. New York, NY: Springer.</ul>
<ul></ul>
<p><strong>
<p>Related Post</p>
<p>
<ol>
<li><a rel="nofollow" href="https://datascienceplus.com/time-series-analysis-in-r-part-1-the-time-series-object/" title="Permanent Link toTime Series Analysis in R Part 1: The Time Series Object" target="_blank">Time Series Analysis in R Part 1: The Time Series Object</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/parsing-text-for-emotion-terms-analysis-visualization-using-r/" title="Permanent Link toParsing Text for Emotion Terms: Analysis &amp; Visualization Using R" target="_blank">Parsing Text for Emotion Terms: Analysis &amp; Visualization Using R</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/using-mongodb-with-r/" title="Permanent Link toUsing MongoDB with R" target="_blank">Using MongoDB with R</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/finding-optimal-number-of-clusters/" title="Permanent Link toFinding Optimal Number of Clusters" target="_blank">Finding Optimal Number of Clusters</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/analyzing-the-first-presidential-debate/" title="Permanent Link toAnalyzing the first Presidential Debate" target="_blank">Analyzing the first Presidential Debate</a></li>
</ol></strong>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://datascienceplus.com/time-series-analysis-in-r-part-2-time-series-transformations/"> R Programming – DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/Maq7N3cdzEs" height="1"></a></div>