<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/why-time-series-forecasts-prediction-intervals-arent-as-good-as-wed-hope/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://ellisp.github.io/blog/2016/12/07/arima-prediction-intervals"> Peter's stats stuff - R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h2>Five different sources of error</h2>
<p>When it comes to time series forecasts from a statistical model we have five sources of error:</p>
<ol>
<li>Random individual errors</li>
<li>Random estimates of parameters (eg the coefficients for each autoregressive term)</li>
<li>Uncertain meta-parameters (eg number of autoregressive terms)</li>
<li>Unsure if the model was right for the historical data</li>
<li>Even given #4, unsure if the model will continue to be right</li>
</ol>
<p>A <em>confidence interval</em> is an estimate of the statistical uncertainty of the estimated parameters in the model. It usually estimates the uncertainty source #2 above, not interested in #1 and conditional on the uncertainty of sources #3, #4 and #5 all being taken out of the picture. A <em>prediction interval</em> should ideally take all five sources into account (see Rob Hyndman for more on <a target="_blank" rel="nofollow" href="http://robjhyndman.com/hyndsight/intervals/">the distinction between prediction and confidence intervals</a>).</p>
<p>Unfortunately, the standard ways of providing time series prediction intervals <a target="_blank" rel="nofollow" href="http://robjhyndman.com/hyndsight/narrow-pi/">typically only take source #1 into account</a> – random individual errors. This differs from standard prediction intervals from more straightforward regression and generalized linear models, which at least usually factor in uncertainty of the estimates of parameters.</p>
<p>The problem is that <strong>for all but the most trivial time series forecasting method there is no simple way of estimating the uncertainty that comes from having estimated the parameters from the data, and much less so the values of meta-parameters</strong> like the amount of differencing needed, how many autoregressive terms, how many moving average terms, etc (those example meta-parameters come from the Box-Jenkins ARIMA approach, but other forecasting methods have their own meta-parameters to estimate too).</p>
<h2>Demonstration of the cost of estimating meta-parameters</h2>
<p>Here’s a simple simulation to show the cost of estimating the meta-parameters, even when sources of error #4 and #5 can be discounted. I generated 10,000 time series, each 100 observations long, from an ARIMA(1, 1, 1) process and split them into a 90 observation training set and 10 observation test set. This creates data that looks like this (just four examples shown):</p>
<p><img alt="examples" src="http://ellisp.github.io/img/0072-examples.svg"></p>
<p>For each one of those datasets, I fit an ARIMA model to the 90 observation test set two ways:</p>
<ul>
<li>using <code>forecast::Arima</code> and advising the algorithm that the model to use is (1,1,1)</li>
<li>using <code>forecast::auto.arima</code> and forcing the algorithm to estimate from the data the correct level of differencing, number of autoregression terms, and number of moving average terms.</li>
</ul>
<p>Then I compared the resulting forecast prediction intervals (at 80% and 95% confidence levels) of the last 10 observations to see what percentage of forecasts actually had the true values in their range. Here’s the result:</p>
<p><img alt="result" src="http://ellisp.github.io/img/0072-results.svg"></p>
<p>As we can see, the <code>Arima</code> model which knew in advance the correct meta-parameter specification didn’t do too badly. Its prediction intervals contain the correct values only just below the promised 80% and 95% of times. But the <code>auto.arima</code> method, which had to estimate the meta-parameters itself, did noticeably worse. Not only that, as the forecast horizon increases, its prediction intervals got increasingly over-optimistic. Having the wrong meta-parameters for your model becomes more of a problem the further you go out (I suspect this is particularly the case for the amount of differencing that needs to be done – an unlucky choice here would have big implications for estimating the trend).</p>
<p>An important thing to remember is that out in the wild, results will generally be worse than this. In this occasion, both methods had the luxury of fitting the same family of models that had generated the data, and knowing that the final ten observations were generated the same way. No outliers, black swans, or changes in data generating process to worry about. One way of partially dealing with this problem with careful use of the method implemented in the <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=forecastHybrid"><code>forecastHybrid</code></a> R package by David Shaub and myself was the subject of <a target="_blank" rel="nofollow" href="http://ellisp.github.io/presentations/ellis-prediction-intervals-for-ensemble-forecasts.pptx">my talk yesterday at the Australian Statistical Conference</a>.</p>
<h2>Code</h2>
<p>Here’s the code that does the simulation above:</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>forecast</span><span>)</span><span>
</span><span>library</span><span>(</span><span>foreach</span><span>)</span><span>
</span><span>library</span><span>(</span><span>doParallel</span><span>)</span><span>
</span><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tidyr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>ggplot2</span><span>)</span><span>
</span><span>library</span><span>(</span><span>scales</span><span>)</span><span>
</span><span>library</span><span>(</span><span>stringr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>grid</span><span>)</span><span>
</span><span>library</span><span>(</span><span>colorspace</span><span>)</span><span>
</span><span>#=============prediction interval functions===================
</span><span>#' takes a forecast object, and the actual results, and returns a data frame
#' binary indicator of success as well as the horizon for each row and an optional
#' &quot;type&quot; column.
</span><span>accuracy_pi</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>fc</span><span>,</span><span> </span><span>actual</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span> </span><span>labs</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;80%&quot;</span><span>,</span><span> </span><span>&quot;95%&quot;</span><span>)){</span><span>
</span><span>if</span><span>(</span><span>nrow</span><span>(</span><span>fc</span><span>$</span><span>upper</span><span>)</span><span> </span><span>!=</span><span> </span><span>length</span><span>(</span><span>actual</span><span>)){</span><span>
</span><span>stop</span><span>(</span><span>&quot;`fc` must be a forecast object with `upper` and `lower` objects that are matrices with number of rows equal to length of `actual`&quot;</span><span>)</span><span>
</span><span>}</span><span>
</span><span>actual</span><span> </span><span>&lt;-</span><span> </span><span>as.vector</span><span>(</span><span>actual</span><span>)</span><span>
</span><span>h</span><span> </span><span>&lt;-</span><span> </span><span>length</span><span>(</span><span>actual</span><span>)</span><span>
</span><span># take advantage of actual being a vector that will recycle, going down each
</span><span> </span><span># column of the two matrices of logical conditions we superimpose here:
</span><span> </span><span>tmp</span><span> </span><span>&lt;-</span><span> </span><span>as.data.frame</span><span>(</span><span>fc</span><span>$</span><span>lower</span><span> </span><span>&lt;</span><span> </span><span>actual</span><span> </span><span>&amp;</span><span> </span><span>fc</span><span>$</span><span>upper</span><span> </span><span>></span><span> </span><span>actual</span><span>)</span><span>
</span><span>names</span><span>(</span><span>tmp</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>labs</span><span>
</span><span>tmp</span><span> </span><span>&lt;-</span><span> </span><span>cbind</span><span>(</span><span>tmp</span><span>,</span><span> </span><span>&quot;h&quot;</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>h</span><span>,</span><span>
</span><span>&quot;type&quot;</span><span> </span><span>=</span><span> </span><span>rep</span><span>(</span><span>type</span><span>,</span><span> </span><span>h</span><span>))</span><span>
</span><span>return</span><span>(</span><span>tmp</span><span>)</span><span>
</span><span>}</span><span>
</span><span>#===================simulations=======================
</span><span>n</span><span> </span><span>&lt;-</span><span> </span><span>100</span><span> </span><span># size of full series
</span><span>h</span><span> </span><span>&lt;-</span><span> </span><span>10</span><span> </span><span># how much of the series to forecast forward
</span><span>R</span><span> </span><span>&lt;-</span><span> </span><span>10000</span><span> </span><span># how many datasets to try this with
</span><span>the_model</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>ar</span><span> </span><span>=</span><span> </span><span>0.5</span><span>,</span><span> </span><span>ma</span><span> </span><span>=</span><span> </span><span>0.5</span><span>)</span><span>
</span><span>cluster</span><span> </span><span>&lt;-</span><span> </span><span>makeCluster</span><span>(</span><span>7</span><span>)</span><span> </span><span># only any good if you have at least 7 processors :)
</span><span>registerDoParallel</span><span>(</span><span>cluster</span><span>)</span><span>
</span><span>clusterEvalQ</span><span>(</span><span>cluster</span><span>,</span><span> </span><span>{</span><span>
</span><span>library</span><span>(</span><span>forecast</span><span>)</span><span>
</span><span>})</span><span>
</span><span>results</span><span> </span><span>&lt;-</span><span> </span><span>foreach</span><span>(</span><span>1</span><span>:</span><span>R</span><span>,</span><span> </span><span>.combine</span><span> </span><span>=</span><span> </span><span>rbind</span><span>)</span><span> </span><span>%dopar%</span><span> </span><span>{</span><span>
</span><span>sim_data</span><span> </span><span>&lt;-</span><span> </span><span>ts</span><span>(</span><span>cumsum</span><span>(</span><span>arima.sim</span><span>(</span><span>model</span><span> </span><span>=</span><span> </span><span>the_model</span><span>,</span><span> </span><span>n</span><span> </span><span>=</span><span> </span><span>n</span><span>)))</span><span>
</span><span>x</span><span> </span><span>&lt;-</span><span> </span><span>sim_data</span><span>[</span><span>1</span><span>:</span><span>(</span><span>n</span><span> </span><span>-</span><span> </span><span>h</span><span>)]</span><span>
</span><span>actual</span><span> </span><span>&lt;-</span><span> </span><span>sim_data</span><span>[(</span><span>n</span><span> </span><span>-</span><span> </span><span>h</span><span> </span><span>+</span><span> </span><span>1</span><span>)</span><span>:</span><span>n</span><span>]</span><span>
</span><span>mod1</span><span> </span><span>&lt;-</span><span> </span><span>auto.arima</span><span>(</span><span>x</span><span>)</span><span>
</span><span>fc1</span><span> </span><span>&lt;-</span><span> </span><span>forecast</span><span>(</span><span>mod1</span><span>,</span><span> </span><span>h</span><span> </span><span>=</span><span> </span><span>h</span><span>)</span><span>
</span><span>mod2</span><span> </span><span>&lt;-</span><span> </span><span>Arima</span><span>(</span><span>x</span><span>,</span><span> </span><span>order</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>1</span><span>))</span><span>
</span><span>fc2</span><span> </span><span>&lt;-</span><span> </span><span>forecast</span><span>(</span><span>mod2</span><span>,</span><span> </span><span>h</span><span> </span><span>=</span><span> </span><span>h</span><span>)</span><span>
</span><span>rbind</span><span>(</span><span>
</span><span>accuracy_pi</span><span>(</span><span>fc1</span><span>,</span><span> </span><span>actual</span><span> </span><span>=</span><span> </span><span>actual</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;auto&quot;</span><span>),</span><span>
</span><span>accuracy_pi</span><span>(</span><span>fc2</span><span>,</span><span> </span><span>actual</span><span> </span><span>=</span><span> </span><span>actual</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;pre-knowledge&quot;</span><span>)</span><span>
</span><span>)</span><span>
</span><span>}</span><span>
</span><span>stopCluster</span><span>(</span><span>cluster</span><span>)</span><span>
</span><span>#==========present results==================
</span><span>pal</span><span> </span><span>&lt;-</span><span> </span><span>rainbow_hcl</span><span>(</span><span>2</span><span>)</span><span>
</span><span>targetlines</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>
</span><span>limit</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;80%&quot;</span><span>,</span><span> </span><span>&quot;95%&quot;</span><span>),</span><span>
</span><span>yintercept</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>0.8</span><span>,</span><span> </span><span>0.95</span><span>)</span><span>
</span><span>)</span><span>
</span><span>results</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>limit</span><span>,</span><span> </span><span>success</span><span>,</span><span> </span><span>-</span><span>h</span><span>,</span><span> </span><span>-</span><span>type</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>h</span><span>,</span><span> </span><span>type</span><span>,</span><span> </span><span>limit</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarise</span><span>(</span><span>success</span><span> </span><span>=</span><span> </span><span>mean</span><span>(</span><span>success</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>h</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>success</span><span>,</span><span> </span><span>colour</span><span> </span><span>=</span><span> </span><span>type</span><span>))</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>~</span><span>limit</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_y_continuous</span><span>(</span><span>label</span><span> </span><span>=</span><span> </span><span>percent</span><span>)</span><span> </span><span>+</span><span>
</span><span>ggtitle</span><span>(</span><span>&quot;Prediction interval coverage of two modelling methods&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Comparing auto.arima with pre-knowledge of the true data generating process&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Number of periods forecast forward&quot;</span><span>,</span><span>
</span><span>caption</span><span> </span><span>=</span><span> </span><span>&quot;Data generated with an ARIMA(1,1,1) process, which is much more regular than real life data.&quot;</span><span>,</span><span>
</span><span>colour</span><span> </span><span>=</span><span> </span><span>&quot;Modelling approach:&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_colour_manual</span><span>(</span><span>values</span><span> </span><span>=</span><span> </span><span>pal</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_x_continuous</span><span>(</span><span>breaks</span><span> </span><span>=</span><span> </span><span>0</span><span>:</span><span>5</span><span> </span><span>*</span><span> </span><span>2</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_hline</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>targetlines</span><span>,</span><span> </span><span>aes</span><span>(</span><span>yintercept</span><span> </span><span>=</span><span> </span><span>yintercept</span><span>),</span><span> </span><span>colour</span><span> </span><span>=</span><span> </span><span>&quot;violet&quot;</span><span>,</span><span> </span><span>linetype</span><span> </span><span>=</span><span> </span><span>2</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1.1</span><span>)</span><span>
</span><span>grid.text</span><span>(</span><span>0.3</span><span>,</span><span> </span><span>y</span><span>=</span><span> </span><span>0.65</span><span>,</span><span>
</span><span>label</span><span> </span><span>=</span><span> </span><span>str_wrap</span><span>(</span><span>&quot;When the true model family and the true meta-parameters p, d, q are known, coverage is close to the 80% and 95% promised.&quot;</span><span>,</span><span> </span><span>33</span><span>),</span><span>
</span><span>gp</span><span> </span><span>=</span><span> </span><span>gpar</span><span>(</span><span>family</span><span> </span><span>=</span><span> </span><span>&quot;myfont&quot;</span><span>,</span><span> </span><span>col</span><span> </span><span>=</span><span> </span><span>pal</span><span>[</span><span>2</span><span>],</span><span> </span><span>cex</span><span> </span><span>=</span><span> </span><span>0.8</span><span>))</span><span>
</span><span>grid.text</span><span>(</span><span>0.8</span><span>,</span><span> </span><span>y</span><span>=</span><span> </span><span>0.45</span><span>,</span><span>
</span><span>label</span><span> </span><span>=</span><span> </span><span>str_wrap</span><span>(</span><span>&quot;When the meta parameters p, d and q are estimated from the data, coverage is materially less.&quot;</span><span>,</span><span> </span><span>33</span><span>),</span><span>
</span><span>gp</span><span> </span><span>=</span><span> </span><span>gpar</span><span>(</span><span>family</span><span> </span><span>=</span><span> </span><span>&quot;myfont&quot;</span><span>,</span><span> </span><span>col</span><span> </span><span>=</span><span> </span><span>pal</span><span>[</span><span>1</span><span>],</span><span> </span><span>cex</span><span> </span><span>=</span><span> </span><span>0.8</span><span>))</span><span>
</span><span>grid.text</span><span>(</span><span>0.9</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>0.779</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>&quot;Gap from estimating\nmeta-parameters&quot;</span><span>,</span><span>
</span><span>gp</span><span> </span><span>=</span><span> </span><span>gpar</span><span>(</span><span>family</span><span> </span><span>=</span><span> </span><span>&quot;myfont&quot;</span><span>,</span><span> </span><span>col</span><span> </span><span>=</span><span> </span><span>&quot;grey50&quot;</span><span>,</span><span> </span><span>cex</span><span> </span><span>=</span><span> </span><span>0.7</span><span>))</span><span>
</span><span>grid.text</span><span>(</span><span>0.85</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>0.840</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>&quot;Gap from estimating parameters&quot;</span><span>,</span><span>
</span><span>gp</span><span> </span><span>=</span><span> </span><span>gpar</span><span>(</span><span>family</span><span> </span><span>=</span><span> </span><span>&quot;myfont&quot;</span><span>,</span><span> </span><span>col</span><span> </span><span>=</span><span> </span><span>&quot;grey50&quot;</span><span>,</span><span> </span><span>cex</span><span> </span><span>=</span><span> </span><span>0.7</span><span>))</span><span>
</span><span>#=================demo data============
# four example datasets
</span><span>set.seed</span><span>(</span><span>123</span><span>)</span><span> </span><span># for reproducibility
</span><span>par</span><span>(</span><span>mfrow</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>2</span><span>,</span><span> </span><span>2</span><span>),</span><span> </span><span>bty</span><span> </span><span>=</span><span> </span><span>&quot;l&quot;</span><span>,</span><span> </span><span>font.main</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span><span>for</span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>4</span><span>){</span><span>
</span><span>sim_data</span><span> </span><span>&lt;-</span><span> </span><span>ts</span><span>(</span><span>cumsum</span><span>(</span><span>arima.sim</span><span>(</span><span>model</span><span> </span><span>=</span><span> </span><span>the_model</span><span>,</span><span> </span><span>n</span><span> </span><span>=</span><span> </span><span>n</span><span>)))</span><span>
</span><span>x</span><span> </span><span>&lt;-</span><span> </span><span>sim_data</span><span>[</span><span>1</span><span>:</span><span>(</span><span>n</span><span> </span><span>-</span><span> </span><span>h</span><span>)]</span><span>
</span><span>actual</span><span> </span><span>&lt;-</span><span> </span><span>sim_data</span><span>[(</span><span>n</span><span> </span><span>-</span><span> </span><span>h</span><span> </span><span>+</span><span> </span><span>1</span><span>)</span><span>:</span><span>n</span><span>]</span><span>
</span><span>mod2</span><span> </span><span>&lt;-</span><span> </span><span>Arima</span><span>(</span><span>x</span><span>,</span><span> </span><span>order</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>1</span><span>))</span><span>
</span><span>fc2</span><span> </span><span>&lt;-</span><span> </span><span>forecast</span><span>(</span><span>mod2</span><span>,</span><span> </span><span>h</span><span> </span><span>=</span><span> </span><span>h</span><span>)</span><span> </span><span>plot</span><span>(</span><span>fc2</span><span>,</span><span> </span><span>main</span><span> </span><span>=</span><span> </span><span>paste0</span><span>(</span><span>&quot;Simulated ARIMA(1,1,1) data: &quot;</span><span>,</span><span> </span><span>i</span><span>))</span><span>
</span><span>lines</span><span>((</span><span>n</span><span> </span><span>-</span><span> </span><span>h</span><span> </span><span>+</span><span> </span><span>1</span><span>)</span><span>:</span><span>n</span><span>,</span><span> </span><span>actual</span><span>,</span><span> </span><span>col</span><span> </span><span>=</span><span> </span><span>&quot;red&quot;</span><span>)</span><span>
</span><span>}</span></code></pre>
</figure>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://ellisp.github.io/blog/2016/12/07/arima-prediction-intervals"> Peter's stats stuff - R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/t2k16G6NVC0"></a></div>