<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/my-kiss-attempt-to-rstatsgoes10k-contest/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://jkunst.com/r/my-KISS-attempt-to-rstatsgoes10k-contest/"> Jkunst - R category</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Last year <a target="_blank" rel="nofollow" href="https://blog.eoda.de/2016/12/21/rstatsgoes10k-eoda-is-hosting-a-contest-in-celebration-of-r/">eoda.de</a><br>
launched a contest to predict when the R packages will be 10k. So this<br>
is a really good opportunity to use (finally) the <code>forecastHybrid</code> package developed by<br>
<a target="_blank" rel="nofollow" href="http://ellisp.github.io/">Peter Ellis</a> and David Shaub.</p>
<blockquote>
<p>This will be a really KISS-naive-simply-raw try to get a<br>
reasonable prediction. No transformations, no CV. etc. But you can do better!<br>
– <cite>The writer.</cite></p>
</blockquote>
<p><img title="Naive Attempt to rstatsgoes10k" alt="" src="https://i1.wp.com/jkunst.com/images/featured-image/rstatsgoes10k.gif?w=456" data-recalc-dims="1"></p>
<p>Let’s load the packages!</p>
<div>
<pre><code><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>rvest</span><span>)</span><span>
</span><span>library</span><span>(</span><span>janitor</span><span>)</span><span>
</span><span>library</span><span>(</span><span>lubridate</span><span>)</span><span>
</span><span>library</span><span>(</span><span>highcharter</span><span>)</span><span>
</span><span>library</span><span>(</span><span>forecast</span><span>)</span><span>
</span><span>library</span><span>(</span><span>forecastHybrid</span><span>)</span><span>
</span><span>library</span><span>(</span><span>zoo</span><span>)</span><span>
</span><span>options</span><span>(</span><span>highcharter.theme</span><span> </span><span>=</span><span> </span><span>hc_theme_smpl</span><span>())</span><span>
</span></code></pre>
</div>
<p>The data will be extracted from the list of packages by date from CRAN.<br>
Then we’ll make some wrangling to get the cumulative sum of the packages<br>
by day.</p>
<div>
<pre><code><span>packages</span><span> </span><span>&lt;-</span><span> </span><span>&quot;https://cran.r-project.org/web/packages/available_packages_by_date.html&quot;</span><span> </span><span>%>%</span><span> </span><span>read_html</span><span>()</span><span> </span><span>%>%</span><span> </span><span>html_table</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>.</span><span>[[</span><span>1</span><span>]]</span><span> </span><span>%>%</span><span> </span><span>tbl_df</span><span>()</span><span>
</span><span>names</span><span>(</span><span>packages</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>tolower</span><span>(</span><span>names</span><span>(</span><span>packages</span><span>))</span><span>
</span><span>packages</span><span> </span><span>&lt;-</span><span> </span><span>mutate</span><span>(</span><span>packages</span><span>,</span><span> </span><span>date</span><span> </span><span>=</span><span> </span><span>ymd</span><span>(</span><span>date</span><span>))</span><span>
</span><span>glimpse</span><span>(</span><span>packages</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Observations: 9,858
## Variables: 3
## $ date    &lt;date> 2017-01-07, 2017-01-07, 2017-01-07, 2017-01-07, 2017-...
## $ package &lt;chr> &quot;AER&quot;, &quot;c212&quot;, &quot;caseMatch&quot;, &quot;clustRcompaR&quot;, &quot;dat&quot;, &quot;gd...
## $ title   &lt;chr> &quot;Applied Econometrics with R&quot;, &quot;Methods for Detecting ...
</code></pre>
</div>
<div>
<pre><code><span>c</span><span>(</span><span>min</span><span>(</span><span>packages</span><span>$</span><span>date</span><span>),</span><span> </span><span>max</span><span>(</span><span>packages</span><span>$</span><span>date</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] &quot;2005-10-29&quot; &quot;2017-01-07&quot;
</code></pre>
</div>
<div>
<pre><code><span>data</span><span> </span><span>&lt;-</span><span> </span><span>packages</span><span> </span><span>%>%</span><span> </span><span>group_by</span><span>(</span><span>date</span><span>)</span><span> </span><span>%>%</span><span> </span><span>summarise</span><span>(</span><span>n</span><span> </span><span>=</span><span> </span><span>n</span><span>())</span><span> </span><span>%>%</span><span> </span><span>left_join</span><span>(</span><span>data_frame</span><span>(</span><span>date</span><span> </span><span>=</span><span> </span><span>seq</span><span>(</span><span>min</span><span>(</span><span>packages</span><span>$</span><span>date</span><span>),</span><span> </span><span>max</span><span>(</span><span>packages</span><span>$</span><span>date</span><span>),</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>1</span><span>)),</span><span>
</span><span>.</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;date&quot;</span><span>)</span><span>
</span><span>data</span><span> </span><span>&lt;-</span><span> </span><span>data</span><span> </span><span>%>%</span><span> </span><span>mutate</span><span>(</span><span>
</span><span>n</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>is.na</span><span>(</span><span>n</span><span>),</span><span> </span><span>0</span><span>,</span><span> </span><span>n</span><span>),</span><span>
</span><span>cumsum</span><span> </span><span>=</span><span> </span><span>cumsum</span><span>(</span><span>n</span><span>)</span><span>
</span><span>)</span><span>
</span><span>tail</span><span>(</span><span>data</span><span>)</span><span>
</span></code></pre>
</div>
<table>
<thead>
<tr>
<th>date</th>
<th>n</th>
<th>cumsum</th>
</tr>
</thead>
<tbody>
<tr>
<td>2017-01-02</td>
<td>14</td>
<td>9710</td>
</tr>
<tr>
<td>2017-01-03</td>
<td>29</td>
<td>9739</td>
</tr>
<tr>
<td>2017-01-04</td>
<td>28</td>
<td>9767</td>
</tr>
<tr>
<td>2017-01-05</td>
<td>37</td>
<td>9804</td>
</tr>
<tr>
<td>2017-01-06</td>
<td>46</td>
<td>9850</td>
</tr>
<tr>
<td>2017-01-07</td>
<td>8</td>
<td>9858</td>
</tr>
</tbody>
</table>
<div>
<pre><code><span>hchart</span><span>(</span><span>data</span><span>,</span><span> </span><span>&quot;line&quot;</span><span>,</span><span> </span><span>hcaes</span><span>(</span><span>date</span><span>,</span><span> </span><span>cumsum</span><span>))</span><span> </span><span>%>%</span><span> </span><span>hc_title</span><span>(</span><span>text</span><span> </span><span>=</span><span> </span><span>&quot;Just in CRAN, what if we sum GH, BioC? How many would be?&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><iframe src="http://jkunst.com/htmlwidgets/my-KISS-attempt-to-rstatsgoes10k-contest/highchart_yzadrnp.html"></iframe></p>
<p><a target="_blank" rel="nofollow" href="http://jkunst.com/htmlwidgets/my-KISS-attempt-to-rstatsgoes10k-contest/highchart_yzadrnp.html">open</a></p>
<p>A little weird the effect in the 2014. Let’s drop some past<br>
information and create some auxiliar variables.</p>
<div>
<pre><code><span>data</span><span> </span><span>&lt;-</span><span> </span><span>filter</span><span>(</span><span>data</span><span>,</span><span> </span><span>year</span><span>(</span><span>date</span><span>)</span><span> </span><span>>=</span><span> </span><span>2013</span><span>)</span><span>
</span><span>date_first</span><span> </span><span>&lt;-</span><span> </span><span>first</span><span>(</span><span>data</span><span>$</span><span>date</span><span>)</span><span>
</span><span>date_last</span><span> </span><span>&lt;-</span><span> </span><span>last</span><span>(</span><span>data</span><span>$</span><span>date</span><span>)</span><span>
</span></code></pre>
</div>
<p>To use the package we need first a time series object:</p>
<div>
<pre><code><span>z</span><span> </span><span>&lt;-</span><span> </span><span>zooreg</span><span>(</span><span>data</span><span>$</span><span>cumsum</span><span>,</span><span> </span><span>start</span><span> </span><span>=</span><span> </span><span>date_first</span><span>,</span><span> </span><span>frequency</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span><span>tail</span><span>(</span><span>z</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## 2017-01-02 2017-01-03 2017-01-04 2017-01-05 2017-01-06 2017-01-07 
##       9710       9739       9767       9804       9850       9858
</code></pre>
</div>
<p>Now we can use the <code>forecastHybrid::hybridModel</code> function. In this case<br>
I removed the <code>tbats</code> model due the long time to fit, the long time to<br>
make CV and the long long time to make the predictions (in my previous tests).<br>
So, in the spirit to be parsimonious and KISS we will remove this model<br>
from the fit.</p>
<div>
<pre><code><span># hm &lt;- hybridModel(z, models = &quot;aefns&quot;, weights = &quot;cv.errors&quot;, errorMethod = &quot;MASE&quot;)
# saveRDS(hm, &quot;data/rstatsgoes10k/hm.rds&quot;)
</span><span>hm</span><span> </span><span>&lt;-</span><span> </span><span>readRDS</span><span>(</span><span>&quot;data/rstatsgoes10k/hm.rds&quot;</span><span>)</span><span>
</span><span>hm</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Hybrid forecast model comprised of the following models: auto.arima, ets, thetam, nnetar
## ############
## auto.arima with weight 0.368 
## ############
## ets with weight 0.37 
## ############
## thetam with weight 0.2 
## ############
## nnetar with weight 0.061
</code></pre>
</div>
<p>It is really simple to get the forecasts. After the calculate them we will<br>
create a <code>data_frame</code> to filter and see what day R will have 10k<br>
packages according this methodology.</p>
<div>
<pre><code><span>H</span><span> </span><span>&lt;-</span><span> </span><span>20</span><span>
</span><span>fc</span><span> </span><span>&lt;-</span><span> </span><span>forecast</span><span>(</span><span>hm</span><span>,</span><span> </span><span>h</span><span> </span><span>=</span><span> </span><span>H</span><span>)</span><span>
</span><span>data_fc</span><span> </span><span>&lt;-</span><span> </span><span>fc</span><span> </span><span>%>%</span><span> </span><span>as_data_frame</span><span>()</span><span> </span><span>%>%</span><span> </span><span>mutate</span><span>(</span><span>date</span><span> </span><span>=</span><span> </span><span>date_last</span><span> </span><span>+</span><span> </span><span>days</span><span>(</span><span>1</span><span>:</span><span>H</span><span>))</span><span> </span><span>%>%</span><span> </span><span>clean_names</span><span>()</span><span> </span><span>%>%</span><span> </span><span>tbl_df</span><span>()</span><span>
</span></code></pre>
</div>
<p>So let’t see the point forecast and the <em>optimistic</em> prediction<br>
which is the upper limit from the 95% interval.</p>
<div>
<pre><code><span>data_preds</span><span> </span><span>&lt;-</span><span> </span><span>bind_rows</span><span>(</span><span>
</span><span>data_fc</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>point_forecast</span><span> </span><span>>=</span><span> </span><span>10000</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>name</span><span> </span><span>=</span><span> </span><span>&quot;Prediction&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>head</span><span>(</span><span>1</span><span>)</span><span> </span><span>%>%</span><span> </span><span>rename</span><span>(</span><span>y</span><span> </span><span>=</span><span> </span><span>point_forecast</span><span>),</span><span>
</span><span>data_fc</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>hi_95</span><span> </span><span>>=</span><span> </span><span>10000</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>name</span><span> </span><span>=</span><span> </span><span>&quot;Optimitstic prediction&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>head</span><span>(</span><span>1</span><span>)</span><span> </span><span>%>%</span><span> </span><span>rename</span><span>(</span><span>y</span><span> </span><span>=</span><span> </span><span>hi_95</span><span>)</span><span>
</span><span>)</span><span>
</span><span>select</span><span>(</span><span>data_preds</span><span>,</span><span> </span><span>name</span><span>,</span><span> </span><span>date</span><span>,</span><span> </span><span>y</span><span>)</span><span>
</span></code></pre>
</div>
<table>
<thead>
<tr>
<th>name</th>
<th>date</th>
<th>y</th>
</tr>
</thead>
<tbody>
<tr>
<td>Prediction</td>
<td>2017-01-16</td>
<td>10008</td>
</tr>
<tr>
<td>Optimitstic prediction</td>
<td>2017-01-11</td>
<td>10008</td>
</tr>
</tbody>
</table>
<blockquote>
<p>So soon!! (warning: according to this).</p>
</blockquote>
<p>Now, let’s visualize the result.</p>
<div>
<pre><code><span>highchart</span><span>()</span><span> </span><span>%>%</span><span> </span><span>hc_title</span><span>(</span><span>text</span><span> </span><span>=</span><span> </span><span>&quot;rstatsgoes10k&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>hc_subtitle</span><span>(</span><span>text</span><span> </span><span>=</span><span> </span><span>&quot;Predictions via &lt;code>forecastHybrid&lt;/code> package&quot;</span><span>,</span><span> </span><span>useHTML</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>%>%</span><span> </span><span>hc_xAxis</span><span>(</span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;datetime&quot;</span><span>,</span><span>
</span><span>crosshair</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>zIndex</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>dashStyle</span><span> </span><span>=</span><span> </span><span>&quot;dot&quot;</span><span>,</span><span>
</span><span>snap</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;gray&quot;</span><span>
</span><span>))</span><span> </span><span>%>%</span><span> </span><span>hc_add_series</span><span>(</span><span>filter</span><span>(</span><span>data</span><span>,</span><span> </span><span>date</span><span> </span><span>>=</span><span> </span><span>ymd</span><span>(</span><span>20161001</span><span>)),</span><span> </span><span>&quot;line&quot;</span><span>,</span><span> </span><span>hcaes</span><span>(</span><span>date</span><span>,</span><span> </span><span>cumsum</span><span>),</span><span>
</span><span>name</span><span> </span><span>=</span><span> </span><span>&quot;Packages&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>hc_add_series</span><span>(</span><span>data_fc</span><span>,</span><span> </span><span>&quot;line&quot;</span><span>,</span><span> </span><span>hcaes</span><span>(</span><span>date</span><span>,</span><span> </span><span>point_forecast</span><span>),</span><span>
</span><span>name</span><span> </span><span>=</span><span> </span><span>&quot;Prediction&quot;</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;#75aadb&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>hc_add_series</span><span>(</span><span>data_fc</span><span>,</span><span> </span><span>&quot;arearange&quot;</span><span>,</span><span> </span><span>hcaes</span><span>(</span><span>date</span><span>,</span><span> </span><span>low</span><span> </span><span>=</span><span> </span><span>lo_95</span><span>,</span><span> </span><span>high</span><span> </span><span>=</span><span> </span><span>hi_95</span><span>),</span><span>
</span><span>name</span><span> </span><span>=</span><span> </span><span>&quot;Prediction Interval (95%)&quot;</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;#75aadb&quot;</span><span>,</span><span> </span><span>fillOpacity</span><span> </span><span>=</span><span> </span><span>0.3</span><span>)</span><span> </span><span>%>%</span><span> </span><span>hc_add_series</span><span>(</span><span>data_preds</span><span>,</span><span> </span><span>&quot;scatter&quot;</span><span>,</span><span> </span><span>hcaes</span><span>(</span><span>date</span><span>,</span><span> </span><span>y</span><span>,</span><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>name</span><span>),</span><span>
</span><span>name</span><span> </span><span>=</span><span> </span><span>&quot;Predicctions to 10K&quot;</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;blue&quot;</span><span>,</span><span>
</span><span>tooltip</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>pointFormat</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>),</span><span> </span><span>zIndex</span><span> </span><span>=</span><span> </span><span>-4</span><span>,</span><span>
</span><span>marker</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>symbol</span><span> </span><span>=</span><span> </span><span>&quot;circle&quot;</span><span>,</span><span> </span><span>lineWidth</span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>radius</span><span>=</span><span> </span><span>4</span><span>,</span><span>
</span><span>fillColor</span><span>=</span><span> </span><span>&quot;transparent&quot;</span><span>,</span><span> </span><span>lineColor</span><span>=</span><span> </span><span>NULL</span><span>),</span><span>
</span><span>dataLabels</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>enabled</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>format</span><span> </span><span>=</span><span> </span><span>&quot;{point.name}&lt;br>{point.x:%Y-%m-%d}&quot;</span><span>,</span><span>
</span><span>style</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>fontWeight</span><span> </span><span>=</span><span> </span><span>&quot;normal&quot;</span><span>)))</span><span> </span><span>%>%</span><span> </span><span>hc_tooltip</span><span>(</span><span>shared</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>valueDecimals</span><span> </span><span>=</span><span> </span><span>0</span><span>)</span><span>
</span></code></pre>
</div>
<p><iframe src="http://jkunst.com/htmlwidgets/my-KISS-attempt-to-rstatsgoes10k-contest/highchart_osvdnka.html"></iframe></p>
<p><a target="_blank" rel="nofollow" href="http://jkunst.com/htmlwidgets/my-KISS-attempt-to-rstatsgoes10k-contest/highchart_osvdnka.html">open</a></p>
<p>Simple, right? Maybe the that will not be the day where R hit 10k packages but its<br>
doesn’t matter. The <strong>really important fact here</strong> is all this is product of many many<br>
developers joined by the community, and some Rheroes like HW, JO, JB, GC, JC, BR, MA,<br>
DR, JS, KR and many others who have astonished us package by package or show our work<br>
in the social media . Thanks to everybody I can using this versatile and powerful language<br>
happily day by day.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://jkunst.com/r/my-KISS-attempt-to-rstatsgoes10k-contest/"> Jkunst - R category</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/dPDzsdDAP_o"></a></div>