<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/using-shiny-with-scheduled-and-streaming-data/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rviews.rstudio.com/2017/11/15/shiny-and-scheduled-data-r/"> R Views</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Shiny applications are often backed by fluid, changing data. Data updates can occur at different time scales: from scheduled daily updates to live streaming data and ad-hoc user inputs. This article describes best practices for handling data updates in Shiny, and discusses deployment strategies for automating data updates.</p>
<p><img alt="" src="https://i1.wp.com/rviews.rstudio.com/post/2017-11-15-shiny-and-scheduled-data/rviews_scheduled_shiny.002.jpeg?w=456&ssl=1" data-recalc-dims="1"></p>
<p>This post builds off of a 2017 rstudio::conf talk. The recording of the <a rel="nofollow" href="https://www.rstudio.com/resources/videos/dashboards-made-easy/" target="_blank">original talk</a> and the <a rel="nofollow" href="https://github.com/slopp/scheduledsnow" target="_blank">sample code</a> for this post are available.</p>
<p>The end goal of this example is a dashboard to help skiers in Colorado select a resort to visit. Recommendations are based on:</p>
<ol>
<li>Snow reports that provide useful metrics like number of runs open and amount of new snow. Snow reports are updated <strong>daily</strong>.</li>
<li>Weather data, updated in <strong>near real-time from a live stream</strong>.</li>
<li>User preferences, entered in the dashboard.</li>
</ol>
<p>Visit the <a rel="nofollow" href="http://colorado.rstudio.com/rsc/content/168/" target="_blank">live app</a>!</p>
<p>The backend for the dashboard looks like:</p>
<p><img alt="" src="https://i1.wp.com/rviews.rstudio.com/post/2017-11-15-shiny-and-scheduled-data/rviews_scheduled_shiny.003.jpeg?w=456&ssl=1" data-recalc-dims="1"></p>
<h2>Automate Scheduled Data Updates</h2>
<p>The first challenge is preparing the daily data. In this case, the data preparation requires a series of API requests and then basic data cleansing. The code for this process is written <strong>into an R Markdown document</strong>, alongside process documentation and a few simple graphs that help validate the new data. The R Markdown document ends by saving the cleansed data into a shared data directory. The entire R Markdown document is scheduled for execution.</p>
<p>It may seem odd at first to use a R Markdown document as the scheduled task. However, our team has found it incredibly useful to be able to look back through historical renderings of the “report” to gut-check the process. Using R Markdown also forces us to properly document the scheduled process.</p>
<p>We use RStudio Connect to easily schedule the document, view past historical renderings, and ultimately to host the application. If the job fails, Connect also sends us an email containing <code>stdout</code> from the render, which helps us stay on top of errors. (Connect can optionally send the successfully rendered report, as well.) However, the same scheduling could be accomplished with a workflow tool or even CRON.</p>
<p>Make sure the data, written to shared storage, is readable by the user running the Shiny application – typically a service account like <code>rstudio-connect</code> or <code>shiny</code> can be set as the run-as user to ensure consistent behavior.</p>
<p>Alternatively, instead of writing results to the file system, prepped data can be saved to a view in a database.</p>
<h2>Using Scheduled Data in Shiny</h2>
<p>The dashboard needs to look for updates to the underlying shared data and automatically update when the data changes. (It wouldn’t be a very good dashboard if users had to refresh a page to see new data.) In Shiny, this behavior is accomplished with the <code>reactiveFileReader</code> function:</p>
<pre><code>daily_data &lt;- reactiveFileReader(
  intervalMillis = 100,
  filePath       = 'path/to/shared/data',
  readFunc       = readr::read_cs
)
</code></pre>
<p>The function checks the shared data file’s update timestamp every <code>intervalMillis</code> to see if the data has changed. If the data has changed, the file is re-read using <code>readFunc</code>. The resulting data object, <code>daily_data</code>, is reactive and can be used in downstream functions like <code>render***</code>.</p>
<p>If the cleansed data is stored in a database instead of written to a file in shared storage, use <code>reactivePoll</code>. <code>reactivePoll</code> is similar to <code>reactiveFileReader</code>, but instead of checking the file’s update timestamp, a second function needs to be supplied that identifies when the database is updated. The function’s <a rel="nofollow" href="https://shiny.rstudio.com/reference/shiny/latest/reactivePoll.html" target="_blank">help documentation</a> includes an example.</p>
<h2>Streaming Data</h2>
<p>The second challenge is updating the dashboard with live streaming weather data. One way for Shiny to ingest a stream of data is by turning the stream into “micro-batches”. The <code>invalidateLater</code> function can be used for this purpose:</p>
<pre><code>liveish_data &lt;- reactive({
  invalidateLater(100)
  httr::GET(...)
})
</code></pre>
<p>This causes Shiny to poll the streaming API every 100 milliseconds for new data. The results are available in the reactive data object <code>liveish_data</code>. Picking how often to poll for data depends on a few factors:</p>
<ol>
<li>Does the upstream API enforce rate limits?</li>
<li>How long does a data update take? The application will be blocked while it polls data.</li>
</ol>
<p>The goal is to pick a polling time that balances the user’s desire for “live” data with these two concerns.</p>
<h2>Conclusion</h2>
<p>To summarize, this architecture provides a number of benefits: No more painful, manual running of R code every day! Dashboard code is isolated from data prep code. There is enough flexibility to meet user requirements for live and daily data, while preventing un-necessary number crunching on the backend.</p>
<p> </p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rviews.rstudio.com/2017/11/15/shiny-and-scheduled-data-r/"> R Views</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/82fq1qh6oHI" height="1"></a></div>