<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/taking-a-shot-at-cdcfluview-v0-7-0-a-k-a-the-dangers-of-relying-on-hidden-apis/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/11/06/taking-a-shot-at-cdcfluview-v0-7-0-a-k-a-the-dangers-of-relying-on-hidden-apis/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Unlike <a rel="nofollow" href="http://twitter.com/noamross" target="_blank">@noamross</a>, I am not an epidemiologist (NOTE: Noam battles pandemics before breakfast, so be super nice to him) but I do like to find kindred methodologies in other disciplines to help foster the growth of cybersecurity into something beyond it’s current Barnum &amp; Bailey state. I also love finding and exposing hidden APIs and especially enjoy killing Adobe Flash. How does all that relate to <code>cdcfluview</code>?</p>
<p><a rel="nofollow" href="https://github.com/hrbrmstr/cdcfluview" target="_blank"><code>cdcfluview</code><img alt="🔗" src="https://s.w.org/images/core/emoji/2.3/72x72/1f517.png" class="wp-smiley"></a> was one of my first R packages. Someone, somwewhere, was trying to do something with Selenium to automate downloading of data from the CDC’s <a rel="nofollow" href="https://www.cdc.gov/flu/weekly/fluviewinteractive.htm" target="_blank">FluView Portal</a>. It was — and, some of it still is — a Flash-based site that locked up useful data behind application screens that did little more than burn ones retinas and force folks to keep Flash alive and, hence, their browsers insecure.</p>
<p>Rather than let the requester suffer under the weight of a pretty significant external dependency, I used the magic of the browser “developer tools” inspector to see that it was making fairly innocuous and useful XHR requests for real data. The package sat on GitHub for a while and eventually made its way to CRAN.</p>
<p>Times change and Flash is dying, so the CDC paid some serious benjamins to have the site re-done in HTML, replicating the horrible UX and terrible visualizations (so. many. pie. charts.). Said revamp also caused changes to the back-end APIs and forced breaking changes. Craig McGowan jumped to the rescue and fixed some core functionality issues, but so much changed — <em>and so much was added</em> — that I felt it was time for a modern re-write of the <code>cdcfluview</code> package.</p>
<p>This is a pretty solid, real-world example of how dangerous it is to rely on hidden APIs. If Craig hadn’t both notified me and gone the extra mile to make a PR, I’d’ve been in the dark until I tried to commiserate (I always seem get the flu no matter what I do) with code and found my package erroring out.</p>
<p>Enter: <code>cdcfluview</code> 0.7.0.</p>
<h3>What’s Different?</h3>
<p>Unfortunately, everything; which is one reason I’m writing this post. </p>
<p>First, to have folks that are using current-gen <code>cdcfluview</code> kick the tyres and let me know (via <a rel="nofollow" href="https://github.com/hrbrmstr/cdcfluview/issues" target="_blank">issues</a>) if you need any old API compatibility back. This isn’t anywhere near the most popular package on CRAN, but it does have users (even, I’m told, <em>within</em> the CDC) and I want to make sure I do as little to disrupt them as possible. But, the current package API maps much more closely to the way the revamped portal works and presents data, so I’m hoping it’s a good net-new vs crushing blow to productivity.</p>
<p>Speaking of maps, the package now has actual maps! A new <code>cdc_basemap()</code> function returns the GeoJSON files that the CDC uses in their web views as <code>sf</code> objects. And, there are tons of maps and multi-labeled features to tie data to:</p>
<p><a rel="nofollow" href="https://rud.is/b/2017/11/06/taking-a-shot-at-cdcfluview-v0-7-0-a-k-a-the-dangers-of-relying-on-hidden-apis/unnamed-chunk-5-1/" target="_blank"><img width="150" alt="" src="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/11/unnamed-chunk-5-1.png?fit=150%2C107&ssl=1" height="107"></a><br>
<a rel="nofollow" href="https://rud.is/b/2017/11/06/taking-a-shot-at-cdcfluview-v0-7-0-a-k-a-the-dangers-of-relying-on-hidden-apis/unnamed-chunk-5-2/" target="_blank"><img width="150" alt="" src="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/11/unnamed-chunk-5-2.png?fit=150%2C107&ssl=1" height="107"></a><br>
<a rel="nofollow" href="https://rud.is/b/2017/11/06/taking-a-shot-at-cdcfluview-v0-7-0-a-k-a-the-dangers-of-relying-on-hidden-apis/unnamed-chunk-5-3/" target="_blank"><img width="150" alt="" src="https://i1.wp.com/rud.is/b/wp-content/uploads/2017/11/unnamed-chunk-5-3.png?fit=150%2C107&ssl=1" height="107"></a><br>
<a rel="nofollow" href="https://rud.is/b/2017/11/06/taking-a-shot-at-cdcfluview-v0-7-0-a-k-a-the-dangers-of-relying-on-hidden-apis/unnamed-chunk-5-4/" target="_blank"><img width="150" alt="" src="https://i1.wp.com/rud.is/b/wp-content/uploads/2017/11/unnamed-chunk-5-4.png?fit=150%2C107&ssl=1" height="107"></a><br>
<a rel="nofollow" href="https://rud.is/b/2017/11/06/taking-a-shot-at-cdcfluview-v0-7-0-a-k-a-the-dangers-of-relying-on-hidden-apis/unnamed-chunk-5-5/" target="_blank"><img width="150" alt="" src="https://i1.wp.com/rud.is/b/wp-content/uploads/2017/11/unnamed-chunk-5-5.png?fit=150%2C107&ssl=1" height="107"></a><br>
<a rel="nofollow" href="https://rud.is/b/2017/11/06/taking-a-shot-at-cdcfluview-v0-7-0-a-k-a-the-dangers-of-relying-on-hidden-apis/unnamed-chunk-5-6/" target="_blank"><img width="150" alt="" src="https://i1.wp.com/rud.is/b/wp-content/uploads/2017/11/unnamed-chunk-5-6.png?fit=150%2C107&ssl=1" height="107"></a></p>
<p>Here’s what’s in the tin:</p>
<ul>
<li><code>age_group_distribution</code>: Age Group Distribution of Influenza Positive Tests Reported by Public Health Laboratories</li>
<li><code>cdc_basemap</code>: Retrieve CDC U.S. Basemaps</li>
<li><code>geographic_spread</code>: State and Territorial Epidemiologists Reports of Geographic Spread of Influenza</li>
<li><code>hospitalizations</code>: Laboratory-Confirmed Influenza Hospitalizations</li>
<li><code>ilinet</code>: Retrieve ILINet Surveillance Data</li>
<li><code>ili_weekly_activity_indicators</code>: Retrieve weekly state-level ILI indicators per-state for a given season</li>
<li><code>pi_mortality</code>: Pneumonia and Influenza Mortality Surveillance</li>
<li><code>state_data_providers</code>: Retrieve metadata about U.S. State CDC Provider Data</li>
<li><code>surveillance_areas</code>: Retrieve a list of valid sub-regions for each surveillance area.</li>
<li><code>who_nrevss</code>: Retrieve WHO/NREVSS Surveillance Data</li>
<li><code>mmwr_week</code>: Convert a Date to an MMWR day+week+year</li>
<li><code>mmwr_weekday</code>: Convert a Date to an MMWR weekday</li>
<li><code>mmwr_week_to_date</code>: Convert an MMWR year+week or year+week+day to a Date object</li>
</ul>
<p>Plus there’s a new data object <code>mmwrid_map</code> that makes it super-easy to convert arcane MMWR identifiers to real date object.</p>
<p>The <a rel="nofollow" href="https://github.com/hrbrmstr/cdcfluview/blob/master/README.md" target="_blank">README</a> has plenty of charts and examples, so I won’t take up post-space with said code or images.</p>
<h3>Curiously Enough</h3>
<p>Along the way, I was able to discern that there’s a hidden layer of this new, hidden API. Exposing said layer should be as easy as figuring out the right keyword and I’m hoping a bit of fuzzing will do the trick on that. It will be interesting to see what extra data that unlocks. (<em>Yes</em>, I <em>just</em> said relying on hidden APIs is dangerous; and, relying on <em>hidden</em>, <em>hidden</em> APIs is doubly so. I’m just a glutton for punishment.)</p>
<p>I was also able to discern that multiple people or teams worked on this revamp and said folks did not communicate with each other. The per-app APIs are <em>woefully</em> inconsistent. Furthermore, someone goofed and forgot to expose some pretty critical information from a few data retrieval operations (said data is also missing on the clickable download versions, too). Hopefully they’ll be addressing the issue soon (the site <em>is</em> technically in beta release).</p>
<h3>FIN</h3>
<p>If you’ve been a user of <code>cdcfluview</code> <em>please</em> give the new API a try and file issues with anything you see. All contributors — testers, modders, enhancers — will get full <code>DESCRIPTION</code> credit (so, please also include how you’d like to be cited).</p>
<p>Finally, please do check out the <a rel="nofollow" href="https://www.cdc.gov/flu/weekly/fluviewinteractive.htm" target="_blank">CDC FluView Portal</a>. It’s gosh awful horribad. I know there are some spiffy Shiny experts out there who could run rings around that portal and I’ll be glad to add you as a collaborator if you contribute a Shiny app (or two!) to the package. If you’d rather go your own route with a self-contained, self-published package, just let me know what API changes you’d like and I’ll gladly accommodate. The goal is to help epidemiologists and other researchers keep us all safe.</p>
<p>So, <em>go get your flu shot</em>!!! <strong>Then</strong>, kick the tyres on this package update and don’t hesitate to convey your criticisms, patches or accolades.</p>
<p>Now to get to my promised final review of <a rel="nofollow" href="https://github.com/richfitz/cyphr" target="_blank"><code>cyphr</code></a> (I’ve not forgotten <a rel="nofollow" href="http://twitter.com/ma_salmon" target="_blank">@ma_salmon</a> <img alt="😉" src="https://s.w.org/images/core/emoji/2.3/72x72/1f609.png" class="wp-smiley"></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2017/11/06/taking-a-shot-at-cdcfluview-v0-7-0-a-k-a-the-dangers-of-relying-on-hidden-apis/"> R – rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/MftFXkBb9YQ" height="1"></a></div>