<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/the-most-important-commodity-in-2017-is-data/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/01/04/the-most-important-commodity-in-2017-is-data/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Despite being in cybersecurity nigh forever (a career that quickly turns one into a determined skeptic if you’re doing your job correctly) I have often trusted various (not to be named) news sources, reports and data sources to provide honest and as-unbiased-as-possible information. The debacle in the U.S. in late 2016 has proven (to me) that we’re all on our own when it comes to validating posited truth/fact. I’m teaching two data science college courses this Spring and one motivation for teaching is helping others on the path to data literacy so they don’t have to just believe what’s tossed at them.</p>
<p>It’s also one of the reasons I write R packages and blog/speak (when I can). </p>
<p>Today, I saw a chart in a WSJ article on the <a target="_blank" rel="nofollow" href="http://www.wsj.com/articles/minimum-wages-set-to-increase-in-many-states-in-2017-1483093806">impending minimum wage changes</a> <span>(paywalled, so do the Google hack to read it)</span> set to take effect in 19 states. </p>
<p><a target="_blank" rel="nofollow" href="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/01/BN-RL751_NACM98_9U_20170103102059.jpg?ssl=1"><img alt="" class="wp-image-4827" src="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/01/BN-RL751_NACM98_9U_20170103102059.jpg?resize=456%2C445&ssl=1" data-recalc-dims="1"></a></p>
<p>It’s a great chart and they cite the data source as coming from the <a target="_blank" rel="nofollow" href="http://www.epi.org/">Economic Policy Institute</a>. I found <a target="_blank" rel="nofollow" href="http://www.epi.org/minimum-wage-tracker/">some minimum wage data</a> there and manually went through a bit of it to get enough of a feel that the WSJ wasn’t “mis-truthing” or truth-spinning. (As an aside, the minimum wage should definitely be higher, indexed and adjusted for inflation, but that’s a discussion for another time.)</p>
<p>While on EPI’s site, I <em>did</em> notice they had a “State of Working America Data Library” @ <a target="_blank" rel="nofollow" href="http://www.epi.org/data/">http://www.epi.org/data/</a>. The data there is based on U.S. government published data and I validated that EPI isn’t fabricating anything (but you should not just take my word for it and do your own math from U.S. gov sources). I also noticed that the filtering interactions were delayed a bit and posited said condition was due to the site making AJAX/XHR calls to retrieve data. So, I peeked under the covers and discovered a robust, consistent, hidden API that is now wrapped in <a target="_blank" rel="nofollow" href="https://github.com/hrbrmstr/epidata">an R package</a> dubbed <code>epidata</code>. </p>
<p>You can get the details of what’s available via the EPI site or through package docs.</p>
<h3>What can you do with this data?</h3>
<p>They seem to update the data (pretty much) monthly and it’s based on U.S. gov publications, so you can very easily validate “news” reports, potentially even easier than via packages that wrap the Bureau of Labor Statistics (BLS) API. On a lark, I wanted to compare the unemployment rate vs median wage over time (mostly to test the API and make a connected scatterplot). If you didn’t add the level of detail to the aesthetics I did, the following code is pretty small to do just that:</p>
<pre><code>library(tidyverse)
library(epidata)
library(ggrepel)

unemployment &lt;- get_unemployment()
wages &lt;- get_median_and_mean_wages()

glimpse(wages)
## Observations: 43
## Variables: 3
## $ date    &lt;int> 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, ...
## $ median  &lt;dbl> 16.53, 16.17, 16.05, 16.15, 16.07, 16.36, 16.15, 16.07...
## $ average &lt;dbl> 19.05, 18.67, 18.64, 18.87, 18.77, 18.83, 19.06, 18.66...

glimpse(unemployment)
## Observations: 456
## Variables: 2
## $ date &lt;date> 1978-12-01, 1979-01-01, 1979-02-01, 1979-03-01, 1979-04-...
## $ all  &lt;dbl> 0.061, 0.061, 0.060, 0.060, 0.059, 0.059, 0.059, 0.058, 0...

group_by(unemployment, date=as.integer(lubridate::year(date))) %>%
  summarise(rate=mean(all)) %>%
  left_join(select(wages, date, median), by=&quot;date&quot;) %>%
  filter(!is.na(median)) %>%
  arrange(date) -> df

cols &lt;- ggthemes::tableau_color_pal()(3)

ggplot(df, aes(rate, median)) +
  geom_path(color=cols[1], arrow=arrow(type=&quot;closed&quot;, length=unit(10, &quot;points&quot;))) +
  geom_point() +
  geom_label_repel(aes(label=date),
                   alpha=c(1, rep((4/5), (nrow(df)-2)), 1),
                   size=c(5, rep(3, (nrow(df)-2)), 5),
                   color=c(cols[2],
                           rep(&quot;#2b2b2b&quot;, (nrow(df)-2)),
                           cols[3]),
                   family=&quot;Hind Medium&quot;) +
  scale_x_continuous(name=&quot;Unemployment Rate&quot;, expand=c(0,0.001), label=scales::percent) +
  scale_y_continuous(name=&quot;Median Wage&quot;, expand=c(0,0.25), label=scales::dollar) +
  labs(title=&quot;U.S. Unemployment Rate vs Median Wage Since 1978&quot;,
       subtitle=&quot;Wage data is in 2015 USD&quot;,
       caption=&quot;Source: EPI analysis of Current Population Survey Outgoing Rotation Group microdata&quot;) +
  hrbrmisc::theme_hrbrmstr(grid=&quot;XY&quot;)</code></pre>
<p><a target="_blank" rel="nofollow" href="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/01/Cursor_and_RStudio.png?ssl=1"><img alt="" class="wp-image-4828" src="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/01/Cursor_and_RStudio.png?resize=456%2C378&ssl=1" data-recalc-dims="1"></a></p>
<p>You can build your own “State of Working America Data Library” dashboard with this data (with <code>flexdashboard</code> and/or <code>shiny</code>) and be a bit of a citizen journalist, keeping the actual media in check and staying more informed an engaged.</p>
<p>As usual, drop issues &amp; feature requests in the github repo. If you make some fun things with the data, drop a comment in the blog. Unless something major comes up the package should be on CRAN by the weekend.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2017/01/04/the-most-important-commodity-in-2017-is-data/"> R – rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/7UFS0lXwiMY"></a></div>