<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/how-the-state-of-indiana-uses-r-and-azure-to-forecast-employment/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2016/12/state-of-indiana-employment.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>&quot;Big Data&quot; generates a lot of news these days, but sometimes small data still means <strong>big</strong> computation.</p>
<p><a target="_blank" rel="nofollow" href="https://www.in.gov/dwd/">Indiana's Department of Workforce Development</a> has the responsibility to forecast future employment rates in the State of Indiana. And not just the number of jobs available: the department also needs to forecast the types of jobs that will be available, so the administration can link up future job requirements with training and education policies and make sure a skilled workforce can fill the future demand. </p>
<p>The State of Indiana contracted with analytics professional services firm <a target="_blank" rel="nofollow" href="http://www.inquidia.com/">Inquidia Consulting</a> to <a target="_blank" rel="nofollow" href="https://info.microsoft.com/Indiana-Department-of-Workforce-Development-White-Paper.html">develop a system to generate the forecast</a>, as part of the &quot;Demand-Driven Workforce System&quot; project. After assembling a database of employment data, the team quickly found that this wasn't exactly a big-data problem: each forecast was based on just 1kb to 3kb of data, and could easily be computed using the <a target="_blank" rel="nofollow" href="https://mran.microsoft.com/documents/what-is-r/">R language</a> in less than 2 seconds. The problem was that one the combinations of counties and selected variables were taken into account, <em>millions</em> of different forecasts needed to be made, immediately making this a problem that demanded significant of computing time.</p>
<p>Fortunately, this is exactly the kind of problem that can be tackled using parallel programming: since each of the models can be computed independently, a large number of machines can compute models simultaneously and dramatically reduce the required computation time. Using Microsoft Azure, the team could spin up hundreds of virtual machines to perform the R processing, orchestrated by a <a target="_blank" rel="nofollow" href="http://wiki.pentaho.com/display/EAI/Carte+User+Documentation#CarteUserDocumentation-WhatisCarte%3F">PDI/carte</a> process to implement a simple (yet reliable) fault-tolerant distributed system. (The team considered using HDInsight and Spark for the distribution framework, but decided given the small data sizes that it would be overkill for their needs.)</p>
<p><a target="_blank" rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c8bd4821970b-pi"><img title="Indiana architecture" alt="Indiana architecture" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c8bd4821970b-800wi" border="0"></a></p>
<p>For the actual forecasts, the Inquidia team used R's <a target="_blank" rel="nofollow" href="https://mran.microsoft.com/package/forecast/">forecast package</a>, noting that it offers &quot;a robust set of options for projecting time series&quot; and &quot;allows for easy projections from any set of models for an arbitrarily large forecast period&quot;.  However, vector autoregressive models can only accomodate a relatively small number of variables and there are a large number of macroeconomic variables to choose from to forecast employment. This created a difficult variable-selection problem, which the team solved with a brute-force model forecasting and evaluation process.</p>
<p>Finally, to present the forecasts to State of Indiana administrators, the team created a <a target="_blank" rel="nofollow" href="https://rstudio.github.io/shinydashboard/">shinydashboard</a> library. This dashboard gave administrators flexbility to customize the forecasts by adjusting for seasonality, combining multiple models, and including new employment data. </p>
<p><a target="_blank" rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c8bd4911970b-pi"><img title="Calcengine" alt="Calcengine" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c8bd4911970b-800wi" border="0"></a></p>
<p>As a result, the Indiana DWD now receives a much more finely grained and accurate analysis of job skills and needs in Indiana, and can now view details at the county and metro level, without waiting for months. This is thanks to the system build with R on Microsoft Azure: a computation engine capable of producing more than 1,000 models per second at a fraction of the cost of an on-premises configuration.</p>
<p>To learn more about the forecasting system Inquidia Consulting build for the Indiana DWD, you can watch this <a target="_blank" rel="nofollow" href="https://info.microsoft.com/big-data-deep-dive-ondemand.html">archived webinar presented by the project leads</a>. Or check out the white paper linked below, which gives details about the underlying architectute and implementation.</p>
<p>Microsoft Advanced Analytics and IoT: <a target="_blank" rel="nofollow" href="https://info.microsoft.com/Indiana-Department-of-Workforce-Development-White-Paper.html">White Paper: Dive deep into small data with big data techniques</a> </p>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2016/12/state-of-indiana-employment.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/_H69zyeWtls"></a></div>