<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/uk-government-using-r-to-modernize-reporting-of-official-statistics/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2017/03/uk-statistics.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>Like all governments, the UK government is responsible for producing reports of official statistics on an ongoing basis. That process has traditionally been a highly manual one: extract data from government systems, load it into a mainframe statistical analysis tool and run models and forecasts, extract the results to a spreadsheet to prepare data for presentation, and ultimately combine it all in a manual document editing tool to produce the final report. The process in the UK looks much like this today:</p>
<div><a target="_blank" rel="nofollow" href="http://a3.typepad.com/6a0105360ba1c6970c01b8d26fba2b970c-pi"><img title="Uk-old" alt="Uk-old" src="http://a3.typepad.com/6a0105360ba1c6970c01b8d26fba2b970c-800wi" border="0"></a></div>
<p><a target="_blank" rel="nofollow" href="https://twitter.com/m_a_upson">Matt Upson</a>, a Data Scientist at the UK Government Digital Service, is looking to modernize this process with a <a target="_blank" rel="nofollow" href="https://gdsdata.blog.gov.uk/2017/03/27/reproducible-analytical-pipeline/">reproducible analytical pipeline</a>. This new process, based on the UK Government's <a target="_blank" rel="nofollow" href="https://www.gov.uk/service-manual/technolog">Technology Service Manual</a> for new IT deployments, aims to simplify the process by using R — the open-source <a target="_blank" rel="nofollow" href="https://www.r-project.org/">programming language for statistical analysis</a> — to automate the data extraction, analysis, and document generation tasks.</p>
<p><a target="_blank" rel="nofollow" href="http://a2.typepad.com/6a0105360ba1c6970c01b7c8e54642970b-pi"><img title="Uk-new" alt="Uk-new" src="http://a2.typepad.com/6a0105360ba1c6970c01b7c8e54642970b-800wi" border="0"></a></p>
<p>The development of the new process is underway now, with the first target being a report on <a target="_blank" rel="nofollow" href="https://www.gov.uk/government/statistics/dcms-sectors-economic-estimates-2016">culture and sporting impacts on the UK economy</a>:</p>
<blockquote>
<p>Towards the end of 2016 we embarked on a project with a team in the Department for Culture, Media, and Sport (DCMS) who are responsible for the production of the Economic Estimates for DCMS Sectors Statistical First Release (SFR). Currently this publication is produced with a mix of manual and semi-manual processes. Our aim was to see if we could speed up production of the SFR, whilst maintaining the high standard of the publication and QA.</p>
</blockquote>
<p>Central to the process is automating the report with <a target="_blank" rel="nofollow" href="http://rmarkdown.rstudio.com/">Rmarkdown</a>, a system for programmatically laying out a document while encapsulating all of the R code for data preparation, analysis, tabulation and charting into a collaborative document. As a comprehensive language, R has the functionality to handle all of these tasks natively, without the need to bring other systems into the chain. And it's an ideal tool for the statistical data analysis required for these reports, which must follow the guidance of the <a target="_blank" rel="nofollow" href="https://www.gov.uk/government/publications/the-aqua-book-guidance-on-producing-quality-analysis-for-government">Aqua Book</a>, the UK government's manual for producing quality analysis. (The Aqua Book is well worth reading for any data scientist, with excellent guidance on designing studies in the face of uncertain data.) </p>
<p>Naturally, the government has specific standards for the presentation of data, and the R-based process needs to be able to reproduce that look in the final report. The GDS team has produced a package for R, called <a target="_blank" rel="nofollow" href="https://github.com/ukgovdatascience/govstyle">govstyle</a>, which standardizes the look of data charts while upgrading them to more modern design principles. For example this chart from a report on truancy in UK schools:</p>
<p><a target="_blank" rel="nofollow" href="https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/468965/SFR39_2015_Text.pdf"><img title="Uk-chart-old" alt="Uk-chart-old" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c8e56a56970b-800wi" border="0"></a></p>
<p>has been upgraded and reproduced in R to look like this:</p>
<p><a target="_blank" rel="nofollow" href="https://github.com/ukgovdatascience/govstyle"><img title="Figure1d-1" alt="Figure1d-1" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb0988be31970d-800wi" border="0"></a></p>
<p>An automated workflow allows for the inclusion of modern devops processes, too. Dependency management for R packages is handled with <a target="_blank" rel="nofollow" href="https://rstudio.github.io/packrat/">packrat</a>. Test-driven development comes courtesy of the <a target="_blank" rel="nofollow" href="https://github.com/hadley/testthat">testthat</a> package, and automated testing (including data verification and code coverage analysis) is provided by <a target="_blank" rel="nofollow" href="https://travis-ci.org/">Travis CI</a>. And source code control and collaboration is provided by Github, which is also where the entire process is documented as R package: the <a target="_blank" rel="nofollow" href="https://github.com/ukgovdatascience/eesectors">eesectors package</a>.</p>
<p>For more on the UK government's use of R for official statistics reporting, see the post below from the UK Government Digital Service. </p>
<p>Data at GDS: <a target="_blank" rel="nofollow" href="https://gdsdata.blog.gov.uk/2017/03/27/reproducible-analytical-pipeline/">Reproducible Analytical Pipeline</a></p>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2017/03/uk-statistics.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/qquiSKEwsNE"></a></div>