<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/implementation-of-a-basic-reproducible-data-analysis-workflow/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.jom.link/implementation_basic_reproductible_workflow.html"> Joris Muller's blog - Posts about R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a target="_blank" rel="nofollow" href="http://blog.jom.link/reproducible_analysis_my_principles.html">In a previous post</a>, I described the principles of my basic reproducible data analysis workflow. Today, let’s be more practical and see how to implement it.</p>
<p>Be noted that it is a basic workflow. The goal is to find a good balance between a minimal reproducible analysis and the ease to deploy it on any platform.</p>
<p>This workflow will allow you to run a complete analysis based on multiple files (data files, R script, Rmd files…) just by launching a single R script.</p>
<h1>Summary</h1>
<p>This workflow allows to process raw files in order to produce reports (in html or PDF). There are 3 main components:</p>
<ol>
<li>Sofwares:
<ul>
<li><a target="_blank" rel="nofollow" href="https://www.r-project.org/">R</a> of course.</li>
<li><a target="_blank" rel="nofollow" href="https://www.rstudio.com/products/rstudio/">Rstudio IDE</a> optionnaly. It could save you installation time because it’s come with both <em>Pandoc</em> and <em>rmarkdown</em> built-in.</li>
<li>If not using <em>Rstudio IDE</em>:
<ul>
<li><a target="_blank" rel="nofollow" href="https://github.com/rstudio/rmarkdown">rmarkdown R’s package</a>, to convert R markdown files to markdown, html and PDF. Just type in R <code>install.packages(&quot;rmarkdown&quot;)</code>.</li>
<li>a recent version of <a target="_blank" rel="nofollow" href="https://github.com/rstudio/rmarkdown/blob/master/PANDOC.md">Pandoc</a> to process markdown files.</li>
</ul>
</li>
<li><a target="_blank" rel="nofollow" href="https://git-scm.com/">Git</a> for control version.</li>
</ul>
</li>
<li><a target="_blank" rel="nofollow" href="http://blog.jom.link/implementation_basic_reproductible_workflow.html#file_orga">Files organisation</a> (see above).</li>
<li><a target="_blank" rel="nofollow" href="http://blog.jom.link/implementation_basic_reproductible_workflow.html#run_all">One R script to rule them all</a> (see above).</li>
</ol>
<h1><a name="file_orga"></a>Files organisation</h1>
<p>Most important is the organisation of the project. I understand by <em>project</em> a folder containing every file necessary to run the analysis: raw data, intermediate data, scripts and other assets (pictures, xml…).</p>
<p>My organisation is:</p>
<div>
<pre><code>name_of_the_project
|-- assets
|-- functions
    |-- import_html_helper.R
    |-- make_report.R
|-- plot
|-- produced_data
    |-- imported.rds
    |-- model_result.rds
|-- run_all.R     # *** Most important file ***
|-- raw_data
    |-- FinalDataV3.xlsx
    |-- SomeExternalData.csv
    |-- NomenclatureFromWeb.html
|-- reports
    |-- 01-import_data.html
    |-- 02-data_tidying.html
    |-- 03-descriptive.html
    |-- 04-model1.html
    |-- 99-sysinfo.html
|-- rmds
    |-- import_data.Rmd
    |-- data_tidying.Rmd
    |-- descriptive.Rmd
    |-- model1.Rmd
    |-- sysinfo.Rmd
|-- rscripts
    |-- complicated_model.R
|-- name_of_the_project.Rproj # not mandatory but useful
</code></pre>
</div>
<p>As you can observe, there are some principles :</p>
<ul>
<li>Directory names are as explicit as possible to be understandable by anyone getting these files.</li>
<li>Reports don’t belong to the script, because I don’t produce a report for every script or Rmd (there is child Rmd) and this way, it’s straightforward where people should look at the results (reports).</li>
<li>Reports are sorted using 2-digit numbers. Data science is also an art to tell stories in the right order.</li>
<li>I use <em>rmarkdown</em> files to produce my report, to have both the results and my comments. These comments are fundamental, this is an appreciation of the results by the data scientist.</li>
<li>A rmarkdown file, <em>sysinfo.Rmd</em>, will be used to produce a report keeping trace of the name and version of R’s package used (with <code>sessionInfo()</code>) and some extra information about the OS (<code>Sys.info()</code>). In an ideal workflow, these commands have to be called at the end of each report.</li>
<li>Everything lives in a subfolder except <code>run_all.R</code> (detail above).</li>
</ul>
<h1><a name="run_all"></a> One R script to run them all</h1>
<p>This <code>run_all.R</code> script, as it name explicitly tell, it runs everything necessary to produce all the reports. Here an example:</p>
<div>
<pre><code><span>source</span><span>(</span><span>&quot;functions/make_reports.R&quot;</span><span>)</span><span>
</span><span>report</span><span>(</span><span>&quot;rmds/import_data.Rmd&quot;</span><span>,</span><span> </span><span>n_file</span><span> </span><span>=</span><span> </span><span>&quot;1&quot;</span><span>)</span><span>
</span><span>report</span><span>(</span><span>&quot;rmds/data_tidying.Rmd&quot;</span><span>,</span><span> </span><span>&quot;2&quot;</span><span>)</span><span>
</span><span>report</span><span>(</span><span>&quot;rmds/descriptive.Rmd&quot;</span><span>,</span><span> </span><span>&quot;3&quot;</span><span>)</span><span>
</span><span>report</span><span>(</span><span>&quot;rmds/model1.Rmd&quot;</span><span>,</span><span> </span><span>&quot;4&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p>It’s straightforward: one line by rmarkdown files to process.</p>
<p><code>make_report.R</code> is optional. It’s an help function to produce the report, set their name.</p>
<div>
<pre><code><span># Clean up the environment
</span><span>rm</span><span>(</span><span>list</span><span> </span><span>=</span><span> </span><span>ls</span><span>())</span><span>
</span><span># Load the libraries
</span><span>library</span><span>(</span><span>knitr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>rmarkdown</span><span>)</span><span>
</span><span># Set the root dir because my rmds live in rmds/ subfolder
</span><span>opts_knit</span><span>$</span><span>set</span><span>(</span><span>root.dir</span><span> </span><span>=</span><span> </span><span>'../.'</span><span>)</span><span>
</span><span># By default, don't open the report at the end of processing
</span><span>default_open_file</span><span> </span><span>&lt;-</span><span> </span><span>FALSE</span><span>
</span><span># Main function
</span><span>report</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>file</span><span>,</span><span> </span><span>n_file</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>open_file</span><span> </span><span>=</span><span> </span><span>default_open_file</span><span>,</span><span> </span><span>report_dir</span><span> </span><span>=</span><span> </span><span>&quot;reports&quot;</span><span>)</span><span> </span><span>{</span><span>
</span><span>### Set the name of the report file ###
</span><span> </span><span>base_name</span><span> </span><span>&lt;-</span><span> </span><span>sub</span><span>(</span><span>pattern</span><span> </span><span>=</span><span> </span><span>&quot;.Rmd&quot;</span><span>,</span><span> </span><span>replacement</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>x</span><span> </span><span>=</span><span> </span><span>basename</span><span>(</span><span>file</span><span>))</span><span>
</span><span># Make nfiles with always 2 digits
</span><span> </span><span>n_file</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>as.integer</span><span>(</span><span>n_file</span><span>)</span><span> </span><span>&lt;</span><span> </span><span>10</span><span>,</span><span> </span><span>paste0</span><span>(</span><span>&quot;0&quot;</span><span>,</span><span> </span><span>n_file</span><span>),</span><span> </span><span>n_file</span><span>)</span><span>
</span><span>file_name</span><span> </span><span>&lt;-</span><span> </span><span>paste0</span><span>(</span><span>n_file</span><span>,</span><span> </span><span>&quot;-&quot;</span><span>,</span><span> </span><span>base_name</span><span>,</span><span> </span><span>&quot;.html&quot;</span><span>)</span><span>
</span><span>### Render ###
</span><span> </span><span>render</span><span>(</span><span>
</span><span>input</span><span> </span><span>=</span><span> </span><span>file</span><span>,</span><span>
</span><span>output_format</span><span> </span><span>=</span><span> </span><span>html_document</span><span>(</span><span>
</span><span>toc</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>toc_depth</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span>
</span><span>code_folding</span><span> </span><span>=</span><span> </span><span>&quot;hide&quot;</span><span>
</span><span>),</span><span>
</span><span>output_file</span><span> </span><span>=</span><span> </span><span>file_name</span><span>,</span><span>
</span><span>output_dir</span><span> </span><span>=</span><span> </span><span>report_dir</span><span>,</span><span>
</span><span>envir</span><span> </span><span>=</span><span> </span><span>new.env</span><span>()</span><span>
</span><span>)</span><span>
</span><span>### Under macOS, open the report file ###
</span><span> </span><span>### in firefox at the end of rendering ###
</span><span> </span><span>if</span><span>(</span><span>open_file</span><span> </span><span>&amp;</span><span> </span><span>Sys.info</span><span>()[</span><span>1</span><span>]</span><span> </span><span>==</span><span> </span><span>&quot;Darwin&quot;</span><span>)</span><span> </span><span>{</span><span>
</span><span>result_path</span><span> </span><span>&lt;-</span><span> </span><span>file.path</span><span>(</span><span>report_dir</span><span>,</span><span> </span><span>file_name</span><span>)</span><span>
</span><span>system</span><span>(</span><span>command</span><span> </span><span>=</span><span> </span><span>paste</span><span>(</span><span>&quot;firefox&quot;</span><span>,</span><span> </span><span>result_path</span><span>))</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<h1>Usage</h1>
<p>The core of this workflow is the <a target="_blank" rel="nofollow" href="http://rmarkdown.rstudio.com/">rmarkdown files</a>. Most of the time, I write <em>.Rmd</em> files, mixing comments about what I’m going to do, code, results and comment about these results.</p>
<p>If there is a heavy computation (<em>e.g.</em> models or simulations), I write a script in R and save the results in a <em>.rds</em> file. Often I use remote machine for this kind of computation. Then I insert an R script in a Rmd chunk not evaluated and load the results in an evaluated one.</p>
<div>
<pre><code> ## ```{r heavy_computation, eval=FALSE}
 ## source(&quot;rscript/complicated_model.R&quot;)
 ## ```
 ## ```{r load_results}
 ## mod_results &lt;- readRDS(&quot;model_result.rds&quot;)
 ## ```
</code></pre>
</div>
<p>Because my <em>.Rmd</em> don’t live in the root directory, I add a setup chunk on all these files. This way, I’m to process my <em>.Rmd</em> directly or even use <a target="_blank" rel="nofollow" href="http://rmarkdown.rstudio.com/r_notebooks.html">Rstudio’s notebook feature</a>.</p>
<div>
<pre><code> ## ```{r setup, include=FALSE}
 ## knitr::opts_knit$set(root.dir = '../.')
 ## ```
</code></pre>
</div>
<p>When I have to rebuild all the reports (<em>e.g.</em> due to some raw data changes), I just run the script <code>run_all.R</code>.</p>
<h1>Benefits in real life</h1>
<p>This basic workflow works for me because:</p>
<ul>
<li>It’s pragmatic: it fulfils all the <a target="_blank" rel="nofollow" href="http://blog.jom.link/reproducible_analysis_my_principles.html">constraints and goals I fixed</a> without any bells or whistles.</li>
<li>It uses mainstream tools. Others can easily use it and these tools are not likely to be deprecated in the next decade.</li>
<li>It’s easy to implement and deploy.</li>
<li>It’s straightforward: peoples who want just to see the reports, know where to see, those who want to reproduce the analysis, just have to use the <code>run_all.R</code> file.</li>
</ul>
<h1>Limits</h1>
<p>There are more layers needed for a perfect data analysis reproducibility (as <a target="_blank" rel="nofollow" href="http://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1003285&type=printable">described in this article</a>). The main weakness of my workflow is the lack of back up for the good version of the software I use (R’ packages included). I already wasn’t able rerun an old data analysis due to change in the package used. (<em>e.g.</em> deprecated functions in dplyr or disappearance of a package in the CRAN). To improve this, I already try to add <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/packrat/index.html"><em>packrat</em></a> to my workflow but it’s been a long time ago and the package wasn’t stable enough for my day to day work. Resolution for 2017: give another try to <em>packrat</em>!</p>
<p>Other improvements possible: run the scripts in a <a target="_blank" rel="nofollow" href="https://www.docker.com/">docker container</a> or in a virtual machine. But this adds too much overhead in my daily work. Furthermore, it brake the platform agnostic and simplicity principles.</p>
<h1>Conclusion</h1>
<p>Currently, I’m happy with this workflow. It covers my needs of my daily data analysis reproducibility with few extra work.</p>
<p>In a later post, I will discuss about these possible extension of this basic workflow.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.jom.link/implementation_basic_reproductible_workflow.html"> Joris Muller's blog - Posts about R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/fpErOHl1NZo"></a></div>