<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/the-ten-simple-rules-for-reproducible-computational-research-are-easy-to-reach-for-r-users/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.jom.link/ten_rules_reproductible_research.html"> Joris Muller's blog - Posts about R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a target="_blank" rel="nofollow" href="http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285"><img alt="article_cover" src="https://i2.wp.com/blog.jom.link/assets/article_ten_rules_cover.jpg?w=456" data-recalc-dims="1"></a></p>
<p>“<a target="_blank" rel="nofollow" href="http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285">Ten Simple Rules for Reproducible Computational Research</a>” is a<br>
freely available paper on <a target="_blank" rel="nofollow" href="http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285">PLOS computational biology</a>.</p>
<p>As <a target="_blank" rel="nofollow" href="http://blog.jom.link/reproducible_analysis_my_principles.html">I’m currently very interested</a> on the subject of reproducible data analysis, I will these ten rules and the possible implementation in R with my point of view of epidemiologist interested in healthcare data reuse. I will also check if <a target="_blank" rel="nofollow" href="http://blog.jom.link/implementation_basic_reproductible_workflow.html">my workflow</a> comply with these rules.</p>
<p>For those who are in a hurry, I <a target="_blank" rel="nofollow" href="http://blog.jom.link/ten_rules_reproductible_research.html#summary">summarised these rules and possible implementation in R in a table at the end of this post</a>.</p>
<h1>Introduction</h1>
<p>The author of this paper, Geir Kjetil Sandve, is <a target="_blank" rel="nofollow" href="http://www.mn.uio.no/ifi/english/people/aca/geirksa/index.html">assistant-professor</a> in a Norwegian biomedical informatics lab. It is possible he wrote these rules with R in mind because he cites R in rule 7. However, biomedical informatics is quite far from my practice and he describes other integrated framework allowing reproducibility.</p>
<p>Two successive main goals for reproducible data analysis are pointed out:</p>
<ol>
<li>Be able to <strong>reproduce the results yourself</strong>. It seems to be obvious. However in my experience, without a good framework implemented at the start of the analysis, you can fail to rerun an analysis shortly after haven produce the first version of a report. I experienced this all the time before switching on R (<a target="_blank" rel="nofollow" href="http://blog.jom.link/ten_rules_reproductible_research.html#manual_steps">see “Avoid Manual Data Manipulation Steps”</a>).</li>
<li>Ensure <strong>others are able to reproduce the results</strong> (<em>e.g.</em>: peer reviewers). This is a higher level of complexity because it supposes that’s the way to rerun the analysis is straightforward (<em>e.g.</em>: steps written somewhere).</li>
</ol>
<p>The author highlight the needs of a pragmatic setting that needs a trade-off between the ideals of reproducibility and simplicity. I agree strongly with this point of view <a target="_blank" rel="nofollow" href="http://blog.jom.link/implementation_basic_reproductible_workflow.html#benefits">as I wrote in a previous post</a>.</p>
<h1>The ten rules</h1>
<p>The titles below are the exact reproduction of the rules proposed in the Sandve’s paper.</p>
<h2>1 – <em>For Every Result, Keep Track of How It Was Produced</em></h2>
<p>In other words, one report = one script. But there is an extra step because the ability to reproduce a result may rely on previous steps. Sandve advice to note the exact sequence of steps or even better, to allow the direct execution of everything. I implemented this with the “<a target="_blank" rel="nofollow" href="http://blog.jom.link/implementation_basic_reproductible_workflow.html#report_equal_rmarkdown">One report file = one R markdown file</a>” rule and the “<a target="_blank" rel="nofollow" href="http://blog.jom.link/implementation_basic_reproductible_workflow.html#run_all">one R script to rule them all</a>”. Another interesting proposition is to use Make files in order to link each part, especially when relying on different software. I already use this to produce my MPH dissertation PDF and I found this perfect in this case. Yet, there two drawback: makefiles are not platform-agnostic and it tends to become a truly labyrinthine system.</p>
<h2><a name="manual_steps"></a> 2 – <em>Avoid Manual Data Manipulation Steps</em></h2>
<p>Before mastering R, I used WYSIWYG softwares like Microsoft Excel, LibreOffice Calc or CDC’s EpiInfo. They are easy to use but once the report produced, it was impossible to check the steps or rerun them with new data. Furthermore, human more error-prone than machines. But if you are an R user, the solution is built in : just use R for every step (including data management)! Furthermore, one have to check his coding style and avoid copy-pasting of the code. In my workflow, I embrace <a target="_blank" rel="nofollow" href="http://adv-r.had.co.nz/Functional-programming.html">functional programing</a> as much as I can (with a <a target="_blank" rel="nofollow" href="http://blog.jom.link/implementation_basic_reproductible_workflow.html#folders">dedicated folder</a>).</p>
<h2>3 – <em>Archive the Exact Versions of All External Programs Used</em></h2>
<p>Software changes, sometimes at a very high pace like some R packages. Just try to run again a 4-year-old R script that uses ggplot. There is a good chance you will at the minimum have some warnings and even have your script crash. This threat is even greater when the data format change (<em>e.g.</em> Microsoft Access and Excel before and after 2007).</p>
<p>In this case, Sandve points out the need to, at a minimum, note the version number of all the software use. With R, to reach the minimum is easy: just store the results of <code>sessionInfo()</code>. A better practice is to archive all the software and their dependency. Still easy to do in R, by making a copy of your library folders (use <code>.libPath()</code> to locate them) when archiving the project. A more elegant way is to use the dedicated package <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/packrat/index.html"><em>packrat</em></a> in your workflow.</p>
<p>Yet it could be not enough. The software we use relies on dependencies that may also rely on some operating system component. In this case storing the image of the <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Virtual_machine">virtual machine</a> with it hypervisor (<em>e.g</em> <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/VirtualBox">Virtual Box</a>) could be a solution. Some drawbacks of VM: it take a lot of space (OS + software + R project) and running a complex analysis in a virtual machine could be slow. Another elegant way is to use <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Operating-system-level_virtualization">software containers</a> like <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Docker_(software)">Dockers</a>. I have little with the later then I don’t use it in my workflow but I plan to give Docker another try.</p>
<p>Even with this, there is still a layer missing: the hardware. Recently I tried to open a very old file produced on one of my first computer (a <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Macintosh_LC">Macintosh LC4</a> under <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/System_7">Mac OS 7</a>). I’m still trying a way to open this binary file… The ultimate reproducible archive is maybe to store the hardware in some bunker.</p>
<p><a target="_blank" rel="nofollow" href="https://i2.wp.com/commons.wikimedia.org/wiki/File:Kudrna-lc605b.jpg?ssl=1"><img title="A Macintosh LC stack" src="https://i2.wp.com/blog.jom.link/assets/lc_stack.jpg?w=456" data-recalc-dims="1"></a></p>
<h2>4 – <em>Version Control All Custom Scripts</em></h2>
<p>Did you ever make a change in your script that breaks everything ? Sometimes, a simple “undo” command is enough to fix it. But the working version of the script could be a quite old one. In this case, you need a system to go back in time.</p>
<p>A common practice is to use manually named files (“script_v1.R”, “script_v2.R”, “script_final.R”, “script_finalev2.R”…). It’s better than nothing but hard to maintain in the long term.</p>
<p><a target="_blank" rel="nofollow" href="http://www.phdcomics.com/comics/archive.php?comicid=1531"><img title="Final doc PhD comics" src="https://i1.wp.com/blog.jom.link/assets/phd101212s.gif?w=456" data-recalc-dims="1"></a></p>
<p>Just use a <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Version_control">version control</a> system. If you are using R with Rstudio and don’t already have one, <a target="_blank" rel="nofollow" href="https://git-scm.com/">git</a> is the natural choice. It is well integrated and allow to interact with GitHub that tends to be (<a target="_blank" rel="nofollow" href="https://www.wired.com/2015/06/problem-putting-worlds-code-github/">for good or bad</a>) an unofficial package repository. The learning curve of this software is not steep compared to R.</p>
<h2><a name="rule5"></a>5 – <em>Record All Intermediate Results, When Possible in Standardized Formats</em></h2>
<p>In theory, because we use reproducible data analysis, intermediate results are not mandatory. However a long process could be necessary to produce one results: <em>e.g.</em> data importation, cleaning, databases merging and additional variable creation are the first basic parts in my analysis. If I have to re-run all this stack each time I want to make any analysis, I lose a lot of time.</p>
<p><img alt="Example of intermediate results" src="https://i0.wp.com/blog.jom.link/assets/produced_data.jpg?w=456" data-recalc-dims="1"></p>
<p>In my workflow, intermediate results are mandatory because one R markdown file = one question. The only way to chain all my R markdown files is to use intermediate data files. For example, my first file is almost always “import.Rmd”. This import the raw files (csv, xlsx, mdb, web scrapping…). At the end of this R markdown file, I record the results. This way, the data imported are directly available for the next part of the analysis (most of the time data cleaning).</p>
<p>My preferred format is RDS as <a target="_blank" rel="nofollow" href="http://www.fromthebottomoftheheap.net/2012/04/01/saving-and-loading-r-objects/">suggested some others</a> with <code>saveRDS</code>. Another option is RData. And when it is a really important results, I save it also in .csv, just in case someone else wants to check the data with another software.</p>
<h2>6 – <em>For Analyses That Include Randomness, Note Underlying Random Seeds</em></h2>
<p>Randomness is common: for example when using <a target="_blank" rel="nofollow" href="http://statmethods.net/advstats/bootstrapping.html">bootstrap method</a> to estimate confidence intervals. Consequence: each time you run the analysis you have slightly different results. The solution is easy to implement in R: just use <code>set.seed()</code>.</p>
<p><img alt="Dice" src="https://i1.wp.com/upload.wikimedia.org/wikipedia/commons/thumb/3/37/Dice_%28PSF%29.png/298px-Dice_%28PSF%29.png?w=456&ssl=1" data-recalc-dims="1"></p>
<h2>7 – <em>Always Store Raw Data behind Plots</em></h2>
<p>Plots are important to communicate results. Producing an informative plot takes time and it’s very common necessary to make cosmetic changes in a plot to fit in a final report or in a paper. As many R users, I use mainly <a target="_blank" rel="nofollow" href="http://ggplot2.org/">ggplot2</a> to produce high-quality plots. Before starting a plot with ggplot, I produce the exact data frame needed and simply save it with <code>saveRDS()</code>. This rule is <a target="_blank" rel="nofollow" href="http://blog.jom.link/ten_rules_reproductible_research.html#rule5">almost the same as No.5</a>.</p>
<p><img alt="A plot I don't want to make again from scratch" src="https://i1.wp.com/blog.jom.link/assets/hba1c-pentes.jpg?w=456" data-recalc-dims="1"></p>
<h2>8 – <em>Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected</em></h2>
<p>The basic idea is that a summarised output (plot, table…) has to be linked with detailed value. This allows to understand and debug this result.</p>
<p>In my workflow, I simply print the intermediate results (<em>e.g.</em> for a model with <code>summary(model_name)</code>) in the R Markdown file that will generate the final output. It’s verbose but the people just have to go to the end of the reports to find the summarised output. And because one R markdown file = one question, in theory there is only one summarised output in a report.</p>
<h2>9 – <em>Connect Textual Statements to Underlying Results</em></h2>
<p>A data scientist could greatly increase the value of the result by adding clarification about how to read the results and how these results were produced. To implement this R markdown or Sweave are easy-to-use solutions. The recently introduced <a target="_blank" rel="nofollow" href="http://rmarkdown.rstudio.com/r_notebooks.html">R notebooks</a> add the possibility to see directly code, results and comment when writing the R markdown file.</p>
<h2>10 – <em>Provide Public Access to Scripts, Runs, and Results</em></h2>
<p>In my case, my “Public” are the people who ask me to analyse their data. Then I just send to them the whole project folder and explain them quickly were to find the reports (in the <em>/reports/</em> directory) and plots for their papers (in the <em>plots/</em> directory). It’s straightforward and doesn’t add any extra step.</p>
<p>When possible (for my personal projects), I share my projects with Git, but most of the time it’s just a copy to an USB key or a zipped archive send by email.</p>
<p>This is not exactly the spirit of Sandve advice. He advocate for data and scripts sharing to everyone for the sake of science transparency. The project I’m working on are not always for papers (<em>e.g.</em> analysis for my healthcare facility) and the aim is not to share it to the world. But <a target="_blank" rel="nofollow" href="http://annals.org/aim/pages/authors#data-sharing-and-reproducible-research">more and more medical paper</a> asks for this material and I will be happy to provide it if I have the opportunity to submit a paper.</p>
<h1><a name="summary"></a>Summary</h1>
<table>
<thead>
<tr>
<th> </th>
<th>Rule summary</th>
<th>Possible implementation in R</th>
<th>Implementation in my workflow</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td><em>Keep track</em></td>
<td>Keep every R script and track the sequence to run them</td>
<td><a target="_blank" rel="nofollow" href="http://blog.jom.link/implementation_basic_reproductible_workflow.html#report_equal_rmarkdown">a results = a script</a> and <a target="_blank" rel="nofollow" href="http://blog.jom.link/implementation_basic_reproductible_workflow.html#run_all"><code>run_all.R</code></a></td>
</tr>
<tr>
<td>2</td>
<td><em>No manual manipulation</em></td>
<td>Just use R!</td>
<td>R and functional programming</td>
</tr>
<tr>
<td>3</td>
<td><em>Archive software</em></td>
<td>VM, docker or packrat</td>
<td><a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/packrat/index.html"><em>packrat</em></a> + <code>sessionInfo()</code> + <code>Sys.info()</code></td>
</tr>
<tr>
<td>4</td>
<td><em>Version control</em></td>
<td>Git, SVN, mercury</td>
<td>Git</td>
</tr>
<tr>
<td>5</td>
<td><em>Record intermediate results</em></td>
<td><em>.Rdata</em>, <em>.rds</em>, <em>.csv</em>, <em>.txt</em>, <em>.xlsx</em>…</td>
<td><code>saveRDS()</code> + <code>write.csv()</code></td>
</tr>
<tr>
<td>6</td>
<td><em>Note random seeds</em></td>
<td><code>set.seed()</code></td>
<td><code>set.seed()</code></td>
</tr>
<tr>
<td>7</td>
<td><em>Store plot’s data</em></td>
<td><em>.rds</em></td>
<td><code>saveRDS()</code> before ggplot</td>
</tr>
<tr>
<td>8</td>
<td><em>Hierarchical Analysis output</em></td>
<td>Print intermediate results</td>
<td>rmarkdown chunks</td>
</tr>
<tr>
<td>9</td>
<td><em>Comment the results</em></td>
<td>Link results to text file. Use Sweave or Knitr.</td>
<td>R markdown files in <em>rmds/</em> subdirectory</td>
</tr>
<tr>
<td>10</td>
<td><em>Public access</em></td>
<td>Github, Gitlab, paper’s repository…</td>
<td>Send the whole project’s directory</td>
</tr>
</tbody>
</table>
<h1>Conclusion</h1>
<p>All the 10 rules proposed in <a target="_blank" rel="nofollow" href="http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285">the Sandve paper</a> are reachable for a R user. Just by using R itself, the <em>rmarkdown</em> workflow and some organisational rules cover most of these rules. <a target="_blank" rel="nofollow" href="http://blog.jom.link/implementation_basic_reproductible_workflow.html">My basic reproductible workflow meet almost all the criterias</a> with the notable exceptions of the software archive (but it’s work in progress with packrat) and the lack of public access (but I can’t share everything).</p>
<p>Be welcome to expose in the comments your own way to follow these rules!</p>
<h1>Image copyrigth</h1>
<p>All the images of this post are from <a target="_blank" rel="nofollow" href="https://commons.wikimedia.org/wiki/Main_Page">Wikimedia Commons</a> or are my own with the notable exeption of the PHD comic strip from <a target="_blank" rel="nofollow" href="http://phdcomics.com/">phdcomics.com</a>.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.jom.link/ten_rules_reproductible_research.html"> Joris Muller's blog - Posts about R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/_WafIOzk6QM"></a></div>