<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/building-reproducible-data-packages-with-datapackager/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ropensci.org/blog/2018/09/18/datapackager/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Sharing data sets for collaboration or publication has always been challenging, but it’s become increasingly problematic as complex and high dimensional data sets have become ubiquitous in the life sciences. Studies are large and time consuming; data collection takes time, data analysis is a moving target, as is the software used to carry it out.</p>
<p>In the vaccine space (<a rel="nofollow" href="https://www.fredhutch.org/en/labs/vaccine-and-infectious-disease/vidd-research/biostatistics-bioinformatics-epidemiology/research-overview/visc-faculty.html" target="_blank">where I work</a>) we analyze collections of high-dimensional immunological data sets from a variety of different technologies (RNA sequencing, cytometry, multiplexed antibody binding, and others). These data often arise from clinical trials and can involve tens to hundreds of subjects. The data are analyzed by teams of researchers with a diverse variety of goals.</p>
<p>Data from a single study will lead to multiple manuscripts by different principal investigators, dozens of reports, talks, presentations. There are many different consumers of data sets, and results and conclusions must be consistent and reproducible.</p>
<p>Data processing pipelines tend to be study specific. Even though we have great <a rel="nofollow" href="https://www.tidyverse.org/" target="_blank">tidyverse</a> tools that make data cleaning easier, every new data set has idiosyncracies and unique features that require some bespoke code to convert them from raw to tidy data.</p>
<p>Best practices tell us <strong>raw data are read-only</strong>, and analysis is done on some form of processed and tidied data set. Conventional wisdom is that this data tidying takes a substantial amount of the time and effort in a data analysis project, so once it’s done, why should every consumer of a data set repeat it?</p>
<p>One important reason is we don’t always know what a collaborator did to go from raw data to their version of an analysis-ready data set, so the instinct is to ask for the raw data and do the processing ourselves, which involves shipping around inconveniently large files.</p>
<h3>Objectives</h3>
<p>How can we ensure:</p>
<ul>
<li>Data consumers aren’t reinventing the wheel and repeating work already done.</li>
<li>Data are processed and tidied consistently and reproducibly.</li>
<li>Everyone is using the same versions of data sets for their analyses.</li>
<li>Reports and analyses are tied to specific versions of the data and are updated when data change.</li>
</ul>
<h3>DataPackageR</h3>
<p>Solutions to these issues require reproducible data processing, easy data sharing and versioning of data sets, and a way to verify and track data provenance of raw to tidy data.</p>
<p>Much of this can be accomplished by building <em>R data packages</em>, which are formal R packages whose sole purpose is to contain, access, and / or document data sets. By coopting R’s package system, we get documentation, versioning, testing, and other best practices for free.</p>
<p><code>DataPackageR</code> was built to help with this packaging process.</p>
<h4>Benefits of DataPackageR</h4>
<ul>
<li>
<p>It aims to automate away much of the tedium of packaging data sets without getting too much in the way, and keeps your processing workflow reproducible.</p>
</li>
<li>
<p>It sets up the necessary package structure and files for a data package.</p>
</li>
<li>
<p>It allows you to keep the large, raw data sets separate and only ship the packaged tidy data, saving space and time consumers of your data set need to spend downloading and re-processing it.</p>
</li>
<li>
<p>It maintains a reproducible record (vignettes) of the data processing along with the package. Consumers of the data package can verify how the processing was done, increasing confidence in your data.</p>
</li>
<li>
<p>It automates construction of the documentation and maintains a data set version and an <a rel="nofollow" href="https://en.wikipedia.org/wiki/MD5" target="_blank">md5 fingerprint</a> of each data object in the package. If the data changes and the package is rebuilt, the data version is automatically updated.</p>
</li>
<li>
<p>Data packages can be version controlled on GitHub (or other VCS sites), making for easy change tracking and sharing with collaborators.</p>
</li>
<li>
<p>When a manuscript is submitted based on a specific version of a data package, one can make a <em>GitHub release</em> and automatically push it to sites like <a rel="nofollow" href="http://zenodo.org/" target="_blank">zenodo</a> so that it is permanently archived.</p>
</li>
<li>
<p>Consumers of the data package pull analysis-ready data from a central location, ensuring all downstream results rely on consistently processed data sets.</p>
</li>
</ul>
<h3>Development</h3>
<p>The package developed organically over the span of several years, initially as a proof-of-concept, with features bolted on over time. <a rel="nofollow" href="https://github.com/paulobrecht" target="_blank">Paul Obrecht</a> kindly contributed the data set autodocumentation code and provided in-house testing. As it gained usage internally, we thougt it might be useful to others, and so began the process of releasing it publicly.</p>
<h4>rOpenSci</h4>
<p><code>DataPackageR</code> was a departure from the <a rel="nofollow" href="http://www.gregfinak.com/#software" target="_blank">usual tools</a> we develop, which live mostly on <a rel="nofollow" href="http://bioconductor.org/" target="_blank">Bioconductor</a>. We thought the <a rel="nofollow" href="http://ropensci.org/" target="_blank">rOpenSci</a> community would be a good place to release the package and reach a wider, more diverse audience.</p>
<h4>The onboarding process</h4>
<p>Onboarding was a great experience. <a rel="nofollow" href="https://github.com/maelle" target="_blank">Maëlle Salmon</a>, <a rel="nofollow" href="https://github.com/karawoo" target="_blank">Kara Woo</a>, <a rel="nofollow" href="https://github.com/wlandau" target="_blank">Will Landau</a>, and <a rel="nofollow" href="https://github.com/noamross" target="_blank">Noam Ross</a> volunteered their time to provide in-depth, comprehensive and careful code review, suggestions for features, documentation, and improvements in coding style and unit tests, that ultimately made the software better. It was an impressive effort (you can see the GitHub issue thread for yourself <a rel="nofollow" href="http://www.github.com/ropensci/onboarding/issues/230" target="_blank">here</a>).</p>
<p>The current version of the package has <strong>100% test coverage</strong> and comprehensive documentation. One great benefit is that as I develop new features in the future, I can be confident I’m not inadvertently breaking something else.</p>
<p>rOpenSci encourages a number of other best practices that I had been somewhat sloppy about enforcing on myself. One is maintaining an updated <code>NEWS.md</code> file which tracks major changes and new features, and links them to relevant GitHub issues. I find this particularly useful as I’ve always appreciated an informative NEWS file to help me decide if I should install the latest version of a piece of software. The <a rel="nofollow" href="https://ropensci.github.io/dev_guide/collaboration.html" target="_blank">online handbook for rOpenSci package development, maintenance and peer review</a> is a great resource to learn about what some of those other best practices are and how you can prepare your own software for submission.</p>
<h3>Using DataPackageR</h3>
<p>Here’s a simple example that demonstrates how <code>DataPackageR</code> can be used to construct a data pacakge.</p>
<p>In this example, I’ll tidy the <code>cars</code> data set by giving it more meaningful column names. The original data set has columns <code>speed</code> and <code>dist</code>. We’ll be more verbose and rename them to <code>speed_mph</code> and <code>stopping_distance</code>.</p>
<h4>An outline of the steps involved.</h4>
<ol>
<li>Create a package directory structure. (<code>DataPackageR::datapackage_skeleton()</code>)</li>
<li>We add a raw data set. (<code>DataPackageR::use_raw_dataset()</code>)</li>
<li>We add a processing script to process <code>cars</code> into <code>tidy_cars</code>. (<code>DataPackageR::use_processing_script()</code>)</li>
<li>We define the tidy data object (named <code>tidy_cars</code>) that will be stored in our data package. (<code>DataPackageR::use_data_object()</code>)</li>
</ol>
<h4>Create a data package directory structure.</h4>
<p>The first step is to create the data package directory structure. This package structure is based on <a rel="nofollow" href="https://github.com/ropensci/rrrpkg" target="_blank"><code>rrrpkg</code></a> and some <a rel="nofollow" href="http://r-pkgs.had.co.nz/data.html#data-extdata" target="_blank">conventions introcuded by Hadley Wickham</a>.</p>
<pre><code>datapackage_skeleton(name = &quot;TidyCars&quot;,
                     path = tempdir(),
                     force = TRUE)
</code></pre>
<pre><code>## ✔ Setting active project to '/private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpXXFG1L/TidyCars'
## ✔ Creating 'R/'
## ✔ Creating 'man/'
## ✔ Writing 'DESCRIPTION'
## ✔ Writing 'NAMESPACE'
## ✔ Added DataVersion string to 'DESCRIPTION'
## ✔ Creating 'data-raw/'
## ✔ Creating 'data/'
## ✔ Creating 'inst/extdata/'
## ✔ configured 'datapackager.yml' file
</code></pre>
<div>
We get some informative output as the various directories are created. Notice the message “Setting active project to…”. This is what allows the rest of the workflow below to work as expected. Internally it uses <code>usethis::proj_set</code> and <code>usethis::proj_get</code>, so be aware if you are using that package in your own scripts, mixing it with <code>DataPackageR</code>, there’s a risk of unexpected side-effects.
</div>
<p>The directory structure that’s created is shown below.</p>
<pre><code>/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T//RtmpXXFG1L/TidyCars
├── DESCRIPTION
├── R
├── Read-and-delete-me
├── data
├── data-raw
├── datapackager.yml
├── inst
│   └── extdata
└── man

6 directories, 3 files
</code></pre>
<h4>Add raw data to the package.</h4>
<p>Next, we work interactively, following the paradigm of the <a rel="nofollow" href="https://github.com/r-lib/usethis" target="_blank"><code>usethis</code></a> package.</p>
<pre><code># write our raw data to a csv
write.csv(x = cars,file = file.path(tempdir(),&quot;cars.csv&quot;),row.names = FALSE)
# this works because we called datapackage_skeleton() first.
use_raw_dataset(file.path(tempdir(),&quot;cars.csv&quot;))
</code></pre>
<p><code>use_raw_dataset()</code> moves the file or path in its argument into <code>inst/extdata</code> under the data package source tree. This raw (usually non-tidy) data will be installed with the data pacakge.</p>
<p>For large data sets that you may not want to distribute with the package, you could place them in a directory external to the package source, or place them in <code>inst/extdata</code> but include them in <code>.Rbuildignore</code>. In fact as I write this, there should be an option to add a data set to <code>.Rbuildignore</code> automatically. That would be <a rel="nofollow" href="http://www.github.com/ropensci/DataPackageR/issues/48" target="_blank">a good first issue</a> for anyone who would like to contribute.</p>
<h4>Add a data processing script.</h4>
<p>We add a script to process <code>cars</code> into <code>tidy_cars</code>. The author and title are provided as arguments. They will go into the yaml frontmatter of the Rmd file.</p>
<div>
Note we specify that our script is an <code>Rmd</code> file. This is recommended. Use <a href="https://en.wikipedia.org/wiki/Literate_programming">literate programming</a> to process your data, and the <code>Rmd</code> will appear as a <a href="http://r-pkgs.had.co.nz/vignettes.html">vignette</a> in your final data package.
</div>
<pre><code>use_processing_script(file = &quot;tidy_cars.Rmd&quot;,
                      author = &quot;Greg Finak&quot;,
                      title = &quot;Process cars into tidy_cars.&quot;)
</code></pre>
<pre><code>## configuration:
##   files:
##     tidy_cars.Rmd:
##       enabled: yes
##   objects: []
##   render_root:
##     tmp: '103469'
</code></pre>
<p>The script file <code>tidy_cars.Rmd</code> is created in the <code>data-raw</code> directory of the package source. The output echoed after the command is the contents of the <code>datapackager.yml</code> configuration file. It controls the build process. Here the file is added to the configuration. You can find more information about it in the <a rel="nofollow" href="https://github.com/ropensci/DataPackageR/blob/master/vignettes/YAML_CONFIG.md" target="_blank">configuration vignette</a>.</p>
<h4>Edit your processing script.</h4>
<p>Our script will look like this:</p>
<pre><code>---
title: Process cars into tidy_cars.
author: Greg Finak
date: September 5, 2018
output_format: html_document
---
```{r}
library(dplyr)
cars &lt;- read.csv(project_extdata_path('cars.csv'))
tidy_cars &lt;- cars %>% rename(speed_mph = speed, stopping_distace = dist)
```
Followed by a description of what we are doing.
</code></pre>
<h5>An important note about reading raw data from the processing script.</h5>
<p>In order to read raw data sets in a reproducible manner, <code>DataPackageR</code> provides an API call:<br>
<code>project_extdata_path()</code> that returns the absolute path to its argument in <code>inst/extdata</code> in a reproducible way, independent of the current working directory. There are also <code>project_path()</code> and <code>project_data_path()</code> that will point to the source root and the <code>data</code> directory, respectively.</p>
<div>NOTE that <code>DataPackageR</code> is not compatible with the <code>here</code> package. Rather use the APIs above.</div>
<p>This script creates the <code>tidy_cars</code> object, which is what we want to store in our final package.</p>
<h4>Let DataPackageR know about the data objects to store in the package.</h4>
<p>We let <code>DataPackageR</code> know about this:</p>
<pre><code>use_data_object(&quot;tidy_cars&quot;)
</code></pre>
<pre><code>## configuration:
##   files:
##     tidy_cars.Rmd:
##       enabled: yes
##   objects:
##   - tidy_cars
##   render_root:
##     tmp: '103469'
</code></pre>
<p>Again, the <code>datapackager.yml</code> configuration is echoed, and we see the data set object has been added.</p>
<p>The build process uses this file to know which scripts to run and what data outputs to expect. More information is in the <a rel="nofollow" href="https://github.com/ropensci/DataPackageR/blob/master/vignettes/usingDataPackageR.md" target="_blank">technical vignette</a>.</p>
<p>It will automatically create documentation stubs for the package and for these data objects.</p>
<h4>Build the package (for the first time).</h4>
<p>We build the package. It will automatically generate some documentation for the data sets that we’ll need to go in and edit. There’s also some informative output. <a rel="nofollow" href="https://www.github.com/ropensci/DataPackageR/issues/50" target="_blank">The output has been cleaned up recently</a>, particularly now that the package has stabilized.</p>
<pre><code>options(&quot;DataPackageR_interact&quot; = FALSE)
</code></pre>
<div>
If you run <code>package_build()</code> in interactive mode, you’ll be prompted to fill in one line of information that will be put in the NEWS.md file, describing the changes to the data package. This helps you track changes across versions.<br>
Setting <code>options(&quot;DataPackageR_interact&quot; = FALSE)</code> turns off interactive mode.
</div>
<pre><code>package_build(packageName = file.path(tempdir(),&quot;TidyCars&quot;), install = FALSE)
</code></pre>
<pre><code>## 
## ✔ Setting active project to '/private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpXXFG1L/TidyCars'
## ✔ 1 data set(s) created by tidy_cars.Rmd
## • tidy_cars
## ☘ Built  all datasets!
## Non-interactive NEWS.md file update.
## 
## ✔ Creating 'vignettes/'
## ✔ Creating 'inst/doc/'
</code></pre>
<pre><code>## First time using roxygen2. Upgrading automatically...
</code></pre>
<pre><code>## Updating roxygen version in /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpXXFG1L/TidyCars/DESCRIPTION
</code></pre>
<pre><code>## Writing NAMESPACE
</code></pre>
<pre><code>## Loading TidyCars
</code></pre>
<pre><code>## Writing TidyCars.Rd
## Writing tidy_cars.Rd
</code></pre>
<pre><code>## '/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file  \
##   --no-environ --no-save --no-restore --quiet CMD build  \
##   '/private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpXXFG1L/TidyCars'  \
##   --no-resave-data --no-manual --no-build-vignettes
</code></pre>
<pre><code>## 
</code></pre>
<pre><code>## Next Steps 
## 1. Update your package documentation. 
##    - Edit the documentation.R file in the package source data-raw subdirectory and update the roxygen markup. 
##    - Rebuild the package documentation with  document() . 
## 2. Add your package to source control. 
##    - Call  git init .  in the package source root directory. 
##    -  git add  the package files. 
##    -  git commit  your new package. 
##    - Set up a github repository for your pacakge. 
##    - Add the github repository as a remote of your local package repository. 
##    -  git push  your local repository to gitub.
</code></pre>
<pre><code>## [1] &quot;/private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpXXFG1L/TidyCars_1.0.tar.gz&quot;
</code></pre>
<div>
The argument <code>install = FALSE</code> prevents the package from being automatically installed to the system after building.
</div>
<p>Above, you’ll see the message:</p>
<blockquote>
<p>1 data set(s) created by tidy_cars.Rmd</p>
</blockquote>
<p>Indicating that the build process found the one of the expected data sets. It then lists the specific data set(s) it created in that script.<br>
It stores those data sets in the package <code>data</code> subfolder as <code>.rda</code> files.</p>
<p>You’ll see it also created vignettes, wrote a roxygen stub for the documentation of the <code>TidyCars</code> package and the <code>tidy_cars</code> data object. It created provisional Rd files, and built the package archive. It even provides some information on what to do next.</p>
<h4>Next edit the data set documentation.</h4>
<p>Let’s update our documentation as requested. We edit the <code>documentation.R</code> file under <code>data-raw</code>.</p>
<p>Here are its contents:</p>
<pre><code>#' TidyCars
#' A data package for TidyCars.
#' @docType package
#' @aliases TidyCars-package
#' @title Package Title
#' @name TidyCars
#' @description A description of the data package
#' @details Use \code{data(package='TidyCars')$results[, 3]} tosee a list of availabledata sets in this data package
#'     and/or DataPackageR::load_all
#' _datasets() to load them.
#' @seealso
#' \link{tidy_cars}
NULL



#' Detailed description of the data
#' @name tidy_cars
#' @docType data
#' @title Descriptive data title
#' @format a \code{data.frame} containing the following fields:
#' \describe{
#' \item{speed_mph}{}
#' \item{stopping_distace}{}
#' }
#' @source The data comes from________________________.
#' @seealso
#' \link{TidyCars}
NULL
</code></pre>
<p>This is standard roxygen markup. You can use roxygen or <a rel="nofollow" href="https://cran.r-project.org/web/packages/roxygen2/vignettes/markdown.html" target="_blank">markdown style comments</a>. You should describe your data set, where it comes from, the columns of the data (if applicable), and any other information that can help a user make good use of and understand the data set.</p>
<p>We’ll fill this in and save the resulting file.</p>
<pre><code>## #' TidyCars
## #' A data package for TidyCars.
## #' @docType package
## #' @aliases TidyCars-package
## #' @title A tidied cars data set.
## #' @name TidyCars
## #' @description Cars but better. The variable names are more meaninful.
## #' @details The columns have been renamed to indicate the units and better describe what is measured.
## #' @seealso
## #' \link{tidy_cars}
## NULL
## 
## 
## 
## #' The stopping distances of cars at different speeds.
## #' @name tidy_cars
## #' @docType data
## #' @title The stopping distances of cars traveling at different speeds.
## #' @format a \code{data.frame} containing the following fields:
## #' \describe{
## #' \item{speed_mph}{The speed of the vehicle.}
## #' \item{stopping_distace}{The stopping distance of the vehicle.}
## #' }
## #' @source The data comes from the cars data set distributed with R.
## #' @seealso
## #' \link{TidyCars}
## NULL
</code></pre>
<p>Then we run the <code>document()</code> method in the <code>DataPackageR::</code> namespace to rebuild the documentation.</p>
<pre><code># ensure we run document() from the DataPackageR namespace and not document() from roxygen or devtools.
package_path &lt;- file.path(tempdir(),&quot;TidyCars&quot;)
DataPackageR::document(package_path)
</code></pre>
<pre><code>## 
## ✔ Setting active project to '/private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpXXFG1L/TidyCars'
</code></pre>
<pre><code>## Updating TidyCars documentation
</code></pre>
<pre><code>## Loading TidyCars
</code></pre>
<pre><code>## Writing TidyCars.Rd
## Writing tidy_cars.Rd
</code></pre>
<pre><code>## [1] TRUE
</code></pre>
<h4>Iterate…</h4>
<p>We can add more data sets, more scripts, and so forth, until we’re happy with the package.</p>
<h4>A final build.</h4>
<p>Finally we rebuild the package one last time. The output is suppressed here for brevity.</p>
<pre><code>package_build(file.path(tempdir(),&quot;TidyCars&quot;))
</code></pre>
<h2>Sharing and distributing data packages.</h2>
<p>If you are the data package developer you may consider:</p>
<ul>
<li>Placing the source of the data package under version control (we like <a rel="nofollow" href="https://git-scm.com/downloads" target="_blank">git</a> and <a rel="nofollow" href="http://www.github.com/" target="_blank">GitHub</a>).</li>
<li>Share the package archive (<code>yourpackage-x.y.z.tar.gz</code>)
<ul>
<li>on a public repository.</li>
<li>directly with collaborators.</li>
</ul>
</li>
</ul>
<p>We’ve placed the <a rel="nofollow" href="http://www.github.com/gfinak/TidyCars" target="_blank">TidyCars data package on GitHub</a> so that you can see for yourself how it looks.</p>
<h4>Limitations and future work</h4>
<p>Versions of software dependencies for processing scripts are not tracked. Users <strong>should</strong> use <code>sessionInfo()</code> to keep track of the versions of software used to perform data processing so that the environment can be replicated if a package needs to be rebuilt far in the future.</p>
<p>Tools like <a rel="nofollow" href="https://rstudio.github.io/packrat/" target="_blank"><code>packrat</code></a> and <a rel="nofollow" href="https://www.docker.com/" target="_blank"><code>Docker</code></a> aim to solve these problems, and it is non-trivial. I would love to integrate these tools more closely with <code>DataPackageR</code> in the future.</p>
<h3>Using a data package.</h3>
<p>If you are a user of a data pacakge.</p>
<p>Install a data package in the usual manner of R package installation.</p>
<ul>
<li>From GitHub:</li>
</ul>
<pre><code>devtools::install_github(&quot;gfinak/TidyCars&quot;)
</code></pre>
<ul>
<li>From the command line:</li>
</ul>
<pre><code>R CMD INSTALL TidyCars-0.1.0.tar.gz
</code></pre>
<p>Once the package is installed, we can load the newly built package and access documentation, vignettes, and use the <code>DataVersion</code> of the package in downstream analysis scripts.</p>
<pre><code>library(TidyCars)
browseVignettes(package = &quot;TidyCars&quot;)
</code></pre>
<p>Typing the above in the R console will pop up a browser window where you’ll see the available vignettes in your new <code>TidyCars</code> package.</p>
<p><img alt="Browse TidyCars Vignettes" src="https://i0.wp.com/ropensci.org/img/blog-images/2018-09-18-datapackager/browsevignettes.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Clicking the <code>HTML</code> link gives you access to the output of the processing script, rendered as a vignette. Careful work here will let you come back to your code and understand what you have done.</p>
<p><img alt="Tidy Cars HTML Vignette" src="https://i1.wp.com/ropensci.org/img/blog-images/2018-09-18-datapackager/tidy_cars_vignette.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>We can also view the data set documentation:</p>
<pre><code>?tidy_cars
</code></pre>
<p><img alt="tidy_cars Rd" src="https://i0.wp.com/ropensci.org/img/blog-images/2018-09-18-datapackager/tidy_cars_doc.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>And we can use the <code>assert_data_version()</code> API to test the version of a data package in a downstream analysis that depends on the data.</p>
<pre><code>data_version(&quot;TidyCars&quot;)
</code></pre>
<pre><code>## [1] '0.1.0'
</code></pre>
<pre><code>assert_data_version(&quot;TidyCars&quot;,version_string = &quot;0.1.0&quot;, acceptable = &quot;equal&quot;)
assert_data_version(&quot;TidyCars&quot;,version_string = &quot;0.2.0&quot;, acceptable = &quot;equal&quot;)
</code></pre>
<pre><code>## Error in assert_data_version(&quot;TidyCars&quot;, version_string = &quot;0.2.0&quot;, acceptable = &quot;equal&quot;): Found TidyCars 0.1.0 but  == 0.2.0 is required.
</code></pre>
<p>The first assertion is true, the second is not, throwing an error. In downstream analyses that depend on a version of a data package, this is useful to ensure updated data don’t inadvertently change results, without the user being aware that something unexpected is going on.</p>
<p>A data package can be built into a package archive (<code>.tar.gz</code>) using the standard <code>R CMD build</code> process. The only difference is that the <code>.rda</code> files in <code>/data</code> won’t be re-generated, and the existing vignettes describing the processing won’t be rebuilt. This is useful when the processing of raw data sets is time consuming (like some biological data), or when raw data sets are too large to distribute conveniently.</p>
<h3>To conclude</h3>
<p>With <code>DataPackageR</code> I’ve tried to implement a straightforward workflow for building data packages. One that doesn’t get in the way (based on my own experience) of how people develop their data processing pipelines. The new APIs are limited and only minor changes need to be made to adapt existing code to the workflow.</p>
<p>With a data package in hand, data can be easily shared and distributed. In my view, the greatest benefit of building a data package is that it encourages us to use best practices, like documenting data sets, documenting code, writing unit tests, and using version control. Since we are very often our own worst enemies, these are all Good Things.</p>
<p>We have been eating our own dog food, so to speak.<br>
We use data packages internally in the <a rel="nofollow" href="https://www.fredhutch.org/en/labs/vaccine-and-infectious-disease/services/VISCnetwork.html" target="_blank">Vaccine Immunology Statistical Center</a> to prepare data sets for analysis, publication, and for sharing with collaborators.<br>
We often do so through the <a rel="nofollow" href="https://www.github.com/FredHutch" target="_blank">Fred Hutch GitHub Organization</a> (though most data are private).</p>
<p>The <a rel="nofollow" href="http://rglab.org/" target="_blank">RGLab</a> has used data packages (though not built using <code>DataPackageR</code>) to share data sets together with publications:</p>
<ul>
<li>Combinatorial Polyfunctionality Analysis of Single Cells: <a rel="nofollow" href="https://www.nature.com/articles/nbt.3187" target="_blank">paper</a>, <a rel="nofollow" href="https://doi.org/10.5281/zenodo.46009" target="_blank">data</a></li>
<li>Model-based Analysis of Single-cell Trascriptomics <a rel="nofollow" href="https://genomebiology.biomedcentral.com/articles/10.1186/s13059-015-0844-5" target="_blank">paper</a>, <a rel="nofollow" href="https://doi.org/10.5281/zenodo.31723" target="_blank">data</a></li>
</ul>
<p>In the long run, packaging data saves us time and effort (of the sort expended trying to reproduce results of long ago and far away) by ensuring data processing is reproducible and properly tracked. It’s proved extremely useful to our group and I hope it’s useful to others as well.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://ropensci.org/blog/2018/09/18/datapackager/"> rOpenSci - open tools for open science</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/ThXIRvBcGjY" height="1"></a></div>