<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/rule-your-data-with-tidy-validation-reports-design/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.questionflow.org/2017/11/28/rule-your-data-with-tidy-validation-reports-design/"> QuestionFlow </a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p> The story about design of ruler package: dplyr-style exploration and validation of data frame like objects.</p>
<div>
<h1>Prologue</h1>
<p>Some time ago I had a task to write data validation code. As for most R practitioners, this led to exploration of present solutions. I was looking for a package with the following features:</p>
<ul>
<li>Relatively small amount of time should be spent learning before comfortable usage. Preferably, it should be built with <a rel="nofollow" href="https://www.tidyverse.org/" target="_blank">tidyverse</a> in mind.</li>
<li>It should be quite flexible in terms of types of validation rules.</li>
<li>Package should offer functionality for both validations (with relatively simple output format) and assertions (with relatively flexible behaviour).</li>
<li><a rel="nofollow" href="http://magrittr.tidyverse.org/reference/pipe.html" target="_blank">Pipe</a>-friendliness.</li>
<li>Validating only data frames would be enough.</li>
</ul>
<p>After devoting couple of days to research, I didn’t find any package fully (subjectively) meeting my needs (for a composed list look <a rel="nofollow" href="https://echasnovski.github.io/ruler/#other-packages-for-validation-and-assertions" target="_blank">here</a>). So I decided to write one myself. More precisely, it turned out into not one but two packages: <a rel="nofollow" href="https://echasnovski.github.io/ruler/" target="_blank">ruler</a> and <a rel="nofollow" href="https://echasnovski.github.io/keyholder/" target="_blank">keyholder</a>, which powers some of <code>ruler</code>’s functionality.</p>
<p>This post is a <em>rather long</em> story about key moments in the journey of <code>ruler</code>’s design process. To learn other aspects see its <a rel="nofollow" href="https://echasnovski.github.io/ruler/" target="_blank">README</a> (for relatively brief introduction) or <a rel="nofollow" href="https://echasnovski.github.io/ruler/articles/" target="_blank">vignettes</a> (for more thorough description of package capabilities).</p>
</div>
<div>
<h1>Overview</h1>
<p>In my mind, the whole process of data validation should be performed in the following steps:</p>
<ul>
<li>Create conditions (rules) for data to meet.</li>
<li>Expose data to them and obtain some kind of unified report as a result.</li>
<li>Act based on the report.</li>
</ul>
<p>The design process went through a little different sequence of definition steps:</p>
<ul>
<li><a rel="nofollow" href="http://www.questionflow.org/2017/11/28/rule-your-data-with-tidy-validation-reports-design/#validation-result" target="_blank">Validation result</a></li>
<li><a rel="nofollow" href="http://www.questionflow.org/2017/11/28/rule-your-data-with-tidy-validation-reports-design/#rules-definition" target="_blank">Rules definition</a></li>
<li><a rel="nofollow" href="http://www.questionflow.org/2017/11/28/rule-your-data-with-tidy-validation-reports-design/#exposing-process" target="_blank">Exposing process</a></li>
<li><a rel="nofollow" href="http://www.questionflow.org/2017/11/28/rule-your-data-with-tidy-validation-reports-design/#act-after-exposure" target="_blank">Act after exposure</a></li>
</ul>
<p>Of course, there was switching between these items in order to ensure they would work well together, but I feel this order was decisive for the end result.</p>
<pre><code>suppressMessages(library(dplyr))
suppressMessages(library(purrr))
library(ruler)</code></pre>
</div>
<div>
<h1>Validation result</h1>
<div>
<h2>Dplyr data units</h2>
<p>I started with an attempt of simple and clear formulation of <strong>validation</strong>: it is a process of checking whether <em>something satisfies certain conditions</em>. As it was enough to be only validating data frames, <em>something</em> should be thought of as <em>parts of data frame</em> which I will call <strong>data units</strong>. <em>Certain conditions</em> might be represented as functions, which I will call <strong>rules</strong>, associated with some data unit and which return <code>TRUE</code>, if condition <em>is satisfied</em>, and <code>FALSE</code> otherwise.</p>
<p>I decided to make <a rel="nofollow" href="http://dplyr.tidyverse.org/" target="_blank">dplyr</a> package a default tool for creating rules. The reason is, basically, because it satisfies most conditions I had in mind. Also I tend to use it for interactive validation of data frames, as, I am sure, many more R users. Its pipe-friendliness creates another important feature: interactive code can be transformed into a function just by replacing the initial data frame variable by a dot <code>.</code>. This will create a <em>functional sequence</em>, “a function which applies the entire chain of right-hand sides in turn to its input.”.</p>
<p><code>dplyr</code> offers a set of tools for operating with the following data units (see comments):</p>
<pre><code>is_integerish &lt;- function(x) {all(x == as.integer(x))}
z_score &lt;- function(x) {abs(x - mean(x)) / sd(x)}

mtcars_tbl &lt;- mtcars %>% as_tibble()

# Data frame as a whole
validate_data &lt;- . %>% summarise(nrow_low = n() >= 15,
                                 nrow_up = n() &lt;= 20)
mtcars_tbl %>% validate_data()
## # A tibble: 1 x 2
##   nrow_low nrow_up
##      </code></pre></div></div>