Usage examples of ruler package: dplyr-style exploration and validation of data frame like objects.
Prologue
My previous post tells a story about design of my ruler package, which presents tools for “… creating data validation pipelines and tidy reports”. This package offers a framework for exploring and validating data frame like objects using dplyr grammar of data manipulation.
This post is intended to show some close to reality ruler usage examples. Described methods and approaches reflect package design. Along the way you will learn why Yoda and Jabba the Hutt are “outliers” among core “Star Wars” characters.
For more information see README (for relatively brief comprehensive introduction) or vignettes (for more thorough description of package capabilities).
Beware of a lot of code.
Overview
suppressMessages(library(dplyr))
suppressMessages(library(purrr))
library(ruler)
The general way of performing validation with ruler can be described with following steps:
Formulate a validation task. It is usually stated in the form of a yes-no question or true-false statement about some part (data unit) of an input data frame. Data unit can be one of: data [as a whole], group of rows [as a whole], column [as a whole], row [as a whole], cell. For example, does every column contain elements with sum more than 100?.
Create a dplyr-style validation function (rule pack) which checks desired data unit for obedience to [possibly] several rules:
Create dplyr code for “interactive” validation. Note to use only functions supported by keyholder package. An example with “enough_sum” as a rule name:
mtcars %__% summarise_all(funs(enough_sum = sum(.) __ 100))
Use ruler’s function rules() instead of explicit or implicit usage of funs():
mtcars %__% summarise_all(rules(enough_sum = sum(.) __ 100))
Modify code to create magrittr functional sequence by replacing input variable name by a .:
. %__% summarise_all(rules(enough_sum = sum(.) __ 100))
Wrap with rule specification function to explicitly identify validated data unit and to name rule pack. In this case it is col_packs() for column data unit with “is_enough_sum” as rule pack name:
col_packs(
is_enough_sum = . %__% summarise_all(rules(is_enough = sum(.) __ 100))
)
Expose data to rules to obtain validation result (exposure). Use ruler’s expose() function for that. It doesn’t modify contents of input data frame but creates/updates exposure attribute. Exposure is a list with information about used rule packs (packs_info) and tidy data validation report (report).
Act after exposure. It can be:
Observing validation results with get_exposure(), get_packs_info() or get_report().
Making assertions if specific rules are not followed in desired way.
Imputing input data frame based on report.
In examples we will use starwars data from dplyr package (to celebrate an upcoming new episode). It is a tibble with every row describing one “Star Wars” character. Every example starts with a validation task stated in italic and performs validation from beginning to end.
Create rule packs
Data
■ Does starwars have 1) number of rows 1a) more than 50; 1b) less than 60; 2) number of columns 2a) more than 10; 2b) less than 15?
check_data_dims % summarise(
nrow_low = nrow(.) __= 50, nrow_up = nrow(.) = 10, ncol_up = ncol(.) %
expose(check_data_dims) %__%
get_exposure()
## Exposure
## ## Packs info:
## # A tibble: 1 x 4
## name type fun remove_obeyers
## ## 1 check_dims data_pack TRUE
## ## Tidy data validation report:
## # A tibble: 1 x 5
## pack rule var id value
## ## 1 check_dims nrow_up .all 0 FALSE
The result is interpreted as follows:
Data was exposed to one rule pack for data as a whole (data rule pack) named “check_dims”. For it all obeyers (data units which follow specified rule) were removed from validation report.
Combination of var equals .all and id equals 0 means that data as a whole is validated.
Input data frame doesn’t obey (because value is equal to FALSE) rule nrow_up from rule pack check_dims.
■ Does starwars have enough rows for characters 1) with blond hair; 2) humans; 3) humans with blond hair?
check_enough_rows % filter(hair_color == &quot;blond&quot;) %__%
summarise(is_enough = n() __ 10),
enough_humans = . %__% summarise(
is_enough = sum(species == &quot;Human&quot;, na.rm = TRUE) __ 30
),
ehough_blond_humans = . %__% filter(
hair_color == &quot;blond&quot;, species == &quot;Human&quot;
) %__%
summarise(is_enough = n() __ 5)
)
starwars %__%
expose(check_enough_rows) %__%
get_exposure()
## Exposure
## ## Packs info:
## # A tibble: 3 x 4
## name type fun remove_obeyers
## ## 1 enough_blond data_pack TRUE
## 2 enough_humans data_pack TRUE
## 3 ehough_blond_humans data_pack TRUE
## ## Tidy data validation report:
## # A tibble: 2 x 5
## pack rule var id value
## ## 1 enough_blond is_enough .all 0 FALSE
## 2 ehough_blond_humans is_enough .all 0 FALSE
New information gained from example:
Rule specification functions can be supplied with multiple rule packs all of which will be independently used during exposing.
■ Does starwars have enough numeric columns?
check_enough_num_cols % summarise(
is_enough = sum(map_lgl(., is.numeric)) __ 1
)
)
starwars %__%
expose(check_enough_num_cols) %__%
get_report()
## Tidy data validation report:
## # A tibble: 0 x 5
## # ... with 5 variables: pack , rule , var , id ,
## # value If no breaker is found get_report() returns tibble with zero rows and usual columns.
Group
■ Does group defined by hair color and gender have a member from Tatooine?
has_hair_gender_tatooine %
group_by(hair_color, gender) %__%
summarise(has_tatooine = any(homeworld == &quot;Tatooine&quot;)),
.group_vars = c(&quot;hair_color&quot;, &quot;gender&quot;),
.group_sep = &quot;__&quot;
)
starwars %__%
expose(has_hair_gender_tatooine) %__%
get_report()
## Tidy data validation report:
## # A tibble: 12 x 5
## pack rule var id value
## ## 1 hair_gender_tatooine has_tatooine auburn__female 0 FALSE
## 2 hair_gender_tatooine has_tatooine auburn, grey__male 0 FALSE
## 3 hair_gender_tatooine has_tatooine auburn, white__male 0 FALSE
## 4 hair_gender_tatooine has_tatooine blonde__female 0 FALSE
## 5 hair_gender_tatooine has_tatooine grey__male 0 FALSE
## # ... with 7 more rows
group_packs() needs grouping columns supplied via .group_vars.
Column var of validation report contains levels of grouping columns to identify group. By default their are pasted together with .. To change that supply .group_sep argument.
12 combinations of hair_color and gender don’t have a character from Tatooine. They are “auburn”-“female”, “auburn, grey”-“male” and so on.
Column
■ Does every list-column have 1) enough average length; 2) enough unique elements?
check_list_cols %
summarise_if(
is.list,
rules(
is_enough_mean = mean(map_int(., length)) __= 1,
length(unique(unlist(.))) __= 10
)
)
)
starwars %__%
expose(check_list_cols) %__%
get_report()
## Tidy data validation report:
## # A tibble: 3 x 5
## pack rule var id value
## ## 1 check_list_cols is_enough_mean vehicles 0 FALSE
## 2 check_list_cols is_enough_mean starships 0 FALSE
## 3 check_list_cols rule..2 films 0 FALSE
To specify rule functions inside dplyr’s scoped verbs use ruler::rules(). It powers correct output interpretation during exposing process and imputes missing rule names based on the present rules in current rule pack.
Columns vehicles and starships don’t have enough average length and column films doesn’t have enough unique elements.
■ Are all values of column birth_year non-NA?
starwars %__%
expose(
col_packs(
. %__% summarise_at(
vars(birth_year = &quot;birth_year&quot;),
rules(all_present = all(!is.na(.)))
)
)
) %__%
get_report()
## Tidy data validation report:
## # A tibble: 1 x 5
## pack rule var id value
## ## 1 col_pack..1 all_present birth_year 0 FALSE
To correctly validate one column with scoped dplyr verb it should be a named argument inside vars. It is needed for correct interpretation of rule pack output.
Row
■ Has character appeared in enough films? As character is defined by row, this is a row pack.
has_enough_films % transmute(is_enough = map_int(films, length) __= 3)
)
starwars %__%
expose(has_enough_films) %__%
get_report() %__%
left_join(y = starwars %__% transmute(id = 1:n(), name),
by = &quot;id&quot;) %__%
print(.validate = FALSE)
## Tidy data validation report:
## # A tibble: 64 x 6
## pack rule var id value name
## ## 1 enough_films is_enough .all 8 FALSE R5-D4
## 2 enough_films is_enough .all 9 FALSE Biggs Darklighter
## 3 enough_films is_enough .all 12 FALSE Wilhuff Tarkin
## 4 enough_films is_enough .all 15 FALSE Greedo
## 5 enough_films is_enough .all 18 FALSE Jek Tono Porkins
## # ... with 59 more rows
64 characters haven’t appeared in 3 films or more. Those are characters described in starwars in rows 8, 9, etc. (counting based on input data).
■ Does character with height less than 100 is a droid?
is_short_droid % filter(height __ 100) %__%
transmute(is_droid = species == &quot;Droid&quot;)
)
starwars %__%
expose(is_short_droid) %__%
get_report() %__%
left_join(y = starwars %__% transmute(id = 1:n(), name, height),
by = &quot;id&quot;) %__%
print(.validate = FALSE)
## Tidy data validation report:
## # A tibble: 5 x 7
## pack rule var id value name height
## ## 1 is_short_droid is_droid .all 19 FALSE Yoda 66
## 2 is_short_droid is_droid .all 29 FALSE Wicket Systri Warrick 88
## 3 is_short_droid is_droid .all 45 FALSE Dud Bolt 94
## 4 is_short_droid is_droid .all 72 FALSE Ratts Tyerell 79
## 5 is_short_droid is_droid .all 73 NA R4-P17 96
One can expose only subset of rows by using filter or slice. The value of id column in result will reflect row number in the original input data frame. This feature is powered by keyholder package. In order to use it, rule pack should be created using its supported functions.
value equal to NA is treated as rule breaker.
5 “not tall” characters are not droids.
Cell
■ Is non-NA numeric cell not an outlier based on z-score? This is a bit tricky. To present outliers as rule breakers one should ask whether cell is not outlier.
z_score %
expose(cell_isnt_outlier) %__%
get_report() %__%
left_join(y = starwars %__% transmute(id = 1:n(), name),
by = &quot;id&quot;) %__%
print(.validate = FALSE)
## Tidy data validation report:
## # A tibble: 4 x 6
## pack rule var id value name
## ## 1 dbl_not_outlier isnt_out height 19 FALSE Yoda
## 2 dbl_not_outlier isnt_out mass 16 FALSE Jabba Desilijic Tiure
## 3 dbl_not_outlier isnt_out birth_year 16 FALSE Jabba Desilijic Tiure
## 4 dbl_not_outlier isnt_out birth_year 19 FALSE Yoda
4 non-NA numeric cells appear to be an outlier within their column.
Expose data to rules
■ Do groups defined by species, gender and eye_color (3 different checks) have appropriate size?
starwars %__%
expose(
group_packs(. %__% group_by(species) %__% summarise(isnt_many = n() %
expose(
group_packs(. %__% group_by(gender) %__% summarise(isnt_many = n() %
expose(is_enough_eye_color = . %__% group_by(eye_color) %__%
summarise(isnt_many = n() %
get_exposure() %__%
print(n_report = Inf)
## Exposure
## ## Packs info:
## # A tibble: 3 x 4
## name type fun remove_obeyers
## ## 1 group_pack..1 group_pack TRUE
## 2 group_pack..2 group_pack FALSE
## 3 is_enough_eye_color group_pack TRUE
## ## Tidy data validation report:
## # A tibble: 7 x 5
## pack rule var id value
## ## 1 group_pack..1 isnt_many Human 0 FALSE
## 2 group_pack..2 isnt_many female 0 TRUE
## 3 group_pack..2 isnt_many hermaphrodite 0 TRUE
## 4 group_pack..2 isnt_many male 0 FALSE
## 5 group_pack..2 isnt_many none 0 TRUE
## 6 group_pack..2 isnt_many NA 0 TRUE
## 7 is_enough_eye_color isnt_many brown 0 FALSE
expose() can be applied sequentially which results into updating existing exposure with new information.
expose() imputes names of supplied unnamed rule packs based on the present rule packs for the same data unit type.
expose() by default removes obeyers (rows with data units that obey respective rules) from validation report. To stop doing that use .remove_obeyers = FALSE during expose() call.
expose() by default guesses the type of the supplied rule pack based only on its output. This has some annoying edge cases but is suitable for interactive usage. To turn this feature off use .guess = FALSE as an argument for expose(). Also, to avoid edge cases create rule packs with appropriate wrappers.
■ Perform some previous checks with one expose().
my_packs %
expose(
col_packs(
low_mass = . %__% summarise_at(
vars(mass = &quot;mass&quot;),
rules(is_small_mass = all(. %
assert_any_breaker()
## Breakers report
## Tidy data validation report:
## # A tibble: 1 x 5
## pack rule var id value
## ## 1 low_mass is_small_mass mass 0 FALSE
## Error: assert_any_breaker: Some breakers found in exposure.
assert_any_breaker() is used to assert presence of at least one breaker in validation report.
However, offered solution via column pack doesn’t show rows which break the rule. To do that one can use cell pack:
starwars %__%
expose(
cell_packs(
low_mass = . %__% transmute_at(
vars(mass = &quot;mass&quot;),
rules(is_small_mass = (. %
assert_any_breaker()
## Breakers report
## Tidy data validation report:
## # A tibble: 1 x 5
## pack rule var id value
## ## 1 low_mass is_small_mass mass 16 FALSE
## Error: assert_any_breaker: Some breakers found in exposure.
■ Remove numeric columns with mean value below certain threshold. To achieve that one should formulate rule as “column mean should be above threshold”, identify breakers and act upon this information.
remove_bad_cols %
pull(var) %__%
unique()
.tbl[, setdiff(colnames(.tbl), bad_cols)]
}
starwars %__%
expose(
col_packs(
. %__% summarise_if(is.numeric, rules(mean(., na.rm = TRUE) __= 100))
)
) %__%
act_after_exposure(
.trigger = any_breaker,
.actor = remove_bad_cols
) %__%
remove_exposure()
## # A tibble: 87 x 11
## name height hair_color skin_color eye_color gender homeworld
## ## 1 Luke Skywalker 172 blond fair blue male Tatooine
## 2 C-3PO 167 gold yellow Tatooine
## 3 R2-D2 96 white, blue red Naboo
## 4 Darth Vader 202 none white yellow male Tatooine
## 5 Leia Organa 150 brown light brown female Alderaan
## # ... with 82 more rows, and 4 more variables: species ,
## # films , vehicles , starships act_after_exposure is a wrapper for performing actions after exposing. It takes .trigger function to trigger action and .actor function to perform action and return its result.
any_breaker is a function which return TRUE if tidy validation report attached to it has any breaker and FALSE otherwise.
Conclusions
Yoda and Jabba the Hutt are outliers among other “Star Wars” characters: Yoda is by height and birth year, Jabba is by mass and also birth year.
There are less than 10 “Star Wars” films yet.
ruler offers flexible and extendable functionality for common validation tasks. Validation can be done for data [as a whole], group of rows [as a whole], column [as a whole], row [as a whole] and cell. After exposing data frame of interest to rules and obtaining tidy validation report, one can perform any action based on this information: explore report, throw error, impute input data frame, etc.