Here’s the problem: I have some data with nested time series. Lots of them. It’s
like there’s many, many little datasets inside my data. There are too many
groups to plot all of the time series at once, so I just want to preview a
handful of them.
For a working example, suppose we want to visualize the top 50 American female
baby names over time. I start by adding up the total number of births for each
name, finding the overall top 50 most populous names, and then keeping just the
time series from those top names.
library(ggplot2)
library(dplyr, warn.conflicts = FALSE)
babynames % filter(sex == &quot;F&quot;)
top50 % group_by(name) %__% summarise(total = sum(n)) %__% top_n(50, total) # keep just rows in babynames that match a row in top50
top_names %
semi_join(top50, by = &quot;name&quot;)
Hmm, so what does this look like?
ggplot(top_names) + aes(x = year, y = n) + geom_line() + facet_wrap(&quot;name&quot;)
Aaack, I can’t read anything! Can’t I just see a few of them?
This is a problem I face frequently, so frequently that I wrote a helper
function to handle this problem: sample_n_of(). This is not a very clever name, but it works. Below I call the function from my personal R package
and plot just the data from four names.
# For reproducible blogging
set.seed(20180524)
top_names %__% tjmisc::sample_n_of(4, name) %__% ggplot() + aes(x = year, y = n) + geom_line() + facet_wrap(&quot;name&quot;)
In this post, I walk through how this function works. It’s not very
complicated: It relies on some light tidy evaluation plus one obscure dplyr
function.
Working through the function
As usual, let’s start by sketching out the function we want to write:
sample_n_of #__ 1 1880 F Mary 7065 0.0724 33
#__ 2 1880 F Anna 2604 0.0267 4
#__ 3 1880 F Emma 2003 0.0205 19
#__ 4 1880 F Elizabeth 1939 0.0199 17
#__ 5 1880 F Margaret 1578 0.0162 32
#__ 6 1880 F Sarah 1288 0.0132 45
#__ 7 1880 F Laura 1012 0.0104 29
#__ 8 1880 F Catherine 688 0.00705 11
#__ 9 1880 F Helen 636 0.00652 21
#__ 10 1880 F Frances 605 0.00620 20
#__ # ... with 6,397 more rows
We can randomly sample five of the group indices and keep the rows for just
those groups.
unique_groups # A tibble: 674 x 5
#__ year sex name n prop
#__ #__ 1 1880 F Anna 2604 0.0267 #__ 2 1880 F Helen 636 0.00652 #__ 3 1880 F Frances 605 0.00620 #__ 4 1880 F Samantha 21 0.000215 #__ 5 1881 F Anna 2698 0.0273 #__ 6 1881 F Helen 612 0.00619 #__ 7 1881 F Frances 586 0.00593 #__ 8 1881 F Samantha 12 0.000121 #__ 9 1881 F Karen 6 0.0000607
#__ 10 1882 F Anna 3143 0.0272 #__ # ... with 664 more rows
# Confirm that only five names are in the dataset
subset_of_the_data %__% distinct(name)
#__ # A tibble: 5 x 1
#__ name #__ #__ 1 Anna #__ 2 Helen #__ 3 Frances #__ 4 Samantha
#__ 5 Karen
Putting these steps together, we get:
sample_n_of % print()
#__ # A tibble: 1,326 x 5
#__ year sex name n prop
#__ #__ 1 1880 F Sarah 1288 0.0132 #__ 2 1880 F Frances 605 0.00620 #__ 3 1880 F Rachel 166 0.00170 #__ 4 1880 F Samantha 21 0.000215 #__ 5 1880 F Deborah 12 0.000123 #__ 6 1880 F Shirley 8 0.0000820
#__ 7 1880 F Carol 7 0.0000717
#__ 8 1880 F Jessica 7 0.0000717
#__ 9 1881 F Sarah 1226 0.0124 #__ 10 1881 F Frances 586 0.00593 #__ # ... with 1,316 more rows
ten_names %__% distinct(name)
#__ # A tibble: 10 x 1
#__ name #__ #__ 1 Sarah #__ 2 Frances #__ 3 Rachel #__ 4 Samantha
#__ 5 Deborah #__ 6 Shirley #__ 7 Carol #__ 8 Jessica #__ 9 Patricia
#__ 10 Sharon
We can sample based on multiple columns too. Ten combinations of names and years
should return just ten rows.
top_names %__% sample_n_of(10, name, year) #__ # A tibble: 10 x 5
#__ year sex name n prop
#__ #__ 1 1907 F Jessica 17 0.0000504
#__ 2 1932 F Catherine 5446 0.00492 #__ 3 1951 F Nicole 94 0.0000509
#__ 4 1953 F Janet 17761 0.00921 #__ 5 1970 F Sharon 9174 0.00501 #__ 6 1983 F Melissa 23473 0.0131 #__ 7 1989 F Brenda 2270 0.00114 #__ 8 1989 F Pamela 1334 0.000670 #__ 9 1994 F Samantha 22817 0.0117 #__ 10 2014 F Kimberly 2891 0.00148
Next steps
There are a few tweaks we could make to this function. For example, in my
package’s version, I warn the user when the number of groups is too large.
too_many % tjmisc::sample_n_of(100, name)
#__ Warning: Sample size (100) is larger than number of groups (50). Using size
#__ = 50.
My version also randomly samples n of the rows when there are no grouping
variables provided.
top_names %__% tjmisc::sample_n_of(2)
#__ # A tibble: 2 x 5
#__ year sex name n prop
#__ #__ 1 1934 F Stephanie 128 0.000118
#__ 2 2007 F Mary 3674 0.00174
One open question is how to handle data that’s already grouped. The function we
wrote above fails.
top_names %__% group_by(name) %__% sample_n_of(2, year)
#__ Error in filter_impl(.data, quo): Result must have length 136, not 6407
Is this a problem?
Here I think failure is okay because what do we think should happen? It’s not
obvious. It should randomly choose 2 of the years for each name.
Should it be the same two years? Then this should be fine.
top_names %__% sample_n_of(2, year)
#__ # A tibble: 100 x 5
#__ year sex name n prop
#__ #__ 1 1970 F Jennifer 46160 0.0252 #__ 2 1970 F Lisa 38965 0.0213 #__ 3 1970 F Kimberly 34141 0.0186 #__ 4 1970 F Michelle 34053 0.0186 #__ 5 1970 F Amy 25212 0.0138 #__ 6 1970 F Angela 24926 0.0136 #__ 7 1970 F Melissa 23742 0.0130 #__ 8 1970 F Mary 19204 0.0105 #__ 9 1970 F Karen 16701 0.00912
#__ 10 1970 F Laura 16497 0.00901
#__ # ... with 90 more rows
Or, should those two years be randomly selected for each name? Then, we should
let do() handle that. do() takes some code that returns a dataframe, applies
it to each group, and returns the combined result.
top_names %__% group_by(name) %__% do(sample_n_of(., 2, year))
#__ # A tibble: 100 x 5
#__ # Groups: name [50]
#__ year sex name n prop
#__ #__ 1 1913 F Amanda 346 0.000528 #__ 2 1953 F Amanda 428 0.000222 #__ 3 1899 F Amy 281 0.00114 #__ 4 1964 F Amy 9579 0.00489 #__ 5 1916 F Angela 715 0.000659 #__ 6 2005 F Angela 2893 0.00143 #__ 7 1999 F Anna 9092 0.00467 #__ 8 2011 F Anna 5649 0.00292 #__ 9 1952 F Ashley 24 0.0000126
#__ 10 2006 F Ashley 12340 0.00591 #__ # ... with 90 more rows
I think raising an error and forcing the user to clarify their code is a better
than choosing one of these options and not doing what the user expects.