I used reading a directory of CSVs as the foundational example in my recent post on idioms.
During my exchange with Matt, Hadley and a few others — in the crazy Twitter thread that spawned said post — I mentioned that I’d personally “just use Drill”.
I’ll use this post as a bit of a teaser trailer for the actual post (or, more likely, series of posts) that goes into detail on where to get Apache Drill, basic setup of Drill for standalone workstation use and then organizing data with it.
You can get ahead of those posts by doing two things:
- Download, install and test your Apache Drill setup (it’s literally 10 minutes on any platform)
- Review the U.S. EPA annual air quality data archive (they have individual, annual CSVs that are perfect for the example)
My goals for this post are really to just to pique your interest enough in Drill and parquet files (yes, I’m ultimately trying to socially engineer you into using parquet files) to convince you to read the future post(s) and show that it’s worth your time to do Step #1 above.
Getting EPA Air Quality Data
The EPA has air quality data going back to 1990 (so, 27 files as of this post). They’re ~1-4MB ZIP compressed and ~10-30MB uncompressed.
You can use the following code to grab them all with the caveat that the libcurl method of performing simultaneous downloads caused some pretty severe issues — like R crashing — for some of my students who use Windows. There are plenty of examples for doing sequential downloads of a list of URLs out there that folks should be able to get all the files even if this succinct method does not work on your platform.
dir.create("airq")
urls <- sprintf("https://aqsdr1.epa.gov/aqsweb/aqstmp/airdata/annual_all_%d.zip", 1990L:2016L)
fils <- sprintf("airq/%s", basename(urls))
download.file(urls, fils, method = "libcurl")
I normally shy away from this particular method since it really hammers the remote server, but this is a beefy U.S. government server, the files are relatively small in number and size and I’ve got a super-fast internet connection (no long-lived sockets) so it should be fine.
Putting all those files under the “control” of Drill is what the next post is for. For now, i’m going to show the basic code and benchmarks for reading in all those files and performing a basic query for all the distinct years. Yes, we know that information already, but it’s a nice, compact task that’s easy to walk through and illustrates the file reading and querying in all three idioms: Drill, tidyverse and data.table.
Data Setup
I’ve converted the EPA annual ZIP files into bzip2 format. ZIP is fine for storage and downloads but it’s not a great format for data analysis tasks. gzip would be slightly faster but it’s not easily splittable and — even though I’m not using the data in a Hadoop context — I think it’s wiser to not have to re-process data later on if I ever had to move raw CSV or JSON data into Hadoop. Uncompressed CSVs are the most portable, but there’s no need to waste space.
All the following files are in a regular filesystem directory accessible to both Drill and R:
> (epa_annual_fils <- dir("~/Data/csv/epa/annual", "*.csv.bz2"))
[1] "annual_all_1990.csv.bz2" "annual_all_1991.csv.bz2" "annual_all_1992.csv.bz2"
[4] "annual_all_1993.csv.bz2" "annual_all_1994.csv.bz2" "annual_all_1995.csv.bz2"
[7] "annual_all_1996.csv.bz2" "annual_all_1997.csv.bz2" "annual_all_1998.csv.bz2"
[10] "annual_all_1999.csv.bz2" "annual_all_2000.csv.bz2" "annual_all_2001.csv.bz2"
[13] "annual_all_2002.csv.bz2" "annual_all_2003.csv.bz2" "annual_all_2004.csv.bz2"
[16] "annual_all_2005.csv.bz2" "annual_all_2006.csv.bz2" "annual_all_2007.csv.bz2"
[19] "annual_all_2008.csv.bz2" "annual_all_2009.csv.bz2" "annual_all_2010.csv.bz2"
[22] "annual_all_2011.csv.bz2" "annual_all_2012.csv.bz2" "annual_all_2013.csv.bz2"
[25] "annual_all_2014.csv.bz2" "annual_all_2015.csv.bz2" "annual_all_2016.csv.bz2"
Drill can directly read plain or compressed JSON, CSV and Apache web server log files plus can treat a directory tree of them as a single data source. It can also read parquet & avro files (both are used frequently in distributed “big data” setups) and access MySQL, MongoDB and other JDBC resources as well as query data stored in Amazon S3 and HDFS (I’ve already mentioned it works fine in plain ‘ol filesystems, too).
I’ve tweaked my Drill configuration to support reading column header info from .csv files (which I’ll show in the next post). In environments like Drill or even Spark, CSV columns are usually queried with some type of column index (e.g. COLUMN[0]) so having named columns makes for less verbose query code.
I turned those individual bzip2 files into parquet format with one Drill query:
CREATE TABLE dfs.pq.`/epa/annual.parquet` AS
SELECT * FROM dfs.csv.`/epa/annual/*.csv.bz2`
Future posts will explain the dfs... component but they are likely familiar path specifications for folks used to Spark and are pretty straightforward. The first bit (up to the back-tick) is an internal Drill shortcut to the actual storage path (which is a plain directory in this test) followed by the tail end path spec to the subdirectories and/or target files. That one statement said ‘take all the CSV files in that directory and make one big table out of them”.
The nice thing about parquet files is that they work much like R data frames in that they can be processed on the column level. We’ll see how that speeds up things in a bit.
Benchmark Setup
The tests were performed on a maxed out 2016 13″ MacBook Pro.
There are 55 columns of data in the EPA annual summary files.
To give both read_csv and fread some benchmark boosts, we’ll define the columns up-front and pass those in to each function on data ingestion and I’ll leave them out of this post for brevity (they’re just a cols() specification and colClasses vector). Drill gets no similar help for this at least when it comes to CSV processing.
I’m also disabling progress & verbose reporting in both fread and read_csv despite not stopping Drill from writing out log messages.
Now, we need some setup code to connect to drill and read in the list of files, plus we’ll setup the five benchmark functions to read in all the files and get the list of distinct years from each.
library(sergeant)
library(data.table)
library(tidyverse)
(epa_annual_fils <- dir("~/Data/csv/epa/annual", "*.csv.bz2", full.names = TRUE))
db <- src_drill("localhost")
# Remember, defining ct & ct_dt - the column types specifications - have been left out for brevity
mb_drill_csv <- function() {
epa_annual <- tbl(db, "dfs.csv.`/epa/annual/*.csv.bz2`")
select(epa_annual, Year) %>%
distinct(Year) %>%
collect()
}
mb_drill_parquet <- function() {
epa_annual_pq <- tbl(db, "dfs.pq.`/epa/annual.parquet`")
select(epa_annual_pq, Year) %>%
distinct(Year) %>%
collect()
}
mb_tidyverse <- function() {
map_df(epa_annual_fils, read_csv, col_types = ct, progress = FALSE) -> tmp
unique(tmp$Year)
}
mb_datatable <- function() {
rbindlist(
lapply(
epa_annual_fils, function(x) {
fread(sprintf("bzip2 -c -d %s", x),
colClasses = ct_dt, showProgress = FALSE,
verbose = FALSE) })) -> tmp
unique(tmp$Year)
}
mb_rda <- function() {
read_rds("~/Data/rds/epa/annual.rds") -> tmp
unique(tmp$Year)
}
microbenchmark(
csv = { mb_drill_csv() },
pq = { mb_drill_parquet() },
df = { mb_tidyverse() },
dt = { mb_datatable() },
rda = { mb_rda() },
times = 5
) -> mb
Yep, it’s really as simple as:
tbl(db, "dfs.csv.`/epa/annual/*.csv.bz2`")
to have Drill treat a directory tree as a single table. It’s also not necessary for all the columns to be in all the files (i.e. you get the bind_rows/map_df/rbindlist behaviour for “free”).
I’m only doing 5 evaluations here since I don’t want to make you wait if you’re going to try this at home now or after the Drill series. I’ve run it with a more robust benchmark configuration and the results are aligned with this one.
Unit: milliseconds
expr min lq mean median uq max neval
csv 15473.5576 16851.0985 18445.3905 19586.1893 20087.1620 20228.9450 5
pq 493.7779 513.3704 616.2634 550.5374 732.6553 790.9759 5
df 41666.1929 42361.1423 42701.2682 42661.9521 43110.3041 43706.7498 5
dt 37500.9351 40286.2837 41509.0078 42600.9916 43105.3040 44051.5247 5
rda 9466.6506 9551.7312 10012.8560 9562.9114 9881.8351 11601.1517 5
The R data route, which is the closest to the parquet route, is definitely better than slurping up CSVs all the time. Both parquet and R data files require pre-processing, so they’re not as flexible as having individual CSVs (that may get added hourly or daily to a directory).
Drill’s CSV slurping handily beats the other R methods even with some handicaps the others did not have.
This particular example is gamed a bit, which helped parquet to ultimately “win”. Since Drill can target the singular column (Year) that was asked for, it doesn’t need to read all the extra columns just to compute the final product (the distinct list of years).
IMO both the Drill CSV ingestion and Drill parquet access provide compelling enough use-cases to use them over the other three methods, especially since they are easily transferrable to remote Drill servers or clusters with virtually no code changes. A single node Drillbit (like R) is constrained by the memory on that individual system, so it’s not going to get you out of a memory jam, but it may make it easier to organize and streamline your core data operations before other analysis and visualization tasks.
FIN
I’m sure some member of some other tribe will come up with an example that proves superiority of their particular tribal computations. I’m hoping one of those tribes is the R/Spark tribe so that can get added into the mix (using Spark standalone is much like using Drill, but with more stats/ML functions directly available).
I’m hopeful that this post has showcased enough of Drill’s utility to general R users that you’ll give it a go and consider adding it to your R data analysis toolbox. It can be beneficial having both a precision tools as well as a Swiss Army knife — which is what Drill really is — handy.
You can find the sergeant package on GitHub.
