This is going to be the type of blog posts that would perhaps be better as a gist, but it is easier for me to use my blog as my own personal collection of gists. Plus, someone else might find this useful, so here it is! In this blog post I am going to show a little trick to randomly sample rows from a text file using bash, and then train a model using the {h2o} package. I will also use the {rsparkling} package. From {rsparkling}’s documentation: {rsparkling} is a package that provides an R interface to the H2O Sparkling Water machine learning library. and will be needed to transfer the data from Spark to H2O.
In a previous blog post I used the {sparklyr} package to load a 30GB csv file into R. I created the file by combining around 300 csv files, each around 80MB big. Here, I would like to use the machine learning functions included in the {h2o} packages to train a random forest on this data. However, I only want to have a simple prototype that simply runs, and check if all the packages work well together. If everything is ok, I’ll keep iterating to make the model better (in a possible subsequent post).
For fast prototyping, using 30GB of data is not a good idea, so I am going to sample 500000 from this file using the linux command line (works on macOS too and also on Windows if you installed the linux subsystem). Why not use R to sample 500000 rows? Because on my machine, loading the 30GB file takes 25 minutes. Sampling half a million lines from it would take quite long too. So here are some bash lines that do that directly on the file, without needing to load it into R beforehand:
[18-03-03 21:50] brodriguesco in /Documents/AirOnTimeCSV ➤ get_seeded_random()
{
seed=&quot;$1&quot;
openssl enc -aes-256-ctr -pass pass:&quot;$seed&quot; -nosalt \
/dev/null
}
[18-03-03 21:50] brodriguesco in /Documents/AirOnTimeCSV ➤ sed &quot;1 d&quot; combined.csv | shuf --random-source= small_combined_temp.csv
[18-03-03 21:56] brodriguesco in /Documents/AirOnTimeCSV ➤ head -1 combined.csv __ colnames.csv
[18-03-03 21:56] brodriguesco in /Documents/AirOnTimeCSV ➤ cat colnames.csv small_combined_temp.csv __ small_combined.csv
The first function I took from the gnu coreutils manual which allows me to fix the random seed to reproduce the same sampling of the file. Then I use &quot;sed 1 d&quot; cobmined.csv to remove the first line of combined.csv which is the header of the file. Then, I pipe the result of sed using | to shuf which does the shuffling. The option --random-source= small_combined_temp.csv, saves the result to small_cobmined_temp.csv. Because I need to add back the header, I use head -1 to extract the first line of combined.csv and save it into colnames.csv. Finally, I bind the rows of both files using cat colnames.csv small_combined_temp.csv and save the result into small_combined.cvs. Taken together, all these steps took about 5 minutes (without counting the googling around for finding how to pass a fixed seed to shuf).
Now that I have this small dataset, I can write a small prototype:
First, you need to install {sparklyr}, {rsparkling} and {h2o}. Refer to this to know how to install the packages. I had a mismatch between the version of H2O that was automatically installed when I installed the {h2o} package, and the version of Spark that {sparklyr} installed but thankfully the {h2o} package returns a very helpful error message with the following lines:
detach(&quot;package:rsparkling&quot;, unload = TRUE)
if (&quot;package:h2o&quot; %in% search()) { detach(&quot;package:h2o&quot;, unload = TRUE) }
if (isNamespaceLoaded(&quot;h2o&quot;)){ unloadNamespace(&quot;h2o&quot;) }
remove.packages(&quot;h2o&quot;)
install.packages(&quot;h2o&quot;, type = &quot;source&quot;, repos = &quot;https://h2o-release.s3.amazonaws.com/h2o/rel-weierstrass/2/R&quot;)
which tells you which version to install.
So now, let’s load everything:
library(sparklyr)
library(rsparkling)
library(h2o)
## ## ----------------------------------------------------------------------
## ## Your next step is to start H2O:
## __ h2o.init()
## ## For H2O package documentation, ask for help:
## __ ??h2o
## ## After starting H2O, you can use the Web UI at http://localhost:54321
## For more information visit http://docs.h2o.ai
## ## ----------------------------------------------------------------------
## ## Attaching package: 'h2o'
## The following objects are masked from 'package:stats':
## ## cor, sd, var
## The following objects are masked from 'package:base':
## ## &amp;&amp;, %*%, %in%, ||, apply, as.factor, as.numeric, colnames,
## colnames