<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/fast-data-loading-from-files-to-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.appsilondatascience.com/rstats/2017/04/11/fast-data-load.html"> Appsilon Data Science - R language</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Recently we were building a Shiny App in which we had to load data from a very large dataframe. It was directly impacting the app initialization time, so we had to look into different ways of reading data from files to R (in our case customer provided <strong>csv</strong> files) and identify the best one.</p>
<p>The goal of my post is to compare:</p>
<ol>
<li><code>read.csv</code> from <code>utils</code>, which was the standard way of reading <strong>csv</strong> files to R in RStudio,</li>
<li><code>read_csv</code> from <code>readr</code> which replaced the former method as a standard way of doing it in RStudio,</li>
<li><code>load</code> and <code>readRDS</code> from <code>base</code>, and</li>
<li><code>read_feather</code> from <code>feather</code> and <code>fread</code> from <code>data.table</code>.</li>
</ol>
<h2>Data</h2>
<p>First let’s generate some random data</p>
<figure>
<pre><code data-lang="r"><span>set.seed</span><span>(</span><span>123</span><span>)</span><span>
</span><span>df</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>replicate</span><span>(</span><span>10</span><span>,</span><span> </span><span>sample</span><span>(</span><span>0</span><span>:</span><span>2000</span><span>,</span><span> </span><span>15</span><span> </span><span>*</span><span> </span><span>10</span><span>^</span><span>5</span><span>,</span><span> </span><span>rep</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)),</span><span>
</span><span>replicate</span><span>(</span><span>10</span><span>,</span><span> </span><span>stringi</span><span>::</span><span>stri_rand_strings</span><span>(</span><span>1000</span><span>,</span><span> </span><span>5</span><span>)))</span></code></pre>
</figure>
<p>and save the files on a disk to evaluate the loading time. Besides the <code>csv</code> format we will also need <code>feather</code>, <code>RDS</code> and <code>Rdata</code> files.</p>
<figure>
<pre><code data-lang="r"><span>path_csv</span><span> </span><span>&lt;-</span><span> </span><span>'../assets/data/fast_load/df.csv'</span><span>
</span><span>path_feather</span><span> </span><span>&lt;-</span><span> </span><span>'../assets/data/fast_load/df.feather'</span><span>
</span><span>path_rdata</span><span> </span><span>&lt;-</span><span> </span><span>'../assets/data/fast_load/df.RData'</span><span>
</span><span>path_rds</span><span> </span><span>&lt;-</span><span> </span><span>'../assets/data/fast_load/df.rds'</span></code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>feather</span><span>)</span><span>
</span><span>library</span><span>(</span><span>data.table</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>write.csv</span><span>(</span><span>df</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>path_csv</span><span>,</span><span> </span><span>row.names</span><span> </span><span>=</span><span> </span><span>F</span><span>)</span><span>
</span><span>write_feather</span><span>(</span><span>df</span><span>,</span><span> </span><span>path_feather</span><span>)</span><span>
</span><span>save</span><span>(</span><span>df</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>path_rdata</span><span>)</span><span>
</span><span>saveRDS</span><span>(</span><span>df</span><span>,</span><span> </span><span>path_rds</span><span>)</span></code></pre>
</figure>
<p>Next let’s check our files sizes:</p>
<figure>
<pre><code data-lang="r"><span>files</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>'../assets/data/fast_load/df.csv'</span><span>,</span><span> </span><span>'../assets/data/fast_load/df.feather'</span><span>,</span><span> </span><span>'../assets/data/fast_load/df.RData'</span><span>,</span><span> </span><span>'../assets/data/fast_load/df.rds'</span><span>)</span><span>
</span><span>info</span><span> </span><span>&lt;-</span><span> </span><span>file.info</span><span>(</span><span>files</span><span>)</span><span>
</span><span>info</span><span>$</span><span>size_mb</span><span> </span><span>&lt;-</span><span> </span><span>info</span><span>$</span><span>size</span><span>/</span><span>(</span><span>1024</span><span> </span><span>*</span><span> </span><span>1024</span><span>)</span><span> </span><span>print</span><span>(</span><span>subset</span><span>(</span><span>info</span><span>,</span><span> </span><span>select</span><span>=</span><span>c</span><span>(</span><span>&quot;size_mb&quot;</span><span>)))</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##                                       size_mb
## ../assets/data/fast_load/df.csv     1780.3005
## ../assets/data/fast_load/df.feather 1145.2881
## ../assets/data/fast_load/df.RData    285.4836
## ../assets/data/fast_load/df.rds      285.4837</code></pre>
</figure>
<p>As we can see both <code>csv</code> and <code>feather</code> format files are taking much more storage space. <code>Csv</code> more than 6 times and <code>feather</code> more than 4 times comparing to <code>RDS</code> and <code>RData</code>.</p>
<h2>Benchmark</h2>
<p>We will use <code>microbenchmark</code> library to compare the reading times of the following methods:</p>
<ul>
<li>utils::read.csv</li>
<li>readr::read_csv</li>
<li>data.table::fread</li>
<li>base::load</li>
<li>base::readRDS</li>
<li>feather::read_feather</li>
</ul>
<p>in 10 rounds.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>microbenchmark</span><span>)</span><span>
</span><span>benchmark</span><span> </span><span>&lt;-</span><span> </span><span>microbenchmark</span><span>(</span><span>readCSV</span><span> </span><span>=</span><span> </span><span>utils</span><span>::</span><span>read.csv</span><span>(</span><span>path_csv</span><span>),</span><span>
</span><span>readrCSV</span><span> </span><span>=</span><span> </span><span>readr</span><span>::</span><span>read_csv</span><span>(</span><span>path_csv</span><span>,</span><span> </span><span>progress</span><span> </span><span>=</span><span> </span><span>F</span><span>),</span><span>
</span><span>fread</span><span> </span><span>=</span><span> </span><span>data.table</span><span>::</span><span>fread</span><span>(</span><span>path_csv</span><span>,</span><span> </span><span>showProgress</span><span> </span><span>=</span><span> </span><span>F</span><span>),</span><span>
</span><span>loadRdata</span><span> </span><span>=</span><span> </span><span>base</span><span>::</span><span>load</span><span>(</span><span>path_rdata</span><span>),</span><span>
</span><span>readRds</span><span> </span><span>=</span><span> </span><span>base</span><span>::</span><span>readRDS</span><span>(</span><span>path_rds</span><span>),</span><span>
</span><span>readFeather</span><span> </span><span>=</span><span> </span><span>feather</span><span>::</span><span>read_feather</span><span>(</span><span>path_feather</span><span>),</span><span> </span><span>times</span><span> </span><span>=</span><span> </span><span>10</span><span>)</span><span>
</span><span>print</span><span>(</span><span>benchmark</span><span>,</span><span> </span><span>signif</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##Unit: seconds
##        expr   min    lq       mean median    uq   max neval
##     readCSV 200.0 200.0 211.187125  210.0 220.0 240.0    10
##    readrCSV  27.0  28.0  29.770890   29.0  32.0  33.0    10
##       fread  15.0  16.0  17.250016   17.0  17.0  22.0    10
##   loadRdata   4.4   4.7   5.018918    4.8   5.5   5.9    10
##     readRds   4.6   4.7   5.053674    5.1   5.3   5.6    10
## readFeather   1.5   1.8   2.988021    3.4   3.6   4.1    10</code></pre>
</figure>
<p>And the <strong>winner</strong> is… <code>feather</code>! However, using <code>feather</code> requires prior conversion of the file to the feather format.<br>
Using <code>load</code> or <code>readRDS</code> can improve performance (second and third place in terms of speed) and has a benefit of storing smaller/compressed file. In both cases you will have to convert your file to the proper format first.</p>
<p>When it comes to reading from <code>csv</code> format <code>fread</code> significantly beats <code>read_csv</code> and <code>read.csv</code>, and thus is the best option to read a <code>csv</code> file.</p>
<p>In our case we decided to go with <code>feather</code> file since conversion from <code>csv</code> to this format is just a one time job and we didn’t have a strict limitation on a storage space to consider usage of <code>Rds</code> or <code>RData</code> format.</p>
<p>The final workflow was:</p>
<ol>
<li>reading a <code>csv</code> file provided by our customer using <code>fread</code>,</li>
<li>writing it to <code>feather</code> using <code>write_feather</code>, and</li>
<li>loading a <code>feather</code> file on app initialization using <code>read_feather</code>.</li>
</ol>
<p>First two tasks were done once and outside of a Shiny App context.</p>
<p>There is also quite interesting benchmark done by Hadley <a target="_blank" rel="nofollow" href="https://gist.github.com/hadley/6353939">here</a> on reading complete files to R. Unfortunately, if you use functions defined in that post, you will end up with an character type object, and you will have to apply string manipulations to obtain a commonly and widely used <strong>dataframe</strong>.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.appsilondatascience.com/rstats/2017/04/11/fast-data-load.html"> Appsilon Data Science - R language</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/qV6u8nv_12Y"></a></div>