<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/benchmarking-a-ssd-drive-in-reading-and-writing-files-with-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://msperlin.github.io/2018-06-29-BenchmarkingSSD/"> Marcelo S. Perlin</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p> – </p>
<p>I recently bought a new computer for home and it came with two drives,<br>
one HDD and other SSD. The later is used for the OS and the former for<br>
all of my files. From all computers I had, both home and work, this is<br>
definitely the fastest. While some of the merits are due to the newer<br>
CPUS and RAM, the SSD drive can make all the difference in file<br>
operations.</p>
<p>My research usually deals with large files from financial markets. Being<br>
efficient in reading those files is key to my productivity. Given that,<br>
I was very curious in understanding how much I would benefit in speed<br>
when reading/writing files in my SSD drive instead of the HDD. For that,<br>
I wrote a simple function that will time a particular operation. The<br>
function will take as input the number of rows in the data (1..Inf), the<br>
type of function used to save the file (<em>rds</em>, <em>csv</em>, <em>fst</em>) and the<br>
type of drive (<em>HDD</em> or <em>SSD</em>). See next.</p>
<div>
<div>
<pre><code>bench.fct &lt;- function(N = 2500000, type.file = 'rds', type.hd = 'HDD') {
  # Function for timing read and write operations
  #
  # INPUT: N - Number of rows in dataframe to be read and write
  #        type.file - format of output file (rds, csv, fst)
  #        type.hd - where to save (hdd or ssd)
  #
  # OUTPUT: A dataframe with results
  require(tidyverse)
  require(fst)
  
  my.df &lt;- data_frame(x = runif(N),
                      char.vec = sample(letters, size = N, 
                                        replace = TRUE))
  
  path.file &lt;- switch(type.hd,
                      'SSD' = '~',
                      'HDD' = '/mnt/HDD/')
  
  my.file &lt;- file.path(path.file, 
                       switch (type.file,
                               'rds-base' = 'temp_rds.rds',
                               'rds-readr' = 'temp_rds.rds',
                               'fst' = 'temp_fst.fst',
                               'csv-readr' = 'temp_csv.csv',
                               'csv-base' = 'temp_csv.csv'))
  
  if (type.file == 'rds-base') {
    time.write &lt;- system.time(saveRDS(my.df, my.file, compress = FALSE))
    time.read &lt;- system.time(readRDS(my.file))
  } else if (type.file == 'rds-readr') {
    time.write &lt;- system.time(write_rds(x = my.df, path =  my.file, compress = 'none'))
    time.read &lt;- system.time(read_rds(path = my.file ))
  } else if (type.file == 'fst') {
    time.write &lt;- system.time(write.fst(x = my.df, path = my.file))
    time.read &lt;- system.time(read_fst(my.file))
  } else if (type.file == 'csv-readr') {
    time.write &lt;- system.time(write_csv(x = my.df, path = my.file))
    time.read &lt;- system.time(read_csv(file = my.file, col_types = cols(x = col_double(),
                                                                       char.vec = col_character())))
  } else if (type.file == 'csv-base') {
    time.write &lt;- system.time(write.csv(x = my.df, file = my.file))
    time.read &lt;- system.time(read.csv(file = my.file))
  }
  
  # clean up
  file.remove(my.file)
  
  # save output
  df.out &lt;- data_frame(type.file = type.file,
                       type.hd = type.hd,
                       N = N,
                       type.time = c('write', 
                                     'read'),
                       times = c(time.write[3], 
                                 time.read[3]))
  
  return(df.out)
  
}
</code></pre>
</div>
</div>
<p>Now that we have my function, its time to use it for all combinations<br>
between number of rows, the formats of the file and type of drive:</p>
<div>
<div>
<pre><code>library(purrr)
df.grid &lt;- expand.grid(N = seq(1, 500000, by = 50000), 
                       type.file = c('rds-readr', 'rds-base', 'fst', 'csv-readr', 'csv-base'), 
                       type.hd = c('HDD', 'SSD'), stringsAsFactors = F)

l.out &lt;- pmap(list(N = df.grid$N,
               type.file = df.grid$type.file,
               type.hd = df.grid$type.hd), .f = bench.fct)

df.res &lt;- do.call(what = bind_rows, args = l.out)
</code></pre>
</div>
</div>
<p>Lets check the result in a nice plot:</p>
<div>
<div>
<pre><code>library(ggplot2)

p &lt;- ggplot(df.res, aes(x = N, y = times, linetype = type.hd)) + 
  geom_line() + facet_grid(type.file ~ type.time)

print(p)
</code></pre>
</div>
</div>
<p><img alt="" src="https://i0.wp.com/msperlin.github.io/img/2018-06-29-BenchmarkingSSD_files/figure-markdown_strict/unnamed-chunk-3-1.png?w=456" data-recalc-dims="1"></p>
<p>As you can see, the <code>csv-base</code> format is messing with the y axis. Let’s<br>
remove it for better visualization:</p>
<div>
<div>
<pre><code>library(ggplot2)

p &lt;- ggplot(filter(df.res, !(type.file %in% c('csv-base'))),
            aes(x = N, y = times, linetype = type.hd)) + 
  geom_line() + facet_grid(type.file ~ type.time)

print(p)
</code></pre>
</div>
</div>
<p><img alt="" src="https://i1.wp.com/msperlin.github.io/img/2018-06-29-BenchmarkingSSD_files/figure-markdown_strict/unnamed-chunk-4-1.png?w=456" data-recalc-dims="1"></p>
<p>When it comes to the file format, we learn:</p>
<ul>
<li>
<p><strong>By far, the <code>fst</code> format is the best</strong>. It takes less time to read<br>
and write than the others. However, it’s probably unfair to compare<br>
it to <code>csv</code> and <code>rds</code> as it uses many of the 16 cores of my<br>
computer.</p>
</li>
<li>
<p><strong><code>readr</code> is a great package for writing and reading <em>csv</em> files</strong>.<br>
You can see a large difference of time from using the <code>base</code><br>
functions. This is likely due to the use of low level functions to<br>
write and read the text files.</p>
</li>
<li>
<p><strong>When using the <em>rds</em> format, the base function do not differ much<br>
from the <code>readr</code> functions</strong>.</p>
</li>
</ul>
<p>As for the effect of using SSD, its clear that it <strong>DOES NOT</strong> effect<br>
the time of reading and writing. The differences between using HDD and<br>
SSD looks like noise. Seeking to provide a more robust analysis, let’s<br>
formally test this hypothesis using a simple t-test for the means:</p>
<div>
<div>
<pre><code>tab &lt;- df.res %>%
  group_by(type.file, type.time) %>%
  summarise(mean.HDD = mean(times[type.hd == 'HDD']),
            mean.SSD = mean(times[type.hd == 'SSD']),
            p.value = t.test(times[type.hd == 'SSD'],
                             times[type.hd == 'HDD'])$p.value)


print(tab)

## # A tibble: 10 x 5
## # Groups:   type.file [?]
##    type.file type.time mean.HDD mean.SSD p.value
##    </code></pre></div></div>