<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/reading-and-analysing-log-files-in-the-rrd-database-format/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rviews.rstudio.com/2018/06/20/reading-rrd-files/"> R Views</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I have frequent conversations with R champions and Systems Administrators responsible for R, in which they ask how they can measure and analyze the usage of their servers. Among the many solutions to this problem, one of the my favourites is to use an <strong>RRD</strong> database and <strong>RRDtool</strong>.</p>
<p>From <a rel="nofollow" href="https://en.wikipedia.org/wiki/RRDtool" target="_blank">Wikipedia</a>:</p>
<blockquote>
<p><strong>RRDtool</strong> (<em>round-robin database tool</em>) aims to handle time series data such as network bandwidth, temperatures or CPU load. The data is stored in a <a rel="nofollow" href="https://en.wikipedia.org/wiki/Circular_buffer" target="_blank">circular buffer</a> based <a rel="nofollow" href="https://en.wikipedia.org/wiki/Database" target="_blank">database</a>, thus the system storage footprint remains constant over time.</p>
</blockquote>
<p><a rel="nofollow" href="https://oss.oetiker.ch/rrdtool/index.en.html" target="_blank">RRDtool</a> is a library written in C, with implementations that can also be accessed from the Linux command line. This makes it convenient for system development, but makes it difficult for R users to extract and analyze this data.</p>
<p>I am pleased to announce that I’ve been working on the <code>rrd</code> <a rel="nofollow" href="https://github.com/andrie/rrd" target="_blank">R package</a> to import RRD files directly into <code>tibble</code> objects, thus making it easy to analyze your metrics.</p>
<p>As an aside, the RStudio Pro products (specifically <a rel="nofollow" href="https://www.rstudio.com/products/rstudio-server-pro/" target="_blank">RStudio Server Pro</a> and <a rel="nofollow" href="https://www.rstudio.com/products/connect/" target="_blank">RStudio Connect</a>) also make use of RRD to store metrics – more about this later.</p>
<h2>Understanding the RRD format as an R user</h2>
<p>The name RRD is an initialism of <strong>R</strong>ound <strong>R</strong>obin <strong>D</strong>atabase. The “round robin” refers to the fact that the database is always fixed in size, and as a new entry enters the database, the oldest entry is discarded. In practical terms, the database collects data for a fixed period of time, and information that is older than the threshold gets removed.</p>
<p><img alt="Image from loriotpro(https://bit.ly/2tk2MFa)" src="https://i2.wp.com/rviews.rstudio.com/post/2018-06-21-analysing-rrd-files_files/rra.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>A second quality of RRD databases is that each datum is stored in different “consolidation data points”, where every data point is an aggregation over time. For example, a data point can represent an average value for the time period, or a maximum over the period. Typical consolidation functions include <code>average</code>, <code>min</code> and <code>max</code>.</p>
<p>The third quality is that every RRD database file typically consists of multiple archives. Each archive measures data for a different time period. For instance, the archives can capture data for intervals of 10 seconds, 30 seconds, 1 minute or 5 minutes.</p>
<p>As an example, here is a description of an RRD file that originated in RStudio Connect:</p>
<pre><code>describe_rrd(&quot;rrd_cpu_0&quot;)
#> A RRD file with 10 RRA arrays and step size 60
#> [1] AVERAGE_60 (43200 rows)
#> [2] AVERAGE_300 (25920 rows)
#> [3] MIN_300 (25920 rows)
#> [4] MAX_300 (25920 rows)
#> [5] AVERAGE_3600 (8760 rows)
#> [6] MIN_3600 (8760 rows)
#> [7] MAX_3600 (8760 rows)
#> [8] AVERAGE_86400 (1825 rows)
#> [9] MIN_86400 (1825 rows)
#> [10] MAX_86400 (1825 rows)
</code></pre>
<p>This <code>RRD</code> file contains data for the properties of CPU 0 of the system. In this example, the first <code>RRA</code> archive contains averaged metrics for one minute (60s) intervals, while the second <code>RRA</code> measures the same metric, but averaged over five minutes. The same metrics are also available for intervals of one hour and one day.</p>
<p>Notice also that every archive has a different number of rows, representing a different historical period where the data is kept. For example, the <em>per minute</em> data <code>AVERAGE_60</code> is retained for 43,200 periods (12 days) while the <em>daily</em> data <code>MAX_86400</code> is retained for 1,825 periods (5 years).</p>
<p>If you want to know more, please read the excellent <a rel="nofollow" href="https://oss.oetiker.ch/rrdtool/tut/rrdtutorial.en.html" target="_blank">introduction tutorial</a> to RRD database.</p>
<h2>Introducing the <code>rrd</code> package</h2>
<p>Until recently, it wasn’t easy to import RRD files into R. But I was pleased to discover that a <a rel="nofollow" href="https://www.google-melange.com/archive/gsoc/2014" target="_blank">Google Summer of Code 2014</a> project created a proof-of-concept R package to read these files. The author of this package is <a rel="nofollow" href="http://plamendimitrov.net/" target="_blank">Plamen Dimitrov</a>, who published the code on <a rel="nofollow" href="https://github.com/pldimitrov/Rrd" target="_blank">GitHub</a> and also wrote an <a rel="nofollow" href="http://plamendimitrov.net/blog/2014/08/09/r-package-for-working-with-rrd-files/" target="_blank">explanatory blog post</a>.</p>
<p>Because I had to provide some suggestions to our customers, I decided to update the package, provide some example code, and generally improve the reliability.</p>
<p>The result is not yet on CRAN, but you can install the development version of package from <a rel="nofollow" href="https://github.com/andrie/rrd" target="_blank">github</a>.</p>
<h3>Installing the package</h3>
<p>To build the package from source, you first need to install <a rel="nofollow" href="http://oss.oetiker.ch/rrdtool/doc/librrd.en.html" target="_blank">librrd</a>. Installing <a rel="nofollow" href="http://oss.oetiker.ch/rrdtool/" target="_blank">RRDtool</a> from your Linux package manager will usually also install this library.</p>
<p>Using Ubuntu:</p>
<pre><code>sudo apt-get install rrdtool librrd-dev
</code></pre>
<p>Using RHEL / CentOS:</p>
<pre><code>sudo yum install rrdtool rrdtool-devel
</code></pre>
<p>Once you have the system requirements in place, you can install the development version of the R package from <a rel="nofollow" href="https://github.com/andrie/rrd" target="_blank">GitHub</a> using:</p>
<pre><code># install.packages(&quot;devtools&quot;)
devtools::install_github(&quot;andrie/rrd&quot;)
</code></pre>
<h3>Limitations</h3>
<p>The package is not yet available for Windows.</p>
<h3>Using the package</h3>
<p>Once you’ve installed the package, you can start to use it. The package itself contains some built-in RRD files, so you should be able to run the following code directly.</p>
<pre><code>library(rrd)
</code></pre>
<h4>Describing the contents of a RRD</h4>
<p>To describe the contents of an RRD file, use <code>describe_rrd()</code>. This function reports information about the names of each archive (RRA) file, the consolidation function, and the number of observations:</p>
<pre><code>rrd_cpu_0 &lt;- system.file(&quot;extdata/cpu-0.rrd&quot;, package = &quot;rrd&quot;)

describe_rrd(rrd_cpu_0)
#> A RRD file with 10 RRA arrays and step size 60
#> [1] AVERAGE_60 (43200 rows)
#> [2] AVERAGE_300 (25920 rows)
#> [3] MIN_300 (25920 rows)
#> [4] MAX_300 (25920 rows)
#> [5] AVERAGE_3600 (8760 rows)
#> [6] MIN_3600 (8760 rows)
#> [7] MAX_3600 (8760 rows)
#> [8] AVERAGE_86400 (1825 rows)
#> [9] MIN_86400 (1825 rows)
#> [10] MAX_86400 (1825 rows)
</code></pre>
<h4>Reading an entire RRD file</h4>
<p>To read an entire RRD file, i.e. all of the RRA archives, use <code>read_rrd()</code>. This returns a list of <code>tibble</code> objects:</p>
<pre><code>cpu &lt;- read_rrd(rrd_cpu_0)

str(cpu, max.level = 1)
#> List of 10
#>  $ AVERAGE60   :Classes 'tbl_df', 'tbl' and 'data.frame':    43199 obs. of  9 variables:
#>  $ AVERAGE300  :Classes 'tbl_df', 'tbl' and 'data.frame':    25919 obs. of  9 variables:
#>  $ MIN300      :Classes 'tbl_df', 'tbl' and 'data.frame':    25919 obs. of  9 variables:
#>  $ MAX300      :Classes 'tbl_df', 'tbl' and 'data.frame':    25919 obs. of  9 variables:
#>  $ AVERAGE3600 :Classes 'tbl_df', 'tbl' and 'data.frame':    8759 obs. of  9 variables:
#>  $ MIN3600     :Classes 'tbl_df', 'tbl' and 'data.frame':    8759 obs. of  9 variables:
#>  $ MAX3600     :Classes 'tbl_df', 'tbl' and 'data.frame':    8759 obs. of  9 variables:
#>  $ AVERAGE86400:Classes 'tbl_df', 'tbl' and 'data.frame':    1824 obs. of  9 variables:
#>  $ MIN86400    :Classes 'tbl_df', 'tbl' and 'data.frame':    1824 obs. of  9 variables:
#>  $ MAX86400    :Classes 'tbl_df', 'tbl' and 'data.frame':    1824 obs. of  9 variables:
</code></pre>
<p>Since the resulting object is a list of <code>tibble</code> objects, you can easily use R functions to work with an individual archive:</p>
<pre><code>names(cpu)
#>  [1] &quot;AVERAGE60&quot;    &quot;AVERAGE300&quot;   &quot;MIN300&quot;       &quot;MAX300&quot;      
#>  [5] &quot;AVERAGE3600&quot;  &quot;MIN3600&quot;      &quot;MAX3600&quot;      &quot;AVERAGE86400&quot;
#>  [9] &quot;MIN86400&quot;     &quot;MAX86400&quot;
</code></pre>
<p>To inspect the contents of the first archive (<code>AVERAGE60</code>), simply print the object – since it’s a <code>tibble</code>, you get 10 lines of output.</p>
<p>For example, the CPU metrics contains a time stamp and metrics for average <em>user</em> and <em>sys</em> usage, as well as the <a rel="nofollow" href="https://en.wikipedia.org/wiki/Nice_(Unix)" target="_blank"><em>nice</em></a> value, <em>idle</em> time, <a rel="nofollow" href="https://en.wikipedia.org/wiki/Interrupt_request_(PC_architecture)" target="_blank"><em>interrupt requests</em></a> and <em>soft interrupt requests</em>:</p>
<pre><code>cpu[[1]]
#> # A tibble: 43,199 x 9
#>    timestamp              user     sys  nice  idle  wait   irq softirq
#>  * </code></pre>