<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/preview-altrep-promises-to-bring-major-performance-improvements-to-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2017/09/altrep-preview.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>Changes are coming to the internals of the R engine which promise to improve performance and reduce memory use, with dramatic impacts in some circumstances. The changes were first proposed by Gabe Becker at the DSC Conference in <a rel="nofollow" href="https://www.r-project.org/dsc/2016/slides/customvectors.html" target="_blank">2016</a> (and updated in <a rel="nofollow" href="https://www.r-project.org/dsc/2017/slides/dsc2017.pdf" target="_blank">2017</a>), and the implementation by Luke Tierney and Gabe Becker is <a rel="nofollow" href="https://twitter.com/groundwalkergmb/status/907720231413456896" target="_blank">now making its way</a> into the development branches of the R sources.</p>
<p>ALTREP is an alternative way of representing R objects internally. (The details are described in this <a rel="nofollow" href="https://svn.r-project.org/R/branches/ALTREP/ALTREP.html" target="_blank">draft vignette</a>, and there are a few more examples in this <a rel="nofollow" href="https://gmbecker.github.io/jsm2017/jsm2017.html" target="_blank">presentation</a>.) It's easiest to illustrate by example.</p>
<p>Today, a vector of numbers in R is represented as a contiguous block of memory in RAM. In other words, if you create the sequence of a million integers <code>1:1000000</code>, R creates a block of memory 4Mb in size (4 bytes per number) to store each number in the sequence. But what if we could use a more efficient representation just for sequences like this? WIth ALTREP, a sequence like this is instead represented by <a rel="nofollow" href="https://svn.r-project.org/R/branches/ALTREP/ALTREP.html#compact_integer_vectors" target="_blank">just its start and end values</a>, which takes up almost no memory at all. That means, in the future you'll be able to write loops like this:</p>
<div>
<pre><span>for</span> (i <span>in</span> <span>1</span><span>:</span><span>1e10</span>) do_something()
</pre>
</div>
<p>without getting errors like this:</p>
<div>
<pre>Error<span>:</span> cannot allocate vector of size <span>74.5</span> Gb
</pre>
</div>
<p>ALTREP has the potential to make many other operations faster or even instantaneous, even on very large objects. Here are a few examples of functions that could be sped up:</p>
<ul>
<li><code>is.na(x)</code> — ALTREP will keep track of whether a vector includes NAs or not, so that R no longer has to inspect the entire vector</li>
<li><code>sort(x)</code> — ALTREP will keep track of whether a vector is already sorted, and sorting will be instantaneous in this case</li>
<li><code>x &lt; 5</code> — knowing that the vector is already sorted, ALTREP can find the breakpoint very quickly (in O(log <em>n</em>) time), and return a &quot;sequence&quot; logical vector that consumes basically no memory</li>
<li><code>match(y,x)</code> — if ALTREP knows that x is already sorted, matching is also much faster</li>
<li><code>as.character(numeric_vector)</code> — ALTREP can defer converting a numeric vector to characters until the character representation is actually needed</li>
</ul>
<p>That last benefit will likely have a large impact on the handling of data frames, which carry around a column of character row labels which start out as a numeric sequence. Development builds are already demonstrating a huge performance gain in the linear regression function <code>lm()</code> as a result:</p>
<div>
<pre><span>></span> n <span>&lt;-</span> <span>10000000</span>
<span>></span> x <span>&lt;-</span> rnorm(n)
<span>></span> y <span>&lt;-</span> rnorm(n)

<span># With R 3.4</span>
<span>></span> system.time(lm(y <span>~</span> x))
   user  system elapsed 
  <span>9.225</span>   <span>0.703</span>   <span>9.929</span> 

<span># With ALTREP</span>
<span>></span> system.time(lm(y <span>~</span> x))
   user  system elapsed 
  <span>1.886</span>   <span>0.610</span>   <span>2.496</span> 
</pre>
</div>
<p>The ALTREP framework is designed to be extensible, so that package authors can define their own alternative representations of standard R objects. For example, an R vector could be represented as a distributed object as in a system like Spark, while still behaving like an ordinary R vector to the R engine. An example package, <a rel="nofollow" href="https://github.com/ltierney/Rpkg-simplemmap" target="_blank">simplemmap</a>, illustrates this concept by defining an implementation of vectors as memory-mapped objects that live on disk instead of in RAM.</p>
<p>There's no definite date yet when ALTREP will be in an official R release, and my guess is that there's likely to be an extensive testing period to shake out any bugs caused by changing the internal representation of R objects. But the fact that the implementation is already making its way into the R sources is hugely promising, and I look forward to testing out the real-world impacts. You can read more about the current state of ALTREP in the draft vignette by Luke Tierney, Gabe Becker and Tomas Kalibera linked below.</p>
<p>R-Project.org: <a rel="nofollow" href="https://svn.r-project.org/R/branches/ALTREP/ALTREP.html" target="_blank">ALTREP: Alternative Representations for R Objects</a></p>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2017/09/altrep-preview.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/qn6R1DrDF8o" height="1"></a></div>