<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/faster-arrays-and-matrices-in-jsonlite-0-9-20/"></div></div><div><a data-url="http://www.r-bloggers.com/faster-arrays-and-matrices-in-jsonlite-0-9-20/" data-counturl="http://www.r-bloggers.com/faster-arrays-and-matrices-in-jsonlite-0-9-20/" data-count="horizontal" data-text="Faster arrays and matrices in jsonlite 0.9.20" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://www.opencpu.org/posts/jsonlite-0-9-20/"> OpenCPU</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p> <a target="_blank" href="https://www.opencpu.org/posts/jsonlite-0-9-20"><img alt="opencpu logo" src="https://www.opencpu.org/images/mariokart.jpg"></a></p>
<p>Yesterday a new version of the <a target="_blank" href="https://cran.r-project.org/web/packages/jsonlite/vignettes/json-aaquickstart.html">jsonlite</a> package was released to CRAN. This update includes no new features, it only introduces performance optimizations.</p>
<h2>Large Matrices</h2>
<p>The jsonlite package was already highly optimized for converting vectors and data frames to json. However Gregory Jefferis and Duncan Murdoch had found that conversion of tall matrices as used by <a target="_blank" href="https://cran.r-project.org/web/packages/rglwidget/index.html">rglwidget</a> was slower than expected.</p>
<p>It turned out this was indeed an edge case that I had overlooked. The new version of jsonlite fixes this problem and matrix conversion should be about 200 times faster than before. Technical details follow below; first a benchmark:</p>
<div>
<pre><code><span># Old version!
</span><span>></span> <span>system.time</span><span>(</span><span>j</span><span>&lt;-</span><span>toJSON</span><span>(</span><span>matrix</span><span>(</span><span>1L</span><span>,</span> <span>ncol</span> <span>=</span> <span>3</span><span>,</span> <span>nrow</span> <span>=</span> <span>50000</span><span>)))</span>
   <span>user</span>  <span>system</span> <span>elapsed</span>
  <span>4.715</span>   <span>0.015</span>   <span>4.729</span>

<span># New version!
</span><span>></span> <span>system.time</span><span>(</span><span>j</span><span>&lt;-</span><span>toJSON</span><span>(</span><span>matrix</span><span>(</span><span>1L</span><span>,</span> <span>ncol</span> <span>=</span> <span>3</span><span>,</span> <span>nrow</span> <span>=</span> <span>50000</span><span>)))</span>
   <span>user</span>  <span>system</span> <span>elapsed</span>
  <span>0.022</span>   <span>0.002</span>   <span>0.023</span>
</code></pre>
</div>
<p>This artificial example (every field has the number 1) highlights the improvement. The relative improvement might be less for matrices with actual data because of additional time spent on number formatting double/integer values (which was already optimized in jsonlite a <a target="_blank" href="https://www.opencpu.org/posts/jsonlite-release-0-9-13/">while ago</a>).</p>
<h2>Technical Details</h2>
<p>So what was the problem? The previous version of jsonlite had an elegant solution that would recurse through the dimensions of a matrix/array and apply json conversion on each of its elements. E.g. for a matrix (2D array) it would convert each row to json, and then combine the results. However it turns out that the <code>apply</code> call below is really slow.</p>
<div>
<pre><code><span># Technical example, don't use this code !
</span><span>x</span> <span>&lt;-</span> <span>matrix</span><span>(</span><span>1L</span><span>,</span> <span>ncol</span> <span>=</span> <span>3</span><span>,</span> <span>nrow</span> <span>=</span> <span>50000</span><span>)</span>
<span>rows</span> <span>&lt;-</span> <span>apply</span><span>(</span><span>x</span><span>,</span> <span>1</span><span>,</span> <span>jsonlite</span><span>:::</span><span>asJSON</span><span>)</span>
<span>json</span> <span>&lt;-</span> <span>jsonlite</span><span>:::</span><span>collapse</span><span>(</span><span>rows</span><span>,</span> <span>indent</span> <span>=</span> <span>NA</span><span>)</span>
</code></pre>
</div>
<p>The new version exploits the fact that matrices and arrays are homogenous (i.e. all elements have the same type). It first removes the dimensions from the array using <code>c(x)</code> and converts all of the individual elements to json with a single call to <code>asJSON</code>. This results in a significant speedup because <code>asJSON</code> is only called once rather than <code>n</code> times.</p>
<div>
<pre><code><span># Technical example, don't use this code !
</span><span>str</span> <span>&lt;-</span> <span>jsonlite</span><span>:::</span><span>asJSON</span><span>(</span><span>c</span><span>(</span><span>x</span><span>),</span> <span>collapse</span> <span>=</span> <span>FALSE</span><span>)</span>
<span>dim</span><span>(</span><span>str</span><span>)</span> <span>&lt;-</span> <span>dim</span><span>(</span><span>x</span><span>)</span>
<span>rows</span> <span>&lt;-</span> <span>apply</span><span>(</span><span>str</span><span>,</span> <span>1</span><span>,</span> <span>jsonlite</span><span>:::</span><span>collapse</span><span>,</span> <span>indent</span> <span>=</span> <span>NA</span><span>)</span>
<span>json</span> <span>&lt;-</span> <span>jsonlite</span><span>:::</span><span>collapse</span><span>(</span><span>rows</span><span>,</span> <span>indent</span> <span>=</span> <span>NA</span><span>)</span>
</code></pre>
</div>
<p>Things get a bit more complicated for higher dimensional arrays, especially with <code>toJSON(x, pretty = TRUE)</code> but this illustrates the core issue.</p>
<p>You might be thinking: can we avoid <code>apply</code> alltogether? Yes! For the important case of 2 dimensional arrays jsonlite has a complete C implementation which makes <code>toJSON</code> on matrices is extra fast. For higher dimensional arrays it currently still uses the solution above, which performs quite well. We might be able to further optimize this case by porting this to C as well, but working with high dimensional arrays in C makes my head hurt.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/faster-arrays-and-matrices-in-jsonlite-0-9-20/"></div></div><div><a data-url="http://www.r-bloggers.com/faster-arrays-and-matrices-in-jsonlite-0-9-20/" data-counturl="http://www.r-bloggers.com/faster-arrays-and-matrices-in-jsonlite-0-9-20/" data-count="vertical" data-text="Faster arrays and matrices in jsonlite 0.9.20" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.opencpu.org/posts/jsonlite-0-9-20/"> OpenCPU</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/k88wB67e7pw"></a></div>