<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/datashader-is-a-big-deal/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2017/03/datashader-is-a-big-deal/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I recently got back from Strata West 2017 (where I ran a very well received <a target="_blank" rel="nofollow" href="https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/55791">workshop</a> on <a target="_blank" rel="nofollow" href="https://cran.r-project.org/"><code>R</code></a> and <a target="_blank" rel="nofollow" href="http://spark.apache.org/"><code>Spark</code></a>). One thing that really stood out for me at the exhibition hall was <a target="_blank" rel="nofollow" href="http://bokeh.pydata.org/en/latest/"><code>Bokeh</code></a> plus <a target="_blank" rel="nofollow" href="http://datashader.readthedocs.io/en/latest/"><code>datashader</code></a> from <a target="_blank" rel="nofollow" href="https://www.continuum.io/">Continuum Analytics</a>.</p>
<p>I had the privilege of having <a target="_blank" rel="nofollow" href="https://twitter.com/pwang">Peter Wang</a> himself demonstrate <code>datashader</code> for me and answer a few of my questions.</p>
<p>I am so excited about <code>datashader</code> capabilities I literally <em>will not wait</em> for the functionality to be exposed in <code>R</code> through <a target="_blank" rel="nofollow" href="http://hafen.github.io/rbokeh/"><code>rbokeh</code></a>. I am going to leave my usual <code>knitr</code>/<code>rmarkdown</code> world and dust off <a target="_blank" rel="nofollow" href="http://jupyter.org/"><code>Jupyter Notebook</code></a> just to use <code>datashader</code> plotting. This is worth trying, even for diehard <code>R</code> users.<span></span></p>
<p>
<h2>datashader</h2>
<p>Every plotting system has two important ends: the <em>grammar</em> where you specify the plot, and the <em>rendering pipeline</em> that executes the presentation. Switching plotting systems means switching how you specify plots and can be unpleasant (this is one of the reasons we wrap our most re-used plots in <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=WVPlots">WVPlots</a> to hide or decouple how the plots are specified from the results you get). Given the convenience of the <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=ggplot2">ggplot2</a> grammar, I am always reluctant to move to other plotting systems unless they bring me something big (and even then sometimes you don’t have to leave: for example the absolutely amazing adapter <a target="_blank" rel="nofollow" href="https://plot.ly/ggplot2/"><code>plotly::ggplotly</code></a>).</p>
<p>Currently, to use <code>datashader</code> you must talk directly to <code>Python</code> and <code>Bokeh</code> (i.e. learn a different language). But what that buys you is massive: in-pixel analytics. Let me clarify that.</p>
<p><code>datashader</code> makes points and pixels first class entities in the graphics rendering pipeline. It admits they exist (many plotting systems render to an imaginary infinite resolution abstract plane) and allows the user to specify scale dependent calculations and re-calculations over them. It is easiest to show by example.</p>
<p>Please take a look at these stills from the <a target="_blank" rel="nofollow" href="https://anaconda.org/jbednar/census/notebook"><code>datashader</code> US Census example</a>. We can ask pixels to be colored by the majority race in the region of Lake Michigan:</p>
<p><img title="NewImage.png" alt="NewImage" src="https://i0.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/03/NewImage-4.png?resize=449%2C262" border="0" data-recalc-dims="1"></p>
<p>If we were to use the interactive version of this graph we could zoom in on Chicago and the majorities are re-calculated based on the new scale:</p>
<p><img title="NewImage.png" alt="NewImage" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/03/NewImage-5.png?resize=449%2C262" border="0" data-recalc-dims="1"></p>
<p>What is important to understand is that is this is vastly more powerful than zooming in on a low-resolution rendering:</p>
<p><img title="Unknown.png" alt="Unknown" src="https://i1.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/03/Unknown.png?resize=453%2C262" border="0" data-recalc-dims="1"></p>
<p>and even more powerful than zooming out on a static high-resolution rendering:</p>
<p><img title="Unknown-2.png" alt="Unknown 2" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/03/Unknown-2.png?resize=449%2C262" border="0" data-recalc-dims="1"></p>
<p><code>datashader</code> can redo aggregations and analytics on the fly. It can recompute histograms and renormalize them relative to what is visible to maintain contrast. It can find patterns that emerge as we change scale: think of zooming in on a grey pixel that resolves into a black and white checkerboard. </p>
<p>You need to run <code>datashader</code> to really see the effect. The <code>html</code> exports, while interactive, sometimes do not correctly perform in all web browsers. </p>
<h2>An R example</h2>
<p>I am going to share a simple <code>datashader</code> example <a target="_blank" rel="nofollow" href="http://winvector.github.io/datashader/RPyExample.html">here</a>. Again, to see the full effect you would have to copy it into an <code>Jupyter</code> notebook and run it. But I will use it to show my point.</p>
<p>After going through the steps to install <code>Anaconda</code> and <code>Juputer notebook</code> (plus some more <code>conda install</code> steps to include necessary packages) we can make a plot of the <code>ggplot2</code> data example <code>diamonds</code></p>
<p><code>ggplot2</code> renderings of <code>diamonds</code> typically look like the following (and show of the power and convenience of the grammar):</p>
<p><img title="NewImage.png" alt="NewImage" src="https://i1.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/03/NewImage-6.png?w=660" border="0" data-recalc-dims="1"></p>
<p>A <code>datashader</code> rendering looks like the following:</p>
<p><img title="NewImage.png" alt="NewImage" src="https://i1.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/03/NewImage-7.png?w=660" border="0" data-recalc-dims="1"></p>
<p>If we use the interactive rectangle selector to zoom in on the apparently isolated point around $18300 and 3.025 carats we get the following dynamic re-render:</p>
<p><img title="NewImage.png" alt="NewImage" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/03/NewImage-8.png?w=660" border="0" data-recalc-dims="1"></p>
<p>Notice the points shrunk (and didn’t subdivide) and there are some extremely faint points. There is something wrong with that as a presentation; but it isn’t because of <code>datashader</code>! It is something unexpected in the data which is now jumping out at us.</p>
<p><code>datashader</code> is shading proportional to aggregated count. So the small point staying very dark (and being so dark it causes other point to render near transparent) means there are multiple observations in this tiny neighborhood. Going back to <code>R</code> we can look directly at the data:</p>
<pre>
> library(&quot;dplyr&quot;)
> diamonds %>% filter(carat>=3, carat&lt;=3.05, price>=18200, price&lt;=18400)

# A tibble: 5 × 10
  carat     cut color clarity depth table price     x     y     z
  &lt;dbl>   &lt;ord> &lt;ord>   &lt;ord> &lt;dbl> &lt;dbl> &lt;int> &lt;dbl> &lt;dbl> &lt;dbl>
1  3.01 Premium     I     SI2  60.2    59 18242  9.36  9.31  5.62
2  3.01    Fair     I     SI2  65.8    56 18242  8.99  8.94  5.90
3  3.01    Fair     I     SI2  65.8    56 18242  8.99  8.94  5.90
4  3.01    Good     I     SI2  63.9    60 18242  9.06  9.01  5.77
5  3.01    Good     I     SI2  63.9    60 18242  9.06  9.01  5.77
</pre>
<p>There are actually 5 rows with the exact carat and pricing indicated by the chosen point. The point stood out at fine scale because it indicated something subtle in the data (repetitions) that the analyst may not have known about or expected. The “ugly” presentation was an important warning. This is hands on the data, the quickest path to correct results.</p>
<p>For some web browsers, you don’t always see proper scaling, yielding artifacts like the following:</p>
<p><img title="NewImage.png" alt="NewImage" src="https://i1.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/03/NewImage-9.png?resize=300%2C300" border="0" data-recalc-dims="1"></p>
<p>The <code>Jupyter notebooks</code> always work, and web-browsers usually work (I am assuming it is security or ad-blocking that is causing the effect, not a <code>datashader</code> issue).</p>
<h2>Conclusion</h2>
<p><code>datashader</code> brings to production resolution dependent per-pixel analytics. This is a very powerful style of interaction that is going to appear more and more places. This is something that the Continuum Analytics team has <a target="_blank" rel="nofollow" href="http://www.crest.iu.edu/publications/prints/2014/Cottam2014OutOfCore.pdf">written about</a> before and requires some interesting cross-compiling (<a target="_blank" rel="nofollow" href="http://numba.pydata.org/">Numba</a>) to implement at scale. Now that analysts have seen this in action they are going to want this and ask for this.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2017/03/datashader-is-a-big-deal/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/vymm-6TCvdQ"></a></div>