<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/binning-outliers-in-a-histogram/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://edwinth.github.io/blog/outlier-bin/"> That’s so Random</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I guess we all use it, the good old histogram. One of the first things we are taught in <em>Introduction to Statistics</em> and routinely applied whenever coming across a new continuous variable. However, it easily gets messed up by outliers. Putting most of the data into a single bin or a few bins, and scattering the outliers barely visible over the x-axis. This distribution might look familiar</p>
<div>
<pre><code><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>set.seed</span><span>(</span><span>42</span><span>)</span><span>
</span><span>hist_data</span><span> </span><span>&lt;-</span><span> </span><span>data_frame</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>rexp</span><span>(</span><span>1000</span><span>,</span><span> </span><span>.5</span><span>),</span><span>
</span><span>runif</span><span>(</span><span>50</span><span>,</span><span> </span><span>0</span><span>,</span><span> </span><span>500</span><span>)))</span><span>
</span><span>ggplot</span><span>(</span><span>hist_data</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span>))</span><span> </span><span>+</span><span> </span><span>geom_histogram</span><span>(</span><span>binwidth</span><span> </span><span>=</span><span> </span><span>.1</span><span>,</span><span> </span><span>col</span><span> </span><span>=</span><span> </span><span>&quot;black&quot;</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;cornflowerblue&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="plot of chunk unnamed-chunk-2" src="https://i1.wp.com/edwinth.github.io/figure/source/2017-04-26-outlier-bin/unnamed-chunk-2-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Two strategies that make the above into something more interpretable are taking the logarithm of the variable, or omitting the outliers. Both do not show the original distribution, however. Another way to go, is to create one bin for all the outlier values. This way we would see the original distribution where the density is the highest, while at the same time getting a feel for the number of outliers. A quick and dirty implementation of this would be</p>
<div>
<pre><code><span>hist_data</span><span> </span><span>%>%</span><span> </span><span>mutate</span><span>(</span><span>x_new</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>x</span><span> </span><span>></span><span> </span><span>10</span><span>,</span><span> </span><span>10</span><span>,</span><span> </span><span>x</span><span>))</span><span> </span><span>%>%</span><span> </span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x_new</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_histogram</span><span>(</span><span>binwidth</span><span> </span><span>=</span><span> </span><span>.1</span><span>,</span><span> </span><span>col</span><span> </span><span>=</span><span> </span><span>&quot;black&quot;</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;cornflowerblue&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="plot of chunk unnamed-chunk-3" src="https://i2.wp.com/edwinth.github.io/figure/source/2017-04-26-outlier-bin/unnamed-chunk-3-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Not bad. However, it now suggests incorrectly that many observations are exactly 10. I routinely make these plots for my own information, but they cannot be shared without explaining what happened to the outliers and what there original range was. Since a plot with a manual is not that great either, I recently did a hacking session into the <code>ggplot</code> object. The resulting <code>gg_outlier_bin</code> function not only indicates the range of the last bin, it also allows for a different fill color of the bin. Now we are clearly distinguishing the outlier aggregation</p>
<div>
<pre><code><span>gg_outlier_bin</span><span>(</span><span>hist_data</span><span>,</span><span> </span><span>&quot;x&quot;</span><span>,</span><span> </span><span>cut_off_floor</span><span> </span><span>=</span><span> </span><span>NA</span><span>,</span><span>
</span><span>cut_off_ceiling</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span>
</span><span>binwidth</span><span> </span><span>=</span><span> </span><span>0.1</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="plot of chunk unnamed-chunk-4" src="https://i1.wp.com/edwinth.github.io/figure/source/2017-04-26-outlier-bin/unnamed-chunk-4-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>It is still a bit experimental, but it seems to work in most situations. Below you find the function code for making histograms with outlier bins. You can also get it by installing the package accompanying this blog <code>devtools::install_github(&quot;edwinth/thatssorandom&quot;)</code>. By the way, it works on both floor and ceiling outliers. Like in the following</p>
<div>
<pre><code><span>data_frame</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>runif</span><span>(</span><span>100</span><span>,</span><span> </span><span>0</span><span>,</span><span> </span><span>100</span><span>),</span><span> </span><span>rnorm</span><span>(</span><span>1000</span><span>,</span><span> </span><span>50</span><span>,</span><span> </span><span>2</span><span>)))</span><span> </span><span>%>%</span><span> </span><span>gg_outlier_bin</span><span>(</span><span>&quot;x&quot;</span><span>,</span><span> </span><span>45</span><span>,</span><span> </span><span>55</span><span>,</span><span> </span><span>binwidth</span><span> </span><span>=</span><span> </span><span>.1</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="plot of chunk unnamed-chunk-5" src="https://i0.wp.com/edwinth.github.io/figure/source/2017-04-26-outlier-bin/unnamed-chunk-5-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>gg_outlier_bin</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>x</span><span>,</span><span>
</span><span>var_name</span><span>,</span><span>
</span><span>cut_off_floor</span><span>,</span><span>
</span><span>cut_off_ceiling</span><span>,</span><span>
</span><span>col</span><span> </span><span>=</span><span> </span><span>&quot;black&quot;</span><span>,</span><span>
</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;cornflowerblue&quot;</span><span>,</span><span>
</span><span>fill_outlier_bins</span><span> </span><span>=</span><span> </span><span>&quot;forestgreen&quot;</span><span>,</span><span>
</span><span>binwidth</span><span> </span><span>=</span><span> </span><span>NULL</span><span>)</span><span> </span><span>{</span><span>
</span><span>printing_min_max</span><span> </span><span>&lt;-</span><span> </span><span>x</span><span> </span><span>%>%</span><span> </span><span>summarise_</span><span>(</span><span>sprintf</span><span>(</span><span>&quot;round(min(%s, na.rm = TRUE), 1)&quot;</span><span>,</span><span> </span><span>var_name</span><span>),</span><span>
</span><span>sprintf</span><span>(</span><span>&quot;round(max(%s, na.rm = TRUE), 1)&quot;</span><span>,</span><span> </span><span>var_name</span><span>))</span><span>
</span><span>ceiling_filter</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>!</span><span>is.na</span><span>(</span><span>cut_off_ceiling</span><span>),</span><span>
</span><span>sprintf</span><span>(</span><span>&quot;%s &lt; %f&quot;</span><span>,</span><span> </span><span>var_name</span><span>,</span><span> </span><span>cut_off_ceiling</span><span>),</span><span>
</span><span>&quot;1 == 1&quot;</span><span>)</span><span> </span><span>floor_filter</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>!</span><span>is.na</span><span>(</span><span>cut_off_floor</span><span>),</span><span>
</span><span>sprintf</span><span>(</span><span>&quot;%s > %f&quot;</span><span>,</span><span> </span><span>var_name</span><span>,</span><span> </span><span>cut_off_floor</span><span>),</span><span>
</span><span>&quot;1 == 1&quot;</span><span>)</span><span>
</span><span>x_regular</span><span> </span><span>&lt;-</span><span> </span><span>x</span><span> </span><span>%>%</span><span> </span><span>filter_</span><span>(</span><span>ceiling_filter</span><span>,</span><span> </span><span>floor_filter</span><span>)</span><span> </span><span>%>%</span><span> </span><span>select_</span><span>(</span><span>var_name</span><span>)</span><span>
</span><span>x_to_roll_ceiling</span><span> </span><span>&lt;-</span><span> </span><span>x</span><span> </span><span>%>%</span><span> </span><span>filter_</span><span>(</span><span>
</span><span>sprintf</span><span>(</span><span>&quot;%s >= %f&quot;</span><span>,</span><span> </span><span>var_name</span><span>,</span><span> </span><span>cut_off_ceiling</span><span>))</span><span> </span><span>%>%</span><span> </span><span>select_</span><span>(</span><span>var_name</span><span>)</span><span>
</span><span>if</span><span> </span><span>(</span><span>!</span><span>is.na</span><span>(</span><span>cut_off_ceiling</span><span>))</span><span> </span><span>x_to_roll_ceiling</span><span>[,</span><span> </span><span>1</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>cut_off_ceiling</span><span>
</span><span>x_to_roll_floor</span><span> </span><span>&lt;-</span><span> </span><span>x</span><span> </span><span>%>%</span><span> </span><span>filter_</span><span>(</span><span>
</span><span>sprintf</span><span>(</span><span>&quot;%s &lt;= %f&quot;</span><span>,</span><span> </span><span>var_name</span><span>,</span><span> </span><span>cut_off_floor</span><span>))</span><span> </span><span>%>%</span><span> </span><span>select_</span><span>(</span><span>var_name</span><span>)</span><span>
</span><span>if</span><span> </span><span>(</span><span>!</span><span>is.na</span><span>(</span><span>cut_off_floor</span><span>))</span><span> </span><span>x_to_roll_floor</span><span>[,</span><span> </span><span>1</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>cut_off_floor</span><span>
</span><span>plot_obj</span><span> </span><span>&lt;-</span><span> </span><span>ggplot</span><span>(</span><span>x_regular</span><span>,</span><span> </span><span>aes_string</span><span>(</span><span>var_name</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_histogram</span><span>(</span><span>col</span><span> </span><span>=</span><span> </span><span>col</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>fill</span><span>,</span><span> </span><span>binwidth</span><span> </span><span>=</span><span> </span><span>binwidth</span><span>)</span><span>
</span><span>if</span><span> </span><span>(</span><span>!</span><span>is.na</span><span>(</span><span>cut_off_ceiling</span><span>))</span><span> </span><span>{</span><span>
</span><span>ticks_for_ceiling</span><span> </span><span>&lt;-</span><span> </span><span>update_tickmarks_ceiling</span><span>(</span><span>plot_obj</span><span>,</span><span> </span><span>cut_off_ceiling</span><span>,</span><span>
</span><span>printing_min_max</span><span>[</span><span>1</span><span>,</span><span>2</span><span>])</span><span>
</span><span>plot_obj</span><span> </span><span>&lt;-</span><span> </span><span>plot_obj</span><span> </span><span>+</span><span>
</span><span>geom_histogram</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>x_to_roll_ceiling</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>fill_outlier_bins</span><span>,</span><span> </span><span>col</span><span> </span><span>=</span><span> </span><span>col</span><span>,</span><span>
</span><span>binwidth</span><span> </span><span>=</span><span> </span><span>binwidth</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_x_continuous</span><span>(</span><span>breaks</span><span> </span><span>=</span><span> </span><span>ticks_for_ceiling</span><span>$</span><span>tick_positions</span><span>,</span><span>
</span><span>labels</span><span> </span><span>=</span><span> </span><span>ticks_for_ceiling</span><span>$</span><span>tick_labels</span><span>)</span><span>
</span><span>}</span><span>
</span><span>if</span><span> </span><span>(</span><span>!</span><span>is.na</span><span>(</span><span>cut_off_floor</span><span>))</span><span> </span><span>{</span><span>
</span><span>ticks_for_floor</span><span> </span><span>&lt;-</span><span> </span><span>update_tickmarks_floor</span><span>(</span><span>plot_obj</span><span>,</span><span> </span><span>cut_off_floor</span><span>,</span><span>
</span><span>printing_min_max</span><span>[</span><span>1</span><span>,</span><span>1</span><span>])</span><span>
</span><span>plot_obj</span><span> </span><span>&lt;-</span><span> </span><span>plot_obj</span><span> </span><span>+</span><span>
</span><span>geom_histogram</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>x_to_roll_floor</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>fill_outlier_bins</span><span>,</span><span> </span><span>col</span><span> </span><span>=</span><span> </span><span>col</span><span>,</span><span> </span><span>binwidth</span><span> </span><span>=</span><span> </span><span>binwidth</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_x_continuous</span><span>(</span><span>breaks</span><span> </span><span>=</span><span> </span><span>ticks_for_floor</span><span>$</span><span>tick_positions</span><span>,</span><span>
</span><span>labels</span><span> </span><span>=</span><span> </span><span>ticks_for_floor</span><span>$</span><span>tick_labels</span><span>)</span><span>
</span><span>}</span><span>
</span><span>return</span><span>(</span><span>plot_obj</span><span>)</span><span>
</span><span>}</span><span>
</span><span>update_tickmarks_ceiling</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>gg_obj</span><span>,</span><span> </span><span>co</span><span>,</span><span> </span><span>max_print</span><span>)</span><span> </span><span>{</span><span>
</span><span>ranges</span><span> </span><span>&lt;-</span><span> </span><span>suppressMessages</span><span>(</span><span>
</span><span>ggplot_build</span><span>(</span><span>gg_obj</span><span>)</span><span>$</span><span>layout</span><span>$</span><span>panel_ranges</span><span>[[</span><span>1</span><span>]])</span><span>
</span><span>label_to_add</span><span> </span><span>&lt;-</span><span> </span><span>sprintf</span><span>(</span><span>&quot;(%s , %s)&quot;</span><span>,</span><span> </span><span>round</span><span>(</span><span>co</span><span>,</span><span> </span><span>1</span><span>),</span><span> </span><span>max_print</span><span>)</span><span>
</span><span>tick_positions</span><span> </span><span>&lt;-</span><span> </span><span>ranges</span><span>$</span><span>x.major_source</span><span>
</span><span>tick_labels</span><span> </span><span>&lt;-</span><span> </span><span>ranges</span><span>$</span><span>x.labels</span><span>
</span><span>if</span><span> </span><span>(</span><span>overlap_ceiling</span><span>(</span><span>tick_positions</span><span>,</span><span> </span><span>co</span><span>))</span><span> </span><span>{</span><span>
</span><span>tick_positions</span><span> </span><span>&lt;-</span><span> </span><span>tick_positions</span><span>[</span><span>-</span><span>length</span><span>(</span><span>tick_positions</span><span>)]</span><span>
</span><span>tick_labels</span><span> </span><span>&lt;-</span><span> </span><span>tick_labels</span><span>[</span><span>-</span><span>length</span><span>(</span><span>tick_labels</span><span>)]</span><span>
</span><span>}</span><span>
</span><span>return</span><span>(</span><span>list</span><span>(</span><span>tick_positions</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>tick_positions</span><span>,</span><span> </span><span>co</span><span>),</span><span>
</span><span>tick_labels</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>tick_labels</span><span>,</span><span> </span><span>label_to_add</span><span>)))</span><span>
</span><span>}</span><span>
</span><span>overlap_ceiling</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>positions</span><span>,</span><span> </span><span>cut_off</span><span>)</span><span> </span><span>{</span><span>
</span><span>n</span><span> </span><span>&lt;-</span><span> </span><span>length</span><span>(</span><span>positions</span><span>)</span><span>
</span><span>ticks_dif</span><span> </span><span>&lt;-</span><span> </span><span>positions</span><span>[</span><span>n</span><span>]</span><span> </span><span>-</span><span> </span><span>positions</span><span>[</span><span>n</span><span>-1</span><span>]</span><span>
</span><span>(</span><span>cut_off</span><span> </span><span>-</span><span> </span><span>positions</span><span>[</span><span>n</span><span>])</span><span> </span><span>/</span><span> </span><span>ticks_dif</span><span> </span><span>&lt;</span><span> </span><span>0.25</span><span>
</span><span>}</span><span>
</span><span>update_tickmarks_floor</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>gg_obj</span><span>,</span><span> </span><span>co</span><span>,</span><span> </span><span>min_print</span><span>)</span><span> </span><span>{</span><span>
</span><span>ranges</span><span> </span><span>&lt;-</span><span> </span><span>suppressMessages</span><span>(</span><span>
</span><span>ggplot_build</span><span>(</span><span>gg_obj</span><span>)</span><span>$</span><span>layout</span><span>$</span><span>panel_ranges</span><span>[[</span><span>1</span><span>]])</span><span>
</span><span>label_to_add</span><span> </span><span>&lt;-</span><span> </span><span>sprintf</span><span>(</span><span>&quot;(%s , %s)&quot;</span><span>,</span><span> </span><span>min_print</span><span>,</span><span> </span><span>round</span><span>(</span><span>co</span><span>,</span><span> </span><span>1</span><span>))</span><span>
</span><span>tick_positions</span><span> </span><span>&lt;-</span><span> </span><span>ranges</span><span>$</span><span>x.major_source</span><span>
</span><span>tick_labels</span><span> </span><span>&lt;-</span><span> </span><span>ranges</span><span>$</span><span>x.labels</span><span>
</span><span>if</span><span> </span><span>(</span><span>overlap_floor</span><span>(</span><span>tick_positions</span><span>,</span><span> </span><span>co</span><span>))</span><span> </span><span>{</span><span>
</span><span>tick_positions</span><span> </span><span>&lt;-</span><span> </span><span>tick_positions</span><span>[</span><span>-1</span><span>]</span><span>
</span><span>tick_labels</span><span> </span><span>&lt;-</span><span> </span><span>tick_labels</span><span>[</span><span>-1</span><span>]</span><span>
</span><span>}</span><span>
</span><span>return</span><span>(</span><span>list</span><span>(</span><span>tick_positions</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>co</span><span>,</span><span> </span><span>tick_positions</span><span>),</span><span>
</span><span>tick_labels</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>label_to_add</span><span>,</span><span> </span><span>tick_labels</span><span>)))</span><span>
</span><span>}</span><span>
</span><span>overlap_floor</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>positions</span><span>,</span><span> </span><span>cut_off</span><span>)</span><span> </span><span>{</span><span>
</span><span>ticks_dif</span><span> </span><span>&lt;-</span><span> </span><span>positions</span><span>[</span><span>2</span><span>]</span><span> </span><span>-</span><span> </span><span>positions</span><span>[</span><span>1</span><span>]</span><span>
</span><span>(</span><span>positions</span><span>[</span><span>1</span><span>]</span><span> </span><span>-</span><span> </span><span>cut_off</span><span>)</span><span> </span><span>/</span><span> </span><span>ticks_dif</span><span> </span><span>&lt;</span><span> </span><span>0.25</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://edwinth.github.io/blog/outlier-bin/"> That’s so Random</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/yS_o0Ti8ul0"></a></div>