<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/handling-outliers-with-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://madstatbr.wordpress.com/2018/07/02/handling-outliers-with-r/"> R – MAD-Stat</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Recently, I attended a presentation where the following graph was shown illustrating the response to stimulation with Thalidomide among a cohort of HIV-1 patients. The biomarker used to measure the response in this case was TNF (tumor necrosis factor) and the response was measured at four time points: time of drug administration and 3, 11 and 23 weeks after administration. This graph appears in a published article, which I won’t cite directly because except for this one statistical problem, the research is outstanding.</p>
<p><img data-orig-file="https://madstatbr.files.wordpress.com/2018/07/tv_outlier_2.png?w=456" data-orig-size="609,460" data-medium-file="https://madstatbr.files.wordpress.com/2018/07/tv_outlier_2.png?w=456?w=300" data-attachment-id="974" data-image-meta="{" data-image-description="" src="https://madstatbr.files.wordpress.com/2018/07/tv_outlier_2.png?w=456" data-permalink="https://madstatbr.wordpress.com/2018/07/02/handling-outliers-with-r/tv_outlier_2/" data-large-file="https://madstatbr.files.wordpress.com/2018/07/tv_outlier_2.png?w=456?w=609" alt="TV_outlier_2" data-image-title="TV_outlier_2" sizes="(max-width: 609px) 100vw, 609px" data-comments-opened="1" class="wp-image-974" data-recalc-dims="1"></p>
<p> </p>
<p>Part of the problem in this graph comes from the software being used. Clearly not R and clearly not ggplot. It comes from one of the highly automated programs that assumes you, the scientist, are a total jerk and incapable of specifying a graph on your own. I have highlighted the problem I saw. That very high value in week 23 seems questionable and could be distorting the summary measures that are being used in the inferences in the article. The following table of the data gives some detail to the problem:</p>
<p><img data-orig-file="https://madstatbr.files.wordpress.com/2018/07/cases_table.png?w=456" data-orig-size="487,364" data-medium-file="https://madstatbr.files.wordpress.com/2018/07/cases_table.png?w=456?w=300" data-attachment-id="975" data-image-meta="{" data-image-description="" src="https://madstatbr.files.wordpress.com/2018/07/cases_table.png?w=456" data-permalink="https://madstatbr.wordpress.com/2018/07/02/handling-outliers-with-r/cases_table/" data-large-file="https://madstatbr.files.wordpress.com/2018/07/cases_table.png?w=456?w=487" alt="cases_table" data-image-title="cases_table" sizes="(max-width: 487px) 100vw, 487px" data-comments-opened="1" class="wp-image-975" data-recalc-dims="1"></p>
<p>Beside the large number of missing values, we can see that the value for Patient 25 in week 23 (visit 4), 1339.552, is 10 times higher than the next value (130.997). The fact that there a number of similar digits between it and the next lower value also raises suspicion that this could simply be a data entry error or an error translating the data from its original home in a spreadsheet to the statistical program used. (I will also not mention the name of the program used, since I try to follow the English comedian Frankie Howerd’s advice: “Don’t mock the afflicted”.)</p>
<p>Is such a high value reasonable? And, if it it is simply a data error, could the use of this value lead to the wrong treatment recommendation based on this cohort.</p>
<h2>First Question — Is It an Outlier?</h2>
<p>The first question that we need to address is whether this value, 1339.552, is really an outlier that needs attention and, if so, how should we modify it?</p>
<p>What’s an outlier? The most common definition is that it is a value that lies far away from the main body of observations for a variable and could distort summaries of the distribution of values. In practice this translates to a value that is 1.5 times the IQR (interquartile range) more extreme than the quartiles of the distribution.</p>
<p>The most effective way to see an outlier is to use a boxplot. The following figure relates the parts of a boxplot to a distribution and its histogram. I have taken it from the excellent book on R by Hadley Wickham and Garrett Grolemund, <strong>R for Data Science</strong>, which is available for reading <a rel="nofollow" href="http://r4ds.had.co.nz/exploratory-data-analysis.html#cat-cont" target="_blank">here</a>.</p>
<p><img data-orig-file="https://madstatbr.files.wordpress.com/2018/07/r4ds-boxplot-image.png?w=456" data-orig-size="2133,1183" data-medium-file="https://madstatbr.files.wordpress.com/2018/07/r4ds-boxplot-image.png?w=456?w=300" data-attachment-id="976" data-image-meta="{" data-image-description="" src="https://madstatbr.files.wordpress.com/2018/07/r4ds-boxplot-image.png?w=456" data-permalink="https://madstatbr.wordpress.com/2018/07/02/handling-outliers-with-r/r4ds-boxplot-image/" data-large-file="https://madstatbr.files.wordpress.com/2018/07/r4ds-boxplot-image.png?w=456?w=640" alt="r4ds boxplot image" data-image-title="r4ds boxplot image" sizes="(max-width: 640px) 100vw, 640px" data-comments-opened="1" class="wp-image-976" data-recalc-dims="1"></p>
<p>Boxplots are an excellent way to identify outliers and other data anomalies. We can draw them either with the base R function <em>boxplot() </em>or the <em>ggplot2</em> geometry <em>geom_boxplot(). </em>Here, I am going to use the <em>ggboxplot()</em> function from the <em>ggpubr</em> package. I find that the functions from <em>ggpubr</em> keep me from making many mistakes in specifying parameters for the equivalent <em>ggplot2</em> functions. It makes life a little simpler.</p>
<p>Here is the code that read in the original data and tidied the data to facilitate the graphing and subsequent analysis.</p>
<pre>library(tidyverse)
library(DescTools)
library(ggpubr)
tnf &lt;- readRDS(&quot;../tnf.rds&quot;)
tnf_long &lt;- tnf %>% 
 gather(vis0:vis4, key = &quot;visit&quot;, value = &quot;value&quot;) %>% 
 group_by(visit)</pre>
<p>The next step is to look at the boxplot, which I have created with the following code:</p>
<pre>gr_tnf_vis2 &lt;- tnf_long %>% 
 filter(!is.na(visit) &amp; !is.na(value)) %>% 
 ggboxplot(x = &quot;visit&quot;, y = &quot;value&quot;,
 palette = &quot;aaas&quot;, 
 title = &quot;TNF Levels by Visit - Thalidomide Group&quot;,
 xlab = &quot;Visit&quot;,
 ylab = &quot;TNF level&quot;,
 ggtheme = theme_gray())
suppressMessages(gr_tnf_vis2)</pre>
<p><img data-orig-file="https://madstatbr.files.wordpress.com/2018/07/tnf_boxplot.png?w=450&amp;h=499&amp;fit=640%2C499" data-orig-size="640,499" data-medium-file="https://madstatbr.files.wordpress.com/2018/07/tnf_boxplot.png?w=450&amp;h=499&amp;fit=640%2C499?w=300" data-attachment-id="977" data-image-meta="{" data-image-description="" src="https://madstatbr.files.wordpress.com/2018/07/tnf_boxplot.png?w=450&h=499&fit=640%2C499" data-permalink="https://madstatbr.wordpress.com/2018/07/02/handling-outliers-with-r/tnf_boxplot/" data-large-file="https://madstatbr.files.wordpress.com/2018/07/tnf_boxplot.png?w=450&amp;h=499&amp;fit=640%2C499?w=640" alt="tnf_boxplot" data-image-title="tnf_boxplot" sizes="(max-width: 640px) 100vw, 640px" data-comments-opened="1" class="wp-image-977" data-recalc-dims="1"></p>
<p>While the graph shows three other outliers, they are not well away from the quartile boxes. However, the value at 1339.552 really stands out as an outlier.</p>
<p>So, what are we going to do about it?</p>
<h1>What to Do about Outliers</h1>
<p>Clearly, if a value has a reasonable scientific basis, it should not be modified even if it lies well outside the range of the other values for the sample or cohort. I normally refer to this as needing to pass the “compelling reason” test. There should be a compelling biological or medical reason to retain the data point.</p>
<p>If not, there are three commonly accepted ways of modifying outlier values.</p>
<ol>
<li>Remove the case. If you have many cases and there does not appear to be an explanation for the appearance of this value, or if the explanation is that it is in error, you can simply get rid of it.</li>
<li>Assign the next value nearer to the median in place of the outlier value. Here that would be 130.997, the next lower value. You will find that this approach leaves the distribution close to what it would be without the value. You can use this approach if you have few cases and are trying to maintain the number of observations you do have.</li>
<li>Calculate the mean of the remaining values without the outlier and assign that to the outlier case. While I have seen this frequently, I don’t really understand its justification and I think it distorts the distribution more than the previous solution.</li>
</ol>
<p>Because we want to keep as many of the few values that we have (a commonplace concern in medical studies), I would choose the second option in this case and assign the value 130.997 to the outlier case. Here is the code that would do that. Patient 7 was the person who represented the value we seek and we are putting the value of that case into the slot for case 25.</p>
<pre># substitute
tnf2 &lt;- tnf_long
# case no. of next case
tnf2$value[tnf2$id == &quot;25&quot; &amp; tnf2$visit == &quot;vis4&quot;] &lt;- 
 tnf2$value[tnf$id == &quot;7&quot; &amp; tnf2$visit == &quot;vis4&quot;]</pre>
<p>The boxplot now shows that the extreme outlier has been controlled.</p>
<p><img data-orig-file="https://madstatbr.files.wordpress.com/2018/07/tnf2_box.png?w=450&amp;h=473&amp;fit=640%2C473" data-orig-size="640,473" data-medium-file="https://madstatbr.files.wordpress.com/2018/07/tnf2_box.png?w=450&amp;h=473&amp;fit=640%2C473?w=300" data-attachment-id="978" data-image-meta="{" data-image-description="" src="https://madstatbr.files.wordpress.com/2018/07/tnf2_box.png?w=450&h=473&fit=640%2C473" data-permalink="https://madstatbr.wordpress.com/2018/07/02/handling-outliers-with-r/tnf2_box/" data-large-file="https://madstatbr.files.wordpress.com/2018/07/tnf2_box.png?w=450&amp;h=473&amp;fit=640%2C473?w=640" alt="tnf2_box" data-image-title="tnf2_box" sizes="(max-width: 640px) 100vw, 640px" data-comments-opened="1" class="wp-image-978" data-recalc-dims="1"></p>
<h1>Problem Resolved?</h1>
<p>Yes and No. We now have values for tnf that fit in the distribution more comfortably. However, we needed to manipulate a value to cause that to happen.</p>
<p>There are three conclusions I would suggest from this type of analysis:</p>
<ol>
<li>Data always needs to be checked for outliers. Without doing this, you are likely to introduce a bias that could distort the results of your study.</li>
<li>Conduct your analysis on the data both with and without the outlier. If the results are very close, you can use the original data without too many qualms. If not, you need to follow the next recommendation even more closely.</li>
<li>When you find an outlier, you should report it in your findings and everything you have done to correct for it including differences between analyses that you conducted with and without the outlier value.</li>
</ol>
<p>A last note is that canned statistical or data analysis software tends not to alert you to the presence and danger of outliers. You need to be especially vigilant when using these programs to avoid getting lulled into believing that everything is ok when it’s not. By forcing you to write your program, R avoids this problem but does not totally eliminate. You still need to take responsibility to going on an outlier hunt as you move through the data cleaning process.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://madstatbr.wordpress.com/2018/07/02/handling-outliers-with-r/"> R – MAD-Stat</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/CECUgmOdecQ" height="1"></a></div>