<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/descriptive-analytics-part-3-outlier-treatment/"></div></div><div><a data-url="https://www.r-bloggers.com/descriptive-analytics-part-3-outlier-treatment/" data-counturl="https://www.r-bloggers.com/descriptive-analytics-part-3-outlier-treatment/" data-count="horizontal" data-text="Descriptive Analytics-Part 3 : Outlier treatment" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://r-exercises.com/2016/11/10/descriptive-analytics-part-3-outlier-treatment/"> R-exercises</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a target="_blank" rel="nofollow" href="https://i0.wp.com/r-exercises.com/wp-content/uploads/2016/10/download.jpg?resize=278%2C181"><img alt="download" class="wp-image-2350" src="https://i0.wp.com/r-exercises.com/wp-content/uploads/2016/10/download.jpg?resize=278%2C181" data-recalc-dims="1"></a>Descriptive Analytics is the examination of data or content, usually manually performed, to answer the question “What happened?”.</p>
<p>In order to be able to solve this set of exercises you should have solved the <a target="_blank" rel="nofollow" href="http://r-exercises.com/2016/10/19/descriptive-analytics-part-0-data-exploration/">part 0</a> and’<a target="_blank" rel="nofollow" href="http://r-exercises.com/2016/10/26/descriptive-analytics-part-1-data-formatting/">part 1</a> , and <a target="_blank" rel="nofollow" href="http://r-exercises.com/2016/11/02/descriptive-analytics-part-2-data-imputation/">part 2</a> of this series but also you should run this <a target="_blank" rel="nofollow" href="https://github.com/VasTsak/r-exercises/blob/master/descriptive-analytics-part-3-completed">script</a> which contain some more data cleaning. In case you haven’t, run this <a target="_blank" rel="nofollow" href="https://github.com/VasTsak/r-exercises/blob/master/Descriptive%20Analytics-Part%203"> script</a> in your machine which contains the lines of code we used to modify our data set. This is the fourth set of exercise of a series of exercises that aims to provide a descriptive analytics solution to the ‘2008’ data set from <a target="_blank" rel="nofollow" href="http://stat-computing.org/dataexpo/2009/the-data.html">here</a>. This data set which contains the arrival and departure information for all domestic flights in the US from 2008 has become the “iris” data set for Big Data. Outliers treatment is a vital part of descriptive analytics since outliers can lead to misleading conclusions regarding our data. So it is an important skill to have in your skill set. The following exercise demonstrates some of the basic and fairly simplistic methods of treating outliers. For more sophisticated methods of dealing with outliers check out <a target="_blank" rel="nofollow" href="http://www.dbs.ifi.lmu.de/~zimek/publications/KDD2010/kdd10-outlier-tutorial.pdf">this </a>. But keep in mind that many people claim that ‘eyes beat maths’ when it comes to outliers. Before proceeding, it might be helpful to look over the help pages for the <code> table</code>, <code> subset</code>,<code>boxplot.stats</code>, <code> %in%</code>, <code> ifelse</code>, <code> rp.outlier</code>, <code> scores</code>.</p>
<p>For this set of exercises you will need to install and load the package <code> rapportools</code>,<code> outliers</code>.</p>
<p><code>install.packages('rapportools')</code><br>
<code>library(rapportools)</code><br>
<code>install.packages('outliers')</code><br>
<code>library(outliers)</code></p>
<p>Answers to the exercises are available <a target="_blank" rel="nofollow" href="http://r-exercises.com/2016/11/10/descriptive-analytics-part-3-outlier-treatment-solutions/">here</a>.</p>
<p>If you obtained a different (correct) answer than those listed on the solutions page, please feel free to post your answer as a comment on that page.</p>
<p><strong>Exercise 1</strong><br>
Print the summary statistics and the structure of the dataset in order to see the type of variables and their extreme values, whether it makes sense or not .</p>
<p><strong>Exercise 2</strong><br>
When it comes to categorical variables, outliers are considered to be the values of which frequency is less than 10% , <code>barplot</code> of <code>flights$UniqueCarrier</code> and <code>flights$CancellationCode</code>. What do you think? There are more categorical variables , so I encourage you to try them out as well. </p>
<p><strong>Exercise 3</strong><br>
Remove the outliers that you have noticed at the barplots of the previous exercise, consider the function <code>subset</code>. </p>
<p><strong>Exercise 4</strong><br>
A good way of detecting outliers from numerical variables is <code>boxplot</code>, make one with <code>flights$ActualElapsedTime</code>. </p>
<p><strong>Exercise 5</strong><br>
Remove the outliers of <code>flights$ActualElapsedTime</code> using <code>boxplot.stats</code> .</p>
<p><strong>Exercise 6</strong><br>
Remove outliers from <code>flights</code> using the <code>subset</code> function ,where <code> TaxiIn </code> is greater than 0 and less than 120.</p>
<p><strong>Exercise 7</strong><br>
Remove outliers from <code>flights</code> using the <code>subset</code> function ,where <code> TaxiOut </code> is greater than 0 and less than 50.</p>
<p><strong>Exercise 8</strong><br>
Assign NA value if the value is an outlier of <code>flights_exp$ArrDelay</code> using the <code>ifelse</code> function.</p>
<p><strong>Exercise 9</strong><br>
Use the <code>rp.outlier</code> to detect and remove the outliers using the Lund Test from <code>flights_exp$Distance</code> , use the <code>rapportools</code>.</p>
<p><strong>Exercise 10</strong><br>
Find the 2% most extreme values of <code>flights$CRSElapsedTime</code> using the <code>scores</code> with chi-square method.</p>
<div>
<h4>Please help us to improve R-exercises:</h4>

</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/descriptive-analytics-part-3-outlier-treatment/"></div></div><div><a data-url="https://www.r-bloggers.com/descriptive-analytics-part-3-outlier-treatment/" data-counturl="https://www.r-bloggers.com/descriptive-analytics-part-3-outlier-treatment/" data-count="vertical" data-text="Descriptive Analytics-Part 3 : Outlier treatment" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://r-exercises.com/2016/11/10/descriptive-analytics-part-3-outlier-treatment/"> R-exercises</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/R60h5gJjIUk"></a></div>