<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/outlier-app-an-interactive-visualization-of-outlier-algorithms/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://datascienceplus.com/outlier-app-an-interactive-visualization-of-outlier-algorithms/"> DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I was recently trying various outlier detection algorithms. For me, the best way to understand an algorithm is to tinker with it. I built a shiny app that allows you to play around with various outlier algorithms and wanted to share it with everyone.</p>
<p>The <a target="_blank" rel="nofollow" href="http://projects.rajivshah.com/shiny/outlier/">shiny app is available on my site</a>, but even better, <a target="_blank" rel="nofollow" href="https://github.com/rajshah4/outliers_shiny">the code is on github</a> for you to run locally or improve! Let me give you a quick tour of the app in this post. If you prefer, I have also <a title="" data-rel="nofollow lightbox-0" target="_blank" rel="nofollow" href="https://www.youtube.com/watch?v=1zPuRAgr1F4">posted a video</a> that provides background on the app. Another tutorial how to build a interactive web <a target="_blank" rel="nofollow" href="http://datascienceplus.com/building-interactive-web-apps-with-shiny/">apps with shiny</a> is published at DataScience+.</p>
<h2>Background</h2>
<p><strong>The available algorithms include:</strong></p>
<blockquote><p>
– Hierarchical Clustering (DMwR)<br>
– Kmeans Euclidean Distance<br>
– Kmeans Mahalanobis<br>
– Kmeans Manhattan<br>
– Fuzzy kmeans – Gustafson and Kessel<br>
– Fuzzy k-medoids<br>
– Fuzzy k-means with polynomial fuzzifier<br>
– Local Outlier Factor (dbscan)<br>
– RandomForest (proximity from randomForest)<br>
– Isolation Forest (IsolationForest)<br>
– Autoencoder (Autoencoder)<br>
– FBOD and SOD (HighDimOut)</p>
</blockquote>
<p><a target="_blank" rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2016/12/algorithms.gif"><img alt="" class="wp-image-5022" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2016/12/algorithms.gif?resize=456%2C237" data-recalc-dims="1"></a></p>
<p>There are also a wide range of datasets to try as well. They include randomly scattered points, defined clusters, and some more unusual patterns like the smiley face and spirals. Additionally, you can use your mouse to add and/or remove points by clicking directly within the visualization. This allows you to create your own dataset.</p>
<p><a title="" data-rel="lightbox-1" target="_blank" rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2016/12/datasets.png"><img sizes="(max-width: 490px) 100vw, 490px" alt="" class="wp-image-5023" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2016/12/datasets-490x283.png?resize=456%2C263" data-recalc-dims="1"></a></p>
<h2>Using the app</h2>
<p>Once the data is loaded, you can start exploring. One thing you can do is look at the effect scaling can have. In this example, you can see how outliers differ when scaling is used with kmeans. The values on the far right no longer dominate the distance measurements, and there are now outliers from other areas:</p>
<p><a title="" data-rel="lightbox-2" target="_blank" rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2016/12/scaling.gif"><img alt="" class="wp-image-5024" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2016/12/scaling.gif?resize=456%2C250" data-recalc-dims="1"></a></p>
<p>It quickly becomes apparent that different algorithms may select different outliers. In this case, you see a difference between the outliers selected using an autoencoder versus isolation forest.<br>
<a title="" data-rel="lightbox-3" target="_blank" rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2016/12/auto_iso.gif"><img alt="" class="wp-image-5025" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2016/12/auto_iso.gif?resize=456%2C253" data-recalc-dims="1"></a></p>
<p>Another example here is the difference between chosen outliers using kmeans versus fuzzy kmeans:<br>
<a title="" data-rel="lightbox-4" target="_blank" rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2016/12/fuzzy.gif"><img alt="" class="wp-image-5026" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2016/12/fuzzy.gif?resize=456%2C250" data-recalc-dims="1"></a></p>
<p>A density based algorithm can also select different outliers versus a distance based algorithm. This example nicely shows the difference between kmeans and lof (local outlier factor from dbscan)<br>
<a title="" data-rel="lightbox-5" target="_blank" rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2016/12/density.gif"><img alt="" class="wp-image-5027" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2016/12/density.gif?resize=456%2C253" data-recalc-dims="1"></a></p>
<p>An important part of using this visualization is studying the distance numbers that are calculated. Are these numbers meshing with your intuition? How big of a quantitative difference is there between outliers and other points?<br>
<a title="" data-rel="lightbox-6" target="_blank" rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2016/12/outlier_table.png"><img sizes="(max-width: 490px) 100vw, 490px" alt="" class="wp-image-5028" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2016/12/outlier_table-490x383.png?resize=456%2C356" data-recalc-dims="1"></a></p>
<h2>3D+ App?</h2>
<p>The next thing is whether to expand this to larger datasets. This is something that you would run locally (large datasets take too long to run for my shiny server). The downside of larger datasets is that it gets tricker to visualize them. For now, I am using a TSNE plot. I am open to suggestions, but the intent here is a way to evaluate outlier algorithms on a variety of datasets.<br>
<a title="" data-rel="lightbox-7" target="_blank" rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2016/12/multiD.gif"><img alt="" class="wp-image-5029" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2016/12/multiD.gif?resize=456%2C304" data-recalc-dims="1"></a></p>
<h2>Source Code</h2>
<p>The source code for the outlier app <a target="_blank" rel="nofollow" href="https://github.com/rajshah4/outliers_shiny">is on github</a>. The app is built off a variety of R packages and could easily be extended to new packages or incorporate additional datasets. Please send me bug fixes, additional algorithms, tighter code, or ideas for improving the app.</p>
<ul></ul>
<p><strong>
<p>Related Post</p>
<p>
<ol>
<li><a title="Permanent Link toCreating an animation using R" target="_blank" rel="nofollow" href="http://datascienceplus.com/creating-an-animation-using-r/">Creating an animation using R</a></li>
<li><a title="Permanent Link toThe importance of Data Visualization" target="_blank" rel="nofollow" href="http://datascienceplus.com/the-importance-of-data-visualization/">The importance of Data Visualization</a></li>
<li><a title="Permanent Link toggplot2 themes examples" target="_blank" rel="nofollow" href="http://datascienceplus.com/ggplot2-themes-examples/">ggplot2 themes examples</a></li>
<li><a title="Permanent Link toMap the Life Expectancy in United States with data from Wikipedia" target="_blank" rel="nofollow" href="http://datascienceplus.com/map-the-life-expectancy-in-united-states-with-data-from-wikipedia/">Map the Life Expectancy in United States with data from Wikipedia</a></li>
<li><a title="Permanent Link toWhat can we learn from the statistics of the EURO 2016 – Application of factor analysis" target="_blank" rel="nofollow" href="http://datascienceplus.com/what-can-we-learn-from-the-statistics-of-the-euro-2016-application-of-factor-analysis/">What can we learn from the statistics of the EURO 2016 – Application of factor analysis</a></li>
</ol></strong>
<div>
<a target="_blank" rel="nofollow" href="http://feeds.feedburner.com/~ff/datascienceplus?a=u7hEEpHVoTc:bmcEsVIDnkw:D7DqB2pKExk"><img src="http://feeds.feedburner.com/~ff/datascienceplus?i=u7hEEpHVoTc:bmcEsVIDnkw:D7DqB2pKExk" border="0"></a> <a target="_blank" rel="nofollow" href="http://feeds.feedburner.com/~ff/datascienceplus?a=u7hEEpHVoTc:bmcEsVIDnkw:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/datascienceplus?d=qj6IDK7rITs" border="0"></a> <a target="_blank" rel="nofollow" href="http://feeds.feedburner.com/~ff/datascienceplus?a=u7hEEpHVoTc:bmcEsVIDnkw:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/datascienceplus?d=yIl2AUoC8zA" border="0"></a>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://datascienceplus.com/outlier-app-an-interactive-visualization-of-outlier-algorithms/"> DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/UTKStT0RpsA"></a></div>