<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/dataexplorer-fast-data-exploration-with-minimum-code/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2018/02/dataexplorer.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p><em>by Boxuan Cui, Data Scientist at Smarter Travel</em></p>
<p>Once upon a time, there was a <a rel="nofollow" href="https://twitter.com/bigdataborat/status/306596352991830016?lang=en" target="_blank">joke</a>:</p>
<blockquote class="twitter-tweet" data-lang="en">
<p dir="ltr" lang="en">In Data Science, 80% of time spent prepare data, 20% of time spent complain about need for prepare data.</p>
<p>— Big Data Borat (@BigDataBorat) <a rel="nofollow" href="https://twitter.com/BigDataBorat/status/306596352991830016?ref_src=twsrc%5Etfw" target="_blank">February 27, 2013</a></p></blockquote>
<p></p>
<p>According to a <a rel="nofollow" href="https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says" target="_blank">Forbes article</a>, cleaning and organizing data is the most time-consuming and least enjoyable data science task. Of all the resources out there, <a rel="nofollow" href="https://mran.microsoft.com/package/DataExplorer" target="_blank">DataExplorer</a> is one of them, with its sole mission to minimize the 80%, and make it enjoyable. As a result, one fundamental design principle is to be extremely user-friendly. Most of the time, one function call is all you need.</p>
<p>Data manipulation is powered by <a rel="nofollow" href="https://cran.r-project.org/package=data.table" target="_blank">data.table</a>, so tasks involving big datasets usually complete in a few seconds. In addition, the package is flexible enough with input data classes, so you should be able to throw in any <code>data.frame</code>-like objects. However, certain functions require a <code>data.table</code> class object as input due to the <a rel="nofollow" href="https://cran.r-project.org/package=data.table/vignettes/datatable-reference-semantics.html" target="_blank">update-by-reference</a> feature, which I will cover in later part of the post.</p>
<p>Now enough said and let's look at some code, shall we?</p>
<hr>
<p>Take the <code>BostonHousing</code> dataset from the <code>mlbench</code> library:</p>
<div>
<pre><code>library(mlbench)
data(&quot;BostonHousing&quot;, package = &quot;mlbench&quot;)</code></pre>
</div>
<h3>Initial Visualization</h3>
<p>Without knowing anything about the data, my first 3 tasks are almost always:</p>
<div>
<pre><code>library(DataExplorer)
plot_missing(BostonHousing) ## Are there missing values, and what is the missing data profile?
plot_bar(BostonHousing) ## How does the categorical frequency for each discrete variable look like?
plot_histogram(BostonHousing) ## What is the distribution of each continuous variable?</code></pre>
</div>
<p>While there are not many interesting insights from <code>plot_missing</code> and <code>plot_bar</code>, below is the output from <code>plot_histogram</code>.</p>
<p><a rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d2d79d35970c-pi" target="_blank"><img border="0" alt="Histogram" title="Histogram" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d2d79d35970c-800wi"></a></p>
<p>Upon scrutiny, the variable <strong>rad</strong> looks like discrete, and I want to group <strong>crim</strong>, <strong>zn</strong>, <strong>indus</strong> and <strong>b</strong> into bins as well. Let's do so:</p>
<div>
<pre><code>## Set `rad` to factor
BostonHousing$rad &lt;- as.factor(BostonHousing$rad)

## Create new discrete variables
for (col in c(&quot;crim&quot;, &quot;zn&quot;, &quot;indus&quot;, &quot;b&quot;)) <br>  BostonHousing[[paste0(col, &quot;_d&quot;)]] &lt;- as.factor(ggplot2::cut_interval(BostonHousing[[col]], 2))

## Plot bar chart for all discrete variables
plot_bar(BostonHousing)</code></pre>
</div>
<p><a rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c94d2fa0970b-pi" target="_blank"><img border="0" alt="Bar" title="Bar" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c94d2fa0970b-800wi"></a></p>
<p>At this point, we have much better understanding of the data distribution. Now assume we are interested in <strong>medv</strong> (median value of owner-occupied homes in USD 1000's), and would like to build a model to predict it. Let's plot it against all other variables:</p>
<div>
<pre><code>plot_boxplot(BostonHousing, by = &quot;medv&quot;)    </code></pre>
</div>
<p><a rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c94d2fbc970b-pi" target="_blank"><img border="0" alt="Boxplot" title="Boxplot" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c94d2fbc970b-800wi"></a></p>
<div>
<pre><code>plot_scatterplot(<br>  subset(BostonHousing, select = -c(crim, zn, indus, b)), <br>  by = &quot;medv&quot;, size = 0.5)</code></pre>
</div>
<p><a rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c94d2fc6970b-pi" target="_blank"><img border="0" alt="Scatterplot_1" title="Scatterplot_1" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c94d2fc6970b-800wi"></a><br> <a rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c94d2fd4970b-pi" target="_blank"><img border="0" alt="Scatterplot_2" title="Scatterplot_2" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c94d2fd4970b-800wi"></a></p>
<div>
<pre><code>plot_correlation(BostonHousing)</code></pre>
</div>
<p><a rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb09f098c7970d-pi" target="_blank"><img border="0" alt="Correlation" title="Correlation" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb09f098c7970d-800wi"></a></p>
<p>And this is how you slice &amp; dice your data, and analyze correlation with merely 3 lines of code.</p>
<h3>Feature Engineering</h3>
<p>Feature engineering is a crucial step in building better models. DataExplorer provides a couple of functions to ease the process. All of them require a <code>data.table</code> as the input object, because it is lightning fast. However, if you don't feel like coding in <code>data.table</code> syntax, you may adopt the following process:</p>
<div>
<pre><code>## Set your data to `data.table` first
your_data &lt;- data.table(your_data)

## Apply DataExplorer functions
group_category(your_data, ...)
drop_columns(your_data, ...)
set_missing(your_data, ...)

## Set data back to the original object
class(your_data) &lt;- &quot;original_object_name&quot;</code></pre>
</div>
<p>Let's return to the <code>BostonHousing</code> dataset. For the rest of this section, we'll assume the data has been converted to a <code>data.table</code> already.</p>
<div>
<pre><code>library(data.table)
BostonHousingDT &lt;- data.table(BostonHousing)</code></pre>
</div>
<p>Remember those transformed continuous variables? Let's drop them:</p>
<div>
<pre><code>drop_columns(BostonHousingDT, c(&quot;crim&quot;, &quot;zn&quot;, &quot;indus&quot;, &quot;b&quot;))</code></pre>
</div>
<p>Note: Because <code>data.table</code> updates by reference, the original object is updated without the need to re-assign a returned object.</p>
<p>Let's take a look at the discrete variable <strong>rad</strong>:</p>
<div>
<pre><code>plot_bar(BostonHousingDT$rad)</code></pre>
</div>
<p><a rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c94d2fee970b-pi" target="_blank"><img border="0" alt="Rad_bar" title="Rad_bar" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c94d2fee970b-800wi"></a></p>
<p>I think categories other than 4, 5 and 24 are too sparse, and might skew my model fit. How could I group all the sparse categories together?</p>
<div>
<pre><code>group_category(BostonHousingDT, &quot;rad&quot;, 0.25, update = FALSE)

#    rad cnt       pct   cum_pct
# 1:  24 132 0.2608696 0.2608696
# 2:   5 115 0.2272727 0.4881423
# 3:   4 110 0.2173913 0.7055336</code></pre>
</div>
<p>Looks like grouping by bottom 25% of <strong>rad</strong> would give me what I need. Let's do so:</p>
<div>
<pre><code>group_category(BostonHousingDT, &quot;rad&quot;, 0.25, update = TRUE)
plot_bar(BostonHousingDT$rad)</code></pre>
</div>
<p><a rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c94d2ff8970b-pi" target="_blank"><img border="0" alt="Grouped_rad_bar" title="Grouped_rad_bar" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c94d2ff8970b-800wi"></a></p>
<p>In addition to categorical frequency, you may also play with the <code>measure</code> argument to group by the sum of a different variable. See <code>?group_category</code> for more example use cases.</p>
<h3>Data Report</h3>
<p>To generate a report of your data:</p>
<div>
<pre><code>create_report(BostonHousing)</code></pre>
</div>
<p>Currently, there is not much to do with this, but it is my plan to support <a rel="nofollow" href="https://github.com/boxuancui/DataExplorer/issues/41" target="_blank">customization</a> of the generated report, so stay tuned for more features!</p>
<hr>
<p>I hope you enjoyed exploring the Boston housing data with me, and finally here are some additional resources about the DataExplorer package:</p>
<ul>
<li>CRAN: <a rel="nofollow" href="https://cran.r-project.org/package=DataExplorer" target="_blank">https://cran.r-project.org/package=DataExplorer</a></li>
<li>Vignette: <a rel="nofollow" href="https://cran.r-project.org/package=DataExplorer/vignettes/dataexplorer-intro.html" target="_blank">https://cran.r-project.org/package=DataExplorer/vignettes/dataexplorer-intro.html</a></li>
<li>GitHub: <a rel="nofollow" href="https://github.com/boxuancui/DataExplorer" target="_blank">https://github.com/boxuancui/DataExplorer</a></li>
<li>Issues/Ideas/Suggestions: <a rel="nofollow" href="https://github.com/boxuancui/DataExplorer/issues" target="_blank">https://github.com/boxuancui/DataExplorer/issues</a></li>
</ul>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2018/02/dataexplorer.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/MB0WElVubJA" height="1"></a></div>