<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/preparing-the-data-for-modelling-with-r/"></div></div><div><a data-url="http://www.r-bloggers.com/preparing-the-data-for-modelling-with-r/" data-counturl="http://www.r-bloggers.com/preparing-the-data-for-modelling-with-r/" data-count="horizontal" data-text="Preparing the data for modelling with R" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.milanor.net/blog/preparing-the-data-for-modelling-with-r/"> MilanoR</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<div>
<h3>Why data preparation</h3>
</div>
<div>
<p>One of the first things which I came across while studying about data science was that three important steps in a data science project is data preparation, creating &amp; testing the model and reporting. It is a widely accepted fact that data preparation takes up most of the time followed by creating the model and then reporting. There were opinions which says we should try to reduce the time taken for data preparation which we can use for creating and testing the model. But a model is only as good as the data on which it is created. A simpler model based on clean data will most likely outperform a complicated model based on dirty or ambiguous data.</p>
</div>
<div>
<h3>Big Mart Sales dataset</h3>
<p>With an example from a regression problem to predict the sales, we can go through some of the common situations we might face while creating a good data set. The dataset which I am using is taken from <a target="_blank" href="http://www.analyticsvidhya.com/">http://www.analyticsvidhya.com/</a> Big Mart Sales prediction problem which I have modified a bit to include some outliers in the response variable <code>Item_Outlet_Sales</code></p>
<div>
<h3>Data description</h3>
<p><strong>Item_Identifier</strong> : Unique product ID<br>
<strong>Item_Weight</strong> : Weight of product<br>
<strong>Item_Fat_Content</strong> : Whether the product is low fat or not<br>
<strong>Item_Visibility</strong> : The % of total display area of all products in a store allocated to the particular product<br>
<strong>Item_Type</strong> : The category to which the product belongs<br>
<strong>Item_MRP</strong> : Maximum Retail Price (list price) of the product<br>
<strong>Outlet_Identifier</strong> : Unique store ID<br>
<strong>Outlet_Establishment_Year</strong> : The year in which store was established<br>
<strong>Outlet_Size</strong> : The size of the store in terms of ground area covered<br>
<strong>Outlet_Location_Type</strong> : The type of city in which the store is located<br>
<strong>Outlet_Type</strong> : Whether the outlet is just a grocery store or some sort of supermarket<br>
<strong>Item_Outlet_Sales</strong> : Sales of the product in the particulat store. This is the outcome variable to be predicted</p>
</div>
</div>
<div>
<h2>Dealing with missing values</h2>
<p>First let’s take a look at the <strong>missing values</strong>. If number of observations with missing values are much lower than the total number of observations, then there’s not much loss of information by dropping them. I am using the function <code>complete.cases()</code> to check for rows without missing values. The function returns a logical vector indicating which cases are complete, i.e., have no missing values. Please note that this function looks for NULL/NA value and there might be missing values in other forms like blanks in character factor columns.</p>
<pre>nrows &lt;- nrow(Data)
ncomplete &lt;- sum(complete.cases(Data))
ncomplete</pre>

<pre>## [1] 7060</pre>

<pre>ncomplete/nrows</pre>

<pre>## [1] 0.8283468</pre>
<p>Here we can see that by dropping all the rows with missing values, we are losing about 18% of data. So we cannot drop them.</p>
</div>
<div>
<h2>Data Exploration</h2>
<p>Now let’s have a proper look into the data set. We can begin with the response variable. From the information we have, it is a continuos variable. I am using the <code>ggplot2</code> package for data visualization which I believe most of you would be familiar with. I will be showing the distribution of the dependent variable <code>Item_Outlet_Sales</code></p>
<pre>library(ggplot2)
#Plotting the dependent variable distribution
pl1 &lt;- ggplot(Data, aes(Item_Outlet_Sales))
pl1 + geom_density(fill = &quot;red&quot;, alpha = &quot;0.7&quot;)</pre>
<p> </p>
<p><a target="_blank" rel="attachment wp-att-2294" href="http://i1.wp.com/www.milanor.net/blog/wp-content/uploads/2016/05/1-data-preparation.png"><img sizes="(max-width: 642px) 100vw, 642px" srcset="http://www.milanor.net/blog/wp-content/uploads/2016/05/1-data-preparation-300x214.png 300w, http://www.milanor.net/blog/wp-content/uploads/2016/05/1-data-preparation-768x549.png 768w, http://www.milanor.net/blog/wp-content/uploads/2016/05/1-data-preparation-1024x731.png 1024w, http://i1.wp.com/www.milanor.net/blog/wp-content/uploads/2016/05/1-data-preparation.png?resize=456%2C325 1344w" alt="1-data-preparation" class="wp-image-2294" src="http://i1.wp.com/www.milanor.net/blog/wp-content/uploads/2016/05/1-data-preparation.png?resize=456%2C325" data-recalc-dims="1"></a></p>
<p>Here we can see that the distribution looks similar to a half normal distribution. If we take a closer look, we can see that there is a sudden spike towards the right end of the graph. This might possibly be a sentinel value. A <strong>sentinel value</strong> is a special kind of bad numerical value: a value that used to represent “unknown” or “not applicable”” or other special cases in numeric data. One way to detect sentinel values is to look for sudden jumps in an otherwise smooth distribution of values. We can now take a look into the summary of the <code>Item_Outlet_Sales</code> variable to confirm this</p>
<pre>summary(Data$Item_Outlet_Sales)</pre>

<pre>##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
##    33.29   834.90  1798.00  2218.00  3104.00 33330.00</pre>
<p>Here we can see that the maximum value is 33333 which is nowhere closer to the other values.<br>
We can now examine these values to check whether they follow a pattern. If not, we can drop them.<br>
I am now using the <a target="_blank" href="https://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html"><code>dplyr</code> package</a>. If anyone is not familiar with it, please go through the package help. The documentation is comprehensive<br>
The <code>filter()</code> function in dplyr helps us to subset the data based column values</p>
<pre>library(dplyr)
#Creating a data frame with only the outliers
outlier &lt;- Data %>% filter(Item_Outlet_Sales == 33333)
outlier</pre>

<pre>##    Item_Identifier Item_Weight Item_Fat_Content Item_Visibility
## 1            DRI11          NA          Low Fat      0.03423768
## 2            FDP25      15.200          Low Fat      0.02132748
## 3            FDU55      16.200          Low Fat      0.03598410
## 4            FDF44       7.170          Regular      0.05997133
## 5            FDE41          NA              reg      0.00000000
## 6            FDK04       7.360          Low Fat      0.05260793
## 7            FDJ53      10.500          Low Fat      0.07125791
## 8            FDX08      12.850          Low Fat      0.02264989
## 9            FDE53      10.895          Low Fat      0.02703220
## 10           FDA31       7.100          Low Fat      0.11023479
##                Item_Type Item_MRP Outlet_Identifier
## 1            Hard Drinks 113.2834            OUT027
## 2                 Canned 216.8824            OUT017
## 3  Fruits and Vegetables 260.6278            OUT045
## 4  Fruits and Vegetables 132.1968            OUT018
## 5           Frozen Foods  83.7566            OUT019
## 6           Frozen Foods  56.3588            OUT017
## 7           Frozen Foods 121.3098            OUT046
## 8  Fruits and Vegetables 179.3318            OUT045
## 9           Frozen Foods 106.3280            OUT017
## 10 Fruits and Vegetables 171.7080            OUT045
##    Outlet_Establishment_Year Outlet_Size Outlet_Location_Type
## 1                       1985      Medium               Tier 3
## 2                       2007                           Tier 2
## 3                       2002                           Tier 2
## 4                       2009      Medium               Tier 3
## 5                       1985       Small               Tier 1
## 6                       2007                           Tier 2
## 7                       1997       Small               Tier 1
## 8                       2002                           Tier 2
## 9                       2007                           Tier 2
## 10                      2002                           Tier 2
##          Outlet_Type Item_Outlet_Sales
## 1  Supermarket Type3             33333
## 2  Supermarket Type1             33333
## 3  Supermarket Type1             33333
## 4  Supermarket Type2             33333
## 5      Grocery Store             33333
## 6  Supermarket Type1             33333
## 7  Supermarket Type1             33333
## 8  Supermarket Type1             33333
## 9  Supermarket Type1             33333
## 10 Supermarket Type1             33333</pre>
<p>There are only 10 observations which has these sentinel values. We can see that <code>Item_Type</code>, <code>Item_MRP</code>, <code>Outlet_Location_Type</code>, <code>Item_Weight</code>, <code>Outlet_Type</code> are all different among these outliers. So, it does not look like wrong data from a particular store or location. Let`s drop them as them. Dealing with these type of values normally require domain knowledge.</p>
<pre>#Removing the observations with sentinel value
Data &lt;- Data %>% filter(Item_Outlet_Sales != 33333)
pl2 &lt;- ggplot(Data, aes(Item_Outlet_Sales))
pl2 + geom_density(fill = &quot;blue&quot;, alpha = &quot;0.5&quot;)</pre>
<p><a target="_blank" rel="attachment wp-att-2295" href="http://i1.wp.com/www.milanor.net/blog/wp-content/uploads/2016/05/2-data-preparation.png"><img sizes="(max-width: 1344px) 100vw, 1344px" srcset="http://www.milanor.net/blog/wp-content/uploads/2016/05/2-data-preparation-300x214.png 300w, http://www.milanor.net/blog/wp-content/uploads/2016/05/2-data-preparation-768x549.png 768w, http://www.milanor.net/blog/wp-content/uploads/2016/05/2-data-preparation-1024x731.png 1024w, http://i1.wp.com/www.milanor.net/blog/wp-content/uploads/2016/05/2-data-preparation.png?resize=456%2C326 1344w" alt="2-data-preparation" class="wp-image-2295" src="http://i1.wp.com/www.milanor.net/blog/wp-content/uploads/2016/05/2-data-preparation.png?resize=456%2C326" data-recalc-dims="1"></a></p>
<p> </p>
<pre>summary(Data$Item_Outlet_Sales)</pre>

<pre>##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
##    33.29   833.60  1794.00  2182.00  3101.00 13090.00</pre>
<p>Now we can explore the remaining variables.</p>
<pre>summary(Data)</pre>

<pre>##  Item_Identifier  Item_Weight     Item_Fat_Content Item_Visibility  
##  FDG33  :  10    Min.   : 4.555   LF     : 316     Min.   :0.00000  
##  FDW13  :  10    1st Qu.: 8.775   low fat: 112     1st Qu.:0.02699  
##  DRE49  :   9    Median :12.600   Low Fat:5081     Median :0.05395  
##  DRN47  :   9    Mean   :12.860   reg    : 116     Mean   :0.06616  
##  FDD38  :   9    3rd Qu.:16.850   Regular:2888     3rd Qu.:0.09466  
##  FDF52  :   9    Max.   :21.350                    Max.   :0.32839  
##  (Other):8457    NA's   :1461                                       
##                  Item_Type       Item_MRP      Outlet_Identifier
##  Fruits and Vegetables:1228   Min.   : 31.29   OUT027 : 934     
##  Snack Foods          :1200   1st Qu.: 93.81   OUT013 : 932     
##  Household            : 910   Median :143.02   OUT035 : 930     
##  Frozen Foods         : 852   Mean   :140.99   OUT049 : 930     
##  Dairy                : 682   3rd Qu.:185.66   OUT046 : 929     
##  Baking Goods         : 648   Max.   :266.89   OUT018 : 927     
##  (Other)              :2993                    (Other):2931     
##  Outlet_Establishment_Year Outlet_Size   Outlet_Location_Type
##  Min.   :1985                    :2404   Tier 1:2386         
##  1st Qu.:1987              High  : 932   Tier 2:2779         
##  Median :1999              Medium:2791   Tier 3:3348         
##  Mean   :1998              Small :2386                       
##  3rd Qu.:2004                                                
##  Max.   :2009                                                
##                                                              
##             Outlet_Type   Item_Outlet_Sales 
##  Grocery Store    :1082   Min.   :   33.29  
##  Supermarket Type1:5570   1st Qu.:  833.58  
##  Supermarket Type2: 927   Median : 1794.33  
##  Supermarket Type3: 934   Mean   : 2181.56  
##                           3rd Qu.: 3101.30  
##                           Max.   :13086.97  
##</pre>
<p>Looking at the <code>Item_Weight</code> variable, we can see that there are 1461 missing values. We can also see that the <code>Item_Fat_Content</code> variable is coded incorrectly. The same factor levels are coded in different ways.<br>
There are 2404 missing values in the <code>Outlet_Size</code> variable. Another interesting thing is about the <code>Item_Visibility</code> variable. In my opinion, there can’t be any item with 0 visibility as no item in a supermarket or grocery store is supposed to be completely invisible to customers.</p>
<p>Let’s treat them one by one:</p>
<p>First let’s recode the <code>Item_Fat_Content</code> variable. There are 2 levels- <code>Regular</code> and <code>low fat</code> which are coded into 5 different levels named as <code>LF</code>, <code>low fat</code>, <code>Low Fat</code>, <code>reg</code> and <code>Regular</code>. We can <strong>recode</strong> them into lowfat</p>
<pre>#with gsub replacing the levels with Regular or lowfat as required
Data$Item_Fat_Content &lt;- gsub(&quot;LF&quot;, &quot;lowfat&quot;,Data$Item_Fat_Content)
Data$Item_Fat_Content &lt;- gsub(&quot;low fat&quot;, &quot;lowfat&quot;,Data$Item_Fat_Content)
Data$Item_Fat_Content &lt;- gsub(&quot;Low Fat&quot;, &quot;lowfat&quot;,Data$Item_Fat_Content)
Data$Item_Fat_Content &lt;- gsub(&quot;reg&quot;, &quot;Regular&quot;,Data$Item_Fat_Content)
Data$Item_Fat_Content &lt;- as.factor(Data$Item_Fat_Content)
summary(Data$Item_Fat_Content)</pre>

<pre>##  lowfat Regular 
##    5509    3004</pre>
<p>Now let us <strong>replace the missing values</strong> in the <code>Item_Weight</code> variable. There are many ways to deal with missing values in a continuous variable which includes mean replacement, median replacement, replacing with an arbitrary constant, regression methods etc. I will be using mean replacement and regression in this example. I am using mean replacement for <code>Item_Weight</code> and regression for <code>Item_Visibility.</code> In real projects, these methods are chosen based on requirements. Normally we use mean replacement for variable which have lower predictive power for the final response variable.</p>
<pre>#Using mean to replace the missing values in Item_Weight variable
MeanItem_Weight &lt;- mean(Data$Item_Weight[!is.na(Data$Item_Weight)])
Data$Item_Weight[is.na(Data$Item_Weight)] &lt;- MeanItem_Weight
#Using regression to replace the zeros in Item_visibility variable
Data_1 &lt;- Data %>% filter(Item_Visibility != 0)
visibility_model &lt;- lm(Item_Visibility ~ Item_Weight + Item_Fat_Content +
                         Item_Type + Item_MRP + 
                         Outlet_Establishment_Year + Outlet_Size + 
                         Outlet_Location_Type + Item_Outlet_Sales,
                         data = Data_1)
Data$Item_Visibility[Data$Item_Visibility == 0] &lt;- 
  predict(visibility_model,newdata = Data[Data$Item_Visibility == 0,])</pre>
<p>Finally we have to <strong>classify the missing values</strong> in the <code>Outlet_Size variable</code>.<br>
I am using the random forest algorithm for classification. In my experience, the <strong>random forest algorithm</strong> has worked well for classification models as it has the advantage of being an ensemble model. I am using the <code>randomForest</code> package as it has a very good implementation of the random forest algorithm. The dataset is split to train and test set using the package <code>caTools</code>. The caTools package is a very good tool for splitting our dataset for machine learning algorithms.<br>
The function <code>sample.split()</code> is used for splitting. Two subsets are made which are classified as <code>TRUE</code> and <code>FALSE</code>. Normally we use the <code>TRUE</code> subset for training and <code>FALSE</code> subset for testing</p>
<pre>library(caTools)
set.seed(100)
Data$Outlet_Size &lt;- as.character(Data$Outlet_Size)
Storetypes &lt;- subset(Data, Outlet_Size != &quot;&quot;)
spl &lt;- sample.split(Storetypes$Outlet_Size, SplitRatio = 0.8)
Train &lt;- subset(Storetypes, spl == TRUE)
Test &lt;- subset(Storetypes, spl == FALSE)
###Using Random Forest for classification
library(randomForest)
Train$Outlet_Size &lt;- as.factor(Train$Outlet_Size)
Test$Outlet_Size &lt;- as.factor(Test$Outlet_Size)
###Creating the model
SizeForest &lt;- randomForest(Outlet_Size ~.-Item_Outlet_Sales -Item_Identifier,
                           data =  Train,nodesize = 25, ntree = 100)  
###Predicting on the test set
PredictForest &lt;- predict(SizeForest, newdata = Test)
#Confusion matrix for accuracy
table(Test$Outlet_Size, PredictForest)</pre>

<pre>##         PredictForest
##          High Medium Small
##   High    186      0     0
##   Medium    0    558     0
##   Small     0      0   477</pre>

<pre>###Classifying the missing values in the dataset
Data$Outlet_Size &lt;- predict(SizeForest, newdata =Data)
######</pre>
<p>Now we can check the complete dataset once more. We can see that the problem of missing values are resolved and the factors are well coded.</p>
<pre>summary(Data)</pre>

<pre>##  Item_Identifier  Item_Weight     Item_Fat_Content Item_Visibility   
##  FDG33  :  10    Min.   : 4.555   lowfat :5509     Min.   :0.003575  
##  FDW13  :  10    1st Qu.: 9.310   Regular:3004     1st Qu.:0.033088  
##  DRE49  :   9    Median :12.860                    Median :0.060615  
##  DRN47  :   9    Mean   :12.860                    Mean   :0.070478  
##  FDD38  :   9    3rd Qu.:16.000                    3rd Qu.:0.096000  
##  FDF52  :   9    Max.   :21.350                    Max.   :0.328391  
##  (Other):8457                                                        
##                  Item_Type       Item_MRP      Outlet_Identifier
##  Fruits and Vegetables:1228   Min.   : 31.29   OUT027 : 934     
##  Snack Foods          :1200   1st Qu.: 93.81   OUT013 : 932     
##  Household            : 910   Median :143.02   OUT035 : 930     
##  Frozen Foods         : 852   Mean   :140.99   OUT049 : 930     
##  Dairy                : 682   3rd Qu.:185.66   OUT046 : 929     
##  Baking Goods         : 648   Max.   :266.89   OUT018 : 927     
##  (Other)              :2993                    (Other):2931     
##  Outlet_Establishment_Year Outlet_Size   Outlet_Location_Type
##  Min.   :1985              High  : 932   Tier 1:2386         
##  1st Qu.:1987              Medium:5195   Tier 2:2779         
##  Median :1999              Small :2386   Tier 3:3348         
##  Mean   :1998                                                
##  3rd Qu.:2004                                                
##  Max.   :2009                                                
##                                                              
##             Outlet_Type   Item_Outlet_Sales 
##  Grocery Store    :1082   Min.   :   33.29  
##  Supermarket Type1:5570   1st Qu.:  833.58  
##  Supermarket Type2: 927   Median : 1794.33  
##  Supermarket Type3: 934   Mean   : 2181.56  
##                           3rd Qu.: 3101.30  
##                           Max.   :13086.97  
##</pre>
<p>The dataset is now ready for modelling….</p>
</div>
</div>
<p>The post <a target="_blank" rel="nofollow" href="http://www.milanor.net/blog/preparing-the-data-for-modelling-with-r/">Preparing the data for modelling with R</a> appeared first on <a target="_blank" rel="nofollow" href="http://www.milanor.net/blog">MilanoR</a>.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/preparing-the-data-for-modelling-with-r/"></div></div><div><a data-url="http://www.r-bloggers.com/preparing-the-data-for-modelling-with-r/" data-counturl="http://www.r-bloggers.com/preparing-the-data-for-modelling-with-r/" data-count="vertical" data-text="Preparing the data for modelling with R" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.milanor.net/blog/preparing-the-data-for-modelling-with-r/"> MilanoR</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/D6CAiwntHUc"></a></div>