<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/autoencoders-and-anomaly-detection-with-machine-learning-in-fraud-analytics/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://shiring.github.io/machine_learning/2017/05/01/fraud"> Shirin's playgRound</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>All my previous posts on machine learning have dealt with supervised learning. But we can also use machine learning for unsupervised learning. The latter are e.g. used for clustering and (non-linear) dimensionality reduction.</p>
<p>For this task, I am using <a target="_blank" rel="nofollow" href="https://www.kaggle.com/dalpozz/creditcardfraud">Kaggle’s credit card fraud dataset</a> from the following study:</p>
<blockquote>
<p>Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi. Calibrating Probability with Undersampling for Unbalanced Classification. In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015</p>
</blockquote>
<p>The dataset gives > 280,000 instances of credit card use and for each transaction, we know whether it was fraudulent or not.</p>
<p>Datasets like this needs special treatment when performing machine learning because they are severely unbalanced: in this case, only 0.17% of all transactions are fraudulent.</p>
<p>While we could try to work with classifiers, like random forests or support vector machines, by applying <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2017/04/02/unbalanced">over- or under-sampling techniques</a>, we can alternatively try to find anomalies in the data (assuming we expect our fraud cases to be anomalies within the whole dataset).</p>
<p>When dealing with such a severe unbalance of response labels, we also need to be careful when measuring model performance. Because there are only a handful of fraudulent instances, a model that predicts everything as non-fraud will already achieve a > 99% accuracy. But despite its high accuracy, such a model won’t necessarily help us find fraudulent cases – the proverbial “needle-in-a-haystack” – that we actually want to find!</p>
<p>Below, I will show how you can use autoencoders and anomaly detection, how you can use autoencoders to pre-train a classification model and how you can measure model performance on unbalanced data.</p>
<p></p>
<h2>Exploring the data</h2>
<div>
<pre><code><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span># download from https://www.kaggle.com/dalpozz/creditcardfraud
</span><span>creditcard</span><span> </span><span>&lt;-</span><span> </span><span>read.csv</span><span>(</span><span>&quot;creditcard.csv&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p>The dataset contains numerical input variables V1 to V28, which are the result of a PCA transformation of the original features (which could not be provided due to confidentiality issues).</p>
<p>The response variable <em>Class</em> tell us whether a transaction was fraudulent (value = 1) or not (value = 0).</p>
<div>
<pre><code><span>creditcard</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>Class</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;lightgrey&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_bw</span><span>()</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/machine_learning/2017/05/01/fraud_files/figure-markdown_github/unnamed-chunk-5-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>There are two additional features, <em>Time</em> (time in seconds between each transaction and the first transaction) and <em>Amount</em> (how much money was transferred in this transaction).</p>
<p>Because <em>Time</em> only tells us the order in which transactions have been done, it doesn’t actually tell us anything about the actual times (i.e. time of day) of the transaction. Therefore, I am normalizing them by day and bin them into four groups according to time of day.</p>
<div>
<pre><code><span>summary</span><span>(</span><span>creditcard</span><span>$</span><span>Time</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##       0   54202   84692   94814  139320  172792
</code></pre>
</div>
<div>
<pre><code><span># how many seconds are 24 hours
# 1 hr = 60 mins = 60 x 60 s = 3600 s
</span><span>3600</span><span> </span><span>*</span><span> </span><span>24</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 86400
</code></pre>
</div>
<div>
<pre><code><span># separate transactions by day
</span><span>creditcard</span><span>$</span><span>day</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>creditcard</span><span>$</span><span>Time</span><span> </span><span>></span><span> </span><span>3600</span><span> </span><span>*</span><span> </span><span>24</span><span>,</span><span> </span><span>&quot;day2&quot;</span><span>,</span><span> </span><span>&quot;day1&quot;</span><span>)</span><span>
</span><span># make transaction relative to day
</span><span>creditcard</span><span>$</span><span>Time_day</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>creditcard</span><span>$</span><span>day</span><span> </span><span>==</span><span> </span><span>&quot;day2&quot;</span><span>,</span><span> </span><span>creditcard</span><span>$</span><span>Time</span><span> </span><span>-</span><span> </span><span>86400</span><span>,</span><span> </span><span>creditcard</span><span>$</span><span>Time</span><span>)</span><span>
</span><span>summary</span><span>(</span><span>creditcard</span><span>[</span><span>creditcard</span><span>$</span><span>day</span><span> </span><span>==</span><span> </span><span>&quot;day1&quot;</span><span>,</span><span> </span><span>]</span><span>$</span><span>Time_day</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##       0   38432   54689   52948   70977   86400
</code></pre>
</div>
<div>
<pre><code><span>summary</span><span>(</span><span>creditcard</span><span>[</span><span>creditcard</span><span>$</span><span>day</span><span> </span><span>==</span><span> </span><span>&quot;day2&quot;</span><span>,</span><span> </span><span>]</span><span>$</span><span>Time_day</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##       1   37843   53425   51705   68182   86392
</code></pre>
</div>
<div>
<pre><code><span># bin transactions according to time of day
</span><span>creditcard</span><span>$</span><span>Time</span><span> </span><span>&lt;-</span><span> </span><span>as.factor</span><span>(</span><span>ifelse</span><span>(</span><span>creditcard</span><span>$</span><span>Time_day</span><span> </span><span>&lt;=</span><span> </span><span>38138</span><span>,</span><span> </span><span>&quot;gr1&quot;</span><span>,</span><span> </span><span># mean 1st Qu.
</span><span> </span><span>ifelse</span><span>(</span><span>creditcard</span><span>$</span><span>Time_day</span><span> </span><span>&lt;=</span><span> </span><span>52327</span><span>,</span><span> </span><span>&quot;gr2&quot;</span><span>,</span><span> </span><span># mean mean
</span><span> </span><span>ifelse</span><span>(</span><span>creditcard</span><span>$</span><span>Time_day</span><span> </span><span>&lt;=</span><span> </span><span>69580</span><span>,</span><span> </span><span>&quot;gr3&quot;</span><span>,</span><span> </span><span># mean 3rd Qu
</span><span> </span><span>&quot;gr4&quot;</span><span>))))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>creditcard</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>day</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;lightgrey&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_bw</span><span>()</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i2.wp.com/shiring.github.io/machine_learning/2017/05/01/fraud_files/figure-markdown_github/unnamed-chunk-7-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>We can see now that the transactions in this dataset have all been recorded on two consecutive days and there are roughly the same number of transactions on these two days.</p>
<p>Now, I remove the columns I used to create the <em>Time</em> bins.</p>
<div>
<pre><code><span>creditcard</span><span> </span><span>&lt;-</span><span> </span><span>select</span><span>(</span><span>creditcard</span><span>,</span><span> </span><span>-</span><span>Time_day</span><span>,</span><span> </span><span>-</span><span>day</span><span>)</span><span>
</span><span># convert class variable to factor
</span><span>creditcard</span><span>$</span><span>Class</span><span> </span><span>&lt;-</span><span> </span><span>factor</span><span>(</span><span>creditcard</span><span>$</span><span>Class</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>creditcard</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>Time</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;lightgrey&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_bw</span><span>()</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span> </span><span>~</span><span> </span><span>Class</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i1.wp.com/shiring.github.io/machine_learning/2017/05/01/fraud_files/figure-markdown_github/unnamed-chunk-9-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>The distribution of transactions over the four <em>Time</em> bins shows, that the majority of fraud cases have happened in group 1 (although, I can’t say when exactly because the original <em>Time</em> feature did not tell us when the first transaction occurred).</p>
<p>I also want to look at the distribution of the amounts of money that were transferred:</p>
<div>
<pre><code><span>summary</span><span>(</span><span>creditcard</span><span>[</span><span>creditcard</span><span>$</span><span>Class</span><span> </span><span>==</span><span> </span><span>&quot;0&quot;</span><span>,</span><span> </span><span>]</span><span>$</span><span>Amount</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
##     0.00     5.65    22.00    88.29    77.05 25691.16
</code></pre>
</div>
<div>
<pre><code><span>summary</span><span>(</span><span>creditcard</span><span>[</span><span>creditcard</span><span>$</span><span>Class</span><span> </span><span>==</span><span> </span><span>&quot;1&quot;</span><span>,</span><span> </span><span>]</span><span>$</span><span>Amount</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    0.00    1.00    9.25  122.21  105.89 2125.87
</code></pre>
</div>
<div>
<pre><code><span>creditcard</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>Amount</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_histogram</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;lightgrey&quot;</span><span>,</span><span> </span><span>bins</span><span> </span><span>=</span><span> </span><span>50</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_bw</span><span>()</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span> </span><span>~</span><span> </span><span>Class</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/machine_learning/2017/05/01/fraud_files/figure-markdown_github/unnamed-chunk-10-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Interestingly, fraudulent credit card transactions had a higher mean amount of money that was transferred, but the maximum amount was much lower compared to regular transactions.</p>
<p></p>
<h2>Modeling</h2>
<p>For modeling, I am using R’s H2O implementation with the <em>h2o</em> package. For more details and other examples, see my posts on <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2017/03/31/webinar_code">my machine learning webinar</a>, on <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2017/02/27/h2o">building neural nets with h2o</a> and on <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2017/03/07/grid_search">performing grid search for hyperparameter tuning</a>.</p>
<div>
<pre><code><span>library</span><span>(</span><span>h</span><span>2</span><span>o</span><span>)</span><span>
</span><span>h</span><span>2</span><span>o.init</span><span>(</span><span>nthreads</span><span> </span><span>=</span><span> </span><span>-1</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##  Connection successful!
## 
## R is connected to the H2O cluster: 
##     H2O cluster uptime:         7 minutes 6 seconds 
##     H2O cluster version:        3.10.4.4 
##     H2O cluster version age:    16 days  
##     H2O cluster name:           H2O_started_from_R_Shirin_nly512 
##     H2O cluster total nodes:    1 
##     H2O cluster total memory:   1.60 GB 
##     H2O cluster total cores:    2 
##     H2O cluster allowed cores:  2 
##     H2O cluster healthy:        TRUE 
##     H2O Connection ip:          localhost 
##     H2O Connection port:        54321 
##     H2O Connection proxy:       NA 
##     H2O Internal Security:      FALSE 
##     R Version:                  R version 3.4.0 (2017-04-21)
</code></pre>
</div>
<div>
<pre><code><span># convert data to H2OFrame
</span><span>creditcard_hf</span><span> </span><span>&lt;-</span><span> </span><span>as.h2o</span><span>(</span><span>creditcard</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<p>Then, I am splitting the dataset into training and test sets. Because I want to check how a pre-trained model with perform, I am splitting my data into two separate training sets and one independent test set for final model comparison.</p>
<div>
<pre><code><span>splits</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.splitFrame</span><span>(</span><span>creditcard_hf</span><span>,</span><span> </span><span>ratios</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>0.4</span><span>,</span><span> </span><span>0.4</span><span>),</span><span> </span><span>seed</span><span> </span><span>=</span><span> </span><span>42</span><span>)</span><span>
</span><span>train_unsupervised</span><span> </span><span>&lt;-</span><span> </span><span>splits</span><span>[[</span><span>1</span><span>]]</span><span>
</span><span>train_supervised</span><span> </span><span>&lt;-</span><span> </span><span>splits</span><span>[[</span><span>2</span><span>]]</span><span>
</span><span>test</span><span> </span><span>&lt;-</span><span> </span><span>splits</span><span>[[</span><span>3</span><span>]]</span><span>
</span><span>response</span><span> </span><span>&lt;-</span><span> </span><span>&quot;Class&quot;</span><span>
</span><span>features</span><span> </span><span>&lt;-</span><span> </span><span>setdiff</span><span>(</span><span>colnames</span><span>(</span><span>train_unsupervised</span><span>),</span><span> </span><span>response</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<h3>Autoencoders</h3>
<p>First, I am training the unsupervised neural network model using deep learning autoencoders. With <em>h2o</em>, we can simply set <code>autoencoder = TRUE</code>.</p>
<p>Here, I am applying a technique called “bottleneck” training, where the hidden layer in the middle is very small. This means that my model will have to reduce the dimensionality of the input data (in this case, down to 2 nodes/dimensions).</p>
<p>The autoencoder model will then learn the patterns of the input data irrespective of given class labels. Here, it will learn, which credit card transactions are similar and which transactions are outliers or anomalies. We need to keep in mind though, that autoencoder models will be sensitive to outliers in our data, which might throw off otherwise typical patterns.</p>
<div>
<pre><code><span>model_nn</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.deeplearning</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>features</span><span>,</span><span>
</span><span>training_frame</span><span> </span><span>=</span><span> </span><span>train_unsupervised</span><span>,</span><span>
</span><span>model_id</span><span> </span><span>=</span><span> </span><span>&quot;model_nn&quot;</span><span>,</span><span>
</span><span>autoencoder</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>reproducible</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>#slow - turn off for real problems
</span><span> </span><span>ignore_const_cols</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span>
</span><span>seed</span><span> </span><span>=</span><span> </span><span>42</span><span>,</span><span>
</span><span>hidden</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>10</span><span>,</span><span> </span><span>2</span><span>,</span><span> </span><span>10</span><span>),</span><span> </span><span>epochs</span><span> </span><span>=</span><span> </span><span>100</span><span>,</span><span>
</span><span>activation</span><span> </span><span>=</span><span> </span><span>&quot;Tanh&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p>Because training can take a while, I am saving the model:</p>
<div>
<pre><code><span>h</span><span>2</span><span>o.saveModel</span><span>(</span><span>model_nn</span><span>,</span><span> </span><span>path</span><span>=</span><span>&quot;model_nn&quot;</span><span>,</span><span> </span><span>force</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>model_nn</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.loadModel</span><span>(</span><span>&quot;model_nn&quot;</span><span>)</span><span>
</span><span>model_nn</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Model Details:
## ==============
## 
## H2OAutoEncoderModel: deeplearning
## Model ID:  model_nn 
## Status of Neuron Layers: auto-encoder, gaussian distribution, Quadratic loss, 776 weights/biases, 16.0 KB, 2,622,851 training samples, mini-batch size 1
##   layer units  type dropout       l1       l2 mean_rate rate_rms momentum
## 1     1    34 Input  0.00 %                                              
## 2     2    10  Tanh  0.00 % 0.000000 0.000000  0.709865 0.320108 0.000000
## 3     3     2  Tanh  0.00 % 0.000000 0.000000  0.048458 0.109033 0.000000
## 4     4    10  Tanh  0.00 % 0.000000 0.000000  0.164717 0.192053 0.000000
## 5     5    34  Tanh         0.000000 0.000000  0.369681 0.425672 0.000000
##   mean_weight weight_rms mean_bias bias_rms
## 1                                          
## 2   -0.039307   0.691302  0.008052 0.965178
## 3   -0.097383   0.314106  0.226376 0.067917
## 4    0.227664   1.089589  0.112032 0.672444
## 5    0.011072   0.605586  0.091124 0.722602
## 
## 
## H2OAutoEncoderMetrics: deeplearning
## ** Reported on training data. **
## 
## Training Set Metrics: 
## =====================
## 
## MSE: (Extract with `h2o.mse`) 0.001472071
## RMSE: (Extract with `h2o.rmse`) 0.03836757
</code></pre>
</div>
<div>
<pre><code><span>#Convert to autoencoded representation
</span><span>test_autoenc</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.predict</span><span>(</span><span>model_nn</span><span>,</span><span> </span><span>test</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<h3>Dimensionality reduction with hidden layers</h3>
<p>Because I had used a bottleneck model with two nodes in the hidden layer in the middle, we can use this dimensionality reduction to explore our feature space (similar to what to we could do with a principal component analysis). We can extract this hidden feature with the <code>h2o.deepfeatures()</code> function and plot it to show the reduced representation of the input data.</p>
<div>
<pre><code><span>train_features</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.deepfeatures</span><span>(</span><span>model_nn</span><span>,</span><span> </span><span>train_unsupervised</span><span>,</span><span> </span><span>layer</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>as.data.frame</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Class</span><span> </span><span>=</span><span> </span><span>as.vector</span><span>(</span><span>train_unsupervised</span><span>[,</span><span> </span><span>31</span><span>]))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>ggplot</span><span>(</span><span>train_features</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>DF.L2.C1</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>DF.L2.C2</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>Class</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.1</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i2.wp.com/shiring.github.io/machine_learning/2017/05/01/fraud_files/figure-markdown_github/unnamed-chunk-17-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Here, we do not see a cluster of fraudulent transactions that is distinct from non-fraud instances, so dimensionality reduction with our autoencoder model alone is not sufficient to identify fraud in this dataset.</p>
<p>But we could use the reduced dimensionality representation of one of the hidden layers as features for model training. An example would be to use the 10 features from the first or third hidden layer:</p>
<div>
<pre><code><span># let's take the third hidden layer
</span><span>train_features</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.deepfeatures</span><span>(</span><span>model_nn</span><span>,</span><span> </span><span>train_unsupervised</span><span>,</span><span> </span><span>layer</span><span> </span><span>=</span><span> </span><span>3</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>as.data.frame</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Class</span><span> </span><span>=</span><span> </span><span>as.factor</span><span>(</span><span>as.vector</span><span>(</span><span>train_unsupervised</span><span>[,</span><span> </span><span>31</span><span>])))</span><span> </span><span>%>%</span><span>
</span><span>as.h2o</span><span>()</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>features_dim</span><span> </span><span>&lt;-</span><span> </span><span>setdiff</span><span>(</span><span>colnames</span><span>(</span><span>train_features</span><span>),</span><span> </span><span>response</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>model_nn_dim</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.deeplearning</span><span>(</span><span>y</span><span> </span><span>=</span><span> </span><span>response</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>features_dim</span><span>,</span><span>
</span><span>training_frame</span><span> </span><span>=</span><span> </span><span>train_features</span><span>,</span><span>
</span><span>reproducible</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>#slow - turn off for real problems
</span><span> </span><span>balance_classes</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>ignore_const_cols</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span>
</span><span>seed</span><span> </span><span>=</span><span> </span><span>42</span><span>,</span><span>
</span><span>hidden</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>10</span><span>,</span><span> </span><span>2</span><span>,</span><span> </span><span>10</span><span>),</span><span> </span><span>epochs</span><span> </span><span>=</span><span> </span><span>100</span><span>,</span><span>
</span><span>activation</span><span> </span><span>=</span><span> </span><span>&quot;Tanh&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>h</span><span>2</span><span>o.saveModel</span><span>(</span><span>model_nn_dim</span><span>,</span><span> </span><span>path</span><span>=</span><span>&quot;model_nn_dim&quot;</span><span>,</span><span> </span><span>force</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>model_nn_dim</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.loadModel</span><span>(</span><span>&quot;model_nn_dim/DeepLearning_model_R_1493574057843_49&quot;</span><span>)</span><span>
</span><span>model_nn_dim</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Model Details:
## ==============
## 
## H2OBinomialModel: deeplearning
## Model ID:  DeepLearning_model_R_1493574057843_49 
## Status of Neuron Layers: predicting Class, 2-class classification, bernoulli distribution, CrossEntropy loss, 184 weights/biases, 7.1 KB, 4,098,024 training samples, mini-batch size 1
##   layer units    type dropout       l1       l2 mean_rate rate_rms
## 1     1    10   Input  0.00 %                                     
## 2     2    10    Tanh  0.00 % 0.000000 0.000000  0.299549 0.363303
## 3     3     2    Tanh  0.00 % 0.000000 0.000000  0.009907 0.012193
## 4     4    10    Tanh  0.00 % 0.000000 0.000000  0.248854 0.126722
## 5     5     2 Softmax         0.000000 0.000000  0.160990 0.078139
##   momentum mean_weight weight_rms mean_bias bias_rms
## 1                                                   
## 2 0.000000   -0.539949   4.601841 -0.060170 2.765655
## 3 0.000000   -0.384247   1.763649 -1.429832 0.570700
## 4 0.000000   -0.004865   0.774441 -0.320202 0.668747
## 5 0.000000    0.563951   1.365125  0.000000 0.397460
## 
## 
## H2OBinomialMetrics: deeplearning
## ** Reported on training data. **
## ** Metrics reported on temporary training frame with 9974 samples **
## 
## MSE:  0.4905394
## RMSE:  0.7003852
## LogLoss:  2.763673
## Mean Per-Class Error:  0.3856454
## AUC:  0.6656933
## Gini:  0.3313867
## 
## Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
##           0    1    Error        Rate
## 0      1438 3594 0.714229  =3594/5032
## 1       282 4660 0.057062   =282/4942
## Totals 1720 8254 0.388610  =3876/9974
## 
## Maximum Metrics: Maximum metrics at their respective thresholds
##                         metric threshold    value idx
## 1                       max f1  0.002173 0.706275 300
## 2                       max f2  0.001540 0.842820 335
## 3                 max f0point5  0.002531 0.619024 280
## 4                 max accuracy  0.002531 0.619711 280
## 5                max precision  0.030005 1.000000   0
## 6                   max recall  0.000496 1.000000 388
## 7              max specificity  0.030005 1.000000   0
## 8             max absolute_mcc  0.002173 0.302697 300
## 9   max min_per_class_accuracy  0.002590 0.554634 277
## 10 max mean_per_class_accuracy  0.002531 0.622276 280
## 
## Gains/Lift Table: Extract with `h2o.gainsLift(&lt;model>, &lt;data>)` or `h2o.gainsLift(&lt;model>, valid=&lt;T/F>, xval=&lt;T/F>)`
</code></pre>
</div>
<p></p>
<p>For measuring model performance on test data, we need to convert the test data to the same reduced dimensions as the trainings data:</p>
<div>
<pre><code><span>test_dim</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.deepfeatures</span><span>(</span><span>model_nn</span><span>,</span><span> </span><span>test</span><span>,</span><span> </span><span>layer</span><span> </span><span>=</span><span> </span><span>3</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>h</span><span>2</span><span>o.predict</span><span>(</span><span>model_nn_dim</span><span>,</span><span> </span><span>test_dim</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>as.data.frame</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>actual</span><span> </span><span>=</span><span> </span><span>as.vector</span><span>(</span><span>test</span><span>[,</span><span> </span><span>31</span><span>]))</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>actual</span><span>,</span><span> </span><span>predict</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarise</span><span>(</span><span>n</span><span> </span><span>=</span><span> </span><span>n</span><span>())</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>freq</span><span> </span><span>=</span><span> </span><span>n</span><span> </span><span>/</span><span> </span><span>sum</span><span>(</span><span>n</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Source: local data frame [4 x 4]
## Groups: actual [2]
## 
##   actual predict     n       freq
##    &lt;chr>  &lt;fctr> &lt;int>      &lt;dbl>
## 1      0       0 16710 0.29506286
## 2      0       1 39922 0.70493714
## 3      1       0     7 0.07608696
## 4      1       1    85 0.92391304
</code></pre>
</div>
<p>Now, this actually looks quite good in terms of identifying fraud cases: 92% of fraud cases were identified! However, many non-fraud cases were also classified as fraud. For real-life application, this wouldn’t be a good model. Let’s try some other techniques…</p>
<p></p>
<h3>Anomaly detection</h3>
<p>We can also ask which instances were considered outliers or anomalies within our test data, using the <code>h2o.anomaly()</code> function. Based on the autoencoder model that was trained before, the input data will be reconstructed and for each instance, the mean squared error (MSE) between actual value and reconstruction is calculated.</p>
<p>I am also calculating the mean MSE for both class labels.</p>
<div>
<pre><code><span>anomaly</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.anomaly</span><span>(</span><span>model_nn</span><span>,</span><span> </span><span>test</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>as.data.frame</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>tibble</span><span>::</span><span>rownames_to_column</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Class</span><span> </span><span>=</span><span> </span><span>as.vector</span><span>(</span><span>test</span><span>[,</span><span> </span><span>31</span><span>]))</span><span>
</span><span>mean_mse</span><span> </span><span>&lt;-</span><span> </span><span>anomaly</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>Class</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarise</span><span>(</span><span>mean</span><span> </span><span>=</span><span> </span><span>mean</span><span>(</span><span>Reconstruction.MSE</span><span>))</span><span>
</span></code></pre>
</div>
<p>This, we can now plot:</p>
<div>
<pre><code><span>ggplot</span><span>(</span><span>anomaly</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>rowname</span><span>),</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>Reconstruction.MSE</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>as.factor</span><span>(</span><span>Class</span><span>)))</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.3</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_hline</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>mean_mse</span><span>,</span><span> </span><span>aes</span><span>(</span><span>yintercept</span><span> </span><span>=</span><span> </span><span>mean</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>Class</span><span>))</span><span> </span><span>+</span><span>
</span><span>scale_color_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;instance number&quot;</span><span>,</span><span>
</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;Class&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i2.wp.com/shiring.github.io/machine_learning/2017/05/01/fraud_files/figure-markdown_github/unnamed-chunk-24-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>As we can see in the plot, there is no perfect classification into fraud and non-fraud cases but the mean MSE is definitely higher for fraudulent transactions than for regular ones.</p>
<p>We can now identify outlier instances by applying an MSE threshold for what we consider outliers. We could e.g. say that we consider every instance with an MSE > 0.02 (chosen according to the plot above) to be an anomaly/outlier.</p>
<div>
<pre><code><span>anomaly</span><span> </span><span>&lt;-</span><span> </span><span>anomaly</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>outlier</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>Reconstruction.MSE</span><span> </span><span>></span><span> </span><span>0.02</span><span>,</span><span> </span><span>&quot;outlier&quot;</span><span>,</span><span> </span><span>&quot;no_outlier&quot;</span><span>))</span><span>
</span><span>anomaly</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>Class</span><span>,</span><span> </span><span>outlier</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarise</span><span>(</span><span>n</span><span> </span><span>=</span><span> </span><span>n</span><span>())</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>freq</span><span> </span><span>=</span><span> </span><span>n</span><span> </span><span>/</span><span> </span><span>sum</span><span>(</span><span>n</span><span>))</span><span> </span></code></pre>
</div>
<div>
<pre><code>## Source: local data frame [4 x 4]
## Groups: Class [2]
## 
##   Class    outlier     n         freq
##   &lt;chr>      &lt;chr> &lt;int>        &lt;dbl>
## 1     0 no_outlier 56602 0.9994702642
## 2     0    outlier    30 0.0005297358
## 3     1 no_outlier    60 0.6521739130
## 4     1    outlier    32 0.3478260870
</code></pre>
</div>
<p>As we can see, outlier detection is not sufficient to correctly classify fraudulent credit card transactions either (at least not with this dataset).</p>
<p></p>
<h3>Pre-trained supervised model</h3>
<p>We can now try using the autoencoder model as a pre-training input for a supervised model. Here, I am again using a neural network. This model will now use the weights from the autoencoder for model fitting.</p>
<div>
<pre><code><span>model_nn_2</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.deeplearning</span><span>(</span><span>y</span><span> </span><span>=</span><span> </span><span>response</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>features</span><span>,</span><span>
</span><span>training_frame</span><span> </span><span>=</span><span> </span><span>train_supervised</span><span>,</span><span>
</span><span>pretrained_autoencoder</span><span> </span><span>=</span><span> </span><span>&quot;model_nn&quot;</span><span>,</span><span>
</span><span>reproducible</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>#slow - turn off for real problems
</span><span> </span><span>balance_classes</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>ignore_const_cols</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span>
</span><span>seed</span><span> </span><span>=</span><span> </span><span>42</span><span>,</span><span>
</span><span>hidden</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>10</span><span>,</span><span> </span><span>2</span><span>,</span><span> </span><span>10</span><span>),</span><span> </span><span>epochs</span><span> </span><span>=</span><span> </span><span>100</span><span>,</span><span>
</span><span>activation</span><span> </span><span>=</span><span> </span><span>&quot;Tanh&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>h</span><span>2</span><span>o.saveModel</span><span>(</span><span>model_nn_2</span><span>,</span><span> </span><span>path</span><span>=</span><span>&quot;model_nn_2&quot;</span><span>,</span><span> </span><span>force</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>model_nn_2</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.loadModel</span><span>(</span><span>&quot;model_nn_2/DeepLearning_model_R_1493574057843_9&quot;</span><span>)</span><span>
</span><span>model_nn_2</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Model Details:
## ==============
## 
## H2OBinomialModel: deeplearning
## Model ID:  DeepLearning_model_R_1493574057843_9 
## Status of Neuron Layers: predicting Class, 2-class classification, bernoulli distribution, CrossEntropy loss, 424 weights/biases, 11.7 KB, 3,643,248 training samples, mini-batch size 1
##   layer units    type dropout       l1       l2 mean_rate rate_rms
## 1     1    34   Input  0.00 %                                     
## 2     2    10    Tanh  0.00 % 0.000000 0.000000  0.274110 0.291925
## 3     3     2    Tanh  0.00 % 0.000000 0.000000  0.004480 0.002651
## 4     4    10    Tanh  0.00 % 0.000000 0.000000  0.210337 0.326239
## 5     5     2 Softmax         0.000000 0.000000  0.137694 0.123830
##   momentum mean_weight weight_rms mean_bias bias_rms
## 1                                                   
## 2 0.000000   -0.006945   1.058583 -0.501068 1.576798
## 3 0.000000    0.021012   0.309286  0.360286 0.105522
## 4 0.000000    0.934361   1.759343  0.304729 0.532527
## 5 0.000000   -0.133149   2.641065  2.392999 3.137845
## 
## 
## H2OBinomialMetrics: deeplearning
## ** Reported on training data. **
## ** Metrics reported on temporary training frame with 9973 samples **
## 
## MSE:  0.01500211
## RMSE:  0.1224831
## LogLoss:  0.04408663
## Mean Per-Class Error:  0.00169424
## AUC:  0.9995075
## Gini:  0.9990149
## 
## Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
##           0    1    Error      Rate
## 0      5000   17 0.003388  =17/5017
## 1         0 4956 0.000000   =0/4956
## Totals 5000 4973 0.001705  =17/9973
## 
## Maximum Metrics: Maximum metrics at their respective thresholds
##                         metric threshold    value idx
## 1                       max f1  0.134320 0.998288  96
## 2                       max f2  0.134320 0.999314  96
## 3                 max f0point5  0.134320 0.997263  96
## 4                 max accuracy  0.134320 0.998295  96
## 5                max precision  1.000000 1.000000   0
## 6                   max recall  0.134320 1.000000  96
## 7              max specificity  1.000000 1.000000   0
## 8             max absolute_mcc  0.134320 0.996597  96
## 9   max min_per_class_accuracy  0.134320 0.996612  96
## 10 max mean_per_class_accuracy  0.134320 0.998306  96
## 
## Gains/Lift Table: Extract with `h2o.gainsLift(&lt;model>, &lt;data>)` or `h2o.gainsLift(&lt;model>, valid=&lt;T/F>, xval=&lt;T/F>)`
</code></pre>
</div>
<div>
<pre><code><span>pred</span><span> </span><span>&lt;-</span><span> </span><span>as.data.frame</span><span>(</span><span>h</span><span>2</span><span>o.predict</span><span>(</span><span>object</span><span> </span><span>=</span><span> </span><span>model_nn_2</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>actual</span><span> </span><span>=</span><span> </span><span>as.vector</span><span>(</span><span>test</span><span>[,</span><span> </span><span>31</span><span>]))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>pred</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>actual</span><span>,</span><span> </span><span>predict</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarise</span><span>(</span><span>n</span><span> </span><span>=</span><span> </span><span>n</span><span>())</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>freq</span><span> </span><span>=</span><span> </span><span>n</span><span> </span><span>/</span><span> </span><span>sum</span><span>(</span><span>n</span><span>))</span><span> </span></code></pre>
</div>
<div>
<pre><code>## Source: local data frame [4 x 4]
## Groups: actual [2]
## 
##   actual predict     n        freq
##    &lt;chr>  &lt;fctr> &lt;int>       &lt;dbl>
## 1      0       0 56512 0.997881057
## 2      0       1   120 0.002118943
## 3      1       0    16 0.173913043
## 4      1       1    76 0.826086957
</code></pre>
</div>
<div>
<pre><code><span>pred</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>actual</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>predict</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>()</span><span> </span><span>+</span><span>
</span><span>theme_bw</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span> </span><span>~</span><span> </span><span>actual</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i1.wp.com/shiring.github.io/machine_learning/2017/05/01/fraud_files/figure-markdown_github/unnamed-chunk-31-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Now, this looks much better! We did miss 17% of the fraud cases but we also did not mis-classify too many of the non-fraud cases.</p>
<p>In real-life, we would now spend some more time trying to improve the model by e.g. <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2017/03/07/grid_search">performing grid search for hyperparameter tuning</a>, going back to the original features (which we did not have here) and trying different engineered features and/or trying different algorithms. But here, I will leave it at that.</p>
<p></p>
<h3>Measuring model performance on highly unbalanced data</h3>
<p>Because of the severe bias towards non-fraud cases, we can not use performance measures like accuracy or area under the curve (AUC), as they would give overly optimistic results based on the high percentage of correct classifications of the majority class.</p>
<p>An alternative to AUC is to use the precision-recall curve or the sensitivity (recall)-specificity curve. To calculate and plot these metrics, we can use the <strong>ROCR</strong> package. There are different ways to calculate the area under a curve (see the <strong>PRROC</strong> package for details) but I am going to use a simple function that calculates the area between every consecutive points-pair of x (i.e. x1 – x0, x2 – x1, etc.) under the corresponding values of y.</p>
<div>
<pre><code><span>library</span><span>(</span><span>ROCR</span><span>)</span><span>
</span><span># http://stackoverflow.com/questions/24563061/computing-integral-of-a-line-plot-in-r
</span><span>line_integral</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>x</span><span>,</span><span> </span><span>y</span><span>)</span><span> </span><span>{</span><span>
</span><span>dx</span><span> </span><span>&lt;-</span><span> </span><span>diff</span><span>(</span><span>x</span><span>)</span><span>
</span><span>end</span><span> </span><span>&lt;-</span><span> </span><span>length</span><span>(</span><span>y</span><span>)</span><span>
</span><span>my</span><span> </span><span>&lt;-</span><span> </span><span>(</span><span>y</span><span>[</span><span>1</span><span>:</span><span>(</span><span>end</span><span> </span><span>-</span><span> </span><span>1</span><span>)]</span><span> </span><span>+</span><span> </span><span>y</span><span>[</span><span>2</span><span>:</span><span>end</span><span>])</span><span> </span><span>/</span><span> </span><span>2</span><span>
</span><span>sum</span><span>(</span><span>dx</span><span> </span><span>*</span><span> </span><span>my</span><span>)</span><span>
</span><span>}</span><span> </span><span>prediction_obj</span><span> </span><span>&lt;-</span><span> </span><span>prediction</span><span>(</span><span>pred</span><span>$</span><span>p</span><span>1</span><span>,</span><span> </span><span>pred</span><span>$</span><span>actual</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>par</span><span>(</span><span>mfrow</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>2</span><span>))</span><span>
</span><span>par</span><span>(</span><span>mar</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>5.1</span><span>,</span><span>4.1</span><span>,</span><span>4.1</span><span>,</span><span>2.1</span><span>))</span><span>
</span><span># precision-recall curve
</span><span>perf1</span><span> </span><span>&lt;-</span><span> </span><span>performance</span><span>(</span><span>prediction_obj</span><span>,</span><span> </span><span>measure</span><span> </span><span>=</span><span> </span><span>&quot;prec&quot;</span><span>,</span><span> </span><span>x.measure</span><span> </span><span>=</span><span> </span><span>&quot;rec&quot;</span><span>)</span><span> </span><span>x</span><span> </span><span>&lt;-</span><span> </span><span>perf1</span><span>@</span><span>x.values</span><span>[[</span><span>1</span><span>]]</span><span>
</span><span>y</span><span> </span><span>&lt;-</span><span> </span><span>perf1</span><span>@</span><span>y.values</span><span>[[</span><span>1</span><span>]]</span><span>
</span><span>y</span><span>[</span><span>1</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>0</span><span>
</span><span>plot</span><span>(</span><span>perf1</span><span>,</span><span> </span><span>main</span><span> </span><span>=</span><span> </span><span>paste</span><span>(</span><span>&quot;Area Under the\nPrecision-Recall Curve:\n&quot;</span><span>,</span><span> </span><span>round</span><span>(</span><span>abs</span><span>(</span><span>line_integral</span><span>(</span><span>x</span><span>,</span><span>y</span><span>)),</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>3</span><span>)))</span><span>
</span><span># sensitivity-specificity curve
</span><span>perf2</span><span> </span><span>&lt;-</span><span> </span><span>performance</span><span>(</span><span>prediction_obj</span><span>,</span><span> </span><span>measure</span><span> </span><span>=</span><span> </span><span>&quot;sens&quot;</span><span>,</span><span> </span><span>x.measure</span><span> </span><span>=</span><span> </span><span>&quot;spec&quot;</span><span>)</span><span> </span><span>x</span><span> </span><span>&lt;-</span><span> </span><span>perf2</span><span>@</span><span>x.values</span><span>[[</span><span>1</span><span>]]</span><span>
</span><span>y</span><span> </span><span>&lt;-</span><span> </span><span>perf2</span><span>@</span><span>y.values</span><span>[[</span><span>1</span><span>]]</span><span>
</span><span>y</span><span>[</span><span>1</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>0</span><span>
</span><span>plot</span><span>(</span><span>perf2</span><span>,</span><span> </span><span>main</span><span> </span><span>=</span><span> </span><span>paste</span><span>(</span><span>&quot;Area Under the\nSensitivity-Specificity Curve:\n&quot;</span><span>,</span><span> </span><span>round</span><span>(</span><span>abs</span><span>(</span><span>line_integral</span><span>(</span><span>x</span><span>,</span><span>y</span><span>)),</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>3</span><span>)))</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i2.wp.com/shiring.github.io/machine_learning/2017/05/01/fraud_files/figure-markdown_github/unnamed-chunk-34-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Precision is the proportion of test cases predicted to be fraud that were indeed fraudulent (i.e. the true positive predictions), while recall or sensitivity is the proportion of fraud cases that were identified as fraud. And specificity is the proportion of non-fraud cases that are identified as non-fraud.</p>
<p>The precision-recall curve tells us the relationship between correct fraud predictions and the proportion of fraud cases that were detected (e.g. if all or most fraud cases were identified, we also have many non-fraud cases predicted as fraud and vice versa). The sensitivity-specificity curve thus tell us the relationship between correctly identified classes of both labels (e.g. if we have 100% correctly classified fraud cases, we will have no correctly classified non-fraud cases and vice versa).</p>
<p>We can also look at this a little bit differently, by manually going through different prediction thresholds and calculating how many cases were correctly classified in the two classes:</p>
<div>
<pre><code><span>thresholds</span><span> </span><span>&lt;-</span><span> </span><span>seq</span><span>(</span><span>from</span><span> </span><span>=</span><span> </span><span>0</span><span>,</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>0.1</span><span>)</span><span>
</span><span>pred_thresholds</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>actual</span><span> </span><span>=</span><span> </span><span>pred</span><span>$</span><span>actual</span><span>)</span><span>
</span><span>for</span><span> </span><span>(</span><span>threshold</span><span> </span><span>in</span><span> </span><span>thresholds</span><span>)</span><span> </span><span>{</span><span>
</span><span>prediction</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>pred</span><span>$</span><span>p</span><span>1</span><span> </span><span>></span><span> </span><span>threshold</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>0</span><span>)</span><span>
</span><span>prediction_true</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>pred_thresholds</span><span>$</span><span>actual</span><span> </span><span>==</span><span> </span><span>prediction</span><span>,</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>pred_thresholds</span><span> </span><span>&lt;-</span><span> </span><span>cbind</span><span>(</span><span>pred_thresholds</span><span>,</span><span> </span><span>prediction_true</span><span>)</span><span>
</span><span>}</span><span>
</span><span>colnames</span><span>(</span><span>pred_thresholds</span><span>)[</span><span>-1</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>thresholds</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>pred_thresholds</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>x</span><span>,</span><span> </span><span>y</span><span>,</span><span> </span><span>2</span><span>:</span><span>ncol</span><span>(</span><span>pred_thresholds</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>actual</span><span>,</span><span> </span><span>x</span><span>,</span><span> </span><span>y</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarise</span><span>(</span><span>n</span><span> </span><span>=</span><span> </span><span>n</span><span>())</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>x</span><span>),</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>n</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>actual</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_vline</span><span>(</span><span>xintercept</span><span> </span><span>=</span><span> </span><span>0.6</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.5</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>()</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.5</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_bw</span><span>()</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>actual</span><span> </span><span>~</span><span> </span><span>y</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;prediction threshold&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;number of instances&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/machine_learning/2017/05/01/fraud_files/figure-markdown_github/unnamed-chunk-37-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>This plot tells us that we can increase the number of correctly classified non-fraud cases without loosing correctly classified fraud cases when we increase the prediction threshold from the default 0.5 to 0.6:</p>
<div>
<pre><code><span>pred</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>predict</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>pred</span><span>$</span><span>p</span><span>1</span><span> </span><span>></span><span> </span><span>0.6</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>0</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>actual</span><span>,</span><span> </span><span>predict</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarise</span><span>(</span><span>n</span><span> </span><span>=</span><span> </span><span>n</span><span>())</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>freq</span><span> </span><span>=</span><span> </span><span>n</span><span> </span><span>/</span><span> </span><span>sum</span><span>(</span><span>n</span><span>))</span><span> </span></code></pre>
</div>
<div>
<pre><code>## Source: local data frame [4 x 4]
## Groups: actual [2]
## 
##   actual predict     n        freq
##    &lt;chr>   &lt;dbl> &lt;int>       &lt;dbl>
## 1      0       0 56558 0.998693318
## 2      0       1    74 0.001306682
## 3      1       0    16 0.173913043
## 4      1       1    76 0.826086957
</code></pre>
</div>
<p>Our final model now correctly identified 83% of fraud cases and almost 100% of non-fraud cases.</p>
<hr>
<p>If you are interested in more machine learning posts, check out <a target="_blank" rel="nofollow" href="https://shiring.github.io/categories.html#machine_learning-ref">the category listing for <strong>machine_learning</strong> on my blog</a>.</p>
<hr>
<div>
<pre><code><span>sessionInfo</span><span>()</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## R version 3.4.0 (2017-04-21)
## Platform: x86_64-apple-darwin15.6.0 (64-bit)
## Running under: macOS Sierra 10.12.3
## 
## Matrix products: default
## BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] ROCR_1.0-7      gplots_3.0.1    h2o_3.10.4.4    dplyr_0.5.0    
##  [5] purrr_0.2.2     readr_1.1.0     tidyr_0.6.1     tibble_1.3.0   
##  [9] ggplot2_2.2.1   tidyverse_1.1.1
## 
## loaded via a namespace (and not attached):
##  [1] gtools_3.5.0       reshape2_1.4.2     haven_1.0.0       
##  [4] lattice_0.20-35    colorspace_1.3-2   htmltools_0.3.6   
##  [7] yaml_2.1.14        foreign_0.8-68     DBI_0.6-1         
## [10] RColorBrewer_1.1-2 modelr_0.1.0       readxl_1.0.0      
## [13] plyr_1.8.4         stringr_1.2.0      munsell_0.4.3     
## [16] gtable_0.2.0       cellranger_1.1.0   rvest_0.3.2       
## [19] caTools_1.17.1     psych_1.7.3.21     evaluate_0.10     
## [22] labeling_0.3       knitr_1.15.1       forcats_0.2.0     
## [25] parallel_3.4.0     broom_0.4.2        Rcpp_0.12.10      
## [28] KernSmooth_2.23-15 scales_0.4.1       backports_1.0.5   
## [31] gdata_2.17.0       jsonlite_1.4       mnormt_1.5-5      
## [34] hms_0.3            digest_0.6.12      stringi_1.1.5     
## [37] grid_3.4.0         rprojroot_1.2      tools_3.4.0       
## [40] bitops_1.0-6       magrittr_1.5       lazyeval_0.2.0    
## [43] RCurl_1.95-4.8     xml2_1.1.1         lubridate_1.6.0   
## [46] assertthat_0.2.0   rmarkdown_1.5      httr_1.2.1        
## [49] R6_2.2.0           nlme_3.1-131       compiler_3.4.0
</code></pre>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://shiring.github.io/machine_learning/2017/05/01/fraud"> Shirin's playgRound</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/zEj7IUz2GJo"></a></div>