<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/detecting-outliers-and-fraud-with-r-and-sql-server-on-my-bank-account-data-part-1/"></div></div><div><a data-url="https://www.r-bloggers.com/detecting-outliers-and-fraud-with-r-and-sql-server-on-my-bank-account-data-part-1/" data-counturl="https://www.r-bloggers.com/detecting-outliers-and-fraud-with-r-and-sql-server-on-my-bank-account-data-part-1/" data-count="horizontal" data-text="Detecting outliers and fraud with R and SQL Server on my bank account data – Part 1" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://tomaztsql.wordpress.com/2016/10/31/detecting-outliers-and-fraud-with-r-and-sql-server-on-my-bank-account-data-part-1/"> R – TomazTsql</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Detecting outliers and fraudulent behaviour (transactions, purchases, events, actions, triggers, etc.) takes a large amount of experiences and statistical/mathetmatical background.</p>
<p>One of the <a target="_blank" rel="nofollow" href="https://gallery.cortanaintelligence.com/Tutorial/Online-Fraud-Detection-Template-with-SQL-Server-R-Services-1">samples</a> Microsoft provided with release of new SQL Server 2016 was using simple logic of <a target="_blank" rel="nofollow" href="http://mathworld.wolfram.com/BenfordsLaw.html">Benford’s law</a>. This law works great with naturally occurring numbers and can be applied across any kind of problem. By naturally occurring, it is meant a number that is not generated generically such as a page number in a book, incremented number in your SQL Table, sequence number of any kind, but numbers that are occurring irrespective from each other, in nature (length or width of trees, mountains, rivers), length of the roads in the cities, addresses in your home town, city/country populations, etc. The law calculates the log distribution of numbers from 1 to 9 and stipulates that number one will occur 30% of times, number two will occur 17% of time, number three will occur 12% of the time and so on. Randomly generated numbers will most certainly generate distribution for each number from 1 to 9 with probability of 1/9. It might also not work with restrictions; for example height expressed in inches will surely not produce Benford function. My height is 188 which is 74 inches or 6ft2. All three numbers will not generate correct distribution, even though height is natural phenomena.</p>
<p>So Probability of number starting with number <strong>n</strong> equals to <strong>log(n+1) – log(n)</strong> with base 10. Keeping in mind this formula. So what is probability that a number starts with 29, is log(30) – log(29) = 1.4771 – 1.4623 = 0.0148. But keep in mind that this law applies only to numbers where number 1 appears approx. 30% of the time.</p>
<p>So just a quick example to support this theory, we will take</p>
<pre>benfordL &lt;- function(d){return(log(1+1/(1:9))/log(10))}
#distribution of numbers according to formula
benfordL(1:9)
#plot with numbers
plot(benfordL(1:9))</pre>
<p>Scatterplot with  log distribution of numbers from 1 to 9:</p>
<p><img sizes="(max-width: 545px) 100vw, 545px" alt="2016-10-31-09_29_58-plot-zoom" class="wp-image-2980" src="https://i0.wp.com/tomaztsql.files.wordpress.com/2016/10/2016-10-31-09_29_58-plot-zoom1.png?resize=456%2C312&ssl=1" data-recalc-dims="1"></p>
<p>Code for Benford’s Law is available at <a target="_blank" rel="nofollow" href="https://rosettacode.org/wiki/Benford%27s_law">RosettaCode</a> for your favorite programming language.</p>
<p>Now I want to check this rule agains the data in WideWorldImporters. I will use the following query to test Benford’s Law:</p>
<pre>SELECT TransactionAmount
FROM [WideWorldImporters].[Sales].[CustomerTransactions]
WHERE
    TransactionAmount > 0</pre>
<p>For this manner I will execute R Script:</p>
<pre>
DECLARE @RScript NVARCHAR(MAX)
SET @RScript = N'
            WWWTrans &lt;- InputDataSet
            get_first_num &lt;- function(number){return(as.numeric(substr(number,1,1)))}
            pbenford &lt;- function(d){return(log10(1+(1/d)))}
            lead_dig &lt;- mapply(get_first_num,WWWTrans$TransactionAmount)
            obs_freq_WWWTrans &lt;- table(lead_dig)/1000
            OutputDataSet &lt;- data.frame(obs_freq_WWWTrans)'

DECLARE @SQLScript NVARCHAR(MAX)
SET @SQLScript = N'
        SELECT 
            TransactionAmount
        FROM [WideWorldImporters].[Sales].[CustomerTransactions]
        WHERE 
            TransactionAmount > 0'

 EXECUTE sp_execute_external_script
          @language = N'R'
        , @script = @RScript
        , @input_data_1 = @SQLScript
        
WITH result SETS ((
                     lead_dig INT
                    ,Freq DECIMAL(10,2)
));</pre>
<p> </p>
<p>By comparing distribution of general Benford’s Law and TransactionAmount there are some discrepancies but in general they can follow same log distribution.</p>
<p><img sizes="(max-width: 542px) 100vw, 542px" alt="2016-10-31-11_06_22-plot-zoom" class="wp-image-3000" src="https://i0.wp.com/tomaztsql.files.wordpress.com/2016/10/2016-10-31-11_06_22-plot-zoom.png?resize=456%2C310&ssl=1" data-recalc-dims="1"></p>
<p>In Part 2 I will cover outlier detection with GLM and Random forest using my bank account dataset.</p>
<p>For Benford’s Law there are couple of R packages available <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/benford.analysis/benford.analysis.pdf">Benford.Analysis</a> and <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/BenfordTests/BenfordTests.pdf">BenfordTests</a>.</p>
<p> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/tomaztsql.wordpress.com/2945/"><img alt="" src="http://feeds.wordpress.com/1.0/comments/tomaztsql.wordpress.com/2945/" border="0"></a> <img alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" border="0" data-recalc-dims="1"></p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/detecting-outliers-and-fraud-with-r-and-sql-server-on-my-bank-account-data-part-1/"></div></div><div><a data-url="https://www.r-bloggers.com/detecting-outliers-and-fraud-with-r-and-sql-server-on-my-bank-account-data-part-1/" data-counturl="https://www.r-bloggers.com/detecting-outliers-and-fraud-with-r-and-sql-server-on-my-bank-account-data-part-1/" data-count="vertical" data-text="Detecting outliers and fraud with R and SQL Server on my bank account data – Part 1" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://tomaztsql.wordpress.com/2016/10/31/detecting-outliers-and-fraud-with-r-and-sql-server-on-my-bank-account-data-part-1/"> R – TomazTsql</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/HathrOxCE5k"></a></div>