<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/information-security-anomaly-detection-and-threat-hunting-with-anomalize/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.business-science.io/business/2018/06/10/infosec-anomalize-threat-hunting.html"> business-science.io - Articles</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a rel="nofollow" href="https://en.wikipedia.org/wiki/Information_security" target="_blank">Information Security (InfoSec)</a> is critical to a business. For those new to <strong>InfoSec, it is the state of being protected against the unauthorized use of information, especially electronic data. A single malicious threat can cause massive damage to a firm, large or small.</strong> It’s this reason when I (Matt Dancho) saw <a rel="nofollow" href="https://twitter.com/holisticinfosec" target="_blank">Russ McRee’s</a> article, <a rel="nofollow" href="https://holisticinfosec.blogspot.com/2018/06/toolsmith-133-anomaly-detection-threat.html" target="_blank">“Anomaly Detection &amp; Threat Hunting with Anomalize”</a>, that I asked him to repost on the <a rel="nofollow" href="http://www.business-science.io/blog/index.html" target="_blank">Business Science blog</a>. In his article, Russ speaks to use of our new R package, <code>anomalize</code>, as a way to detect threats (aka <strong>“threat hunting”</strong>). Russ is Group Program Manager of the <a rel="nofollow" href="https://danielmiessler.com/study/red-blue-purple-teams/" target="_blank">Blue Team</a> (the internal security team that defends against real attackers) for Microsoft’s Windows and Devices Group (WDG), now part of the Cloud and AI (C+AI) organization. He writes <a rel="nofollow" href="https://holisticinfosec.blogspot.com/" target="_blank">toolsmith</a>, a monthly column for information security practitioners, and has written for other publications including Information Security, (IN)SECURE, SysAdmin, and Linux Magazine. <strong>The data Russ routinely deals with is massive in scale</strong>: He processes <a rel="nofollow" href="https://en.wikipedia.org/wiki/Telemetry" target="_blank">security event telemetry</a> of all types (operating systems, network, applications, service layer) for all of Windows, Xbox, the Universal Store (transactions/purchases), and a few others. Billions of events in short order.</p>
<p><span data-sumome-listbuilder-embed-id="6cf8523a01e2faac60392073d460d72402c5971ce4821a8a8e81b28cde43f056"></span></p>
<h2>Learning Trajectory</h2>
<p>This is a great article from <strong>master in information security, Russ McCree</strong>. You’ll learn how Russ is using our new package for time series anomaly detection, <code>anomalize</code>, within his Blue Team (internal thread-defending team) work at Microsoft. He provides real-world examples of “threat hunting”, or the act of identifying malicious attacks on servers and how <code>anomalize</code> can help to <em>algorithmically</em> identify threats. Specifically, Russ shows you how to detect anomalies in security event logs as shown below.</p>
<p><img alt="Security Event Log Anomalies" src="https://i1.wp.com/www.business-science.io/assets/2018-06-10-infosec-anomalize/security_event_anomalies.png?w=456" data-recalc-dims="1"></p>
<h2>Threat Hunting With Anomalize</h2>
<blockquote>
<p>By Russ McCree, Group Program Manager of Microsoft’s Windows and Devices Group</p>
</blockquote>
<p>When, in <a rel="nofollow" href="https://holisticinfosec.blogspot.com/2017/10/toolsmith-128-dfir-redefined-deeper.html" target="_blank">October</a> and <a rel="nofollow" href="https://holisticinfosec.blogspot.com/2017/11/toolsmith-129-dfir-redefined-deeper.html" target="_blank">November</a>’s <a rel="nofollow" href="https://holisticinfosec.blogspot.com/" target="_blank">toolsmith</a> posts, I (Russ McRee) redefined <em>DFIR</em> under the premise of <strong>Deeper Functionality for Investigators in R</strong>, I discovered a “tip of the iceberg” scenario.<br>
To that end, I’d like to revisit the DFIR concept with an additional discovery and opportunity. In reality, this is really a case of Deeper Functionality for Investigators in R within the original and paramount activity of <a rel="nofollow" href="https://en.wikipedia.org/wiki/Digital_forensics" target="_blank"><strong>D</strong>igital <strong>F</strong>orensics/<strong>I</strong>ncident <strong>R</strong>esponse (DFIR)</a>.</p>
<h3>Massive Data Requires Algorithmic Methods</h3>
<p>Those of us in the DFIR practice, and Blue Teaming at large, are overwhelmed by data and scale. <strong>Success truly requires algorithmic methods</strong>. If you’re not already invested here I have an <strong>immediately applicable case study for you in tidy anomaly detection with anomalize</strong>.</p>
<p>First, let me give credit where entirely due for the work that follows. Everything I discuss and provide is immediately derivative from <em>Business Science</em> (<a rel="nofollow" href="https://twitter.com/bizScienc" target="_blank">@bizScienc</a>), specifically Matt Dancho (<a rel="nofollow" href="https://twitter.com/mdancho84" target="_blank">@mdancho84</a>). When a client asked <em>Business Science</em> to build an open source anomaly detection algorithm that suited their needs, he created <a rel="nofollow" href="https://business-science.github.io/anomalize/" target="_blank">anomalize</a>:</p>
<blockquote>
<p>“<em>a tidy anomaly detection algorithm that’s time-based (built on top of <code>tibbletime</code>) and scalable from one to many time series</em>,”</p>
</blockquote>
<p>I’d say he responded beautifully.</p>
<h3>Anomalizing in InfoSec: Threat Hunting At Scale</h3>
<p>When his <a rel="nofollow" href="http://www.business-science.io/code-tools/2018/04/08/introducing-anomalize.html" target="_blank">blogpost introducing anomalize</a> hit my (Russ’s) radar via <a rel="nofollow" href="https://www.r-bloggers.com/" target="_blank">R-Bloggers</a> it lived as an open tab in my browser for more than a month until generating <a rel="nofollow" href="https://holisticinfosec.blogspot.com/2018/06/toolsmith-133-anomaly-detection-threat.html" target="_blank">this <em>toolsmith</em> article</a>. Please consider Matt’s post a mandatory read as step one of the process here. I’ll quote Matt specifically before shifting context:</p>
<blockquote>
<p>“<em>Our client had a challenging problem: detecting anomalies in time series on daily or weekly data at scale. Anomalies indicate exceptional events, which could be increased web traffic in the marketing domain or a malfunctioning server in the IT domain. Regardless, it’s important to flag these unusual occurrences to ensure the business is running smoothly. One of the challenges was that the client deals with not one time series but thousands that need to be analyzed for these extreme events.</em>”</p>
<p>-Matt Dancho, creator of <code>anomalize</code></p>
</blockquote>
<p><strong>Key takeaway: Detecting anomalies in time series on daily or weekly data at scale. Anomalies indicate exceptional events.</strong></p>
<p>Now shift context with me to security-specific events and incidents, as they pertain to security monitoring, incident response, and threat hunting. In my <a rel="nofollow" href="https://holisticinfosec.blogspot.com/2017/11/toolsmith-129-dfir-redefined-deeper.html" target="_blank">November 2017 post on redefining DFIR</a>, I discussed <strong>Time Series Regression (TSR) with the Holt-Winters method and a focus on seasonality and trends</strong>. Unfortunately, I couldn’t share the code for how we applied TSR, but pointed out alternate methods, including <strong>Seasonal and Trend Decomposition using Loess (STL)</strong>, which:</p>
<ul>
<li>Handles any type of seasonality (which can change over time)</li>
<li>Automatically smooths the trend-cycle (this can also be controlled by the user)</li>
<li>Is robust to outliers (high leverage points that can shift the mean)</li>
</ul>
<p>Here now, Matt has created a means to immediately apply the <strong>STL method</strong>, along with the <strong>Twitter method</strong> (<a rel="nofollow" href="https://business-science.github.io/anomalize/reference/time_decompose.html" target="_blank">reference page</a>), as part of his <code>time_decompose()</code> function, one of three functions specific to the <code>anomalize</code> package. The <a rel="nofollow" href="https://github.com/business-science/anomalize" target="_blank">anomalize</a> package includes the following main functions:</p>
<ul>
<li>
<p><code>time_decompose()</code>: Separates the time series into seasonal, trend, and remainder components. The methods used including <strong>STL</strong> and <strong>Twitter</strong> are described in Matt’s <a rel="nofollow" href="https://business-science.github.io/anomalize/reference/time_decompose.html" target="_blank">reference page</a>.</p>
</li>
<li>
<p><code>anomalize()</code>: Applies anomaly detection methods to the remainder component. The methods used including <strong>IQR</strong> and <strong>GESD</strong> are described in Matt’s <a rel="nofollow" href="https://business-science.github.io/anomalize/reference/anomalize.html" target="_blank">reference page</a>.</p>
</li>
<li>
<p><code>time_recompose()</code>: Calculates limits that separate the “normal” data from the anomalies.</p>
</li>
</ul>
<p>Matt ultimately set out to build a scalable adaptation of <a rel="nofollow" href="https://github.com/twitter/AnomalyDetection" target="_blank">Twitter’s AnomalyDetection package</a> in order to address his client’s challenges in dealing with not one time series but thousands needing to be analyzed for extreme events. You’ll note that Matt describes <code>anomalize</code> using a dataset (<code>tidyverse_cran_downloads</code> that ships with <code>anomalize</code>) that contains the daily download counts of the 15 <a rel="nofollow" href="https://www.tidyverse.org/" target="_blank">tidyverse</a> packages from <a rel="nofollow" href="https://cran.r-project.org/" target="_blank">CRAN</a>, relevant as he leverages the tidyverse package.</p>
<p>I initially toyed with tweaking Matt’s demo to model downloads for security-specific R packages (yes, there are such things) from CRAN, including:</p>
<ul>
<li><a rel="nofollow" href="https://cran.r-project.org/web/packages/RAppArmor/index.html" target="_blank">RAppArmor</a></li>
<li><a rel="nofollow" href="https://cran.r-project.org/web/packages/net.security/index.html" target="_blank">net.security</a></li>
<li><a rel="nofollow" href="https://cran.r-project.org/web/packages/securitytxt/index.html" target="_blank">securitytxt</a></li>
<li><a rel="nofollow" href="https://cran.r-project.org/web/packages/cymruservices/index.html" target="_blank">cymruservices</a></li>
</ul>
<p>The latter two packages are courtesy of <a rel="nofollow" href="https://twitter.com/hrbrmstr" target="_blank">Bob Rudis</a> of our beloved <a rel="nofollow" href="https://holisticinfosec.blogspot.com/2014/09/toolsmith-jay-and-bob-strike-back-data.html" target="_blank">Data-Driven Security: Analysis, Visualization and Dashboards</a>. Alas, this was a mere rip and replace, and really didn’t exhibit the use of <code>anomalize</code> in a deserving, varied, truly security-specific context. That said, I was able to generate immediate results doing so, as seen in <strong>Figure 1: Security R Anomalies</strong>.</p>
<p><img alt="Security R Anomalies" src="https://i2.wp.com/www.business-science.io/assets/2018-06-10-infosec-anomalize/Figure1.PNG?w=456" data-recalc-dims="1"></p>
<p></p>
<p>As an initial experiment you can replace packages names with those of your choosing in <a rel="nofollow" href="https://business-science.github.io/anomalize/reference/tidyverse_cran_downloads.html" target="_blank">tidyverse_cran_downloads</a>, run it in R Studio, then tweak variable names and labels in the code per Matt’s <a rel="nofollow" href="https://github.com/business-science/anomalize" target="_blank">README page</a>.</p>
<h3>Code Tutorial: Anomalizing A Real Security Scenario</h3>
<p>I wanted to run <code>anomalize</code> against <strong>a real security data scenario</strong>, so I went back to the dataset from the original DFIR articles where I’d utilized counts of 4624 Event IDs per day, per user, on a given set of servers. As utilized originally, I’d represented results specific to only one device and user, but herein is the beauty of anomalize. We can achieve quick results across multiple times series (multiple systems/users). This premise is but one of many where time series analysis and seasonality can be applied to security data.</p>
<p>For those that would like to follow along, load the following libraries.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tibbletime</span><span>)</span><span>
</span><span>library</span><span>(</span><span>anomalize</span><span>)</span></code></pre>
</figure>
<p>I originally tried to write log data from <code>log.csv</code> straight to an <code>anomalize.R</code> script with <code>logs = read_csv(&quot;log.csv&quot;)</code> into a <a rel="nofollow" href="https://tibble.tidyverse.org/index.html" target="_blank">tibble</a> (ready your troubles with tibbles jokes), which was not being parsed accurately, particularly time attributes. To correct this, from Matt’s GitHub I grabbed <code>tidyverse_cran_downloads.R</code>, and modified it as follows:</p>
<figure>
<pre><code data-lang="r"><span># Path to security log data</span><span>
</span><span>logs_path</span><span> </span><span>&lt;-</span><span> </span><span>&quot;https://raw.githubusercontent.com/holisticinfosec/toolsmith_R/master/anomalize/log.csv&quot;</span><span>
</span><span># Group by server, Convert to tibbletime</span><span>
</span><span>security_access_logs</span><span> </span><span>&lt;-</span><span> </span><span>read_csv</span><span>(</span><span>logs_path</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>server</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>as_tbl_time</span><span>(</span><span>date</span><span>)</span><span>
</span><span>security_access_logs</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A time tibble: 198 x 3
## # Index:  date
## # Groups: server [3]
##    date       count server       
##    </code></pre></figure>