<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/10/09/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I’ve blogged a bit about <code>robots.txt</code> — the rules file that documents a sites “robots exclusion” standard that instructs web crawlers what they can and cannot do (and how frequently they should do things when they are allowed to). This is a well-known and well-defined standard, but it’s not mandatory and often ignored by crawlers and content owners alike.</p>
<p>There’s an <a rel="nofollow" href="https://tools.ietf.org/html/draft-foudil-securitytxt-00" target="_blank">emerging IETF draft</a> for a different type of site metadata that content owners should absolutely consider adopting. This one defines “web security policies” for a given site and has much in common with robots exclusion standard, including the name (<code>security.txt</code>) and format (policy directives are defined with simple syntax — see Chapter 5 of the <a rel="nofollow" href="https://www.debian.org/doc/debian-policy/#document-ch-controlfields" target="_blank">Debian Policy Manual</a>).</p>
<p>One core difference is that this file is intended for humans. If you are are a general user and visit a site and notice something “off” (security-wise) or if you are an honest, honorable security researcher who found a vulnerability or weakness on a site, this <code>security.txt</code> file should make it easier to contact the appropriate folks at the site to help them identify and resolve security issues. The IETF abstract summarizes the intent well:</p>
<div>When security risks in web services are discovered by independent security researchers who understand the severity of the risk, they often lack the channels to properly disclose them. As a result, security issues may be left unreported. Security.txt defines a standard to help organizations define the process for security researchers to securely disclose security vulnerabilities.</div>
<p>A big change from <code>robots.txt</code> is <em>where</em> the <code>security.txt</code> file goes. The IETF standard is still in draft state so the location may change, but the current thinking is to have it go into <code>/.well-known/security.txt</code> vs being placed in the top level root (i.e. it’s not supposed to be in <code>/security.txt</code>). If you aren’t familiar with the <code>.well-known</code> directory, give <a rel="nofollow" href="https://tools.ietf.org/html/rfc5785" target="_blank">RFC 5785</a> a read.</p>
<p>You can visit the <a rel="nofollow" href="https://securitytxt.org/" target="_blank">general information site</a> to find out more and install a development version of a <a rel="nofollow" href="https://github.com/securitytxt/Extension" target="_blank">Chrome extension</a> that will make it easier for pull up this info in your browser if you find an issue.</p>
<p>Here’s the <code>security.txt</code> for my site:</p>
<pre><code>Contact: bob@rud.is
Encryption: https://keybase.io/hrbrmstr/pgp_keys.asc?fingerprint=e5388172b81c210906f5e5605879179645de9399
Disclosure: Full</code></pre>
<p>With that info, you know where to contact me, have the ability to encrypt your message and know that I’ll give you credit and will disclose the bugs openly.</p>
<h3>So, Why the [R] tag?</h3>
<p>Ah, yes. This post is in the <code>R</code> RSS category feed for a reason. I do at-scale analysis of the web for a living and will be tracking the adoption of <code>security.txt</code> across the internet (initially with the Umbrella Top 1m and a choice list of sites with more categorical data associated with them) over time. My esteemed colleague <a rel="nofollow" href="http://twitter.com/jhartftw" target="_blank">@jhartftw</a> is handling the crawling part, but I needed a way to speedily read in these files for a broader analysis. So, I made an R package: <a rel="nofollow" href="https://github.com/hrbrmstr/securitytxt" target="_blank"><code>securitytxt</code><img alt="🔗" src="https://s.w.org/images/core/emoji/2/72x72/1f517.png" class="wp-smiley"></a>.</p>
<p>It’s pretty easy to use. Here’s how to install it and use one of the functions to generate a <code>security.txt</code> target URL for a site:</p>
<pre><code>devtools::install_github(&quot;hrbrmstr/securitytxt&quot;)

library(securitytxt)

(xurl &lt;- sectxt_url(&quot;https://rud.is/b&quot;))
## [1] &quot;https://rud.is/.well-known/security.txt&quot;</code></pre>
<p>This is how you read in and parse a <code>security.txt</code> file:</p>
<pre><code>(x &lt;- sectxt(url(xurl)))
## </code></pre>