{"id":6685,"date":"2017-10-09T14:59:44","date_gmt":"2017-10-09T19:59:44","guid":{"rendered":"https:\/\/rud.is\/b\/?p=6685"},"modified":"2018-03-10T08:01:08","modified_gmt":"2018-03-10T13:01:08","slug":"enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r","status":"publish","type":"post","link":"https:\/\/rud.is\/b\/2017\/10\/09\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\/","title":{"rendered":"Enabling Concerned Visitors & Ethical Security Researchers with security.txt Web Security Policies (plus analyze them at-scale with R)"},"content":{"rendered":"

I’ve blogged a bit about robots.txt<\/code> — the rules file that documents a sites “robots exclusion” standard that instructs web crawlers what they can and cannot do (and how frequently they should do things when they are allowed to). This is a well-known and well-defined standard, but it’s not mandatory and often ignored by crawlers and content owners alike.<\/p>\n

There’s an emerging IETF draft<\/a> for a different type of site metadata that content owners should absolutely consider adopting. This one defines “web security policies” for a given site and has much in common with robots exclusion standard, including the name (security.txt<\/code>) and format (policy directives are defined with simple syntax — see Chapter 5 of the Debian Policy Manual<\/a>).<\/p>\n

One core difference is that this file is intended for humans. If you are are a general user and visit a site and notice something “off” (security-wise) or if you are an honest, honorable security researcher who found a vulnerability or weakness on a site, this security.txt<\/code> file should make it easier to contact the appropriate folks at the site to help them identify and resolve security issues. The IETF abstract summarizes the intent well:<\/p>\n

When security risks in web services are discovered by independent security researchers who understand the severity of the risk, they often lack the channels to properly disclose them. As a result, security issues may be left unreported. Security.txt defines a standard to help organizations define the process for security researchers to securely disclose security vulnerabilities.<\/div>\n

A big change from robots.txt<\/code> is where<\/em> the security.txt<\/code> file goes. The IETF standard is still in draft state so the location may change, but the current thinking is to have it go into \/.well-known\/security.txt<\/code> vs being placed in the top level root (i.e. it’s not supposed to be in \/security.txt<\/code>). If you aren’t familiar with the .well-known<\/code> directory, give RFC 5785<\/a> a read.<\/p>\n

You can visit the general information site<\/a> to find out more and install a development version of a Chrome extension<\/span> that will make it easier for pull up this info in your browser if you find an issue.<\/p>\n

Here’s the security.txt<\/code> for my site:<\/p>\n

Contact: bob@rud.is\r\nEncryption: https:\/\/keybase.io\/hrbrmstr\/pgp_keys.asc?fingerprint=e5388172b81c210906f5e5605879179645de9399\r\nDisclosure: Full<\/code><\/pre>\n
With that info, you know where to contact me, have the ability to encrypt your message and know that I’ll give you credit and will disclose the bugs openly.<\/p>\n
So, Why the [R] tag?<\/h3>\n
Ah, yes. This post is in the R<\/code> RSS category feed for a reason. I do at-scale analysis of the web for a living and will be tracking the adoption of security.txt<\/code> across the internet (initially with the Umbrella Top 1m and a choice list of sites with more categorical data associated with them) over time. My esteemed colleague @jhartftw is handling the crawling part, but I needed a way to speedily read in these files for a broader analysis. So, I made an R package: securitytxt<\/code>?<\/a>.<\/p>\n
It’s pretty easy to use. Here’s how to install it and use one of the functions to generate a security.txt<\/code> target URL for a site:<\/p>\n
devtools::install_github("hrbrmstr\/securitytxt")\r\n\r\nlibrary(securitytxt)\r\n\r\n(xurl <- sectxt_url("https:\/\/rud.is\/b"))\r\n## [1] "https:\/\/rud.is\/.well-known\/security.txt"<\/code><\/pre>\n
This is how you read in and parse a security.txt<\/code> file:<\/p>\n
(x <- sectxt(url(xurl)))\r\n## <Web Security Policies Object>\r\n## Contact: bob@rud.is\r\n## Encryption: https:\/\/keybase.io\/hrbrmstr\/pgp_keys.asc?fingerprint=e5388172b81c210906f5e5605879179645de9399\r\n## Disclosure: Full<\/code><\/pre>\n
And, this is how you turn that into a usable data frame:<\/p>\n
sectxt_info(x)\r\n##          key                                                                                         value\r\n## 1    contact                                                                                    bob@rud.is\r\n## 2 encryption https:\/\/keybase.io\/hrbrmstr\/pgp_keys.asc?fingerprint=e5388172b81c210906f5e5605879179645de9399\r\n## 3 disclosure                                                                                          Full<\/code><\/pre>\n
There’s also a function to validate that the keys are within the current IETF standard. That will become more useful once the standard moves out of draft status.<\/p>\n
FIN<\/h3>\n
So, definitely adopt the standard<\/em> and feel invited to kick the tyres on the package. Don’t hesitate to jump on board if you have ideas for how you’d like to extend the package, and drop a note in the comments if you have questions on it or on adopting the standard for your site.<\/p>\n","protected":false},"excerpt":{"rendered":"
I’ve blogged a bit about robots.txt — the rules file that documents a sites “robots exclusion” standard that instructs web crawlers what they can and cannot do (and how frequently they should do things when they are allowed to). This is a well-known and well-defined standard, but it’s not mandatory and often ignored by crawlers […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":3,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":""},"categories":[3,91],"tags":[810],"class_list":["post-6685","post","type-post","status-publish","format-standard","hentry","category-information-security","category-r","tag-post"],"yoast_head":"\nEnabling Concerned Visitors & Ethical Security Researchers with security.txt Web Security Policies (plus analyze them at-scale with R) - rud.is<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/rud.is\/b\/2017\/10\/09\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Enabling Concerned Visitors & Ethical Security Researchers with security.txt Web Security Policies (plus analyze them at-scale with R) - rud.is\" \/>\n<meta property=\"og:description\" content=\"I’ve blogged a bit about robots.txt — the rules file that documents a sites “robots exclusion” standard that instructs web crawlers what they can and cannot do (and how frequently they should do things when they are allowed to). This is a well-known and well-defined standard, but it’s not mandatory and often ignored by crawlers […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/rud.is\/b\/2017\/10\/09\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\/\" \/>\n<meta property=\"og:site_name\" content=\"rud.is\" \/>\n<meta property=\"article:published_time\" content=\"2017-10-09T19:59:44+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-03-10T13:01:08+00:00\" \/>\n<meta name=\"author\" content=\"hrbrmstr\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"hrbrmstr\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2017\\\/10\\\/09\\\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2017\\\/10\\\/09\\\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\\\/\"},\"author\":{\"name\":\"hrbrmstr\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\"},\"headline\":\"Enabling Concerned Visitors & Ethical Security Researchers with security.txt Web Security Policies (plus analyze them at-scale with R)\",\"datePublished\":\"2017-10-09T19:59:44+00:00\",\"dateModified\":\"2018-03-10T13:01:08+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2017\\\/10\\\/09\\\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\\\/\"},\"wordCount\":645,\"commentCount\":2,\"publisher\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\"},\"keywords\":[\"post\"],\"articleSection\":[\"Information Security\",\"R\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/rud.is\\\/b\\\/2017\\\/10\\\/09\\\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2017\\\/10\\\/09\\\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\\\/\",\"url\":\"https:\\\/\\\/rud.is\\\/b\\\/2017\\\/10\\\/09\\\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\\\/\",\"name\":\"Enabling Concerned Visitors & Ethical Security Researchers with security.txt Web Security Policies (plus analyze them at-scale with R) - rud.is\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#website\"},\"datePublished\":\"2017-10-09T19:59:44+00:00\",\"dateModified\":\"2018-03-10T13:01:08+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2017\\\/10\\\/09\\\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/rud.is\\\/b\\\/2017\\\/10\\\/09\\\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2017\\\/10\\\/09\\\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/rud.is\\\/b\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Enabling Concerned Visitors & Ethical Security Researchers with security.txt Web Security Policies (plus analyze them at-scale with R)\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#website\",\"url\":\"https:\\\/\\\/rud.is\\\/b\\\/\",\"name\":\"rud.is\",\"description\":\""In God we trust. All others must bring data"\",\"publisher\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/rud.is\\\/b\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\",\"name\":\"hrbrmstr\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\",\"width\":460,\"height\":460,\"caption\":\"hrbrmstr\"},\"logo\":{\"@id\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\"},\"description\":\"Don't look at me\u2026I do what he does \u2014 just slower. #rstats avuncular \u2022 ?Resistance Fighter \u2022 Cook \u2022 Christian \u2022 [Master] Chef des Donn\u00e9es de S\u00e9curit\u00e9 @ @rapid7\",\"sameAs\":[\"http:\\\/\\\/rud.is\"],\"url\":\"https:\\\/\\\/rud.is\\\/b\\\/author\\\/hrbrmstr\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Enabling Concerned Visitors & Ethical Security Researchers with security.txt Web Security Policies (plus analyze them at-scale with R) - rud.is","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/rud.is\/b\/2017\/10\/09\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\/","og_locale":"en_US","og_type":"article","og_title":"Enabling Concerned Visitors & Ethical Security Researchers with security.txt Web Security Policies (plus analyze them at-scale with R) - rud.is","og_description":"I’ve blogged a bit about robots.txt — the rules file that documents a sites “robots exclusion” standard that instructs web crawlers what they can and cannot do (and how frequently they should do things when they are allowed to). This is a well-known and well-defined standard, but it’s not mandatory and often ignored by crawlers […]","og_url":"https:\/\/rud.is\/b\/2017\/10\/09\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\/","og_site_name":"rud.is","article_published_time":"2017-10-09T19:59:44+00:00","article_modified_time":"2018-03-10T13:01:08+00:00","author":"hrbrmstr","twitter_card":"summary_large_image","twitter_misc":{"Written by":"hrbrmstr","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/rud.is\/b\/2017\/10\/09\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\/#article","isPartOf":{"@id":"https:\/\/rud.is\/b\/2017\/10\/09\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\/"},"author":{"name":"hrbrmstr","@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"headline":"Enabling Concerned Visitors & Ethical Security Researchers with security.txt Web Security Policies (plus analyze them at-scale with R)","datePublished":"2017-10-09T19:59:44+00:00","dateModified":"2018-03-10T13:01:08+00:00","mainEntityOfPage":{"@id":"https:\/\/rud.is\/b\/2017\/10\/09\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\/"},"wordCount":645,"commentCount":2,"publisher":{"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"keywords":["post"],"articleSection":["Information Security","R"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/rud.is\/b\/2017\/10\/09\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/rud.is\/b\/2017\/10\/09\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\/","url":"https:\/\/rud.is\/b\/2017\/10\/09\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\/","name":"Enabling Concerned Visitors & Ethical Security Researchers with security.txt Web Security Policies (plus analyze them at-scale with R) - rud.is","isPartOf":{"@id":"https:\/\/rud.is\/b\/#website"},"datePublished":"2017-10-09T19:59:44+00:00","dateModified":"2018-03-10T13:01:08+00:00","breadcrumb":{"@id":"https:\/\/rud.is\/b\/2017\/10\/09\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/rud.is\/b\/2017\/10\/09\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/rud.is\/b\/2017\/10\/09\/enabling-concerned-visitors-ethical-security-researchers-with-security-txt-web-security-policies-plus-analyze-them-at-scale-with-r\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/rud.is\/b\/"},{"@type":"ListItem","position":2,"name":"Enabling Concerned Visitors & Ethical Security Researchers with security.txt Web Security Policies (plus analyze them at-scale with R)"}]},{"@type":"WebSite","@id":"https:\/\/rud.is\/b\/#website","url":"https:\/\/rud.is\/b\/","name":"rud.is","description":""In God we trust. All others must bring data"","publisher":{"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/rud.is\/b\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886","name":"hrbrmstr","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","url":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","contentUrl":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","width":460,"height":460,"caption":"hrbrmstr"},"logo":{"@id":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1"},"description":"Don't look at me\u2026I do what he does \u2014 just slower. #rstats avuncular \u2022 ?Resistance Fighter \u2022 Cook \u2022 Christian \u2022 [Master] Chef des Donn\u00e9es de S\u00e9curit\u00e9 @ @rapid7","sameAs":["http:\/\/rud.is"],"url":"https:\/\/rud.is\/b\/author\/hrbrmstr\/"}]}},"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p23idr-1JP","jetpack_likes_enabled":true,"jetpack-related-posts":[{"id":6945,"url":"https:\/\/rud.is\/b\/2017\/11\/03\/i-for-one-welcome-our-forthcoming-new-robots-txt-overlords\/","url_meta":{"origin":6685,"position":0},"title":"I, For One, Welcome Our Forthcoming New robots.txt Overlords","author":"hrbrmstr","date":"2017-11-03","format":false,"excerpt":"Despite my week-long Twitter consumption sabbatical (helped --- in part --- by the nigh week-long internet and power outage here in Maine), I still catch useful snippets from folks. My cow-orker @dabdine shunted a tweet by @terrencehart into a Slack channel this morning, and said tweet contained a link to\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/11\/andy-kelly-402111.jpg?fit=1080%2C720&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/11\/andy-kelly-402111.jpg?fit=1080%2C720&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/11\/andy-kelly-402111.jpg?fit=1080%2C720&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/11\/andy-kelly-402111.jpg?fit=1080%2C720&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/11\/andy-kelly-402111.jpg?fit=1080%2C720&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":12060,"url":"https:\/\/rud.is\/b\/2019\/03\/05\/heads-up-roll-your-own-http-headers-investigations-with-the-hdrs-package\/","url_meta":{"origin":6685,"position":1},"title":"Head’s Up! Roll Your Own HTTP Headers Investigations with the ‘hdrs’ Package","author":"hrbrmstr","date":"2019-03-05","format":false,"excerpt":"I blathered alot about HTTP headers in the last post. In the event you wanted to dig deeper I threw together a small package that will let you grab HTTP headers from a given URL and take a look at them. The README has examples for most things but we'll\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":6465,"url":"https:\/\/rud.is\/b\/2017\/09\/25\/speeding-up-digital-arachinds\/","url_meta":{"origin":6685,"position":2},"title":"Speeding Up Digital Arachnids","author":"hrbrmstr","date":"2017-09-25","format":false,"excerpt":"spiderbar, spiderbar Reads robots rules from afar. Crawls the web, any size; Fetches with respect, never lies. Look Out! Here comes the spiderbar. Is it fast? Listen bud, It's got C++ under the hood. Can you scrape, from a site? Test with can_fetch(), TRUE == alright Hey, there There goes\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/spiderbar_1-1.png?fit=1200%2C720&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/spiderbar_1-1.png?fit=1200%2C720&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/spiderbar_1-1.png?fit=1200%2C720&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/spiderbar_1-1.png?fit=1200%2C720&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/spiderbar_1-1.png?fit=1200%2C720&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":6385,"url":"https:\/\/rud.is\/b\/2017\/09\/19\/pirating-web-content-responsibly-with-r\/","url_meta":{"origin":6685,"position":3},"title":"Pirating Web Content Responsibly With R","author":"hrbrmstr","date":"2017-09-19","format":false,"excerpt":"International Code Talk Like A Pirate Day almost slipped by without me noticing (September has been a crazy busy month), but it popped up in the calendar notifications today and I was glad that I had prepped the meat of a post a few weeks back. There will be no\u2026","rel":"","context":"In "data wrangling"","block_context":{"text":"data wrangling","link":"https:\/\/rud.is\/b\/category\/data-wrangling\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Plot_Zoom-2.png?fit=1200%2C917&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Plot_Zoom-2.png?fit=1200%2C917&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Plot_Zoom-2.png?fit=1200%2C917&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Plot_Zoom-2.png?fit=1200%2C917&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Plot_Zoom-2.png?fit=1200%2C917&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":12142,"url":"https:\/\/rud.is\/b\/2019\/04\/12\/a-note-to-our-community-on-how-to-hide-your-content-from-search-engines\/","url_meta":{"origin":6685,"position":4},"title":"A Note to Our Community On How To Hide Your Content From Search Engines","author":"hrbrmstr","date":"2019-04-12","format":false,"excerpt":"UPDATE 2019-04-17 \u2014 The example at the bottom which shows that the, er, randomly chosen site has the offending <meta> tag present is an old result. As of this update timestamp, that robots noindex tag is not on the site. Since the presence status of that tag is in flux,\u2026","rel":"","context":"In "Leadership"","block_context":{"text":"Leadership","link":"https:\/\/rud.is\/b\/category\/leadership\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":133,"url":"https:\/\/rud.is\/b\/2011\/02\/14\/metricon-measuring-metrics-programs-why-arent-we\/","url_meta":{"origin":6685,"position":5},"title":"Metricon: Measuring Metrics Programs (Why Aren’t We?)","author":"hrbrmstr","date":"2011-02-14","format":false,"excerpt":"Speaker: Jared Pfost (@JaredPfost) Framing: IT Security Metrics in an Enterprise \u00a0 If metrics are valuable, why aren't we measuring them. Virtually no research on them. \u00a0 The Chase Measuring metric program maturity would be easy, but not valuable Metric programs aren't a priority for enough CISOs for a benchmark\u2026","rel":"","context":"In "Information Security"","block_context":{"text":"Information Security","link":"https:\/\/rud.is\/b\/category\/information-security\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts\/6685","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/comments?post=6685"}],"version-history":[{"count":0,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts\/6685\/revisions"}],"wp:attachment":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/media?parent=6685"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/categories?post=6685"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/tags?post=6685"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}