{"id":6945,"date":"2017-11-03T08:18:12","date_gmt":"2017-11-03T13:18:12","guid":{"rendered":"https:\/\/rud.is\/b\/?p=6945"},"modified":"2018-03-07T16:57:13","modified_gmt":"2018-03-07T21:57:13","slug":"i-for-one-welcome-our-forthcoming-new-robots-txt-overlords","status":"publish","type":"post","link":"https:\/\/rud.is\/b\/2017\/11\/03\/i-for-one-welcome-our-forthcoming-new-robots-txt-overlords\/","title":{"rendered":"I, For One, Welcome Our Forthcoming New robots.txt Overlords"},"content":{"rendered":"

\"\"<\/a><\/p>\n

Despite my week-long Twitter consumption sabbatical (helped — in part — by the nigh week-long internet and power outage here in Maine), I still catch useful snippets from folks. My cow-orker @dabdine shunted a tweet by @terrencehart into a Slack channel this morning, and said tweet contained a link to this little gem<\/a>. Said gem is the text of a very recent ruling from a District Court in Texas and deals with a favourite subject of mine: robots.txt<\/code>.<\/p>\n

The background of the case is that there were two parties who both ran websites for oil and gas professionals that include job postings. One party filed a lawsuit against the other asserting that the they hacked into their system and accessed and used various information in violation of the Computer Fraud and Abuse Act (CFAA), the Stored Wire and Electronic Communications and Transactional Records Access Act (SWECTRA), the Racketeer Influenced and Corrupt Organizations Act (RICO), the Texas Harmful Access by Computer Act (THACA), the Texas Theft Liability Act (TTLA), and the Texas Uniform Trade Secrets Act (TUTS). They also asserted common law claims of misappropriation of confidential information, conversion, trespass to chattels, fraud, breach of fiduciary duty, unfair competition, tortious interference with present and prospective business relationships, civil conspiracy, and aiding and abetting.<\/p>\n

The other party filed a motion for dismissal on a number of grounds involving legalese on Terms & Conditions, a rebuttal of CFAA claims and really gnarly legalese around copyrights. There are more than a few paragraphs that make me glad none of my offspring have gone into or have a desire to go into the legal profession. One TLDR here is that T&Cs do, in fact, matter (though that is definitely dependent upon the legal climate where you live or have a case filed against you). We’re going to focus on the DMCA claim which leads us to the robots.txt<\/code> part.<\/p>\n

I shall also preface the rest with “IANAL”, but I don’t think a review of this case requires a law degree.<\/p>\n

Command-Shift-R<\/h3>\n

To refresh memories (or create lasting new ones), robots.txt<\/code> is a file that is placed at the top of a web site domain (i.e. https:\/\/rud.is\/robots.txt<\/a>) that contains robots exclusion standard<\/a> rules. These rules tell bots (NOTE: if you write a scraper, you’ve written a scraping bot) what they can or cannot scrape and what — if any — delay should be placed between scraping efforts by said bot.<\/p>\n

R has two CRAN packages for dealing with these files\/rules: robotstxt<\/code><\/a> by Peter Meissner and spiderbar<\/code><\/a> by me. They are not competitors, but are designed much like Reese’s Peanut Butter cups — to go together (though Peter did some wicked good testing<\/em> and noted a possible error in the underlying C++ library I use that could generate Type I or Type II in certain circumstances) and each has some specialization. I note them now because you don’t have an excuse not to check robots.txt<\/code> given two CRAN packages being available. Python folks have (at a minimum) robotparser<\/code> and reppy<\/code>. Node, Go and other, modern languages all have at least one module\/library\/package available as well. No. Excuses.<\/p>\n

Your Point?<\/h3>\n

(Y’all are always in a rush, eh?)<\/em><\/p>\n

This October, 2017 Texas ruling references a 2007 ruling by a District Court in Pennsylvania<\/a>. I dug in a bit through searchable Federal case law<\/a> for mentions of robots.txt<\/code> and there aren’t many U.S. cases that mention this control, though I am amused a small cadre of paralegals had to type robots.txt<\/code> over-and-over again.<\/p>\n

The dismissal request on the grounds that the CFAA did not apply was summarily rejected<\/em>. Why? The defendant provided proof that they monitor for scraping activity that violates the robots.txt<\/code> rules and that they use the Windows Firewall (ugh, they use Windows for web serving) to block offending IP addresses when they discover them.<\/p>\n

Nuances came out further along in the dismissal text noting that user-interactive viewing of the member profiles on the site was well-within the T&Cs but that the defendant “never authorized [the use of] automated bots to download over 500,000 profiles”<\/em> nor to have that data used for commercial purposes.<\/p>\n

The kicker (for me, anyway) is the last paragraph of the document in the Conclusion where the defendant asserts that:<\/p>\n