{"id":11540,"date":"2018-09-19T09:26:17","date_gmt":"2018-09-19T14:26:17","guid":{"rendered":"https:\/\/rud.is\/b\/?p=11540"},"modified":"2018-09-19T11:57:07","modified_gmt":"2018-09-19T16:57:07","slug":"taking-a-tour-of-the-pirate-ship-github-dmca-with-r","status":"publish","type":"post","link":"https:\/\/rud.is\/b\/2018\/09\/19\/taking-a-tour-of-the-pirate-ship-github-dmca-with-r\/","title":{"rendered":"Taking a Tour of the Pirate Ship ‘GitHub DMCA’ with R"},"content":{"rendered":"

Despite having sailed through the core components of this year’s Talk Like A Pirate Day<\/a> R post a few months ago, time has been an enemy of late so this will be a short post that others can build off of, especially since there’s lots more knife work<\/strike> ground to cover from the data.<\/p>\n

DMC-WhAt?<\/h3>\n

Since this is TLAPD, I’ll pilfer some of the explanation from GitHub itself:<\/p>\n

The Digital Millennium Copyright Act (DMCA) <start_of_current_pilfer><\/span>“provides a safe harbor for service providers that host user-generated content. Since even a single claim of copyright infringement can carry statutory damages of up to $150,000, the possibility of being held liable for user-generated content could be very harmful for service providers. With potential damages multiplied across millions of users, cloud-computing and user-generated content sites like YouTube, Facebook, or GitHub probably never would have existed without the DMCA (or at least not without passing some of that cost downstream to their users).”<\/em><\/p>\n

“The DMCA addresses this issue by creating a copyright liability safe harbor for internet service providers hosting allegedly infringing user-generated content. Essentially, so long as a service provider follows the DMCA’s notice-and-takedown rules, it won’t be liable for copyright infringement based on user-generated content. Because of this, it is important for GitHub to maintain its DMCA safe-harbor status.”<\/em><\/end_of_current_pilfer><\/span><\/p>\n

(I’ll save you from a long fact- and opinion-based diatribe on the DMCA, but suffice it to say it’s done far more harm than good IMO. Also, hopefully the “piracy” connection makes sense, now :-)<\/p>\n

If your initial reaction was “What does the DMCA have to do with GitHub?”<\/em> it likely (quickly) turned to “Oh…GitHub is really just a version-controlled file sharing service…”<\/em>. As such it has to have a robust takedown policy<\/a> and process.<\/p>\n

I don’t know if Microsoft is going to keep the practice of being open about DMCA requests now that they own GitHub nor do I know if they’ll use the same process on themselves (since, as we’ll see, they have issued DMCA requests to GitHub in the past). For now, we’ll assume they will, thus making the code from this post usable in the future to check on the status of DMCA requests over a longer period of time. But first we need the data.<\/p>\n

Hunting for treasure in the data hoard<\/h3>\n

Unsurprisingly, GitHub stores DMCA data on GitHub<\/a>. Ironically, they store it openly — in-part — to shine a light on what giant, global megacorps like Microsoft are doing. Feel free to use one of the many R packages to clone the repo, but a simple command-line git clone git@github.com:github\/dmca.git<\/code> is quick and efficient (not everything needs to be done from R).<\/p>\n

The directory structure looks like this:<\/p>\n

\u251c\u2500\u2500 2011\n\u251c\u2500\u2500 2012\n\u251c\u2500\u2500 2013\n\u251c\u2500\u2500 2014\n\u251c\u2500\u2500 2015\n\u251c\u2500\u2500 2016\n\u251c\u2500\u2500 2017\n\u251c\u2500\u2500 2017-02-01-RBoyApps-2.md\n\u251c\u2500\u2500 2017-02-15-DeutscheBank.md\n\u251c\u2500\u2500 2017-03-13-Jetbrains.md\n\u251c\u2500\u2500 2017-06-26-Wipro-Counternotice.md\n\u251c\u2500\u2500 2017-06-30-AdflyLink.md\n\u251c\u2500\u2500 2017-07-28-Toontown-2.md\n\u251c\u2500\u2500 2017-08-31-Tourzan.md\n\u251c\u2500\u2500 2017-09-04-Random-House.md\n\u251c\u2500\u2500 2017-09-05-RandomHouse-2.md\n\u251c\u2500\u2500 2017-09-18-RandomHouse.md\n\u251c\u2500\u2500 2017-09-19-Ragnarok.md\n\u251c\u2500\u2500 2017-10-10-Broadcom.md\n\u251c\u2500\u2500 2018\n\u251c\u2500\u2500 2018-02-01-NihonAdSystems.md\n\u251c\u2500\u2500 2018-03-03-TuneIn.md\n\u251c\u2500\u2500 2018-03-16-Wabg.md\n\u251c\u2500\u2500 2018-05-17-Packt.md\n\u251c\u2500\u2500 2018-06-12-Suning.md\n\u251c\u2500\u2500 2018-07-31-Pearson.md\n\u251c\u2500\u2500 CONTRIBUTING.md\n\u251c\u2500\u2500 data\n\u2514\u2500\u2500 README.md<\/code><\/pre>\n
Unfortunately, the data<\/code> directory contains fools’ gold (it’s just high-level summary data).<\/p>\n
We want DMCA filer names, repo names, file names and the DMCA notice text (though we’ll be leaving NLP projects up to the intrepid readers). For that, it will mean processing the directories of notices.<\/p>\n
Notices are named (sadly, with some inconsistency) like this: 2018-03-15-Microsoft.md<\/code>. Year, month, date and name of org. The contents are text-versions of correspondence (usually email text) that have some requirements<\/a> in order to be processed. There’s also an online form<\/a> one can fill out but it’s pretty much a free text field with some semblance of structure. It’s up to humans to follow that structure and — as such — there is inconsistency in the text as well. (Perhaps this is a great lesson that non-constrained inputs and human-originated filenames aren’t a great plan for curating data stores.)<\/p>\n
You may have seen what look like takedown files in the top level of the repo. I have no idea if they are legit (since they aren’t in the structured directories) so we’ll be ignoring them.<\/p>\n
When I took a look at the directories, some files end in .markdown<\/code> but most end in .md<\/code>. We’ll cover both instances (you’ll need to replace \/data\/github\/dmca<\/code> with the prefix where you stored the repo:<\/p>\n
library(tools)\nlibrary(stringi)\nlibrary(hrbrthemes)\nlibrary(tidyverse)\n\nlist.files(\n  path = sprintf(\"\/data\/github\/dmca\/%s\", 2011:2018), \n  pattern = \"\\\\.md$|\\\\.markdown$\",\n  full.names = TRUE\n) -> dmca_files<\/code><\/pre>\n
As noted previously, we’re going to focus on DMCA views over time, look at organizations who filed DMCA notices and the notice content. It turns out the filenames also distinguish whether a notice is a takedown request or a counter-notice<\/a> (i.e. an “oops…my bad…” by a takedown originator) or a retraction, so we’ll collect that metadata as well. Finally, we’ll slurp up the text along the way.<\/p>\n
Again, I’ve taken a pass at this and found out the following:<\/p>\n