{"id":1948,"date":"2013-01-27T18:29:45","date_gmt":"2013-01-27T23:29:45","guid":{"rendered":"http:\/\/rud.is\/b\/?p=1948"},"modified":"2017-04-02T22:51:38","modified_gmt":"2017-04-03T03:51:38","slug":"once-more-into-the-prc-aggregated-breaches","status":"publish","type":"post","link":"https:\/\/rud.is\/b\/2013\/01\/27\/once-more-into-the-prc-aggregated-breaches\/","title":{"rendered":"Once More Into The [PRC Aggregated] Breaches"},"content":{"rendered":"

If you’re not on the SecurityMetrics<\/span>.org mailing list you missed an interaction about the Privacy Rights Clearinghouse Chronology of Data Breaches<\/a> data source started by Lance Spitzner (@lspitzner). You’ll need to subscribe to the list see the thread, but one innocent question put me down the path to taking a look at the aggregated data with the intent of helping folks understand the overall utility\/efficacy of it when trying to craft messages from it. <\/p>\n

Before delving into the data, please note that PRC does an excellent job<\/a> detailing source material for the data. They fully acknowledge some of the challenges with it, but a picture (or two) is worth a thousand caveats. (NOTE: Charts & numbers have been produced from January 20th, 2013 data)<\/em>.<\/p>\n

The first thing I did was try to get a feel for overall volume:<\/p>\n

Total breach record entries across all years (2005-present): 3573<\/b>
\nNumber of entries with ‘Total Records Lost’ filled in: 751<\/b>
\n% of entries with ‘Total Records Lost’ filled in: 21.0%<\/b><\/div>\n

Takeaway #1:<\/strong> Be very wary of using any “Total Records Breached<\/em>” data from this data set.<\/p><\/blockquote>\n

It may help to see that computation broken down by reporting source over the years that the data file spans:<\/p>\n

\"complete-records-by-source-across-years\"<\/center><\/p>\n

This view also gives us:<\/p>\n

Takeaway #2:<\/strong> Not all data sources span all years and some have very little data.<\/p><\/blockquote>\n

However, Lance’s original goal was to compare “human error” vs “technical hack”. To do this, he combined DISC<\/code>, PHYS<\/code>, PORT<\/code> & STAT<\/code> into one category (accidental\/human :: ACC-HUM<\/code>) and HACK<\/code>, CARD<\/code> & INSD<\/code> into another (malicious\/attack :: MAL-ATT<\/code>). Here’s what that looks like when broken down across reporting sources across time:<\/p>\n

\"breach-count-metatype-year\"<\/a>

(click to enlarge)<\/p><\/div><\/center><\/p>\n

This view provides another indicator that one might not want to place a great deal of faith on the PRC’s aggregation efforts. Why? It’s highly unlikely that DatalossDB had virtually no breach recordings in 2011 (in fact, it’s more than unlikely, it’s not true<\/span>). Further views will show some other potential misses from DatalossDB.<\/p>\n

Takeaway #3:<\/strong> Do not assume the components of this aggregated data set are complete.<\/p><\/blockquote>\n

We can get a further feel for data quality and also which reporting sources are weighted more heavily (i.e. which ones have more records, thus implicitly placing a greater reliance on them for any calculations) by looking at how many records they each contributed to the aggregated population each year:<\/p>\n

\"(click<\/a>

(click to enlarge)<\/p><\/div><\/center><\/p>\n

I’m not sure why 2008 & 2009 have such small bars for Databreaches.net and PHIPrivacy.net, and you can see the 2011 gap in the DatalossDB graph.<\/p>\n

At this point, I’d (maybe) trust some aggregate analysis of the HHS (via PHI), CA Attorney General & Media data, but would need to caveat any conclusions with the obvious biases introduced by each.<\/p>\n

Even with these issues, I really wanted a “big picture” view of the entire set and ended up creating the following two charts:<\/p>\n

\"(click<\/a>

(click to enlarge)<\/p><\/div><\/p>\n

\"(click<\/a>

(click to enlarge)<\/p><\/div><\/center><\/p>\n

(You’ll probably want to view the PDF documents of each : [1<\/a>] [2<\/a>] given how big they are.)<\/p>\n

Those charts show the number of breaches-by-type by month across the 2005-2013 span by reporting source. The only difference between the two is that the latter one is grouped by Lance’s “meta type” definition. These views enable us to see gaps in reporting by month (note the additional aggregation issue at the tail end of 2010 for DatalossDB) and also to get a feel for the trends of each band (note the significant increase in “unknown” in 2012 for DatalossDB).<\/p>\n

Takeaway #4:<\/strong> Do not ignore the “unknown” classification when performing analysis with this data set.<\/p><\/blockquote>\n

We can see other data issues if we look at it from other angles, such as the state the breach was recorded in:<\/p>\n

\"(click<\/a>

(click to enlarge)<\/p><\/div><\/center><\/p>\n

We can see at least three issues (missing value and occurrences recorded not in the US) from this view, but it seems the number of breaches mostly aligns with population<\/a> (discrepancies make sense given the lack of uniform breach reporting requirements).<\/p>\n

It’s also very difficult to do any organizational analysis (I’m a big fan of looking at “repeat offenders” in general) with this data set without some significant data cleansing\/normalization. For example, all of these are “Bank of America<\/em>“:<\/p>\n

[1] \"Bank of America\"                                                             \r\n[2] \"Wachovia, Bank of America, PNC Financial Services Group and Commerce Bancorp\"\r\n[3] \"Bank of America Corp.\"                                                       \r\n[4] \"Citigroup, Inc., Bank of America, Corp.\"<\/pre>\n
Without any cleansing, here are the orgs with two or more reported breaches since 2005:<\/p>\n