{"id":3940,"date":"2016-02-14T08:17:25","date_gmt":"2016-02-14T13:17:25","guid":{"rendered":"http:\/\/rud.is\/b\/?p=3940"},"modified":"2018-03-13T07:35:14","modified_gmt":"2018-03-13T12:35:14","slug":"making-faceted-heatmaps-with-ggplot2","status":"publish","type":"post","link":"https:\/\/rud.is\/b\/2016\/02\/14\/making-faceted-heatmaps-with-ggplot2\/","title":{"rendered":"Making Faceted Heatmaps with ggplot2"},"content":{"rendered":"

We were doing some exploratory data analysis on some attacker data at work and one of the things I was interested is what were “working hours” by country. Now, I don’t put a great deal of faith in the precision of geolocated IP addresses since every geolocation database that exists thinks I live in Vermont (I don’t) and I know that these databases rely on a pretty “meh” distributed process for getting this local data. However, at a country level, the errors are tolerable provided you use a decent geolocation provider. Since a rant about the precision of IP address geolocation was not<\/em> the point of this post, let’s move on.<\/p>\n

One of the best ways to visualize these “working hours” is a temporal heatmap. Jay & I made a couple<\/a> as part of our inaugural Data-Driven Security Book blog post to show how much of our collected lives were lost during the creation of our tome.<\/p>\n

I have some paired-down, simulated data based on the attacker data we were looking at. Rather than the complete data set, I’m providing 200,000 “events” (RDP login attempts, to be precise) in the eventlog.csv<\/code> file in the data\/<\/code> directory that have the timestamp<\/code>, and the source_country<\/code> ISO 3166-1 alpha-2 country code (which is the source of the attack) plus the tz<\/code> time zone of the source IP address. Let’s have a look:<\/p>\n

library(data.table)  # faster fread() and better weekdays()\r\nlibrary(dplyr)       # consistent data.frame operations\r\nlibrary(purrr)       # consistent & safe list\/vector munging\r\nlibrary(tidyr)       # consistent data.frame cleaning\r\nlibrary(lubridate)   # date manipulation\r\nlibrary(countrycode) # turn country codes into pretty names\r\nlibrary(ggplot2)     # base plots are for Coursera professors\r\nlibrary(scales)      # pairs nicely with ggplot2 for plot label formatting\r\nlibrary(gridExtra)   # a helper for arranging individual ggplot objects\r\nlibrary(ggthemes)    # has a clean theme for ggplot2\r\nlibrary(viridis)     # best. color. palette. evar.\r\nlibrary(knitr)       # kable : prettier data.frame output\r\n\r\nattacks <- tbl_df(fread("data\/eventlog.csv"))\r\n\r\nkable(head(attacks))<\/code><\/pre>\n\n\n\n\n\n\n\n\n\n\n
timestamp<\/th>\nsource_country<\/th>\ntz<\/th>\n<\/tr>\n<\/thead>\n
2015-03-12T15:59:16.718901Z<\/td>\nCN<\/td>\nAsia\/Shanghai<\/td>\n<\/tr>\n
2015-03-12T16:00:48.841746Z<\/td>\nFR<\/td>\nEurope\/Paris<\/td>\n<\/tr>\n
2015-03-12T16:02:26.731256Z<\/td>\nCN<\/td>\nAsia\/Shanghai<\/td>\n<\/tr>\n
2015-03-12T16:02:38.469907Z<\/td>\nUS<\/td>\nAmerica\/Chicago<\/td>\n<\/tr>\n
2015-03-12T16:03:22.201903Z<\/td>\nCN<\/td>\nAsia\/Shanghai<\/td>\n<\/tr>\n
2015-03-12T16:03:45.984616Z<\/td>\nCN<\/td>\nAsia\/Shanghai<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
For a temporal heatmap, we’re going to need the weekday and hour (or as granular as you want to get). I use a factor<\/code> here so I can have ordered weekdays. I need the source timezone weekday\/hour so we have to get a bit creative since the time zone parameter to virtually every date\/time operation in R only handles a single element vector.<\/p>\n
make_hr_wkday <- function(cc, ts, tz) {\r\n\r\n  real_times <- ymd_hms(ts, tz=tz[1], quiet=TRUE)\r\n\r\n  data_frame(source_country=cc,\r\n             wkday=weekdays(as.Date(real_times, tz=tz[1])),\r\n             hour=format(real_times, "%H", tz=tz[1]))\r\n\r\n}\r\n\r\ngroup_by(attacks, tz) %>%\r\n  do(make_hr_wkday(.$source_country, .$timestamp, .$tz)) %>%\r\n  ungroup() %>%\r\n  mutate(wkday=factor(wkday,\r\n                      levels=levels(weekdays(0, FALSE)))) -> attacks\r\nkable(head(attacks))<\/code><\/pre>\n\n\n\n\n\n\n\n\n\n\n
tz<\/th>\nsource_country<\/th>\nwkday<\/th>\nhour<\/th>\n<\/tr>\n<\/thead>\n
Africa\/Cairo<\/td>\nBG<\/td>\nSaturday<\/td>\n22<\/td>\n<\/tr>\n
Africa\/Cairo<\/td>\nTW<\/td>\nSunday<\/td>\n08<\/td>\n<\/tr>\n
Africa\/Cairo<\/td>\nTW<\/td>\nSunday<\/td>\n10<\/td>\n<\/tr>\n
Africa\/Cairo<\/td>\nCN<\/td>\nSunday<\/td>\n13<\/td>\n<\/tr>\n
Africa\/Cairo<\/td>\nUS<\/td>\nSunday<\/td>\n17<\/td>\n<\/tr>\n
Africa\/Cairo<\/td>\nCA<\/td>\nMonday<\/td>\n13<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
It’s pretty straightforward to make an overall heatmap of activity. Group & count the number of “attacks” by weekday and hour then use geom_tile()<\/code>. I’m going to clutter up the pristine ggplot2 commands with some explanation for those still learning ggplot2:<\/p>\n
wkdays <- count(attacks, wkday, hour)\r\n\r\nkable(head(wkdays))<\/code><\/pre>\n\n\n\n\n\n\n\n\n\n\n
wkday<\/th>\nhour<\/th>\nn<\/th>\n<\/tr>\n<\/thead>\n
Sunday<\/td>\n00<\/td>\n1076<\/td>\n<\/tr>\n
Sunday<\/td>\n01<\/td>\n1307<\/td>\n<\/tr>\n
Sunday<\/td>\n02<\/td>\n1189<\/td>\n<\/tr>\n
Sunday<\/td>\n03<\/td>\n1301<\/td>\n<\/tr>\n
Sunday<\/td>\n04<\/td>\n1145<\/td>\n<\/tr>\n
Sunday<\/td>\n05<\/td>\n1313<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Here, we’re just feeding in the new data.frame<\/code> we just created to ggplot and telling it we want to use the hour<\/code> column for the x-axis, the wkday<\/code> column for the y-axis and that we are doing a continuous scale fill by the n<\/code> aggregated count:<\/p>\n
gg <- ggplot(wkdays, aes(x=hour, y=wkday, fill=n))<\/code><\/pre>\n
This does all the hard work. geom_tile()<\/code> will make tiles at each x&y location we’ve already specified. I knew we had events for every hour, but if you had missing days or hours, you could use tidyr::complete()<\/code> to fill those in. We’re also telling it to use a thin (0.1 units) white border to separate the tiles.<\/p>\n
gg <- gg + geom_tile(color="white", size=0.1)<\/code><\/pre>\n
this has some additional magic in that it’s an awesome color scale. Read the viridis<\/code> package vignette for more info. By specifying a name<\/code> here, we get a nice label on the legend.<\/p>\n
gg <- gg + scale_fill_viridis(name="# Events", label=comma)<\/code><\/pre>\n
This ensures the plot will have a 1:1 aspect ratio (i.e. geom_tile()<\/code>–which draws rectangles–will draw nice squares).<\/p>\n
gg <- gg + coord_equal()<\/code><\/pre>\n
This tells ggplot to not use an x- or y-axis label and to also not reserve any space for them. I used a pretty bland but descriptive title. If I worked for some other security company I’d’ve added “ZOMGOSH CHINA!”<\/strong> to it.<\/p>\n
gg <- gg + labs(x=NULL, y=NULL, title="Events per weekday & time of day")<\/code><\/pre>\n
Here’s what makes the plot look really<\/em> nice. I customize a number of theme elements, starting with a base theme of theme_tufte()<\/code> from the ggthemes package. It removes alot of chart junk without having to do it manually.<\/p>\n
gg <- gg + theme_tufte(base_family="Helvetica")<\/code><\/pre>\n
I like my plot titles left-aligned. For hjust<\/code>:<\/p>\n