The BBC did a pretty good job [live tracking the Scotland secession vote](http://www.bbc.com/news/events/scotland-decides/results), but I really didn’t like the color scheme they chose and decided to use the final tally site as the basis for another tutorial using the tools from the Hadleyverse and taking advantage of the fact that newer `gdal` libraries can read in [TopoJSON](https://github.com/mbostock/topojson)/GeoJSON files, meaning we can use _most_ of the maps the D3-ers create/use right in R.
We’ll need a few R packages to help us get, clean, format and chart the data:
library(rvest) library(dplyr) library(httr) # >0.5 library(tidyr) library(gpclib) library(rgeos) library(sp) library(maptools) library(rgdal) # needs gdal > 1.11.0 library(ggplot2) library(reshape2) library(gridExtra)
The new `rvest` package makes it super-fun (and easy) to get data out of web pages (as I’ve [mentioned on the sister blog](http://datadrivensecurity.info/blog/posts/2014/Sep/migrating-to-rvest/)), but said data is still web page data, usually geared towards making things render well in a browser, and we end up having to clean up the extracted fields to get useful data. Since we usually want a data frame from the extraction, an `rvest` idiom I’ve been playing with involves bundling the element extraction & cleanup code into one function and then using that to build the columns:
# extract data from rvest-ed <div>'s and clean it up a bit # pass in the rvested HTML object and the CSS selector to extract, also # indicating whether we want a number or character vector returned extractAndCleanup <- function(data, selector, make_numeric=FALSE) { x <- data %>% html_nodes(selector) %>% html_text() x <- gsub("^[[:punct:][:space:]]*|[[:punct:][:space:]]*$", "", x) if (make_numeric) x <- as.numeric(gsub("[,[:space:]]*", "", x)) x } bbc_vote <- html("http://www.bbc.com/news/events/scotland-decides/results") secede <- data.frame( council=bbc_vote %>% extractAndCleanup(".body-row__cell--council"), electorate=bbc_vote %>% extractAndCleanup(".body-row__cell--electorate", TRUE), yes=bbc_vote %>% extractAndCleanup(".body-row__cell--yes", TRUE), no=bbc_vote %>% extractAndCleanup(".body-row__cell--no", TRUE), stringsAsFactors=FALSE)
We can then compute whether the vote tally was to secede or not and assign a color in the event we choose to use base graphics for plotting (we won’t for this tutorial). I chose a muted version of the Union Jack red and the official Scottish blue for this exercise.
secede <- secede %>% mutate(gone=yes>no, color=ifelse(gone, "#0065BD", "#CF142B77"))
Getting the map from the BBC site is just as simple. An inspection of the site in any decent browser with a “Developer” mode lets us see the elements being downloaded. For the BBC map, it reads the data from: `http://static.bbci.co.uk/news/1.49.0-1192/js/data/maps/l/scotland-elections.js` which is a TopoJSON object wrapped in two lines of extra javascript code. We’ll grab that file, clean it up and read the map into R using `httr`’s new-ish ability to save to a data file:
GET("http://static.bbci.co.uk/news/1.49.0-1192/js/data/maps/l/scotland-elections.js", write_disk("data/scotland.json"), progress()) tmp <- readLines("data/scotland.json") dir.create("data") writeLines(tmp[2], "data/scotland.json") map <- readOGR("data/scotland.json", "scotland-elections")
We’ll want to work with the map using Council names, so we need to ensure the names from the extracted `div` elements match what’s in the TopoJSON file:
secede$council %in% map@data$name ## [1] TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE ## [13] FALSE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE TRUE TRUE FALSE ## [25] TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE
It looks like we’ll need to clean the names up a bit, but thankfully the names aren’t too far off:
secede$council <- gsub("&", "and", secede$council) secede[secede$council=="Edinburgh",]$council = "City of Edinburgh" secede[secede$council=="Glasgow",]$council = "Glasgow City" secede[secede$council=="Comhairle nan Eilean Siar",]$council = "Na h-Eileanan an Iar"
If we were using base graphics for plotting, we’d also have to ensure the data was in the right order:
secede$council <- factor(secede$council, map@data$name, ordered=TRUE) secede <- secede %>% arrange(council)
We’re going to use `ggplot` for the mapping portion, but the normal `fortify` process didn’t work on this TopoJSON file (some polygon errors emerged), so we’ll take another route and do the data Council name↔id mapping after the `fortify` call and merge the rest of our data into the map data frame:
map_df <- fortify(map) # manually associate the map id's with the Council names and vote data councils <- data.frame(id=0:(length(map@data$name)-1), council=as.character(map@data$name)) map_df <- merge(map_df, councils, by="id") map_df <- merge(map_df, secede, by="council")
Now we can generate the choropleth:
gg <- ggplot() gg <- gg + geom_map(data=map_df, map=map_df, aes(map_id=id, x=long, y=lat, group=group, fill=color), color="white", size=0.25) gg <- gg + scale_fill_manual(values=rev(unique(secede$color)), labels=c("Yes", "No"), name="Secede?") gg <- gg + xlim(extendrange(r=range(coordinates(map)[,1]), f=0.15)) gg <- gg + ylim(extendrange(r=range(coordinates(map)[,2]), f=0.07)) gg <- gg + coord_map() gg <- gg + labs(x="", y="") gg <- gg + theme_bw() gg <- gg + theme(panel.grid=element_blank()) gg <- gg + theme(legend.position="none") gg <- gg + theme(panel.border=element_blank()) gg <- gg + theme(axis.ticks=element_blank()) gg <- gg + theme(axis.text=element_blank())
A choropleth is all well-and-good, but—since we have the data–let’s add the bar chart to complete the presentation. We’ll combine some `dplyr` and `tidyr` calls to melt and subset our data frame:
secede_m <- secede %>% gather(variable, value, -council) %>% filter(variable %in% c("yes", "no")) %>% mutate(value=as.numeric(value))
For this exercise, we’ll plot the 100% stacked bars in order of the “No” votes, and we’ll pre-process this ordering to make the `ggplot` code easier on the eyes. We start by merging some data back into our melted data frame so we can build the sorted factor by the “No” value column and then make sure the Councils will be in that order:
secede_m <- merge(secede_m, secede, by="council") secede_m$variable <- factor(secede_m$variable, levels=c("yes", "no"), ordered=TRUE) secede_m <- secede_m %>% arrange(no, variable) secede_m$council <- factor(secede_m$council, unique(secede_m$council), ordered=TRUE)
Finally, we can create the 100% stacked bar plot and combine it with the choropleth to build the final product:
gg1 <- ggplot(secede_m, aes(x=council, y=value, fill=factor(variable))) gg1 <- gg1 + geom_bar(stat="identity", position="fill") gg1 <- gg1 + scale_fill_manual(values=rev(unique(secede$color)), labels=c("Yes", "No"), name="Secede?") gg1 <- gg1 + geom_hline(yintercept=0.50, color="gray80") gg1 <- gg1 + geom_text(aes(label=percent(yes/100)), y=0.08, color="white", size=3) gg1 <- gg1 + geom_text(aes(label=percent(no/100)), y=0.92, color="white", size=3) gg1 <- gg1 + coord_flip() gg1 <- gg1 + labs(x="", y="") gg1 <- gg1 + theme_bw() gg1 <- gg1 + theme(panel.grid=element_blank()) gg1 <- gg1 + theme(legend.position="top") gg1 <- gg1 + theme(panel.border=element_blank()) gg1 <- gg1 + theme(axis.ticks=element_blank()) gg1 <- gg1 + theme(axis.text.x=element_blank()) vote <- arrangeGrob(gg1, gg, ncol=2, main=textGrob("Scotland Votes", gp=gpar(fontsize=20)))
(Click for larger version)
I’ve bundled this code up into it’s own [github repo](https://github.com/hrbrmstr/secede-2014). The full project example has a few extra features as
– it shows how to save the resultant data frame to an R data file (in case the BBC nukes the site)
– also saves the cleaned-up JSON (getting minimal Scotland shapefiles is tricky so this one’s a keeper even with the polygon errors)
– wraps all that in `if` statements so future analysis/vis can work with or without the live data being available.
Hadley really has to stop making R so fun to work with :-)
UPDATE
Based on a comment by Paul Drake suggesting that the BBC choropleth (and, hence, my direct clone of it) could be made more informative by showing the vote difference. Since it’s just changing two lines of code, here it is in-situ vs creating a new post.
gg <- gg + geom_map(data=map_df, map=map_df, aes(map_id=id, x=long, y=lat, group=group, fill=yes-no), color="white", size=0.25) gg <- gg + scale_fill_gradient(low="#CF142B", high="#0065BD", name="Secede?\n(vote margin)", guide="legend")
