2016-08-13 UPDATE: Fortune
has a story on this and it does seem to be tax-related vs ideology. @thosjleeper suggested something similar as well about a week ago.
If you’re even remotely following the super insane U.S. 2016 POTUS circus election you’ve no doubt seen a resurgence of _”if X gets elected, I’m moving to Y”_ claims by folks who are “anti” one candidate or another. The [Washington Examiner](http://www.washingtonexaminer.com/americans-renouncing-citizenship-near-record-highs/article/2598074) did a story on last quarter’s U.S. expatriation numbers. I didn’t realize we had a department in charge of tracking and posting that data, but we do thanks to inane bureaucratic compliance laws.
I should have put _”posting that data”_ in quotes as it’s collected quarterly and posted ~2 months later in non-uniform HTML and PDF form across individual posts in a unique/custom Federal Register publishing system. How’s that hope and change in “open government data” working out for y’all?
The data is organized enough that we can take a look at the history of expatriation with some help from R. Along the way we’ll:
– see how to make parameterized web requests a bit cleaner with `httr`
– get even _more_ practice using the `purrr` package
– perhaps learn a new “trick” when using the `stringi` package
– show how we can “make do” living in a non-XPath 2 world (it’s actually pretty much at XPath 3 now, too #sigh)
A manual hunt on that system will eventually reveal a search URL that you can use in a `read.csv()` (to grab a list of URLs with the data, not the data itself #ugh). Those URLs are _gnarly_ (you’ll see what I mean if you do the hunt) but we can take advantage of the standardized URL query parameter that are used in the egregiously long URLs in a far more readable fashion if we use `httr::GET()` directly, especially since `httr::content()` will auto-convert the resultant CSV to a `tibble` for us since the site sets the response MIME type appropriately.
Unfortunately, when using the `6039G` search parameter (the expatriate tracking form ID) we do need to filter out non-quarterly report documents since the bureaucrats must have their ancillary TPS reports.
library(dplyr)
library(httr)
library(rvest)
library(purrr)
library(lubridate)
library(ggplot2) # devtools::install_github("hadley/ggplot2")
library(hrbrmisc) # devtools::install_github("hrbrmstr/hrbrmisc")
library(ggalt)
library(grid)
library(scales)
library(magrittr)
library(stringi)
GET("https://www.federalregister.gov/articles/search.csv",
query=list(`conditions[agency_ids][]`=254,
`conditions[publication_date][gte]`="01/01/2006",
`conditions[publication_date][lte]`="7/29/2016",
`conditions[term]`="6039G",
`conditions[type][]`="NOTICE")) %>%
content("parsed") %>%
filter(grepl("^Quarterly", title)) -> register
glimpse(register)
## Observations: 44
## Variables: 9
## $ citation <chr> "81 FR 50058", "81 FR 27198", "81 FR 65...
## $ document_number <chr> "2016-18029", "2016-10578", "2016-02312...
## $ title <chr> "Quarterly Publication of Individuals, ...
## $ publication_date <chr> "07/29/2016", "05/05/2016", "02/08/2016...
## $ type <chr> "Notice", "Notice", "Notice", "Notice",...
## $ agency_names <chr> "Treasury Department; Internal Revenue ...
## $ html_url <chr> "https://www.federalregister.gov/articl...
## $ page_length <int> 9, 17, 16, 20, 8, 20, 16, 12, 9, 15, 8,...
## $ qtr <date> 2016-06-30, 2016-03-31, 2015-12-31, 20...
Now, we grab the content at each of the `html_url`s and save them off to be kind to bandwidth and/or folks with slow connections (so you don’t have to re-grab the HTML):
docs <- map(register$html_url, read_html)
saveRDS(docs, file="deserters.rds")
That generates a list of parsed HTML documents.
The reporting dates aren’t 100% consistent (i.e. not always “n” weeks from the collection date), but the data collection dates _embedded textually in the report_ are (mostly…some vary in the use of upper/lower case). So, we use the fact that these are boring legal documents that use the same language for various phrases and extract the “quarter ending” dates so we know what year/quarter the data is relevant for:
register %<>%
mutate(qtr=map_chr(docs, ~stri_match_all_regex(html_text(.), "quarter ending ([[:alnum:], ]+)\\.",
opts_regex=stri_opts_regex(case_insensitive=TRUE))[[1]][,2]),
qtr=mdy(qtr))
I don’t often use that particular `magrittr` pipe, but it “feels right” in this case and is handy in a pinch.
If you visit some of the URLs directly, you’ll see that there are tables and/or lists of names of the expats. However, there are woefully inconsistent naming & formatting conventions for these lists of names *and* (as I noted earlier) there’s no XPath 2 support in R. Therefore, we have to make a slightly more verbose XPath query to target the necessary table for scraping since we need to account for vastly different column name structures for the tables we are targeting.
NOTE: Older HTML pages may not have HTML tables at all and some only reference PDFs, so don’t rely on this code working beyond these particular dates (at least consistently).
We’ll also tidy up the data into a neat `tibble` for plotting.
map(docs, ~html_nodes(., xpath=".//table[contains(., 'First name') or
contains(., 'FIRST NAME') or
contains(., 'FNAME')]")) %>%
map(~html_table(.)[[1]]) -> tabs
data_frame(date=register$qtr, count=map_int(tabs, nrow)) %>%
filter(format(as.Date(date), "%Y") >= 2006) -> left
With the data wrangling work out of the way, we can tally up the throngs of folks desperate for greener pastures. First, by quarter:
gg <- ggplot(left, aes(date, count))
gg <- gg + geom_lollipop()
gg <- gg + geom_label(data=data.frame(),
aes(x=min(left$date), y=1500, label="# individuals"),
family="Arial Narrow", fontface="italic", size=3, label.size=0, hjust=0)
gg <- gg + scale_x_date(expand=c(0,14), limits=range(left$date))
gg <- gg + scale_y_continuous(expand=c(0,0), label=comma, limits=c(0,1520))
gg <- gg + labs(x=NULL, y=NULL,
title="A Decade of Desertion",
subtitle="Quarterly counts of U.S. individuals who have chosen to expatriate (2006-2016)",
caption="Source: https://www.federalregister.gov/")
gg <- gg + theme_hrbrmstr_an(grid="Y")
gg

and, then annually:
left %>%
mutate(year=format(date, "%Y")) %>%
count(year, wt=count) %>%
ggplot(aes(year, n)) -> gg
gg <- gg + geom_bar(stat="identity", width=0.6)
gg <- gg + geom_label(data=data.frame(), aes(x=0, y=5000, label="# individuals"),
family="Arial Narrow", fontface="italic", size=3, label.size=0, hjust=0)
gg <- gg + scale_y_continuous(expand=c(0,0), label=comma, limits=c(0,5100))
gg <- gg + labs(x=NULL, y=NULL,
title="A Decade of Desertion",
subtitle="Annual counts of U.S. individuals who have chosen to expatriate (2006-2016)",
caption="Source: https://www.federalregister.gov/")
gg <- gg + theme_hrbrmstr_an(grid="Y")
gg

The exodus isn’t _massive_ but it’s actually more than I expected. It’d be interesting to track various US tax code laws, enactment of other compliance regulations and general news events to see if there are underlying reasons for the overall annual increases but also the dips in some quarters (which could just be data collection hiccups by the Feds…after all, this is government work). If you want to do all the math for correcting survey errors, it’d also be interesting to normalize this by population and track all the data back to 1996 (when HIPPA mandated the creation & publication of this quarterly list) and then see if you can predict where we’ll be at the end of this year (though I suspect political events are a motivator for at least a decent fraction of some of the quarters).
The Devil is in the Details
The [first public informational video](https://www.greatagain.gov/news/message-president-elect-donald-j-trump.html) from the PEOTUS didn’t add a full transcript of the video to the web site and did not provide (at least as of 0700 EST on 2016-11-22) their own text annotations/captions to the video.
Google’s (YouTube’s) auto-captioning (for the most part) worked and it’s most likely “just good enough” to enable the PETOUS’s team to avoid an all-out A.D.A. violation (which is probably the least of their upcoming legal worries). This is a forgotten small detail in an ever-growing list of forgotten small and large details. I’m also surprised that no progressive web site bothered to put up a transcription for those that need it.
Since “the press” did a terrible job holding the outgoing POTUS accountable during his two terms and also woefully botched their coverage of the 2016 election (NOTE: I didn’t vote for either major party candidate, but did write-in folks for POTUS & veep), I guess it’s up to all of us to help document history.
Here’s a moderately cleaned up version from the auto-generated Google SRT stream, presented without any commentary:
A Message from President-Elect Donald J. Trump
Today I would like to provide the American people with an update on the White House transition and our policy plans for the first 100 days.
Our transition team is working very smoothly, efficiently and effectively. Truly great and talented men and women – patriots — indeed are being brought in and many will soon be a part of our government helping us to make America great again.
My agenda will be based on a simple core principle: putting America first whether it’s producing steel, building cars or curing disease. I want the next generation of production and innovation to happen right here in our great homeland America creating wealth and jobs for American workers.
As part of this plan I’ve asked my transition team to develop a list of executive actions we can take on day one to restore our laws and bring back our jobs (about time) these include the following on trade: I’m going to issue our notification of intent to withdraw from the trans-pacific partnership — a potential disaster for our country. Instead we will negotiate fair bilateral trade deals that bring jobs and industry back onto American shores.
On energy: I will cancel job-killing restrictions on the production of American energy including shale energy and clean coal creating many millions of high-paying jobs. That’s what we want, that’s what we’ve been waiting for.
On regulation I will formulate a role which says that for every one new regulation two old regulations must be eliminated (so important).
For national security I will ask the Department of Defense and the chairman of the Joint Chiefs of Staff to develop a comprehensive plan to protect America’s vital infrastructure from cyberattacks and all other form of attacks.
On immigration: I will direct the Department of Labor to investigate all abuses of visa programs that undercut the American worker.
On ethics reform: as part of our plan to “drain the swamp” we will impose a five-year ban executive officials becoming lobbyists after they leave the administration and a lifetime ban on executive officials lobbying on behalf of a foreign government.
These are just a few of the steps we will take to reform Washington and rebuild our middle class I will provide more updates in the coming days as we work together to make America great again for everyone and I mean everyone.
For those technically inclined, you can grab that feed using the following R code. You’ll first need to ensure Developer Tools is open in your browser and tick the “CC” button in the video before starting the video. Look for a network request that begins with `https://www.youtube.com/api/timedtext` and use the “Copy as cURL” feature (all three major, cross-platform browsers support it), then run the code immediately after it (the `straighten()` function will take the data from the clipboard).
In this emergent “post-truth” world, we’re all going to need to be citizen data journalists and it’s going to be even more important than ever to ensure any data-driven pieces we write are honest, well-sourced, open to review & civil commentary, plus be fully reproducible. We all will also need to hold those in power — including the media — accountable. Despite whatever leanings you have, cherry-picking data, slicing a bit off the edges of the truth and injecting a bias because of your passionate views that you believe are right are only going to exacerbate the existing problems.