I was enthused to see a mention of this on the GDELT blog since I’ve been working on an R package dubbed newsflash
to work with the API that the form front-ends.
Given the current climate, I feel compelled to note that I’m neither a Clinton supporter/defender/advocate nor a ? supporter/defender/advocate) in any way, shape or form. I’m only using the example for replication and I’m very glad the article author stayed (pretty much) non-partisan apart from some color commentary about the predictability of network coverage of certain topics.
For now, the newsflash
package is configured to grab raw count data, not the percent summaries since folks using R to grab this data probably want to do their own work with it. I used the following to try to replicate the author’s findings:
library(newsflash)
library(ggalt) # github version
library(hrbrmisc) # github only
library(tidyverse)
starts <- seq(as.Date("2015-01-01"), (as.Date("2017-01-26")-30), "30 days")
ends <- as.character(starts + 29)
ends[length(ends)] <- ""
pb <- progress_estimated(length(starts))
emails <- map2(starts, ends, function(x, y) {
pb$tick()$print()
query_tv("clinton", "email,emails,server", timespan="custom", start_date=x, end_date=y)
})
clinton_timeline <- map_df(emails, "timeline")
sum(clinton_timeline$value)
## [1] 34778
count(clinton_timeline, station, wt=value, sort=TRUE) %>%
mutate(pct=n/sum(n), pct_lab=sprintf("%s (%s)", scales::comma(n), scales::percent(pct)),
station=factor(station, levels=rev(station))) -> timeline_df
timeline_df
## # A tibble: 7 × 4
## station n pct pct_lab
## <fctr> <int> <dbl> <chr>
## 1 FOX News 14807 0.425757663 14,807 (42.6%)
## 2 FOX Business 7607 0.218730232 7,607 (21.9%)
## 3 CNN 5434 0.156248203 5,434 (15.6%)
## 4 MSNBC 4413 0.126890563 4,413 (12.7%)
## 5 Aljazeera America 1234 0.035482201 1,234 (3.5%)
## 6 Bloomberg 980 0.028178734 980 (2.8%)
## 7 CNBC 303 0.008712404 303 (0.9%)
NOTE: I had to break up the queries since the bulk one across the two dates bump up against the API limits and may be providing helper functions for that before CRAN release.
While my package matches the total from the news article and sample query: 34,778 results my percentages are different since it’s the percentages across the raw counts for the included stations. “Percent of Sentences” (result “n” divided by the number of all sentences for each station in the time frame) — which the author used — seems to have some utility so I’ll probably add that as a query parameter or add a new function.
Tidy news text
The package also is designed to work with the tidytext
package (it’s on CRAN) and provides a top_text()
function which can return a tidytext
-ready tibble or a plain character vector for use in other text processing packages. If you were curious as to whether this API has good data behind it, we can take a naive peek with the help of tidytext
:
library(tidytext)
tops <- map_df(emails, top_text)
anti_join(tops, stop_words) %>%
filter(!(word %in% c("clinton", "hillary", "server", "emails", "mail", "email",
"mails", "secretary", "clinton's", "secretary"))) %>%
count(word, sort=TRUE) %>%
print(n=20)
## # A tibble: 26,861 × 2
## word n
## <chr> <int>
## 1 private 12683
## 2 department 9262
## 3 fbi 7250
## 4 campaign 6790
## 5 classified 6337
## 6 trump 6228
## 7 information 6147
## 8 investigation 5111
## 9 people 5029
## 10 time 4739
## 11 personal 4514
## 12 president 4448
## 13 donald 4011
## 14 foundation 3972
## 15 news 3918
## 16 questions 3043
## 17 top 2862
## 18 government 2799
## 19 bill 2698
## 20 reporter 2684
I’d say the API is doing just fine.
Fin
The package also has some other bits from the API in it and if this has piqued your interest, please leave all package feature requests or problems as a github issue.
Many thanks to the Internet Archive / GDELT for making this API possible. Data like this would be amazing in any time, but is almost invaluable now.
The ? Resistance
I need to be up-front about something: I’m somewhat partially at fault for ? being elected. While I did not vote for him, I could not in any good conscience vote for his Democratic rival. I wrote in a ticket that had one Democrat and one Republican on it. The “who” doesn’t matter and my district in Maine went abundantly for ?’s opponent, so there was no real impact of my direct choice but I did actively point out the massive flaws in his opponent. Said flaws were many and I believe we’d be in a different bad place, but not equally as bad of a place now with her. But, that’s in the past and we’ve got a new reality to deal with, now.
This is a (hopefully) brief post about finding a way out of this mess we’re in. It’s far from comprehensive, but there’s honest-to-goodness evil afoot that needs to be met head on.
Brand Damage
You’ll note I’m not using either of their names. Branding is extremely important to both of them, but is the almost singular focus of ?. His name is his hotel brand, company brand and global identifier. Using it continues to add it to the history books and can only help inflate the power of that brand. First and foremost, do not use his name in public posts, articles, papers, etc. “POTUS”, “The President”, “The Commander in Chief”, “?” (chosen to match his skin/hair color, complexion and that comb-over tuft) are all sufficient references since there is date-context with virtually anything we post these days. Don’t help build up his brand. Don’t populate historical repositories with his name. Don’t give him what he wants most of all: attention.
Document and Defend with Data
Speaking of the historical record, we need to be blogging and publishing regularly the actual facts based on data. We also need to save data as there’s signs of a deliberate government purge going on. I’m not sure how successful said purge will be in the long run and I suspect that the long-term effects of data purging and corruption by this administration will have lasting unintended consequences.
Join/support @datarefuge to save data & preserve the historical record.
Install the Wayback Machine plugin and take the 2 seconds per site you visit to click it.
Create blog posts, tweets, news articles and papers that counter bad facts with good/accurate/honest ones. Don’t make stuff up (even a little). Validate your posits before publishing. Write said posts in a respectful tone.
Support the Media
When the POTUS’ Chief Strategist says things like “The media should be embarrassed and humiliated and keep its mouth shut and just listen for a while” it’s a deliberate attempt to curtail the Press and eventually there will be more actions to actually suppress Press freedom.
I’m not a liberal (I probably have no convenient definition) and I think the Press gave Obama a free ride during his eight year rule. They are definitely making up for that now, mostly because their very livelihoods are at stake.
The problem with them is that they are continuing to let themselves be manipulated by ?. He’s a master at this manipulation. Creating a story about the size of his hands in a picture delegitimizes you as a purveyor of news, especially when — as you’re watching his hands — he’s separating families, normalizing bigotry and undermining the Constitution. Forget about the hands and even forget about the hotels (for now). There was even a recent story trying to compare email servers (the comparison is very flawed). Stop it.
Encourage reporters to focus on things that actually matter and provide pointers to verifiable data they can use to call out the lack of veracity in ?’s policies. Personal blog posts are fleeting things but an NYT, WSJ (etc) story will live on.
Be Kind
I’ve heard and read some terrible language about rural America from what I can only classify as “liberals” in the week this post was written. Intellectual hubris and actual, visceral disdain for those who don’t think a certain way were two major reasons why ? got elected. The actual reasons he got elected are diverse and very nuanced.
Regardless of political leaning, pick your head up from your glowing rectangles and go out of your way to regularly talk to someone who doesn’t look, dress, think, eat, etc like you. Engage everyone with compassion. Regularly challenge your own beliefs.
There is a wedge that I estimate is about 1/8th of the way into the core of America now. Perpetuating this ideological “us vs them” mindset is only going to fuel the fires that created the conditions we’re in now and drive the wedge in further. The only way out is through compassion.
Remember: all life matters. Your degree, profession, bank balance or faith alignment doesn’t give you the right to believe you are better than anyone else.
FIN (for now)
I’ll probably move most of future opines to a new medium (not uppercase Medium) as you may be getting this drivel when you want recipes or R code (even though there are separate feeds for them).