I was until recently subscribed to an email list, ALLSTAT, “A UK-based worldwide e-mail broadcast system for the statistical community, operated by ICSE for HEA Statistics.” created in 1998. That’s how I saw the ad for my previous job in Barcelona! Now, I dislike emails more and more so I unsubscribed, but I’d still check out the archives any time I need a job, since many messages are related to openings. Nowadays, I probably identify more as a research software engineer or data scientist than a statistician… which made me wonder, when did ALLSTAT start featuring data scientist jobs? How do their frequency compare to those of statisticians?
In this post, I’ll webscrape and analyse meta-data of ALLSTAT emails. It’ll also be the occasion for me to take the wonderful new polite package for a ride, that helps respectful webscraping!
Webscraping ALLSTAT
Life on the edge of easy responsible webscraping
As underlined in The Ethical Scraper’s principles by James Densmore (that I found in the slides of Hanjo Odendaal’s useR! 2018 webscraping tutorial), a first step is to research whether you can get the data without webscraping (I found no API). Note that an ideal case would be to have received emails since the list’s creation in 1998, and having stored them, so that one could use this local copy. Or one could email the list maintainers of course!
Then, there is some good practice to keep in mind whilst webscraping, also underlined in the principles list, and most importantly all encompassed in the awesome new package polite by Dmytro Perepolkin! I knew the packages wrapped by Dmytro, but tended to only use robotstxt e.g. here when webscraping, (but using others in API packages). Now with very little effort, with polite you will
Seek permissions via robotstxt,
Introduce yourself with an user agent, by default the package’s name and URL,
Take it slowly thanks to ratelimitr,
Never ask twice thanks to memoise.
The polite package is brand-new, still in development, which in general means you might want to stay away from it for a while, but I was eager to try it and pleased by its working very well! Being an early adopter also means I saw my issues promptly closed with some solution/new code by Dmytro!
I started work by bowing which in polite’s workflow means both creating a session (with user-agent, delay between calls) and checking that the path is allowed.
home_url %
rvest::xml_nodes(&quot;a&quot;) %__%
rvest::html_attr(&quot;href&quot;) %__%
purrr::keep(function(x) stringr::str_detect(x, &quot;\\/cgi-bin\\/webadmin\\?A1\\=&quot;)) %__%
stringr::str_remove(&quot;\\/cgi\\-bin\\/webadmin\\?A1\\=ind&quot;) %__%
stringr::str_remove(&quot;\\&amp;L\\=ALLSTAT&quot;) -__ date_strings
This is not very elegant but this got me only the “1807” and such I needed for the rest of the scraping. polite::scrape is a wrapper to both httr::GET and httr::content and does the rate limiting (by default 1 call every 5 seconds, delay parameter of polite::bow) and memoising. I actually ended up only scraping emails metadata from 2007 because it took ages to parse the 2006 page. It made me, according to a search via the website, miss these data scientists job openings: one in 2000 and one in 2004.
date_strings