<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/allstatisticians-in-decline-a-polite-look-at-allstat-email-archives/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://masalmon.eu/2018/07/31/alldatascience/"> Posts on Maëlle's R blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I was until recently subscribed to an email list, <a rel="nofollow" href="https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=ALLSTAT" target="_blank">ALLSTAT</a>, “A UK-based worldwide e-mail broadcast system for the statistical community, operated by ICSE for HEA Statistics.” created in 1998. That’s how I saw <a rel="nofollow" href="https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=ind1505&L=ALLSTAT&P=R59128&1=ALLSTAT&9=A&J=on&K=2&d=No+Match%3BMatch%3BMatches&z=4" target="_blank">the ad for my previous job</a> in Barcelona! Now, I dislike emails more and more so I unsubscribed, but I’d still check out the archives any time I need a job, since many messages are related to openings. Nowadays, I probably <a rel="nofollow" href="http://masalmon.eu/bio/" target="_blank">identify more as a research software engineer</a> or data scientist than a statistician… which made me wonder, when did ALLSTAT start featuring data scientist jobs? How do their frequency compare to those of statisticians?</p>
<p>In this post, I’ll webscrape and analyse meta-data of ALLSTAT emails. It’ll also be the occasion for me to take the wonderful new <a rel="nofollow" href="https://github.com/dmi3kno/polite" target="_blank"><code>polite</code> package</a> for a ride, that helps respectful webscraping!</p>
<p><span></span></p>
<div>
<h1>Webscraping ALLSTAT</h1>
<div>
<h2>Life on the edge of easy responsible webscraping</h2>
<p>As underlined in <a rel="nofollow" href="https://towardsdatascience.com/ethics-in-web-scraping-b96b18136f01" target="_blank">The Ethical Scraper’s principles by James Densmore</a> (that I found in the <a rel="nofollow" href="https://hanjostudy.github.io/Presentations/UseR2018/Rvest/rvest.html#1" target="_blank">slides of Hanjo Odendaal’s useR! 2018 webscraping tutorial</a>), a first step is to research whether you can get the data without webscraping (I found no API). Note that an ideal case would be to have received emails since the list’s creation in 1998, and having stored them, so that one could use this local copy. Or one could email the list maintainers of course!</p>
<p>Then, there is some good practice to keep in mind whilst webscraping, also underlined in the principles list, and most importantly all encompassed in the <a rel="nofollow" href="https://github.com/dmi3kno/polite" target="_blank">awesome new package <code>polite</code> by Dmytro Perepolkin</a>! I knew the packages wrapped by Dmytro, but tended to only use <code>robotstxt</code> e.g. <a rel="nofollow" href="https://masalmon.eu/2018/06/18/mathtree/" target="_blank">here</a> when webscraping, (but <a rel="nofollow" href="https://itsalocke.com/blog/some-web-api-package-development-lessons-from-hibpwned/" target="_blank">using others in API packages</a>). Now with very little effort, with <code>polite</code> you will</p>
<ul>
<li>
<p>Seek permissions via <a rel="nofollow" href="https://github.com/ropensci/robotstxt" target="_blank"><code>robotstxt</code></a>,</p>
</li>
<li>
<p>Introduce yourself with an <a rel="nofollow" href="https://en.wikipedia.org/wiki/User_agent" target="_blank">user agent</a>, by default the package’s name and URL,</p>
</li>
<li>
<p><em>Take it slowly</em> thanks to <a rel="nofollow" href="https://github.com/tarakc02/ratelimitr" target="_blank"><code>ratelimitr</code></a>,</p>
</li>
<li>
<p><em>Never ask twice</em> thanks to <a rel="nofollow" href="https://github.com/r-lib/memoise" target="_blank"><code>memoise</code></a>.</p>
</li>
</ul>
<p>The <code>polite</code> package is brand-new, still in development, which in general means you might want to stay away from it for a while, but I was eager to try it and pleased by its working very well! Being an early adopter also means I saw <a rel="nofollow" href="https://github.com/dmi3kno/polite/issues?utf8=%E2%9C%93&q=is%3Aissue+author%3Amaelle+" target="_blank">my issues</a> promptly closed with some solution/new code by Dmytro!</p>
<p>I started work by <em>bowing</em> which in <code>polite</code>’s workflow means both creating a session (with user-agent, delay between calls) and checking that the path is allowed.</p>
<pre><code>home_url &lt;- &quot;https://www.jiscmail.ac.uk/cgi-bin/webadmin&quot;

session &lt;- polite::bow(home_url,
                       user_agent = &quot;Maëlle Salmon https://masalmon.eu/&quot;)</code></pre>
<p>And then it was time to scrape and parse…</p>
</div>
<div>
<h2>Actual webscraping a.k.a solving XPath puzzles</h2>
<p>When confronted with a page I’d like to extract info from, I try to identify how I can write <a rel="nofollow" href="https://www.w3schools.com/xml/xpath_intro.asp" target="_blank">XPath</a> to get the elements I want, which means looking at the page source. I used to transform whole pages to text before using regexp on them, which was clearly suboptimal, thanks to <a rel="nofollow" href="https://twitter.com/expersso/" target="_blank">Eric Persson</a> for <a rel="nofollow" href="https://masalmon.eu/2017/04/23/radioswissclassic/" target="_blank">making me switch to XPath</a>.</p>
<p>My strategy here was to get the subject, date, sender and size of each email from archive pages. <a rel="nofollow" href="https://www.jiscmail.ac.uk/cgi-bin/webadmin?A1=ind0703&L=ALLSTAT" target="_blank">Here</a> is such an archive page. Nowadays there is one archive page by month, but there used to be one by year, so I got the list of archive pages from <a rel="nofollow" href="https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=ALLSTAT" target="_blank">this general page</a>.</p>
<pre><code>library(&quot;magrittr&quot;)

polite::scrape(session, params = &quot;?A0=ALLSTAT&quot;) %>%
  rvest::xml_nodes(&quot;li&quot;) %>%
  rvest::xml_nodes(&quot;a&quot;) %>%
  rvest::html_attr(&quot;href&quot;) %>%
  purrr::keep(function(x) stringr::str_detect(x, &quot;\\/cgi-bin\\/webadmin\\?A1\\=&quot;)) %>%
  stringr::str_remove(&quot;\\/cgi\\-bin\\/webadmin\\?A1\\=ind&quot;) %>%
  stringr::str_remove(&quot;\\&amp;L\\=ALLSTAT&quot;) -> date_strings</code></pre>
<p>This is not very elegant but this got me only the “1807” and such I needed for the rest of the scraping. <code>polite::scrape</code> is a wrapper to both <code>httr::GET</code> and <code>httr::content</code> and does the rate limiting (by default 1 call every 5 seconds, <code>delay</code> parameter of <code>polite::bow</code>) and memoising. I actually ended up only scraping emails metadata from 2007 because it took ages to parse the 2006 page. It made me, according to a search via the website, miss these data scientists job openings: <a rel="nofollow" href="https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=ind00&L=ALLSTAT&P=R80005&1=ALLSTAT&9=A&I=-3&J=on&K=8&d=No+Match%3BMatch%3BMatches&z=4" target="_blank">one in 2000</a> and <a rel="nofollow" href="https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=ind04&L=ALLSTAT&P=R83293&1=ALLSTAT&9=A&I=-3&J=on&K=8&d=No+Match%3BMatch%3BMatches&z=4" target="_blank">one in 2004</a>.</p>
<pre><code>date_strings &lt;- date_strings[stringr::str_length(date_strings) != 2]
# or
date_strings &lt;- purrr::discard(date_strings, function(x) stringr::str_length(x) == 2)</code></pre>
<p>I created a function getting the metadata out of each archive page. The trickiest points here were:</p>
<ul>
<li>
<p>That the rows of the archive table could have two classes, which is the way alternate coloring was obtained. I therefore used <code>|</code> in XPath <code>'//tr[@class=&quot;normalgroup&quot;]|//tr[@class=&quot;emphasizedgroup&quot;]'</code>.</p>
</li>
<li>
<p>That there was no different class/formatting for subject, date, sender, so I got all of them at once, and then used the modulo operator, <code>%%</code>, to assign them to the right vector.</p>
</li>
</ul>
<pre><code>get_emails_meta_by_date &lt;- function(date_string, session){
  message(date_string)
  params &lt;- glue::glue(&quot;?A1=ind{date_string}&amp;L=ALLSTAT&amp;F=&amp;S=&amp;O=T&amp;H=0&amp;D=0&amp;T=0&quot;)
  
  everything &lt;- try(polite::scrape(session, params = params),
                    silent = TRUE)
  
  # at the time of writing one couldn't pass encoding to scrape
  # but now one can https://github.com/dmi3kno/polite/issues/6#issuecomment-409268730
  if(is(everything, &quot;try-error&quot;)){
    everything &lt;- httr::GET(paste0(home_url,
                                   params)) %>%
      httr::content(encoding = &quot;latin1&quot;)
  }
  
  everything &lt;- everything %>%
    # there are two classes that correspond
    # to the table having two colours of rows!
    rvest::xml_nodes(XPath = '//tr[@class=&quot;normalgroup&quot;]|//tr[@class=&quot;emphasizedgroup&quot;]') %>%
    rvest::xml_nodes(&quot;span&quot;)
  
  everything %>%
    rvest::xml_nodes(XPath = &quot;//td&quot;) %>%
    rvest::xml_nodes(&quot;span&quot;) %>%
    rvest::xml_nodes(&quot;a&quot;) %>%
    rvest::html_text() -> subjects
  
  everything %>%
    rvest::xml_nodes(XPath = &quot;//td[@nowrap]&quot;) %>%
    rvest::xml_nodes(XPath = &quot;p[@class='archive']&quot;) %>%
    rvest::html_text() -> big_mess
  
  senders &lt;- big_mess[seq_along(big_mess) %% 3 == 1]
  senders &lt;- stringr::str_remove(senders, &quot; \\&lt;\\[log in to unmask\\]\\>&quot;)
  
  dates &lt;- big_mess[seq_along(big_mess) %% 3 == 2]
  dates &lt;- lubridate::dmy_hms(dates, tz = &quot;UTC&quot;)
  
  sizes &lt;- big_mess[seq_along(big_mess) %% 3 == 0]
  sizes &lt;- stringr::str_remove(sizes, &quot; lines&quot;)
  sizes &lt;- as.numeric(sizes)
  
  tibble::tibble(subject = subjects,
                 sender = senders,
                 date = dates,
                 size = sizes) %>%
    readr::write_csv(glue::glue(&quot;data/emails_meta{date_string}.csv&quot;))
  
}</code></pre>
<p>I chose to save the metadata of each archive page in its own csv in order to make my workflow less breakable. I could have used <code>purrr::map_df</code> but then it’d be harder to re-start, and it was hard on memory apparently.</p>
<pre><code>fs::dir_create(&quot;data&quot;)
purrr::walk(date_strings,
              get_emails_meta_by_date,
              session = session)</code></pre>
</div>
</div>
<div>
<h1>Analyzing ALLSTAT jobs</h1>
<div>
<h2>Filtering jobs</h2>
<p>ALLSTAT encourages you to use keywords in emails’ subjects, so many job openings contain some variant of “job”, and that’s the sample on which I shall work.</p>
<pre><code>library(&quot;magrittr&quot;)
library(&quot;magrittr&quot;)

fs::dir_ls(&quot;../../static/data/allstat&quot;) %>%
  purrr::map_df(readr::read_csv) -> emails

jobs &lt;- dplyr::filter(emails, 
                      stringr::str_detect(subject,
                                          &quot;[Jj][Oo][Bb]&quot;))</code></pre>
<p>Out of 32761 emails I got 9245 job openings.</p>
<p>I created two dummy variables to indicate the presence of data scientist or statistician in the description. With the definition below, the “statistician” category might contain “biostatitisticians” which is fine by me.</p>
<pre><code>jobs &lt;- dplyr::mutate(jobs,
                      data_scientist = stringr::str_detect(subject, 
                                                           &quot;[Dd]ata [Ss]cientist&quot;),
                      statistician = stringr::str_detect(subject, 
                                                          &quot;[Ss]tatistician&quot;))</code></pre>
<p>176 subjects contain the word “data scientist”, 2546 the word “statistician.”, 20 both.</p>
<pre><code>dplyr::filter(jobs, data_scientist, statistician) %>%
  dplyr::select(subject, sender, date) %>%
  knitr::kable()</code></pre>
<table>
<thead>
<tr>
<th align="left">subject</th>
<th align="left">sender</th>
<th align="left">date</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">JOB: Statisticians/Data Scientists with Unilever</td>
<td align="left">Murray, Peter</td>
<td align="left">2008-02-12 08:47:56</td>
</tr>
<tr>
<td align="left">Jobs: Senior Data Scientists/Statisticians</td>
<td align="left">Cox, Trevor</td>
<td align="left">2008-03-12 16:24:48</td>
</tr>
<tr>
<td align="left">20 new job ads for data scientists, statisticians</td>
<td align="left">Vincent Granville</td>
<td align="left">2013-06-16 00:18:27</td>
</tr>
<tr>
<td align="left">23 new jobs for statisticians, data scientists</td>
<td align="left">Vincent Granville</td>
<td align="left">2013-07-27 20:05:48</td>
</tr>
<tr>
<td align="left">19 new job ads for statisticians and data scientists</td>
<td align="left">Vincent Granville</td>
<td align="left">2013-10-12 23:21:23</td>
</tr>
<tr>
<td align="left">Job Data scientist / Statistician</td>
<td align="left">Andrea Schirru</td>
<td align="left">2014-07-22 16:38:59</td>
</tr>
<tr>
<td align="left">JOB: Computational Statistician / Data Scientist</td>
<td align="left">David Hastie</td>
<td align="left">2014-07-16 11:36:34</td>
</tr>
<tr>
<td align="left">Job Openings: Statisticians and Data Scientists at Open Analytics (Belgium)</td>
<td align="left">Tobias Verbeke</td>
<td align="left">2015-06-30 09:53:21</td>
</tr>
<tr>
<td align="left">JOBS x 2: Data Scientist/Medical Statistician at University of Manchester</td>
<td align="left">Matthew Sperrin</td>
<td align="left">2016-07-28 10:42:52</td>
</tr>
<tr>
<td align="left">JOB: Data scientist – Statistician (KTP associate) @ University of Essex</td>
<td align="left">Aris Perperoglou</td>
<td align="left">2016-09-12 10:33:32</td>
</tr>
<tr>
<td align="left">JOB: Principal Statistician/ Data Scientist- Pharma – Perm – Centralised Monitoring- Data Safety – Global Search-UK</td>
<td align="left">Sabrina Andresen</td>
<td align="left">2016-10-17 16:05:12</td>
</tr>
<tr>
<td align="left">JOB: Research Associate – Statistician / Data Scientist, University of Manchester (Centre for Musculoskeletal Research)</td>
<td align="left">Jamie Sergeant</td>
<td align="left">2016-11-01 10:21:07</td>
</tr>
<tr>
<td align="left">JOB – Statistician / Data Scientist at Cefas, UK</td>
<td align="left">David Maxwell</td>
<td align="left">2017-08-04 16:15:25</td>
</tr>
<tr>
<td align="left">Job: Data Scientist/Statistician at Essex University</td>
<td align="left">Aris Perperoglou</td>
<td align="left">2017-08-21 13:43:43</td>
</tr>
<tr>
<td align="left">Job: Statistician/Data Scientist</td>
<td align="left">Roisin McCarthy</td>
<td align="left">2017-11-06 16:49:18</td>
</tr>
<tr>
<td align="left">Job Openings: Statisticians and Data Scientists at Open Analytics (Belgium)</td>
<td align="left">Tobias Verbeke</td>
<td align="left">2017-12-05 20:48:39</td>
</tr>
<tr>
<td align="left">JOB | StatsJobs – Data Scientist/Statistician, Zurich Insurance</td>
<td align="left">James Phillips</td>
<td align="left">2018-01-30 08:59:49</td>
</tr>
<tr>
<td align="left">JOB | Junior Medical statistician / Real-world data scientist – Centre of Excellence for Retrospective Studies, IQVIA (London)</td>
<td align="left">Venerus, Alessandra</td>
<td align="left">2018-03-05 10:33:09</td>
</tr>
<tr>
<td align="left">JOB | Senior Medical statistician / Real-world data scientist – Centre of Excellence for Retrospective Studies, IQVIA (London)</td>
<td align="left">Venerus, Alessandra</td>
<td align="left">2018-03-05 10:32:51</td>
</tr>
<tr>
<td align="left">Job: Catalyst Project Research Data Scientist / Statistician</td>
<td align="left">Aris Perperoglou</td>
<td align="left">2018-06-25 14:17:18</td>
</tr>
</tbody>
</table>
<p>Are the job titles synonymous for the organizations using slashes? I am especially puzzled by “Senior Medical statistician / Real-world data scientist”! I filtered them out and created a <code>category</code> variable.</p>
<pre><code>jobs &lt;- dplyr::filter(jobs,
                      !(data_scientist&amp;statistician))
jobs &lt;- dplyr::mutate(jobs,
                      category = dplyr::case_when(data_scientist ~ &quot;data scientist&quot;,
                                                  statistician ~ &quot;statistician&quot;,
                                                  TRUE ~ &quot;other&quot;),
                      category = factor(category,
                                        levels = c(&quot;statistician&quot;,
                                                   &quot;data scientist&quot;,
                                                   &quot;other&quot;),
                                        ordered = TRUE))

jobs &lt;- dplyr::mutate(jobs,
                      year = lubridate::year(date))</code></pre>
<p>Here are some examples of positions for each:</p>
<pre><code>head(jobs$subject[jobs$category==&quot;statistician&quot;])</code></pre>
<pre><code>## [1] &quot;FW: JOB: Senior/Lead Statistician - GlaxoSmithKline&quot;                   
## [2] &quot;JOB - Clinical Trial Statisticians and SAS Programmers&quot;                
## [3] &quot;JOB - Medical statistician in Salvador, Brazil (2-3 years)&quot;            
## [4] &quot;JOB - Medical Statistician, Oxford&quot;                                    
## [5] &quot;JOB OPPORTUNITY - 39408 - Entry-Level Statisticians - GSK&quot;             
## [6] &quot;JOB: Biostatistician - Contract/Consultancy Opportunity - Switzer land&quot;</code></pre>
<pre><code>head(jobs$subject[jobs$category==&quot;data scientist&quot;])</code></pre>
<pre><code>## [1] &quot;JOB: Data Scientist, Unilever&quot;                                 
## [2] &quot;JOB: Data Scientist, Unilever&quot;                                 
## [3] &quot;JOB: Data Scientist / Machine Learning (BELFAST)&quot;              
## [4] &quot;JOB: Data Scientists in Belfast&quot;                               
## [5] &quot;JOBS: Data Scientists&quot;                                         
## [6] &quot;[Job] Hacker or Botnet Developer, Data Scientist (Telecommute)&quot;</code></pre>
<pre><code>head(jobs$subject[jobs$category==&quot;other&quot;])</code></pre>
<pre><code>## [1] &quot;Job - Data manager Dept of Haematology, Imperial College, London&quot;                  
## [2] &quot;Re: JOB - Health Economic Modelling and Value Demonstrations&quot;                      
## [3] &quot;JOB - Lectureship / readership&quot;                                                    
## [4] &quot;JOB - Mathematical Modeller at the Health Protection Agency, Centre for Infections&quot;
## [5] &quot;JOB - PhD position&quot;                                                                
## [6] &quot;JOB - SAS Senior Statistical Programmer, Oxford&quot;</code></pre>
</div>
<div>
<h2>Are data scientists on the rise?</h2>
<pre><code>library(&quot;ggplot2&quot;)

ggplot(jobs) +
  geom_bar(aes(year, fill = category)) +
  viridis::scale_fill_viridis(discrete = TRUE) +
  theme(legend.position = &quot;bottom&quot;) +
  hrbrthemes::theme_ipsum(base_size = 14) +
  xlab(&quot;Year (2018 not complete yet)&quot;) +
  ylab(&quot;Number of job openings&quot;) +
  ggtitle(&quot;ALLSTAT mailing list 2007-2018&quot;)</code></pre>
<p><img src="https://i2.wp.com/masalmon.eu/post/2018-07-31-alldatascience_files/figure-html/unnamed-chunk-6-1.png?w=450&ssl=1" data-recalc-dims="1"></p>
<p>I like this bar plot that shows how the total number of job openings fluctuates, but it’s hard to see differences in proportions.</p>
<pre><code>ggplot(jobs) +
  geom_bar(aes(year, fill = category),
           position = &quot;fill&quot;) +
  viridis::scale_fill_viridis(discrete = TRUE) +
  theme(legend.position = &quot;bottom&quot;) +
  hrbrthemes::theme_ipsum(base_size = 14) +
  xlab(&quot;Year (2018 not complete yet)&quot;) +
  ylab(&quot;Number of job openings&quot;) +
  ggtitle(&quot;ALLSTAT mailing list 2007-2018&quot;)</code></pre>
<p><img src="https://i0.wp.com/masalmon.eu/post/2018-07-31-alldatascience_files/figure-html/unnamed-chunk-7-1.png?w=450&ssl=1" data-recalc-dims="1"></p>
<p>According to this plot, although there seems to be more and more data scientists’ jobs advertised on ALLSTAT… Statisticians don’t need to get worried just yet.</p>
</div>
<div>
<h2>Who offers data scientists’ jobs?</h2>
<pre><code>dplyr::count(jobs, category,
                sender) %>%
  dplyr::group_by(category) %>%
  dplyr::arrange(category, - n) %>%
  dplyr::filter(sender %in% sender[1:5])</code></pre>
<pre><code>## # A tibble: 15 x 3
## # Groups:   category [3]
##    category       sender                     n
##    </code></pre></div></div>