<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/building-the-olympics-blog-tidy-data-preparation/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://edwinth.github.io/olympics-dataprep/"> That’s so Random</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Last week I did an analysis of <a rel="nofollow" href="https://edwinth.github.io/analyzing-olympics/" target="_blank">gold diggers at the Olympics</a>. Here I take you along how I scraped the data from wikepedia and subsequently used the tools of the tidyverse to get the data in a format in which they can be analyzed. You are invited to copy-paste the code and see how the data is gradually getting into a shape in which we can analyze it. I will often not show the data in the intermediate steps, because this will clog the blog. Copy the code to find out for yourself.</p>
<h2>Scraping the data</h2>
<p>Web scraping in R can be done using the <code>rvest</code> package. There are several introductions to this package, such as <a rel="nofollow" href="https://stat4701.github.io/edav/2015/04/02/rvest_tutorial/" target="_blank">this one</a>. You can use either the Inspector Gadget to find out the html tags of the data you want to scrape, or inspect elements in the web browser. I prefer the latter. These data are published on <a rel="nofollow" href="https://en.wikipedia.org/wiki/List_of_2018_Winter_Olympics_medal_winners" target="_blank">Wikipedia</a>, for each sport you will find the results in a separate table. It appears that the names of the sports all have <code>h2</code> tags, the data of the medal winners all have a <code>table</code> class. First we obtain the names of the sports</p>
<div>
<div>
<pre><code><span>library</span><span>(</span><span>rvest</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>library</span><span>(</span><span>stringr</span><span>)</span><span>
</span><span>Olympics_2018_wiki</span><span> </span><span>&lt;-</span><span> </span><span>read_html</span><span>(</span><span>&quot;https://en.wikipedia.org/wiki/List_of_2018_Winter_Olympics_medal_winners&quot;</span><span>)</span><span>
</span><span>sports</span><span> </span><span>&lt;-</span><span> </span><span>html_nodes</span><span>(</span><span>Olympics_2018_wiki</span><span>,</span><span> </span><span>&quot;h2&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>html_text</span><span>()</span><span>
</span><span>sports</span><span> </span><span>%>%</span><span> </span><span>head</span><span>(</span><span>3</span><span>)</span><span>
</span></code></pre>
</div>
</div>
<div>
<div>
<pre><code>## [1] &quot;Alpine skiing[edit]&quot; &quot;Biathlon[edit]&quot;      &quot;Bobsleigh[edit]&quot;
</code></pre>
</div>
</div>
<p>Note that you have to use <code>html_text</code> to convert the xml nodes to regular R characters. I am not showing the full output here, but it appears that the first fifteen elements contain the sport’s names. Also, we still need to get rid of the “[edit]” part of the strings.</p>
<div>
<div>
<pre><code><span>sports</span><span> </span><span>&lt;-</span><span> </span><span>sports</span><span> </span><span>%>%</span><span> </span><span>`[`</span><span>(</span><span>1</span><span>:</span><span>15</span><span>)</span><span> </span><span>%>%</span><span> </span><span>str_split</span><span>(</span><span>&quot;\\[&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>map_chr</span><span>(</span><span>1</span><span>)</span><span>
</span></code></pre>
</div>
</div>
<p>In the above I call the subsetting operator <code>[</code> as a prefix function, rather than its usual usage <code>object[index]</code>. (Remember everything in R is a function, even when it appears not be). This to make it pipeable. Every infix operator can be used in this way. If you find this unaesthetic, <code>magrittr</code> has alternative functions, such as <code>extract</code> to do subsetting. I personally find this a nice use of infixes. Next, we need to get rid of the “[edit]” after each name. We use stringr’s <code>str_split</code> to split on each “[”. This results in a list of character vectors, each vector of length 2. <code>purrr::map_chr</code> will select the first element of each vector and store the result in a single character vector.</p>
<p>Now, the tables with the sports</p>
<div>
<div>
<pre><code><span>medals</span><span> </span><span>&lt;-</span><span> </span><span>html_nodes</span><span>(</span><span>Olympics_2018_wiki</span><span>,</span><span> </span><span>&quot;table&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>html_table</span><span>()</span><span> </span><span>%>%</span><span> </span><span>`[`</span><span>(</span><span>3</span><span>:</span><span>26</span><span>)</span><span>
</span></code></pre>
</div>
</div>
<p>I checked manually that the 3rd up until the 26th table contain the results. Now, if you visit the wiki page you will notice that some sports have a single results table (such as Curling), while others have several (for women’s, for men’s, and some even for mixed events). I counted on the website the number of tables for each sport.</p>
<div>
<div>
<pre><code><span>tables_by_sport</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>3</span><span>,</span><span> </span><span>3</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>2</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>2</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>2</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>2</span><span>,</span><span> </span><span>2</span><span>)</span><span>
</span></code></pre>
</div>
</div>
<p>The next step is flatten these little data frames to one, and add the sport of each event as a column.</p>
<div>
<div>
<pre><code><span>medals_tbl</span><span> </span><span>&lt;-</span><span> </span><span>medals</span><span> </span><span>%>%</span><span>
</span><span>map</span><span>(</span><span>~</span><span>select</span><span>(</span><span>.x</span><span>,</span><span> </span><span>Gold</span><span>,</span><span> </span><span>Silver</span><span>,</span><span> </span><span>Bronze</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>map2_df</span><span>(</span><span>rep</span><span>(</span><span>sports</span><span>,</span><span> </span><span>tables_by_sport</span><span>),</span><span>
</span><span>~</span><span>mutate</span><span>(</span><span>.x</span><span>,</span><span> </span><span>sport</span><span> </span><span>=</span><span> </span><span>.y</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>as_data_frame</span><span>()</span><span>
</span></code></pre>
</div>
</div>
<p>At the first step we apply dplyr’s select to obtain the three columns we are interested in, with <code>map</code> this is applied to each of the little data frames. Next we use <code>map2_df</code>, this will iterate over two vectors instead of one, to add the sports names to each of the data frames with <code>mutate</code>. Note that by using <code>map2_df</code>, we bind all the little data frames into one big data frame right away. The result is of class <code>data.frame</code>. Since I like to work with tibbles, I coerce it at the final line.</p>
<div>
<div>
<pre><code><span>medals_tbl</span><span> </span><span>%>%</span><span> </span><span>select</span><span>(</span><span>Gold</span><span>)</span><span> </span><span>%>%</span><span> </span><span>slice</span><span>(</span><span>c</span><span>(</span><span>4</span><span>,</span><span> </span><span>16</span><span>))</span><span>
</span></code></pre>
</div>
</div>
<div>
<div>
<pre><code>## # A tibble: 2 x 1
##                                                                          Gold
##                                                                         </code></pre></div></div>