<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/radio-edit-an-improved-scraping-of-and-look-at-radio-swiss-classic-program/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://maelle.github.io/2017/04/30/radioedit/"> Maëlle</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Last week I published a post about <a target="_blank" rel="nofollow" href="http://www.masalmon.eu/2017/04/23/radioswissclassic/">scraping Radio Swiss Classic program</a>. After that, Bob Rudis wrote <a target="_blank" rel="nofollow" href="https://rud.is/b/2017/04/23/decomposing-composers-with-r/">an extremely useful post</a> improving my code a lot and teaching me cool stuff. I don’t know why I forgot to add pauses between requests… Really bad behaviour! I will use his code today for re-scraping the data.</p>
<p>Why re-scrape the data? I mentioned broken links in my post. In fact, each time I hit a broken page, Radio Swiss Classic webmaster received an email. That person received <em>a lot</em> of emails because of me. They repaired the bug explaining these broken pages and contacted me because someone had turned me in (I feel super famous or spied on now), very kindly mentioning they had fixed all pages, and not holding any grudge against me. So let’s scrape everything again!</p>
<p><span></span></p>
<h1>Scraping the program</h1>
<p>So this is mostly Bob Rudis’ code with my clumsy comments alongside his comments. His are always in-line. I really recommend you to read <a target="_blank" rel="nofollow" href="https://rud.is/b/2017/04/23/decomposing-composers-with-r/">his post</a>!</p>
<div>
<pre><code><span>library</span><span>(</span><span>&quot;rvest&quot;</span><span>)</span><span>
</span><span>library</span><span>(</span><span>&quot;purrr&quot;</span><span>)</span><span>
</span><span>library</span><span>(</span><span>&quot;stringi&quot;</span><span>)</span><span>
</span><span>library</span><span>(</span><span>&quot;lubridate&quot;</span><span>)</span><span>
</span><span>library</span><span>(</span><span>&quot;tidyverse&quot;</span><span>)</span><span>
</span><span># using purrr::safely is cool because if the page # is broken you get NULL as an output
</span><span>s_read_html</span><span> </span><span>&lt;-</span><span> </span><span>purrr</span><span>::</span><span>safely</span><span>(</span><span>read_html</span><span>)</span><span>
</span><span># helper for brevity
</span><span>xtract_nodes</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>node</span><span>,</span><span> </span><span>css</span><span>)</span><span> </span><span>{</span><span>
</span><span>html_nodes</span><span>(</span><span>node</span><span>,</span><span> </span><span>css</span><span>)</span><span> </span><span>%>%</span><span> </span><span>html_text</span><span>(</span><span>trim</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>}</span><span>
</span><span>get_one_day_program</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>date</span><span>=</span><span>Sys.Date</span><span>(),</span><span>
</span><span>base_url</span><span>=</span><span>&quot;http://www.radioswissclassic.ch/en/music-programme/search/%s&quot;</span><span>,</span><span>
</span><span>pb</span><span>=</span><span>NULL</span><span>)</span><span> </span><span>{</span><span>
</span><span># progress bar magic!
</span><span> </span><span>if</span><span> </span><span>(</span><span>!</span><span>is.null</span><span>(</span><span>pb</span><span>))</span><span> </span><span>pb</span><span>$</span><span>tick</span><span>()</span><span>$</span><span>print</span><span>()</span><span>
</span><span># that's the part where you're nice towards the website
</span><span> </span><span>Sys.sleep</span><span>(</span><span>sample</span><span>(</span><span>seq</span><span>(</span><span>0</span><span>,</span><span>1</span><span>,</span><span>0.25</span><span>),</span><span> </span><span>1</span><span>))</span><span> </span><span># ideally, make this sample(5,1)
</span><span>
</span><span>date</span><span> </span><span>&lt;-</span><span> </span><span>ymd</span><span>(</span><span>date</span><span>)</span><span> </span><span># handles case where input is character ISO date
</span><span>
</span><span>pg</span><span> </span><span>&lt;-</span><span> </span><span>s_read_html</span><span>(</span><span>sprintf</span><span>(</span><span>base_url</span><span>,</span><span> </span><span>format</span><span>(</span><span>date</span><span>,</span><span> </span><span>&quot;%Y%m%d&quot;</span><span>)))</span><span>
</span><span>if</span><span> </span><span>(</span><span>!</span><span>is.null</span><span>(</span><span>pg</span><span>$</span><span>result</span><span>))</span><span> </span><span>{</span><span>
</span><span># go read Bob Rudis' post, in particular to read about
# the extra selector for &quot;playlist&quot;
</span><span> </span><span>dplyr</span><span>::</span><span>data_frame</span><span>(</span><span>
</span><span>date</span><span> </span><span>=</span><span> </span><span>date</span><span>,</span><span>
</span><span>time</span><span> </span><span>=</span><span> </span><span>xtract_nodes</span><span>(</span><span>pg</span><span>$</span><span>result</span><span>,</span><span> </span><span>'div[class=&quot;playlist&quot;] *
span[class=&quot;time hidden-xs&quot;]'</span><span>)</span><span> </span><span>%>%</span><span> </span><span>hm</span><span>(),</span><span>
</span><span>datetime</span><span> </span><span>=</span><span> </span><span>update</span><span>(</span><span>date</span><span>,</span><span> </span><span>hour</span><span> </span><span>=</span><span> </span><span>hour</span><span>(</span><span>time</span><span>),</span><span> </span><span>minute</span><span> </span><span>=</span><span> </span><span>minute</span><span>(</span><span>time</span><span>)),</span><span>
</span><span>artist</span><span> </span><span>=</span><span> </span><span>xtract_nodes</span><span>(</span><span>pg</span><span>$</span><span>result</span><span>,</span><span> </span><span>'div[class=&quot;playlist&quot;] * span[class=&quot;titletag&quot;]'</span><span>),</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>xtract_nodes</span><span>(</span><span>pg</span><span>$</span><span>result</span><span>,</span><span> </span><span>'div[class=&quot;playlist&quot;] * span[class=&quot;artist&quot;]'</span><span>)</span><span>
</span><span>)</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>closeAllConnections</span><span>()</span><span>
</span><span>NULL</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span><span>search_dates</span><span> </span><span>&lt;-</span><span> </span><span>seq</span><span>(</span><span>from</span><span> </span><span>=</span><span> </span><span>ymd</span><span>(</span><span>&quot;2008-09-01&quot;</span><span>),</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>ymd</span><span>(</span><span>&quot;2017-04-22&quot;</span><span>),</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;1 day&quot;</span><span>)</span><span>
</span><span># how you can make the progress bar work,
# estimate the time necessary for 5 requests
</span><span>pb</span><span> </span><span>&lt;-</span><span> </span><span>dplyr</span><span>::</span><span>progress_estimated</span><span>(</span><span>length</span><span>(</span><span>search_dates</span><span>))</span><span>
</span><span>programs_df</span><span> </span><span>&lt;-</span><span> </span><span>map_df</span><span>(</span><span>search_dates</span><span>,</span><span> </span><span>get_one_day_program</span><span>,</span><span> </span><span>pb</span><span>=</span><span>pb</span><span>)</span><span>
</span><span>programs_df</span><span> </span><span>&lt;-</span><span> </span><span>programs_df</span><span> </span><span>%>%</span><span>
</span><span>dplyr</span><span>::</span><span>select</span><span>(</span><span>-</span><span> </span><span>time</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>dplyr</span><span>::</span><span>mutate</span><span>(</span><span>datetime</span><span> </span><span>=</span><span> </span><span>force_tz</span><span>(</span><span>datetime</span><span>,</span><span> </span><span>tz</span><span> </span><span>=</span><span> </span><span>&quot;Europe/Zurich&quot;</span><span>))</span><span>
</span><span>programs_df</span><span>
</span><span>save</span><span>(</span><span>programs_df</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;data/radioswissclassic_programs_radioedit.RData&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>load</span><span>(</span><span>&quot;data/radioswissclassic_programs_radioedit.RData&quot;</span><span>)</span><span>
</span><span>search_dates</span><span> </span><span>&lt;-</span><span> </span><span>seq</span><span>(</span><span>from</span><span> </span><span>=</span><span> </span><span>lubridate</span><span>::</span><span>ymd</span><span>(</span><span>&quot;2008-09-01&quot;</span><span>),</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>lubridate</span><span>::</span><span>ymd</span><span>(</span><span>&quot;2017-04-22&quot;</span><span>),</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;1 day&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p>This time I got 100% of programs!</p>
<p>Now I can improve my analysis of the data: My <a target="_blank" rel="nofollow" href="https://twitter.com/MilaCarles">colleague Carles</a> (who miiight have been receiving the link to each of my posts because I feel he needs to know about all of them… poor Carles) left a comment on my last post with two nice suggestions.</p>
<h1>Applying quantile regression to the duration of pieces</h1>
<p>What Carles said: “Regarding differences throughout the day, I agree that there are no major differences on the mean, but there are in the upper quantiles. I’d suggest quantile regression or quantile smoothing in case you want to try ;)”. Let’s not comment on the absence of nose of this smiley, my colleague is a nice guy so I’ll try to not freak out and to concentrate on the statistics.</p>
<div>
<pre><code><span>library</span><span>(</span><span>&quot;magrittr&quot;</span><span>)</span><span>
</span><span>programs_df</span><span> </span><span>&lt;-</span><span> </span><span>dplyr</span><span>::</span><span>arrange</span><span>(</span><span>programs_df</span><span>,</span><span> </span><span>datetime</span><span>)</span><span>
</span><span>programs_df</span><span> </span><span>&lt;-</span><span> </span><span>dplyr</span><span>::</span><span>mutate</span><span>(</span><span>programs_df</span><span>,</span><span>
</span><span>duration</span><span> </span><span>=</span><span> </span><span>difftime</span><span>(</span><span>lead</span><span>(</span><span>datetime</span><span>,</span><span> </span><span>1</span><span>),</span><span>
</span><span>datetime</span><span>,</span><span>
</span><span>units</span><span> </span><span>=</span><span> </span><span>&quot;min&quot;</span><span>))</span><span>
</span><span>programs_df</span><span> </span><span>&lt;-</span><span> </span><span>dplyr</span><span>::</span><span>mutate</span><span>(</span><span>programs_df</span><span>,</span><span>
</span><span>duration</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>duration</span><span> </span><span>></span><span> </span><span>60</span><span>,</span><span>
</span><span>NA</span><span>,</span><span> </span><span>duration</span><span>))</span><span>
</span><span>programs_df</span><span> </span><span>&lt;-</span><span> </span><span>dplyr</span><span>::</span><span>mutate</span><span>(</span><span>programs_df</span><span>,</span><span>
</span><span>hour</span><span> </span><span>=</span><span> </span><span>as.factor</span><span>(</span><span>lubridate</span><span>::</span><span>hour</span><span>(</span><span>datetime</span><span>)))</span><span>
</span><span>programs_df</span><span> </span><span>&lt;-</span><span> </span><span>dplyr</span><span>::</span><span>mutate</span><span>(</span><span>programs_df</span><span>,</span><span>
</span><span>night</span><span> </span><span>=</span><span> </span><span>(</span><span>lubridate</span><span>::</span><span>hour</span><span>(</span><span>datetime</span><span>)</span><span> </span><span>&lt;=</span><span> </span><span>4</span><span> </span><span>|</span><span> </span><span>lubridate</span><span>::</span><span>hour</span><span>(</span><span>datetime</span><span>)</span><span> </span><span>>=</span><span> </span><span>20</span><span>))</span><span>
</span><span>quantiles</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>x</span><span>){</span><span>
</span><span>probs</span><span> </span><span>&lt;-</span><span> </span><span>seq</span><span>(</span><span>from</span><span> </span><span>=</span><span> </span><span>0.5</span><span>,</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>0.95</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>0.05</span><span>)</span><span>
</span><span>quantiles_names</span><span> </span><span>&lt;-</span><span> </span><span>paste0</span><span>(</span><span>&quot;q&quot;</span><span>,</span><span> </span><span>probs</span><span>*</span><span>100</span><span>)</span><span>
</span><span>quantiles</span><span> </span><span>&lt;-</span><span> </span><span>purrr</span><span>::</span><span>map_dbl</span><span>(</span><span>probs</span><span>,</span><span> </span><span>function</span><span>(</span><span>p</span><span>,</span><span> </span><span>v</span><span> </span><span>=</span><span> </span><span>x</span><span>){</span><span>
</span><span>quantile</span><span>(</span><span>v</span><span>,</span><span> </span><span>probs</span><span> </span><span>=</span><span> </span><span>p</span><span>,</span><span> </span><span>na.rm</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>})</span><span>
</span><span>df</span><span> </span><span>&lt;-</span><span> </span><span>tibble</span><span>::</span><span>as.tibble</span><span>(</span><span>t</span><span>(</span><span>quantiles</span><span>))</span><span>
</span><span>names</span><span>(</span><span>df</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>quantiles_names</span><span>
</span><span>return</span><span>(</span><span>df</span><span>)</span><span>
</span><span>}</span><span>
</span><span>programs_df</span><span> </span><span>%>%</span><span>
</span><span>dplyr</span><span>::</span><span>group_by</span><span>(</span><span>night</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>dplyr</span><span>::</span><span>summarize</span><span>(</span><span>duration_q</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>quantiles</span><span>(</span><span>duration</span><span>)))</span><span> </span><span>%>%</span><span>
</span><span>tidyr</span><span>::</span><span>unnest</span><span>(</span><span>duration_q</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>knitr</span><span>::</span><span>kable</span><span>()</span><span>
</span></code></pre>
</div>
<table>
<thead>
<tr>
<th>night</th>
<th>q50</th>
<th>q55</th>
<th>q60</th>
<th>q65</th>
<th>q70</th>
<th>q75</th>
<th>q80</th>
<th>q85</th>
<th>q90</th>
<th>q95</th>
</tr>
</thead>
<tbody>
<tr>
<td>FALSE</td>
<td>7</td>
<td>8</td>
<td>8</td>
<td>9</td>
<td>9</td>
<td>10</td>
<td>10</td>
<td>11</td>
<td>13</td>
<td>16</td>
</tr>
<tr>
<td>TRUE</td>
<td>8</td>
<td>8</td>
<td>9</td>
<td>9</td>
<td>10</td>
<td>11</td>
<td>13</td>
<td>16</td>
<td>20</td>
<td>26</td>
</tr>
</tbody>
</table>
<p>Looking at this table above, it’s true that the upper quantiles of piece duration seem to be longet at night.</p>
<div>
<pre><code><span>library</span><span>(</span><span>&quot;ggplot2&quot;</span><span>)</span><span>
</span><span>library</span><span>(</span><span>&quot;hrbrthemes&quot;</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>programs_df</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_histogram</span><span>(</span><span>aes</span><span>(</span><span>duration</span><span>))</span><span> </span><span>+</span><span>
</span><span>facet_grid</span><span>(</span><span>night</span><span> </span><span>~</span><span> </span><span>.</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_ipsum</span><span>()</span><span> </span><span>+</span><span>
</span><span>xlab</span><span>(</span><span>&quot;Duration (minutes)&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="plot of chunk unnamed-chunk-4" src="https://i0.wp.com/maelle.github.io/figure/source/2017-04-30-radioedit/unnamed-chunk-4-1.png?w=456" data-recalc-dims="1"></p>
<p>And I think these histograms are actually clearer than the boxplots of last post.</p>
<p>So now let’s try quantile regression, which I admit I had never used. The <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/quantreg/vignettes/rq.pdf">vignette of the <code>quantreg</code> package</a> is really nice, it feels a bit old school with a real introduction and a section called “What is a vignette?”. The vignette starts with this sentence “Beran’s (2003) provocative definition of statistics as ‘the study of algorithms for<br>
data analysis’ elevates computational considerations to the forefront of the field.”. <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Roger_Koenker">Roger Koenker</a> also writes “ Although there is now some functionality for quantile regression in most of the major commercial statistical packages, I have a natural predilection for the R environment and the software that I have developed for R.” which made me smile. Last nice point about the vignette I noticed, it encourages a contributor mindset by saying “When such modifications appear to be of general applicability,<br>
it is desirable to communicate them to the package author, so they could be shared with the larger community.” Ok a real last point about that package, the list of authors includes Brian D Ripley with this note from Roger Koenker “Initial (2001) R port from S (to my everlasting shame – how could I have been so slow to adopt R!”.</p>
<p>In the following I will just look at the effect of night on the duration for various quantiles, nothing fancy.</p>
<div>
<pre><code><span>library</span><span>(</span><span>&quot;quantreg&quot;</span><span>)</span><span>
</span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>rq</span><span>(</span><span>duration</span><span> </span><span>~</span><span> </span><span>night</span><span>,</span><span> </span><span>tau</span><span> </span><span>=</span><span> </span><span>seq</span><span>(</span><span>from</span><span> </span><span>=</span><span> </span><span>0.5</span><span>,</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>0.95</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>0.05</span><span>),</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>programs_df</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;pfn&quot;</span><span>)</span><span>
</span><span>summaries</span><span> </span><span>&lt;-</span><span> </span><span>summary</span><span>(</span><span>model</span><span>,</span><span> </span><span>se</span><span> </span><span>=</span><span> </span><span>&quot;ker&quot;</span><span>)</span><span>
</span><span>transform_summary</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>summ</span><span>){</span><span>
</span><span>coeffs</span><span> </span><span>&lt;-</span><span> </span><span>tibble</span><span>::</span><span>as.tibble</span><span>(</span><span>summ</span><span>$</span><span>coefficients</span><span>)</span><span>
</span><span>coeffs</span><span> </span><span>&lt;-</span><span> </span><span>dplyr</span><span>::</span><span>mutate_</span><span>(</span><span>coeffs</span><span>,</span><span> </span><span>tau</span><span> </span><span>=</span><span> </span><span>~</span><span>summ</span><span>$</span><span>tau</span><span>)</span><span>
</span><span>coeffs</span><span>[</span><span>2</span><span>,]</span><span>
</span><span>}</span><span>
</span><span>purrr</span><span>::</span><span>map</span><span>(</span><span>summaries</span><span>,</span><span> </span><span>transform_summary</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>dplyr</span><span>::</span><span>bind_rows</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>knitr</span><span>::</span><span>kable</span><span>()</span><span>
</span></code></pre>
</div>
<table>
<thead>
<tr>
<th>Value</th>
<th>Std. Error</th>
<th>t value</th>
<th>Pr(>|t|)</th>
<th>tau</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>0.0082075</td>
<td>121.83907</td>
<td>0</td>
<td>0.50</td>
</tr>
<tr>
<td>0</td>
<td>0.0082689</td>
<td>0.00000</td>
<td>1</td>
<td>0.55</td>
</tr>
<tr>
<td>1</td>
<td>0.0094450</td>
<td>105.87571</td>
<td>0</td>
<td>0.60</td>
</tr>
<tr>
<td>0</td>
<td>0.0093682</td>
<td>0.00000</td>
<td>1</td>
<td>0.65</td>
</tr>
<tr>
<td>1</td>
<td>0.0106465</td>
<td>93.92756</td>
<td>0</td>
<td>0.70</td>
</tr>
<tr>
<td>1</td>
<td>0.0122657</td>
<td>81.52811</td>
<td>0</td>
<td>0.75</td>
</tr>
<tr>
<td>3</td>
<td>0.0216791</td>
<td>138.38211</td>
<td>0</td>
<td>0.80</td>
</tr>
<tr>
<td>5</td>
<td>0.0298266</td>
<td>167.63566</td>
<td>0</td>
<td>0.85</td>
</tr>
<tr>
<td>7</td>
<td>0.0428316</td>
<td>163.43057</td>
<td>0</td>
<td>0.90</td>
</tr>
<tr>
<td>10</td>
<td>0.0605325</td>
<td>165.20043</td>
<td>0</td>
<td>0.95</td>
</tr>
</tbody>
</table>
<p>This seems to support the idea that upper quantiles of duration are bigger at night. This shift of the distribution of duration at night compared to its distribution during daytime means that there are more long pieces at night… which is what the website claims, so we’re good. Just for info, we are looking at 171668 pieces played at night and 355487 pieces played during the day, that is to say a lot.</p>
<h1>Using keywords to identify the speed of pieces</h1>
<p>Carles said: “For romantic music the type of composition “Scherzo”/”Waltz”/”Nocturne” may tell you something about the mood of the composition”. Okay, time to analyse some text data then! Instead of looking at the mood of the music as suggested by Carles, I’ll look at the tempo of the pieces instead (from <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Tempo">Wikipedia page</a>, see “Italian tempo markings”).</p>
<p>To identify the tempo of each piece indicating one, I shall use <a target="_blank" rel="nofollow" href="https://github.com/dgrtwo/fuzzyjoin"><code>fuzzyjoin</code></a>, a very handy package for joining tables based on say, close geographical coordinates or regular expressions.</p>
<div>
<pre><code><span>tempos</span><span> </span><span>&lt;-</span><span> </span><span>tolower</span><span>(</span><span>c</span><span>(</span><span>&quot;Larghissimo&quot;</span><span>,</span><span> </span><span>&quot;Grave&quot;</span><span>,</span><span> </span><span>&quot;Largo&quot;</span><span>,</span><span>
</span><span>&quot;Lento&quot;</span><span>,</span><span> </span><span>&quot;Larghetto&quot;</span><span>,</span><span> </span><span>&quot;Adagio&quot;</span><span>,</span><span>
</span><span>&quot;Adagietto&quot;</span><span>,</span><span> </span><span>&quot;Andante&quot;</span><span>,</span><span> </span><span>&quot;Andantino&quot;</span><span>,</span><span>
</span><span>&quot;Marcia moderato&quot;</span><span>,</span><span> </span><span>&quot;Andante moderato&quot;</span><span>,</span><span>
</span><span>&quot;Moderato&quot;</span><span>,</span><span> </span><span>&quot;Allegretto&quot;</span><span>,</span><span> </span><span>&quot;Allegro moderato&quot;</span><span>,</span><span>
</span><span>&quot;Allegro&quot;</span><span>,</span><span> </span><span>&quot;Vivace&quot;</span><span>,</span><span> </span><span>&quot;Vivacissimo&quot;</span><span>,</span><span>
</span><span>&quot;Allegrissimo&quot;</span><span>,</span><span> </span><span>&quot;Presto&quot;</span><span>,</span><span> </span><span>&quot;Prestissimo&quot;</span><span>))</span><span>
</span><span>tempos</span><span> </span><span>&lt;-</span><span> </span><span>tibble</span><span>::</span><span>tibble</span><span>(</span><span>tempo</span><span> </span><span>=</span><span> </span><span>tempos</span><span>,</span><span>
</span><span>speed</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>length</span><span>(</span><span>tempos</span><span>))</span><span>
</span><span>programs</span><span> </span><span>&lt;-</span><span> </span><span>dplyr</span><span>::</span><span>mutate</span><span>(</span><span>programs_df</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>tolower</span><span>(</span><span>title</span><span>))</span><span>
</span><span>library</span><span>(</span><span>&quot;fuzzyjoin&quot;</span><span>)</span><span>
</span><span>programs</span><span> </span><span>&lt;-</span><span> </span><span>regex_inner_join</span><span>(</span><span>programs</span><span>,</span><span> </span><span>tempos</span><span>,</span><span>
</span><span>by</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;tempo&quot;</span><span>))</span><span>
</span></code></pre>
</div>
<p>Now I have to take care of pieces that got several tempos, for instance because their title is “grave and allegro blabla” or “something moderato” thus yielding “something” and “something moderato” as tempos. For the first case I’ll randomly sample one of the tempos, for the second case I’ll keep the most precise tempo.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>1</span><span>)</span><span>
</span><span>nrow</span><span>(</span><span>programs</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 120052
</code></pre>
</div>
<div>
<pre><code><span>programs</span><span> </span><span>&lt;-</span><span> </span><span>split</span><span>(</span><span>programs</span><span>,</span><span> </span><span>programs</span><span>$</span><span>datetime</span><span>)</span><span>
</span><span>clean_tempos</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>df</span><span>){</span><span>
</span><span>if</span><span>(</span><span>nrow</span><span>(</span><span>df</span><span>)</span><span> </span><span>></span><span> </span><span>1</span><span>){</span><span>
</span><span>if</span><span>(</span><span>any</span><span>(</span><span>stringr</span><span>::</span><span>str_detect</span><span>(</span><span>df</span><span>$</span><span>tempo</span><span>,</span><span> </span><span>&quot; moderato&quot;</span><span>))){</span><span>
</span><span>df</span><span> </span><span>&lt;-</span><span> </span><span>dplyr</span><span>::</span><span>filter_</span><span>(</span><span>df</span><span>,</span><span> </span><span>lazyeval</span><span>::</span><span>interp</span><span>(</span><span>~</span><span> </span><span>stringr</span><span>::</span><span>str_detect</span><span>(</span><span>df</span><span>$</span><span>tempo</span><span>,</span><span> </span><span>&quot; moderato&quot;</span><span>)))</span><span>
</span><span>}</span><span>else</span><span>{</span><span>
</span><span>df</span><span> </span><span>&lt;-</span><span> </span><span>dplyr</span><span>::</span><span>sample_n</span><span>(</span><span>df</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span><span>return</span><span>(</span><span>df</span><span>)</span><span>
</span><span>}</span><span>
</span><span>programs</span><span> </span><span>&lt;-</span><span> </span><span>purrr</span><span>::</span><span>map</span><span>(</span><span>programs</span><span>,</span><span> </span><span>clean_tempos</span><span>)</span><span>
</span><span>programs</span><span> </span><span>&lt;-</span><span> </span><span>dplyr</span><span>::</span><span>bind_rows</span><span>(</span><span>programs</span><span>)</span><span>
</span><span>nrow</span><span>(</span><span>programs</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 102439
</code></pre>
</div>
<p>Now that we have 102439 pieces with a known speed out of 527168, let’s look at speed distribution depending on the hour of the day.</p>
<div>
<pre><code><span>programs</span><span> </span><span>%>%</span><span>
</span><span>dplyr</span><span>::</span><span>group_by</span><span>(</span><span>night</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>dplyr</span><span>::</span><span>summarize</span><span>(</span><span>mean_speed</span><span> </span><span>=</span><span> </span><span>mean</span><span>(</span><span>speed</span><span>),</span><span>
</span><span>median_speed</span><span> </span><span>=</span><span> </span><span>median</span><span>(</span><span>speed</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>knitr</span><span>::</span><span>kable</span><span>()</span><span>
</span></code></pre>
</div>
<table>
<thead>
<tr>
<th>night</th>
<th>mean_speed</th>
<th>median_speed</th>
</tr>
</thead>
<tbody>
<tr>
<td>FALSE</td>
<td>10.453073</td>
<td>9</td>
</tr>
<tr>
<td>TRUE</td>
<td>9.888101</td>
<td>8</td>
</tr>
</tbody>
</table>
<div>
<pre><code><span>programs</span><span> </span><span>&lt;-</span><span> </span><span>dplyr</span><span>::</span><span>mutate</span><span>(</span><span>programs</span><span>,</span><span> </span><span>speed</span><span> </span><span>=</span><span> </span><span>as.factor</span><span>(</span><span>speed</span><span>))</span><span>
</span><span>library</span><span>(</span><span>&quot;viridis&quot;</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>programs</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>aes</span><span>(</span><span>hour</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>speed</span><span>),</span><span>
</span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;fill&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_viridis</span><span>(</span><span>discrete</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_ipsum</span><span>()</span><span>
</span></code></pre>
</div>
<p><img alt="plot of chunk unnamed-chunk-8" src="https://i0.wp.com/maelle.github.io/figure/source/2017-04-30-radioedit/unnamed-chunk-8-1.png?w=456" data-recalc-dims="1"></p>
<div>
<pre><code><span>ggplot</span><span>(</span><span>programs</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>aes</span><span>(</span><span>night</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>speed</span><span>),</span><span>
</span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;fill&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_viridis</span><span>(</span><span>discrete</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_ipsum</span><span>()</span><span>
</span></code></pre>
</div>
<p><img alt="plot of chunk unnamed-chunk-8" src="https://i0.wp.com/maelle.github.io/figure/source/2017-04-30-radioedit/unnamed-chunk-8-2.png?w=456" data-recalc-dims="1"></p>
<p>Honestly, I don’t really see a change of speed at night, or only a really small one, so my method was probably not a very good way of estimating the liveliness of pieces.</p>
<h1>Conclusion</h1>
<p>I’d like to say thanks again to Bob Rudis, to Radio Swiss Classic webmaster and to <a target="_blank" rel="nofollow" href="https://twitter.com/MilaCarles">my colleague Carles</a>. I was also surprised and happy to receive an email from a university music theory teacher, who told me he could use such data in class to look at popular music pieces. I really like how this blogging thing makes me discover new fields!</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://maelle.github.io/2017/04/30/radioedit/"> Maëlle</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/Ucqeanxyl5A"></a></div>