<div>Sunday night was a big night for Taylor Swift – not only was she nominated for multiple <a rel="nofollow" href="https://www.billboard.com/articles/news/bbma/8456842/billboard-music-awards-2018-winners-list-bbmas" target="_blank">Billboard Music Awards</a>; she took home Top Female Artist and Top Selling Album. So I thought it was a good time for some <a rel="nofollow" href="http://www.deeplytrivial.com/2018/05/statistics-sunday-taylor-swift-vs-lorde.html" target="_blank">more Taylor Swift-themed statistical analysis</a>.
<p>When I started this blog back in 2011, my goal was to write deep thoughts on trivial topics – specifically, to overthink and overanalyze pop culture and related topics that appear fluffy until you really dig into them. Recently, I’ve been blogging more about statistics, research, R, and data science, and I’ve loved getting to teach and share.</p>
<p>But sometimes, you just want to overthink and overanalyze pop culture.</p>
<p>So in a similar vein to the <a href="http://www.deeplytrivial.com/2018/05/statistics-sunday-taylor-swift-vs-lorde.html">text analysis</a> I’ve been <a href="https://www.deeplytrivial.com/2018/05/statistics-sunday-welcome-to-sentiment.html">demonstrating on my blog</a>, I decided to answer a question I’m sure we all have – as Taylor Swift moved from country sweetheart to mega pop star, how have the words she uses in her songs changed?</p>
<p>I’ve used the geniusR package on a couple posts, and I’ll be using it again today to answer this question. I’ll be pulling in some additional code, some based on code from the <a rel="nofollow" href="https://www.amazon.com/Text-Mining-R-Tidy-Approach-ebook/dp/B071K9RT9Z/ref=sr_1_fkmr0_1?ie=UTF8&qid=1526611640&sr=8-1-fkmr0&keywords=text+analysis+in+R+julia+silge" target="_blank"><i>Text Mining with R: A Tidy Approach</i></a> book I recently devoured, some written to try to tackle this problem I’ve created for myself to solve. I’ve shared all my code and tried to credit those who helped me write it where I can.</p>
<p>First, we want to pull in the names of Taylor Swift’s 6 studio albums. I found these and their release dates on Wikipedia. While there are only 6 and I could easily copy and paste them to create my data frame, I wanted to pull that data directly from Wikipedia, to write code that could be used on a larger set in the future. Thanks to <a href="https://www.r-bloggers.com/using-rvest-to-scrape-an-html-table/">this post</a>, I could, with a couple small tweaks.</p>
<div>
<div>
<div>
<pre><span>library</span><span>(rvest)</span><br></pre>
</div>
<div>
<pre>## Loading required package: xml2<br></pre>
</div>
<div>
<pre><span>TSdisc</span> <span>&lt;-</span> <span>'https://en.wikipedia.org/wiki/Taylor_Swift_discography'</span><br><br><span>disc</span> <span>&lt;-</span> <span>TSdisc</span> <span>%>%</span><br>  <span>read_html</span><span>()</span> <span>%>%</span><br>  <span>html_nodes</span><span>(</span><span>xpath</span> <span>=</span> <span>'//*[@id=&quot;mw-content-text&quot;]/div/table[2]'</span><span>)</span> <span>%>%</span><br>  <span>html_table</span><span>(</span><span>fill</span> <span>=</span> <span>TRUE</span><span>)</span><br></pre>
</div>
</div>
</div>
<p>Since html() is deprecated, I replaced it with read_html(), and I got errors if I didn’t add fill = TRUE. The result is a list of 1, with an 8 by 14 data frame within that single list object. I can pull that out as a separate data frame.</p>
<div>
<div>
<div>
<pre><span>TS_albums</span> <span>&lt;-</span> <span>disc[[</span><span>1</span><span>]]</span><br></pre>
</div>
</div>
</div>
<p>The data frame requires a little cleaning. First up, there are 8 rows, but only 6 albums. Because the Wikipedia table had a double header, the second header was read in as a row of data, so I want to delete that, because I only care about the first two columns anyway. The last row contains a footnote that was included with the table. So I removed those two rows, first and last, and dropped the columns I don’t need. Second, the information I want with release date was in a table cell along with record label and formats (e.g., CD, vinyl). I don’t need those for my purposes, so I’ll only pull out the information I want and drop the rest. Finally, I converted year from character to numeric – this becomes important later on.</p>
<div>
<div>
<div>
<pre><span>library</span><span>(tidyverse)</span><br></pre>
</div>
<div>
<pre><span>TS_albums</span><span>&lt;-</span><span>TS_albums[</span><span>2</span><span>:</span><span>7</span><span>,</span><span>1</span><span>:</span><span>2</span><span>]</span><br><br><span>TS_albums</span> <span>&lt;-</span> <span>TS_albums</span> <span>%>%</span><br>  <span>separate</span><span>(`Album details`,</span> <span>c</span><span>(</span><span>&quot;Released&quot;</span><span>,</span><span>&quot;Month&quot;</span><span>,</span><span>&quot;Day&quot;</span><span>,</span><span>&quot;Year&quot;</span><span>),</span><br>           <span>extra</span><span>=</span><span>'drop'</span><span>)</span> <span>%>%</span><br>  <span>select</span><span>(</span><span>c</span><span>(</span><span>&quot;Title&quot;</span><span>,</span><span>&quot;Year&quot;</span><span>))</span><br><br><span>TS_albums</span><span>$</span><span>Year</span><span>&lt;-</span><span>as.numeric</span><span>(TS_albums</span><span>$</span><span>Year)</span><br></pre>
</div>
</div>
</div>
<p>I asked geniusR to download lyrics for all 6 albums. (Note: this code may take a couple minutes to run.) It nests all of the individual album data, including lyrics, into a single column, so I just need to unnest that to create a long file, with album title and release year applied to each unnested line.</p>
<div>
<div>
<div>
<pre><span>library</span><span>(geniusR)</span><br><br><span>TS_lyrics</span> <span>&lt;-</span> <span>TS_albums</span> <span>%>%</span><br>  <span>mutate</span><span>(</span><span>tracks</span> <span>=</span> <span>map2</span><span>(</span><span>&quot;Taylor Swift&quot;</span><span>, Title, genius_album))</span><br></pre>
</div>
<div>
<pre>## Joining, by = c(&quot;track_title&quot;, &quot;track_n&quot;, &quot;track_url&quot;)<br>## Joining, by = c(&quot;track_title&quot;, &quot;track_n&quot;, &quot;track_url&quot;)<br>## Joining, by = c(&quot;track_title&quot;, &quot;track_n&quot;, &quot;track_url&quot;)<br>## Joining, by = c(&quot;track_title&quot;, &quot;track_n&quot;, &quot;track_url&quot;)<br>## Joining, by = c(&quot;track_title&quot;, &quot;track_n&quot;, &quot;track_url&quot;)<br>## Joining, by = c(&quot;track_title&quot;, &quot;track_n&quot;, &quot;track_url&quot;)<br></pre>
</div>
<div>
<pre><span>TS_lyrics</span> <span>&lt;-</span> <span>TS_lyrics</span> <span>%>%</span><br>  <span>unnest</span><span>(tracks)</span><br></pre>
</div>
</div>
</div>
<p>Now we’ll tokenize our lyrics data frame, and start doing our word analysis.</p>
<div>
<div>
<div>
<pre><span>library</span><span>(tidytext)</span><br><br><span>tidy_TS</span> <span>&lt;-</span> <span>TS_lyrics</span> <span>%>%</span><br>  <span>unnest_tokens</span><span>(word, lyric)</span> <span>%>%</span><br>  <span>anti_join</span><span>(stop_words)</span><br></pre>
</div>
<div>
<pre>## Joining, by = &quot;word&quot;<br></pre>
</div>
<div>
<pre><span>tidy_TS</span> <span>%>%</span><br>  <span>count</span><span>(word,</span> <span>sort</span> <span>=</span> <span>TRUE</span><span>)</span><br></pre>
</div>
<div>
<pre>## # A tibble: 2,024 x 2<br>##    word      n<br>##    </pre></div></div></div></div>