<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/prophets-of-gloom-using-nlp-to-analyze-radiohead-lyrics/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://feedproxy.google.com/~r/My-misadventures-In-R-Programming/~3/C1wA3txjchA/"> my (mis)adventures in R programming</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>From the first time I listened to Radiohead’s <em>The Bends</em>, the band has been my favorite. I was a grad student in England at the time, and I recall listening to “Fake Plastic Trees” on repeat as I made my way to and from the library each day. By the time <em>OK Computer</em> came out, I was hooked. I remain hooked to this day.</p>
<p>As a writer-turned-data-science practitioner, text mining and natural language processing fascinate me. So in this post I’ll combine these two passions—words and Radiohead—to analyze the band’s lyrics.</p>
<p>The first thing to do is create the dataset, using <a rel="nofollow" href="https://github.com/JosiahParry/geniusR" target="_blank">Josiah Parry’s wonderful <code>geniusR</code> package</a> to obtain the lyrics. I’ve seen other people get the discography from Wikipedia, but it’s a short enough list that it would be more trouble for me to do it that way than to just create the list from scratch.</p>

<pre>library(geniusR)
library(tidytext)
library(tidyverse)
library(wordcloud)
library(dplyr)
library(tidyr)
library(ggplot2)
library(purrr)
library(broom)
theme_set(theme_minimal())

# Create dataframe
album &lt;- c(&quot;Pablo Honey&quot;, &quot;The Bends&quot;, &quot;OK Computer&quot;, &quot;Kid A&quot;,
           &quot;Amnesiac&quot;, &quot;Hail to the Thief&quot;, &quot;In Rainbows&quot;,
           &quot;The King of Limbs&quot;, &quot;A Moon Shaped Pool&quot;)
year &lt;- c(1993, 1995, 1997, 2000, 2001, 2003, 2007, 2011, 2016)
Radiohead_albums &lt;- data.frame(album, year)

# Get lyrics
Radiohead_lyrics &lt;- Radiohead_albums %>%
  mutate(tracks = map2(&quot;Radiohead&quot;, album, genius_album))

Radiohead_lyrics &lt;-  Radiohead_lyrics %>%
  unnest(tracks)

Radiohead_words &lt;- Radiohead_lyrics %>%
  unnest_tokens(word, lyric) %>%
  anti_join(stop_words) %>%
  select(album, word, year, track_title, track_n, line) # get variables in order for 

# convert everything to lower case
Radiohead_words$word &lt;- sapply(Decemberists_words$word, tolower)
head(Radiohead_words)</pre>

<pre>album    word year track_title track_n line
1 Pablo Honey     sun 1993         You       1    2
2 Pablo Honey    moon 1993         You       1    2
3 Pablo Honey   stars 1993         You       1    3
4 Pablo Honey     run 1993         You       1    4
5 Pablo Honey chaotic 1993         You       1    7
6 Pablo Honey   world 1993         You       1   11</pre>

<p>So the first thing I want to do is just get a sense of overall word frequency, so I’ll use the <code>wordcloud</code> package.</p>

<pre>pal &lt;- brewer.pal(7, &quot;GnBu&quot;)

Radiohead_words %>%
  count(word, sort = TRUE) %>%
  with(wordcloud(word, n, max.words = 100, rot.per = 0.35, colors = pal))</pre>
<p><img sizes="(max-width: 600px) 100vw, 600px" src="https://i1.wp.com/lesleylathrop.com/blog/wp-content/uploads/2018/10/radiohead-wordcloud.png?w=450" alt="" class="wp-image-135" data-recalc-dims="1"></p>
<p>If you’ve listened to Radiohead for any length of time, you don’t need me to tell you that their songs are often dark. This is evidenced in the word cloud of their lyrics. The word cloud, though, includes all of their albums together, which can’t then give us a real sense of what each album is about or of the progression over time. Even doing a word cloud for each one isn’t going to give us an adequate picture of meaning because often a word that is unimportant to meaning will nonetheless occur frequently. So, for example, the word <em>there’ll</em> appears as a frequent word in the above word cloud, but without context that word is virtually meaningless. And while we could add that to our list of stopwords to remove, that’s a pretty inelegant and unsophisticated way to adjust term frequency.</p>
<p>But term frequency (tf) is only one way of discerning the importance of specific terms. Another way is to look at inverse document frequency (idf), a method of weighting infrequent words higher than frequent words within a collection or corpus of documents. In turn, term frequency-inverse document frequency (tf-idf), which multiplies the two scores together, is a widely regarded statistical measure for identifying terms important to a document within a collection of documents. Importance increases in proportion to the number of occurrences in a document, but it is then offset by its frequency in the collection as a whole. Hence, a word that is rare in a collection, but frequent in a document, will have a higher weight. </p>
<p>So let’s run the model and use <code>ggplot2</code> to visualize the results.</p>

<pre>Radiohead_tf_idf &lt;- Radiohead_lyrics %>%
  unnest_tokens(word, lyric) %>%
  count(album, word, sort = TRUE) %>%
  ungroup()

plot_radiohead &lt;- Radiohead_tf_idf %>%
  bind_tf_idf(word, album, n) %>%
  arrange(desc(tf_idf)) %>%
  mutate(word = factor(word, levels = rev(unique(word)))) %>%
  mutate(album = factor(album, levels = c(&quot;Pablo Honey&quot;, &quot;The Bends&quot;, &quot;OK Computer&quot;, &quot;Kid A&quot;,
                                           &quot;Amnesiac&quot;, &quot;Hail to the Thief&quot;, &quot;In Rainbows&quot;,
                                           &quot;The King of Limbs&quot;, &quot;A Moon Shaped Pool&quot;)))

plot_radiohead %>% 
  group_by(album) %>% 
  top_n(5, tf_idf) %>% 
  ungroup() %>%
  mutate(word = reorder(word, tf_idf)) %>%
  ggplot(aes(word, tf_idf, fill = album)) +
  geom_col(show.legend = FALSE) +
  labs(x = NULL, y = &quot;tf-idf&quot;) +
  facet_wrap(~album, ncol = 3, scales = &quot;free&quot;) +
  coord_flip()</pre>

<p><img sizes="(max-width: 720px) 100vw, 720px" src="https://i2.wp.com/lesleylathrop.com/blog/wp-content/uploads/2018/10/radiohead-tfidf.png?w=450" alt="" class="wp-image-137" data-recalc-dims="1"></p>
<p>We can see that the term <em>raindrops</em>, which appeared as the most prominent term in the word cloud, is indeed quite important for the <em>Hail to the Thief</em> record, with the highest weight of any term in the entire corpus. We can also see that the terms <em>hurt</em> and <em>haunt</em> from <em>The King of Limbs</em> are also highly weighted for that album. Lastly, we can see that the most important words on <em>A Moon-shaped Pool</em> are non-English words. So let’s look at one of them to see it’s context.</p>

<pre>efil &lt;- Radiohead_lyrics[grep(&quot;efil&quot;, Radiohead_lyrics$lyric),]

efil &lt;- efil %>%
  select(album, track_title, lyric) %>%
  distinct()
efil</pre>

<pre>album track_title           lyric
1 A Moon Shaped Pool Daydreaming ​efil ym fo flaH</pre>

<p>Next, I’m curious about how the band has evolved, so I want to look at which words the band has used at a higher or lower rate over time. I’m relying on code that Julia Silge and David Robinson developed <a rel="nofollow" href="https://www.tidytextmining.com/twitter.html#changes-in-word-use" target="_blank">to analyze changes in their own Twitter output</a> for the book <em>Text Mining with R</em>. As I create the dataset, I’m going to <code>filter()</code> the results so that I’m only keeping words that occur 15 or more times.</p>

<pre>Radiohead_by_time &lt;- Radiohead_words %>%
  count(year, word) %>%
  mutate(time_total = sum(n)) %>%
  group_by(word) %>%
  mutate(word_total = sum(n)) %>%
  ungroup() %>%
  rename(count = n) %>%
  dplyr::filter(word_total > 15)

Radiohead_by_time</pre>

<pre># A tibble: 138 x 5
    year word   count time_total word_total
   </pre>