<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/tidy-text-mining-of-david-bowies-lyrics/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.mango-solutions.com/wp/2017/02/tidy-text-mining-of-david-bowies-lyrics/"> Mango Solutions » R Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><em>by Nic Crane</em></p>
<p>After hearing about <strong>tidytext</strong> from colleagues who went to the RStudio conference last month, I decided to try it out. If you want to try out tidy text mining for yourself, check out the official website at <a target="_blank" rel="nofollow" href="http://tidytextmining.com/">http://tidytextmining.com</a>.</p>
<p>Legendary performer David Bowie would have been 70 earlier this month, and as a tribute to his impressive career, I decided to take a look at his lyrics.</p>
<p>First, I used the <strong>rvest</strong> package to scrape all the lyrical content from all albums on DavidBowie.com. Only albums from before 2016 which had lyrics present on this website and weren’t re-releases were included (i.e. the eponymous album of 1969 was omitted as it was re-released under the title “Space Oddity”).</p>
<p>Let’s take a look at the structure of the lyrics dataset.</p>
<div>
<pre>str(bowieLyrics)</pre>
</div>
<pre>## 'data.frame':    221 obs. of  4 variables:
##  $ lyrics  : chr  &quot; Pushing thru&lt;U+0092> the market square\nSo many mothers sighing,\nNews had just come over,\nWe had five years left to cry in.\nNews g&quot;| __truncated__ &quot; Stone love, she kneels before the grave\nA brave son who gave his life to save the slogan\nThat hovers between the headstone a&quot;| __truncated__ &quot; I&lt;U+0092>m an alligator\nI&lt;U+0092>m a mama-papa comin&lt;U+0092> for you.\nI&lt;U+0092>m the space invader\nI&lt;U+0092>ll be a rock n rollin bitch for you.\nKeep your mo&quot;| __truncated__ &quot; Didn&lt;U+0092>t know what time it was\nThe lights were low,\nI leaned back on my radio,\nSome cat was layin&lt;U+0092> down some rock n roll,\nLo&quot;| __truncated__ ...
##  $ album   : chr  &quot;The Rise And Fall Of Ziggy Stardust And The Spiders From Mars&quot; &quot;The Rise And Fall Of Ziggy Stardust And The Spiders From Mars&quot; &quot;The Rise And Fall Of Ziggy Stardust And The Spiders From Mars&quot; &quot;The Rise And Fall Of Ziggy Stardust And The Spiders From Mars&quot; ...
##  $ albumNum: int  1 1 1 1 1 1 1 1 1 1 ...
##  $ trackNum: int  1 2 3 4 5 6 7 8 9 10 ...
</pre>
<p>The dataset contains a character string for each track, with the lyrics for the whole song. It looks good, but it’s a bit of a mess, and definitely needs more work before we can use it for any kind of analysis.</p>
<p>The first job is to tidy up the dataset. The <code>unnest_tokens</code> function in the tidytext package makes easy work of this for us.</p>
<div>
<pre># tidy data so it has one-token-per-document-per-row format
unnested &lt;- unnest_tokens(bowieLyrics, word, lyrics)
kable(head(unnested, 5))</pre>
</div>
<pre>album                           albumNum   trackNum  word </pre>
<hr>
<p>1 The Rise And Fall Of Ziggy Stardust And The Spiders From Mars 1 1 pushing 1.1 The Rise And Fall Of Ziggy Stardust And The Spiders From Mars 1 1 thru<br>
1.2 The Rise And Fall Of Ziggy Stardust And The Spiders From Mars 1 1 the<br>
1.3 The Rise And Fall Of Ziggy Stardust And The Spiders From Mars 1 1 market<br>
1.4 The Rise And Fall Of Ziggy Stardust And The Spiders From Mars 1 1 square</p>
<p>Looking good, although these lyrics are displayed in the order in which they occur. Let’s sort them and take a look at what the most common words are.</p>
<div>
<pre># What are the most common words? 
count(unnested, word, sort=TRUE)</pre>
</div>
<pre>## # A tibble: 4,596 × 2
##     word     n
##    &lt;chr> &lt;int>
## 1    the  1828
## 2    and  1021
## 3    you   972
## 4      i   940
## 5      a   897
## 6     to   788
## 7     in   591
## 8     of   587
## 9     my   501
## 10    me   453
## # ... with 4,586 more rows
</pre>
<p>That’s great, but not so interesting. Let’s remove words like ‘the’ and ‘and’, and take another look.</p>
<div>
<pre>
data(&quot;stop_words&quot;)
unnested &lt;- unnested %>% anti_join(stop_words)
</pre>
</div>
<pre>
## Joining, by = &quot;word&quot;
</pre>
<div>
<pre>
# Cool, let's try again!
count(unnested, word, sort=TRUE)
</pre>
</div>
<pre>
## # A tibble: 4,129 × 2
##     word     n
##    &lt;chr> &lt;int>
## 1   love   221
## 2   time   141
## 3   life   102
## 4   eyes    98
## 5   baby    93
## 6     ya    87
## 7    day    86
## 8  world    69
## 9   feel    62
## 10 night    61
## # ... with 4,119 more rows
</pre>
<p>OK, this is reminding me a lot more of the Bowie songs that I know and love!</p>
<p>I bet this would look better as a word cloud though…</p>
<div>
<pre>
library(wordcloud)

# Some interesting results - one of which is the word &quot;ch&quot; as in &quot;ch ch changes&quot;!
unnested %>%
  anti_join(stop_words) %>%
  count(word) %>%
  with(wordcloud(word, n, max.words = 120, colors=&quot;purple4&quot;))
</pre>
</div>
<p><a target="_blank" rel="nofollow" href="https://i0.wp.com/github.com/MangoTheCat/blog_bowie_tidytext/blob/master/bowie_blog_post_files/figure-html/unnamed-chunk-6-1.png?ssl=1"><img alt="" src="https://i2.wp.com/github.com/MangoTheCat/blog_bowie_tidytext/raw/master/bowie_blog_post_files/figure-html/unnamed-chunk-6-1.png?w=456&ssl=1" data-recalc-dims="1"></a></p>
<p>Finally, let’s take a quick look at words used to express sentiments in Bowie’s lyrics.</p>
<p>The <code>get_sentiments</code> function allows us to retrieve sentiment lexicons. They’re in a tidy format so can easily be joined with our tidy text data.</p>
<p>First, we join the datasets and create a word cloud of only the positive words.</p>
<div>
<pre>
# Sentiment analysis of lyrical content
bing &lt;- get_sentiments(&quot;bing&quot;)

byEmotion &lt;- unnested %>%
  inner_join(bing)

byEmotion %>%
  filter(sentiment == &quot;positive&quot;) %>%
  count(word) %>%
  with(wordcloud(word, n, max.words = 50, colors=&quot;red&quot;))
</pre>
</div>
<p><a target="_blank" rel="nofollow" href="https://i2.wp.com/github.com/MangoTheCat/blog_bowie_tidytext/blob/master/bowie_blog_post_files/figure-html/unnamed-chunk-7-1.png?ssl=1"><img alt="" src="https://i1.wp.com/github.com/MangoTheCat/blog_bowie_tidytext/raw/master/bowie_blog_post_files/figure-html/unnamed-chunk-7-1.png?w=456&ssl=1" data-recalc-dims="1"></a></p>
<p>By far the most common positive word in David Bowie’s lyrics was ‘love’.</p>
<p>Now let’s take a look at the negative words.</p>
<div>
<pre>
byEmotion %>%
  filter(sentiment == &quot;negative&quot;) %>%
  count(word) %>%
  with(wordcloud(word, n, max.words = 50, colors=&quot;black&quot;))
</pre>
</div>
<p><a target="_blank" rel="nofollow" href="https://i0.wp.com/github.com/MangoTheCat/blog_bowie_tidytext/blob/master/bowie_blog_post_files/figure-html/unnamed-chunk-8-1.png?ssl=1"><img alt="" src="https://i1.wp.com/github.com/MangoTheCat/blog_bowie_tidytext/raw/master/bowie_blog_post_files/figure-html/unnamed-chunk-8-1.png?w=456&ssl=1" data-recalc-dims="1"></a></p>
<p>I think I prefer the first word cloud!</p>
<p>This was just a quick look at the tidytext package, but we’re a big fan of how easy it is to quickly analyse text data. Check out <a target="_blank" rel="nofollow" href="http://tidytextmining.com/">http://tidytextmining.com/</a> to see what else this package can do!</p>
<p>This blogpost (with code) can also be found on <a target="_blank" rel="nofollow" href="https://github.com/MangoTheCat/blog_bowie_tidytext">GitHub</a>.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.mango-solutions.com/wp/2017/02/tidy-text-mining-of-david-bowies-lyrics/"> Mango Solutions » R Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/zdsgXWvImqo"></a></div>