<div>As promised in <a href="http://www.deeplytrivial.com/2018/05/statistics-sunday-taylor-swift-vs-lorde.html">last week’s post</a>, this week: sentiment analysis, also with song lyrics.
<p>Sentiment analysis is a method of natural language processing that involves classifying words in a document based on whether a word is positive or negative, or whether it is related to a set of basic human emotions; the exact results differ based on the sentiment analysis method selected. The <a rel="nofollow" href="https://cran.r-project.org/web/packages/tidytext/vignettes/tidytext.html" target="_blank">tidytext</a> R package has 4 different sentiment analysis methods:</p>
<ul>
<li>“AFINN” for <a href="http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010">Finn Årup Nielsen</a> – which classifies words from -5 to +5 in terms of negative or positive valence</li>
<li>“bing” for <a href="https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html">Bing Liu and colleagues</a> – which classifies words as either positive or negative</li>
<li>“loughran” for <a href="https://sraf.nd.edu/textual-analysis/resources/">Loughran-McDonald</a> – mostly for financial and nonfiction works, which classifies as positive or negative, as well as topics of uncertainty, litigious, modal, and constraining</li>
<li>“nrc” for the <a href="http://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm">NRC lexicon</a> – which classifies words into eight <a href="http://www.deeplytrivial.com/2016/05/on-feelings-woah-oh-oh-feelings.html">basic emotions</a> (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) as well as positive or negative sentiment</li>
</ul>
<p>Sentiment analysis works on unigrams – single words – but you can aggregate across multiple words to look at sentiment across a text.</p>
<p>To demonstrate sentiment analysis, I’ll use one of my favorite songs: “Hotel California” by the Eagles.</p>
<p>I know, I know.</p>
<div><a rel="nofollow" href="https://i1.wp.com/3.bp.blogspot.com/-E8RqtLBZX1U/Wv46lGHnMBI/AAAAAAAALkQ/Ewq0bsYAUSw5ea-o4sFKKw0Nd9aGJQKmACLcBGAs/s1600/eagles.gif?ssl=1" target="_blank"><img border="0" src="https://i2.wp.com/3.bp.blogspot.com/-E8RqtLBZX1U/Wv46lGHnMBI/AAAAAAAALkQ/Ewq0bsYAUSw5ea-o4sFKKw0Nd9aGJQKmACLcBGAs/s320/eagles.gif?resize=320%2C161&ssl=1" data-original-width="450" width="320" data-original-height="252" data-recalc-dims="1" height="161"></a></div>
<p>Using similar code as last week, let’s pull in the lyrics of the song.</p>
<div>
<div>
<div>
<pre><span>library</span><span>(geniusR)</span><br><span>library</span><span>(tidyverse)</span><br></pre>
</div>
<div>
<pre><span>hotel_calif</span> <span>&lt;-</span> <span>genius_lyrics</span><span>(</span><span>artist</span> <span>=</span> <span>&quot;Eagles&quot;</span><span>,</span> <span>song</span> <span>=</span> <span>&quot;Hotel California&quot;</span><span>)</span> <span>%>%</span><br>  <span>mutate</span><span>(</span><span>line</span> <span>=</span> <span>row_number</span><span>())</span><br></pre>
</div>
</div>
</div>
<p>First, we’ll chop up these 43 lines into individual words, using the tidytext package and unnest_tokens function. </p>
<div>
<div>
<div>
<pre><span>library</span><span>(tidytext)</span><br><span>tidy_hc</span> <span>&lt;-</span> <span>hotel_calif</span> <span>%>%</span><br>  <span>unnest_tokens</span><span>(word,lyric)</span><br></pre>
</div>
</div>
</div>
<p>This is also probably the point I would remove stop words with anti_join. But these common words are very unlikely to have a sentiment attached to them, so I’ll leave them in, knowing they’ll be filtered out anyway by this analysis. We have 4 lexicons to choose from. Loughran is more financial and textual, but we’ll still see how well it can classify the words anyway. First, let’s create a data frame of our 4 sentiment lexicons.</p>
<div>
<div>
<div>
<pre><span>new_sentiments</span> <span>&lt;-</span> <span>sentiments</span> <span>%>%</span><br>  <span>mutate</span><span>(</span> <span>sentiment</span> <span>=</span> <span>ifelse</span><span>(lexicon</span> <span>==</span> <span>&quot;AFINN&quot;</span> <span>&amp;</span> <span>score</span> <span>>=</span> <span>0</span><span>,</span> <span>&quot;positive&quot;</span><span>,</span><br>                             <span>ifelse</span><span>(lexicon</span> <span>==</span> <span>&quot;AFINN&quot;</span> <span>&amp;</span> <span>score</span> <span></span> <span>0</span><span>,</span><br>                                    <span>&quot;negative&quot;</span><span>, sentiment)))</span> <span>%>%</span><br>  <span>group_by</span><span>(lexicon)</span> <span>%>%</span><br>  <span>mutate</span><span>(</span><span>words_in_lexicon</span> <span>=</span> <span>n_distinct</span><span>(word))</span> <span>%>%</span><br>  <span>ungroup</span><span>()</span><br></pre>
</div>
</div>
</div>
<p>Now, we’ll see how well the 4 lexicons match up with the words in the lyrics. Big thanks to <a href="https://www.datacamp.com/community/tutorials/sentiment-analysis-R">Debbie Liske at Data Camp</a> for this piece of code (and several other pieces used in this post):</p>
<div>
<div>
<div>
<pre><span>my_kable_styling</span> <span>&lt;-</span> <span>function</span><span>(</span><span>dat</span><span>,</span> <span>caption</span><span>) {</span><br>  <span>kable</span><span>(dat,</span> <span>&quot;html&quot;</span><span>,</span> <span>escape</span> <span>=</span> <span>FALSE</span><span>,</span> <span>caption</span> <span>= caption)</span> <span>%>%</span><br>    <span>kable_styling</span><span>(</span><span>bootstrap_options</span> <span>=</span> <span>c</span><span>(</span><span>&quot;striped&quot;</span><span>,</span> <span>&quot;condensed&quot;</span><span>,</span> <span>&quot;bordered&quot;</span><span>),</span><br>                  <span>full_width</span> <span>=</span> <span>FALSE</span><span>)</span><br><span>}</span><br><br><br><span>library</span><span>(kableExtra)</span><br><span>library</span><span>(formattable)</span><br><span>library</span><span>(yarrr)</span><br></pre>
</div>
<div>
<pre><span>tidy_hc</span> <span>%>%</span><br>  <span>mutate</span><span>(</span><span>words_in_lyrics</span> <span>=</span> <span>n_distinct</span><span>(word))</span> <span>%>%</span><br>  <span>inner_join</span><span>(new_sentiments)</span> <span>%>%</span><br>  <span>group_by</span><span>(lexicon, words_in_lyrics, words_in_lexicon)</span> <span>%>%</span><br>  <span>summarise</span><span>(</span><span>lex_match_words</span> <span>=</span> <span>n_distinct</span><span>(word))</span> <span>%>%</span><br>  <span>ungroup</span><span>()</span> <span>%>%</span><br>  <span>mutate</span><span>(</span><span>total_match_words</span> <span>=</span> <span>sum</span><span>(lex_match_words),</span><br>         <span>match_ratio</span> <span>= lex_match_words</span><span>/</span><span>words_in_lyrics)</span> <span>%>%</span><br>  <span>select</span><span>(lexicon, lex_match_words, words_in_lyrics, match_ratio)</span> <span>%>%</span><br>  <span>mutate</span><span>(</span><span>lex_match_words</span> <span>=</span> <span>color_bar</span><span>(</span><span>&quot;lightblue&quot;</span><span>)(lex_match_words),</span><br>         <span>lexicon</span> <span>=</span> <span>color_tile</span><span>(</span><span>&quot;lightgreen&quot;</span><span>,</span><span>&quot;lightgreen&quot;</span><span>)(lexicon))</span> <span>%>%</span><br>  <span>my_kable_styling</span><span>(</span><span>caption</span> <span>=</span> <span>&quot;Lyrics Found In Lexicons&quot;</span><span>)</span><br></pre>
</div>
<div>
<pre>## Joining, by = &quot;word&quot;<br></pre>
</div>
<table>
<caption>Lyrics Found In Lexicons</caption>
<thead>
<tr>
<th> lexicon </th>
<th> lex_match_words </th>
<th> words_in_lyrics </th>
<th> match_ratio </th>
</tr>
</thead>
<tbody>
<tr>
<td> <span>AFINN </span> </td>
<td> <span>18</span> </td>
<td> 175 </td>
<td> 0.1028571 </td>
</tr>
<tr>
<td> <span>bing </span> </td>
<td> <span>18</span> </td>
<td> 175 </td>
<td> 0.1028571 </td>
</tr>
<tr>
<td> <span>loughran</span> </td>
<td> <span>1</span> </td>
<td> 175 </td>
<td> 0.0057143 </td>
</tr>
<tr>
<td> <span>nrc </span> </td>
<td> <span>23</span> </td>
<td> 175 </td>
<td> 0.1314286 </td>
</tr>
</tbody>
</table></div>
</div>
<p>NRC offers the best match, classifying about 13% of the words in the lyrics. (It’s not unusual to have such a low percentage. Not all words have a sentiment.)</p>
<div>
<div>
<div>
<pre><span>hcsentiment</span> <span>&lt;-</span> <span>tidy_hc</span> <span>%>%</span><br>  <span>inner_join</span><span>(</span><span>get_sentiments</span><span>(</span><span>&quot;nrc&quot;</span><span>),</span> <span>by</span> <span>=</span> <span>&quot;word&quot;</span><span>)</span><br><br><span>hcsentiment</span><br></pre>
</div>
<div>
<pre>## # A tibble: 103 x 4<br>##    track_title       line word   sentiment<br>##    </pre></div></div></div></div>