<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/using-tidytext-to-make-sentiment-analysis-easy/"></div></div><div><a data-url="https://www.r-bloggers.com/using-tidytext-to-make-sentiment-analysis-easy/" data-counturl="https://www.r-bloggers.com/using-tidytext-to-make-sentiment-analysis-easy/" data-count="horizontal" data-text="Using tidytext to make sentiment analysis easy" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://jacobsimmering.com/2016/11/15/tidytext/"> jacobsimmering.com</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>

<p></p>

<p><br>
</p>
<p><br>
</p>
<p></p>
<p>Last week I discovered the R package <a target="_blank" rel="nofollow" href="https://github.com/juliasilge/tidytext"><code>tidytext</code></a> and its very nice <a target="_blank" rel="nofollow" href="http://tidytextmining.com/index.html">e-book</a> detailing usage. Julia Silge and David Robinson have significantly reduced the effort it takes for me to “grok” text mining by making it “tidy.”</p>
<p>It certainly helped that a lot of the examples are from Pride and Prejudice and other books by Jane Austen, my most beloved author. Julia Silge’s examples on her blog doing <a target="_blank" rel="nofollow" href="http://juliasilge.com/blog/You-Must-Allow-Me/">NLP</a> and <a target="_blank" rel="nofollow" href="http://juliasilge.com/blog/If-I-Loved-NLP-Less/">sentiment analysis</a> alone would have made me a life-long fan. The gifs from P&amp;P (mostly the 1995 mini-series, to be honest) on her posts and the references in the titles made me very excited. My brain automatically started playing the theme and made me smile.</p>
<p>Okay, enough of that. Moving on.</p>
<p>Seeing her work, I started wondering what I can model to get some insight into my own life. I have a database of 92,372 text messages (basically every message sent to or from me from sometime in 2011/2012 to 2015) but text messages were weird (lots of “lol” and “haha”’s). I think there is some interesting insights there, but probably not what I wanted to cover today.</p>
<p>So I started thinking what other plain text data did I have that might be interesting. And then I realized I have a 149 page dissertation (excluding boilerplate and references) <em>and</em> it was in LaTeX (so easy to parse) <em>and</em> it was written in 5 different files that relate directly to the chapters (intro, lit review, methods, results and a discussion). I could do something with that!</p>
<p>My thesis is currently under embargo while I chop it into its respective papers (one under review, one soon to be under review and one undergoing a final revision. So close.), so I can’t link to it. However, it relates the seasonality of two infectious diseases and local weather patterns.</p>
<p>I wonder how my sentiment changes across the thesis. To do this, I’ll use the <code>tidytext</code> package. Let’s import the relevant packages now.</p>
<pre><code>library(tidyverse)
library(tidytext)
library(stringr)</code></pre>
<p>The <code>tidyverse</code> ecosystem and <code>tidytext</code> play well together (no surprises there) and so I also import <code>tidyverse</code>. The <code>stringr</code> package is useful for filtering out the LaTeX specific code and also for dropping words that have numbers in them (like <code>jefferson1776</code> as a reference or <code>0.05</code>).</p>
<p>Now let’s read in the data (the tex files)</p>
<pre><code>thesis_words &lt;- data_frame(file = paste0(&quot;~/thesis/thesis/&quot;, 
                         c(&quot;introduction.tex&quot;, &quot;lit-review.tex&quot;, &quot;methods.tex&quot;, 
                           &quot;results.tex&quot;, &quot;discussion.tex&quot;))) %>%
  mutate(text = map(file, read_lines))
thesis_words</code></pre>
<pre><code>## # A tibble: 5 × 2
##                               file          text
##                              &lt;chr>        &lt;list>
## 1 ~/thesis/thesis/introduction.tex   &lt;chr [125]>
## 2   ~/thesis/thesis/lit-review.tex &lt;chr [1,386]>
## 3      ~/thesis/thesis/methods.tex   &lt;chr [625]>
## 4      ~/thesis/thesis/results.tex &lt;chr [1,351]>
## 5   ~/thesis/thesis/discussion.tex   &lt;chr [649]></code></pre>
<p>The resulting tibble has a variable <code>file</code> that is the name of the file that created that row and a list-column of the text of that file.</p>
<p>We want to <code>unnest()</code> that tibble, remove the lines that are LaTeX crude (either start with <code>\[A-Z]</code> or <code>\[a-z]</code>, like <code>\section</code> or <code>\figure</code>) and compute a line number.</p>
<pre><code>thesis_words &lt;- thesis_words %>%
  unnest() %>%
  filter(text != &quot;%!TEX root = thesis.tex&quot;) %>%
  filter(!str_detect(text, &quot;^(\\\\[A-Z,a-z])&quot;),
         text != &quot;&quot;) %>%
  mutate(line_number = 1:n(),
         file = str_sub(basename(file), 1, -5))
thesis_words$file &lt;- forcats::fct_relevel(thesis_words$file, c(&quot;introduction&quot;,
                                                  &quot;lit-review&quot;,
                                                  &quot;methods&quot;,
                                                  &quot;results&quot;,
                                                  &quot;discussion&quot;))</code></pre>
<p>Now we have a tibble with <code>file</code> giving us the chapter, <code>text</code> giving us the line of text from the tex files (when I wrote it, I strived to keep my line lengths under 80 characters, hence the relatively short value in <code>text</code>) and <code>line_number</code> giving a counter of the number of lines since the start of the thesis.</p>
<p>Now we want to tokenize (strip each word of any formatting and reduce down to the root word, if possible). This is easy with <code>unnest_tokens()</code>. I’ve also played around with the results and came up with some other words that needed to be deleted (stats terms like <code>ci</code> or <code>p</code>, LaTeX terms like <code>_i</code> or <code>tabular</code> and references/numbers).</p>
<pre><code>thesis_words &lt;- thesis_words %>%
  unnest_tokens(word, text) %>%
  filter(!str_detect(word, &quot;[0-9]&quot;),
         word != &quot;fismanreview&quot;,
         word != &quot;multicolumn&quot;,
         word != &quot;p&quot;,
         word != &quot;_i&quot;,
         word != &quot;c&quot;, 
         word != &quot;ci&quot;,
         word != &quot;al&quot;,
         word != &quot;dowellsars&quot;,
         word != &quot;h&quot;,
         word != &quot;tabular&quot;,
         word != &quot;t&quot;,
         word != &quot;ref&quot;,
         word != &quot;cite&quot;,
         !str_detect(word, &quot;[a-z]_&quot;),
         !str_detect(word, &quot;:&quot;),
         word != &quot;bar&quot;,
         word != &quot;emph&quot;,
         !str_detect(word, &quot;textless&quot;))
thesis_words</code></pre>
<pre><code>## # A tibble: 27,787 × 3
##            file line_number        word
##          &lt;fctr>       &lt;int>       &lt;chr>
## 1  introduction           1 seasonality
## 2  introduction           1          or
## 3  introduction           1         the
## 4  introduction           1    periodic
## 5  introduction           1      surges
## 6  introduction           1         and
## 7  introduction           1       lulls
## 8  introduction           1          in
## 9  introduction           1   incidence
## 10 introduction           1          is
## # ... with 27,777 more rows</code></pre>
<p>Now to compute the sentiment using the words written per line in the thesis. <code>tidytext</code> comes with three sentiment lexicons, <code>affin</code>, <code>bing</code> and <code>nrc</code>. <code>affin</code> provides a score ranging from -5 (very negative) to +5 (very positive) fr 2,476 words. <code>bing</code> provides a label of “negative” or “positive” for 6,788 words. <code>nrc</code> provides a label (anger, anticipation, disgust, fear, joy, negative, positive, sadness, surprise or trust) for 13,901 words. None of these account for negation (“I’m not sad” is a negative sentiment, not a positive one).</p>
<p>Using the <code>nrc</code> lexicon, let’s see how the emotions of my words change over the thesis.</p>
<pre><code>thesis_words %>%
  inner_join(get_sentiments(&quot;nrc&quot;)) %>%
  group_by(index = line_number %/% 25, file, sentiment) %>%
  summarize(n = n()) %>%
  ggplot(aes(x = index, y = n, fill = file)) + 
  geom_bar(stat = &quot;identity&quot;, alpha = 0.8) + 
  facet_wrap(~ sentiment, ncol = 5) </code></pre>
<p><img src="https://i2.wp.com/jacobsimmering.com/2016/img/2016-11-17/nrc_thesis_fig-1.png?w=456" data-recalc-dims="1"></p>
<p>I wasn’t surprised, but at least I wasn’t sad? It looks like I used more “fear” and “negative” words in the lit-review than the other sections. However, it looks like “infectious” as in “infectious diseases” is a fear/negative word. I used that word a lot more in the lit review than other sections.</p>
<p>I can use the <code>bing</code> and <code>afinn</code> lexicons to look at how the sentiment of the words changed over the course of the thesis.</p>
<pre><code>thesis_words %>% 
  left_join(get_sentiments(&quot;bing&quot;)) %>%
  left_join(get_sentiments(&quot;afinn&quot;)) %>%
  group_by(index = line_number %/% 25, file) %>%
  summarize(afinn = mean(score, na.rm = TRUE), 
            bing = sum(sentiment == &quot;positive&quot;, na.rm = TRUE) - sum(sentiment == &quot;negative&quot;, na.rm = TRUE)) %>%
  gather(lexicon, lexicon_score, afinn, bing) %>% 
  ggplot(aes(x = index, y = lexicon_score, fill = file)) +
    geom_bar(stat = &quot;identity&quot;) + 
    facet_wrap(~ lexicon, scale = &quot;free_y&quot;) +
    scale_x_continuous(&quot;Location in thesis&quot;, breaks = NULL) +
    scale_y_continuous(&quot;Lexicon Score&quot;)</code></pre>
<p><img src="https://i0.wp.com/jacobsimmering.com/2016/img/2016-11-17/thesis_sentiment_fig-1.png?w=456" data-recalc-dims="1"></p>
<p>Looking at the two lexicon’s scoring of my thesis, the <code>bing</code> lexicon seems a little more stable if we assume local correlation of sentiments is likely. It seems like I started out all doom and gloom (hey, I needed to convince my committee that it was a real problem!), moved onto more doom and gloom (did I mention this is a problem and my question hasn’t been resolved?), the methods were more neutral, results were more doom and gloom but with a slight uplift at the end followed by more doom and gloom (this really is a problem guys!) and a little bit of hope at the end (now that we know, we can fix this?).</p>
<p>This got me thinking about what a typical academic paper looks like. My mental model for a paper is:</p>
<ol>
<li>show that the problem is really a problem (“</ol>