<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/sentiment-analysis-of-the-lord-of-the-rings-with-tidytext/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.jakubglinka.com/2017-03-01-text_mining_part1/"> Jakub Glinka's Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>

<p></p>
<p>You got me thinking about Watson and its unprecedented flexibility in analyzing different data sources (at least according to IBM). So how difficult it would be to analyse sentiment of one of my favorites books using R? Pretty easy actually – all thanks to new package <strong>tidytext</strong> by Julia Silge and David Robinson…</p>
<h2>The tidy text format</h2>
<p>Tidy text format is defined as a table with one-term-per-row. In short it is just tidy textual data which enables to use <strong>tidyverse</strong> tools to clean and transform data. More about conceptual underpinnings creation of tidytext package can be found in the online book:</p>
<p><a target="_blank" rel="nofollow" href="http://tidytextmining.com/tidytext.html">http://tidytextmining.com/tidytext.html</a></p>
<h2>Parsing data to the tibble format</h2>
<p>I parsed one of the websites that offers LOTR trilogy online. After some pretty basic operations on html source I extracted full text along with chapter names and book parts and book titles:</p>
<div>
<pre><code>## # A tibble: 6 × 4
##                        book   part               chapter
##                       &lt;chr>  &lt;chr>                &lt;fctr>
## 1 1. Fellowship of the Ring Book I A Long-expected Party
## 2 1. Fellowship of the Ring Book I A Long-expected Party
## 3 1. Fellowship of the Ring Book I A Long-expected Party
## 4 1. Fellowship of the Ring Book I A Long-expected Party
## 5 1. Fellowship of the Ring Book I A Long-expected Party
## 6 1. Fellowship of the Ring Book I A Long-expected Party
## # ... with 1 more variables: text &lt;chr>
</code></pre>
</div>
<p>with field <em>text</em> simply containing lines of text from the books:</p>
<div>
<pre><code>## # A tibble: 6 × 1
##                                                    text
##                                                   &lt;chr>
## 1 When Mr. Bilbo Baggins of Bag End announced that h...
## 2 celebrating his eleventy-first birthday with a par...
## 3    there was much talk and excitement in Hobbiton....
## 4 Bilbo was very rich and very peculiar, and had bee...
## 5 for sixty years, ever since his remarkable disappe...
## 6 The riches he had brought back from his travels ha...
</code></pre>
</div>
<h2>Sentiment analysis</h2>
<p>One way to analyze the sentiment of a text is to consider the text as a combination of its individual words and the sentiment content of the whole text as the sum of the sentiment content of the individual words. Since LOTR is naturally divided into chapters we can apply sentiment analysis to them and plot their sentiment scores.</p>
<p>The three general-purpose lexicons available in <strong>tidytext</strong> package are</p>
<ul>
<li>AFINN from Finn Årup Nielsen,</li>
<li>bing from Bing Liu and collaborators, and</li>
<li>nrc from Saif Mohammad and Peter Turney.</li>
</ul>
<p>I will use <strong>Bing</strong> lexicon which is simply a <em>tibble</em> with words and positive and negative words:</p>
<div>
<pre><code><span>get_sentiments</span><span>(</span><span>&quot;bing&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>head</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## # A tibble: 6 × 2
##         word sentiment
##        &lt;chr>     &lt;chr>
## 1    2-faced  negative
## 2    2-faces  negative
## 3         a+  positive
## 4   abnormal  negative
## 5    abolish  negative
## 6 abominable  negative
</code></pre>
</div>
<p>and this is how you run sentiment analysis <em>tidytext</em> way:</p>
<div>
<pre><code><span>lotr</span><span> </span><span>%>%</span><span>
</span><span># split text into words
</span><span> </span><span>unnest_tokens</span><span>(</span><span>word</span><span>,</span><span> </span><span>text</span><span>)</span><span> </span><span>%>%</span><span>
</span><span># remove stop words
</span><span> </span><span>anti_join</span><span>(</span><span>stop_words</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;word&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span># add sentiment scores to words
</span><span> </span><span>left_join</span><span>(</span><span>get_sentiments</span><span>(</span><span>&quot;bing&quot;</span><span>),</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;word&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span># count number of negative and positive words
</span><span> </span><span>count</span><span>(</span><span>chapter</span><span>,</span><span> </span><span>book</span><span>,</span><span> </span><span>sentiment</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>spread</span><span>(</span><span>key</span><span> </span><span>=</span><span> </span><span>sentiment</span><span>,</span><span> </span><span>value</span><span> </span><span>=</span><span> </span><span>n</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ungroup</span><span> </span><span>%>%</span><span>
</span><span># create centered score
</span><span> </span><span>mutate</span><span>(</span><span>sentiment</span><span> </span><span>=</span><span> </span><span>positive</span><span> </span><span>-</span><span> </span><span>negative</span><span> </span><span>-</span><span> </span><span>mean</span><span>(</span><span>positive</span><span> </span><span>-</span><span> </span><span>negative</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>book</span><span>,</span><span> </span><span>chapter</span><span>,</span><span> </span><span>sentiment</span><span>)</span><span> </span><span>%>%</span><span>
</span><span># reorder chapter levels
</span><span> </span><span>mutate</span><span>(</span><span>chapter</span><span> </span><span>=</span><span> </span><span>factor</span><span>(</span><span>as.character</span><span>(</span><span>chapter</span><span>),</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>levels</span><span>(</span><span>chapter</span><span>)[</span><span>61</span><span>:</span><span>1</span><span>]))</span><span> </span><span>%>%</span><span>
</span><span># plot
</span><span> </span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>chapter</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>sentiment</span><span>))</span><span> </span><span>+</span><span> </span><span>geom_bar</span><span>(</span><span>stat</span><span> </span><span>=</span><span> </span><span>&quot;identity&quot;</span><span>,</span><span> </span><span>aes</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>book</span><span>))</span><span> </span><span>+</span><span> </span><span>theme_classic</span><span>()</span><span> </span><span>+</span><span> </span><span>theme</span><span>(</span><span>axis.text.x</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>angle</span><span> </span><span>=</span><span> </span><span>90</span><span>))</span><span> </span><span>+</span><span> </span><span>coord_flip</span><span>()</span><span> </span><span>+</span><span> </span><span>ylim</span><span>(</span><span>-250</span><span>,</span><span> </span><span>250</span><span>)</span><span> </span><span>+</span><span>
</span><span>ggtitle</span><span>(</span><span>&quot;Centered sentiment scores&quot;</span><span>,</span><span> </span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;for LOTR chapters&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="plot of chunk unnamed-chunk-4" src="https://i1.wp.com/www.jakubglinka.com/img/2017-03-01-text_mining_part1-unnamed-chunk-4-1.png?w=456" data-recalc-dims="1"></p>
<p>It’s pretty neat if you ask me.</p>
<hr>
<p>Code for this post can be found here:<br>
<a target="_blank" rel="nofollow" href="https://github.com/jakubglinka/posts/tree/master/text_mining_part1">github</a></p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.jakubglinka.com/2017-03-01-text_mining_part1/"> Jakub Glinka's Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/G9oIEXr_0D8"></a></div>