<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/parsing-text-for-emotion-terms-analysis-visualization-using-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://datascienceplus.com/parsing-text-for-emotion-terms-analysis-visualization-using-r/"> R Programming – DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Recently, I read a <a target="_blank" rel="nofollow" href="http://blog.revolutionanalytics.com/2017/04/warren-buffet-sentiment.html">post</a> regarding a sentiment analysis of Mr Warren Buffett’s annual shareholder letters in the past 40 years written by Michael Toth. In this <a target="_blank" rel="nofollow" href="http://http//michaeltoth.me/sentiment-analysis-of-warren-buffetts-letters-to-shareholders.html">post</a>, only five of the annual shareholder letters showed negative net sentiment scores, whereas a majority of the letters (88%) displayed a positive net sentiment score. Toth noted that the years with negative net sentiment scores (1987, 1990, 2001, 2002 and 2008), coincided with lower annual returns on investments and global market decline. This observation caught my attention and triggered my curiosity about emotion words in those same shareholder letters, and whether or not emotion words were differentially expressed among the 40 letters. </p>
<p>With the explosion of digital and social media, there are various emoticons and emojis that can be embedded in text messages, emails, or other various social media communications, for use in expressing personal feelings or emotions. Emotions may also be expressed in textual forms using words. R offers the <code>get_nrc_sentiment</code> function via the <code>Tidy</code> or <code>Syuzhet</code> packages for analysis of emotion words expressed in text. Both packages implemented Saif Mohammad’s <a target="_blank" rel="nofollow" href="http://www.purl.org/net/NRCemotionlexicon">NRC Emotion lexicon</a>, comprised of eight basic words of emotion expressions for anger, fear, anticipation, trust, surprise, sadness, joy, and disgust. </p>
<p>I have two companion posts on this subject; this post is the first part. The motivations for this post are to illustrate the applications of some of the R tools and approaches:</p>
<blockquote>
<ul>
<li>Analysis of emotion words in textual data</li>
<li>Visualization and presentation of outputs and results </li>
</ul>
</blockquote>
<p>In the second part, unsupervised learning and differential expression of emotion words using R will be attempted.</p>
<h2>The Dataset</h2>
<p>Mr. Warren Buffett’s annual shareholder letters in the past 40-years (1977 – 2016) were downloaded from <a target="_blank" rel="nofollow" href="http://%27http//www.berkshirehathaway.com/letters">this site</a> using the an R code obtained from <a target="_blank" rel="nofollow" href="http://michaeltoth.me/sentiment-analysis-of-warren-buffetts-letters-to-shareholders.html">here</a>.</p>
<pre>
## The R code snippet to retrieve the letters was obtained from Michel Toth's post.
library(pdftools)      
library(rvest)       
library(XML)
# Getting &amp; Reading in HTML Letters
urls_77_97 &lt;- paste('http://www.berkshirehathaway.com/letters/', seq(1977, 1997), '.html', sep='')
html_urls &lt;- c(urls_77_97,
               'http://www.berkshirehathaway.com/letters/1998htm.html',
               'http://www.berkshirehathaway.com/letters/1999htm.html',
               'http://www.berkshirehathaway.com/2000ar/2000letter.html',
               'http://www.berkshirehathaway.com/2001ar/2001letter.html')

letters_html % html_text())
# Getting &amp; Reading in PDF Letters
urls_03_16 &lt;- paste('http://www.berkshirehathaway.com/letters/', seq(2003, 2016), 'ltr.pdf', sep = '')
pdf_urls &lt;- data.frame('year' = seq(2002, 2016),
                       'link' = c('http://www.berkshirehathaway.com/letters/2002pdf.pdf', urls_03_16))
download_pdfs &lt;- function(x) {
  myfile = paste0(x['year'], '.pdf')
  download.file(url = x['link'], destfile = myfile, mode = 'wb')
  return(myfile)
}
pdfs &lt;- apply(pdf_urls, 1, download_pdfs)
letters_pdf % paste(collapse=&quot; &quot;))
tmp &lt;- lapply(pdfs, function(x) if(file.exists(x)) file.remove(x)) 
# Combine letters in a data frame
letters &lt;- do.call(rbind, Map(data.frame, year=seq(1977, 2016), text=c(letters_html, letters_pdf)))
letters$text &lt;- as.character(letters$text)
</pre>
<h3>Load additional required packages</h3>
<pre>
require(tidyverse)
require(tidytext)
require(RColorBrewer)
require(gplots)
theme_set(theme_bw(12))
</pre>
<h2>Descriptive Statistics</h2>
<p>Analysis steps of emotion terms in textual data included word tokenization, pre-processing of tokens to exclude stop words and numbers and then invoking the <code>get_sentiment</code> function using the <code>Tidy</code> package, followed by aggregation and presentation of results. Word tokenization is the process of separating text into single words or unigrams. </p>
<h3>Emotion words frequency and proportions</h3>
<pre>
total_words_count &lt;- letters %>%
    unnest_tokens(word, text) %>%  
    anti_join(stop_words, by = &quot;word&quot;) %>%                  
    filter(!grepl('[0-9]', word)) %>%
   left_join(get_sentiments(&quot;nrc&quot;), by = &quot;word&quot;) %>%                        
    group_by(year) %>%
    summarize(total= n()) %>%
    ungroup()

emotion_words_count &lt;- letters %>% 
  unnest_tokens(word, text) %>%                           
  anti_join(stop_words, by = &quot;word&quot;) %>%                  
  filter(!grepl('[0-9]', word)) %>%
  left_join(get_sentiments(&quot;nrc&quot;), by = &quot;word&quot;) %>%
  filter(!(sentiment == &quot;negative&quot; | sentiment == &quot;positive&quot; | sentiment == &quot;NA&quot;)) %>%
  group_by(year) %>%
  summarize(emotions= n()) %>%
  ungroup()

emotions_to_total_words &lt;- total_words_count %>%
     left_join(emotion_words_count, by=&quot;year&quot;) %>%
               mutate(percent_emotions=round((emotions/total)*100,1))

ggplot(emotions_to_total_words, aes(x=year, y=percent_emotions)) +
     geom_line(size=1) +
     scale_y_continuous(limits = c(0, 35), breaks = c(0, 5, 10, 15, 20, 25, 30, 35)) +
     xlab(&quot;Year&quot;) + 
     ylab(&quot;Emotion terms / total words (%)&quot;) + theme(legend.position=&quot;none&quot;) +
     ggtitle(&quot;Proportion of emotion words usage \n in Mr. Buffett's annual shareholder letters&quot;)
</pre>
<p>Gives this plot:<br>
<a title="" data-rel="lightbox-0" target="_blank" rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/05/Fig1.png?ssl=1"><img sizes="(max-width: 490px) 100vw, 490px" alt="" class="wp-image-6472" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/05/Fig1-490x328.png?w=450&ssl=1" data-recalc-dims="1"></a></p>
<p>Emotion words in the annual shareholder letters accounted for approximately 20% – 25% of the total words count (excluding stop words and numbers). The median emotion count was ~22% of the total words count.</p>
<h2>Depicting distribution of emotion words usage</h2>
<pre>
### pull emotion words and aggregate by year and emotion terms
emotions &lt;- letters %>% 
  unnest_tokens(word, text) %>%                           
  anti_join(stop_words, by = &quot;word&quot;) %>%                  
  filter(!grepl('[0-9]', word)) %>%
  left_join(get_sentiments(&quot;nrc&quot;), by = &quot;word&quot;) %>%
  filter(!(sentiment == &quot;negative&quot; | sentiment == &quot;positive&quot;)) %>%
  group_by(year, sentiment) %>%
  summarize( freq = n()) %>%
  mutate(percent=round(freq/sum(freq)*100)) %>%
  select(-freq) %>%
  ungroup()
### need to convert the data structure to a wide format
emo_box = emotions %>%
spread(sentiment, percent, fill=0) %>%
ungroup()
### color scheme for the box plots (This step is optional)
cols  &lt;- colorRampPalette(brewer.pal(7, &quot;Set3&quot;), alpha=TRUE)(8)
boxplot2(emo_box[,c(2:9)], col=cols, lty=1, shrink=0.8, textcolor=&quot;red&quot;,        xlab=&quot;Emotion Terms&quot;, ylab=&quot;Emotion words count (%)&quot;, main=&quot;Distribution of emotion words count in annual shareholder letters (1978 - 2016&quot;)
</pre>
<p>Which gives this plot:<br>
<a title="" data-rel="lightbox-1" target="_blank" rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/05/Fig2_boxplot.png?ssl=1"><img sizes="(max-width: 490px) 100vw, 490px" alt="" class="wp-image-6473" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/05/Fig2_boxplot-490x314.png?w=450&ssl=1" data-recalc-dims="1"></a></p>
<p>Terms for all eight emotions types were expressed albeit at variable rates. Looking at the box plot, anger, sadness, surprise and trust showed outliers. Besides, anger, disgust and surprise were skewed to the left, whereas Joy was skewed to the right. The n= below each box plot indicates the number of observations that contributed to the distribution of the box plot above it. </p>
<h2>Emotion words usage over time</h2>
<pre>
## yearly line chart
ggplot(emotions, aes(x=year, y=percent, color=sentiment, group=sentiment)) +
geom_line(size=1) +
geom_point(size=0.5) +
xlab(&quot;Year&quot;) +
  ylab(&quot;Emotion words count (%)&quot;) +
  ggtitle(&quot;Emotion words expressed in Mr. Buffett's \n annual shareholder letters&quot;)
</pre>
<p>Gives this plot:<br>
<a title="" data-rel="lightbox-2" target="_blank" rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/05/Fig3_Linechart.png?ssl=1"><img sizes="(max-width: 490px) 100vw, 490px" alt="" class="wp-image-6474" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/05/Fig3_Linechart-490x314.png?w=450&ssl=1" data-recalc-dims="1"></a><br>
Clearly emotion terms referring to trust and anticipation were expressed consistently higher than the other emotion terms in all of the annual shareholder letters. Emotion terms referring to disgust, anger and surprise were expressed consistently lower than the other emotion terms at almost all time points.</p>
<h2> Average emotion words expression using bar charts with error bars </h2>
<pre>
### calculate overall averages and standard deviations for each emotion term
overall_mean_sd &lt;- emotions %>%
     group_by(sentiment) %>%
     summarize(overall_mean=mean(percent), sd=sd(percent))
### draw a bar graph with error bars
ggplot(overall_mean_sd, aes(x = reorder(sentiment, -overall_mean), y=overall_mean)) +
     geom_bar(stat=&quot;identity&quot;, fill=&quot;darkgreen&quot;, alpha=0.7) + 
     geom_errorbar(aes(ymin=overall_mean-sd, ymax=overall_mean+sd), width=0.2,position=position_dodge(.9)) +
     xlab(&quot;Emotion Terms&quot;) +
     ylab(&quot;Emotion words count (%)&quot;) +
     ggtitle(&quot;Emotion words expressed in Mr. Buffett's \n annual shareholder letters (1977 – 2016)&quot;) + 
     theme(axis.text.x=element_text(angle=45, hjust=1)) +
     coord_flip( )
</pre>
<p>Which gives this plot:<br>
<a title="" data-rel="lightbox-3" target="_blank" rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/05/Fig4_Barchart.png?ssl=1"><img sizes="(max-width: 490px) 100vw, 490px" alt="" class="wp-image-6475" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/05/Fig4_Barchart-490x314.png?w=450&ssl=1" data-recalc-dims="1"></a><br>
Emotion words referring to trust, anticipation and joy were over-represented and accounted on average for approximately 60% of all emotion words in all shareholder letters. On the other hand, disgust, surprise and anger were the least expressed emotion terms and accounted on average for approximately 18% of all emotion terms in all shareholder letters.</p>
<h2>Emotion terms usage over time compared to 40-years averages</h2>
<p>For the figure below, the 40-year averages of each emotion terms shown in the above bar chart were subtracted from the yearly percent emotions for any given year. The results were showing higher or lower than average emotion expression levels for the respective years. </p>
<pre>
## Hi / Low plots compared to the 40-years average
emotions_diff &lt;- emotions  %>%
     left_join(overall_mean_sd, by=&quot;sentiment&quot;) %>%
     mutate(difference=percent-overall_mean)

ggplot(emotions_diff, aes(x=year, y=difference, colour=difference>0)) +
geom_segment(aes(x=year, xend=year, y=0, yend=difference),
size=1.1, alpha=0.8) +
geom_point(size=1.0) +
xlab(&quot;Emotion Terms&quot;) +
     ylab(&quot;Net emotion words count (%)&quot;) +
     ggtitle(&quot;Emotion words expressed in Mr. Buffett's \n annual shareholder letters (1977 - 2016)&quot;) + 
theme(legend.position=&quot;none&quot;) +
facet_wrap(~sentiment, ncol=4)
</pre>
<p>Gives this plot:<br>
<a title="" data-rel="lightbox-4" target="_blank" rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/05/Fig5_HiLow_plots.png?ssl=1"><img sizes="(max-width: 490px) 100vw, 490px" alt="" class="wp-image-6476" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/05/Fig5_HiLow_plots-490x314.png?w=450&ssl=1" data-recalc-dims="1"></a></p>
<p>Red lines show lower than the 40-year average emotion expression levels, while blue lines indicate higher than the 40-year average emotion expression levels for the respective years.</p>
<h2>Concluding Remarks</h2>
<p>Excluding stop words and numbers, approximately 1 in 4 words in the annual shareholder letters represented emotion terms. Clearly emotion terms referring to trust, anticipation and joy accounted for approximately 60% of all emotion terms. There were also very limited emotions of fear (approximately 1 in 10 emotion terms). In conclusion, R offers several packages and functions for the evaluation and analyses of emotions words in textual data, as well as visualization and presentation of analysis results. Some of those packages and functions have been illustrated in this post. Hopefully, you find this post and analyses and visualization examples helpful. </p>
<ul></ul>
<p><strong>
<p>Related Post</p>
<p>
<ol>
<li><a title="Permanent Link toUsing MongoDB with R" target="_blank" rel="nofollow" href="https://datascienceplus.com/using-mongodb-with-r/">Using MongoDB with R</a></li>
<li><a title="Permanent Link toFinding Optimal Number of Clusters" target="_blank" rel="nofollow" href="https://datascienceplus.com/finding-optimal-number-of-clusters/">Finding Optimal Number of Clusters</a></li>
<li><a title="Permanent Link toAnalyzing the first Presidential Debate" target="_blank" rel="nofollow" href="https://datascienceplus.com/analyzing-the-first-presidential-debate/">Analyzing the first Presidential Debate</a></li>
<li><a title="Permanent Link toGoodReads: Machine Learning (Part 3)" target="_blank" rel="nofollow" href="https://datascienceplus.com/goodreads-machine-learning-part-3/">GoodReads: Machine Learning (Part 3)</a></li>
<li><a title="Permanent Link toMachine Learning for Drug Adverse Event Discovery" target="_blank" rel="nofollow" href="https://datascienceplus.com/machine-learning-for-drug-adverse-event-discovery/">Machine Learning for Drug Adverse Event Discovery</a></li>
</ol></strong>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://datascienceplus.com/parsing-text-for-emotion-terms-analysis-visualization-using-r/"> R Programming – DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/xa89N8oIGxk"></a></div>