<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/single-word-analysis-of-early-19th-century-poetry-using-tidytext/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://eighty20.github.io//package%20exploration/2017/06/12/sentiment-blog-post/"> Pareto's Playground</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>When reading poetry, it feels as if you are reading emotion. A good poet can use words to bring to their reader whichever emotions they choose. To do this, these poets often have to look deep within themselves, using their own lives as fuel. For this reason you would expect that poets and as a result, their poetry, would be influenced by the events happening around them. If certain events caused the poet’s view of the world to be an unhappy one, you’d expect their poetry to be unhappy, and of course the same for the opposite.</p>
<p>In this analysis I attempt to show this relation between world events and the poets of those times.</p>
<p>Gathering the required data – this being poems and their dates – proved to be a very difficult task. I had initially hoped to be able to show an analysis of the entire 20th century (this may still happen when I find more time and/or sources), but in the end I settled for the years between 1890 – 1925. I managed to collect an overall number of 97 poems (a few of which are excluded from the analysis due to being written in 1888 or 1889) from 6 of the most well known poets of the time. These poets being:</p>
<ul>
<li>William Yeats</li>
<li>T. S. Eliot</li>
<li>Robert Frost</li>
<li>Wallace Stevens</li>
<li>E. E. Cummings</li>
<li>D. H. Lawrence</li>
</ul>
<p>I will only be going through the preparation code for Yeats. The rest is similar but not exactly the same, and can be found in my github repository (https://github.com/Laurence-Sonnenberg/single_word_poetry_analysis) if anyone would like to re-create anything I have done. The actual analysis is done using all poets/poems combined.</p>
<p>Firstly I load the packages I’m going to need. I have included a small function that other people might find useful.</p>
<figure>
<pre><code data-lang="r"><span>load_pkg</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>packagelist</span><span>)</span><span> </span><span>{</span><span>
</span><span># Check if any package needs installation:
</span><span> </span><span>PackagesNeedingInstall</span><span> </span><span>&lt;-</span><span> </span><span>packagelist</span><span>[</span><span>!</span><span>(</span><span>packagelist</span><span> </span><span>%in%</span><span> </span><span>installed.packages</span><span>()[,</span><span>&quot;Package&quot;</span><span>])]</span><span>
</span><span>if</span><span>(</span><span>length</span><span>(</span><span>PackagesNeedingInstall</span><span>))</span><span>
</span><span>{</span><span>
</span><span>cat</span><span>(</span><span>&quot;\nInstalling necesarry packages: &quot;</span><span>,</span><span> </span><span>paste0</span><span>(</span><span>PackagesNeedingInstall</span><span>,</span><span>collapse</span><span> </span><span>=</span><span> </span><span>&quot;, &quot;</span><span>),</span><span> </span><span>&quot;\n&quot;</span><span>)</span><span>
</span><span>install.packages</span><span>(</span><span>PackagesNeedingInstall</span><span>,</span><span> </span><span>dependencies</span><span> </span><span>=</span><span> </span><span>T</span><span>,</span><span> </span><span>repos</span><span> </span><span>=</span><span> </span><span>&quot;http://cran-mirror.cs.uu.nl/&quot;</span><span>)</span><span>
</span><span>}</span><span>
</span><span># load packages into R:
</span><span> </span><span>x</span><span> </span><span>&lt;-</span><span> </span><span>suppressWarnings</span><span>(</span><span>lapply</span><span>(</span><span>packagelist</span><span>,</span><span> </span><span>require</span><span>,</span><span> </span><span>character.only</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>quietly</span><span> </span><span>=</span><span> </span><span>T</span><span>))</span><span>
</span><span>}</span><span>
</span><span>pkg_list</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;tidyverse&quot;</span><span>,</span><span>
</span><span>&quot;tidytext&quot;</span><span>,</span><span>
</span><span>&quot;stringr&quot;</span><span>,</span><span>
</span><span>&quot;rvest&quot;</span><span>,</span><span>
</span><span>&quot;lubridate&quot;</span><span>,</span><span>
</span><span>&quot;purrr&quot;</span><span>)</span><span>
</span><span>load_pkg</span><span>(</span><span>pkg_list</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## 
## Installing necesarry packages:  tidytext</code></pre>
</figure>
<p>The website I use for scraping has a page for each poet which lists the poems available – then it has a seperate page for each of the poems in the list, along with their dates. I have chosen this website specifically because having each poem dated is a must for my analysis. I run an initial scrape using <code>rvest</code> to get the names which are then cleaned so they can be used in the urls to get the poems and dates. Some names are not exactly the same as their url versions, so they need to be manually changed.</p>
<figure>
<pre><code data-lang="r"><span># page url
</span><span>poemNameUrl</span><span> </span><span>&lt;-</span><span> </span><span>&quot;http://www.poetry-archive.com/y/yeats_w_b.html&quot;</span><span>
</span><span># scrape for poem names
</span><span>poemName</span><span> </span><span>&lt;-</span><span> </span><span>poemNameUrl</span><span> </span><span>%>%</span><span>
</span><span>read_html</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>html_nodes</span><span>(</span><span>&quot;a font&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>html_text</span><span>()</span><span> </span><span># clean names
</span><span>poemName</span><span> </span><span>&lt;-</span><span> </span><span>poemName</span><span>[</span><span>1</span><span>:</span><span>50</span><span>]</span><span> </span><span>%>%</span><span>
</span><span>str_replace_all</span><span>(</span><span>pattern</span><span> </span><span>=</span><span> </span><span>&quot;\r&quot;</span><span>,</span><span> </span><span>replacement</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>str_replace_all</span><span>(</span><span>pattern</span><span> </span><span>=</span><span> </span><span>&quot;\n&quot;</span><span>,</span><span> </span><span>replacement</span><span> </span><span>=</span><span> </span><span>&quot; &quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>str_replace_all</span><span>(</span><span>pattern</span><span> </span><span>=</span><span> </span><span>&quot;[ ]{2}&quot;</span><span>,</span><span> </span><span>replacement</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>str_replace_all</span><span>(</span><span>pattern</span><span> </span><span>=</span><span> </span><span>&quot;[[:punct:]]&quot;</span><span>,</span><span> </span><span>replacement</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>tolower</span><span>()</span><span>
</span><span># hardcode 2 poem names to fit in with used url name
</span><span>poemName</span><span>[</span><span>9</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>&quot;he mourns for the change&quot;</span><span>
</span><span>poemName</span><span>[</span><span>24</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>&quot;the old men admiring themselves&quot;</span><span>
</span><span># display
</span><span>head</span><span>(</span><span>poemName</span><span>,</span><span> </span><span>10</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##  [1] &quot;at galway races&quot;                 &quot;the blessed&quot;                    
##  [3] &quot;the cap and bells&quot;               &quot;the cat and the moon&quot;           
##  [5] &quot;down by the salley gardens&quot;      &quot;dream of a blessed spirit&quot;      
##  [7] &quot;the falling of the leaves&quot;       &quot;he bids his beloved be at peace&quot;
##  [9] &quot;he mourns for the change&quot;        &quot;he remembers forgotten beauty&quot;</code></pre>
</figure>
<p>Next I need to scrape the website for the poems and their dates. To do this I use a function that takes a poem name – and again using <code>rvest</code> – scrapes for the given poem and its date, then cleans the text and waits before returning. The wait is so that I don’t send too many requests to the website in quick succession.</p>
<figure>
<pre><code data-lang="r"><span># function to scrape for poem and its date using poem names </span><span>GetPoemAndDate</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>poemName</span><span>)</span><span> </span><span>{</span><span>
</span><span># split string at empty space and unlist the result </span><span> </span><span>nameVec</span><span> </span><span>&lt;-</span><span> </span><span>unlist</span><span>(</span><span>str_split</span><span>(</span><span>poemName</span><span>,</span><span> </span><span>pattern</span><span> </span><span>=</span><span> </span><span>&quot; &quot;</span><span>))</span><span>
</span><span># use result in url </span><span> </span><span>url</span><span> </span><span>&lt;-</span><span> </span><span>str_c</span><span>(</span><span>&quot;http://www.poetry-archive.com/y/&quot;</span><span>,</span><span> </span><span>paste</span><span>(</span><span>nameVec</span><span>,</span><span> </span><span>collapse</span><span> </span><span>=</span><span> </span><span>&quot;_&quot;</span><span>),</span><span>
</span><span>&quot;.html&quot;</span><span>)</span><span> </span><span># scrape for poem and clean
</span><span> </span><span>poem</span><span> </span><span>&lt;-</span><span> </span><span>url</span><span> </span><span>%>%</span><span>
</span><span>read_html</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>html_nodes</span><span>(</span><span>&quot;dl&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>html_text</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>str_replace_all</span><span>(</span><span>pattern</span><span> </span><span>=</span><span> </span><span>&quot;\r&quot;</span><span>,</span><span> </span><span>replacement</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>str_replace_all</span><span>(</span><span>pattern</span><span> </span><span>=</span><span> </span><span>&quot;\n&quot;</span><span>,</span><span> </span><span>replacement</span><span> </span><span>=</span><span> </span><span>&quot; &quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>str_replace_all</span><span>(</span><span>pattern</span><span> </span><span>=</span><span> </span><span>&quot;[ ]{2}&quot;</span><span>,</span><span> </span><span>replacement</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>)</span><span>
</span><span># scrape for date </span><span> </span><span>date</span><span> </span><span>&lt;-</span><span> </span><span>url</span><span> </span><span>%>%</span><span>
</span><span>read_html</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>html_nodes</span><span>(</span><span>&quot;td font&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>html_text</span><span>()</span><span> </span><span># clean dates
</span><span> </span><span>date</span><span> </span><span>&lt;-</span><span> </span><span>date</span><span>[</span><span>3</span><span>]</span><span> </span><span>%>%</span><span>
</span><span>str_extract</span><span>(</span><span>pattern</span><span> </span><span>=</span><span> </span><span>&quot;[0-9]+&quot;</span><span>)</span><span>
</span><span># pause before function return
</span><span> </span><span>Sys.sleep</span><span>(</span><span>runif</span><span>(</span><span>1</span><span>,</span><span> </span><span>0</span><span>,</span><span> </span><span>1</span><span>))</span><span>
</span><span>return</span><span>(</span><span>list</span><span>(</span><span>poem</span><span> </span><span>=</span><span> </span><span>poem</span><span>,</span><span> </span><span>date</span><span> </span><span>=</span><span> </span><span>date</span><span>))</span><span>
</span><span>}</span></code></pre>
</figure>
<p>I then use the previous function in a for loop which loops through each poems name and scrapes for that particular poem and its date. With each loop I also add the data frame of poem and date to a list. Once the for loop has completed I <code>rbind</code> all the data frames together. A complete data frame of all poems and dates will be needed for the next step.</p>
<figure>
<pre><code data-lang="r"><span># get poems and dates
</span><span>poemDataFrame</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>()</span><span>
</span><span>count</span><span> </span><span>&lt;-</span><span> </span><span>1</span><span>
</span><span>for</span><span> </span><span>(</span><span>name</span><span> </span><span>in</span><span> </span><span>poemName</span><span>)</span><span> </span><span>{</span><span>
</span><span># get poem and date for given poem name
</span><span> </span><span>poemNameDate</span><span> </span><span>&lt;-</span><span> </span><span>GetPoemAndDate</span><span>(</span><span>name</span><span>)</span><span>
</span><span># create data frame of name and date and add it to list
</span><span> </span><span>poemDataFrame</span><span>[[</span><span>count</span><span>]]</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>poem</span><span> </span><span>=</span><span> </span><span>poemNameDate</span><span>$</span><span>poem</span><span>,</span><span> </span><span>date</span><span> </span><span>=</span><span> </span><span>poemNameDate</span><span>$</span><span>date</span><span>,</span><span>
</span><span>stringsAsFactors</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>count</span><span> </span><span>&lt;-</span><span> </span><span>count</span><span> </span><span>+</span><span> </span><span>1</span><span>
</span><span>}</span><span>
</span><span># rbind all poems and dates
</span><span>poemDataFrame</span><span> </span><span>&lt;-</span><span> </span><span>do.call</span><span>(</span><span>rbind</span><span>,</span><span> </span><span>poemDataFrame</span><span>)</span></code></pre>
</figure>
<p>I then combine the names from the original scrape, the dates, and the poems into a single data frame so they will be ready for me to work with. I also take a look at all the text, checking for and correcting any errors that could negatively affect my analysis. In this case I have found an error in a date.</p>
<figure>
<pre><code data-lang="r"><span># create data frame of names, poems and dates
</span><span>poemDataFrame</span><span> </span><span>&lt;-</span><span> </span><span>cbind</span><span>(</span><span>poemName</span><span>,</span><span> </span><span>poemDataFrame</span><span>)</span><span>
</span><span># hardcode single date with error
</span><span>poemDataFrame</span><span>$</span><span>date</span><span>[</span><span>40</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>&quot;1916&quot;</span></code></pre>
</figure>
<p>After this, we have a dataset of dimension 50×3 with column headings <code>poemName</code>, <code>poem</code> and <code>date</code>.</p>
<p>The next function I have written takes a row index, and using the combined data frame of poems, names and dates, along with <code>tidytext</code>, tokenizes the poem found at that index into single words. It then uses <code>anti_join</code> from <code>dplyr</code>, along with the <code>stop_words</code> dataset from <code>tidytext</code>, to create a tibble – this tibble will now be missing a list of stop words which will not be helpful in the analysis. This list of words for each poem is then used to create a sentiment score. This is done using <code>inner_join</code> and the <code>sentiment</code> dataset filtered on the bing lexicon, again from <code>tidytext</code>. The function calculates the sentiment score value for the poem depending on the number of negative and positive words in the list – negative words get a value of -1 and positive get a value of 1, these values are then summed and the result returned as a percentage of the number of words in that particular poem.</p>
<figure>
<pre><code data-lang="r"><span># function to analyse poems and return scores using bing lexicon
</span><span>bingAnalysePoems</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>i</span><span>)</span><span> </span><span>{</span><span>
</span><span># get poem a index i
</span><span> </span><span>poem</span><span> </span><span>&lt;-</span><span> </span><span>data_frame</span><span>(</span><span>poem</span><span> </span><span>=</span><span> </span><span>poemDataFrame</span><span>$</span><span>poem</span><span>[</span><span>i</span><span>])</span><span>
</span><span># tokenize into words
</span><span> </span><span>textTokenized</span><span> </span><span>&lt;-</span><span> </span><span>poem</span><span> </span><span>%>%</span><span>
</span><span>unnest_tokens</span><span>(</span><span>word</span><span>,</span><span> </span><span>poem</span><span>)</span><span>
</span><span># load stop words dataset
</span><span> </span><span>data</span><span>(</span><span>&quot;stop_words&quot;</span><span>)</span><span>
</span><span># anti join on stop words
</span><span> </span><span>tidyPoem</span><span> </span><span>&lt;-</span><span> </span><span>textTokenized</span><span> </span><span>%>%</span><span>
</span><span>anti_join</span><span>(</span><span>stop_words</span><span>)</span><span>
</span><span># filter on bing lexicon
</span><span> </span><span>bing</span><span> </span><span>&lt;-</span><span> </span><span>sentiments</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>lexicon</span><span> </span><span>==</span><span> </span><span>&quot;bing&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>-</span><span>score</span><span>)</span><span>
</span><span># join on bing to get whether words are positive or negative
</span><span> </span><span>poemSentiment</span><span> </span><span>&lt;-</span><span> </span><span>tidyPoem</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>bing</span><span>)</span><span> </span><span># get score for poem
</span><span> </span><span>poemSentiment</span><span> </span><span>&lt;-</span><span> </span><span>poemSentiment</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>score</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>poemSentiment</span><span>$</span><span>sentiment</span><span> </span><span>==</span><span> </span><span>&quot;positive&quot;</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>-1</span><span>))</span><span>
</span><span># get score as a percentage of total words in poem
</span><span> </span><span>finalScore</span><span> </span><span>&lt;-</span><span> </span><span>(</span><span>sum</span><span>(</span><span>poemSentiment</span><span>$</span><span>score</span><span>)</span><span>/</span><span>length</span><span>(</span><span>textTokenized</span><span>$</span><span>word</span><span>))</span><span>*</span><span>10</span><span>
</span><span>return</span><span>(</span><span>finalScore</span><span>)</span><span>
</span><span>}</span></code></pre>
</figure>
<p>Now, using the previous function and <code>sapply</code> I get the score for each poem and create a data frame with the poems score and its date. These scores will be used later to show the change in sentiment for each year.</p>
<figure>
<pre><code data-lang="r"><span># get scores
</span><span>scores</span><span> </span><span>&lt;-</span><span> </span><span>sapply</span><span>(</span><span>1</span><span>:</span><span>length</span><span>(</span><span>poemDataFrame</span><span>$</span><span>poem</span><span>),</span><span> </span><span>bingAnalysePoems</span><span>)</span><span>
</span><span># create data frame with data and scores
</span><span>dateAndScore</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>scores</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>date</span><span> </span><span>=</span><span> </span><span>year</span><span>(</span><span>ymd</span><span>(</span><span>str_c</span><span>(</span><span>poemDataFrame</span><span>$</span><span>date</span><span>,</span><span> </span><span>&quot;/01/01&quot;</span><span>))))</span><span>
</span><span># display
</span><span>head</span><span>(</span><span>dateAndScore</span><span>,</span><span> </span><span>10</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##         scores date
## 1  -0.20618557 1910
## 2  -0.03649635 1899
## 3   0.16528926 1899
## 4  -0.13157895 1919
## 5   0.32258065 1921
## 6   0.31250000 1921
## 7  -0.42253521 1889
## 8  -0.73394495 1899
## 9  -0.23255814 1899
## 10 -0.44025157 1899</code></pre>
</figure>
<p>The next function I have written takes a particular sentiment, which in this case can also be called an emotion, from a number of sentiments found in the NRC lexicon of the <code>sentiment</code> dataset; it then again creates a tibble of words not including the stop words, and using <code>semi_join</code> and the <code>sentiment</code> dataset filtered on the NRC lexicon and the particular sentiment given, returns the sum of the number of words relating to that sentiment. These sums will be used to show which emotions were high (or low) in a particular year.</p>
<figure>
<pre><code data-lang="r"><span># NRC analysis function
</span><span>nrcAnalysePoems</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>sent</span><span>)</span><span> </span><span>{</span><span>
</span><span># nrcPoem used from global environment
</span><span> </span><span>textTokenized</span><span> </span><span>&lt;-</span><span> </span><span>nrcPoem</span><span> </span><span>%>%</span><span>
</span><span>unnest_tokens</span><span>(</span><span>word</span><span>,</span><span> </span><span>poem</span><span>)</span><span>
</span><span># load stop words dataset
</span><span> </span><span>data</span><span>(</span><span>&quot;stop_words&quot;</span><span>)</span><span>
</span><span>tidyPoem</span><span> </span><span>&lt;-</span><span> </span><span>textTokenized</span><span> </span><span>%>%</span><span>
</span><span>anti_join</span><span>(</span><span>stop_words</span><span>)</span><span>
</span><span># filter on NRC lexicon and sentiment
</span><span> </span><span>nrcSentiment</span><span> </span><span>&lt;-</span><span> </span><span>sentiments</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>lexicon</span><span> </span><span>==</span><span> </span><span>&quot;nrc&quot;</span><span>,</span><span> </span><span>sentiment</span><span> </span><span>==</span><span> </span><span>sent</span><span>)</span><span>
</span><span># join on sentiment and count words
</span><span> </span><span>sentimentInPoem</span><span> </span><span>&lt;-</span><span> </span><span>tidyPoem</span><span> </span><span>%>%</span><span>
</span><span>semi_join</span><span>(</span><span>nrcSentiment</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>word</span><span>,</span><span> </span><span>sort</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span># return the sum of the counted words
</span><span> </span><span>return</span><span>(</span><span>sum</span><span>(</span><span>sentimentInPoem</span><span>$</span><span>n</span><span>))</span><span>
</span><span>}</span></code></pre>
</figure>
<p>Using the above function and <code>sapply</code>, I get the sums for each different sentiment in my sentiments list; this is done within a for loop, which loops through each poem, creating a data frame with each row consisting of a sentiment, the sum of words for that sentiment, the name of the poem the sentiment relates to, and the poems date. I then add this data frame to a list which will be used in my next step.</p>
<figure>
<pre><code data-lang="r"><span># list of used setiments found in NRC lexicon
</span><span>sentimentsVec</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;anger&quot;</span><span>,</span><span> </span><span>&quot;anticipation&quot;</span><span>,</span><span> </span><span>&quot;disgust&quot;</span><span>,</span><span> </span><span>&quot;fear&quot;</span><span>,</span><span>
</span><span>&quot;joy&quot;</span><span>,</span><span>&quot;sadness&quot;</span><span>,</span><span> </span><span>&quot;surprise&quot;</span><span>,</span><span> </span><span>&quot;trust&quot;</span><span>)</span><span>
</span><span># create empty list </span><span>nrcAnalysisDataFrame</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>()</span><span>
</span><span># get a frequency percetage for each sentiment found in the NRC lexicon </span><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>length</span><span>(</span><span>poemDataFrame</span><span>$</span><span>poem</span><span>))</span><span> </span><span>{</span><span>
</span><span># poem at index i - to be used in nrcAnalysePoems function environment
</span><span> </span><span>nrcPoem</span><span> </span><span>&lt;-</span><span> </span><span>data_frame</span><span>(</span><span>poem</span><span> </span><span>=</span><span> </span><span>poemDataFrame</span><span>$</span><span>poem</span><span>[</span><span>i</span><span>])</span><span>
</span><span># create data frame for each poem of all sentiment sums
</span><span> </span><span>nrcDataFrame</span><span> </span><span>&lt;-</span><span> </span><span>as.data.frame</span><span>(</span><span>sapply</span><span>(</span><span>sentimentsVec</span><span>,</span><span> </span><span>nrcAnalysePoems</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>rownames_to_column</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>sentiment</span><span> </span><span>=</span><span> </span><span>rowname</span><span>,</span><span> </span><span>value</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>name</span><span> </span><span>=</span><span> </span><span>poemDataFrame</span><span>$</span><span>poemName</span><span>[</span><span>i</span><span>],</span><span> </span><span>date</span><span> </span><span>=</span><span> </span><span>poemDataFrame</span><span>$</span><span>date</span><span>[</span><span>i</span><span>])</span><span>
</span><span># add data frame to list
</span><span> </span><span>nrcAnalysisDataFrame</span><span>[[</span><span>i</span><span>]]</span><span> </span><span>&lt;-</span><span> </span><span>nrcDataFrame</span><span>
</span><span>}</span></code></pre>
</figure>
<p>Once the for loop has completed I <code>rbind</code> all the data frames in the list so that I can work with all of them together as a whole.</p>
<figure>
<pre><code data-lang="r"><span># rbind list of all NRC sum values for all poems
</span><span>nrcAnalysisDataFrame</span><span> </span><span>&lt;-</span><span> </span><span>do.call</span><span>(</span><span>rbind</span><span>,</span><span> </span><span>nrcAnalysisDataFrame</span><span>)</span><span>
</span><span># display
</span><span>head</span><span>(</span><span>nrcAnalysisDataFrame</span><span>,</span><span> </span><span>10</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##       sentiment value            name date
## 1         anger     1 at galway races 1910
## 2  anticipation     4 at galway races 1910
## 3       disgust     2 at galway races 1910
## 4          fear     4 at galway races 1910
## 5           joy     2 at galway races 1910
## 6       sadness     4 at galway races 1910
## 7      surprise     2 at galway races 1910
## 8         trust     2 at galway races 1910
## 9         anger     0     the blessed 1899
## 10 anticipation     9     the blessed 1899</code></pre>
</figure>
<p>Now all the preparation code is done and we are ready to begin analyzing. For ease of use I have created a function for each poet to run all the code shown previously; the functions return both the bing and the NRC data frames, ready to be used. These functions are run but will not be shown here.</p>
<p>Firstly I will need to call each poet’s function and assign the returned data frames to variables so that I can work with them as needed.</p>
<figure>
<pre><code data-lang="r"><span>yeats</span><span> </span><span>&lt;-</span><span> </span><span>Yeats</span><span>()</span><span>
</span><span>eliot</span><span> </span><span>&lt;-</span><span> </span><span>Eliot</span><span>()</span><span>
</span><span>frost</span><span> </span><span>&lt;-</span><span> </span><span>Frost</span><span>()</span><span>
</span><span>wallace</span><span> </span><span>&lt;-</span><span> </span><span>Wallace</span><span>()</span><span>
</span><span>cummings</span><span> </span><span>&lt;-</span><span> </span><span>Cummings</span><span>()</span><span>
</span><span>lawrence</span><span> </span><span>&lt;-</span><span> </span><span>Lawrence</span><span>()</span><span>
</span><span>poets</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>yeats</span><span>,</span><span> </span><span>eliot</span><span>,</span><span> </span><span>frost</span><span>,</span><span> </span><span>wallace</span><span>,</span><span> </span><span>cummings</span><span>,</span><span> </span><span>lawrence</span><span>)</span></code></pre>
</figure>
<p>Having collected all the poems from our five poets, we plot the count of poems per year for each of our selected sample:</p>
<p><img alt="center" src="https://i2.wp.com/eighty20.github.io/figures/sentiment_blog_post/unnamed-chunk-3-1.png?w=456" data-recalc-dims="1"></p>
<p>From this we see that Yeats dominates the earlier period in our sample, while Frost plays a large part in the second half of the sample of text.</p>
<p>Next I will join all the bing scores for all poems using <code>rbind</code>, I do this so that I can work with all the data at once. I then group by date and get a mean score for each year in the complete data frame. This returns a tibble of scores and dates which I can now plot.</p>
<figure>
<pre><code data-lang="r"><span># rbind data frames and summarize
</span><span>completedataFramebing</span><span> </span><span>&lt;-</span><span> </span><span>poets</span><span> </span><span>%>%</span><span> </span><span>map</span><span>(</span><span>~</span><span>.x</span><span> </span><span>%>%</span><span> </span><span>.</span><span>[[</span><span>1</span><span>]]</span><span> </span><span>%>%</span><span> </span><span>data.frame</span><span>)</span><span> </span><span>%>%</span><span> </span><span>reduce</span><span>(</span><span>rbind</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>date</span><span> </span><span>>=</span><span> </span><span>1890</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>date</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarize</span><span>(</span><span>score</span><span> </span><span>=</span><span> </span><span>mean</span><span>(</span><span>scores</span><span>))</span><span> </span><span># display
</span><span>head</span><span>(</span><span>completedataFramebing</span><span>,</span><span> </span><span>10</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 10 × 2
##     date       score
##    </code></pre></figure>