<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/examining-the-arc-of-100000-stories-a-tidy-analysis/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://varianceexplained.org/r/tidytext-plots/"> Variance Explained</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I recently came across a <a target="_blank" rel="nofollow" href="https://github.com/markriedl/WikiPlots">great natural language dataset</a> from <a target="_blank" rel="nofollow" href="https://research.cc.gatech.edu/inc/mark-riedl">Mark Riedel</a>: 112,000 plots of stories downloaded from English language Wikipedia. This includes books, movies, TV episodes, video games- anything that has a <strong>Plot</strong> section on a Wikipedia page.</p>
<p>This offers a great opportunity to analyze story structure quantitatively. In this post I’ll do a simple analysis, examining what words tend to occur at particular points within a story, including words that characterize the beginning, middle, or end.</p>
<p>As I usually do for text analysis, I’ll be using the <a target="_blank" rel="nofollow" href="https://github.com/juliasilge/tidytext">tidytext package</a> Julia Silge and I developed last year. To learn more about analyzing datasets like this, see our online book <a target="_blank" rel="nofollow" href="http://tidytextmining.com/">Text Mining with R: A Tidy Approach</a>, soon to be <a target="_blank" rel="nofollow" href="https://www.amazon.com/Text-Mining-R-tidy-approach/dp/1491981652">published by O’Reilly</a>. I’ll provide code for the text mining sections so you can follow along. I don’t show the code for most of the visualizations to keep the post concise, but as with all of my posts the code can be found <a target="_blank" rel="nofollow" href="https://github.com/dgrtwo/dgrtwo.github.com/tree/master/_R/2017-04-26-tidytext-plots.Rmd">here on GitHub</a>.</p>
<h3>Setup</h3>
<p>I downloaded and unzipped the <a target="_blank" rel="nofollow" href="https://gtvault-my.sharepoint.com/personal/mriedl3_gatech_edu/_layouts/15/guestaccess.aspx?docid=09511cca101bd45859859fe1765c7a84d&authkey=Af3ym3JQ-UIXbyKYwyEe0hU&expiration=2017-06-19T13%3A37%3A21.000Z">plots.zip</a> file from the link on the GitHub repository. We then read the files into R, and combined them using dplyr.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>readr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span># Plots and titles are in separate files
</span><span>plots</span><span> </span><span>&lt;-</span><span> </span><span>read_lines</span><span>(</span><span>&quot;~/Downloads/plots/plots&quot;</span><span>,</span><span> </span><span>progress</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>titles</span><span> </span><span>&lt;-</span><span> </span><span>read_lines</span><span>(</span><span>&quot;~/Downloads/plots/titles&quot;</span><span>,</span><span> </span><span>progress</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span># Each story ends with an &lt;EOS> line
</span><span>plot_text</span><span> </span><span>&lt;-</span><span> </span><span>data_frame</span><span>(</span><span>text</span><span> </span><span>=</span><span> </span><span>plots</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>story_number</span><span> </span><span>=</span><span> </span><span>cumsum</span><span>(</span><span>text</span><span> </span><span>==</span><span> </span><span>&quot;&lt;EOS>&quot;</span><span>)</span><span> </span><span>+</span><span> </span><span>1</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>titles</span><span>[</span><span>story_number</span><span>])</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>text</span><span> </span><span>!=</span><span> </span><span>&quot;&lt;EOS>&quot;</span><span>)</span></code></pre>
</figure>
<p>We can then use the <a target="_blank" rel="nofollow" href="http://tidytextmining.com/">tidytext</a> package to unnest the plots into a tidy format, with one token per line.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>tidytext</span><span>)</span><span>
</span><span>plot_words</span><span> </span><span>&lt;-</span><span> </span><span>plot_text</span><span> </span><span>%>%</span><span>
</span><span>unnest_tokens</span><span>(</span><span>word</span><span>,</span><span> </span><span>text</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>plot_words</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 40,330,086 × 3
##    story_number       title    word
##           &lt;dbl>       &lt;chr>   &lt;chr>
## 1             1 Animal Farm     old
## 2             1 Animal Farm   major
## 3             1 Animal Farm     the
## 4             1 Animal Farm     old
## 5             1 Animal Farm    boar
## 6             1 Animal Farm      on
## 7             1 Animal Farm     the
## 8             1 Animal Farm   manor
## 9             1 Animal Farm    farm
## 10            1 Animal Farm summons
## # ... with 40,330,076 more rows</code></pre>
</figure>
<p>This dataset contains over 40 million words across 112,000 stories.</p>
<h3>Words at the beginning or end of stories</h3>
<p>Joseph Campbell introduced the idea of a <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Hero%27s_journey">“hero’s journey”</a>, that every story follows the same structure. Whether or not you buy into his theory, you can agree it’d be surprising if a plot started with a climactic fight, or ended by introducing new characters.</p>
<p>That structure is reflected quantitatively in what words are used at which point in a story: there are some words you’d expect would appear at the start, and others at the end.</p>
<p>As a simple measure of where a word occurs within a plot, we’ll record the <strong>median</strong> position of each word, along with the number of times it appears.</p>
<figure>
<pre><code data-lang="r"><span>word_averages</span><span> </span><span>&lt;-</span><span> </span><span>plot_words</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>title</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>word_position</span><span> </span><span>=</span><span> </span><span>row_number</span><span>()</span><span> </span><span>/</span><span> </span><span>n</span><span>())</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>word</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarize</span><span>(</span><span>median_position</span><span> </span><span>=</span><span> </span><span>median</span><span>(</span><span>word_position</span><span>),</span><span>
</span><span>number</span><span> </span><span>=</span><span> </span><span>n</span><span>())</span></code></pre>
</figure>
<p>We’re not interested in rare words that occurred in only a few plot descriptions, so we’ll filter for ones occurring at least 2,500 times.</p>
<figure>
<pre><code data-lang="r"><span>word_averages</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>number</span><span> </span><span>>=</span><span> </span><span>2500</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>median_position</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 1,640 × 3
##           word median_position number
##          &lt;chr>           &lt;dbl>  &lt;int>
## 1    fictional       0.1193618   2688
## 2         year       0.2013554  18692
## 3  protagonist       0.2029450   3222
## 4      century       0.2096774   3583
## 5      wealthy       0.2356817   5686
## 6        opens       0.2408638   7319
## 7   california       0.2423856   2656
## 8      angeles       0.2580645   2889
## 9          los       0.2661747   3110
## 10     student       0.2692308   6961
## # ... with 1,630 more rows</code></pre>
</figure>
<p>For example, we can see that the word “fictional” was used about 2700 times, and that half of its uses were before the 12% mark of the story: it’s highly shifted towards the beginning.</p>
<p>What were were the words most shifted towards the beginning or end of a story?</p>
<p><img alt="center" src="https://i2.wp.com/varianceexplained.org/figs/2017-04-26-tidytext-plots/word_average_plot-1.png?w=456" data-recalc-dims="1"></p>
<p>The words shifted towards the beginning of a story tend to describe a setting: “The story <strong>opens</strong> on the <strong>protagonist</strong>, a <strong>wealthy</strong> <strong>young</strong> 19th <strong>century</strong> <strong>student</strong> <strong>recently</strong> graduated from the <strong>fictional University College</strong> in <strong>Los Angeles, California.</strong>”. Most are therefore nouns and adjectives that can be used to specify and describe a person, location, or time period.</p>
<p>In contrast, the words shifted towards the end of a story are packed with excitement! There are a few housekeeping terms you’d expect to find at the end of a plot description (“ending”, “final”), but also a number of verbs suggestive of a climax. “The hero <strong>shoots</strong> the villain and <strong>rushes</strong> to the heroine, and <strong>apologizes</strong>. The two <strong>reunited</strong>, they <strong>kiss</strong>.”</p>
<h3>Visualizing trends of words</h3>
<p>The median gives us a useful summary statistic of where a word appears within a story, but let’s take a closer look at a few. First we’ll divide each story into deciles (first 10%, second 10%, etc), and count the appearances of each word within each decile.</p>
<figure>
<pre><code data-lang="r"><span>decile_counts</span><span> </span><span>&lt;-</span><span> </span><span>plot_words</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>title</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>word_position</span><span> </span><span>=</span><span> </span><span>row_number</span><span>()</span><span> </span><span>/</span><span> </span><span>n</span><span>())</span><span> </span><span>%>%</span><span>
</span><span>ungroup</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>decile</span><span> </span><span>=</span><span> </span><span>ceiling</span><span>(</span><span>word_position</span><span> </span><span>*</span><span> </span><span>10</span><span>)</span><span> </span><span>/</span><span> </span><span>10</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>decile</span><span>,</span><span> </span><span>word</span><span>)</span></code></pre>
</figure>
<p>This lets us visualize the frequency of a word across the length of plot descriptions. We may want to look at the most extreme start/end ones:</p>
<p><img alt="center" src="https://i0.wp.com/varianceexplained.org/figs/2017-04-26-tidytext-plots/progress_counts_filtered-1.png?w=456" data-recalc-dims="1"></p>
<p>No word happens <em>exclusively</em> at the start or end of a story. Some, like “happily”, remain steady throughout and then spike up at the end (“lived happily ever after”). Other words, like “truth”, or “apologizes”, show a constant rise in frequency over the course of the story, which makes sense: a character generally wouldn’t “apologize” or “realize the truth” right at the start of the story. Similarly, words that establish settings like “wealthy” become steadily rarer the course of the story, as it becomes less likely the plot will introduce new characters.</p>
<p>One interesting feature of the above graph is that while most words peak either at the beginning or end, words like “grabs”, “rushes”, and “shoots” were most common at the 90% point. This might represent the <strong>climax</strong> of the story.</p>
<h3>Words appearing in the middle of a story</h3>
<p>Inspired by this examination of words that might occur at a climax, let’s consider what words were most likely to appear at particular points in the middle, rather than being shifted towards the beginning or end.</p>
<figure>
<pre><code data-lang="r"><span>peak_decile</span><span> </span><span>&lt;-</span><span> </span><span>decile_counts</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>word_averages</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;word&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>number</span><span> </span><span>>=</span><span> </span><span>2500</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>transmute</span><span>(</span><span>peak_decile</span><span> </span><span>=</span><span> </span><span>decile</span><span>,</span><span>
</span><span>word</span><span>,</span><span>
</span><span>number</span><span>,</span><span>
</span><span>fraction_peak</span><span> </span><span>=</span><span> </span><span>n</span><span> </span><span>/</span><span> </span><span>number</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>desc</span><span>(</span><span>fraction_peak</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>distinct</span><span>(</span><span>word</span><span>,</span><span> </span><span>.keep_all</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>peak_decile</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 1,640 × 4
##    peak_decile        word number fraction_peak
##          &lt;dbl>       &lt;chr>  &lt;int>         &lt;dbl>
## 1          0.1   fictional   2688     0.4676339
## 2          1.0     happily   2895     0.4601036
## 3          1.0        ends  18523     0.4036603
## 4          0.1       opens   7319     0.3913103
## 5          1.0    reunited   2660     0.3853383
## 6          0.1 protagonist   3222     0.3764742
## 7          1.0      ending   4181     0.3721598
## 8          0.1        year  18692     0.3578536
## 9          0.1     century   3583     0.3530561
## 10         0.1       story  37248     0.3257356
## # ... with 1,630 more rows</code></pre>
</figure>
<p>Each decile of the book (the start, the end, the 30% point, etc) therefore has some some words that peak within it. What words were most characteristic of particular deciles?</p>
<p><img alt="center" src="https://i2.wp.com/varianceexplained.org/figs/2017-04-26-tidytext-plots/word_decile_percent-1.png?w=456" data-recalc-dims="1"></p>
<p>We see that the words in the start and the end are the most specific to their particular deciles: for example, almost half of the occurrences of the word “fictional” occurred in the first 10% of the story. The middle sections have words that are more spread out (having, say, 14% of their occurrences in that section rather than the expected 10%), but they still are words that make sense in the story structure.</p>
<p>Let’s visualize the full trend for the words overrepreseted at each point.</p>
<p><img alt="center" src="https://i0.wp.com/varianceexplained.org/figs/2017-04-26-tidytext-plots/sparklines-1.png?w=456" data-recalc-dims="1"></p>
<p>Try reading the 24 word story laid out by the subgraph titles. Our protagonist is “attracted”, then “suspicious”, followed by “jealous”, “drunk”, and ultimately “furious”. A shame that once they “confront” the problem, they run into a “trap” and are “wounded”. If you ignore the repetitive words and the lack of syntax, you can see the rising tension of a story just in these sparklines.</p>
<h3>Sentiment analysis</h3>
<p>As one more confirmation of our hypothesis about rising tension and conflict within a story, we can use <a target="_blank" rel="nofollow" href="http://tidytextmining.com/sentiment.html">sentiment analysis</a> to find the average sentiment within each piece of a story.</p>
<figure>
<pre><code data-lang="r"><span>decile_counts</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>get_sentiments</span><span>(</span><span>&quot;afinn&quot;</span><span>),</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;word&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>decile</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarize</span><span>(</span><span>score</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>score</span><span> </span><span>*</span><span> </span><span>n</span><span>)</span><span> </span><span>/</span><span> </span><span>sum</span><span>(</span><span>n</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>decile</span><span>,</span><span> </span><span>score</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_x_continuous</span><span>(</span><span>labels</span><span> </span><span>=</span><span> </span><span>percent_format</span><span>())</span><span> </span><span>+</span><span>
</span><span>expand_limits</span><span>(</span><span>y</span><span> </span><span>=</span><span> </span><span>0</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Position within a story&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Average AFINN sentiment score&quot;</span><span>)</span></code></pre>
</figure>
<p><img alt="center" src="https://i2.wp.com/varianceexplained.org/figs/2017-04-26-tidytext-plots/sentiment_story_plot-1.png?w=456" data-recalc-dims="1"></p>
<p>Plot descriptions have a negative average AFINN score at all points in the story (which makes sense, since stories focus on conflict. But it might start with a relatively peaceful beginning, before the conflict increases over the course of the plot, until it hits a maximum around the climax, 80-90%. It’s then often followed by a resolution, which contains words like “happily”, “rescues”, and “reunited” that return it to a higher sentiment score.</p>
<p>In short, if we had to summarize the <em>average</em> story that humans tell, it would go something like <strong>Things get worse and worse until at the last minute they get better.</strong></p>
<h3>To be continued</h3>
<p>This was a pretty simple analysis of story arcs (for a more in-depth example, see the <a target="_blank" rel="nofollow" href="https://www.theatlantic.com/technology/archive/2016/07/the-six-main-arcs-in-storytelling-identified-by-a-computer/490733/">research described here</a>), and it doesn’t tell us too much we wouldn’t have been able to guess.<br>
(Except perhaps that characters are most likely to be drunk right in the middle of a story. How can we monetize that insight?)</p>
<p>What I like about this approach is how quickly you can gain insights with simple quantitative methods (counting, taking the median) applied to a large text dataset. In future posts, I’ll be diving deeper into these plots and showing what else we can learn.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://varianceexplained.org/r/tidytext-plots/"> Variance Explained</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/Z7GDIJjJpgU"></a></div>