<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/edinbr-text-mining-with-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://blog.jumpingrivers.com/posts/2018/tidytext_edinbr_2018/"> R on The Jumping Rivers Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img src="https://i0.wp.com/blog.jumpingrivers.com/img/2018/tidytext_edinbr.jpg?w=400&ssl=1" data-recalc-dims="1"></p>
<p>During a very quick tour of Edinburgh (and in particular some distilleries), <a rel="nofollow" href="http://www.twitter.com/drob" target="_blank">Dave Robinson</a> (Tidytext fame), was able to drop by the <a rel="nofollow" href="http://edinbr.org/" target="_blank">Edinburgh R</a> meet-up group to give a very neat talk on tidy text. The first part of the talk set the scene</p>
<ul>
<li>What does does text mean?</li>
<li>Why make text tidy?</li>
<li>What sort of problems can you solve.</li>
</ul>
<p>This was a very neat overview of the topic and gave persuasive arguments around the idea of using a data frame for manipulating text. Most of the details are in Julie’s and his book on <a rel="nofollow" href="https://www.tidytextmining.com/" target="_blank">Text Mining with R</a>.</p>
<p>Personally I found the second part of his talk the most interesting, where Dave did an “off the cuff” demonstration of a tidy text analysis of the “Scottish play” (see <a rel="nofollow" href="https://www.youtube.com/watch?v=h--HR7PWfp0" target="_blank">Blackadder</a> for details on the “Scottish play”).</p>
<p>After loading a few packages</p>
<pre><code>library(&quot;gutenbergr&quot;)
library(&quot;tidyverse&quot;)
library(&quot;tidytext&quot;)
library(&quot;zoo&quot;)</code></pre>
<p>He downloaded the “Scottish Play” via the <a rel="nofollow" href="https://blog.jumpingrivers.com/posts/2018/tidytext_edinbr_2018/CRAN" target="_blank">Gutenbergr</a> package</p>
<pre><code>macbeth = gutenberg_works(title == &quot;Macbeth&quot;) %>%
  gutenberg_download()</code></pre>
<p>Then proceeded to generate a bar chart of the top <span>\(10\)</span> words (excluding stop words such as <em>and</em>, <em>to</em>), via</p>
<pre><code>macbeth %>%
  unnest_tokens(word, text) %>% # Make text tidy
  count(word, sort = TRUE) %>% # Count occurances
  anti_join(stop_words, by = &quot;word&quot;) %>% # Remove stop words
  head(10) %>% # Select top 10
  ggplot(aes(word, n)) + # Plot
  geom_col() </code></pre>
<p><img width="450" src="https://blog.jumpingrivers.com/posts/2018/2018-02-23-edinbr_files/figure-html/unnamed-chunk-4-1.svg"></p>
<p>The two key parts of this code are</p>
<ul>
<li><code>unnest_tokens()</code> – used to tidy the text;</li>
<li><code>anti_join()</code> – remove any <code>stop_words</code>.</li>
</ul>
<p>Since this analysis was “off the cuff”, Dave noticed that we could easily extract the speaker. This is clearly something you would want to store and can be achieved via a some <code>mutate()</code> magic</p>
<pre><code>speaker_words = macbeth %>%
  mutate(is_speaker = str_detect(text, &quot;^[A-Z ]+\\.$&quot;), # Detect capital letters
         speaker = ifelse(is_speaker, text, NA),
         speaker = na.locf(speaker, na.rm = FALSE))</code></pre>
<p>The <code>str_detect()</code> uses a simple regular expression to determine if the text are capital letters (theyby indicating a scene). Any expression of length zero is replaced, by a missing value <code>NA</code>. Before finishing with the <strong>zoo</strong> <code>na.locf()</code> function to carry the last observation forward, thereby filling the blanks.</p>
<p>The resulting tibble is then cleaned using</p>
<pre><code>speaker_words = speaker_words %>%
  filter(!is_speaker, !is.na(speaker)) %>%
  select(-is_speaker, -gutenberg_id) %>%
  unnest_tokens(word, text) %>%
  anti_join(stop_words, by = &quot;word&quot;) </code></pre>
<p>A further bit of analysis gives</p>
<pre><code>speaker_words %>%
  count(speaker, word, sort = TRUE) %>%
  bind_tf_idf(word, speaker, n) %>%
  arrange(desc(tf_idf)) %>%
  filter(n >= 5)</code></pre>
<pre><code>## # A tibble: 107 x 6
##    speaker       word         n      tf   idf tf_idf
##    </code></pre>