<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/rick-and-morty-and-tidy-data-principles-part-2/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://pachamaltese.github.io/2017-10-22_rick_and_morty_tidy_data_2.html"> Pachá (Batteries Included)</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h1>Motivation</h1>
<p>The <a rel="nofollow" href="http://pacha.hk/2017-10-13_rick_and_morty_tidy_data.html" target="_blank">first part</a> left an open door to analyze Rick and Morty contents using tf-idf, bag-of-words or some other NLP techniques. Here I’m also taking a lot of ideas from <a rel="nofollow" href="https://juliasilge.com/" target="_blank">Julia Silge</a>‘s blog. </p>
<p><img src="https://i1.wp.com/pachamaltese.github.io/images/2017-10-13_rick_and_morty_tidy_data/intro.jpg?w=456" data-recalc-dims="1"></p>
<p><em>Note: If some images appear too small on your screen you can open them in a new tab to show them in their original size.</em></p>
<h1>Term Frequency</h1>
<p>The most basic measure in natural language processing is obviously to just count words. This is a crude way of knowing what a document is about. The problem with counting words, however, is that there are some words (called stopwords) that are always too common, like “the” or “that”. So to create a more meaningful representation what people usually do is to compare the word counts observed in a document with that of a larger body of text.</p>
<p>Tf-idf is the frequency of a term adjusted for how rarely it is used. It is intended to measure how important a word is to a document in a collection (or corpus) of documents.</p>
<p>The inverse document frequency for any given term is defined as:
</p>
<div>$$idf(\text{term}) = \ln{\left(\frac{n_{\text{documents}}}{n_{\text{documents containing term}}}\right)}$$</div>
<p>
We can use tidy data principles to approach tf-idf analysis and use consistent, effective tools to quantify how important various terms are in a document that is part of a collection.</p>
<h1>What do Rick and Morty say?</h1>
<p>Let’s start by looking at Rick and Morty dialogues and examine first term frequency, then tf-idf. I’ll analyze this removing stopwords beforehand.</p>
<div>
<pre><span></span><span>if</span> <span>(</span><span>!</span><span>require</span><span>(</span><span>&quot;pacman&quot;</span><span>))</span> install.packages<span>(</span><span>&quot;pacman&quot;</span><span>)</span>
p_load<span>(</span>data.table<span>,</span>tidyr<span>,</span>stringr<span>,</span>tidytext<span>,</span>dplyr<span>,</span>janitor<span>,</span>ggplot2<span>,</span>viridis<span>,</span>ggstance<span>,</span>igraph<span>)</span>
p_load_gh<span>(</span><span>&quot;thomasp85/ggraph&quot;</span><span>,</span><span>&quot;dgrtwo/widyr&quot;</span><span>)</span>

rick_and_morty_subs <span>=</span> as_tibble<span>(</span>fread<span>(</span><span>&quot;2017-10-13_rick_and_morty_tidy_data/rick_and_morty_subs.csv&quot;</span><span>))</span>

rick_and_morty_subs_tidy <span>=</span> rick_and_morty_subs <span>%>%</span> 
  unnest_tokens<span>(</span>word<span>,</span>text<span>)</span> <span>%>%</span> 
  anti_join<span>(</span>stop_words<span>)</span> <span>%>%</span> 
  count<span>(</span>season<span>,</span> word<span>,</span> sort <span>=</span> <span>TRUE</span><span>)</span>

total_words <span>&lt;-</span> rick_and_morty_subs_tidy <span>%>%</span> group_by<span>(</span>season<span>)</span> <span>%>%</span> summarize<span>(</span>total <span>=</span> <span>sum</span><span>(</span>n<span>))</span>
season_words <span>&lt;-</span> left_join<span>(</span>rick_and_morty_subs_tidy<span>,</span> total_words<span>)</span>

season_words
</pre>
</div>
<div>
<pre><span></span># A tibble: 11,933 x 4
   season   word     n total
    </pre></div>