Motivation
The first part left an open door to analyze Rick and Morty contents using tf-idf, bag-of-words or some other NLP techniques. Here I'm also taking a lot of ideas from Julia Silge's blog. Note: If some images appear too small on your screen you can open them in a new tab to show them in their original size.
Term Frequency
The most basic measure in natural language processing is obviously to just count words. This is a crude way of knowing what a document is about. The problem with counting words, however, is that there are some words (called stopwords) that are always too common, like “the” or “that”. So to create a more meaningful representation what people usually do is to compare the word counts observed in a document with that of a larger body of text.
Tf-idf is the frequency of a term adjusted for how rarely it is used. It is intended to measure how important a word is to a document in a collection (or corpus) of documents.
The inverse document frequency for any given term is defined as:
$$idf(\text{term}) = \ln{\left(\frac{n_{\text{documents}}}{n_{\text{documents containing term}}}\right)}$$
We can use tidy data principles to approach tf-idf analysis and use consistent, effective tools to quantify how important various terms are in a document that is part of a collection.
What do Rick and Morty say?
Let’s start by looking at Rick and Morty dialogues and examine first term frequency, then tf-idf. I'll analyze this removing stopwords beforehand.
if (!require(&quot;pacman&quot;)) install.packages(&quot;pacman&quot;)
p_load(data.table,tidyr,stringr,tidytext,dplyr,janitor,ggplot2,viridis,ggstance,igraph)
p_load_gh(&quot;thomasp85/ggraph&quot;,&quot;dgrtwo/widyr&quot;)
rick_and_morty_subs = as_tibble(fread(&quot;2017-10-13_rick_and_morty_tidy_data/rick_and_morty_subs.csv&quot;))
rick_and_morty_subs_tidy = rick_and_morty_subs %__% unnest_tokens(word,text) %__% anti_join(stop_words) %__% count(season, word, sort = TRUE)
total_words % group_by(season) %__% summarize(total = sum(n))
season_words %
select(-total) %__%
arrange(desc(tf_idf)) # A tibble: 11,933 x 6
season word n tf idf tf_idf
1 S03 pickle 43 0.003305658 1.098612 0.003631637
2 S02 unity 32 0.002494349 1.098612 0.002740322
3 S01 meeseeks 44 0.002270265 1.098612 0.002494141
4 S03 vindicators 26 0.001998770 1.098612 0.002195873
5 S02 purge 25 0.001948710 1.098612 0.002140877
6 S01 flu 32 0.001651102 1.098612 0.001813921
7 S01 crystals 30 0.001547908 1.098612 0.001700550
8 S03 tommy 20 0.001537515 1.098612 0.001689133
9 S02 deer 19 0.001481020 1.098612 0.001627066
10 S03 noob 18 0.001383764 1.098612 0.001520220
# ... with 11,923 more rows
Curious about &quot;pickle&quot;? You'd better watch Picle Rick episode if you don't get why &quot;pickle&quot; is the highest tf-idf ranked term. &quot;Vindicator&quot; is another term that is concentrated in one episode where Vindicators appear. There's even an episode where flu is a part of the central problem and Rick has to use his mind to try to solve a flu of of control because of his inventions.
Some of the values for idf are the same for different terms because there are 6 documents in this corpus and we are seeing the numerical value for ln(6/1), ln(6/2), etc. Let’s look at a visualization for these high tf-idf words.
plot_tfidf %
arrange(desc(tf_idf)) %__%
mutate(word = factor(word, levels = rev(unique(word))))
ggplot(plot_tfidf[1:20,], aes(tf_idf, word, fill = season, alpha = tf_idf)) +
geom_barh(stat = &quot;identity&quot;) +
labs(title = &quot;Highest tf-idf words in Rick and Morty' Seasons&quot;,
y = NULL, x = &quot;tf-idf&quot;) +
theme_minimal(base_size = 13) +
scale_alpha_continuous(range = c(0.6, 1), guide = FALSE) +
scale_x_continuous(expand=c(0,0)) +
scale_fill_viridis(end = 0.85, discrete=TRUE) +
theme(legend.title=element_blank()) +
theme(legend.justification=c(1,0), legend.position=c(1,0))
Let’s look at the seasons individually.
plot_tfidf % group_by(season) %__% top_n(15) %__% ungroup()
ggplot(plot_tfidf, aes(tf_idf, word, fill = season, alpha = tf_idf)) +
geom_barh(stat = &quot;identity&quot;, show.legend = FALSE) +
labs(title = &quot;Highest tf-idf words in Rick and Morty' Seasons&quot;,
y = NULL, x = &quot;tf-idf&quot;) +
facet_wrap(~season, nrow = 3, scales = &quot;free&quot;) +
theme_minimal(base_size = 13) +
scale_alpha_continuous(range = c(0.6, 1)) +
scale_x_continuous(expand=c(0,0)) +
scale_fill_viridis(end = 0.85, discrete=TRUE) +
theme(strip.text=element_text(hjust=0)) +
theme(strip.text = element_text(face = &quot;italic&quot;))