Motivation
The first and second part of this analysis gave the idea that I did too much scrapping and processing and that deserves more analysis to use that information well. In this third and final part I'm also taking a lot of ideas from Julia Silge's blog. In the GitHub repo of this project you shall find not just Rick and Morty processed subs, but also for Archer, Bojack Horseman, Gravity Falls and Stranger Things. Why? In post post I'm gonna compare the different shows.
Note: If some images appear too small on your screen you can open them in a new tab to show them in their original size.
Word Frequencies
Comparing frequencies across different shows can tell us how similar Gravity Falls, for example, is similar to Rick and Morty. I'll use the subtitles from different shows that I scraped using the same procedure I did with Rick and Morty.
if (!require(&quot;pacman&quot;)) install.packages(&quot;pacman&quot;)
p_load(data.table,tidyr,tidytext,dplyr,ggplot2,viridis,ggstance,stringr,scales)
p_load_gh(&quot;dgrtwo/widyr&quot;)
rick_and_morty_subs = as_tibble(fread(&quot;2017-10-13_rick_and_morty_tidy_data/rick_and_morty_subs.csv&quot;))
archer_subs = as_tibble(fread(&quot;2017-10-13_rick_and_morty_tidy_data/archer_subs.csv&quot;))
bojack_horseman_subs = as_tibble(fread(&quot;2017-10-13_rick_and_morty_tidy_data/bojack_horseman_subs.csv&quot;))
gravity_falls_subs = as_tibble(fread(&quot;2017-10-13_rick_and_morty_tidy_data/gravity_falls_subs.csv&quot;))
stranger_things_subs = as_tibble(fread(&quot;2017-10-13_rick_and_morty_tidy_data/stranger_things_subs.csv&quot;))
rick_and_morty_subs_tidy = rick_and_morty_subs %__% unnest_tokens(word,text) %__% anti_join(stop_words)
archer_subs_tidy = archer_subs %__% unnest_tokens(word,text) %__% anti_join(stop_words)
bojack_horseman_subs_tidy = bojack_horseman_subs %__% unnest_tokens(word,text) %__% anti_join(stop_words)
gravity_falls_subs_tidy = gravity_falls_subs %__% unnest_tokens(word,text) %__% anti_join(stop_words)
stranger_things_subs_tidy = stranger_things_subs %__% unnest_tokens(word,text) %__% anti_join(stop_words)
With this processing we can compare frequencies across different shows. Here's an example of the top ten words for each show:
bind_cols(rick_and_morty_subs_tidy %__% count(word, sort = TRUE) %__% filter(row_number() % count(word, sort = TRUE) %__% filter(row_number() % count(word, sort = TRUE) %__% filter(row_number() % count(word, sort = TRUE) %__% filter(row_number() % count(word, sort = TRUE) %__% filter(row_number() % setNames(., c(&quot;rm_word&quot;,&quot;rm_n&quot;,&quot;a_word&quot;,&quot;a_n&quot;,&quot;bh_word&quot;,&quot;bh_n&quot;,&quot;gf_word&quot;,&quot;gf_n&quot;,&quot;st_word&quot;,&quot;st_n&quot;))
# A tibble: 10 x 10
rm_word rm_n a_word a_n bh_word bh_n gf_word gf_n st_word st_n
1 morty 1898 archer 4548 bojack 956 mabel 457 yeah 485
2 rick 1691 lana 2800 yeah 704 hey 453 hey 318
3 jerry 646 yeah 1478 hey 575 ha 416 mike 271
4 yeah 484 cyril 1473 gonna 522 stan 369 sighs 262
5 gonna 421 malory 1462 time 451 dipper 347 uh 189
6 summer 409 pam 1300 uh 382 gonna 345 dustin 179
7 hey 391 god 878 na 373 time 314 lucas 173
8 uh 331 wait 846 diane 345 yeah 293 gonna 172
9 time 319 uh 835 todd 339 uh 265 joyce 161
10 beth 301 gonna 748 love 309 guys 244 mom 157
There are common words such as &quot;yeah&quot; for example.
Now I'll combine the frequencies of all the shows and I'll plot the top 50 frequencies to see similitudes with Rick and Morty:
tidy_others = bind_rows(mutate(archer_subs_tidy, show = &quot;Archer&quot;),
mutate(bojack_horseman_subs_tidy, show = &quot;Bojack Horseman&quot;),
mutate(gravity_falls_subs_tidy, show = &quot;Gravity Falls&quot;),
mutate(stranger_things_subs_tidy, show = &quot;Stranger Things&quot;))
frequency = tidy_others %__%
mutate(word = str_extract(word, &quot;[a-z]+&quot;)) %__%
count(show, word) %__%
rename(other = n) %__%
inner_join(count(rick_and_morty_subs_tidy, word)) %__%
rename(rick_and_morty = n) %__%
mutate(other = other / sum(other),
rick_and_morty = rick_and_morty / sum(rick_and_morty)) %__%
ungroup() frequency_top_50 = frequency %__% group_by(show) %__% arrange(-other,-rick_and_morty) %__% filter(row_number()