Statistics Sunday: Tokenizing TextI recently started working my way through Text Mining with R: A Tidy Approach by Julia Silge and David Robinson. There are many interesting ways text analysis can be used, not only for research interests but for marketing and business insights. Today, I'd like to introduce one of the basic concepts necessary for conducting text analysis: tokens.In text analysis, a token is any kind of meaningful text unit that can be analyzed. Frequently, a token is a word and the process of tokenizing splits up the text into individual words and counts up how many times each word appears in the text. But a token could also be a phrase (such as each two-word combination present in a text, which is called a bi-gram), a sentence, a paragraph, even a whole chapter. Obviously, the size of the token you choose impacts what kind of analysis you can do. Generally, people choose smaller tokens, like words.Let's use R to download the text of a classic book (which I did previously in this post, but today, I'll do in an easier way) and tokenize it by word.Any text available in the Project Gutenberg repository can be downloaded, with header and footer information stripped out, with the guternbergr package.install.packages(&quot;gutenbergr&quot;)library(gutenbergr)The package comes with a dataset, called gutenberg_metadata, that contains a list, by ID, of all text available. Let's use The War of the Worlds by H.G. Wells as our target book. We can find our target book like this:library(tidyverse)## ── Attaching packages ────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──## ✔ ggplot2 2.2.1 ✔ purrr 0.2.4## ✔ tibble 1.4.2 ✔ dplyr 0.7.4## ✔ tidyr 0.8.0 ✔ stringr 1.3.0## ✔ readr 1.1.1 ✔ forcats 0.3.0## ── Conflicts ───────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──## ✖ dplyr::filter() masks stats::filter()## ✖ dplyr::lag() masks stats::lag()gutenberg_metadata %__%filter(title == &quot;The War of the Worlds&quot;)## # A tibble: 3 x 8## gutenberg_id title author gutenberg_autho… language gutenberg_books…## ## 1 36 The Wa… Wells, … 30 en Movie Books/Sci…## 2 8976 The Wa… Wells, … 30 en Movie Books/Sci…## 3 26291 The Wa… Wells, … 30 en ## # ... with 2 more variables: rights , has_text The ID for The War of the Worlds is 36. Now I can use that information to download the text of the book into a data frame, using the gutenbergr function, gutenberg_download.warofworlds