Last week I saw Chris Moody’s post on the Stitch Fix blog about calculating word vectors from a corpus of text using word counts and matrix factorization, and I was so excited! This blog post illustrates how to implement that approach to find word vector representations in R using tidy data principles and sparse matrices.
Word vectors, or word embeddings, are typically calculated using neural networks; that is what word2vec is. (GloVe embeddings are trained a little differently than word2vec.) By contrast, the approach from Chris’s post that I’m implementing here uses only counting and some linear algebra. Deep learning is great, but I am super excited about this approach because it allows practitioners to find word vectors for their own collections of text (no need to rely on pre-trained vectors) using familiar techniques that are not difficult to understand. And it doesn’t take too long computationally!
Getting some data
Let’s download half a million observations from… the Hacker News corpus.
via GIPHY
I know, right? But it’s the dataset that Chris uses in his blog post and it gives me an opportunity to use the bigrquery package for the first time.
library(bigrquery)
library(tidyverse)
project %
mutate(text = str_replace_all(text, &quot;'|&quot;|/&quot;, &quot;'&quot;), ## weird encoding
text = str_replace_all(text, &quot;&quot;, &quot; &quot;), ## links text = str_replace_all(text, &quot;>|&lt;&quot;, &quot; &quot;), ## html yuck
text = str_replace_all(text, &quot;]*__&quot;, &quot; &quot;), ## mmmmm, more html yuck
postID = row_number())
Unigram probabilities
First, let’s calculate the unigram probabilities, how often we see each word in this corpus. This is straightforward using unnest_tokens() from the tidytext package and then just count() and mutate() from dplyr.
library(tidytext)
unigram_probs %
unnest_tokens(word, text) %__%
count(word, sort = TRUE) %__%
mutate(p = n / sum(n))
unigram_probs
## # A tibble: 318,503 x 3
## word n p
## ## 1 the 1101045 0.04190098
## 2 to 761658 0.02898539
## 3 a 669067 0.02546178
## 4 of 541937 0.02062376
## 5 and 532592 0.02026813
## 6 is 431841 0.01643399
## 7 that 406506 0.01546985
## 8 i 401545 0.01528105
## 9 in 368045 0.01400619
## 10 it 333441 0.01268931
## # ... with 318,493 more rows
Skipgram probabilities
Next, we need to calculate the skipgram probabilities, how often we find each word near each other word. We do this by defining a fixed-size moving window that centers around each word. Do we see word1 and word2 together within this window? I take the approach here of using unnest_tokens() once with token = &quot;ngrams&quot; to find all the windows I need, then using unnest_tokens() again to tidy these n-grams. After that, I can use pairwise_count() from the widyr package to count up cooccuring pairs within each n-gram/sliding window.
I’m not sure what the ideal value for window size is here for the skipgrams. This value determines the sliding window that we move through the text, counting up bigrams that we find within the window. When this window is bigger, the process of counting skipgrams takes longer, obviously. I experimented a bit and windows of 8 words seem to work pretty well. Probably more work needed here! I’d be happy to be pointed to more resources on this topic.
Finding all the skipgrams is a computationally expensive part of this process. Not something that just runs instantly!
library(widyr)
tidy_skipgrams %
unnest_tokens(ngram, text, token = &quot;ngrams&quot;, n = 8) %__%
mutate(ngramID = row_number()) %__% unite(skipgramID, postID, ngramID) %__%
unnest_tokens(word, ngram)
tidy_skipgrams
## # A tibble: 183,434,296 x 2
## skipgramID word
## ## 1 1_1 i
## 2 1_1 bet
## 3 1_1 taking
## 4 1_1 a
## 5 1_1 few
## 6 1_1 months
## 7 1_1 off
## 8 1_1 from
## 9 1_2 bet
## 10 1_2 taking
## # ... with 183,434,286 more rows
skipgram_probs %
pairwise_count(word, skipgramID, diag = TRUE, sort = TRUE) %__%
mutate(p = n / sum(n))
Normalized skipgram probability
We now know how often words occur on their own, and how often words occur together with other words. We can calculate which words occurred together more often than expected based on how often they occurred on their own. When this number is high (greater than 1), the two words are associated with each other, likely to occur together. When this number is low (less than 1), the two words are not associated with each other, unlikely to occur together.
normalized_prob %
filter(n __ 20) %__%
rename(word1 = item1, word2 = item2) %__%
left_join(unigram_probs %__%
select(word1 = word, p1 = p),
by = &quot;word1&quot;) %__%
left_join(unigram_probs %__%
select(word2 = word, p2 = p),
by = &quot;word2&quot;) %__%
mutate(p_together = p / p1 / p2)
What are the words most associated with Facebook on Hacker News?
normalized_prob %__% filter(word1 == &quot;facebook&quot;) %__%
arrange(-p_together)
## # A tibble: 1,746 x 7
## word1 word2 n p p1 p2 p_together
## ## 1 facebook facebook 52883 3.757919e-05 0.0003340144 3.340144e-04 336.83488
## 2 facebook messenger 348 2.472923e-07 0.0003340144 1.122642e-05 65.94840
## 3 facebook zuckerburg 23 1.634403e-08 0.0003340144 8.752799e-07 55.90454
## 4 facebook statuses 40 2.842440e-08 0.0003340144 1.522226e-06 55.90454
## 5 facebook myspace 330 2.345013e-07 0.0003340144 1.331948e-05 52.70999
## 6 facebook newsfeed 32 2.273952e-08 0.0003340144 1.370003e-06 49.69292
## 7 facebook hiphop 29 2.060769e-08 0.0003340144 1.484170e-06 41.57004
## 8 facebook mashable.com 24 1.705464e-08 0.0003340144 1.293892e-06 39.46203
## 9 facebook friendster 30 2.131830e-08 0.0003340144 1.636393e-06 39.00317
## 10 facebook gtalk 22 1.563342e-08 0.0003340144 1.217781e-06 38.43437
## # ... with 1,736 more rows
What about the programming language Scala?
normalized_prob %__% filter(word1 == &quot;scala&quot;) %__%
arrange(-p_together)
## # A tibble: 445 x 7
## word1 word2 n p p1 p2 p_together
## ## 1 scala scala 9159 6.508478e-06 5.449568e-05 5.449568e-05 2191.56919
## 2 scala odersky 47 3.339867e-08 5.449568e-05 1.103614e-06 555.32856
## 3 scala akka 74 5.258514e-08 5.449568e-05 3.006396e-06 320.96285
## 4 scala sbt 31 2.202891e-08 5.449568e-05 1.446115e-06 279.52988
## 5 scala groovy 88 6.253369e-08 5.449568e-05 5.594180e-06 205.12353
## 6 scala mu 23 1.634403e-08 5.449568e-05 1.826671e-06 164.18624
## 7 scala kotlin 92 6.537613e-08 5.449568e-05 7.382795e-06 162.49359
## 8 scala clojure 472 3.354080e-07 5.449568e-05 5.719764e-05 107.60518
## 9 scala constructors 23 1.634403e-08 5.449568e-05 2.816118e-06 106.49918
## 10 scala idiomatic 54 3.837294e-08 5.449568e-05 7.573073e-06 92.98028
## # ... with 435 more rows
Looks good!
Cast to a sparse matrix
We want to do matrix factorization, so we should probably make a matrix. We can use cast_sparse() from the tidytext package to transform our tidy data frame to a matrix.
pmi_matrix %
mutate(pmi = log10(p_together)) %__%
cast_sparse(word1, word2, pmi)
What is the type of this object?
class(pmi_matrix)
## [1] &quot;dgCMatrix&quot;
## attr(,&quot;package&quot;)
## [1] &quot;Matrix&quot;
The dgCMatrix class is a class of sparse numeric matrices in R. Text data like this represented in matrix form usually has lots and lots of zeroes, so we want to make use of sparse data structures to save us time and memory and all that.
Reduce the matrix dimensionality
We want to get information out of this giant matrix in a more useful form, so it’s time for singular value decomposition. Since we have a sparse matrix, we don’t want to use base R’s svd function, which casts the input to a plain old matrix (not sparse) first thing. Instead we will use the fast SVD algorithm for sparse matrices in the irlba package.
library(irlba)
pmi_svd %
mutate(selected = &quot;haskell&quot;)) %__%
group_by(selected) %__%
top_n(15, similarity) %__%
ungroup %__%
mutate(token = reorder(token, similarity)) %__%
ggplot(aes(token, similarity, fill = selected)) +
geom_col(show.legend = FALSE) +
facet_wrap(~selected, scales = &quot;free&quot;) +
coord_flip() +
theme(strip.text=element_text(hjust=0, family=&quot;Roboto-Bold&quot;, size=12)) +
scale_y_continuous(expand = c(0,0)) +
labs(x = NULL, title = &quot;What word vectors are most similar to Facebook or Haskell?&quot;,
subtitle = &quot;Based on the Hacker News corpus, calculated using counts and matrix factorization&quot;)
We can also do the familiar WORD MATH that is so fun with the output of word2vec; you have probably seen examples such as King - Man + Woman = Queen and such. We can just add and subtract our word vectors, and then search the matrix we built!
If the iPhone is an important product associated with Apple, as discussed on Hacker News, what is an important product associated with Microsoft?
mystery_product