<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/word-vectors-with-tidy-data-principles/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://juliasilge.com/blog/tidy-word-vectors/"> Rstats on Julia Silge</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Last week I saw Chris Moody’s <a rel="nofollow" href="http://multithreaded.stitchfix.com/blog/2017/10/18/stop-using-word2vec/" target="_blank">post on the Stitch Fix blog</a> about calculating word vectors from a corpus of text using word counts and matrix factorization, and I was so excited! This blog post illustrates how to implement that approach to find word vector representations in R using tidy data principles and sparse matrices.</p>
<p>Word vectors, or word embeddings, are typically calculated using neural networks; that is what <a rel="nofollow" href="https://en.wikipedia.org/wiki/Word2vec" target="_blank">word2vec</a> is. (<a rel="nofollow" href="https://nlp.stanford.edu/projects/glove/" target="_blank">GloVe embeddings</a> are trained a little differently than word2vec.) By contrast, the approach from Chris’s post that I’m implementing here uses only counting and some linear algebra. Deep learning is great, but I am super excited about this approach because it allows practitioners to find word vectors for their own collections of text (no need to rely on pre-trained vectors) using familiar techniques that are not difficult to understand. And it doesn’t take too long computationally!</p>
<div>
<h2>Getting some data</h2>
<p>Let’s download half a million observations from… the <a rel="nofollow" href="https://cloud.google.com/bigquery/public-data/hacker-news" target="_blank">Hacker News corpus</a>.</p>
<p><iframe width="450" allowfullscreen src="https://giphy.com/embed/uwZhzLqlV0cZq"><br>
</iframe></p>
<p>
<a rel="nofollow" href="https://giphy.com/gifs/buster-keaton-uwZhzLqlV0cZq" target="_blank">via GIPHY</a>
</p>
<p>I know, right? But it’s the dataset that Chris uses in his blog post and it gives me an opportunity to use the <a rel="nofollow" href="https://cran.r-project.org/package=bigrquery" target="_blank">bigrquery</a> package for the first time.</p>
<pre><code>library(bigrquery)
library(tidyverse)

project &lt;- &quot;my-first-project-184003&quot;

sql &lt;- &quot;#legacySQL
SELECT
  stories.title AS title,
  stories.text AS text
FROM
  [bigquery-public-data:hacker_news.full] AS stories
WHERE
  stories.deleted IS NULL
LIMIT
  500000&quot;

hacker_news_raw &lt;- query_exec(sql, project = project, max_pages = Inf)</code></pre>
<p>Next, let’s clean this text up to take care of some of the messy ways it has gotten encoded.</p>
<pre><code>library(stringr)

hacker_news_text &lt;- hacker_news_raw %>%
    as_tibble() %>%
    mutate(title = na_if(title, &quot;&quot;),
           text = coalesce(title, text)) %>%
    select(-title) %>%
    mutate(text = str_replace_all(text, &quot;'|&quot;|/&quot;, &quot;'&quot;),   ## weird encoding
           text = str_replace_all(text, &quot;&lt;a(.*?)>&quot;, &quot; &quot;),               ## links 
           text = str_replace_all(text, &quot;>|&lt;&quot;, &quot; &quot;),              ## html yuck
           text = str_replace_all(text, &quot;&lt;[^>]*>&quot;, &quot; &quot;),                ## mmmmm, more html yuck
           postID = row_number())</code></pre>
</div>
<div>
<h2>Unigram probabilities</h2>
<p>First, let’s calculate the unigram probabilities, how often we see each word in this corpus. This is straightforward using <code>unnest_tokens()</code> from the <a rel="nofollow" href="https://github.com/juliasilge/tidytext" target="_blank">tidytext</a> package and then just <code>count()</code> and <code>mutate()</code> from dplyr.</p>
<pre><code>library(tidytext)

unigram_probs &lt;- hacker_news_text %>%
    unnest_tokens(word, text) %>%
    count(word, sort = TRUE) %>%
    mutate(p = n / sum(n))

unigram_probs</code></pre>
<pre><code>## # A tibble: 318,503 x 3
##     word       n          p
##    </code></pre></div>