<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/training-evaluating-and-interpreting-topic-models/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://juliasilge.com/blog/evaluating-stm/"> Rstats on Julia Silge</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>At the beginning of this year, I wrote a blog post about how to get started with the <a rel="nofollow" href="https://juliasilge.com/blog/sherlock-holmes-stm/" target="_blank">stm and tidytext packages for topic modeling</a>. I have been doing more topic modeling in various projects, so I wanted to share some workflows I have found useful for</p>
<ul>
<li>training many topic models at one time,</li>
<li>evaluating topic models and understanding model diagnostics, and</li>
<li>exploring and interpreting the content of topic models.</li>
</ul>
<p>I’ve been doing all my topic modeling with <a rel="nofollow" href="http://www.structuraltopicmodel.com/" target="_blank">Structural Topic Models</a> and the <a rel="nofollow" href="https://cran.r-project.org/package=stm" target="_blank">stm</a> package lately, and it has been ✨GREAT✨. One thing I am not going to cover in this blog post is how to use document-level covariates in topic modeling, i.e., how to train a model with topics that can vary with some continuous or categorical characteristic of your documents. I hope to build up some posts about that, but in the meantime, you can check out the <a rel="nofollow" href="https://cran.r-project.org/web/packages/stm/vignettes/stmVignette.pdf" target="_blank">stm vignette</a> and perhaps <a rel="nofollow" href="https://github.com/methodds/stminsights" target="_blank">Carsten Schwemmer’s Shiny app</a> for more details on this.</p>
<div>
<h2>Modeling the Hacker News corpus</h2>
<p>In my last blog post, I demonstrated how to get started with about a book’s worth of text, which is a TEENY TINY amount of text for a topic model. This time around, I’d like to demonstrate how to go about interpreting results with a more realistic set of text, something more like what you might actually want to model topics with in the real world, so let’s turn to the <a rel="nofollow" href="https://cloud.google.com/bigquery/public-data/hacker-news" target="_blank">Hacker news corpus</a> and download 100,000 texts using the <a rel="nofollow" href="https://cran.r-project.org/package=bigrquery" target="_blank">bigrquery</a> package.</p>
<pre><code>library(bigrquery)
library(tidyverse)

sql &lt;- &quot;#legacySQL
SELECT
  stories.title AS title,
  stories.text AS text,
FROM
  [bigquery-public-data:hacker_news.full] AS stories
WHERE
  stories.deleted IS NULL
LIMIT
  100000&quot;

hacker_news_raw &lt;- query_exec(sql, project = project, max_pages = Inf)</code></pre>
<p>After we have the text downloaded, let’s clean the text and make a data frame containing only the text, plus an ID to identify each “document”, i.e., post.</p>
<pre><code>hacker_news_text &lt;- hacker_news_raw %>%
  as_tibble() %>%
  mutate(title = na_if(title, &quot;&quot;),
         text = coalesce(title, text)) %>%
  select(-title) %>%
  mutate(text = str_replace_all(text, &quot;'|&quot;|/&quot;, &quot;'&quot;), ## weird encoding
         text = str_replace_all(text, &quot;&lt;a(.*?)>&quot;, &quot; &quot;),             ## links 
         text = str_replace_all(text, &quot;>|&lt;|&amp;&quot;, &quot; &quot;),      ## html yuck
         text = str_replace_all(text, &quot;&amp;#[:digit:]+;&quot;, &quot; &quot;),        ## html yuck
         text = str_remove_all(text, &quot;&lt;[^>]*>&quot;),                    ## mmmmm, more html yuck
         postID = row_number()) </code></pre>
<p>Now it’s time to tokenize and tidy the text, remove some stop words (and numbers, although this is an analytical choice that you might want to try in a different way), and then cast to a sparse matrix. I’m using the <code>token = &quot;tweets&quot;</code> option for tokenizing because it often performs the most sensibly with text from online forums, such as Hacker News (and Stack Overflow, and Reddit, and so on). In my <a rel="nofollow" href="https://juliasilge.com/blog/sherlock-holmes-stm/" target="_blank">previous blog post</a>, I used a quanteda <code>dfm</code> as the input to the topic modeling algorithm, but here I’m using a plain old sparse matrix. Either one works.</p>
<pre><code>library(tidytext)

tidy_hacker_news &lt;- hacker_news_text %>%
  unnest_tokens(word, text, token = &quot;tweets&quot;) %>%
  anti_join(get_stopwords()) %>%
  filter(!str_detect(word, &quot;[0-9]+&quot;)) %>%
  add_count(word) %>%
  filter(n > 100) %>%
  select(-n)

hacker_news_sparse &lt;- tidy_hacker_news %>%
  count(postID, word) %>%
  cast_sparse(postID, word, n)</code></pre>
</div>
<div>
<h2>Train and evaluate topic models</h2>
<p>Now it’s time to train some topic models! 💪 You can check out that <a rel="nofollow" href="https://juliasilge.com/blog/sherlock-holmes-stm/" target="_blank">previous blog post on stm</a> for some details on how to get started, but in this post, we’re going to go to the next level. We’re not going to train just one topic model, but a whole group of them, with different numbers of topics, and then evaluate these models. In topic modeling, like with k-means clustering, we don’t know ahead of time how many topics we should use, and research in this area says there is no “right” answer for the number of topics that is appropriate for any given corpus. Here, let’s try a number of different values for <span>\(K\)</span> (the number of topics) from 20 to 100.</p>
<p>With 100,000 texts this modeling takes a while 😩 so I have used I have used the <a rel="nofollow" href="https://github.com/DavisVaughan/furrr" target="_blank">furrr</a> package (and <a rel="nofollow" href="https://github.com/HenrikBengtsson/future" target="_blank">future</a>) for parallel processing.</p>
<pre><code>library(stm)
library(furrr)
plan(multiprocess)

many_models &lt;- data_frame(K = c(20, 40, 50, 60, 70, 80, 100)) %>%
  mutate(topic_model = future_map(K, ~stm(hacker_news_sparse, K = .,
                                          verbose = FALSE)))</code></pre>
<p>Now that we’ve fit all these topic models with different numbers of topics, we can explore how many topics are appropriate/good/“best”. The code below to find <code>k_result</code> is similar to stm’s own <a rel="nofollow" href="https://github.com/bstewart/stm/blob/master/R/searchK.R" target="_blank"><code>searchK()</code></a> function, but it allows you to evaluate models trained on a sparse matrix (or a quanteda <code>dfm</code>) instead of only stm’s corpus data structure, as well as to dig into the model diagnostics yourself in detail. Some of these functions were not originally flexible enough to take a sparse matrix or <code>dfm</code> as input, so I’d like to send huge thanks to Brandon Stewart, stm’s developer, for <a rel="nofollow" href="https://github.com/bstewart/stm/issues/134" target="_blank">adding this functionality</a>.</p>
<pre><code>heldout &lt;- make.heldout(hacker_news_sparse)

k_result &lt;- many_models %>%
  mutate(exclusivity = map(topic_model, exclusivity),
         semantic_coherence = map(topic_model, semanticCoherence, hacker_news_sparse),
         eval_heldout = map(topic_model, eval.heldout, heldout$missing),
         residual = map(topic_model, checkResiduals, hacker_news_sparse),
         bound =  map_dbl(topic_model, function(x) max(x$convergence$bound)),
         lfact = map_dbl(topic_model, function(x) lfactorial(x$settings$dim$K)),
         lbound = bound + lfact,
         iterations = map_dbl(topic_model, function(x) length(x$convergence$bound)))

k_result</code></pre>
<pre><code>## # A tibble: 7 x 10
##       K topic_model exclusivity semantic_coherence eval_heldout residual        bound lfact     lbound iterations
##   </code></pre></div>