<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/cross-validation-of-topic-modelling/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://ellisp.github.io/blog/2017/01/05/topic-model-cv"> Peter's stats stuff - R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><center><img src="https://i0.wp.com/ellisp.github.io/img/0077-AssociatedPress.gif?w=400" data-recalc-dims="1"></center></p>
<h2>Determining the number of “topics” in a corpus of documents</h2>
<p>In my <a target="_blank" rel="nofollow" href="http://ellisp.github.io/blog/2016/12/31/sparse-bags">last post</a> I finished by topic modelling a set of political blogs from 2004. I made a passing comment that it’s a challenge to know how many topics to set; the R <code>topicmodels</code> package doesn’t do this for you. There’s quite a discussion on this out there, but nearly all the extant approaches amount to fitting your model with lots of different values of k (where k is the number of latent topics to identify) and seeing which is “best”. Naturally, this begs the question of what is meant by “best”.</p>
<p>A literature has grown around this question, and the convenient <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=ldatuning"><code>ldatuning</code> R package</a> by Nikita Murzintcev provides metrics using four of the methods. A <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/ldatuning/vignettes/topics.html">helpful vignette</a> shows this in action with the <code>AssociatedPress</code> dataset of 2,246 articles and 10,473 words. This dataset was presented at the First Text Retrieval Conference (TREC-1) in 1992 and is distributed with several of the commonly used R text-mining packages. The <code>ldatuning</code> approach is super-simple to run; here’s the all the code needed to fit calculate three of the measures for a range of different topic numbers from 10 to 350. To save computational effort, I omitted a fourth method that the vignette shows not to work well with this particular data.</p>
<figure>
<pre><code data-lang="r"><span># Load up R packages including a few we only need later:
</span><span>library</span><span>(</span><span>topicmodels</span><span>)</span><span>
</span><span>library</span><span>(</span><span>doParallel</span><span>)</span><span>
</span><span>library</span><span>(</span><span>ggplot2</span><span>)</span><span>
</span><span>library</span><span>(</span><span>scales</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>library</span><span>(</span><span>RColorBrewer</span><span>)</span><span>
</span><span>library</span><span>(</span><span>wordcloud</span><span>)</span><span>
</span><span>data</span><span>(</span><span>&quot;AssociatedPress&quot;</span><span>,</span><span> </span><span>package</span><span>=</span><span>&quot;topicmodels&quot;</span><span>)</span><span>
</span><span>full_data</span><span> </span><span>&lt;-</span><span> </span><span>AssociatedPress</span><span>
</span><span>system.time</span><span>({</span><span>
</span><span>tunes</span><span> </span><span>&lt;-</span><span> </span><span>FindTopicsNumber</span><span>(</span><span>
</span><span>full_data</span><span>,</span><span>
</span><span>topics</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>1</span><span>:</span><span>10</span><span> </span><span>*</span><span> </span><span>10</span><span>,</span><span> </span><span>120</span><span>,</span><span> </span><span>140</span><span>,</span><span> </span><span>160</span><span>,</span><span> </span><span>180</span><span>,</span><span> </span><span>0</span><span>:</span><span>3</span><span> </span><span>*</span><span> </span><span>50</span><span> </span><span>+</span><span> </span><span>200</span><span>),</span><span>
</span><span>metrics</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Griffiths2004&quot;</span><span>,</span><span> </span><span>&quot;CaoJuan2009&quot;</span><span>,</span><span> </span><span>&quot;Arun2010&quot;</span><span>),</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;Gibbs&quot;</span><span>,</span><span>
</span><span>control</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>seed</span><span> </span><span>=</span><span> </span><span>77</span><span>),</span><span>
</span><span>mc.cores</span><span> </span><span>=</span><span> </span><span>4L</span><span>,</span><span>
</span><span>verbose</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>
</span><span>)</span><span>
</span><span>})</span><span>
</span><span>FindTopicsNumber_plot</span><span>(</span><span>tunes</span><span>)</span></code></pre>
</figure>
<p>and here’s the results.</p>
<p><img alt="ldatuning" src="http://ellisp.github.io/img/0077-ldatuning.svg"></p>
<p>OK, pretty convincing, and a good literature behind it to justify these methods. The various methods agree that somewhere between 90 and 140 topics is optimal for this dataset. Time consuming, but rigorous. The code above took 10 hours to run on an old machine with four CPUs. The <code>ldatuning</code> documentation has references and links to the underlying articles.</p>
<h2>Cross-validation</h2>
<p>The abstracts of the (mostly paywalled unfortunately) articles implemented by <code>ldatuning</code> look like the metrics they suggest are based on assessing maximising <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Likelihood_function">likelihood</a>, minimising <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence">Kullback-Leibler divergence</a> or similar, using the same dataset that the model was trained on (rather than cross-validation).</p>
<p>While I was researching this I came across <a target="_blank" rel="nofollow" href="http://stackoverflow.com/questions/21355156/topic-models-cross-validation-with-loglikelihood-or-perplexity">this question on Stack Overflow</a> as well as a few mentions elsewhere on the web of cross-validation of topic models to choose optimal number of topics. As part of my familiarisation with the whole approach to topic modelling I decided to look further into this. For example, the <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/topicmodels/vignettes/topicmodels.pdf">vignette for topicmodels</a> says the number of topics in their worked example was chosen by 10-fold cross-validation, but doesn’t say how this was done other than pointing out that it is possible in “a few lines of R code”.</p>
<p>A more complete description is given in this <a target="_blank" rel="nofollow" href="http://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-16-S13-S8">conference paper by Zhao, Chen, Perkins, Liu, Ge, Ding and Zou</a>:</p>
<blockquote>
<p>“Lacking such a heuristic to choose the number of topics, researchers have no recourse beyond an informed guess or time-consuming trial and error evaluation. For trial and error evaluation, an iterative approach is typical based on presenting different models with different numbers of topics, normally developed using cross-validation on held-out document sets, and selecting the number of topics for which the model is least perplexed by the test sets… Using the identified appropriate number of topics, LDA is performed on the whole dataset to obtain the topics for the corpus. We refer to this as the perplexity-based method. Although the perplexity-based method may generate meaningful results in some cases, it is not stable and the results vary with the selected seeds even for the same dataset.”</p>
</blockquote>
<p>The key idea of cross-validation is that you divide the data into different numbers of subsets – conventionally 5 or 10, let’s say 5 from now on – and take turns at using one of the five as a validation set while the remaining four are used as a training set. This way each data point gets one turn as part of the hold-out validation, and four turns as part of the training set. It’s useful for assessing the overall validity of the model on data that wasn’t involved in its training, and also for identifying good values of tuning hyper-parameters. For today, I want to use it to find the best value of the number of topics hyperparameter “k”.</p>
<p>I’m going to use the <code>perplexity</code> measure for the applicability of a topic model to new data. <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Perplexity">Perplexity</a> is a measure of how well a probability model predicts a sample. Given the <code>perplexity</code> function comes in the <code>topicmodels</code> R package and is the obvious way supplied to test a trained model on new data, I think this is most likely what Grun and Hornick did for cross-validation in the <code>topicmodels</code> vignette; and it is mentioned by Zhao et al as the method used in “time-consuming trial and error evaluation”. I’m not particularly worried for now about what Zhao et al say about the stability of cross-validation with perplexity; I might come back to that in a later post (my intuition is that this is unlikely to be a problem, but I’ll want to check that some time!).</p>
<p>Let’s do this one step at a time, building up to the full cross-validation at different values of k:</p>
<ul>
<li>first, I’m going to do validation (not cross-validation), on a single hold-out set of one fifth of the data, at a single value of k</li>
<li>then I’m going to use cross-validation at a single value of k</li>
<li>finally I’m going to use cross-validation at many values of k</li>
</ul>
<p>Cross-validation is very computing-intensive and embarrassingly parallel so I’m going to parallelize the second and third of the efforts above.</p>
<h3>Simple validation with a single number of topics</h3>
<p>For the simple validation, I make a single <code>train_set</code> version of 75% of rows from the <code>AssociatedPress</code> document-term-matrix, randomly chosen; and a <code>valid_set</code> of the remaining 25%. The model is fit with the <code>LDA</code> function, and then (final two lines of code in the chunk below) I estimate how perplexed the resulting model is with both the training data set and the validation set.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>topicmodels</span><span>)</span><span>
</span><span>library</span><span>(</span><span>doParallel</span><span>)</span><span>
</span><span>library</span><span>(</span><span>ggplot2</span><span>)</span><span>
</span><span>library</span><span>(</span><span>scales</span><span>)</span><span>
</span><span>data</span><span>(</span><span>&quot;AssociatedPress&quot;</span><span>,</span><span> </span><span>package</span><span> </span><span>=</span><span> </span><span>&quot;topicmodels&quot;</span><span>)</span><span>
</span><span>burnin</span><span> </span><span>=</span><span> </span><span>1000</span><span>
</span><span>iter</span><span> </span><span>=</span><span> </span><span>1000</span><span>
</span><span>keep</span><span> </span><span>=</span><span> </span><span>50</span><span>
</span><span># define our &quot;full data&quot; - during development I pretend the full dataset is # just the first 500 AP articles, which is enough for a reasonable test while not taking
# forever to run. When I ran the final model, I came back and removed the &quot;1:500&quot; from below
</span><span>full_data</span><span> </span><span>&lt;-</span><span> </span><span>AssociatedPress</span><span>[</span><span>1</span><span>:</span><span>500</span><span> </span><span>,</span><span> </span><span>]</span><span>
</span><span>n</span><span> </span><span>&lt;-</span><span> </span><span>nrow</span><span>(</span><span>full_data</span><span>)</span><span>
</span><span>#-----------validation--------
</span><span>k</span><span> </span><span>&lt;-</span><span> </span><span>5</span><span> </span><span># number of topics
</span><span>
</span><span>splitter</span><span> </span><span>&lt;-</span><span> </span><span>sample</span><span>(</span><span>1</span><span>:</span><span>n</span><span>,</span><span> </span><span>round</span><span>(</span><span>n</span><span> </span><span>*</span><span> </span><span>0.75</span><span>))</span><span>
</span><span>train_set</span><span> </span><span>&lt;-</span><span> </span><span>full_data</span><span>[</span><span>splitter</span><span>,</span><span> </span><span>]</span><span>
</span><span>valid_set</span><span> </span><span>&lt;-</span><span> </span><span>full_data</span><span>[</span><span>-</span><span>splitter</span><span>,</span><span> </span><span>]</span><span>
</span><span>fitted</span><span> </span><span>&lt;-</span><span> </span><span>LDA</span><span>(</span><span>train_set</span><span>,</span><span> </span><span>k</span><span> </span><span>=</span><span> </span><span>k</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;Gibbs&quot;</span><span>,</span><span>
</span><span>control</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>burnin</span><span> </span><span>=</span><span> </span><span>burnin</span><span>,</span><span> </span><span>iter</span><span> </span><span>=</span><span> </span><span>iter</span><span>,</span><span> </span><span>keep</span><span> </span><span>=</span><span> </span><span>keep</span><span>)</span><span> </span><span>)</span><span>
</span><span>perplexity</span><span>(</span><span>fitted</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>train_set</span><span>)</span><span>
</span><span>perplexity</span><span>(</span><span>fitted</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>valid_set</span><span>)</span></code></pre>
</figure>
<p>Perplexity was 2728 on the training data, and a much higher 4280 on the validation set. This is to be expected – the model has been fit to the training set and naturally finds the new validation data more perplexing – that’s the whole reason for confronting a model with fresh data. BTW, to check whether this was impacted on by the number of documents, I also tried the above with equally sized training and validation sets and got similar results.</p>
<h3>Cross validation with a single number of topics</h3>
<p>Next step is do the validation, with a single value of k, but splitting the data into five so each row of the data gets a turn in the validation set. Here’s how I did that:</p>
<figure>
<pre><code data-lang="r"><span>#---------------5-fold cross-validation---------------------
</span><span>folds</span><span> </span><span>&lt;-</span><span> </span><span>5</span><span>
</span><span>splitfolds</span><span> </span><span>&lt;-</span><span> </span><span>sample</span><span>(</span><span>1</span><span>:</span><span>folds</span><span>,</span><span> </span><span>n</span><span>,</span><span> </span><span>replace</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>cluster</span><span> </span><span>&lt;-</span><span> </span><span>makeCluster</span><span>(</span><span>detectCores</span><span>(</span><span>logical</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>-</span><span> </span><span>1</span><span>)</span><span> </span><span># leave one CPU spare...
</span><span>registerDoParallel</span><span>(</span><span>cluster</span><span>)</span><span>
</span><span>clusterEvalQ</span><span>(</span><span>cluster</span><span>,</span><span> </span><span>{</span><span>
</span><span>library</span><span>(</span><span>topicmodels</span><span>)</span><span>
</span><span>})</span><span>
</span><span>clusterExport</span><span>(</span><span>cluster</span><span>,</span><span> </span><span>c</span><span>(</span><span>&quot;full_data&quot;</span><span>,</span><span> </span><span>&quot;k&quot;</span><span>,</span><span> </span><span>&quot;burnin&quot;</span><span>,</span><span> </span><span>&quot;iter&quot;</span><span>,</span><span> </span><span>&quot;keep&quot;</span><span>,</span><span> </span><span>&quot;splitfolds&quot;</span><span>))</span><span>
</span><span>results</span><span> </span><span>&lt;-</span><span> </span><span>foreach</span><span>(</span><span>i</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>folds</span><span>)</span><span> </span><span>%dopar%</span><span> </span><span>{</span><span>
</span><span>train_set</span><span> </span><span>&lt;-</span><span> </span><span>full_data</span><span>[</span><span>splitfolds</span><span> </span><span>!=</span><span> </span><span>i</span><span> </span><span>,</span><span> </span><span>]</span><span>
</span><span>valid_set</span><span> </span><span>&lt;-</span><span> </span><span>full_data</span><span>[</span><span>splitfolds</span><span> </span><span>==</span><span> </span><span>i</span><span>,</span><span> </span><span>]</span><span>
</span><span>fitted</span><span> </span><span>&lt;-</span><span> </span><span>LDA</span><span>(</span><span>train_set</span><span>,</span><span> </span><span>k</span><span> </span><span>=</span><span> </span><span>k</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;Gibbs&quot;</span><span>,</span><span>
</span><span>control</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>burnin</span><span> </span><span>=</span><span> </span><span>burnin</span><span>,</span><span> </span><span>iter</span><span> </span><span>=</span><span> </span><span>iter</span><span>,</span><span> </span><span>keep</span><span> </span><span>=</span><span> </span><span>keep</span><span>)</span><span> </span><span>)</span><span>
</span><span>return</span><span>(</span><span>perplexity</span><span>(</span><span>fitted</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>valid_set</span><span>))</span><span>
</span><span>}</span><span>
</span><span>stopCluster</span><span>(</span><span>cluster</span><span>)</span></code></pre>
</figure>
<p>The results aren’t particularly interesting – just 5 numbers – so I won’t show here.</p>
<h3>Cross validation with many candidate numbers of topics</h3>
<p>Finally, we now do this cross-validation many times, with different values of the number of latent topics to estimate. Here’s how I did that:</p>
<figure>
<pre><code data-lang="r"><span>#----------------5-fold cross-validation, different numbers of topics----------------
</span><span>cluster</span><span> </span><span>&lt;-</span><span> </span><span>makeCluster</span><span>(</span><span>detectCores</span><span>(</span><span>logical</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>-</span><span> </span><span>1</span><span>)</span><span> </span><span># leave one CPU spare...
</span><span>registerDoParallel</span><span>(</span><span>cluster</span><span>)</span><span>
</span><span>clusterEvalQ</span><span>(</span><span>cluster</span><span>,</span><span> </span><span>{</span><span>
</span><span>library</span><span>(</span><span>topicmodels</span><span>)</span><span>
</span><span>})</span><span>
</span><span>folds</span><span> </span><span>&lt;-</span><span> </span><span>5</span><span>
</span><span>splitfolds</span><span> </span><span>&lt;-</span><span> </span><span>sample</span><span>(</span><span>1</span><span>:</span><span>folds</span><span>,</span><span> </span><span>n</span><span>,</span><span> </span><span>replace</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>candidate_k</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>2</span><span>,</span><span> </span><span>3</span><span>,</span><span> </span><span>4</span><span>,</span><span> </span><span>5</span><span>,</span><span> </span><span>10</span><span>,</span><span> </span><span>20</span><span>,</span><span> </span><span>30</span><span>,</span><span> </span><span>40</span><span>,</span><span> </span><span>50</span><span>,</span><span> </span><span>75</span><span>,</span><span> </span><span>100</span><span>,</span><span> </span><span>200</span><span>,</span><span> </span><span>300</span><span>)</span><span> </span><span># candidates for how many topics
</span><span>clusterExport</span><span>(</span><span>cluster</span><span>,</span><span> </span><span>c</span><span>(</span><span>&quot;full_data&quot;</span><span>,</span><span> </span><span>&quot;burnin&quot;</span><span>,</span><span> </span><span>&quot;iter&quot;</span><span>,</span><span> </span><span>&quot;keep&quot;</span><span>,</span><span> </span><span>&quot;splitfolds&quot;</span><span>,</span><span> </span><span>&quot;folds&quot;</span><span>,</span><span> </span><span>&quot;candidate_k&quot;</span><span>))</span><span>
</span><span># we parallelize by the different number of topics. A processor is allocated a value
# of k, and does the cross-validation serially. This is because it is assumed there
# are more candidate values of k than there are cross-validation folds, hence it
# will be more efficient to parallelise
</span><span>system.time</span><span>({</span><span>
</span><span>results</span><span> </span><span>&lt;-</span><span> </span><span>foreach</span><span>(</span><span>j</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>length</span><span>(</span><span>candidate_k</span><span>),</span><span> </span><span>.combine</span><span> </span><span>=</span><span> </span><span>rbind</span><span>)</span><span> </span><span>%dopar%</span><span>{</span><span>
</span><span>k</span><span> </span><span>&lt;-</span><span> </span><span>candidate_k</span><span>[</span><span>j</span><span>]</span><span>
</span><span>results_1k</span><span> </span><span>&lt;-</span><span> </span><span>matrix</span><span>(</span><span>0</span><span>,</span><span> </span><span>nrow</span><span> </span><span>=</span><span> </span><span>folds</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span><span>colnames</span><span>(</span><span>results_1k</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;k&quot;</span><span>,</span><span> </span><span>&quot;perplexity&quot;</span><span>)</span><span>
</span><span>for</span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>folds</span><span>){</span><span>
</span><span>train_set</span><span> </span><span>&lt;-</span><span> </span><span>full_data</span><span>[</span><span>splitfolds</span><span> </span><span>!=</span><span> </span><span>i</span><span> </span><span>,</span><span> </span><span>]</span><span>
</span><span>valid_set</span><span> </span><span>&lt;-</span><span> </span><span>full_data</span><span>[</span><span>splitfolds</span><span> </span><span>==</span><span> </span><span>i</span><span>,</span><span> </span><span>]</span><span>
</span><span>fitted</span><span> </span><span>&lt;-</span><span> </span><span>LDA</span><span>(</span><span>train_set</span><span>,</span><span> </span><span>k</span><span> </span><span>=</span><span> </span><span>k</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;Gibbs&quot;</span><span>,</span><span>
</span><span>control</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>burnin</span><span> </span><span>=</span><span> </span><span>burnin</span><span>,</span><span> </span><span>iter</span><span> </span><span>=</span><span> </span><span>iter</span><span>,</span><span> </span><span>keep</span><span> </span><span>=</span><span> </span><span>keep</span><span>)</span><span> </span><span>)</span><span>
</span><span>results_1k</span><span>[</span><span>i</span><span>,]</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>k</span><span>,</span><span> </span><span>perplexity</span><span>(</span><span>fitted</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>valid_set</span><span>))</span><span>
</span><span>}</span><span>
</span><span>return</span><span>(</span><span>results_1k</span><span>)</span><span>
</span><span>}</span><span>
</span><span>})</span><span>
</span><span>stopCluster</span><span>(</span><span>cluster</span><span>)</span><span>
</span><span>results_df</span><span> </span><span>&lt;-</span><span> </span><span>as.data.frame</span><span>(</span><span>results</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>results_df</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>k</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>perplexity</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>()</span><span> </span><span>+</span><span>
</span><span>geom_smooth</span><span>(</span><span>se</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span> </span><span>+</span><span>
</span><span>ggtitle</span><span>(</span><span>&quot;5-fold cross-validation of topic modelling with the 'Associated Press' dataset&quot;</span><span>,</span><span>
</span><span>&quot;(ie five different models fit for each candidate number of topics)&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Candidate number of topics&quot;</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Perplexity when fitting the trained model to the hold-out set&quot;</span><span>)</span></code></pre>
</figure>
<p>This took 75 minutes to run on a modern machine (different to that used for <code>ldatuning</code>) when I stopped the <code>candidate_k</code> at 100. When I added the final values of 200 and 300 to <code>candidate_k</code> it took 210 minutes. I did notice that towards the end, not all processors were being used at maximum capacity, which suggests the way I’ve parallelised it may not be optimal.</p>
<p>Exactly how number of documents, topics and words contribute to the time cost of fitting a topic model is a subject for a future post I think.</p>
<p>Here’s the results from the cross-validation using perplexity:</p>
<p><img alt="cv" src="http://ellisp.github.io/img/0077-cv.svg"></p>
<p>We get something that at least is consistent with the measures from the <code>ldatuning</code> package; there’s a distinct flattening out of the cross-validated perplexity somewhere between 50 and 200 topics. By the time we have 200 topics there has definitely been over-fitting, and the model is starting to get worse when tested on the hold-out validation sets. There’s still judgement required as to exactly how many topics to use, but 100 looks a good consensus number that all three methods tried from <code>ldatuning</code> support as well as the perplexity cross-validation measure.</p>
<h2>Presentation of final model</h2>
<p>I opted to fit a model with 90 topics. Here’s an animation of the results:<br>
<img alt="anim2" src="https://i0.wp.com/ellisp.github.io/img/0077-AssociatedPress.gif?w=456" data-recalc-dims="1"></p>
<p>Code for creating the animation:</p>
<figure>
<pre><code data-lang="r"><span>#=====================Presentation of results===============
</span><span>FinalModel</span><span> </span><span>&lt;-</span><span> </span><span>LDA</span><span>(</span><span>full_data</span><span>,</span><span> </span><span>k</span><span> </span><span>=</span><span> </span><span>90</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;Gibbs&quot;</span><span>,</span><span>
</span><span>control</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>burnin</span><span> </span><span>=</span><span> </span><span>burnin</span><span>,</span><span> </span><span>iter</span><span> </span><span>=</span><span> </span><span>iter</span><span>,</span><span> </span><span>keep</span><span> </span><span>=</span><span> </span><span>keep</span><span>)</span><span> </span><span>)</span><span>
</span><span># approach to drawing word clouds of all topics from an object created with LDA,
# taken from /blog/2016/12/31/sparse-bags
</span><span>n</span><span> </span><span>&lt;-</span><span> </span><span>100</span><span>;</span><span> </span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Greens&quot;</span><span>;</span><span> </span><span>lda</span><span> </span><span>&lt;-</span><span> </span><span>FinalModel</span><span>
</span><span>p</span><span> </span><span>&lt;-</span><span> </span><span>posterior</span><span>(</span><span>lda</span><span>)</span><span>
</span><span>w</span><span>1</span><span> </span><span>&lt;-</span><span> </span><span>as.data.frame</span><span>(</span><span>t</span><span>(</span><span>p</span><span>$</span><span>terms</span><span>))</span><span> </span><span>w</span><span>2</span><span> </span><span>&lt;-</span><span> </span><span>w</span><span>1</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>word</span><span> </span><span>=</span><span> </span><span>rownames</span><span>(</span><span>w</span><span>1</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>topic</span><span>,</span><span> </span><span>weight</span><span>,</span><span> </span><span>-</span><span>word</span><span>)</span><span> </span><span>pal</span><span> </span><span>&lt;-</span><span> </span><span>rep</span><span>(</span><span>brewer.pal</span><span>(</span><span>9</span><span>,</span><span> </span><span>palette</span><span>),</span><span> </span><span>each</span><span> </span><span>=</span><span> </span><span>ceiling</span><span>(</span><span>n</span><span> </span><span>/</span><span> </span><span>9</span><span>))[</span><span>n</span><span>:</span><span>1</span><span>]</span><span>
</span><span>wd</span><span> </span><span>&lt;-</span><span> </span><span>setwd</span><span>(</span><span>tempdir</span><span>())</span><span>
</span><span># need to be careful for warnings that words didn't fit in, # in which case make the png device larger
# remove any PNG files sitting around in the temp folder
</span><span>unlink</span><span>(</span><span>&quot;*.png&quot;</span><span>)</span><span>
</span><span># create a PNG for each frame of the animation
</span><span>for</span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>ncol</span><span>(</span><span>w</span><span>1</span><span>)){</span><span>
</span><span>png</span><span>(</span><span>paste0</span><span>(</span><span>i</span><span> </span><span>+</span><span> </span><span>1000</span><span>,</span><span> </span><span>&quot;.png&quot;</span><span>),</span><span> </span><span>8</span><span> </span><span>*</span><span> </span><span>100</span><span>,</span><span> </span><span>8</span><span> </span><span>*</span><span> </span><span>100</span><span>,</span><span> </span><span>res</span><span> </span><span>=</span><span> </span><span>100</span><span>)</span><span>
</span><span>par</span><span>(</span><span>bg</span><span> </span><span>=</span><span> </span><span>&quot;grey95&quot;</span><span>)</span><span>
</span><span>w</span><span>3</span><span> </span><span>&lt;-</span><span> </span><span>w</span><span>2</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>topic</span><span> </span><span>==</span><span> </span><span>i</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>desc</span><span>(</span><span>weight</span><span>))</span><span>
</span><span>with</span><span>(</span><span>w</span><span>3</span><span>[</span><span>1</span><span>:</span><span>n</span><span>,</span><span> </span><span>],</span><span> </span><span>wordcloud</span><span>(</span><span>word</span><span>,</span><span> </span><span>freq</span><span> </span><span>=</span><span> </span><span>weight</span><span>,</span><span> </span><span>random.order</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>ordered.colors</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>colors</span><span> </span><span>=</span><span> </span><span>pal</span><span>))</span><span>
</span><span>title</span><span>(</span><span>paste</span><span>(</span><span>&quot;Associated Press Topic&quot;</span><span>,</span><span> </span><span>i</span><span>))</span><span>
</span><span>dev.off</span><span>()</span><span>
</span><span>}</span><span>
</span><span># combine all those png frames into a single animated GIF
</span><span>system</span><span>(</span><span>'magick -loop 0 -delay 70 *.png &quot;AssociatedPress.gif&quot;'</span><span>)</span><span>
</span><span># move the asset over to where needed for the blog. This is specific to my folder structure.
</span><span>file.copy</span><span>(</span><span>&quot;AssociatedPress.gif&quot;</span><span>,</span><span> </span><span>paste0</span><span>(</span><span>wd</span><span>,</span><span> </span><span>&quot;/..http://ellisp.github.io/img/0077-AssociatedPress.gif&quot;</span><span>),</span><span> </span><span>overwrite</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span># cleanup
</span><span>unlink</span><span>(</span><span>c</span><span>(</span><span>&quot;*.png&quot;</span><span>,</span><span> </span><span>&quot;AssociatedPress.gif&quot;</span><span>))</span><span>
</span><span>setwd</span><span>(</span><span>wd</span><span>)</span></code></pre>
</figure>
<h2>A small note on <code>topicmodels</code> in linux.</h2>
<p>In writing this post I used both Windows and Linux (Ubuntu) machines, and it was my first time using the R <code>topicmodels</code> pacakge on Linux. I had some frustrations getting it installed; basically the same problem with missing GNU Scientific Library in <a target="_blank" rel="nofollow" href="http://tinyheero.github.io/2016/02/20/install-r-topicmodels.html">this blog post</a> and <a target="_blank" rel="nofollow" href="http://stackoverflow.com/questions/24172188/how-can-i-install-topicmodels-package-in-r">this Stack Overflow Q&amp;A</a>.</p>
<p>The trick was I needed not just <code>gsl-bin</code> but also <code>libsg10-dev</code>. Also the <code>mpfr</code> library which is “a GNU portable C library for arbitrary-precision binary floating-point computation with correct rounding, based on GNU Multi-Precision Library.” In Ubuntu:</p>
<figure>
<pre><code data-lang="bash">sudo apt-get install gsl-bin libgsl0-dev
sudo apt-get install libmpfr-dev libmpfr-doc libmpfr4 libmpfr4-dbg</code></pre>
</figure>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://ellisp.github.io/blog/2017/01/05/topic-model-cv"> Peter's stats stuff - R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/9vD2pTcga4Q"></a></div>