<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/sparse-matrices-k-means-clustering-topic-modelling-with-posts-on-the-2004-us-presidential-election/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://ellisp.github.io/blog/2016/12/31/sparse-bags"> Peter's stats stuff - R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h2>Daily Kos bags of words from the time of the 2004 Presidential election</h2>
<p>This is a bit of a rambly blog entry today. My original motivation was to just explore moving data around from R into the H2O machine learning software. While successful on this, and more on it below, I got a bit interested in my example data in its own right. The best initial exploratory analysis of that data turns out on my opinion not to be one of the approaches implemented in H2O, so by the end of the post I’m back in R illustrating the use of topic modelling for analysing blog posts. As a teaser, this is where we’re going to end up:</p>
<p><img width="730" src="http://ellisp.github.io/img/0076-topics-8.svg"></p>
<p>That graphic represents eight latent “topics” identified in 3,430 posts on the liberal (as per US terminology) <a target="_blank" rel="nofollow" href="http://www.dailykos.com/">Daily Kos site</a> from the time of the 2004 US Presidential Election. The data are available at the <a target="_blank" rel="nofollow" href="https://archive.ics.uci.edu/ml/datasets/Bag+of+Words">Machine Learning Repository</a> of the Univerity of California’s Center for Machine Learning and Intelligent Systems. The metadata are low on context (or perhaps I was just looking in the wrong spot) – the origin is just described as “KOS blog entries … orig source: dailykos.com”, and I’m presuming from their content that they are from 2004. More on what it all means down below.</p>
<h2>Sparse matrices</h2>
<p>I chose this data because it’s a nice example of a medium sized sparse matrix. The data has the number of occurences of 6,906 words in 3,430 documents so the obvious conceptual format is a matrix with 3,430 rows and 6,906 columns, where most cells are empty (meaning zero occurences of that word in that document) and the filled cells are counts. Of course, we try to avoid moving data around like that. Instead of that super-wide matrix, the data is distributed as a text file with three columns that looks like this:</p>
<figure>
<pre><code data-lang="c"><span>1</span> <span>61</span> <span>2</span>
<span>1</span> <span>76</span> <span>1</span>
<span>1</span> <span>89</span> <span>1</span>
<span>1</span> <span>211</span> <span>1</span>
<span>1</span> <span>296</span> <span>1</span>
<span>1</span> <span>335</span> <span>1</span>
<span>1</span> <span>404</span> <span>1</span>
<span>1</span> <span>441</span> <span>1</span>
<span>1</span> <span>454</span> <span>2</span></code></pre>
</figure>
<p>In this case, it means that the first document has two counts of word 61; one count of word 76; one count of word 89; and so on. A separate file tells us that word 61 is “action”, word 76 is “added” and so on. In fact, the two files for distributing these data are just the two tables that you’d represent them with in a relational database. This is the optimally efficient way of storing data of this sort.</p>
<p>Now, I’ve been exploring the machine learning software <a target="_blank" rel="nofollow" href="http://www.h2o.ai/">H2O</a> which provides fast and efficient estimation of models such as :</p>
<ul>
<li>deep learning neural networks</li>
<li>random forests</li>
<li>gradient boosting machines</li>
<li>generalized linear models with elastic net regularisation</li>
<li>k-means clustering</li>
</ul>
<p>All these tools need data to be represented in a wide format, rather than the efficient long format you’d have in a relational database. In this case, that would mean a row for each document (or observation) and a column for each word, which would be used as “features” in machine learning terminology. That is, it needs the sort of operation you’d use <code>PIVOT</code> for in SQL Server, or <code>tidyr::spread</code> in R (or, before the coming of the tidyverse, <code>reshape</code> or <code>reshape2::cast</code>).</p>
<p>H2O is expected to work with another tool for data persistence eg a Hadoop cluster. One thing I’m still getting my head around is the several variants on making data actually available to H2O. I’m particularly interested in the case of medium-sized data that can be handled on a single machine in efficient sparse format, when a Hadoop cluster isn’t available. So the Daily Kos data was my small familiarisation example in this space.</p>
<p>A common workflow I expect with this sort of data is a bunch of pre-processing that will need a general data munging tool like R or Python before upload into H2O. For example, with text data, I might want to do one or more of remove common “stopwords” (like ‘the’, ‘and’, etc); reduce words to their “stems” (so “abandons” and “abandon” are treated as the same); match words to sentiment; etc. This is the sort of thing done with <code>NLTK</code> in Python or <code>tm</code>, <code>SnowballC</code> and the brilliant new <code>tidytext</code> in R.</p>
<p><a target="_blank" rel="nofollow" href="http://tidytextmining.com/"><code>tidytext</code> by Julia Silge and David Robinson</a> makes it possible to analyse textual data using the general data manipulation grammar familiar to users of Hadley Wickham’s <code>tidyverse</code>. One verb tidytext offers is <code>cast_sparse</code>, which converts data in long tidy format into a wide but sparse <code>Matrix</code> format (using Doug Bates and Martin Maechler’s <code>Matrix</code> package, which is distributed with base R) that is suitable for a range of statistical and machine learning models. So I was pleased to note that the <a target="_blank" rel="nofollow" href="http://h2o-release.s3.amazonaws.com/h2o/rel-tutte/1/index.html">latest stable version of H2O</a> (not yet on CRAN at the time of writing) and its accompanying R interface expands the <code>h2o::as.H2O</code> function to support the sparse <code>Matrix</code> format.</p>
<h2>Importing and modifying the data</h2>
<p>So here’s how I set about exploring a workflow based on the following key tools</p>
<ul>
<li>Pre-processing in R
<ul>
<li>Import data with <code>data.table::fread</code> – the fastest way to get data in. While <code>readr</code> or even <code>utils</code> functions would also do the job well in this modest-sized case, I want to scale up in the future and <code>fread</code> is the best there is for getting large, regular text data into R.</li>
<li>Modify with tools from the <code>tidyverse</code> and <code>tidytext</code>, calling in specialist packages for stemming etc as necessary</li>
<li>Cast to sparse <code>Matrix</code> format and export to H2O with <code>h2o::as.h2o</code></li>
</ul>
</li>
<li>Analyse in H2O</li>
<li>Export the summary results back to R for presentation</li>
</ul>
<p>I had my doubts about whether <code>h2o::as.h2o</code> is right for the job, because I understand it doesn’t make the most of H2O’s own fast data importing abilities. But it is definitely a candidate for medium sized data.</p>
<p>Here’s how I got started. This first chunk just loads the functionality used in the rest of the post and downloads the main table, if it hasn’t been downloaded before (I developed this script in little chunks of time between doing other stuff, and didn’t want to download the file every time I ran it or have to bother with selectively running bits of the script). Lots of R packages needed for today’s exploration:</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>ggplot2</span><span>)</span><span>
</span><span>library</span><span>(</span><span>scales</span><span>)</span><span>
</span><span>library</span><span>(</span><span>R.utils</span><span>)</span><span> </span><span># for gunzip
</span><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tidytext</span><span>)</span><span>
</span><span>library</span><span>(</span><span>Matrix</span><span>)</span><span> </span><span># for sparse matrices
</span><span>library</span><span>(</span><span>data.table</span><span>)</span><span> </span><span># for fread, fast version of read.table
</span><span>library</span><span>(</span><span>SnowballC</span><span>)</span><span> </span><span># for wordStem
</span><span>library</span><span>(</span><span>foreach</span><span>)</span><span>
</span><span>library</span><span>(</span><span>doParallel</span><span>)</span><span>
</span><span>library</span><span>(</span><span>testthat</span><span>)</span><span> </span><span># used for various checks and tests along the way
</span><span>library</span><span>(</span><span>topicmodels</span><span>)</span><span>
</span><span>library</span><span>(</span><span>slam</span><span>)</span><span> </span><span># for other types of sparse matrices, used with topicmodels
</span><span>library</span><span>(</span><span>wordcloud</span><span>)</span><span>
</span><span>library</span><span>(</span><span>grid</span><span>)</span><span> </span><span># for annotating graphics </span><span>
</span><span># latest version of h2o needed at time of writing (December 2016) for sparse matrix support
# install.packages(&quot;h2o&quot;, type=&quot;source&quot;, repos=(c(&quot;http://h2o-release.s3.amazonaws.com/h2o/rel-tutte/1/R&quot;)))
</span><span>library</span><span>(</span><span>h</span><span>2</span><span>o</span><span>)</span><span>
</span><span># https://archive.ics.uci.edu/ml/datasets/Bag+of+Words
# download the smallest example bag of words for first trial
</span><span>if</span><span>(</span><span>!</span><span>&quot;docword.kos.txt&quot;</span><span> </span><span>%in%</span><span> </span><span>list.files</span><span>()){</span><span>
</span><span>download.file</span><span>(</span><span>&quot;https://archive.ics.uci.edu/ml/machine-learning-databases/bag-of-words/docword.kos.txt.gz&quot;</span><span>,</span><span>
</span><span>destfile</span><span> </span><span>=</span><span> </span><span>&quot;docword.kos.txt.gz&quot;</span><span>,</span><span> </span><span>mode</span><span> </span><span>=</span><span> </span><span>&quot;wb&quot;</span><span>)</span><span>
</span><span>gunzip</span><span>(</span><span>&quot;docword.kos.txt.gz&quot;</span><span>)</span><span>
</span><span>}</span></code></pre>
</figure>
<p>Importing the data is super fast with <code>data.table::fread</code>. Combining it with the vocabulary look up table, removing stop words and performing stemming is made super-easy by <code>tidytext</code> in combination with text mining specialist tools, in this case <code>SnowballC::wordStem</code>:</p>
<figure>
<pre><code data-lang="r"><span>kos</span><span> </span><span>&lt;-</span><span> </span><span>fread</span><span>(</span><span>&quot;docword.kos.txt&quot;</span><span>,</span><span> </span><span>skip</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span> </span><span>sep</span><span> </span><span>=</span><span> </span><span>&quot; &quot;</span><span>)</span><span> </span><span>names</span><span>(</span><span>kos</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;doc&quot;</span><span>,</span><span> </span><span>&quot;word&quot;</span><span>,</span><span> </span><span>&quot;count&quot;</span><span>)</span><span>
</span><span>expect_equal</span><span>(</span><span>sum</span><span>(</span><span>kos</span><span>$</span><span>count</span><span>),</span><span> </span><span>467714</span><span>)</span><span>
</span><span>kos_vocab</span><span> </span><span>&lt;-</span><span> </span><span>read.table</span><span>(</span><span>&quot;https://archive.ics.uci.edu/ml/machine-learning-databases/bag-of-words/vocab.kos.txt&quot;</span><span>)</span><span>$</span><span>V</span><span>1</span><span>
</span><span>kos2</span><span> </span><span>&lt;-</span><span> </span><span>kos</span><span> </span><span>%>%</span><span>
</span><span># attach the names of words back to the words
</span><span> </span><span>mutate</span><span>(</span><span>word</span><span> </span><span>=</span><span> </span><span>factor</span><span>(</span><span>word</span><span>,</span><span> </span><span>labels</span><span> </span><span>=</span><span> </span><span>kos_vocab</span><span>))</span><span> </span><span>%>%</span><span>
</span><span># stopwords are meant to be already removed but no harm in checking (note - makes no difference):
</span><span> </span><span>anti_join</span><span>(</span><span>stop_words</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;word&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span># reduce words to their stems (otherwise abandon, abandoning, abandons all treated as different, etc),
</span><span> </span><span>mutate</span><span>(</span><span>word</span><span> </span><span>=</span><span> </span><span>wordStem</span><span>(</span><span>word</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>word</span><span> </span><span>=</span><span> </span><span>gsub</span><span>(</span><span>&quot;^iraq$&quot;</span><span>,</span><span> </span><span>&quot;iraqi&quot;</span><span>,</span><span> </span><span>word</span><span>))</span><span> </span><span>%>%</span><span>
</span><span># ... and aggregate back up by the new stemmed words to remove duplicates
</span><span> </span><span>group_by</span><span>(</span><span>doc</span><span>,</span><span> </span><span>word</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarise</span><span>(</span><span>count</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>count</span><span>))</span><span> </span><span>%>%</span><span>
</span><span># correct some names that went wrong in the stemming:
</span><span> </span><span>mutate</span><span>(</span><span>word</span><span> </span><span>=</span><span> </span><span>gsub</span><span>(</span><span>&quot;^chenei$&quot;</span><span>,</span><span> </span><span>&quot;cheney&quot;</span><span>,</span><span> </span><span>word</span><span>),</span><span>
</span><span>word</span><span> </span><span>=</span><span> </span><span>gsub</span><span>(</span><span>&quot;^kerri$&quot;</span><span>,</span><span> </span><span>&quot;kerry&quot;</span><span>,</span><span> </span><span>word</span><span>),</span><span>
</span><span>word</span><span> </span><span>=</span><span> </span><span>gsub</span><span>(</span><span>&quot;^georg$&quot;</span><span>,</span><span> </span><span>&quot;george&quot;</span><span>,</span><span> </span><span>word</span><span>),</span><span>
</span><span>word</span><span> </span><span>=</span><span> </span><span>gsub</span><span>(</span><span>&quot;^john$&quot;</span><span>,</span><span> </span><span>&quot;John&quot;</span><span>,</span><span> </span><span>word</span><span>))</span><span>
</span><span># how many words per post?
</span><span>wordcount</span><span> </span><span>&lt;-</span><span> </span><span>kos2</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>doc</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarise</span><span>(</span><span>words</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>count</span><span>))</span><span>
</span><span>par</span><span>(</span><span>font.main</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span><span>hist</span><span>(</span><span>wordcount</span><span>$</span><span>words</span><span>,</span><span> </span><span>col</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>,</span><span> </span><span>fg</span><span> </span><span>=</span><span> </span><span>&quot;grey80&quot;</span><span>,</span><span> </span><span>border</span><span> </span><span>=</span><span> </span><span>&quot;grey95&quot;</span><span>,</span><span>
</span><span>main</span><span> </span><span>=</span><span> </span><span>&quot;Words per post in the KOS data set&quot;</span><span>,</span><span> </span><span>xlab</span><span> </span><span>=</span><span> </span><span>&quot;Number of words&quot;</span><span>)</span></code></pre>
</figure>
<p><img alt="histogram" src="http://ellisp.github.io/img/0076-words-per-post.svg"></p>
<p>Because the post lengths varied in the predictable way (histogram above), I decided to convert all the counts to proportions of words in each original document. The next chunk does this, and then casts the data out of long format into wide, analysis-ready format. I create both sparse and dense versions of the wide matrix to check that things are working the way I want (this is one of the reasons for checking things out on a fairly small dataset). In the chunk below, note that I had a bit of a fiddle (and then built in a test to check I got it right) to get the columns in the same order with <code>cast_sparse</code> as when I used <code>spread</code> for the dense version.</p>
<figure>
<pre><code data-lang="r"><span># OK, quite a variety of words per post.
# Convert counts to *relative* count (ie proportion of words in each doc)
# need to do this as otherwise the main dimension in the data is just the # length of each blog post
</span><span>kos3</span><span> </span><span>&lt;-</span><span> </span><span>kos2</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>doc</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>count</span><span> </span><span>=</span><span> </span><span>count</span><span> </span><span>/</span><span> </span><span>sum</span><span>(</span><span>count</span><span>))</span><span>
</span><span># Sparse version
</span><span>kos_sparse</span><span> </span><span>&lt;-</span><span> </span><span>kos3</span><span> </span><span>%>%</span><span>
</span><span># arrange it so the colnames after cast_sparse are in alphabetical order:
</span><span> </span><span>arrange</span><span>(</span><span>word</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>cast_sparse</span><span>(</span><span>doc</span><span>,</span><span> </span><span>word</span><span>,</span><span> </span><span>count</span><span>)</span><span>
</span><span># Dense version
</span><span>kos_df</span><span> </span><span>&lt;-</span><span> </span><span>kos3</span><span> </span><span>%>%</span><span>
</span><span>spread</span><span>(</span><span>word</span><span>,</span><span> </span><span>count</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>0</span><span>)</span><span>
</span><span>kos_dense</span><span> </span><span>&lt;-</span><span> </span><span>as.matrix</span><span>(</span><span>kos_df</span><span>[</span><span> </span><span>,</span><span> </span><span>-1</span><span>])</span><span>
</span><span>expect_equal</span><span>(</span><span>colnames</span><span>(</span><span>kos_dense</span><span>),</span><span> </span><span>colnames</span><span>(</span><span>kos_sparse</span><span>))</span><span>
</span><span>dim</span><span>(</span><span>kos_sparse</span><span>)</span><span> </span><span># would be 3430 documents and 6906 words except that there's been more processing, so only 4566 words
</span><span>object.size</span><span>(</span><span>kos</span><span>)</span><span> </span><span># about 4MB
</span><span>object.size</span><span>(</span><span>kos_sparse</span><span>)</span><span> </span><span># about the same. Slightly less efficient
</span><span>object.size</span><span>(</span><span>kos_dense</span><span>)</span><span> </span><span>#</span><span> </span><span>120</span><span>+</span><span> </span><span>MB</span></code></pre>
</figure>
<h2>k-means cluster analysis in R</h2>
<p>As this exercise started as essentially just a test of getting data into H2O, I needed an H2O algorithm to test on the data. With just bags of words and no other metadata (such as how popular each post was, or who wrote it, or anything else of that sort) there’s no obvious response variable to test predictive analytics methods on, so I chose <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/K-means_clustering">k-means clustering</a> as my test algorithm instead. There’s reasons why this method isn’t particularly useful for this sort of high-dimensional (ie lots of columns) data, but they’re not enough to stop it being a test of sorts.</p>
<p>One of the challenges of cluster analysis is to know how many groups (k) to divide the data into. So give R and H2O a decent run, I applied the k-means clustering to the data (using the dense version of the matrix) with each value of k from 1 to 25.</p>
<p>Close readers – if there are any – may note that I haven’t first scaled all the columns in the matrix before applying the k-means algorithm, even though this is often recommended to avoid columns with high variance dominating the clustering. This is a deliberate choice from me, applicable to this particular dataset. As all the columns are on the same scale (“this word’s frequency as a proportion of total words in the document for this row”) I thought it made sense to let the high variance columns sway the results, rather than (in effect) let low variance columns for rarely-used words get equal votes.</p>
<figure>
<pre><code data-lang="r"><span>#========k-means cluster analysis in R===========
# set up parallel cluster for the R version of analysis # (sorry for there being two uses of the word &quot;cluster&quot;...)
</span><span>cluster</span><span> </span><span>&lt;-</span><span> </span><span>makeCluster</span><span>(</span><span>7</span><span>)</span><span> </span><span># only any good if you have at least 7 processors :)
</span><span>registerDoParallel</span><span>(</span><span>cluster</span><span>)</span><span>
</span><span>clusterExport</span><span>(</span><span>cluster</span><span>,</span><span> </span><span>&quot;kos_dense&quot;</span><span>)</span><span>
</span><span># How to work out how many clusters to find in the data?
# great answer at http://stackoverflow.com/questions/15376075/cluster-analysis-in-r-determine-the-optimal-number-of-clusters # But I only use the simplest method - visual inspection of a scree plot of the total within squares
</span><span>
</span><span># takes some time, even with the parallel processing:
</span><span>max_groups</span><span> </span><span>&lt;-</span><span> </span><span>25</span><span>
</span><span>wss</span><span> </span><span>&lt;-</span><span> </span><span>foreach</span><span> </span><span>(</span><span>i</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>max_groups</span><span>)</span><span> </span><span>%dopar%</span><span> </span><span>{</span><span>
</span><span>kmeans</span><span>(</span><span>kos_dense</span><span>,</span><span> </span><span>centers</span><span> </span><span>=</span><span> </span><span>i</span><span>)</span><span>$</span><span>tot.withinss</span><span>
</span><span>}</span><span>
</span><span>par</span><span>(</span><span>font.main</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span><span>plot</span><span>(</span><span>1</span><span>:</span><span>max_groups</span><span>,</span><span> </span><span>wss</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;b&quot;</span><span>,</span><span> </span><span>xlab</span><span>=</span><span>&quot;Number of groups&quot;</span><span>,</span><span>
</span><span>ylab</span><span>=</span><span>&quot;Within groups sum of squares&quot;</span><span>,</span><span> </span><span>bty</span><span> </span><span>=</span><span> </span><span>&quot;l&quot;</span><span>)</span><span>
</span><span>grid</span><span>()</span><span>
</span><span>stopCluster</span><span>(</span><span>cluster</span><span>)</span></code></pre>
</figure>
<p>This graphic shows the total within-groups sum of squares for each value of k. The within-groups sum of squares is an indicator of how homogenous the points within each group are. Inevitably it goes down as the number of groups dividing the data increases; we draw plots like this one to see if there is a visually-obvious “levelling-off” point to indicate when we are getting diminishing marginal returns from adding more groups.</p>
<p><img alt="scree" src="http://ellisp.github.io/img/0076-base-scree.svg"></p>
<p>I won’t worry too much about interpreting this as I think the method is of dubious value anyway. But my gut feel is that either there is 1, 2, or about 20 groups in this data.</p>
<h2>Getting the data into H2O</h2>
<p>For the H2O comparison, first step is to initiate an H2O cluster and get the data into it. With the newly enhanced <code>as.h2o</code> function working with sparse <code>Matrix</code> objects this is easy and takes only 3 seconds (compared to 22 seconds for uploading the dense version of the matrix, code not shown).</p>
<blockquote>
<p>There’s a funny quirk to watch out for; the column names don’t get imported, and only 100 column names are applied.</p>
</blockquote>
<p>I suspect the <code>colnames</code> issue after importing a sparse Matrix to H2O is a bug. but will make sure before I report it as an issue.</p>
<p>Having imported the data, I do a basic check. k-means with a single group should return just the mean of each column as the p-dimensional centroid of a single big hyper-sphere containing all the data. I want to check that the various versions of the data in R and H2O all return the same values for this.</p>
<figure>
<pre><code data-lang="r"><span>#==============cluster analysis in h2o======================
</span><span>h</span><span>2</span><span>o.init</span><span>(</span><span>nthreads</span><span> </span><span>=</span><span> </span><span>-1</span><span>,</span><span> </span><span>max_mem_size</span><span> </span><span>=</span><span> </span><span>&quot;10G&quot;</span><span>)</span><span>
</span><span>system.time</span><span>({</span><span> </span><span>kos_h1</span><span> </span><span>&lt;-</span><span> </span><span>as.h2o</span><span>(</span><span>kos_sparse</span><span>)})</span><span> </span><span># 3 seconds
</span><span>dim</span><span>(</span><span>kos_h1</span><span>)</span><span> </span><span># correct
</span><span>colnames</span><span>(</span><span>kos_h1</span><span>)</span><span> </span><span># for some reason, only 100 columnames, c1:100
</span><span>colnames</span><span>(</span><span>kos_h1</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>colnames</span><span>(</span><span>kos_sparse</span><span>)</span><span>
</span><span>#--------------------checking we have the same data in all its different versions--------------------
# Compare the simplest single cluster case - should just be the mean of each column
</span><span>simple_r</span><span> </span><span>&lt;-</span><span> </span><span>kmeans</span><span>(</span><span>kos_dense</span><span>,</span><span> </span><span>centers</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span><span>simple_h</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.kmeans</span><span>(</span><span>kos_h1</span><span>,</span><span> </span><span>x</span><span> </span><span>=</span><span> </span><span>colnames</span><span>(</span><span>kos_h1</span><span>),</span><span> </span><span>k</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>standardize</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>simple_m1</span><span> </span><span>&lt;-</span><span> </span><span>apply</span><span>(</span><span>kos_dense</span><span>,</span><span> </span><span>2</span><span>,</span><span> </span><span>mean</span><span>)</span><span>
</span><span>simple_m2</span><span> </span><span>&lt;-</span><span> </span><span>t</span><span>(</span><span>apply</span><span>(</span><span>kos_h1</span><span>,</span><span> </span><span>2</span><span>,</span><span> </span><span>mean</span><span>))</span><span>
</span><span>simple_m3</span><span> </span><span>&lt;-</span><span> </span><span>apply</span><span>(</span><span>kos_sparse</span><span>,</span><span> </span><span>2</span><span>,</span><span> </span><span>mean</span><span>)</span><span>
</span><span>comparison</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>
</span><span>R</span><span> </span><span>=</span><span> </span><span>simple_r</span><span>$</span><span>centers</span><span>[</span><span>1</span><span> </span><span>,],</span><span>
</span><span>manual1</span><span> </span><span>=</span><span> </span><span>simple_m1</span><span>,</span><span>
</span><span>H</span><span>2</span><span>O</span><span> </span><span>=</span><span> </span><span>t</span><span>(</span><span>h</span><span>2</span><span>o.centers</span><span>(</span><span>simple_h</span><span>))[,</span><span>1</span><span>],</span><span>
</span><span>manual2</span><span> </span><span>=</span><span> </span><span>as.vector</span><span>(</span><span>simple_m2</span><span>[,</span><span>1</span><span>]),</span><span>
</span><span>manual3</span><span> </span><span>=</span><span> </span><span>simple_m3</span><span>
</span><span>)</span><span>
</span><span>par</span><span>(</span><span>font.main</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span><span>pairs</span><span>(</span><span>comparison</span><span>,</span><span> </span><span>main</span><span> </span><span>=</span><span> </span><span>&quot;Different methods of estimating the center of the data&quot;</span><span>)</span></code></pre>
</figure>
<p><img src="https://i1.wp.com/ellisp.github.io/img/0076-pairs-centers.png?w=456" data-recalc-dims="1"></p>
<p>And the result is fine. The data import has worked ok. Note that this wasn’t the case when I first did it – after a while I tracked it down to the non-alphabetical ordering of columns created by <code>cast_sparse</code>, which I have now forced to be alphabetical in the code earlier in this post.</p>
<h2>k-means cluster analysis in H2O</h2>
<p>To complete my experiment, I ran the k-means clustering algorithm on the data in H2O in a similar way to how I’d already done it in base R. One advantage of H2O in this context is it lets you use cross-validation for assessing the within-group sum of squares, rather than getting it from the data that was used to set the centers. This takes longer but is a better way of doing it. In fact, I tried it with and without cross validation and got similar results – the cross-validation total within groups sum of squares is always a little higher than that from the training data, but not enough to disrupt patterns.</p>
<figure>
<pre><code data-lang="r"><span>max_groups</span><span> </span><span>&lt;-</span><span> </span><span>25</span><span>
</span><span>wss_h2o</span><span> </span><span>&lt;-</span><span> </span><span>numeric</span><span>(</span><span>max_groups</span><span>)</span><span>
</span><span>for</span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>max_groups</span><span>){</span><span>
</span><span>kos_km1</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.kmeans</span><span>(</span><span>kos_h1</span><span>,</span><span> </span><span>x</span><span>=</span><span> </span><span>colnames</span><span>(</span><span>kos_h1</span><span>),</span><span> </span><span>estimate_k</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>k</span><span> </span><span>=</span><span> </span><span>i</span><span>,</span><span> </span><span>standardize</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>nfolds</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>max_iterations</span><span> </span><span>=</span><span> </span><span>25</span><span>)</span><span> </span><span>wss_h2o</span><span>[</span><span>i</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.tot_withinss</span><span>(</span><span>kos_km1</span><span>,</span><span> </span><span>xval</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span># xval=TRUE means cross validation used
</span><span>}</span><span>
</span><span>par</span><span>(</span><span>font.main</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span><span>plot</span><span>(</span><span>1</span><span>:</span><span>max_groups</span><span>,</span><span> </span><span>wss_h2o</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;b&quot;</span><span>,</span><span> </span><span>xlab</span><span>=</span><span>&quot;Number of Clusters&quot;</span><span>,</span><span>
</span><span>ylab</span><span>=</span><span>&quot;Within groups sum of squares&quot;</span><span>,</span><span> </span><span>bty</span><span> </span><span>=</span><span> </span><span>&quot;l&quot;</span><span>)</span><span>
</span><span>grid</span><span>()</span></code></pre>
</figure>
<p><img alt="h2o-scree" src="http://ellisp.github.io/img/0076-h2o-scree.svg"></p>
<p>Note that the scree plot from H2O is <em>much</em> less regular than that from base R. This was the case whether it was created with cross-validation values of total within group sum of squares, or those from the training data. Basically, the H2O k-means algorithm appears to be a little less steady than that in R. This would worry me except for my concerns with the appropriateness of the method at all to this sort of data. Because of the <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Clustering_high-dimensional_data">“curse of dimensionality”, nearly all points in this 4,000 dimensional space are pretty much equally far from each-other</a>, and it’s not surprising that a technique that aims to efficiently group them into clusters finds itself in unstable territory.</p>
<p>So I’m not going to spend more space on presenting the final model, as I just don’t think the method is a good one in this case. The exercise has served its purpose of showing that the new <code>as.h2o</code> upload of sparse <code>Matrix</code> format data into H2O works. Unfortunately, it didn’t scale up to larger data. The New York Times bags of words in the same repository as the Kos Blog data has about 300,000 documents and a vocabulary of 100,000 words, and although R, <code>tidytext</code> and <code>Matrix</code> handle it fine I couldn’t upload to H2O with <code>as.h2o</code>. So this remains an open problem. If you know how to fix it, please answer my <a target="_blank" rel="nofollow" href="http://stackoverflow.com/questions/41340086/how-to-cast-data-from-long-to-wide-format-in-h2o">question on Stack Overflow!</a> The solution could be either a way of pivoting/spreading long form data in H2O itself, or a better way of handling the upload of pre-pivoted wide sparse data. Or perhaps I just need more computing power.</p>
<h2>Topic modelling in R</h2>
<p>I would have stopped there but I thought it was a waste of a good dataset to use only an admittedly inferior method to analyse it. So I reached into the toolbox and grabbed the <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Topic_model">topic modelling</a> approach, which is a more appropriate tool in this case. Topic modelling is a Bayesian approach that postulates a number of unseen latent “topics”. Each topic has its own probability distribution to generate a bag of words. The probability of each estimated topic generating each observed bag of words conditional on the best estimate of topics is considered, the process improved iteratively, and eventually one is left with a bunch of topics each of which has a probability of generating each word.</p>
<p>As an inferential method I think it’s over-rated, but as an exploratory approach to otherwise intractable large bunches of bags of words it’s pretty useful – sort of the equivalent exploratory analytical/graphical combined method (in my view) to the armoury of density plots, scatter plots and line charts that we use when first exploring continuous data. Oddly enough, the results are usually presented as tables of data listing the words most characteristic of each topic. I think we can do better than that… here’s the graphic from the top of the post again, for convenience:</p>
<p><img width="730" src="http://ellisp.github.io/img/0076-topics-8.svg"></p>
<p>Word clouds can be a silly gimmick, but if used well they are a good use of space to pack text in. Like any graphic, they should facilitate comparisons (in this case, between different topics).</p>
<p>My interpretation of these topics?</p>
<ul>
<li>Topics 1 and 8 might be the more tactically focused discussion of polls and punditry in the leadup to the actual election</li>
<li>Topic 2 stands out clearly as the Iraq War topic</li>
<li>Topic 6 clearly relates to the broader war on terror, intelligence and defence issues</li>
<li>Topic 5 looks to be the general “Bush Administration” discussions</li>
<li>Topic 3 seems to be focused on the Democratic primary process</li>
<li>Topic 4 might be discussion of non-Presidential (Senate, House, Governor) races</li>
<li>Topic 7 might be a more general discussion of the Democratic party, people, politics and parties</li>
<li>The Daily Kos writers are a pretty nationally focused lot. A political blog in Australia or New Zealand might discuss elections in the USA, Britain, and probably France, Germany, Japan and maybe Indonesia as well as their own country, but the Daily Kos has the luxury of focusing on just the USA.</li>
</ul>
<p>Overall, it’s not a bad overview of the subject-matter of 3,400 articles that the computer has read so we don’t need to.</p>
<p>As this is a standard and nicely curated dataset I thought there would be stacks of people who have done this before me but on a cursory search I couldn’t find it. Here are a couple of bits of related analysis:</p>
<ul>
<li><a target="_blank" rel="nofollow" href="https://datamicroscopes.github.io/topic.html">https://datamicroscopes.github.io/topic.html</a></li>
<li><a target="_blank" rel="nofollow" href="http://cseweb.ucsd.edu/~saul/papers/icml13_tilda.pdf">http://cseweb.ucsd.edu/~saul/papers/icml13_tilda.pdf</a></li>
</ul>
<p>There’s a fair bit of discussion and not much clarity on the web on how to choose the correct number of topics for a given corpus of text. I think I’ll save that for another time.</p>
<p>Here’s the R code that created it, no frills or playing around with arguments. Note that I had to put the data into <em>another</em> sparse matrix format for the excellent <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=topicmodels"><code>topicmodels</code> R package</a> by Bettina Gruen to work with. I use the <code>kos2</code> object defined earlier in this post, which was the data after stemming and re-aggregation, but before word frequencies were converted to counts. The most common form of topic modelling uses a method called latent Dirichlet allocation which unless I’m mistaken does that conversion under the hood; but the <code>topicmodels::LDA</code> function I use below needs to be provided counts, not proportions.</p>
<figure>
<pre><code data-lang="r"><span>kos_counts</span><span> </span><span>&lt;-</span><span> </span><span>kos2</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>word</span><span>)</span><span> </span><span>%>%</span><span>
</span><span># take advantage of factor() as a quick way to make number:level pairings
</span><span> </span><span>mutate</span><span>(</span><span>word</span><span> </span><span>=</span><span> </span><span>factor</span><span>(</span><span>word</span><span>))</span><span>
</span><span># convert to triplet (row, column, cell) sparse matrix format:
</span><span>kos_trip</span><span> </span><span>&lt;-</span><span> </span><span>with</span><span>(</span><span>kos_counts</span><span>,</span><span> </span><span>simple_triplet_matrix</span><span>(</span><span>i</span><span> </span><span>=</span><span> </span><span>doc</span><span>,</span><span> </span><span>j</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>word</span><span>),</span><span> </span><span>v</span><span> </span><span>=</span><span> </span><span>count</span><span>))</span><span>
</span><span>dimnames</span><span>(</span><span>kos_trip</span><span>)[[</span><span>2</span><span>]]</span><span> </span><span>&lt;-</span><span> </span><span>levels</span><span>(</span><span>kos_counts</span><span>$</span><span>word</span><span>)</span><span>
</span><span># function for drawing word clouds of all topics from an object created with LDA
</span><span>wclda</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>lda</span><span>,</span><span> </span><span>n</span><span> </span><span>=</span><span> </span><span>100</span><span>,</span><span> </span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Blues&quot;</span><span>,</span><span> </span><span>...</span><span>){</span><span>
</span><span>p</span><span> </span><span>&lt;-</span><span> </span><span>posterior</span><span>(</span><span>lda</span><span>)</span><span>
</span><span>w</span><span>1</span><span> </span><span>&lt;-</span><span> </span><span>as.data.frame</span><span>(</span><span>t</span><span>(</span><span>p</span><span>$</span><span>terms</span><span>))</span><span> </span><span>w</span><span>2</span><span> </span><span>&lt;-</span><span> </span><span>w</span><span>1</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>word</span><span> </span><span>=</span><span> </span><span>rownames</span><span>(</span><span>w</span><span>1</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>topic</span><span>,</span><span> </span><span>weight</span><span>,</span><span> </span><span>-</span><span>word</span><span>)</span><span> </span><span>pal</span><span> </span><span>&lt;-</span><span> </span><span>rep</span><span>(</span><span>brewer.pal</span><span>(</span><span>9</span><span>,</span><span> </span><span>palette</span><span>),</span><span> </span><span>each</span><span> </span><span>=</span><span> </span><span>ceiling</span><span>(</span><span>n</span><span> </span><span>/</span><span> </span><span>9</span><span>))[</span><span>n</span><span>:</span><span>1</span><span>]</span><span>
</span><span>for</span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>ncol</span><span>(</span><span>w</span><span>1</span><span>)){</span><span>
</span><span>w</span><span>3</span><span> </span><span>&lt;-</span><span> </span><span>w</span><span>2</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>topic</span><span> </span><span>==</span><span> </span><span>i</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>desc</span><span>(</span><span>weight</span><span>))</span><span>
</span><span>with</span><span>(</span><span>w</span><span>3</span><span>[</span><span>1</span><span>:</span><span>n</span><span>,</span><span> </span><span>],</span><span> </span><span>wordcloud</span><span>(</span><span>word</span><span>,</span><span> </span><span>freq</span><span> </span><span>=</span><span> </span><span>weight</span><span>,</span><span> </span><span>random.order</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>ordered.colors</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>colors</span><span> </span><span>=</span><span> </span><span>pal</span><span>,</span><span> </span><span>...</span><span>))</span><span>
</span><span>title</span><span>(</span><span>paste</span><span>(</span><span>&quot;Topic&quot;</span><span>,</span><span> </span><span>i</span><span>))</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span><span>system.time</span><span>(</span><span>lda3</span><span> </span><span>&lt;-</span><span> </span><span>LDA</span><span>(</span><span>kos_trip</span><span>,</span><span> </span><span>k</span><span> </span><span>=</span><span> </span><span>8</span><span>))</span><span> </span><span># about 300 seconds
</span><span>
</span><span>par</span><span>(</span><span>mfrow</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>2</span><span>,</span><span> </span><span>4</span><span>),</span><span> </span><span>font.main</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span><span>wclda</span><span>(</span><span>lda3</span><span>)</span><span>
</span><span>grid.text</span><span>(</span><span>0.5</span><span>,</span><span> </span><span>0.55</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>&quot;Latent topics identified in 3,430 dailykos.com blog posts in 2004&quot;</span><span>,</span><span>
</span><span>gp</span><span> </span><span>=</span><span> </span><span>gpar</span><span>(</span><span>fontface</span><span> </span><span>=</span><span> </span><span>&quot;italic&quot;</span><span>))</span><span>
</span><span>grid.text</span><span>(</span><span>0.98</span><span>,</span><span> </span><span>0.02</span><span>,</span><span> </span><span>hjust</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span>
</span><span>label</span><span> </span><span>=</span><span> </span><span>&quot;Source: data from https://archive.ics.uci.edu/ml/datasets/Bag+of+Words\n analysis from https://ellisp.github.io&quot;</span><span>,</span><span>
</span><span>gp</span><span> </span><span>=</span><span> </span><span>gpar</span><span>(</span><span>fontface</span><span> </span><span>=</span><span> </span><span>&quot;italic&quot;</span><span>,</span><span> </span><span>cex</span><span> </span><span>=</span><span> </span><span>0.7</span><span>,</span><span> </span><span>col</span><span> </span><span>=</span><span> </span><span>&quot;grey60&quot;</span><span>))</span></code></pre>
</figure>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://ellisp.github.io/blog/2016/12/31/sparse-bags"> Peter's stats stuff - R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/Bl1AeGxRbew"></a></div>