<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/words-growing-or-shrinking-in-hacker-news-titles-a-tidy-analysis/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://varianceexplained.org/r/hn-trends/"> Variance Explained</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In May, some friends and I built <a rel="nofollow" href="http://taggernews.com/" target="_blank">Tagger News</a>, a real-time automatic classifier of Hacker News articles based on their text (see <a rel="nofollow" href="http://varianceexplained.org/programming/tagger-news/" target="_blank">here</a> for more about how we built it). This process started me down some interesting paths, particularly analyzing trends in titles. By finding words that became more or less common in Hacker News titles over time, we can see what topics were gaining or losing interest in the tech world.</p>
<p>I thought I’d share some of these analyses here. As with most of my text analysis posts, I’ll be analyzing it with the <a rel="nofollow" href="http://tidytextmining.com/" target="_blank">tidytext package</a> written by me and <a rel="nofollow" href="https://juliasilge.com/" target="_blank">Julia Silge</a>. (Check out our soon-to-be-released book on the subject, <a rel="nofollow" href="https://www.amazon.com/Text-Mining-R-Tidy-Approach/dp/1491981652" target="_blank">Text Mining with R</a>!)</p>
<h3>Processing one million Hacker News articles</h3>
<p>The titles (and dates, and links) of Hacker News articles are helpfully stored <a rel="nofollow" href="https://cloud.google.com/bigquery/public-data/hacker-news" target="_blank">on Google Bigquery</a>. We start with a dataset of a million Hacker News article titles, which covers about 3 and a half years of posts. I downloaded it on 2017-06-04 with the following query:</p>
<div>
<pre><code>SELECT id, score, time, title, url FROM [bigquery-public-data:hacker_news.full]
  WHERE deleted IS null AND dead IS null AND type = 'story'
  ORDER BY time DESC
  LIMIT 1000000
</code></pre>
</div>
<p>I’m hosting the dataset so that you can read it yourself (note that it’s a moderately large and compressed file so it may take a minute to read) as follows:</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>library</span><span>(</span><span>lubridate</span><span>)</span><span>
</span><span>stories</span><span> </span><span>&lt;-</span><span> </span><span>read_csv</span><span>(</span><span>&quot;http://varianceexplained.org/files/stories_1000000.csv.gz&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>time</span><span> </span><span>=</span><span> </span><span>as.POSIXct</span><span>(</span><span>time</span><span>,</span><span> </span><span>origin</span><span> </span><span>=</span><span> </span><span>&quot;1970-01-01&quot;</span><span>),</span><span>
</span><span>month</span><span> </span><span>=</span><span> </span><span>round_date</span><span>(</span><span>time</span><span>,</span><span> </span><span>&quot;month&quot;</span><span>))</span></code></pre>
</figure>
<p>Whenever we read in a text dataset, the first step is usually to tokenize it (split it up into words) and remove stopwords (like “the” or “and”). <a rel="nofollow" href="http://tidytextmining.com/tidytext.html" target="_blank">As described here</a>, we can do this with tidytext’s <code>unnest_tokens</code> function.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tidytext</span><span>)</span><span>
</span><span>library</span><span>(</span><span>stringr</span><span>)</span><span>
</span><span>title_words</span><span> </span><span>&lt;-</span><span> </span><span>stories</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>desc</span><span>(</span><span>score</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>distinct</span><span>(</span><span>title</span><span>,</span><span> </span><span>.keep_all</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>unnest_tokens</span><span>(</span><span>word</span><span>,</span><span> </span><span>title</span><span>,</span><span> </span><span>drop</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>distinct</span><span>(</span><span>id</span><span>,</span><span> </span><span>word</span><span>,</span><span> </span><span>.keep_all</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>anti_join</span><span>(</span><span>stop_words</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;word&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>str_detect</span><span>(</span><span>word</span><span>,</span><span> </span><span>&quot;[^\\d]&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>word</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>word_total</span><span> </span><span>=</span><span> </span><span>n</span><span>())</span><span> </span><span>%>%</span><span>
</span><span>ungroup</span><span>()</span></code></pre>
</figure>
<p>This creates a pretty large data frame (4443083 rows) with one row per word per post. For example, we could use this to find and visualize the most common words in Hacker News posts during this 3.5 year period.</p>
<figure>
<pre><code data-lang="r"><span>word_counts</span><span> </span><span>&lt;-</span><span> </span><span>title_words</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>word</span><span>,</span><span> </span><span>sort</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>word_counts</span><span> </span><span>%>%</span><span>
</span><span>head</span><span>(</span><span>25</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>word</span><span> </span><span>=</span><span> </span><span>reorder</span><span>(</span><span>word</span><span>,</span><span> </span><span>n</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>word</span><span>,</span><span> </span><span>n</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_col</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;lightblue&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_y_continuous</span><span>(</span><span>labels</span><span> </span><span>=</span><span> </span><span>comma_format</span><span>())</span><span> </span><span>+</span><span>
</span><span>coord_flip</span><span>()</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Most common words in Hacker News titles&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Among the last million stories; stop words removed&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;# of uses&quot;</span><span>)</span></code></pre>
</figure>
<p><img alt="center" src="https://i0.wp.com/varianceexplained.org/figs/2017-06-08-hn-trends/word_counts_bar-1.png?w=456" data-recalc-dims="1"></p>
<p>The top result, “hn”, comes from the common constructions of “Show HN” and “Ask HN”, which are often prepended to a title. Among the other words, there’s nothing we wouldn’t have expected from following Hacker News. There some notable technology companies like Google, Apple, and Facebook, along with common topics in tech discussions like “app”, “data” and “startup”.</p>
<h3>Change over time</h3>
<p>What words and topics have become more frequent, or less frequent, over time? These could give us a sense of the changing technology ecosystem, and let us predict what topics will continue to grow in relevance.</p>
<p>To achieve this, we’ll first count the occurrences of words in titles by month.</p>
<figure>
<pre><code data-lang="r"><span>stories_per_month</span><span> </span><span>&lt;-</span><span> </span><span>stories</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>month</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarize</span><span>(</span><span>month_total</span><span> </span><span>=</span><span> </span><span>n</span><span>())</span><span>
</span><span>word_month_counts</span><span> </span><span>&lt;-</span><span> </span><span>title_words</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>word_total</span><span> </span><span>>=</span><span> </span><span>1000</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>word</span><span>,</span><span> </span><span>month</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>complete</span><span>(</span><span>word</span><span>,</span><span> </span><span>month</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>n</span><span> </span><span>=</span><span> </span><span>0</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>stories_per_month</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;month&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>percent</span><span> </span><span>=</span><span> </span><span>n</span><span> </span><span>/</span><span> </span><span>month_total</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>year</span><span> </span><span>=</span><span> </span><span>year</span><span>(</span><span>month</span><span>)</span><span> </span><span>+</span><span> </span><span>yday</span><span>(</span><span>month</span><span>)</span><span> </span><span>/</span><span> </span><span>365</span><span>)</span><span>
</span><span>word_month_counts</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 33,480 x 6
##     word      month     n month_total     percent     year
##    </code></pre></figure>