<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/reddit-responds-to-the-election/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://juliasilge.com/blog/Reddit-Responds/"> data science ish</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>It’s been about a month since the U.S. presidential election, with Donald Trump’s victory over Hillary Clinton coming as a surprise to most. Reddit user Jason Baumgartner collected and published <a target="_blank" rel="nofollow" href="https://www.reddit.com/r/datasets/comments/5ch2bq/reddit_raw_election_data_comments_and_submissions/">every submission and comment posted to Reddit</a> on the day of (and a bit surrounding) the U.S. election; let’s explore this data set and see what kinds of things we can learn.</p>
<h2>Data wrangling</h2>
<p>This first bit was the hardest part of this analysis for me, probably because I am not the most experienced JSON person out there. At first, I took an approach of reading in the lines of each text file and parsing each JSON object separately. I <a target="_blank" rel="nofollow" href="https://twitter.com/juliasilge/status/805915916961718272">complained about this on Twitter</a> and got several excellent recommendations of much better approaches, including using <code>stream_in</code> from the <a target="_blank" rel="nofollow" href="https://github.com/jeroenooms/jsonlite">jsonlite</a> package. This works way better and faster than what I was doing before, and now it is easy!</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>jsonlite</span><span>)</span><span>
</span><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>nov8_posts</span><span> </span><span>&lt;-</span><span> </span><span>stream_in</span><span>(</span><span>file</span><span>(</span><span>&quot;RS_2016-11-08&quot;</span><span>),</span><span>
</span><span>verbose</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>-</span><span>preview</span><span>,</span><span> </span><span>-</span><span>secure_media_embed</span><span>,</span><span> </span><span>-</span><span>media</span><span>,</span><span> </span><span>-</span><span>secure_media</span><span>,</span><span> </span><span>-</span><span>media_embed</span><span>)</span><span>
</span><span>nov9_posts</span><span> </span><span>&lt;-</span><span> </span><span>stream_in</span><span>(</span><span>file</span><span>(</span><span>&quot;RS_2016-11-09&quot;</span><span>),</span><span>
</span><span>verbose</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>-</span><span>preview</span><span>,</span><span> </span><span>-</span><span>secure_media_embed</span><span>,</span><span> </span><span>-</span><span>media</span><span>,</span><span> </span><span>-</span><span>secure_media</span><span>,</span><span> </span><span>-</span><span>media_embed</span><span>)</span><span>
</span><span>posts</span><span> </span><span>&lt;-</span><span> </span><span>bind_rows</span><span>(</span><span>nov8_posts</span><span>,</span><span> </span><span>nov9_posts</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>created_utc</span><span> </span><span>=</span><span> </span><span>as.POSIXct</span><span>(</span><span>created_utc</span><span>,</span><span> </span><span>origin</span><span> </span><span>=</span><span> </span><span>&quot;1970-01-01&quot;</span><span>,</span><span> </span><span>tz</span><span> </span><span>=</span><span> </span><span>&quot;UTC&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>created_utc</span><span> </span><span>></span><span> </span><span>as.POSIXct</span><span>(</span><span>&quot;2016-11-08 18:00:00&quot;</span><span>,</span><span> </span><span>tz</span><span> </span><span>=</span><span> </span><span>&quot;UTC&quot;</span><span>))</span></code></pre>
</figure>
<p>Notice here that I am using files from November 8 and 9 in UTC time and I’m filtering out some of the earlier posts. This will end up leaving me with 30 hours of Reddit posts starting at noon on Election Day in the Central Time Zone. Also notice that I am not using the files that include Reddit comments, only the parent submissions. I tried most of the following analysis with both submissions and comments, but the comments dominated the results and included lots of repeated words/phrases that obscured what we would like to see. For the approach I am taking here, it worked better to just use submissions.</p>
<h2>Finding the words</h2>
<p>The submissions include a title and sometimes also some text; sometimes Reddit posts are just the title. Let’s use <code>unnest_tokens</code> from the <a target="_blank" rel="nofollow" href="https://github.com/juliasilge/tidytext">tidytext</a> package to identify all the words in the title and text fields of the submissions and organize them into a tidy data structure.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>tidytext</span><span>)</span><span>
</span><span>posts</span><span> </span><span>&lt;-</span><span> </span><span>bind_rows</span><span>(</span><span>
</span><span>posts</span><span> </span><span>%>%</span><span> </span><span>unnest_tokens</span><span>(</span><span>word</span><span>,</span><span> </span><span>title</span><span>),</span><span>
</span><span>posts</span><span> </span><span>%>%</span><span> </span><span>unnest_tokens</span><span>(</span><span>word</span><span>,</span><span> </span><span>selftext</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>created_utc</span><span>,</span><span> </span><span>subreddit</span><span>,</span><span> </span><span>url</span><span>,</span><span> </span><span>word</span><span>)</span><span>
</span><span>head</span><span>(</span><span>posts</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##           created_utc  subreddit
## 1 2016-11-08 18:00:01 Ebay_deals
## 2 2016-11-08 18:00:01 Ebay_deals
## 3 2016-11-08 18:00:01 Ebay_deals
## 4 2016-11-08 18:00:01 Ebay_deals
## 5 2016-11-08 18:00:01 Ebay_deals
## 6 2016-11-08 18:00:01 Ebay_deals
##                                                 url   word
## 1 https://www.pinterest.com/pin/724587027517110980/ marble
## 2 https://www.pinterest.com/pin/724587027517110980/   grey
## 3 https://www.pinterest.com/pin/724587027517110980/ single
## 4 https://www.pinterest.com/pin/724587027517110980/ double
## 5 https://www.pinterest.com/pin/724587027517110980/  queen
## 6 https://www.pinterest.com/pin/724587027517110980/   king</code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>dim</span><span>(</span><span>posts</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [1] 17652591        4</code></pre>
</figure>
<p>That’s… almost 18 million rows. People on Reddit are busy.</p>
<h2>Which words changed in frequency the fastest?</h2>
<p>Right now we have a data frame that has each word on its own row, with an id (url), the time when it was posted, and the subreddit it came from. Let’s use dplyr operations to calculate how many times each word was mentioned in a particular unit of time, so we can model the change with time. We will calculate <code>minute_total</code>, the total words posted in that time unit so we can compare across times of day when people post different amounts, and <code>word_total</code>, the number of times that word was posted so we can filter out words that are not used much.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>lubridate</span><span>)</span><span>
</span><span>library</span><span>(</span><span>stringr</span><span>)</span><span>
</span><span>words_by_minute</span><span> </span><span>&lt;-</span><span> </span><span>posts</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>str_detect</span><span>(</span><span>word</span><span>,</span><span> </span><span>&quot;[a-z]&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>anti_join</span><span>(</span><span>data_frame</span><span>(</span><span>word</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;ref&quot;</span><span>)))</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>created</span><span> </span><span>=</span><span> </span><span>floor_date</span><span>(</span><span>created_utc</span><span>,</span><span> </span><span>unit</span><span> </span><span>=</span><span> </span><span>&quot;30 minutes&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>distinct</span><span>(</span><span>url</span><span>,</span><span> </span><span>word</span><span>,</span><span> </span><span>.keep_all</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>created</span><span>,</span><span> </span><span>word</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ungroup</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>created</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>minute_total</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>n</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>word</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>word_total</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>n</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ungroup</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>rename</span><span>(</span><span>count</span><span> </span><span>=</span><span> </span><span>n</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>word_total</span><span> </span><span>></span><span> </span><span>500</span><span>)</span><span>
</span><span>head</span><span>(</span><span>words_by_minute</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 6 × 5
##               created  word count minute_total word_total
##                &lt;dttm> &lt;chr> &lt;int>        &lt;int>      &lt;int>
## 1 2016-11-08 18:00:00 1080p    20       231072        652
## 2 2016-11-08 18:00:00  16gb    11       231072        559
## 3 2016-11-08 18:00:00   1st    20       231072        758
## 4 2016-11-08 18:00:00   2nd    21       231072        893
## 5 2016-11-08 18:00:00    2x    14       231072        722
## 6 2016-11-08 18:00:00    3d    12       231072        546</code></pre>
</figure>
<p>This is the data frame we can use for modeling. We can use <code>nest</code> from tidyr to make a data frame with a list column that contains the little miniature data frames for each word and then <code>map</code> from purrr to apply our modeling procedure to each of those little data frames inside our big data frame. Jenny Bryan has put together <a target="_blank" rel="nofollow" href="https://jennybc.github.io/purrr-tutorial/">some resources</a> on using purrr with list columns this way. This is count data (how many words were posted?) so let’s use <code>glm</code> for modeling.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>tidyr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>purrr</span><span>)</span><span>
</span><span>nested_models</span><span> </span><span>&lt;-</span><span> </span><span>words_by_minute</span><span> </span><span>%>%</span><span>
</span><span>nest</span><span>(</span><span>-</span><span>word</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>models</span><span> </span><span>=</span><span> </span><span>map</span><span>(</span><span>data</span><span>,</span><span> </span><span>~</span><span> </span><span>glm</span><span>(</span><span>cbind</span><span>(</span><span>count</span><span>,</span><span> </span><span>minute_total</span><span>)</span><span> </span><span>~</span><span> </span><span>created</span><span>,</span><span> </span><span>.</span><span>,</span><span> </span><span>family</span><span> </span><span>=</span><span> </span><span>&quot;binomial&quot;</span><span>)))</span></code></pre>
</figure>
<p>Now we can use <code>tidy</code> from broom to pull out the slopes for each of these models and find the important ones.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>broom</span><span>)</span><span>
</span><span>slopes</span><span> </span><span>&lt;-</span><span> </span><span>nested_models</span><span> </span><span>%>%</span><span>
</span><span>unnest</span><span>(</span><span>map</span><span>(</span><span>models</span><span>,</span><span> </span><span>tidy</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>term</span><span> </span><span>==</span><span> </span><span>&quot;created&quot;</span><span>)</span></code></pre>
</figure>
<p>Which words decreased in frequency of use the fastest during Election Day? Which words increased in use the fastest?</p>
<figure>
<pre><code data-lang="r"><span>slopes</span><span> </span><span>%>%</span><span> </span><span>arrange</span><span>(</span><span>estimate</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 891 × 6
##        word    term      estimate    std.error statistic       p.value
##       &lt;chr>   &lt;chr>         &lt;dbl>        &lt;dbl>     &lt;dbl>         &lt;dbl>
## 1   polling created -2.195136e-05 1.059199e-06 -20.72449  2.083382e-95
## 2     fraud created -2.022229e-05 1.335022e-06 -15.14753  7.866753e-52
## 3    voting created -1.788212e-05 5.644046e-07 -31.68316 2.651017e-220
## 4    ballot created -1.745250e-05 1.114699e-06 -15.65670  2.990299e-55
## 5        fl created -1.743009e-05 1.211878e-06 -14.38271  6.644389e-47
## 6   florida created -1.712721e-05 6.055286e-07 -28.28472 5.327752e-176
## 7      poll created -1.688132e-05 1.039083e-06 -16.24637  2.369894e-59
## 8     voter created -1.675932e-05 9.359359e-07 -17.90649  1.049562e-71
## 9     polls created -1.615500e-05 7.058145e-07 -22.88845 6.054992e-116
## 10 virginia created -1.493344e-05 1.435504e-06 -10.40293  2.404218e-25
## # ... with 881 more rows</code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>slopes</span><span> </span><span>%>%</span><span> </span><span>arrange</span><span>(</span><span>desc</span><span>(</span><span>estimate</span><span>))</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 891 × 6
##        word    term     estimate    std.error statistic       p.value
##       &lt;chr>   &lt;chr>        &lt;dbl>        &lt;dbl>     &lt;dbl>         &lt;dbl>
## 1   trump’s created 1.758326e-05 1.430901e-06 12.288244  1.047563e-34
## 2      flex created 1.416875e-05 1.290926e-06 10.975647  5.004609e-28
## 3     elect created 1.390813e-05 8.378822e-07 16.599146  7.068929e-62
## 4        da created 1.281702e-05 1.037396e-06 12.354986  4.578350e-35
## 5  congress created 1.225188e-05 1.280413e-06  9.568696  1.082627e-21
## 6   trump's created 1.200055e-05 5.499216e-07 21.822297 1.425196e-105
## 7  policies created 1.156944e-05 1.276465e-06  9.063655  1.261512e-19
## 8  liberals created 1.151984e-05 1.151751e-06 10.002020  1.493199e-23
## 9   climate created 1.092999e-05 1.269681e-06  8.608452  7.405384e-18
## 10   policy created 9.879997e-06 1.038524e-06  9.513494  1.843631e-21
## # ... with 881 more rows</code></pre>
</figure>
<p>Let’s plot these words.</p>
<figure>
<pre><code data-lang="r"><span>top_slopes</span><span> </span><span>&lt;-</span><span> </span><span>slopes</span><span> </span><span>%>%</span><span>
</span><span>top_n</span><span>(</span><span>10</span><span>,</span><span> </span><span>estimate</span><span>)</span><span>
</span><span>words_by_minute</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>top_slopes</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;word&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>created</span><span> </span><span>=</span><span> </span><span>with_tz</span><span>(</span><span>created</span><span>,</span><span> </span><span>tz</span><span> </span><span>=</span><span> </span><span>&quot;America/Chicago&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>created</span><span>,</span><span> </span><span>count</span><span>/</span><span>minute_total</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>word</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1.3</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Time (Central Time Zone)&quot;</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Word frequency&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Words associated with Trump and policies increased in frequency&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Trending words on Reddit on Election Day&quot;</span><span>)</span></code></pre>
</figure>
<p><img alt="center" src="https://i0.wp.com/juliasilge.com/figs/2016-12-06-Reddit-Responds/unnamed-chunk-8-1.png?w=456" data-recalc-dims="1"></p>
<p>There are lots of election-related words here, like “elect”, “liberals”, and “policies”. In fact, I think all of these words are conceivably related to the election with the exception of “flex”. I looked at some of the posts with “flex” in them and they were in fact not election-related. I had a hard time deciphering what they <em>were</em> about, but my best guess is either a) fantasy football or b) some kind of gaming. Why do we see “Trump’s” on this plot twice? It is because there is more than one way of encoding an apostrophe. You can see it on the legend if you look closely.</p>
<p>We don’t see Trump’s name by itself on this plot. How far off from being a top word, by my definition here, was it?</p>
<figure>
<pre><code data-lang="r"><span>slopes</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>word</span><span> </span><span>==</span><span> </span><span>&quot;trump&quot;</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 1 × 6
##    word    term     estimate    std.error statistic      p.value
##   &lt;chr>   &lt;chr>        &lt;dbl>        &lt;dbl>     &lt;dbl>        &lt;dbl>
## 1 trump created 2.328675e-06 1.512196e-07   15.3993 1.654546e-53</code></pre>
</figure>
<p>Trump must have been being discussed at a high level already, so the change was not as big as for the word “Trump’s”.</p>
<p>What about the words that dropped in use the most during this day?</p>
<figure>
<pre><code data-lang="r"><span>low_slopes</span><span> </span><span>&lt;-</span><span> </span><span>slopes</span><span> </span><span>%>%</span><span>
</span><span>top_n</span><span>(</span><span>-10</span><span>,</span><span> </span><span>estimate</span><span>)</span><span>
</span><span>words_by_minute</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>low_slopes</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;word&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>created</span><span> </span><span>=</span><span> </span><span>with_tz</span><span>(</span><span>created</span><span>,</span><span> </span><span>tz</span><span> </span><span>=</span><span> </span><span>&quot;America/Chicago&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>created</span><span>,</span><span> </span><span>count</span><span>/</span><span>minute_total</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>word</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1.3</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Time (Central Time Zone)&quot;</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Word frequency&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Word associated with voting and polls decreased in frequency&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Trending words on Reddit on Election Day&quot;</span><span>)</span></code></pre>
</figure>
<p><img alt="center" src="https://i1.wp.com/juliasilge.com/figs/2016-12-06-Reddit-Responds/unnamed-chunk-10-1.png?w=456" data-recalc-dims="1"></p>
<p>These are maybe even <em>more</em> interesting to me. Look at that spike for Florida the night of November 8 when it seemed like there might be flashbacks to 2000 or something. And people’s interest in discussing voters/voting, polls/polling, and fraud dropped off precipitously as Trump’s victory became obvious.</p>
<h2>Which subreddits demonstrated the most change in sentiment?</h2>
<p>We have looked at which words changed most quickly in use on Election Day; now let’s take a look at changes in sentiment. Are there subreddits that exhibited changes in sentiment over the course of this time period? To look at this, we’ll take a bigger time period (2 hours instead of 30 minutes) since the words with measured sentiment are only a subset of all words. Much of the rest of these dplyr operations are similar. We can use <code>inner_join</code> to do the sentiment analysis, and then calculate the sentiment content of each board in each time period.</p>
<figure>
<pre><code data-lang="r"><span>sentiment_by_minute</span><span> </span><span>&lt;-</span><span> </span><span>posts</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>created</span><span> </span><span>=</span><span> </span><span>floor_date</span><span>(</span><span>created_utc</span><span>,</span><span> </span><span>unit</span><span> </span><span>=</span><span> </span><span>&quot;2 hours&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>distinct</span><span>(</span><span>url</span><span>,</span><span> </span><span>word</span><span>,</span><span> </span><span>.keep_all</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ungroup</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>subreddit</span><span>,</span><span> </span><span>created</span><span>,</span><span> </span><span>word</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>created</span><span>,</span><span> </span><span>subreddit</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>minute_total</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>n</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>subreddit</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>subreddit_total</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>n</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ungroup</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>subreddit_total</span><span> </span><span>></span><span> </span><span>1000</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>get_sentiments</span><span>(</span><span>&quot;afinn&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>subreddit</span><span>,</span><span> </span><span>created</span><span>,</span><span> </span><span>minute_total</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarize</span><span>(</span><span>score</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>n</span><span> </span><span>*</span><span> </span><span>score</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ungroup</span><span>()</span><span>
</span><span>head</span><span>(</span><span>sentiment_by_minute</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 6 × 4
##   subreddit             created minute_total score
##       &lt;chr>              &lt;dttm>        &lt;int> &lt;int>
## 1     1liga 2016-11-08 20:00:00           20     2
## 2     1liga 2016-11-09 02:00:00          135    -7
## 3     1liga 2016-11-09 06:00:00          187    -9
## 4     1liga 2016-11-09 08:00:00          233    -8
## 5     1liga 2016-11-09 10:00:00          155    -2
## 6     1liga 2016-11-09 12:00:00          110    -6</code></pre>
</figure>
<p>Let’s again use <code>nest</code>, but this time we’ll <code>nest</code> by subreddit instead of word. This sentiment score is not really count data (since it can be negative) so we’ll use regular old <code>lm</code> here.</p>
<figure>
<pre><code data-lang="r"><span>nested_models</span><span> </span><span>&lt;-</span><span> </span><span>sentiment_by_minute</span><span> </span><span>%>%</span><span>
</span><span>nest</span><span>(</span><span>-</span><span>subreddit</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>models</span><span> </span><span>=</span><span> </span><span>map</span><span>(</span><span>data</span><span>,</span><span> </span><span>~</span><span> </span><span>lm</span><span>(</span><span>score</span><span>/</span><span>minute_total</span><span> </span><span>~</span><span> </span><span>created</span><span>,</span><span> </span><span>.</span><span>)))</span></code></pre>
</figure>
<p>Let’s again use <code>unnest</code>, <code>map</code>, and <code>tidy</code> to extract out the slopes from the linear models.</p>
<figure>
<pre><code data-lang="r"><span>slopes</span><span> </span><span>&lt;-</span><span> </span><span>nested_models</span><span> </span><span>%>%</span><span>
</span><span>unnest</span><span>(</span><span>map</span><span>(</span><span>models</span><span>,</span><span> </span><span>tidy</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>term</span><span> </span><span>==</span><span> </span><span>&quot;created&quot;</span><span>)</span></code></pre>
</figure>
<p>Which subreddits exhibited the biggest changes in sentiment, in either direction?</p>
<figure>
<pre><code data-lang="r"><span>slopes</span><span> </span><span>%>%</span><span> </span><span>arrange</span><span>(</span><span>estimate</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 64 × 6
##            subreddit    term      estimate    std.error statistic     p.value
##                &lt;chr>   &lt;chr>         &lt;dbl>        &lt;dbl>     &lt;dbl>       &lt;dbl>
## 1        aznidentity created -2.941746e-06 1.129234e-06 -2.605081 0.040386018
## 2             ainbow created -2.576480e-06 8.841939e-07 -2.913932 0.017200835
## 3        TheBluePill created -2.233135e-06 6.868578e-07 -3.251233 0.009977663
## 4    muacjdiscussion created -2.165141e-06 4.838494e-07 -4.474823 0.020801581
## 5            parrots created -2.105393e-06 7.592753e-07 -2.772898 0.024188248
## 6         needadvice created -1.610350e-06 5.277117e-07 -3.051572 0.018542125
## 7        sweepstakes created -1.458346e-06 5.576759e-07 -2.615042 0.022590562
## 8  RandomActsofCards created -1.421334e-06 5.556567e-07 -2.557935 0.026616327
## 9       csgogambling created -1.397576e-06 5.619925e-07 -2.486823 0.027257440
## 10             1liga created -1.305359e-06 4.842398e-07 -2.695688 0.027255456
## # ... with 54 more rows</code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>slopes</span><span> </span><span>%>%</span><span> </span><span>arrange</span><span>(</span><span>desc</span><span>(</span><span>estimate</span><span>))</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 64 × 6
##            subreddit    term     estimate    std.error statistic     p.value
##                &lt;chr>   &lt;chr>        &lt;dbl>        &lt;dbl>     &lt;dbl>       &lt;dbl>
## 1  shoudvebeenbernie created 2.792205e-06 7.339321e-07  3.804446 0.008921294
## 2        TheDickShow created 2.189643e-06 7.900087e-07  2.771670 0.019730686
## 3            orlando created 1.860757e-06 7.313366e-07  2.544324 0.034477936
## 4    DoesAnybodyElse created 1.846480e-06 5.773406e-07  3.198251 0.007657351
## 5    playrustservers created 1.553049e-06 5.623340e-07  2.761791 0.032776294
## 6    GoogleCardboard created 1.449634e-06 5.215326e-07  2.779565 0.032015397
## 7    morbidquestions created 1.423202e-06 4.865300e-07  2.925208 0.019138028
## 8          TrueDoTA2 created 1.388537e-06 3.592288e-07  3.865329 0.011816017
## 9        JapanTravel created 1.365982e-06 3.702370e-07  3.689479 0.005001422
## 10           editors created 1.253335e-06 4.080638e-07  3.071420 0.011812188
## # ... with 54 more rows</code></pre>
</figure>
<p>Let’s plot these!</p>
<figure>
<pre><code data-lang="r"><span>top_slopes</span><span> </span><span>&lt;-</span><span> </span><span>slopes</span><span> </span><span>%>%</span><span>
</span><span>top_n</span><span>(</span><span>10</span><span>,</span><span> </span><span>estimate</span><span>)</span><span>
</span><span>sentiment_by_minute</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>top_slopes</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;subreddit&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>created</span><span> </span><span>=</span><span> </span><span>with_tz</span><span>(</span><span>created</span><span>,</span><span> </span><span>tz</span><span> </span><span>=</span><span> </span><span>&quot;America/Chicago&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>created</span><span>,</span><span> </span><span>score</span><span>/</span><span>minute_total</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>subreddit</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1.3</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Time (Central Time Zone)&quot;</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Sentiment score&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;These subreddits increased in sentiment the most&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Sentiment on Subreddits on Election Day&quot;</span><span>)</span></code></pre>
</figure>
<p><img alt="center" src="https://i1.wp.com/juliasilge.com/figs/2016-12-06-Reddit-Responds/unnamed-chunk-15-1.png?w=456" data-recalc-dims="1"></p>
<p>These relationships are much noisier than the relationships with words were, and you might notice that some p-values are getting kind of high (no adjustment for multiple comparisons has been performed). Also, these subreddits are less related to the election than the quickly changing words were. Really only the shouldvebeenbernie subreddit is that political here.</p>
<figure>
<pre><code data-lang="r"><span>low_slopes</span><span> </span><span>&lt;-</span><span> </span><span>slopes</span><span> </span><span>%>%</span><span>
</span><span>top_n</span><span>(</span><span>-10</span><span>,</span><span> </span><span>estimate</span><span>)</span><span>
</span><span>sentiment_by_minute</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>low_slopes</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;subreddit&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>created</span><span> </span><span>=</span><span> </span><span>with_tz</span><span>(</span><span>created</span><span>,</span><span> </span><span>tz</span><span> </span><span>=</span><span> </span><span>&quot;America/Chicago&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>created</span><span>,</span><span> </span><span>score</span><span>/</span><span>minute_total</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>subreddit</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1.3</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Time (Central Time Zone)&quot;</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Sentiment score&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;These subreddits decreased in sentiment the most&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Sentiment on Subreddits on Election Day&quot;</span><span>)</span></code></pre>
</figure>
<p><img alt="center" src="https://i0.wp.com/juliasilge.com/figs/2016-12-06-Reddit-Responds/unnamed-chunk-16-1.png?w=456" data-recalc-dims="1"></p>
<p>Again, we see that not really any of these are specifically political, although I coudld image that the aznidentity subreddit (Asian identity board) and the ainbow subreddit (LGBT board) could have been feeling down after Trump’s election. The 1liga board is a German language board and ended up here because it used the word “die” a lot. In case you are wondering, the parrots subreddit is, in fact, about parrots; hopefully nothing too terrible was happening to the parrots on Election Day.</p>
<h2>Which subreddits have the most dramatic word use?</h2>
<p>Those last plots demonstrated with subreddits were changing in sentiment the fastest around the time of the election, but perhaps we would like to know which subreddits used the largest proportion of high or low sentiment words overall during this time period. To do that, we don’t need to keep track of the timestamp of the posts. Instead, we just need to <code>count</code> by subreddit and word, then use <code>inner_join</code> to find a sentiment score.</p>
<figure>
<pre><code data-lang="r"><span>sentiment_by_subreddit</span><span> </span><span>&lt;-</span><span> </span><span>posts</span><span> </span><span>%>%</span><span>
</span><span>distinct</span><span>(</span><span>url</span><span>,</span><span> </span><span>word</span><span>,</span><span> </span><span>.keep_all</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>subreddit</span><span>,</span><span> </span><span>word</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ungroup</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>get_sentiments</span><span>(</span><span>&quot;afinn&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>subreddit</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarize</span><span>(</span><span>score</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>score</span><span> </span><span>*</span><span> </span><span>n</span><span>)</span><span> </span><span>/</span><span> </span><span>sum</span><span>(</span><span>n</span><span>))</span></code></pre>
</figure>
<p>I would print some out for you, or plot them or something, but they are almost all extremely NSFW, both the positive and negative sentiment subreddits. I’m sure you can use your imagination.</p>
<h2>The End</h2>
<p>This is just one approach to take with this extremely extensive data set. There is still lots and lots more that could be done with it. I first saw this data set via Jeremy Singer-Vine’s <a target="_blank" rel="nofollow" href="https://tinyletter.com/data-is-plural">Data Is Plural newsletter</a>; this newsletter is an excellent resource and I highly recommend it. The R Markdown file used to make this blog post is available <a target="_blank" rel="nofollow" href="https://github.com/juliasilge/juliasilge.github.io/blob/master/_R/2016-12-06-Reddit-Responds.Rmd">here</a>. I am very happy to hear feedback or questions!</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://juliasilge.com/blog/Reddit-Responds/"> data science ish</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/2s0JONGO13s"></a></div>