<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/characterizing-twitter-followers-with-tidytext/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://shiring.github.io/text_analysis/2017/06/28/twitter_post"> Shirin's playgRound</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Lately, I have been more and more taken with tidy principles of data analysis. They are elegant and make analyses clearer and easier to comprehend. Following the <strong>tidyverse</strong> and <strong>ggraph</strong>, I have been quite intrigued by applying tidy principles to text analysis with <a rel="nofollow" href="https://cran.r-project.org/web/packages/tidytext/vignettes/tidytext.html" target="_blank">Julia Silge and David Robinson’s <strong>tidytext</strong></a>.</p>
<p>In this post, I will explore <strong>tidytext</strong> with an analysis of my Twitter followers’ descriptions to try and learn more about the people who are interested in my tweets, which are mainly about Data Science and Machine Learning.</p>
<p>Resources I found useful for this analysis were <a rel="nofollow" href="http://www.rdatamining.com/docs/twitter-analysis-with-r" target="_blank">http://www.rdatamining.com/docs/twitter-analysis-with-r</a> and <a rel="nofollow" href="http://tidytextmining.com/tidytext.html" target="_blank">http://tidytextmining.com/tidytext.html</a></p>
<p></p>
<h2>Retrieving Twitter data</h2>
<p>I am using <strong>twitteR</strong> to retrieve data from Twitter (I have also tried <strong>rtweet</strong> but for some reason, my API key, secret and token (that worked with <strong>twitteR</strong>) resulted in a “failed to authorize” error with <strong>rtweet</strong>’s functions).</p>
<div>
<pre><code><span>library</span><span>(</span><span>twitteR</span><span>)</span><span>
</span></code></pre>
</div>
<p>Once we have <a rel="nofollow" href="https://dev.twitter.com/rest/public" target="_blank">set up our Twitter REST API</a>, we get the necessary information to authenticate our access.</p>
<div>
<pre><code><span>consumerKey</span><span> </span><span>=</span><span> </span><span>&quot;INSERT KEY HERE&quot;</span><span>
</span><span>consumerSecret</span><span> </span><span>=</span><span> </span><span>&quot;INSERT SECRET KEY HERE&quot;</span><span>
</span><span>accessToken</span><span> </span><span>=</span><span> </span><span>&quot;INSERT TOKEN HERE&quot;</span><span>
</span><span>accessSecret</span><span> </span><span>=</span><span> </span><span>&quot;INSERT SECRET TOKEN HERE&quot;</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>options</span><span>(</span><span>httr_oauth_cache</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>setup_twitter_oauth</span><span>(</span><span>consumer_key</span><span> </span><span>=</span><span> </span><span>consumerKey</span><span>,</span><span> </span><span>consumer_secret</span><span> </span><span>=</span><span> </span><span>consumerSecret</span><span>,</span><span> </span><span>access_token</span><span> </span><span>=</span><span> </span><span>accessToken</span><span>,</span><span> </span><span>access_secret</span><span> </span><span>=</span><span> </span><span>accessSecret</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<p>Now, we can access information from Twitter, like timeline tweets, user timelines, mentions, tweets &amp; retweets, followers, etc.</p>
<p>All the following datasets were retrieved on June 7th 2017, converted to a data frame for tidy analysis and saved for later use:</p>
<ul>
<li>the last 3200 tweets on my timeline</li>
</ul>
<div>
<pre><code><span>my_name</span><span> </span><span>&lt;-</span><span> </span><span>userTimeline</span><span>(</span><span>&quot;ShirinGlander&quot;</span><span>,</span><span> </span><span>n</span><span> </span><span>=</span><span> </span><span>3200</span><span>,</span><span> </span><span>includeRts</span><span>=</span><span>TRUE</span><span>)</span><span>
</span><span>my_name_df</span><span> </span><span>&lt;-</span><span> </span><span>twListToDF</span><span>(</span><span>my_name</span><span>)</span><span>
</span><span>save</span><span>(</span><span>my_name_df</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;my_name.RData&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<ul>
<li>my last 3200 mentions and retweets</li>
</ul>
<div>
<pre><code><span>my_mentions</span><span> </span><span>&lt;-</span><span> </span><span>mentions</span><span>(</span><span>n</span><span> </span><span>=</span><span> </span><span>3200</span><span>)</span><span>
</span><span>my_mentions_df</span><span> </span><span>&lt;-</span><span> </span><span>twListToDF</span><span>(</span><span>my_mentions</span><span>)</span><span>
</span><span>save</span><span>(</span><span>my_mentions_df</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;my_mentions.RData&quot;</span><span>)</span><span>
</span><span>my_retweets</span><span> </span><span>&lt;-</span><span> </span><span>retweetsOfMe</span><span>(</span><span>n</span><span> </span><span>=</span><span> </span><span>3200</span><span>)</span><span>
</span><span>my_retweets_df</span><span> </span><span>&lt;-</span><span> </span><span>twListToDF</span><span>(</span><span>my_retweets</span><span>)</span><span>
</span><span>save</span><span>(</span><span>my_retweets_df</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;my_retweets.RData&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<ul>
<li>the last 3200 tweets to me</li>
</ul>
<div>
<pre><code><span>tweetstome</span><span> </span><span>&lt;-</span><span> </span><span>searchTwitter</span><span>(</span><span>&quot;@ShirinGlander&quot;</span><span>,</span><span> </span><span>n</span><span> </span><span>=</span><span> </span><span>3200</span><span>)</span><span>
</span><span>tweetstome_df</span><span> </span><span>&lt;-</span><span> </span><span>twListToDF</span><span>(</span><span>tweetstome</span><span>)</span><span>
</span><span>save</span><span>(</span><span>tweetstome_df</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;tweetstome.RData&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<ul>
<li>my friends and followers</li>
</ul>
<div>
<pre><code><span>user</span><span> </span><span>&lt;-</span><span> </span><span>getUser</span><span>(</span><span>&quot;ShirinGlander&quot;</span><span>)</span><span>
</span><span>friends</span><span> </span><span>&lt;-</span><span> </span><span>user</span><span>$</span><span>getFriends</span><span>()</span><span> </span><span># who I follow
</span><span>friends_df</span><span> </span><span>&lt;-</span><span> </span><span>twListToDF</span><span>(</span><span>friends</span><span>)</span><span>
</span><span>save</span><span>(</span><span>friends_df</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;my_friends.RData&quot;</span><span>)</span><span>
</span><span>followers</span><span> </span><span>&lt;-</span><span> </span><span>user</span><span>$</span><span>getFollowers</span><span>()</span><span> </span><span># my followers
</span><span>followers_df</span><span> </span><span>&lt;-</span><span> </span><span>twListToDF</span><span>(</span><span>followers</span><span>)</span><span>
</span><span>save</span><span>(</span><span>followers_df</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;my_followers.RData&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<h2>Analyzing friends and followers</h2>
<p>In this post, I will have a look at my friends and followers.</p>
<div>
<pre><code><span>load</span><span>(</span><span>&quot;my_friends.RData&quot;</span><span>)</span><span>
</span><span>load</span><span>(</span><span>&quot;my_followers.RData&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p>I am going to use packages from the <strong>tidyverse</strong> (<strong>tidyquant</strong> for plotting).</p>
<div>
<pre><code><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tidyquant</span><span>)</span><span>
</span></code></pre>
</div>
<ul>
<li>
<p>Number of friends (who I follow on Twitter): 225</p>
</li>
<li>
<p>Number of followers (who follows me on Twitter): 324</p>
</li>
<li>
<p>Number of friends who are also followers: 97</p>
</li>
</ul>
<p></p>
<h3>What languages do my followers speak?</h3>
<p>One of the columns describing my followers is which language they have set for their Twitter account. Not surprisingly, English is by far the most predominant language of my followers, followed by German, Spanish and French.</p>
<div>
<pre><code><span>followers_df</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>lang</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>droplevels</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>reorder</span><span>(</span><span>lang</span><span>,</span><span> </span><span>desc</span><span>(</span><span>n</span><span>)),</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>n</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>stat</span><span> </span><span>=</span><span> </span><span>&quot;identity&quot;</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>],</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>],</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_tq</span><span>()</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>axis.text.x</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>angle</span><span> </span><span>=</span><span> </span><span>45</span><span>,</span><span> </span><span>vjust</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>hjust</span><span> </span><span>=</span><span> </span><span>1</span><span>))</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;language ISO 639-1 code&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;number of followers&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i1.wp.com/shiring.github.io/text_analysis/2017/06/28/twitter_post_files/figure-markdown_github/unnamed-chunk-14-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h3>Who are my most “influential” followers (i.e. followers with the biggest network)?</h3>
<p>I also have information about the number of followers that each of my followers have (2nd degree followers). Most of my followers are followed by up to ~ 1000 people, while only a few have a very large network.</p>
<div>
<pre><code><span>followers_df</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>log2</span><span>(</span><span>followersCount</span><span>)))</span><span> </span><span>+</span><span>
</span><span>geom_density</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>],</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>],</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_tq</span><span>()</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;log2 of number of followers&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;density&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/text_analysis/2017/06/28/twitter_post_files/figure-markdown_github/unnamed-chunk-16-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h3>How active are my followers (i.e. how often do they tweet)</h3>
<p>The followers data frame also tells me how many statuses <a rel="nofollow" href="https://dev.twitter.com/rest/reference/get/statuses/user_timeline" target="_blank">(i.e. tweets)</a> each of followers have. To make the numbers comparable, I am normalizing them by the number of days that they have had their accounts to calculate the average number of tweets per day.</p>
<div>
<pre><code><span>followers_df</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>date</span><span> </span><span>=</span><span> </span><span>as.Date</span><span>(</span><span>created</span><span>,</span><span> </span><span>format</span><span> </span><span>=</span><span> </span><span>&quot;%Y-%m-%d&quot;</span><span>),</span><span>
</span><span>today</span><span> </span><span>=</span><span> </span><span>as.Date</span><span>(</span><span>&quot;2017-06-07&quot;</span><span>,</span><span> </span><span>format</span><span> </span><span>=</span><span> </span><span>&quot;%Y-%m-%d&quot;</span><span>),</span><span>
</span><span>days</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>today</span><span> </span><span>-</span><span> </span><span>date</span><span>),</span><span>
</span><span>statusesCount_pDay</span><span> </span><span>=</span><span> </span><span>statusesCount</span><span> </span><span>/</span><span> </span><span>days</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>log2</span><span>(</span><span>statusesCount_pDay</span><span>)))</span><span> </span><span>+</span><span>
</span><span>geom_density</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>],</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>],</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_tq</span><span>()</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i1.wp.com/shiring.github.io/text_analysis/2017/06/28/twitter_post_files/figure-markdown_github/unnamed-chunk-17-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h4>Who are my followers with the biggest network and who tweet the most?</h4>
<div>
<pre><code><span>followers_df</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>date</span><span> </span><span>=</span><span> </span><span>as.Date</span><span>(</span><span>created</span><span>,</span><span> </span><span>format</span><span> </span><span>=</span><span> </span><span>&quot;%Y-%m-%d&quot;</span><span>),</span><span>
</span><span>today</span><span> </span><span>=</span><span> </span><span>as.Date</span><span>(</span><span>&quot;2017-06-07&quot;</span><span>,</span><span> </span><span>format</span><span> </span><span>=</span><span> </span><span>&quot;%Y-%m-%d&quot;</span><span>),</span><span>
</span><span>days</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>today</span><span> </span><span>-</span><span> </span><span>date</span><span>),</span><span>
</span><span>statusesCount_pDay</span><span> </span><span>=</span><span> </span><span>statusesCount</span><span> </span><span>/</span><span> </span><span>days</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>screenName</span><span>,</span><span> </span><span>followersCount</span><span>,</span><span> </span><span>statusesCount_pDay</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>desc</span><span>(</span><span>followersCount</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>top_n</span><span>(</span><span>10</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##         screenName followersCount statusesCount_pDay
## 1        dr_morton         150937           71.35193
## 2    Scientists4EU          66117           17.64389
## 3       dr_morton_          63467           46.57763
## 4   NewScienceWrld          60092           54.65874
## 5     RubenRabines          42286           25.99592
## 6  machinelearnbot          27427          204.67061
## 7  BecomingDataSci          16807           25.24069
## 8       joelgombin           6566           21.24094
## 9    renato_umeton           1998           19.58387
## 10 FranPatogenLoco            311           28.92593
</code></pre>
</div>
<div>
<pre><code><span>followers_df</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>date</span><span> </span><span>=</span><span> </span><span>as.Date</span><span>(</span><span>created</span><span>,</span><span> </span><span>format</span><span> </span><span>=</span><span> </span><span>&quot;%Y-%m-%d&quot;</span><span>),</span><span>
</span><span>today</span><span> </span><span>=</span><span> </span><span>as.Date</span><span>(</span><span>&quot;2017-06-07&quot;</span><span>,</span><span> </span><span>format</span><span> </span><span>=</span><span> </span><span>&quot;%Y-%m-%d&quot;</span><span>),</span><span>
</span><span>days</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>today</span><span> </span><span>-</span><span> </span><span>date</span><span>),</span><span>
</span><span>statusesCount_pDay</span><span> </span><span>=</span><span> </span><span>statusesCount</span><span> </span><span>/</span><span> </span><span>days</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>screenName</span><span>,</span><span> </span><span>followersCount</span><span>,</span><span> </span><span>statusesCount_pDay</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>desc</span><span>(</span><span>statusesCount_pDay</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>top_n</span><span>(</span><span>10</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##         screenName followersCount statusesCount_pDay
## 1  machinelearnbot          27427          204.67061
## 2        dr_morton         150937           71.35193
## 3   NewScienceWrld          60092           54.65874
## 4       dr_morton_          63467           46.57763
## 5  FranPatogenLoco            311           28.92593
## 6     RubenRabines          42286           25.99592
## 7  BecomingDataSci          16807           25.24069
## 8       joelgombin           6566           21.24094
## 9    renato_umeton           1998           19.58387
## 10   Scientists4EU          66117           17.64389
</code></pre>
</div>
<p></p>
<h4>Is there a correlation between number of followers and number of tweets?</h4>
<p>Indeed, there seems to be a correlation that users with many followers also tend to tweet more often.</p>
<div>
<pre><code><span>followers_df</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>date</span><span> </span><span>=</span><span> </span><span>as.Date</span><span>(</span><span>created</span><span>,</span><span> </span><span>format</span><span> </span><span>=</span><span> </span><span>&quot;%Y-%m-%d&quot;</span><span>),</span><span>
</span><span>today</span><span> </span><span>=</span><span> </span><span>as.Date</span><span>(</span><span>&quot;2017-06-07&quot;</span><span>,</span><span> </span><span>format</span><span> </span><span>=</span><span> </span><span>&quot;%Y-%m-%d&quot;</span><span>),</span><span>
</span><span>days</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>today</span><span> </span><span>-</span><span> </span><span>date</span><span>),</span><span>
</span><span>statusesCount_pDay</span><span> </span><span>=</span><span> </span><span>statusesCount</span><span> </span><span>/</span><span> </span><span>days</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>followersCount</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>statusesCount_pDay</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>days</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_smooth</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;lm&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_color_continuous</span><span>(</span><span>low</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>],</span><span> </span><span>high</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>2</span><span>])</span><span> </span><span>+</span><span>
</span><span>theme_tq</span><span>()</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/text_analysis/2017/06/28/twitter_post_files/figure-markdown_github/unnamed-chunk-20-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h3>Tidy text analysis</h3>
<p>Next, I want to know more about my followers by analyzing their Twitter descriptions with the <strong>tidytext</strong> package.</p>
<div>
<pre><code><span>library</span><span>(</span><span>tidytext</span><span>)</span><span>
</span><span>library</span><span>(</span><span>SnowballC</span><span>)</span><span>
</span></code></pre>
</div>
<p>To prepare the data, I am going to unnest the words (or tokens) in the user descriptions, convert them to the word stem, remove stop words and urls.</p>
<div>
<pre><code><span>data</span><span>(</span><span>stop_words</span><span>)</span><span>
</span><span>tidy_descr</span><span> </span><span>&lt;-</span><span> </span><span>followers_df</span><span> </span><span>%>%</span><span>
</span><span>unnest_tokens</span><span>(</span><span>word</span><span>,</span><span> </span><span>description</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>word_stem</span><span> </span><span>=</span><span> </span><span>wordStem</span><span>(</span><span>word</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>anti_join</span><span>(</span><span>stop_words</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;word&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>!</span><span>grepl</span><span>(</span><span>&quot;\\.|http&quot;</span><span>,</span><span> </span><span>word</span><span>))</span><span>
</span></code></pre>
</div>
<p></p>
<h4>What are the most commonly used words in my followers’ descriptions?</h4>
<p>The first question I want to ask is what words are most common in my followers’ descriptions.</p>
<p>Not surprisingly, the most common word is “data”. I do tweet mostly about data related topics, so it makes sense that my followers are mostly likeminded. The rest is also related to data science, machine learning and R.</p>
<div>
<pre><code><span>tidy_descr</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>word_stem</span><span>,</span><span> </span><span>sort</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>n</span><span> </span><span>></span><span> </span><span>20</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>reorder</span><span>(</span><span>word_stem</span><span>,</span><span> </span><span>n</span><span>),</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>n</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_col</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>],</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>],</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>)</span><span> </span><span>+</span><span>
</span><span>coord_flip</span><span>()</span><span> </span><span>+</span><span>
</span><span>theme_tq</span><span>()</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;count of word stem in all followers' descriptions&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/text_analysis/2017/06/28/twitter_post_files/figure-markdown_github/unnamed-chunk-27-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<p>This, we can also show with a word cloud.</p>
<div>
<pre><code><span>library</span><span>(</span><span>wordcloud</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tm</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>tidy_descr</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>word_stem</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>word_stem</span><span> </span><span>=</span><span> </span><span>removeNumbers</span><span>(</span><span>word_stem</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>with</span><span>(</span><span>wordcloud</span><span>(</span><span>word_stem</span><span>,</span><span> </span><span>n</span><span>,</span><span> </span><span>max.words</span><span> </span><span>=</span><span> </span><span>100</span><span>,</span><span> </span><span>colors</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()))</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i2.wp.com/shiring.github.io/text_analysis/2017/06/28/twitter_post_files/figure-markdown_github/unnamed-chunk-29-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<p>Instead of looking for the most common words, we can also look for the most common ngrams: here, for the most common word pairs (bigrams) in my followers’ descriptions.</p>
<div>
<pre><code><span>tidy_descr_ngrams</span><span> </span><span>&lt;-</span><span> </span><span>followers_df</span><span> </span><span>%>%</span><span>
</span><span>unnest_tokens</span><span>(</span><span>bigram</span><span>,</span><span> </span><span>description</span><span>,</span><span> </span><span>token</span><span> </span><span>=</span><span> </span><span>&quot;ngrams&quot;</span><span>,</span><span> </span><span>n</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>!</span><span>grepl</span><span>(</span><span>&quot;\\.|http&quot;</span><span>,</span><span> </span><span>bigram</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>separate</span><span>(</span><span>bigram</span><span>,</span><span> </span><span>c</span><span>(</span><span>&quot;word1&quot;</span><span>,</span><span> </span><span>&quot;word2&quot;</span><span>),</span><span> </span><span>sep</span><span> </span><span>=</span><span> </span><span>&quot; &quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>!</span><span>word1</span><span> </span><span>%in%</span><span> </span><span>stop_words</span><span>$</span><span>word</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>!</span><span>word2</span><span> </span><span>%in%</span><span> </span><span>stop_words</span><span>$</span><span>word</span><span>)</span><span>
</span><span>bigram_counts</span><span> </span><span>&lt;-</span><span> </span><span>tidy_descr_ngrams</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>word1</span><span>,</span><span> </span><span>word2</span><span>,</span><span> </span><span>sort</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>bigram_counts</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>n</span><span> </span><span>></span><span> </span><span>10</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>reorder</span><span>(</span><span>word1</span><span>,</span><span> </span><span>-</span><span>n</span><span>),</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>reorder</span><span>(</span><span>word2</span><span>,</span><span> </span><span>-</span><span>n</span><span>),</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>n</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_tile</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;white&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_gradientn</span><span>(</span><span>colours</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>palette_light</span><span>()[[</span><span>1</span><span>]],</span><span> </span><span>palette_light</span><span>()[[</span><span>2</span><span>]]))</span><span> </span><span>+</span><span>
</span><span>coord_flip</span><span>()</span><span> </span><span>+</span><span>
</span><span>theme_tq</span><span>()</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>legend.position</span><span> </span><span>=</span><span> </span><span>&quot;right&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>axis.text.x</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>angle</span><span> </span><span>=</span><span> </span><span>45</span><span>,</span><span> </span><span>vjust</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>hjust</span><span> </span><span>=</span><span> </span><span>1</span><span>))</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;first word in pair&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;second word in pair&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i2.wp.com/shiring.github.io/text_analysis/2017/06/28/twitter_post_files/figure-markdown_github/unnamed-chunk-32-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<p>These, we can also show as a graph:</p>
<div>
<pre><code><span>library</span><span>(</span><span>igraph</span><span>)</span><span>
</span><span>library</span><span>(</span><span>ggraph</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>bigram_graph</span><span> </span><span>&lt;-</span><span> </span><span>bigram_counts</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>n</span><span> </span><span>></span><span> </span><span>5</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>graph_from_data_frame</span><span>()</span><span>
</span><span>set.seed</span><span>(</span><span>1</span><span>)</span><span>
</span><span>a</span><span> </span><span>&lt;-</span><span> </span><span>grid</span><span>::</span><span>arrow</span><span>(</span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;closed&quot;</span><span>,</span><span> </span><span>length</span><span> </span><span>=</span><span> </span><span>unit</span><span>(</span><span>.15</span><span>,</span><span> </span><span>&quot;inches&quot;</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>ggraph</span><span>(</span><span>bigram_graph</span><span>,</span><span> </span><span>layout</span><span> </span><span>=</span><span> </span><span>&quot;fr&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_edge_link</span><span>(</span><span>aes</span><span>(</span><span>edge_alpha</span><span> </span><span>=</span><span> </span><span>n</span><span>),</span><span> </span><span>show.legend</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span>
</span><span>arrow</span><span> </span><span>=</span><span> </span><span>a</span><span>,</span><span> </span><span>end_cap</span><span> </span><span>=</span><span> </span><span>circle</span><span>(</span><span>.07</span><span>,</span><span> </span><span>'inches'</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_node_point</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>],</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_node_text</span><span>(</span><span>aes</span><span>(</span><span>label</span><span> </span><span>=</span><span> </span><span>name</span><span>),</span><span> </span><span>vjust</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>hjust</span><span> </span><span>=</span><span> </span><span>0.5</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_void</span><span>()</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i2.wp.com/shiring.github.io/text_analysis/2017/06/28/twitter_post_files/figure-markdown_github/unnamed-chunk-35-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<p>We can also use bigram analysis to identify negated meanings (this will become relevant for sentiment analysis later). So, let’s look at which words are preceded by “not” or “no”.</p>
<div>
<pre><code><span>bigrams_separated</span><span> </span><span>&lt;-</span><span> </span><span>followers_df</span><span> </span><span>%>%</span><span>
</span><span>unnest_tokens</span><span>(</span><span>bigram</span><span>,</span><span> </span><span>description</span><span>,</span><span> </span><span>token</span><span> </span><span>=</span><span> </span><span>&quot;ngrams&quot;</span><span>,</span><span> </span><span>n</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>!</span><span>grepl</span><span>(</span><span>&quot;\\.|http&quot;</span><span>,</span><span> </span><span>bigram</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>separate</span><span>(</span><span>bigram</span><span>,</span><span> </span><span>c</span><span>(</span><span>&quot;word1&quot;</span><span>,</span><span> </span><span>&quot;word2&quot;</span><span>),</span><span> </span><span>sep</span><span> </span><span>=</span><span> </span><span>&quot; &quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>word1</span><span> </span><span>==</span><span> </span><span>&quot;not&quot;</span><span> </span><span>|</span><span> </span><span>word1</span><span> </span><span>==</span><span> </span><span>&quot;no&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>!</span><span>word2</span><span> </span><span>%in%</span><span> </span><span>stop_words</span><span>$</span><span>word</span><span>)</span><span>
</span><span>not_words</span><span> </span><span>&lt;-</span><span> </span><span>bigrams_separated</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>word1</span><span> </span><span>==</span><span> </span><span>&quot;not&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>get_sentiments</span><span>(</span><span>&quot;afinn&quot;</span><span>),</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>word2</span><span> </span><span>=</span><span> </span><span>&quot;word&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>word2</span><span>,</span><span> </span><span>score</span><span>,</span><span> </span><span>sort</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ungroup</span><span>()</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>not_words</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>contribution</span><span> </span><span>=</span><span> </span><span>n</span><span> </span><span>*</span><span> </span><span>score</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>desc</span><span>(</span><span>abs</span><span>(</span><span>contribution</span><span>)))</span><span> </span><span>%>%</span><span>
</span><span>head</span><span>(</span><span>20</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>word2</span><span> </span><span>=</span><span> </span><span>reorder</span><span>(</span><span>word2</span><span>,</span><span> </span><span>contribution</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>word2</span><span>,</span><span> </span><span>n</span><span> </span><span>*</span><span> </span><span>score</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>n</span><span> </span><span>*</span><span> </span><span>score</span><span> </span><span>></span><span> </span><span>0</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_col</span><span>(</span><span>show.legend</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_manual</span><span>(</span><span>values</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>())</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Sentiment score * number of occurrences&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Words preceded by \&quot;not\&quot;&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>coord_flip</span><span>()</span><span> </span><span>+</span><span>
</span><span>theme_tq</span><span>()</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i1.wp.com/shiring.github.io/text_analysis/2017/06/28/twitter_post_files/figure-markdown_github/unnamed-chunk-37-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h4>What’s the predominant sentiment in my followers’ descriptions?</h4>
<p>For sentiment analysis, I will exclude the words with a negated meaning from nrc and switch their positive and negative meanings from bing (although in this case, there was only one negated word, “endorsement”, so it won’t make a real difference).</p>
<div>
<pre><code><span>tidy_descr_sentiment</span><span> </span><span>&lt;-</span><span> </span><span>tidy_descr</span><span> </span><span>%>%</span><span>
</span><span>left_join</span><span>(</span><span>select</span><span>(</span><span>bigrams_separated</span><span>,</span><span> </span><span>word1</span><span>,</span><span> </span><span>word2</span><span>),</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;word&quot;</span><span> </span><span>=</span><span> </span><span>&quot;word2&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>get_sentiments</span><span>(</span><span>&quot;nrc&quot;</span><span>),</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;word&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>get_sentiments</span><span>(</span><span>&quot;bing&quot;</span><span>),</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;word&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>rename</span><span>(</span><span>nrc</span><span> </span><span>=</span><span> </span><span>sentiment.x</span><span>,</span><span> </span><span>bing</span><span> </span><span>=</span><span> </span><span>sentiment.y</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>nrc</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>!</span><span>is.na</span><span>(</span><span>word1</span><span>),</span><span> </span><span>NA</span><span>,</span><span> </span><span>nrc</span><span>),</span><span>
</span><span>bing</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>!</span><span>is.na</span><span>(</span><span>word1</span><span>)</span><span> </span><span>&amp;</span><span> </span><span>bing</span><span> </span><span>==</span><span> </span><span>&quot;positive&quot;</span><span>,</span><span> </span><span>&quot;negative&quot;</span><span>,</span><span> </span><span>ifelse</span><span>(</span><span>!</span><span>is.na</span><span>(</span><span>word1</span><span>)</span><span> </span><span>&amp;</span><span> </span><span>bing</span><span> </span><span>==</span><span> </span><span>&quot;negative&quot;</span><span>,</span><span> </span><span>&quot;positive&quot;</span><span>,</span><span> </span><span>bing</span><span>)))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>tidy_descr_sentiment</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>nrc</span><span> </span><span>!=</span><span> </span><span>&quot;positive&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>nrc</span><span> </span><span>!=</span><span> </span><span>&quot;negative&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>x</span><span>,</span><span> </span><span>y</span><span>,</span><span> </span><span>nrc</span><span>,</span><span> </span><span>bing</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>x</span><span>,</span><span> </span><span>y</span><span>,</span><span> </span><span>sort</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>n</span><span> </span><span>></span><span> </span><span>10</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>reorder</span><span>(</span><span>y</span><span>,</span><span> </span><span>n</span><span>),</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>n</span><span>))</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>~</span><span> </span><span>x</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_col</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>],</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>],</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>)</span><span> </span><span>+</span><span>
</span><span>coord_flip</span><span>()</span><span> </span><span>+</span><span>
</span><span>theme_tq</span><span>()</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;count of sentiment in followers' descriptions&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/text_analysis/2017/06/28/twitter_post_files/figure-markdown_github/unnamed-chunk-39-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h4>Are followers’ descriptions mostly positive or negative?</h4>
<p>The majority of my followers have predominantly positive descriptions.</p>
<div>
<pre><code><span>tidy_descr_sentiment</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>screenName</span><span>,</span><span> </span><span>word</span><span>,</span><span> </span><span>bing</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>screenName</span><span>,</span><span> </span><span>bing</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarise</span><span>(</span><span>sum</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>n</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>spread</span><span>(</span><span>bing</span><span>,</span><span> </span><span>sum</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>0</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>sentiment</span><span> </span><span>=</span><span> </span><span>positive</span><span> </span><span>-</span><span> </span><span>negative</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>sentiment</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_density</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>],</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>],</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_tq</span><span>()</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i2.wp.com/shiring.github.io/text_analysis/2017/06/28/twitter_post_files/figure-markdown_github/unnamed-chunk-40-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h4>What are the most common positive and negative words in followers’ descriptions?</h4>
<div>
<pre><code><span>library</span><span>(</span><span>reshape2</span><span>)</span><span>
</span><span>tidy_descr_sentiment</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>word</span><span>,</span><span> </span><span>bing</span><span>,</span><span> </span><span>sort</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>acast</span><span>(</span><span>word</span><span> </span><span>~</span><span> </span><span>bing</span><span>,</span><span> </span><span>value.var</span><span> </span><span>=</span><span> </span><span>&quot;n&quot;</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>0</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>comparison.cloud</span><span>(</span><span>colors</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>()[</span><span>1</span><span>:</span><span>2</span><span>],</span><span>
</span><span>max.words</span><span> </span><span>=</span><span> </span><span>100</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i1.wp.com/shiring.github.io/text_analysis/2017/06/28/twitter_post_files/figure-markdown_github/unnamed-chunk-41-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h4>Topic modeling: are there groups of followers with specific interests?</h4>
<p>Topic modeling can be used to categorize words into groups. Here, we can use it to see whether (some) of my followers can be grouped into subgroups according to their descriptions.</p>
<div>
<pre><code><span>library</span><span>(</span><span>topicmodels</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>dtm_words_count</span><span> </span><span>&lt;-</span><span> </span><span>tidy_descr</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>word_stem</span><span> </span><span>=</span><span> </span><span>removeNumbers</span><span>(</span><span>word_stem</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>screenName</span><span>,</span><span> </span><span>word_stem</span><span>,</span><span> </span><span>sort</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ungroup</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>word_stem</span><span> </span><span>!=</span><span> </span><span>&quot;&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>cast_dtm</span><span>(</span><span>screenName</span><span>,</span><span> </span><span>word_stem</span><span>,</span><span> </span><span>n</span><span>)</span><span>
</span><span># set a seed so that the output of the model is predictable
</span><span>dtm_lda</span><span> </span><span>&lt;-</span><span> </span><span>LDA</span><span>(</span><span>dtm_words_count</span><span>,</span><span> </span><span>k</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>control</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>seed</span><span> </span><span>=</span><span> </span><span>1234</span><span>))</span><span>
</span><span>topics_beta</span><span> </span><span>&lt;-</span><span> </span><span>tidy</span><span>(</span><span>dtm_lda</span><span>,</span><span> </span><span>matrix</span><span> </span><span>=</span><span> </span><span>&quot;beta&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>p</span><span>1</span><span> </span><span>&lt;-</span><span> </span><span>topics_beta</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>grepl</span><span>(</span><span>&quot;[a-z]+&quot;</span><span>,</span><span> </span><span>term</span><span>))</span><span> </span><span>%>%</span><span> </span><span># some words are Chinese, etc. I don't want these because ggplot doesn't plot them correctly
</span><span> </span><span>group_by</span><span>(</span><span>topic</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>top_n</span><span>(</span><span>10</span><span>,</span><span> </span><span>beta</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ungroup</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>topic</span><span>,</span><span> </span><span>-</span><span>beta</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>term</span><span> </span><span>=</span><span> </span><span>reorder</span><span>(</span><span>term</span><span>,</span><span> </span><span>beta</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>term</span><span>,</span><span> </span><span>beta</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>factor</span><span>(</span><span>topic</span><span>),</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>factor</span><span>(</span><span>topic</span><span>)))</span><span> </span><span>+</span><span>
</span><span>geom_col</span><span>(</span><span>show.legend</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_color_manual</span><span>(</span><span>values</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>())</span><span> </span><span>+</span><span>
</span><span>scale_fill_manual</span><span>(</span><span>values</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>())</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>~</span><span> </span><span>topic</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>5</span><span>)</span><span> </span><span>+</span><span>
</span><span>coord_flip</span><span>()</span><span> </span><span>+</span><span>
</span><span>theme_tq</span><span>()</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;beta (~ occurrence in topics 1-5)&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;The top 10 most characteristic words describe topic categories.&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>user_topic</span><span> </span><span>&lt;-</span><span> </span><span>tidy</span><span>(</span><span>dtm_lda</span><span>,</span><span> </span><span>matrix</span><span> </span><span>=</span><span> </span><span>&quot;gamma&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>desc</span><span>(</span><span>gamma</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>document</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>top_n</span><span>(</span><span>1</span><span>,</span><span> </span><span>gamma</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>p</span><span>2</span><span> </span><span>&lt;-</span><span> </span><span>user_topic</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>topic</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>top_n</span><span>(</span><span>10</span><span>,</span><span> </span><span>gamma</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>reorder</span><span>(</span><span>document</span><span>,</span><span> </span><span>-</span><span>gamma</span><span>),</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>gamma</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>factor</span><span>(</span><span>topic</span><span>)))</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>~</span><span> </span><span>topic</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>5</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>(</span><span>show.legend</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>4</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_color_manual</span><span>(</span><span>values</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>())</span><span> </span><span>+</span><span>
</span><span>scale_fill_manual</span><span>(</span><span>values</span><span> </span><span>=</span><span> </span><span>palette_light</span><span>())</span><span> </span><span>+</span><span>
</span><span>theme_tq</span><span>()</span><span> </span><span>+</span><span>
</span><span>coord_flip</span><span>()</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;gamma\n(~ affiliation with topics 1-5)&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>library</span><span>(</span><span>grid</span><span>)</span><span>
</span><span>library</span><span>(</span><span>gridExtra</span><span>)</span><span>
</span><span>grid.arrange</span><span>(</span><span>p</span><span>1</span><span>,</span><span> </span><span>p</span><span>2</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>heights</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>0.7</span><span>,</span><span> </span><span>0.3</span><span>))</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/text_analysis/2017/06/28/twitter_post_files/figure-markdown_github/unnamed-chunk-52-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>The upper of the two plots above show the words that were most strongly grouped to five topics. The lower plots show my followers with the strongest affiliation with these five topics.</p>
<p>Because in my tweets I only cover a relatively narrow range of topics (i.e. related to data), my followers are not very diverse in terms of their descriptions and the five topics are not very distinct.</p>
<p>If you find yourself in any of the topics, let me know if you agree with the topic that was modeled for you!</p>
<p></p>
<p>For more text analysis, see my <a rel="nofollow" href="https://shiring.github.io/text_analysis/2016/11/06/sysk_earwax" target="_blank">post about text mining and sentiment analysis of a Stuff You Should Know Podcast</a>.</p>
<hr>
<p></p>
<div>
<pre><code><span>sessionInfo</span><span>()</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## R version 3.4.0 (2017-04-21)
## Platform: x86_64-apple-darwin15.6.0 (64-bit)
## Running under: macOS Sierra 10.12.5
## 
## Matrix products: default
## BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] grid      stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] gridExtra_2.2.1               topicmodels_0.2-6            
##  [3] reshape2_1.4.2                ggraph_1.0.0                 
##  [5] igraph_1.0.1                  tm_0.7-1                     
##  [7] NLP_0.1-10                    wordcloud_2.5                
##  [9] RColorBrewer_1.1-2            SnowballC_0.5.1              
## [11] tidytext_0.1.3                bindrcpp_0.2                 
## [13] tidyquant_0.5.1               quantmod_0.4-10              
## [15] TTR_0.23-1                    PerformanceAnalytics_1.4.3541
## [17] xts_0.9-7                     zoo_1.8-0                    
## [19] lubridate_1.6.0               dplyr_0.7.0                  
## [21] purrr_0.2.2.2                 readr_1.1.1                  
## [23] tidyr_0.6.3                   tibble_1.3.3                 
## [25] ggplot2_2.2.1                 tidyverse_1.1.1              
## [27] twitteR_1.1.9                
## 
## loaded via a namespace (and not attached):
##  [1] viridis_0.4.0     httr_1.2.1        viridisLite_0.2.0
##  [4] bit64_0.9-7       jsonlite_1.5      modelr_0.1.0     
##  [7] assertthat_0.2.0  stats4_3.4.0      cellranger_1.1.0 
## [10] ggrepel_0.6.5     yaml_2.1.14       slam_0.1-40      
## [13] backports_1.1.0   lattice_0.20-35   glue_1.1.0       
## [16] digest_0.6.12     rvest_0.3.2       colorspace_1.3-2 
## [19] htmltools_0.3.6   Matrix_1.2-10     plyr_1.8.4       
## [22] psych_1.7.5       pkgconfig_2.0.1   broom_0.4.2      
## [25] haven_1.0.0       scales_0.4.1      tweenr_0.1.5     
## [28] ggforce_0.1.1     lazyeval_0.2.0    mnormt_1.5-5     
## [31] magrittr_1.5      readxl_1.0.0      evaluate_0.10    
## [34] tokenizers_0.1.4  janeaustenr_0.1.5 nlme_3.1-131     
## [37] MASS_7.3-47       forcats_0.2.0     xml2_1.1.1       
## [40] foreign_0.8-68    tools_3.4.0       hms_0.3          
## [43] stringr_1.2.0     munsell_0.4.3     compiler_3.4.0   
## [46] rlang_0.1.1       units_0.4-5       rjson_0.2.15     
## [49] labeling_0.3      rmarkdown_1.6     gtable_0.2.0     
## [52] DBI_0.7           R6_2.2.2          knitr_1.16       
## [55] udunits2_0.13     bit_1.1-12        bindr_0.1        
## [58] rprojroot_1.2     Quandl_2.8.0      modeltools_0.2-21
## [61] stringi_1.1.5     parallel_3.4.0    Rcpp_0.12.11
</code></pre>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://shiring.github.io/text_analysis/2017/06/28/twitter_post"> Shirin's playgRound</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/6eZXOWoc54Y" height="1"></a></div>