<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/extract-twitter-data-automatically-using-scheduler-r-package/"></div></div><div><a data-url="https://www.r-bloggers.com/extract-twitter-data-automatically-using-scheduler-r-package/" data-counturl="https://www.r-bloggers.com/extract-twitter-data-automatically-using-scheduler-r-package/" data-count="horizontal" data-text="Extract Twitter Data Automatically using Scheduler R package" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://datascienceplus.com/extract-twitter-data-automatically-using-scheduler-r-package/"> DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>The following R-script is for download the data automatically from twitter using the package <code>SchedulerR</code>. As first step we download the data using the <code>OAuth protocol</code> and store the data using as a name the date from the download. As second we used the package <code>SchedulerR</code> for to specify how long the script runs. This package has the advantage that we can set the time and also how long will repeat either daily, weekly or monthly.</p>
<p>Load the packages:</p>
<pre>library(twitteR)
library(httr)
library(SchedulerR) </pre>
<p>Also we choose the directory where the RData would be saved:</p>
<pre>setwd(&quot;C:/Users/Documents/schedulerR&quot;)</pre>
<p>OAuth protocol:</p>
<pre>api_key &lt;- &quot;xxxxxxxxxxxxxxxxxxxxxx&quot;
api_secret &lt;- &quot;xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&quot;
access_token &lt;- &quot;xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&quot;
access_token_secret &lt;- &quot;xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&quot;
setup_twitter_oauth(api_key,api_secret,access_token,access_token_secret)</pre>
<p>Extract the tweets using usertimeline function from the Twitter R package:</p>
<pre> clinton_tweets &lt;- userTimeline(&quot;HillaryClinton&quot;, n = 3200)
tweetsc.df &lt;- twListToDF(clinton_tweets)
dim(tweetsc.df)</pre>
<p>Create a variable called date and become to character:</p>
<pre>date&lt;-Sys.Date()
date&lt;-as.character(date)
name&lt;-paste(date,&quot;.RData&quot;)
#Finally we save the Rdata using as name the date from the download
save(tweetsc.df, file =name)</pre>
<p>The script before is for download the data from twitter.<br>
The next step is used the package SchedulerR, the script for this is the following: </p>
<pre>taskscheduler_create(taskname = &quot;taskclinton&quot;, rscript = clintontweets, 
  schedule = &quot;DAILY&quot;, starttime = &quot;11:30&quot;, startdate = format(Sys.Date(), &quot;%d/%m/%Y&quot;)) </pre>
<p>Also we can used the add in interface on Rstudio add in tasks. Feel free to post comment if you have any question or suggestion about this post.</p>
<h2>Examples of Analysis</h2>
<p>We can implement many analysis from text data.</p>
<p>Find words common with a minimun frequency of 45:</p>
<pre>term.freq &lt;- rowSums(as.matrix(tdm))
term.freq = 45)
#we saved the data in a data frame
df &lt;- data.frame(term = names(term.freq), freq = term.freq)
save(df,file=&quot;tqfclinton50.RData&quot;)
#plot with ggplot2
library(ggplot2)
ggplot(df, aes(x = term, y = freq)) + geom_bar(stat = &quot;identity&quot;) +
xlab(&quot;Terms&quot;) + ylab(&quot;Count&quot;) + coord_flip()</pre>
<p>This is the plot:<br>
<a title="" data-rel="lightbox-0" target="_blank" rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2016/10/wordscommon.png"><img sizes="(max-width: 353px) 100vw, 353px" alt="wordscommon" class="wp-image-4891" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2016/10/wordscommon.png?resize=353%2C284" data-recalc-dims="1"></a></p>
<p>Hierarchical cluster:</p>
<pre>
# remove the parse terms
tdm2 &lt;- removeSparseTerms(tdm, sparse = 0.95)
m2 &lt;- as.matrix(tdm2)
# hierarchical cluster
m &lt;- as.matrix(m2)
d&lt;- dist(m2)
#we used the cosine similaruty as distance because is the best option for text data
distcos&lt;-dissimilarity(x=m2,method='cosine')
groups &lt;- hclust(distcos,method=&quot;complete&quot;)
#plot dendograma
plot(groups, hang=-1)
rect.hclust(groups,5)</pre>
<p>This is the plot:<br>
<a title="" data-rel="lightbox-1" target="_blank" rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2016/10/cosenoclint.png"><img sizes="(max-width: 490px) 100vw, 490px" alt="cosenoclint" class="wp-image-4892" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2016/10/cosenoclint-490x319.png?resize=456%2C297" data-recalc-dims="1"></a></p>
<h3>K means clustering</h3>
<p>We combine the Kmeans clustering with Principal components analysis for obtain a better result. As a first we used the elbow method for choose the “K”</p>
<p><a title="" data-rel="lightbox-2" target="_blank" rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2016/10/elbowpcatrump.png"><img sizes="(max-width: 312px) 100vw, 312px" alt="elbowpcatrump" class="wp-image-4893" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2016/10/elbowpcatrump.png?resize=312%2C284" data-recalc-dims="1"></a></p>
<p>We choose K=5 from the plot. Now we proceed to make the K means clustering.</p>
<pre>pc &lt;- princomp(tdm)
plot(pc)
data&lt;-tdm
wss &lt;- (nrow(data)-1)*sum(apply(data,2,var))
for (i in 2:15) wss[i] &lt;- sum(kmeans(data,
centers=i)$withinss)
plot(1:15, wss, type=&quot;b&quot;, xlab=&quot;Number of Clusters&quot;,
ylab=&quot;Within groups sum of squares&quot;)

m3 &lt;- t(tdm2) # transpse from term document matriz
set.seed(1222) # we choose a seed.
k &lt;- 5# number of clusters
kmeansResult &lt;- kmeans(m3, k)
km&lt;-round(kmeansResult$centers, digits = 3) 
#we saved in a csv file.
write.csv(km,file=&quot;kmeansclinton.csv&quot;)</pre>
<p>The following loop is for visualize the clusters.</p>
<pre>for (i in 1:k) {
cat(paste(&quot;cluster &quot;, i, &quot;: &quot;, sep = &quot;&quot;))
s &lt;- sort(kmeansResult$centers[i, ], decreasing = T)
cat(names(s)[1:5], &quot;\n&quot;)
}
write.csv(s,file=&quot;clusterkm.csv&quot;)  </pre>
<p><a title="" data-rel="lightbox-3" target="_blank" rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2016/10/clusterimage.png"><img sizes="(max-width: 490px) 100vw, 490px" alt="clusterimage" class="wp-image-4894" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2016/10/clusterimage-490x76.png?resize=456%2C71" data-recalc-dims="1"></a></p>
<p>From the results we can said from each cluster they are the topics most spoken from the tweets.</p>
<ul></ul>
<p><strong>
<p>Related Post</p>
<p>
<ol>
<li><a title="Permanent Link toAn Introduction to Time Series with JSON Data" target="_blank" rel="nofollow" href="http://datascienceplus.com/introduction-to-time-series-with-json-data/">An Introduction to Time Series with JSON Data</a></li>
<li><a title="Permanent Link toGet Your Data into R: Import Data from SPSS, Stata, SAS, CSV or TXT" target="_blank" rel="nofollow" href="http://datascienceplus.com/get-your-data-into-r/">Get Your Data into R: Import Data from SPSS, Stata, SAS, CSV or TXT</a></li>
<li><a title="Permanent Link toExporting Data from R to TXT, CSV, SPSS or Stata" target="_blank" rel="nofollow" href="http://datascienceplus.com/exporting-data-from-r/">Exporting Data from R to TXT, CSV, SPSS or Stata</a></li>
<li><a title="Permanent Link toFirst Things to Do After You Import the Data into R" target="_blank" rel="nofollow" href="http://datascienceplus.com/first-things-to-do-after-you-import-the-data-into-r/">First Things to Do After You Import the Data into R</a></li>
</ol></strong>
<div>
<a target="_blank" rel="nofollow" href="http://feeds.feedburner.com/~ff/datascienceplus?a=rrVHxWTLwOQ:21nD8tADHOo:D7DqB2pKExk"><img src="http://feeds.feedburner.com/~ff/datascienceplus?i=rrVHxWTLwOQ:21nD8tADHOo:D7DqB2pKExk" border="0"></a> <a target="_blank" rel="nofollow" href="http://feeds.feedburner.com/~ff/datascienceplus?a=rrVHxWTLwOQ:21nD8tADHOo:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/datascienceplus?d=qj6IDK7rITs" border="0"></a> <a target="_blank" rel="nofollow" href="http://feeds.feedburner.com/~ff/datascienceplus?a=rrVHxWTLwOQ:21nD8tADHOo:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/datascienceplus?d=yIl2AUoC8zA" border="0"></a>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/extract-twitter-data-automatically-using-scheduler-r-package/"></div></div><div><a data-url="https://www.r-bloggers.com/extract-twitter-data-automatically-using-scheduler-r-package/" data-counturl="https://www.r-bloggers.com/extract-twitter-data-automatically-using-scheduler-r-package/" data-count="vertical" data-text="Extract Twitter Data Automatically using Scheduler R package" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://datascienceplus.com/extract-twitter-data-automatically-using-scheduler-r-package/"> DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/a0Wjd-yL258"></a></div>