<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/scraping-twitter-data-to-visualize-trending-tweets-in-kuala-lumpur/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://feedproxy.google.com/~r/RStoriesDataSpeak/~3/sK92E2ZrXbI/"> Stories Data Speak</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><em>(Disclaimer: I’ve no grudge against python programming language per se. I think its equally great. In the following post, I’m merely recounting my experience.)</em></p>
<p>It’s been quite a while since I last posted. The reasons are numerous, notable being, unable to decide which programming language to choose for web data scraping. The contenders were data analytic maestro, <code>R</code> and data scraping guru, <code>python</code>. So, I decided to give myself some time to figure out which language will be best for my use case. My use case was, <em>Given some search keywords, scrape twitter for related posts and visualize the result</em>. First, I needed the <em>live data</em>. Again, I was at the cross-roads, “R or Python”. Apparently python has some great packages for twitter data streaming like <code>twython</code>,<code>python-twitter</code>, <code>tweepy</code>. Equivalent R libraries are <code>twitteR</code>,<code>rwteet</code>. I chose the <code>rtweet</code> package for data collection over python for following reasons;</p>
<ul>
<li>I do not have to create a <code>credential file</code> (unlike in python) to log in to my twitter account. However, you do need to authenticate the twitter account when using the <code>rtweet</code> package. This authentication is done just once if using the <code>rtweet</code> package. Your twitter credentials will be stored locally.</li>
<li>Coding and code readability is far more easier as compared to python.</li>
<li>The <code>rtweet</code> package allows for multiple hash tags to be searched for.</li>
<li>To localize the data, the package also allows for specifying geographic coordinates.</li>
</ul>
<p>So, using the following code snippet, I was able to scrape data. The code has following parts;</p>
<ol>
<li>
<p>A custom search for tweets function which will accept the search string. If search string is <code>NULL</code>, it will throw a message and stop, else it will search for hash tags specified in search string and return a data frame as output.</p>
<p> #required libraries</p>
<div>
<div>
<pre><code> > library(rtweet)
 > library(tidytext)
 > library(tidyverse)
 > library(stringr)
 > library(stopwords)
 > library(rtweet) # for search_tweets()
 # Create a function that will accept multiple hashtags and will search the twitter api for related tweets

 > search_tweets_queries &lt;- function(x, n = 100, ...) {
   ## check inputs
   stopifnot(is.atomic(x), is.numeric(n))
   if (length(x) == 0L) {
 stop(&quot;No query found&quot;, call. = FALSE)
   }  
   ## search for each string in column of queries
   rt &lt;- lapply(x, search_tweets, n = n, ...)
   ## add query variable to data frames
   rt &lt;- Map(cbind, rt, query = x, stringsAsFactors = FALSE)
   ## merge users data into one data frame
   rt_users &lt;- do.call(&quot;rbind&quot;, lapply(rt, users_data))
   ## merge tweets data into one data frame
   rt &lt;- do.call(&quot;rbind&quot;, rt)
   ## set users attribute
   attr(rt, &quot;users&quot;) &lt;- rt_users
   ## return tibble (validate = FALSE makes it a bit faster)
   tibble::as_tibble(rt, validate = FALSE)
 }
</code></pre>
</div></div>
</li>
<li>
<p>A data frame containing the search terms. Note, here my search hash-tags are <code>KTM</code>, <code>MRT</code> and <code>monorail</code>.</p>
<div>
<div>
<pre><code> # create data frame with query column
 > df_query &lt;- data.frame(
   query = c(&quot;KTM&quot;, &quot;monorail&quot;,&quot;MRT&quot;),
   n = rnorm(3), # change this number according to the number of searchwords in parameter query. As of now, the parameter got 3 keywords, therefore this nuber is set to 3.
   stringsAsFactors = FALSE
 )
</code></pre>
</div></div>
</li>
<li>
<p>Using the <code>search_tweets_queries</code> defined in step 1, to search for tweets. Note, the usage of <code>retryonratelimit=TRUE</code> indicates if search rate limit reached, then the crawler will sleep for a while and start again. Refer to the <code>rtweet</code> <a rel="nofollow" href="https://rtweet.info/" target="_blank">documentation</a> for more information.</p>
<div>
<div>
<pre><code> # pass query column to the search_tweet_queries function defined above
 > df_collect_tweets &lt;- search_tweets_queries(df_query$query, include_rts = FALSE, retryonratelimit = TRUE, 
 #geocode for Kuala Lumpur
 geocode = &quot;3.14032,101.69466,93.5mi&quot;
 )
</code></pre>
</div></div>
</li>
<li>
<p>Once the data is collected, I’ll keep some selected columns only.</p>
<div>
<div>
<pre><code> # Select and keep only relevant columns
 > df_select_tweets&lt;- df_collect_tweets %>%
   select(c(user_id,created_at,screen_name, !is.na(hashtags),text,
    source,display_text_width>0,lang,!is.na(place_name),
    !is.na(place_full_name),
    !is.na(geo_coords), !is.na(country), !is.na(location),
    retweet_count,account_created_at, account_lang, query)
  )
</code></pre>
</div></div>
</li>
<li>
<p><strong>Text mining</strong>: The collected data need to be cleaned. Therefore, I’ve used the basic <code>gsub()</code> function and <code>str_replace_all()</code> from the <code>stringr</code> library.</p>
<div>
<div>
<pre><code> # Saving the selected columns data
 > df_select_tweets_1 = data.frame(lapply(df_select_tweets, as.character), stringsAsFactors=FALSE)
 ### Text preprocessing
    
 # 1. Remove URL from text
 # collapse to long format
 > clean_tweet&lt;- df_select_tweets_1
    
 #clean_tweet&lt;- paste(df_select_tweets_1, collapse=&quot; &quot;)
 > clean_tweet$text = gsub(&quot;&amp;amp&quot;, &quot;&quot;, clean_tweet$text)
 > clean_tweet$text = gsub(&quot;(RT|via)((?:\\b\\W*@\\w+)+)&quot;, &quot;&quot;, clean_tweet$text)
 > clean_tweet$text = gsub(&quot;@\\w+&quot;, &quot;&quot;, clean_tweet$text)
 > clean_tweet$text = gsub(&quot;[[:punct:]]&quot;, &quot;&quot;, clean_tweet$text)
 > clean_tweet$text = gsub(&quot;[[:digit:]]&quot;, &quot;&quot;, clean_tweet$text)
 > clean_tweet$text = gsub(&quot;http\\w+&quot;, &quot;&quot;, clean_tweet$text)
 > clean_tweet$text = gsub(&quot;[ \t]{2,}&quot;, &quot;&quot;, clean_tweet$text)
 > clean_tweet$text = gsub(&quot;^\\s+|\\s+$&quot;, &quot;&quot;, clean_tweet$text)

 #get rid of unnecessary spaces
 > clean_tweet$text &lt;- str_replace_all(clean_tweet$text,&quot; &quot;,&quot; &quot;)
 # Get rid of URLs
 > clean_tweet$text&lt;- str_replace_all(clean_tweet$text, &quot;https://t.co/[a-z,A-Z,0-9]*&quot;,&quot;&quot;)
 > clean_tweet$text&lt;- str_replace_all(clean_tweet$text, &quot;http://t.co/[a-z,A-Z,0-9]*&quot;,&quot;&quot;)
 # Take out retweet header, there is only one
 > clean_tweet$text &lt;- str_replace(clean_tweet$text,&quot;RT @[a-z,A-Z]*: &quot;,&quot;&quot;)
 # Get rid of hashtags
 > clean_tweet$text &lt;- str_replace_all(clean_tweet$text,&quot;#[a-z,A-Z]*&quot;,&quot;&quot;)
 # Get rid of references to other screennames
 > clean_tweet$text &lt;- str_replace_all(clean_tweet$text,&quot;@[a-z,A-Z]*&quot;,&quot;&quot;) 
</code></pre>
</div></div>
<p>a. Next, I’ll use the <code>tidytext</code> library for <code>token</code> extraction</p>
<div>
<div>
<pre><code> # Unnest the tokens
 > df.clean&lt;- clean_tweet %>%
   unnest_tokens(word, text)
    
 > clean_tweets&lt;- tibble()
 > clean_tweets&lt;- rbind(clean_tweets, df.clean)
    
 # Basic calculations
 # calculate word frequency
 > word_freq &lt;- clean_tweets %>%
   count(word, sort=TRUE)
 > word_freq 

 # A tibble: 5,291 x 2
    wordn
    </code></pre></div></div></ol>