<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/setting-up-twitter-streamr-service-on-an-ubuntu-server/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://amitkohli.com/setting-up-twitter-streamr-service-on-an-ubuntu-server/"> R – AmitKohli.com</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I am working on a<strong> super-secret</strong> project for which I am harvesting a highly confidential source of data: twitter <img alt="🙂" src="https://i0.wp.com/s.w.org/images/core/emoji/2.3/72x72/1f642.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"> The idea is to gather a small amount of twitter data, but for a long time… maybe a year. I tried to use the package <a rel="nofollow" href="https://cran.r-project.org/web/packages/twitteR/index.html" target="_blank">TwitteR</a>, but it can only  grab up to a week of tweets… it’s not really good for a set-it-and-forget-it ongoing capture since it requires user-based authentication, which means (I guess) that a machine can’t authenticate for it. Tangibly this means a human needs to start the process every time. So I could run the script weekly, but of course there’s days you miss, or run at different times… plus it’s just plain annoying…</p>
<blockquote class="twitter-tweet" data-lang="en">
<p dir="ltr" lang="en">does no <a rel="nofollow" href="https://twitter.com/hashtag/rstats?src=hash&ref_src=twsrc%5Etfw" target="_blank">#rstats</a> package useTwitter oauth 2.0? I can’t believe it… this means that automated tweet-scrapers in R aren’t possible? <a rel="nofollow" href="https://t.co/eq9dnrbB6L" target="_blank">pic.twitter.com/eq9dnrbB6L</a></p>
<p>— amit (@VizMonkey) <a rel="nofollow" href="https://twitter.com/VizMonkey/status/895191870208434177?ref_src=twsrc%5Etfw" target="_blank">August 9, 2017</a></p>
</blockquote>
<p></p>
<p>And then I remembered about <a rel="nofollow" href="https://cran.r-project.org/web/packages/streamR/index.html" target="_blank">streamR</a>, which allows exactly for this ongoing scraping. This blog documents my experience this up on my server, using a linux service.</p>
<h2>Let’s Go!</h2>
<p>(Small meta-note: I’m experimenting with a new blogging style: showing more of my errors and my iterative approach to solving problems in order to counter the perception of the perfect analyst… something a bunch of people have been talking about recently. I was exposed to it by <a rel="nofollow" data-user-id="2167059661" href="https://twitter.com/JennyBryan" target="_blank"><span data-aria-label-part dir="ltr">@<b>JennyBryan</b></span></a>  and <a rel="nofollow" data-user-id="24228154" href="https://twitter.com/hspter" target="_blank"><span data-aria-label-part dir="ltr">@<b>hspter</b></span></a> during EARL London, and it’s really got me thinking. Anyway, I do realize that it makes for a messier read full of tangents and dead ends. Love it? Hate it? Please let me know what you think in the comments!)</p>
<p>(metanote 2: The linux bash scripts are available in <a rel="nofollow" href="https://github.com/mexindian/linuxRstuff" target="_blank">their own github repo</a>)</p>
<p>So if you don’t have a linux server of your own, follow <a rel="nofollow" href="http://amitkohli.com/setting-up-twitter-streamr-service-on-an-ubuntu-server/deanattali.com/2015/05/09/setup-rstudio-shiny-server-digital-ocean/" target="_blank">Dean Atalli’s excellent guide</a> to set one up on <a rel="nofollow" href="https://m.do.co/c/d925aae2d09f" target="_blank">Digital Ocean</a>… it’s cheap and totally worth it. Obviously, you’ll need to install <code>streamR</code>, also <code>ROauth</code>. I use other packages in the scripts here, up to you to do it exactly how I do it or not. Also… remember when you install R-Packages on Ubuntu, you have to do it as the superuser in linux, not from R (otherwise that package won’t be available for any other user (like shiny). If you don’t know what I’m talking about then you didn’t read Dean Atalli’s guide like I said above… why are you still here?). Actually, it’s so annoying to have to remember how to correctly install R packages on linux, that I created a little utility for it. save the following into a file called “Rinstaller.sh”:</p>
<p> </p>
<pre data-enlighter-language="shell">!/bin/bash
# Ask the user what package to install
echo what package should I grab?
read varname

echo I assume you mean CRAN, but to use github type &quot;g&quot;
read source

if [ &quot;$source&quot; = &quot;g&quot; ]; then
        echo --------------------------------
        echo Installing $varname from GitHub
        sudo su - -c \\&quot;R -e \&quot;devtools::install_github('$varname')\&quot;\\&quot;
else
        echo --------------------------------
        echo Grabbin $varname from CRAN
        sudo su - -c \\&quot;R -e \&quot;install.packages('$varname', repos='http://cran.$
fi

</pre>
<p>this function will accept an input (the package name) and then will ask if to install from CRON or from github. From github, obviously you need to supply the user account and package name. There! Now we don’t need to remember anything anymore! <img alt="🙂" src="https://i0.wp.com/s.w.org/images/core/emoji/2.3/72x72/1f642.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"> Oh, make sure you <code>chmod 777 Rinstaller.sh</code> (which lets anyone execute the file) and then run it as <code>./Rinstaller.sh</code></p>
<p> </p>
<p>Anyway, I messed around with streamR for a while and figured out how I wanted to structure the files. I think I want 3 files… one to authenticate, one to capture tweets, and the third to do the supersecret analysis. Here they are:</p>
<p> </p>
<h4>Authenticator</h4>
<pre>## Auth
library(ROAuth)
requestURL &lt;- &quot;https://api.twitter.com/oauth/request_token&quot;
accessURL &lt;- &quot;https://api.twitter.com/oauth/access_token&quot;
authURL &lt;- &quot;https://api.twitter.com/oauth/authorize&quot;
consumerKey &lt;- &quot;myKey&quot;
consumerSecret &lt;- &quot;mySecret&quot;

my_oauth &lt;- OAuthFactory$new(consumerKey = consumerKey, consumerSecret = consumerSecret, 
                             requestURL = requestURL, accessURL = accessURL, authURL = authURL)
my_oauth$handshake(cainfo = system.file(&quot;CurlSSL&quot;, &quot;cacert.pem&quot;, package = &quot;RCurl&quot;))

save(my_oauth, file = &quot;my_oauth.Rdata&quot;)</pre>
<p>So we use this file to connect to the Twitter service. You will need to set yourself up with an API… it’s fairly painless. <a rel="nofollow" href="https://apps.twitter.com/" target="_blank">Go do that here</a> and select “Create new app”. It’s fairly painless.</p>
<p>(small caveat: make sure the <code>my_oauth</code> file is saved in the working directory. You can make sure of it by creating a Project for these three files… tho this is a pain… more on this later).</p>
<h4>Tweet-Getter</h4>
<pre data-enlighter-language="null">library(streamR)
library(here)
## Get
load(&quot;/srv/shiny-server/SecretFolder/my_oauth.Rdata&quot;)
filterStream(&quot;/srv/shiny-server/SecretFolder/tweets.json&quot;, track = &quot;SecretTopic&quot;, oauth = my_oauth)</pre>
<p>OK, so we run the authenticator once, then we run can run this file, which just goes out and gathers all tweets related to <code>SecretTopic</code>, and saves them to <code>tweets.json</code>. This works with my stream because it’s relatively small number of tweets… but be careful, if your topic gets tons of hits, the file can grow VERY quickly. You might be interested in splitting up the output into multiple files. Check <a rel="nofollow" href="http://enhancedatascience.com/2017/07/17/twitter-analysis-using-r/" target="_blank">this</a> to see how.</p>
<p> </p>
<p>On working directories, it’s super annoying, it’s typically bad practice to specify a direct path to files in your script, instead, it’s encouraged that you use tech to “know your path”… for example, we can use the <code>here</code> package, or use the Project folder. The problem that arises when running files from some kind of automated cron or scheduler is that it doesn’t know how to read <code>.Rproj</code> files, and therefore doesn’t know what folder to use. I asked this question in the <a rel="nofollow" href="https://community.rstudio.com/t/best-way-to-define-paths-for-a-file-running-on-a-cron/2061" target="_blank">RStudio Community</a> site, which have sparked a large discussion… check it out! Anyway, the last script:</p>
<p> </p>
<h4>Tweet-analyzer</h4>
<pre data-enlighter-language="null">## read
library(streamR)
tweets.df &lt;- parseTweets(&quot;tweets.json&quot;, verbose = FALSE)

## Do the Secret Stuff <img alt="🙂" src="https://i0.wp.com/s.w.org/images/core/emoji/2.3/72x72/1f642.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"></pre>
<p> </p>
<p>Ok, so now we can authenticate, gather tweets, and anaylze the resulting file!</p>
<p> </p>
<p>OK cool! So let’s get the TweetGetter running! As long as it’s running, it will be appending tweets to the <code>json</code> file. We could run it on our laptop, but it’ll stop running when we close our laptop, so that’s a perfect candidate to run on a server. If you don’t know how to get your stuff up into a linux server, I recommend saving your work locally, setting up git, <code>git push</code>ing it up to a private github remote (CAREFUL! This will have your Private Keys so make sure you don’t use a public repo) and then <code>git pull</code>ing it into your server.</p>
<p> </p>
<h2>OK, set it up to run on the server</h2>
<p>(CAVEAT!! I am not a linux expert… far from it! If anyone sees me doing something boneheaded (like the <code>chmod 777</code> above, please leave a comment).</p>
<p>The first time you run the script, it will ask you to authenticate… So I recommend running the Authenticator file from RStudio on the server, which will allow you to grab the auth code and paste it into the Rstudio session. Once you’re done, you should be good to capture tweets on that server. The problem is, if you run the TweetGetter in RStudio… when you close that session, it stops the script.</p>
<p> </p>
<p><strong>Idea 2:</strong> Hrm… let’s try in the shell. So SSH into the server (on windows use Putty), go to the Project folder and type:</p>
<pre data-enlighter-language="shell">Rscript TweetGetter.R</pre>
<p> </p>
<p>It runs, but when I close the SSH session it also stops the script :-\ . I guess that instance is tied to the SSH session? I don’t get it… but whatever, fine.</p>
<p><strong>Idea 3</strong>: set a cronjob to run it! In case you don’t know, cron jobs are the schedulers on linux. Run <code>crontab -e</code> to edit the jobs, and <code>crontab -l</code> to view what jobs you have scheduled. To understand the syntax of the crontabs, <a rel="nofollow" href="http://www.adminschoice.com/crontab-quick-reference" target="_blank">see this</a>.</p>
<p> </p>
<p>So the idea would be to start the task on a schedule… that way it’s not my session that started it… although of course, if it’s set on a schedule and the schedule dictates it’s time to start up again but the file is already running, I don’t want it to run twice… hrm…</p>
<p> </p>
<p>Oh I know! I’ll create a small bash file (like a small executable) that CHECKS if the thingie is running, and if it isn’t then run it, if it is, then don’t do anything! This is what I came up with:</p>
<pre data-enlighter-language="shell">f pgrep -x &quot;Rscript&quot; > /dev/null then
    echo &quot;Running&quot;
else
    echo &quot;Stopped... restarting&quot;
    Rscript &quot;/srv/shiny-server/SecretFolder/newTweetGetter.R&quot;
fi</pre>
<p><span>WARNING! THIS IS WRONG.</span></p>
<p>What this is saying is “check if ‘Rscript’ is running on the server (I assumed I didn’t have  any OTHER running R process at the time, a valid assumption in this case). If it is, then just say ‘Running’, if it’s not, then say ‘Stopped… restarting’ and re-run the file, using <code>Rscript</code>. Then, we can put file on the cron job to run hourly… so hourly I will check if the job is running or not, and if it’s stopped, restart. This is what the cron job looks like:</p>
<p><code>1 * * * * &quot;/srv/shiny-server/SecretFolder/chek.sh&quot;</code></p>
<p>In other words, run the file <code>chek.sh</code> during minute 1 of every hour, every day of money, every month of the year, and every day of the week (ie, every hour :))</p>
<p>OK…. Cool! So I’m good right? Let me check if the json is getting tweets… hrm… no data in the past 10 minutes or so… has nobody tweeted or is it broken? Hrm2… how does one check the cronjob log file? Oh, there is none… but shouldn’t there be? ::google:: I guess there is supposed to be one… ::think:: Oh, it’s because I’m logged in with a user that doesn’t have admin rights, so when it tries to create a log file in a protected folder, it gets rejected… Well Fine! I’ll pipe the output of the run to a file in a folder I know I can write to. (Another option is to set up the cron job as the root admin…. ie instead of <code>crontab -e</code> you would say <code>sudo crontab -e</code>… but if there’s one thing I know about linux is that I don’t know linux and therefore I use admin commands as infrequently as I can get away with). So how do I pipe run contents to a location I can see? Well… google says this is one way:</p>
<p> </p>
<pre data-enlighter-language="shell">40 * * * * &quot;/srv/shiny-server/SecretFolder/chek.sh&quot; >> /home/amit/SecretTweets.log 2>&amp;1</pre>
<p>So what this is doing is running the file just as before, but the <code>>></code> pushes the results to a log file on my home directory. Just a bit of Linux for you… <code>></code> recreates the piped output everytime (ie overwrites), whereas <code>>></code> appends to what was already there. The <code>2>&amp;1</code> part means ‘grab standard output and errors’… if you wanna read more about why, <a rel="nofollow" href="https://stackoverflow.com/questions/818255/in-the-shell-what-does-21-mean" target="_blank">geek out</a>, but I think you’re basically saying “grab any errors and pipe them to standard output and then grab all standard output”.</p>
<p>OK, so after looking at the output, I saw what was happening… during every crontab run, the <code>chek.sh</code> file made it seem like the <code>newTweetGetter.R</code> wasn’t running… so it would restart it, gather 1 tweet and then time out. <img alt="🙁" src="https://i1.wp.com/s.w.org/images/core/emoji/2.3/72x72/1f641.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"> What strange behaviour! Am I over some Twitter quota? No, it can’t be, it’s a streaming service, twitter will feed me whatever it wants, I’m not requesting any amount… so it can’t be that.</p>
<p> </p>
<blockquote>
<p>here is where I threw my hands up and asked <a rel="nofollow" href="https://www.linkedin.com/in/richardstrand/" target="_blank">Richard</a>, my local linux expert for help</p>
</blockquote>
<p>Enter a very useful command: <code>top</code>. This command, and it’s slightly cooler version <code>htop</code> (which doesn’t come in Ubuntu by default but is easy to install… <code>sudo apt install htop</code>) quickly showed me that when you call an R file via <code>Rscript</code>, it doesn’t launch a service called <code>Rscript</code>, it launches a service called <code>/usr/lib/R/bin/exec/R --slave --no-restore --file=/srv/shiny-server/SecretFolder/newTweetGetter.R</code>.  Which explains why <code>chek.sh</code> didn’t think it was running (when it was)… and when the second run would try to connect to the twitter stream, it got rejected (because the first script was already connected). So this is where Richard said “BTW, you should probably set this up as a service…”. And being who I am and Richard who he is, I said: “ok”. (Although I didn’t give up on the cron… see <span>ENDNOTE1</span>).</p>
<p>After a bit of playing around, we found that the shiny-server linux service was probably easy enough to manipulate and get functional (guidance <a rel="nofollow" href="https://www.freedesktop.org/software/systemd/man/systemd.service.html" target="_blank">here</a> and <a rel="nofollow" href="https://www.freedesktop.org/software/systemd/man/systemd.exec.html#" target="_blank">here</a>), so let’s do it!</p>
<h2>Setting up the service:</h2>
<ol>
<li>First of all, go to the folder where all the services live.  <code>cd /etc/systemd/system/</code></li>
<li>Next, copy the shiny service into your new one, called SECRETtweets.service: <code>sudo cp shiny-server.service SECRETtweets.service</code></li>
<li>Now edit the contents! <code>sudo nano SECRETtweets.service</code> and copy paste the following code:</li>
</ol>
<pre data-enlighter-language="shell">[Unit]
Description=SECRETTweets

[Service]
Type=simple
User=amit
ExecStart=/usr/bin/Rscript &quot;/srv/shiny-server/SecretFolder/newTweetGetter.R&quot;
Restart=always
WorkingDirectory= /srv/shiny-server/SecretFolder/
Environment=&quot;LANG=en_US.UTF-8&quot;


[Install]
WantedBy=multi-user.target
</pre>
<p>4. restart the daemon that picks up services? Don’t know why… just do it: <code>sudo systemctl daemon-reload</code></p>
<p>5. Now start the service!! <code>sudo systemctl start SECRETtweets</code></p>
<p> </p>
<p>Now your service is running! You can check the status of it using: <code>systemctl status SECRETtweets.service</code></p>
<p>Where each part does this:</p>
<ul>
<li><strong>Description</strong> is what the thingie does</li>
<li><strong>Type </strong>says how to run it, and “simple” is the default… but check the documentation if u wanna do something more fancy</li>
<li><strong>User</strong> this defines what user is running the service. This is a bit of extra insurance, in case you installed a package as a yourself and not as a superuser (which is the correct way)</li>
<li><strong>ExecStart</strong> is the command to run</li>
<li><strong>Restart</strong> by specifying this to “always”, if the script ever goes down, it’ll automatically restart and start scraping again! <img alt="🙂" src="https://i0.wp.com/s.w.org/images/core/emoji/2.3/72x72/1f642.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"> Super cool, no? <span>WARNING: Not sure about whether this can cause trouble… if twitter is for some reason pissed off and doesn’t want to serve tweets to you anymore, not sure if CONSTANTLY restarting this could get you in trouble. If I get banned, I’ll letchu know… stay tuned</span>)</li>
<li><strong>WorkingDirectory </strong>This part is where the magic happens. Remember earlier on we were worried and worried about HOW to pass the working directory to the R script? This is how!! Now we don’t have to worry about paths on the server anymore!</li>
<li><strong>Environment </strong>is the language</li>
<li><strong>WantedBy</strong> I have no idea what this does and don’t care because it works!</li>
</ul>
<p>So there you go! This is the way to set up a proper service that you can monitor, and treat properly like any formal linux process! Enjoy!</p>
<p> </p>
<p><span>ENDNOTE 1</span></p>
<p>Ok, it’s true… sometimes a Service is the right thing to do, if you have a job that runs for a certain amount of time, finishes, and then you want to run it again discretely later, you should set it up as a cron job. So for those cases, here’s the correct script to check the script is running, even assigning a working directory.</p>
<pre data-enlighter-language="null">if ps aux | grep &quot;R_file_you_want_to_check.R&quot;  | grep -v grep > /dev/null
then
  echo &quot;Running, all good!&quot;
else
  echo &quot;Not running... will restart:&quot;
  cd /path_to_your_working_directory
  Rscript &quot;R_file_you_want_to_check.R&quot; 
fi
</pre>
<p> </p>
<p>save that as <code>chek.sh</code> and assign it to the cron with the output to your home path, like:</p>
<p><code>40 * * * * &quot;/srv/shiny-server/SecretFolder/chek.sh&quot; >> /home/amit/SecretTweets.log 2>&amp;1</code></p>
<p> </p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://amitkohli.com/setting-up-twitter-streamr-service-on-an-ubuntu-server/"> R – AmitKohli.com</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/TV6YS_5kHEY" height="1"></a></div>