<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/launching-webrockets-at-runconf17/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://ropensci.org/blog/blog/2017/07/05/webrockets"> rOpenSci Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>We, <a rel="nofollow" href="https://github.com/AliciaSchep/" target="_blank">Alicia Schep</a> and <a rel="nofollow" href="https://github.com/MilesMcBain" target="_blank">Miles<br>
McBain</a>, drove the <code>webrockets</code> project<br>
at <a rel="nofollow" href="https://ropensci.org/blog/blog/2017/06/02/unconf2017" target="_blank">#runconf17</a>.<br>
To make progress we solicited code, advice, and entertaining anecdotes<br>
from a host of other attendees, whom we humbly thank for helping to make<br>
our project possible.</p>
<p>This post is divided into two sections: First up we'll relate <a rel="nofollow" href="http://ropensci.org/blog/blog/2017/07/05/webrockets#our-experiences" target="_blank">our<br>
experiences</a>, prompted by some questions we wrote for<br>
one another. Second, we'll put the <a rel="nofollow" href="http://ropensci.org/blog/blog/2017/07/05/webrockets#cool-but-what-is-a-websocket" target="_blank"><code>webrockets</code><br>
package</a> into context and walk you<br>
through a fun example where you can live plot streaming sensor data from<br>
a mobile device.</p>
<h1>Our Experiences</h1>
<h2>Alicia Answers Miles' Questions</h2>
<h3>Q1: What motivated you to apply for #runconf17 and then to work on <code>webrockets</code>?</h3>
<p>I first learned about rOpenSci last summer, while doing an internship at<br>
Genentech. <a rel="nofollow" href="https://ropensci.org/about/#leadership" target="_blank">Jenny Bryan</a> came to<br>
give a seminar and also participated in a &quot;women in computing&quot; group<br>
discussion during her visit. During that discussion, she talked about<br>
how she thought organizations like rOpenSci that are <a rel="nofollow" href="https://ropensci.org/blog/blog/2017/06/23/community" target="_blank">welcoming and<br>
inclusive</a> are<br>
great for helping diversify programming (and other) communities. It<br>
sounded like a really cool organization, and I started exploring some of<br>
the rOpenSci <a rel="nofollow" href="https://ropensci.org/packages/" target="_blank">packages</a> and reading the<br>
<a rel="nofollow" href="https://ropensci.org/blog/" target="_blank">blog</a> but wasn't sure of how to take that<br>
next step and become more actively involved. When I saw the blog post<br>
about the unconf, it seemed like a great opportunity to take that next<br>
step so I went ahead and applied, although I didn't think I had much<br>
chance of being selected — it was a very welcome surprise!</p>
<p>When I got to the unconf, I didn't know what I wanted to work on, as a<br>
lot of the projects sounded interesting. I had proposed a project, but<br>
it seemed like someone else was keen to work on a similar idea after the<br>
unconf, so I ended up circling the room trying to figure out what other<br>
project I might want to work on. I chose the <code>webrockets</code> project for<br>
two main reasons: 1) the use cases that you proposed (making<br>
visualizations that updated based on streaming data!) seemed really<br>
neat, and 2) I wanted to push myself to learn something really new, and<br>
I didn't know much about web sockets! This type of project was something<br>
that I would be very hesitant about jumping into on my own, so the<br>
unconf seemed like a great time for it. I figured you could teach me all<br>
I needed to know about web sockets <img alt="😀" src="https://s.w.org/images/core/emoji/2/72x72/1f600.png" class="wp-smiley"> (and if those reading this<br>
don't know what web sockets are, don't worry… later in this post we'll<br>
<a rel="nofollow" href="http://ropensci.org/blog/blog/2017/07/05/webrockets#cool-but-what-is-a-websocket" target="_blank">explain it to you too</a>!)</p>
<h3>Q2: What were the most challenging parts of #runconf17 for you?</h3>
<p>I'm pretty shy and introverted, so I was nervous about being around so<br>
many new people and the packed schedule. Luckily, everyone I met was<br>
extremely friendly! I did still ended up being pretty exhausted by the<br>
end of each day, but had a great time working with you on <code>webrockets</code><br>
and talking about a wide range of topics (R, food, dogs, politics,<br>
life…) with various smaller groups. It was also really incredible to<br>
be able to get help on our project from some of the most knowledgeable<br>
people possible!</p>
<h3>Q3: What things from #runconf17 will stay with you into the future?</h3>
<p>I thing the big thing that will stay with me is knowing how kind and<br>
helpful people in the R community are. I'm often nervous about asking<br>
for help or contributing to a new project, but I feel much more<br>
comfortable now about doing so in the future!</p>
<h2>Miles Answers Alicia's Questions</h2>
<h3>Q1: What inspired you to work on the <code>webrockets</code> package?</h3>
<p>It wasn't my first choice, that was actually <a rel="nofollow" href="https://github.com/ropensci/unconf17/issues/77" target="_blank">your suggestion for<br>
roxygen-like tagging for makefile<br>
generation</a>, which I<br>
still think is an amazing idea. But I went into the unconf with handful<br>
of issues earmarked and websockets was definitely one of them.</p>
<p>The main attraction to websockets for me is about connecting R to new<br>
kinds of data sources. They could be Internet of Things devices or<br>
sources that reside in Virtual Realities. Websockets provide the base<br>
layer for the protocols you need to navigate to stream data from these<br>
sources. Since I'm starting to encounter these types of sources in my<br>
work, I had a feel for a basic unconf-sized use case to attack.</p>
<p>So as the project room was rapidly clearing on Thursday morning, I was<br>
starting to get a bit disheartened that I might have to move to a<br>
project I hadn't really registered an interest in. Then almost at the<br>
last moment Alicia Schep rocks up, keen and backing her C++ skills,<br>
convincing me we might be able to get something done. That as much as<br>
anything inspired me to work on <code>webrockets</code>.</p>
<h3>Q2: How did working on this package during the unconference differ from your usual R development experience?</h3>
<p>In basically all the ways. I usually work independently and I tend to<br>
bang my head against something for a while before reaching for help. The<br>
unconf encourages just the opposite approach and it works amazingly<br>
well. We had web experts like <a rel="nofollow" href="https://github.com/hrbrmstr" target="_blank">Bob Rudis</a><br>
and <a rel="nofollow" href="https://github.com/jcheng5" target="_blank">Joe Cheng</a> sitting literally metres<br>
away from us, and they were only too happy to point us in the right<br>
directions to short cut a lot of time consuming trial and error.</p>
<p>Also, You and I were <a rel="nofollow" href="https://en.wikipedia.org/wiki/Pair_programming" target="_blank">pair<br>
programming</a> for a good<br>
portion of the time, that's something I rarely do. I was surprised it<br>
worked so well given we had only just met. I think you can't overstate<br>
the effect of the tone set by the rOpenSci team in facilitating this. I<br>
can't speak for You, but I didn't feel like the self-consciousness that<br>
can sometimes occur when coding in front of strangers was a noticeable<br>
presence. We were too focussed on getting something happening!</p>
<h3>Q3: What was the favorite thing you learned during the unconference? (Doesn't have to be <code>webrockets</code> related!)</h3>
<p>Wow. That's hard. I mean there's the little known but dedicated scene<br>
inhabited by the R Karaoke People group, the fact that there exists<br>
someone who takes more unadulterated joy in streaming sensor data than<br>
me (hey Bob!), the hypnotic power of sing song southern accents,<br>
<a rel="nofollow" href="https://ropensci.org/about/#leadership" target="_blank">Karthik</a>'s magic Dolittle-level<br>
powers over dogs…</p>
<p>But my absolute favorite thing learned would be aspects of all these<br>
inspiring people who were just previously names or static 2d avatars and<br>
text on the interwebz. As a far-flung Aussie, it's easy to feel on the<br>
outer of the R community. Post unconf I feel like the online community<br>
is so much more alive, and that has to be a result of having had the<br>
opportunity to commune with some of its brightest inhabitants.</p>
<h1>Cool… But what <em>is</em> a websocket?</h1>
<p>Websockets are communication end-points that occupy the same place in<br>
the network stack as HTTP. To shoehorn the protocols needed by modern<br>
web apps into the request-response model of HTTP has required a lot of<br>
fancy footwork involving polling schemes. These schemes are expensive in<br>
bandwidth due to HTTP header information that has to be included in the<br>
most basic of messages. Websockets strip back and abstract away that<br>
machinery, allowing for application level protocol development without<br>
concern for HTTP headers or URL strings!</p>
<p>You can read a great explanation<br>
<a rel="nofollow" href="http://www.websocket.org/quantum.html" target="_blank">here</a>.</p>
<h1>Webrockets <img align="absmiddle" src="https://i1.wp.com/assets-cdn.github.com/images/icons/emoji/unicode/1f680.png?resize=20%2C20&ssl=1" data-recalc-dims="1"></h1>
<p>We've discussed our personal motivations, but the <a rel="nofollow" href="https://github.com/ropensci/unconf17/issues/43" target="_blank">project<br>
issue</a> was actually<br>
proposed by Bob Rudis. He has been using websockets to control headless<br>
Chrome browsers in the context of automated testing. Check out his repo:<br>
<a rel="nofollow" href="https://github.com/hrbrmstr/decapitated" target="_blank"><code>decapitated</code></a>.</p>
<p>Bob had already hunted down a lightweight C++ websocket library that<br>
compiled on Windows, and he was kind enough to share that with us. So<br>
using the wrapper he created for<br>
<a rel="nofollow" href="https://github.com/dhbaird/easywsclient" target="_blank">easywsclient</a> as a template,<br>
we set about creating a general API to target our unconf challenge.</p>
<p>That challenge was: Can we create a <code>shiny</code> app that makes live animated<br>
plots of streaming data received over a websocket connection? To do this<br>
successfully requires a client interface (inbound) for websockets that<br>
is compatible with the reactive programming paradigm in <code>shiny</code>. It also<br>
requires a well understood test source of streaming data to verify the<br>
app is working properly.</p>
<p>A server interface (outbound) for websockets in R has been available for<br>
some time through RStudio's<br>
<a rel="nofollow" href="https://github.com/rstudio/httpuv" target="_blank"><code>httpuv</code></a> package. We were able to<br>
use this to implement our <a rel="nofollow" href="https://github.com/ropenscilabs/webrockets/blob/master/tests/random_coordintate_server.R" target="_blank">test data<br>
source</a>.</p>
<h2>Streaming data as websocket client</h2>
<h3>Initializing the connection</h3>
<p>We assume that as the client, it is our role to initiate a data<br>
streaming session with the server. So we have a way to initialize a<br>
connection:</p>
<div>
<pre><code data-lang="text"><span></span>con &lt;- ws_connect(url = &quot;ws://localhost:5006/&quot;)
</code></pre>
</div>
<p><code>con</code> is a handle to our websocket connection. It is passed to <code>read</code><br>
methods to get data.</p>
<h3>Getting messages</h3>
<p>With <code>easywsclient</code> there is no concept of a buffer messages piling up<br>
for us. A new received message overwrites the one received prior. If we<br>
have not read it, it is lost. This creates some considerations for us as<br>
consumers of websocket data:</p>
<ul>
<li> How often will we check the websocket for new messages?</li>
<li> How will we deal with no messages?</li>
</ul>
<p>Good answers to these questions depend on the characteristics of the<br>
data stream as well as our application. We have to consider how fast and<br>
how predictably data are arriving, and what data we really need (e.g.<br>
latest vs all history). So some degrees of freedom are required, which<br>
is why we created a few methods to read from websockets:</p>
<div>
<pre><code data-lang="text"><span></span>msg &lt;- ws_receive(ws_ptr = con, timeout = 10) # Check for a message waiting up to 10 milliseconds to receive.
msg &lt;- ws_receive_one(ws_ptr = con, frequency = 5) #Return 1 message, check every 5 milliseconds until it arrives.
msg &lt;- ws_receive_multiple(ws_ptr = con, eventlimit = 10) #Do not return until you have 10 messages.
</code></pre>
</div>
<h3>Getting messages in shiny</h3>
<p>To continually receive messages, we could potentially put calls to<br>
receive messages within a while loop. However, that wouldn't work in the<br>
context of <code>shiny</code> since the app would spend all of its time inside the<br>
<code>while</code> loop, ignoring other inputs/outputs. We got a very entertaining<br>
introduction to the <a rel="nofollow" href="https://github.com/HenrikBengtsson/future" target="_blank"><code>future</code><br>
package</a> and how it might<br>
help us get messages asynchronously from <a rel="nofollow" href="https://github.com/HenrikBengtsson" target="_blank">Henrik<br>
Bengtsson</a>, but ultimately <a rel="nofollow" href="https://github.com/carlganz" target="_blank">Carl<br>
Ganz</a> and <a rel="nofollow" href="https://github.com/jcheng5" target="_blank">Joe<br>
Cheng</a> set us on the right path using some<br>
of <code>shiny</code>'s lesser known classes and functions.</p>
<p>To receive websocket messages in a <code>shiny</code> application, we can place our<br>
call to get the message and the handling of the response in an<br>
<code>observeEvent</code> expression. The event we are observing here isn't a real<br>
event; we use the <code>invalidateLater</code> function to specify that we actually<br>
just want to run the code every X milliseconds.</p>
<div>
<pre><code data-lang="text"><span></span>con &lt;- ws_connect(url = &quot;ws://localhost:5006/&quot;)
ui &lt;- fluidPage(
    plotOutput('plot')
)
server &lt;- function(input, output) {
    values &lt;- reactiveValues(x = NULL, y = NULL)

    observeEvent(invalidateLater(100), {

        new_response &lt;- ws_receive(con, 0)
        if (new_response != &quot;&quot;){
            new_point &lt;- fromJSON(new_response)
            values$x &lt;- c(values$x, new_point$x)
            values$y &lt;- c(values$y, new_point$y)
        }
    }, ignoreNULL = FALSE)

    output$plot &lt;- renderPlot({
        ggplot(data.frame(xval = values$x, yval = values$y),
               aes(x = xval, y=yval)) + geom_point()
    })
}
shinyApp(ui = ui, server = server)
</code></pre>
</div>
<p>Here's what the app looks like as new points are received from the<br>
server:</p>
<p><img alt="test
example" src="https://i2.wp.com/ropensci.org/assets/blog-images/2017-07-05_launching_webrockets/wr_coord_example.gif?w=456" data-recalc-dims="1"></p>
<h2>Streaming sensor data</h2>
<p>We were pretty happy to get the proof of concept <code>shiny</code> app that<br>
updates based on a test server up and running — lots of high fives and<br>
cheering when we saw the first few points getting added to the<br>
plot! We then set our sights higher — could we plot some real sensor<br>
data? Miles set up an app on his phone that would start a websockets<br>
server that would send out messages containing accelerometer data from<br>
the phone. We modified our <code>shiny</code> app to read in and plot that real<br>
(live!) data.</p>
<p>We used <a rel="nofollow" href="https://play.google.com/store/apps/details?id=net.collaud.android.sensorvisualisation&hl=en" target="_blank">this<br>
app</a><br>
from the Android app store to stream our sensor data. <em>Hot tip</em>: be sure<br>
to be running your <code>shiny</code> plotting app on the same wifi network as your<br>
phone. To begin sensor data transmission, hit &quot;start&quot; in the sensor data<br>
app interface and make a note of the websocket port and phone ip<br>
address.</p>
<p>The call to establish the connection is:<br>
<code>con &lt;- ws_connect(url = &quot;ws://</code>