<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/inter-operate-with-mqtt-message-brokers-with-r-a-k-a-live-bbc-subtitles/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/12/14/inter-operate-with-mqtt-message-brokers-with-r-a-k-a-live-bbc-subtitles/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Most of us see the internet through the lens of browsers and apps on our laptops, desktops, watches, TVs and mobile devices. These displays are showing us — for the most part — content designed for human consumption. Sure, apps handle API interactions, but even most of that communication happens over ports 80 or 443. But, there are <em>lots</em> of ports out there; <code>0:65535</code>, in fact (at least TCP-wise). And, all of them have some kind of data, and most of that is still targeted to something for us.</p>
<p>What if I told you the machines are also talking to <em>each other</em> using a thin/efficient protocol that allows one, tiny sensor to talk to hundreds — if not <em>thousands</em> — of systems without even a drop of silicon-laced sweat? How can a mere, constrained sensor do that? Well, it doesn’t do it alone. Many of them share their data over a fairly new protocol dubbed <a rel="nofollow" href="http://mqtt.org/" target="_blank">MQTT</a> (Message Queuing Telemetry Transport).</p>
<p>An MQTT <em>broker</em> watches for devices to <em>publish</em> data under various <a rel="nofollow" href="https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics-best-practices" target="_blank"><em>topics</em></a> and then also watches for other systems to <em>subscribe</em> to said topics and handles the rest of the interchange. The protocol is lightweight enough that fairly low-powered (CPU- and literal electric-use-wise) devices can easily send their data chunks up to a broker, and the entire protocol is robust enough to support a plethora of connections and an equal plethora of types of data.</p>
<h2>Why am I telling you all this?</h2>
<p>Devices that publish to MQTT brokers tend to be in the spectrum of what folks sadly call the “Internet of Things”. It’s a terrible, ambiguous name, but it’s all over the media and most folks have some idea what it means. In the context of MQTT, you can think of it as, say, a single temperature sensor publishing it’s data to an MQTT broker so many other things — including programs written by humans to capture, log and analyze that data — can receive it. This is starting to sound like something that might be right up R’s alley.</p>
<p>There are also potential use-cases where an online data processing system might want to publish data to many clients without said clients having to poll a poor, single-threaded R server constantly.</p>
<p>Having MQTT connectivity for R could be really interesting.</p>
<p>And, now we have the beginnings of said connectivity with the <a rel="nofollow" href="https://github.com/hrbrmstr/mqtt" target="_blank"><code>mqtt</code><img alt="🔗" src="https://i1.wp.com/s.w.org/images/core/emoji/2.3/72x72/1f517.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"> package</a>. </p>
<h2>Another Package? Really?</h2>
<p>Yes, really.</p>
<p>Besides the huge potential for having a direct R-bridge to the MQTT world, I’m work-interested in MQTT since we’ve found over 35,000 of them on the default, plaintext port for MQTT (1883) alone:</p>
<p><a rel="nofollow" href="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/12/mqttmap-1.png?ssl=1" target="_blank"><img data-orig-file="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/12/mqttmap-1.png?fit=2797%2C1249&amp;ssl=1" data-orig-size="2797,1249" data-medium-file="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/12/mqttmap-1.png?fit=300%2C134&amp;ssl=1" data-attachment-id="7557" data-image-meta="{" data-image-description="" src="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/12/mqttmap-1.png?resize=780%2C348&ssl=1" data-permalink="https://rud.is/b/2017/12/14/inter-operate-with-mqtt-message-brokers-with-r-a-k-a-live-bbc-subtitles/mqttmap-2/" data-large-file="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/12/mqttmap-1.png?fit=530%2C237&amp;ssl=1" alt="" data-image-title="mqttmap" width="450" data-comments-opened="1" class="wp-image-7557" data-recalc-dims="1"></a></p>
<p>There are enough of them that I don’t even need to show a base map.</p>
<p>Some of these servers require authentication and others aren’t doing much of anything. But, there are a number of them hosted by corporations and individuals that are exposing real data. <a rel="nofollow" href="http://owntracks.org/" target="_blank">OwnTracks</a> seems to be one of the more popular self-/badly-hosted ones.</p>
<p>Then, there are others — like <code>test.mosquitto.org</code> — which deliberately run open MQTT servers for “testing”. There definitely is testing going on there, but there are also real services using it as a production broker. The <code>mqtt</code> package is based on the <a rel="nofollow" href="https://mosquitto.org/documentation/" target="_blank">mosquitto C library</a>, so it’s only fitting that we show a few examples from its own test site here.</p>
<p>For now, there’s really one function: <code>topic_subscribe()</code>. Eventually, R will be able to publish to a broker and do more robust data collection operations (say, to make a live MQTT dashboard in Shiny). The <code>topic_subscribe()</code> function is an all-in one tool that enables you to:</p>
<ul>
<li>connect to a broker</li>
<li>subscribe to a topic</li>
<li>pass in R callback functions which will be executed on connect, disconnect and when new messages come in</li>
</ul>
<p>That’s plenty of functionality to do some fun things. </p>
<h2>What’s the <strike>frequency</strike>temperature, Kenneth?</h2>
<p>The mosquitto test server has one topic — <code>/outbox/crouton-demo/temperature</code> — which is a fake temperature sensor that just sends data periodically so you have something to test with. Let’s capture 50 samples and plot them.</p>
<p>Since we’re using a callback we have to use the tricksy <code>&lt;-</code> operator to store/update variables outside the callback function. And, we should pre-allocate space for said data to avoid needlessly growing objects. Here’s a complete code-block:</p>
<pre><code>library(mqtt) # devtools::install_github(&quot;hrbrmstr/mqtt&quot;)
library(jsonlite)
library(hrbrthemes)
library(tidyverse)

i &lt;- 0 # initialize our counter
max_recs &lt;- 50 # max number of readings to get

readings &lt;- vector(&quot;character&quot;, max_recs)

# our callback function
temp_cb &lt;- function(id, topic, payload, qos, retain) {

  i &lt;- i + 1 # update the counter
  readings[i] &lt;- readBin(payload, &quot;character&quot;) # update our larger-scoped vector

  return(if (i==max_recs) &quot;quit&quot; else &quot;go&quot;) # need to send at least &quot;&quot;. &quot;quit&quot; == done

}

topic_subscribe(
  topic = &quot;/outbox/crouton-demo/temperature&quot;,
  message_callback=temp_cb
)

# each reading looks like this:
# {&quot;update&quot;: {&quot;labels&quot;:[4631],&quot;series&quot;:[[68]]}}
map(readings, fromJSON) %>%
  map(unlist) %>%
  map_df(as.list) %>%
  ggplot(aes(update.labels, update.series)) +
  geom_line() +
  geom_point() +
  labs(x=&quot;Reading&quot;, y=&quot;Temp (F)&quot;, title=&quot;Temperature via MQTT&quot;) +
  theme_ipsum_rc(grid=&quot;XY&quot;)</code></pre>
<p><a rel="nofollow" href="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/12/temp.png?ssl=1" target="_blank"><img data-orig-file="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/12/temp.png?fit=1692%2C812&amp;ssl=1" data-orig-size="1692,812" data-medium-file="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/12/temp.png?fit=300%2C144&amp;ssl=1" data-attachment-id="7560" data-image-meta="{" data-image-description="" src="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/12/temp.png?resize=780%2C374&ssl=1" data-permalink="https://rud.is/b/2017/12/14/inter-operate-with-mqtt-message-brokers-with-r-a-k-a-live-bbc-subtitles/temp/" data-large-file="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/12/temp.png?fit=530%2C254&amp;ssl=1" alt="" data-image-title="temp" width="450" data-comments-opened="1" class="wp-image-7560" data-recalc-dims="1"></a></p>
<p>We setup <code>temp_cb()</code> to be our callback and <code>topic_subscribe()</code> ensures that the underlying mosquitto library will call it every time a new message is published to that topic. The chart really shows how synthetic the data is.</p>
<h2>Subtitles from the Edge</h2>
<p>Temperature sensors are just the sort of thing that MQTT was designed for. But, we don’t need to be stodgy about our use of MQTT.</p>
<p>Just about a year ago from this post, the BBC launched <a rel="nofollow" href="http://www.bbc.co.uk/mediacentre/latestnews/2016/BBC-launches-subtitles-for-live-channels-on-BBC-iPlayer-in-world-first" target="_blank">live subtitles for iPlayer</a>. Residents of the Colonies may not know what iPlayer is, but it’s the “app” that lets UK citizens watch BBC programmes on glowing rectangles that aren’t proper tellys. Live subtitles are <em>hard</em> to produce well (and get right) and the BBC making the effort to do so also on their digital platform is quite commendable. We U.S. folks will likely be charged $0.99 for each set of digital subtitles now that net neutrality is gone.</p>
<p>Now, some clever person(s) wired up some of these live subtitles to MQTT topics. We can wire up our own code in R to read them live:</p>
<pre><code>bbc_callback &lt;- function(id, topic, payload, qos, retain) {
  cat(crayon::green(readBin(payload, &quot;character&quot;)), &quot;\n&quot;, sep=&quot;&quot;)
  return(&quot;&quot;) # ctrl-c will terminate
}

mqtt::topic_subscribe(topic = &quot;bbc/subtitles/bbc_news24/raw&quot;,
                      connection_callback=mqtt::mqtt_silent_connection_callback,
                      message_callback=bbc_callback)</code></pre>
<p>In this case, <code>control-c</code> terminates things (cleanly).</p>
<p><a rel="nofollow" href="https://i1.wp.com/rud.is/b/wp-content/uploads/2017/12/mqtt.gif?ssl=1" target="_blank"><img data-orig-file="https://i1.wp.com/rud.is/b/wp-content/uploads/2017/12/mqtt.gif?fit=1410%2C856&amp;ssl=1" data-orig-size="1410,856" data-medium-file="https://i1.wp.com/rud.is/b/wp-content/uploads/2017/12/mqtt.gif?fit=300%2C182&amp;ssl=1" data-attachment-id="7561" data-image-meta="{" data-image-description="" src="https://i1.wp.com/rud.is/b/wp-content/uploads/2017/12/mqtt.gif?resize=780%2C474&ssl=1" data-permalink="https://rud.is/b/2017/12/14/inter-operate-with-mqtt-message-brokers-with-r-a-k-a-live-bbc-subtitles/mqtt/" data-large-file="https://i1.wp.com/rud.is/b/wp-content/uploads/2017/12/mqtt.gif?fit=530%2C322&amp;ssl=1" alt="" data-image-title="mqtt" width="450" data-comments-opened="1" class="wp-image-7561" data-recalc-dims="1"></a></p>
<p>You could easily modify the above code to have a bot that monitors for certain keywords then sends windowed chunks of subtitled text to some other system (Slack, database, etc). Or, create an online tidy text analysis workflow from them.</p>
<h2>FIN</h2>
<p>The code is <a rel="nofollow" href="https://github.com/hrbrmstr/mqtt" target="_blank">on GitHub</a> and all input/contributions are welcome and encouraged. Some necessary TBDs are authentication &amp; encryption. But, how would <em>you</em> like the API to look for using it, say, in Shiny apps? What should publishing look like? What helper functions would be useful (ones to slice &amp; dice topic names or another to convert raw message text more safely)? Should there be an R MQTT “DSL”? Lots of things to ponder and so many sites to “test”!</p>
<h2>P.S.</h2>
<p>In case you are concerned about the unusually boring R package name, I wanted to use <code>RIoT</code> (lower-cased, of course) but <code>riot</code> is, alas, already taken.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2017/12/14/inter-operate-with-mqtt-message-brokers-with-r-a-k-a-live-bbc-subtitles/"> R – rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/OC_s-mSH3O8" height="1"></a></div>