Skip navigation

Category Archives: Rust

See it live before reading!

This is a Lit + WebR reproduction of the OG Shiny Demo App

Lit is a javascript library that makes it a bit easier to work with Web Components, and is especially well-suited in reactive environments.

My recent hack-y WebR experiments have been using Reef which is an even ligher-weight javascript web components-esque library, and it’s a bit more (initially) accessible than Lit. Lit’s focus on “Web Components-first” means that you are kind of forced into a structure, which is good, since reactive things can explode if not managed well.

I also think this might Shiny folks feel a bit more at home.

This is the structure of our Lit + WebR example (I keep rejiggering this layout, which likely frustrates alot of folks 🙃)_

lit-webr
├── css
│   └── style.css             # you know what this is
├── favicon.ico               # when developing locally I want my icon
├── index.html                # you know what this is
├── main.js                   # the core experiment runner
├── md
│   └── main.md               # the core experiment markdown file
├── r-code
│   └── region-plot.R         # we keep longer bits of R code in source files here
├── r.js                      # place for WebR work
├── renderers.js              # these experiment templates always use markdown
├── themes
│   └── ayu-dark.json         # my fav shiki theme
├── utils.js                  # handy utilities (still pretty bare)
├── wc
│   ├── region-plot.js        # 👉🏼 WEB COMPONENT for the plot
│   ├── select-list.js        # 👉🏼               for the regions popup menu
│   └── status-message.js     # 👉🏼               for the status message
├── webr-serviceworker.js.map # not rly necessary; just for clean DevTools console
└── webr-worker.js.map        # ☝🏽

A great deal has changed (due to using Lit) since the last time you saw one of these experiments. You should scan through the source before continuing.

The core changes to index.html are just us registering our web components:

<script type="module" src="./wc/select-list.js"></script>
<script type="module" src="./wc/region-plot.js"></script>
<script type="module" src="./wc/status-message.js"></script>

We could have rolled them up into one JS file and minified them, but we’re keeping things simple for these experiments.

Web Components (“components” from now on) become an “equal citizen” in terms of HTMLElements, and they’re registered right in the DOM.

The next big change is in this file (the rendered main.md), where we use these new components instead of our <div>s. The whittled down version of it is essentially:

<status-message id="status"></status-message>

<region-plot id="regionsOutput" svgId="lit-regions">
  <select-list label="Select a region:" id="regionsInput"></select-list>
</region-plot>

The intent of those elements is pretty clear (much clearer than the <div> versions), which is one aspect of components I like quite a bit.

You’ll also notice components are - (dash) crazy. That’s part of the Web Components spec and is mandatory.

We’re using pretty focused components. What I mean by that is that they’re not very reusable across other projects without copy/paste. Part of that is on me since I don’t do web stuff for a living. Part of it was also to make it easier to show how to use them with WebR.

With more modular code, plus separating out giant chunks of R source means that we can actually put the entirety of main.js right here (I’ve removed all the annotations; please look at main.js to see them; we will be explaining one thing in depth here, vs there, tho.):

import { renderMarkdownInBody } from "./renderers.js";
import * as d3 from "https://cdn.jsdelivr.net/npm/d3@7/+esm";

await renderMarkdownInBody(
  `main`,
  "ayu-dark",
  [ 'javascript', 'r', 'json', 'md', 'xml', 'console' ],
  false
)

let message = document.getElementById("status");
message.text = "WebR Loading…"

import * as R from "./r.js";

message.text = "Web R Initialized!"

await R.webR.installPackages([ "svglite" ])

await R.library(`svglite`)
await R.library(`datasets`)

const regionRender = await globalThis.webR.evalR(await d3.text("r-code/region-plot.R"))

message.text = "{svglite} installed"

const regions = document.getElementById("regionsInput")
const plotOutput = document.getElementById("regionsOutput")

regions.options = await (await R.webR.evalR(`colnames(WorldPhones)`)).toArray()
plotOutput.region = regions.options[ 0 ]
plotOutput.renderFunction = regionRender
plotOutput.render()

message.text = "Ready"

I want to talk a bit about this line from main.js:

const regionRender = await globalThis.webR.evalR(
  await d3.text("r-code/region-plot.R")
)

That fetches the source of the single R file we have in this app, evaluates it, and returns the evaluated value (which is an R function object) to javascript. This is the script:

renderRegions <- function(region, id = "region-plot") {

  # our base plot theme

  list(
    panel.fill = "#001e38",
    bar.fill = "#4a6d88",
    axis.color = "#c6cdd7",
    label.color = "#c6cdd7",
    subtitle.color = "#c6cdd7",
    title.color = "#c6cdd7",
    ticks.color = "#c6cdd7",
    axis.color = "#c6cdd7"
  ) -> theme

  # get our svg graphics device amp'd
  s <- svgstring(width = 8, height = 4, pointsize = 8, id = id, standalone = FALSE)

  # setup theme stuff we can't do in barplot()
  par(
    bg = theme$panel.fill,
    fg = theme$label.color
  )

  # um, it's a barplot
  barplot(
    WorldPhones[, region],
    main = region,
    col = theme$bar.fill,
    sub = "Data from AT&T (1961) The World's Telephones",
    ylab = "Number of Telephones (K)",
    xlab = "Year",
    border = NA,
    col.axis = theme$axis.color,
    col.lab = theme$label.color,
    col.sub = theme$subtitle.color,
    col.main = theme$title.color
  )

  dev.off()

  # get the stringified SVG
  plot_svg <- s()

  # make it responsive
  plot_svg <- sub("width='\\d+(\\.\\d+)?pt'", "width='100%'", plot_svg)
  plot_svg <- sub("height='\\d+(\\.\\d+)?pt'", "", plot_svg)

  # return it
  plot_svg

}

That R function is callable right from javascript. Creating that ability was super brilliant of George (the Godfather of WebR). We actually end up giving it to the component that plots the barplot (see region-plot.js) right here:

plotOutput.renderFunction = regionRender

We’re getting a bit ahead of ourselves, since we haven’t talked about the components yet. We’ll do so, starting with the easiest one to grok, which is in status-message.js and is represented by the <status-message></status-message> tag.

These custom Lit components get everything HTMLElement has, plus whatever else you provide. I’m not going to show the entire source for status-message.js here as it is (lightly) annotated. We’ll just cover the fundamentals, as Lit components also have alot going on and we’re just using a fraction of what they can do. Here’s the outline of what’s in our status-message:

export class StatusMessage extends LitElement {
  static properties = { /* things you can assign to and read from */ }
  static styles = [ /* component-scoped CSS */ ]
  constructor() { /* initialization bits */
  render() { /* what gets called when things change */ }
}
// register it
customElements.define('status-message', StatusMessage);

Our status-message properties just has one property:

static properties = {
  text: {type: String}, // TypeScript annotations are requried by Lit
};

This means when we do:

let message = document.getElementById("status");
message.text = "WebR Loading…"

we are finding our component in the DOM, then updating the property we defined. That will trigger render() each time, and use any component-restricted CSS we’ve setup.

Things get a tad more complicated in select-list.js. We’ll just cover the highlights, starting with the properties:

static properties = {
  id: { type: String },    // gives us easy access to the id we set
  label: { type: String }, // lets us define the label up front
  options: { type: Array } // where the options for the popup will go
};

If you recall, this is how we used them in the source:

<region-plot id="regionsOutput" svgId="lit-regions">
  <select-list label="Select a region:" id="regionsInput"></select-list>
</region-plot>

The id and label properties will be available right away after the custom element creation.

We start option with an empty list:

constructor() {
  super()
  this.options = []
}

Our render() function places the <label> and <select> tags in the DOM and will eventually populate the menu once it has data:

render() {
  const selectId = `select-list-${this.id}`;
  return html`
  <label for="${selectId}">${this.label} 
    <select id="${selectId}" @change=${this._dispatch}>
      ${this.options.map(option => html`<option>${option}</option>`)}
    </select>
  </label>
  `;
}

Their clever use of JS template strings makes it much easier than ugly string concatenation.

That html in the return is doing alot of work, and not just returning text. You gotta read up on Lit to get more info b/c this is already too long.

The way we wired up reactivity in my Reef examples felt kludgy, and even the nicer way to do it in Reef feels kludgy to me. It’s really nice in Lit. This little addition to the <select> tag:

@change=${this._dispatch}

says to call a function named _dispatch whenever the value changes. That’s in the component as well:

_dispatch(e) {
  const options = {
    detail: e.target,
    bubbles: true,
    composed: true,
  };
  this.dispatchEvent(new CustomEvent(`regionChanged`, options));
}

We setup a data structure and then fire off a custom event that our plot component will listen for. We’ve just linked them together on one side. Now we just need to populate the options list, using some data from R:

const regions = document.getElementById("regionsInput")
regions.options = await (await R.webR.evalR(`colnames(WorldPhones)`)).toArray()

That’ll make the menu appear.

Hearkening back to the main.js plot setup:

const plotOutput = document.getElementById("regionsOutput")
plotOutput.region = regions.options[ 0 ]
plotOutput.renderFunction = regionRender
plotOutput.render()

we see that we:

  • find the element
  • set the default region to the first one in the popup
  • assign our R-created rendering function to it
  • and ask it nicely to render right now vs wait for someone to select something

The other side of that (region-plot.js) is a bit more complex. Let’s start with the properties:

static properties = {
  // we keep a local copy for fun
  region: { type: String },

  // this is where our S
  asyncSvg: { type: String },

  // a DOM-accessible id string (cld be handy)
  svgId: { type: String },

  // the function to be called to render
  renderFunction: { type: Function }
};

WebR === “async”, which is why you see that asyncSvg. Async is great and also a pain. There are way more functions in region-plot.js as a result.

We have to have something in renderFunction before WebR is powered up since the component will be alive before that. We’ll give it an anonymous async function that returns an empty SVG.

this.renderFunction = async () => `<svg></svg>`

Oddly enough, our render function does not call the plotting function. This is what it does:

render() {
  return html`
  <div>
  <slot></slot>
  ${unsafeSVG(this.asyncSvg)}
  </div>`;
}

This bit:

<slot></slot>

just tells render() to take whatever is wrapped in the tag and shove it there (it’s a bit more powerful than just that tho).

This bit:

${unsafeSVG(this.asyncSvg)}

is just taking our string with SVG in it and letting Lit know we really want to live dangerously. Lit does its best to help you avoid security issues and SVGs are dangerous.

So, how do we render the plot? With two new functions:

// this is a special async callback mechanism that 
// lets the component behave normally, but do things 
// asynchronously when necessary.
async connectedCallback() {

  super.connectedCallback();

  // THIS IS WHERE WE CALL THE PLOT FUNCTION
  this.asyncSvg = await this.renderFunction(this.region, this.svgId);

  // We'll catch this event when the SELECT list changes or when
  // WE fire it, like we do down below.
  this.addEventListener('regionChanged', async (e) => {
    this.region = e.detail.value;
    const res = await this.renderFunction(this.region, this.svgId);
    // if the result of the function call is from the R function and
    // not the anonymous one we initialized the oject with
    // we need to tap into the `values` slot that gets return
    // with any call to WebR's `toJs()`
    if (res.values) this.asyncSvg = res.values[ 0 ] ;
  });

}

// special function that will get called when we 
// programmatically ask for a forced update
performUpdate() {
  super.performUpdate();
  const options = {
    detail: { value: this.region },
    bubbles: true,
    composed: true,
  };

  // we fire the event so things happen async
  this.dispatchEvent(new CustomEvent(`regionChanged`, options));
}

That finished the wiring up on the plotting end.

Serving ‘Just’ Desserts (Locally)

I highly recommend using the tiny but awesome Rust-powered miniserve to serve things locally during development:

miniserve \
  --header "Cache-Control: no-cache; max-age=300" \
  --header "Cross-Origin-Embedder-Policy: require-corp" \
  --header "Cross-Origin-Opener-Policy: same-origin" \
  --header "Cross-Origin-Resource-Policy: cross-origin" \
  --index index.html \
  .

You can use that (once installed) from the local justfile, which (presently) has four semantically named actions:

  • install-miniserve
  • serve
  • rsync
  • github

You’ll need to make path changes if you decide to use it.

FIN

I realize this is quite a bit to take in, and — as I keep saying — most folks will be better off using WebR in Shiny (when available) or Quarto.

Lit gives us reactivity without the bloat that comes for the ride with Vue and React, so we get to stay in Vanilla JS land. You’ll notice there’s no “npm” or “bundling” or “rollup” here. You get to code in whatever environment you want, and serving WebR-powered pages is, then, as simple as an rsync.

Drop issues at the repo.

Apple is in the final stages of shuttering the DarkSky service/API. They’ve replaced it with WeatherKit, which has both an xOS framework version as well as a REST API. To use either, you need to be a member of the Apple Developer Program (ADP) — $99.00/USD per-year — and calls to the service via either method are free up to 500K/month. After that, Apple has pricing tears.

As a result of the forced-ADP membership fee, I’m not sure how many folks are going to invest in building anything but freemium native or web apps. DarkSky had a generous free tier that only required an API key.

Since I had a {darksky} R package, I recently made a similar {weatherkit} package —https://rud.is/b/2022/07/07/introducing-weatherkit-the-eventual-replacement-r-package-for-darksky/ — complete with a CLI demo program.

Lots of R folks will disagree with the following, but R is a terrible language for CLI tools if you’re not already invested in the R ecosystem. CRAN makes it a pain to modify the user’s local system, and most R things have a ton of dependencies. So, while I generally code R-first, I do not code R-only, especially for CLI tools.

I like Rust more than Golang, and am also getting used to it over C/C++, so I threw together a Rust-based WeatherKit CLI tool shortly after the R one — https://github.com/hrbrmstr/weatherkit-rust. There’s documentation for how to cross all the t’s and dot all the i’s required to get authentication to work.

The GH releases have a signed macOS universal binary and I’m working on decomposing Starship’s wicked cool Rust release builder that uses the equally cool Google release-please to deal up binaries for virtually every platform.

I may make the Rust version a full WeatherKit API library, but I don’t know if I’m going to invest time into something that may just get shoved aside due to the hate I’m expecting to see pointed in Apple’s direction.

In my M-F newsletter today I mentioned an awesome Rust-based HTML/JS/CSS minifier library that also include batteries for a few other languages.

There was no R port, so I made one using {rextendr}. The {rextendr} package makes is as easy to use Rust code in R packages as {Rcpp} does C/C++ code.

It was as simple as adding some dependencies to the Rust Cargo.toml file and then adding one Rust function to the main lib.rs file, and writing a thin wrapper function ({rextendr} can do that, too, but I wanted some default function parameters) for the shim. It took almost no time, and now you, too, can use the utility:

library(minifyhtml)

'
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta charset="UTF-8"/>
  <meta name="viewport" content="width=device-width, initial-scale=1"/>
  <!-- COMMENT -->
  <style>
    * { color: black; }
  </style>
  <title>TITTLE</title>
  </head>
  <body>
    <p>
       Some text
    </p>
    <script>
      console.log("This is a console log message.");
    </script>
  </body>
</html>
' -> src

cat(minify(src))
## <html xmlns=http://www.w3.org/1999/xhtml><meta charset=UTF-8><meta content=width=device-width,initial-scale=1 name=viewport><style>* { color: black; }</style><title>TITTLE</title><body><p>Some text</p><script>console.log("This is a console log message.");</script>

FIN

I have to work out one kink (due to developing on arm64 macOS) and the utility will also be able to minify CSS and JS embedded in HTML.

You can find {minifyhtml} on GitHub.

After a Twitter convo about weather stations I picked up a WeatherFlow Tempest. Setup was quick, but the sensor package died within 24 hours. I was going to give up on it but I had written an R package (for the REST API & UDP broadcast interfaces) and C++ utility (for just the UDP broadcast interface), and the support staff were both friendly and competent and sent me a replacement super quick.

I’ve blathered about the R package already (on Twitter) so am not going to tag that here, but will link to a few repositories (in various languages) that receive the UDP broadcast messages and at least shove them to stdout.

The C++ one is mostly C but gets the job done (it just posts the messages to stdout). It should run everywhere but I only tested on macOS & Linux, because Windows is a terrible operating system nobody should use.

The Golang one has some structured types to consume about half of the JSON messages (I’ve only seen four in the broadcasts so far, and will add more as I see new ones). It’s only more verbose than the C++ one due to the various record type handling. This should run everywhere, though.

For kicks, I threw together a Swift one that is really just Swift-ified C and is a Frankenstein monster that likely shouldn’t be used. (I’ll be making a SwiftUI macOS/iOS/iPadOS app for the UDP broadcast messages, though, soon).

To round out my obsession I also made a Rust version which I’m just in 💙 with (not because of any skill of my own). It’s the smallest source file and is pretty elegant (100% due to Rust, and, again, not me).

All the code/projects are super small, but the Rust source is so tiny that it won’t be too intrusive to post here:

use std::net::UdpSocket;

fn main() -> std::io::Result<()> {

  let mut buf = [0; 1024]; // 1024 byte buffer is plenty
  let s = UdpSocket::bind("0.0.0.0:50222").expect(r#"{"message":"Could not bind to address/port."}"#);

  loop {

    let (n, _) = s.recv_from(&mut buf).expect(r#"{"message":"No broadcasts received."}"#);

    println!("{}", String::from_utf8(buf[..n].to_vec()).unwrap())

  }

}

FIN

If you’re interested in a low-cost weather station with great DIY programming support, I’d definitely (so far, at least) recommend the Tempest. We’ll see if it survives the forthcoming snowpocalypse.

These are the JSON messages it slings over UDP:

{"serial_number":"HB-00069665","type":"hub_status","firmware_revision":"177","uptime":728643,"rssi":-50,"timestamp":1643246011,"reset_flags":"BOR,PIN,POR","seq":72787,"radio_stats":[25,1,0,3,16637],"mqtt_stats":[10,108]}
{"serial_number":"ST-00055227","type":"rapid_wind","hub_sn":"HB-00069665","ob":[1643246013,0.00,0]}
{"serial_number":"ST-00055227","type":"rapid_wind","hub_sn":"HB-00069665","ob":[1643246015,0.00,0]}
{"serial_number":"ST-00055227","type":"device_status","hub_sn":"HB-00069665","timestamp":1643246016,"uptime":106625,"voltage":2.683,"firmware_revision":165,"rssi":-72,"hub_rssi":-66,"sensor_status":655364,"debug":0}
{"serial_number":"ST-00055227","type":"obs_st","hub_sn":"HB-00069665","obs":[[1643246016,0.00,0.00,0.00,0,3,1024.56,-12.82,47.84,0,0.00,0,0.000000,0,0,0,2.683,1]],"firmware_revision":165}