Daily Drop Example Report

This is more of a blog post than a report, but the principles are the same. I figured it was easier to document the process of making this report in the report itself!

Working With Observable Framework

Observable just leveled up their platform and also gave us a new way to build data-driven documents/projects with their new Framework (before you judge, naming things is super-hard). With it we can build dashboards and reports. The Framework also lets you publish things to Observable's revamped platform. But, we're getting ahead of ourselves.

The example we're going to walk through, today, for the Drop's WPE is how to make a simple report with the Framework. But, to do that, we need to have the Framework!

Working with it requires Node version 20+. One of the better ways to be able to work with different versions of Node is Node Version Manager (nvm). I'm going to assume you have it installed already or will do so after learning that this is an assumption I'm making.

Let's make sure we're using the right version:

$ nvm use 20
Now using node v20.9.0 (npm v10.1.0)

$ node --version
v20.9.0

You'll be prompted to install the version if it's not already installed.

Oh, yeah. ^^ is a "note". The Framework comes with a ton of crunchy good stuff baked in, like pre-defined CSS classes for things like notes.

Now we have to create a new Framework project:

$ npm init @observablehq
Need to install the following packages:
@observablehq/create@1.0.0
Ok to proceed? (y)
…
┌   observable create  v1.0.0
│
◇  Where to create your project?
│  ./drop-report
│
◇  What to title your project?
│  Drop Report
│
◇  Include sample files to help you get started?
│  Yes, include sample files
│
◇  Install dependencies?
│  Yes, via npm
│
◇  Initialize git repository?
│  Yes
│
◇  Installed! 🎉
│
◇  Next steps… ──────╮
│                    │
│  cd ./drop-report  │
│  npm run dev       │
│                    │
├────────────────────╯
│
└  Problems? https://observablehq.com/framework/getting-started

They've done a pretty solid job with the CLI interface, and you are well-informed on what's going on with every command/subcommand you run.

I always try to break/explore new toys when I get them, so let's see what comes in the observable CLI's tin:

$ cd drop-report
$ npm run observable

> observable
> observable

usage: observable <command>
  create       create a new project from a template
  preview      start the preview server
  build        generate a static site
  login        sign-in to Observable
  logout       sign-out of Observable
  deploy       deploy a project to Observable
  whoami       check authentication status
  convert      convert an Observable notebook to Markdown
  help         print usage information
  version      print the version

Whoa. So much to explore! What is this login I see?

$ npm run observable login

> convert-test@1.0.0 observable
> observable login

┌   observable login

Attention: Observable Framework collects anonymous telemetry to help us improve
           the product. See https://observablehq.com/framework/telemetry for details.
           Set `OBSERVABLE_TELEMETRY_DISABLE=true` to disable.
│
◇  Your confirmation code is UNIQUECODE
│  Open https://observablehq.com/auth-device?code=UNIQUECODE
│  in your browser, and confirm the code matches.
│
◇  You are logged into observablehq.com as boB Rudis (@hrbrmstr).
│
◇   ─────────────────────────────────────────────╮
│                                                │
│  You have access to the following workspaces:  │
│                                                │
│   * GreyNoise Intelligence (@greynoise)        │
│   * boB Rudis (@hrbrmstr)                      │
│                                                │
├────────────────────────────────────────────────╯
│
└  Logged in

Did it really work?

$ npm run observable whoami

> convert-test@1.0.0 observable
> observable whoami


You are logged into observablehq.com as boB Rudis (@hrbrmstr).

You have access to the following workspaces:
 * GreyNoise Intelligence (@greynoise)
 * boB Rudis (@hrbrmstr)

Yep!

OK. Let's see what comes along for the ride in the sample Framework project:

$ tree -AFL 3 -I node_modules
./
├── docs/
│   ├── components/
│   │   └── timeline.js
│   ├── data/
│   │   ├── events.json
│   │   └── launches.csv.js
│   ├── example-dashboard.md
│   ├── example-report.md
│   └── index.md
├── observablehq.config.ts
├── package-lock.json
└── package.json

4 directories, 9 files

docs/components is a place where you can put javascript modules with exported functions/objects so you can keep your report markdown a bit neater (though, you can make and use as many directories as you like)
docs/data is where static data and data loader scripts go. The loader scripts run on build. We'll talk more about them in a bit.
any markdown files in docs will be auto-added to a sidebar in dashboard mode and you can organize them better in 👇🏼
observablehq.config.ts holds configuration options for the project like the title, theme, header/footer, pages organization, etc.

YMMV, but he first thing I'd do if I were you is edit package.json and disable the auto-launching of a browser every time you preview project with the dev command:

  "dev": "observable preview --no-open",

I'm using VS Code (yeah, yeah) for this example and prefer using the Browse Lite plugin for dev-mode previews.

Let's see what we get as the default example:

$ npm run dev

You can read their tutorial for this example. We're going to invoke Iron Man's Clean Slate Protocol and focus on making this simple, sample report.

Creating A Basic Report

Delete the docs/example-dashboard.md and docs/example-report.md files, along with the static JSON and JS loader in data. You can also delete the timeline component in components.

You can now delete everything in index.md, create an empty style.css, and — finally — edit observablehq.config.ts. I made mine look like this:

export default {
  title: "Drop Report",
  theme: "coffee",
  toc: false,
  pager: false,
  footer: `Built with 💙 by @hrbrmstr with Observable's new Framework.`
};

This tells the Framework builder to use the coffee theme, disable the table of contents, disable the paging buttons that would normally be at the bottom of the pages, and set a footer. That title will be appended to the HTML title tag as "| Drop Report".

I'm setting as little as possible in the config file since you have to re-run the dev subcommand each time you make a change and that's kind of a pain (you can upvote the issue to fix this on the Framework's GH, if you like).

In index.md, put this in the YAML header:

---
style: style.css
---

That will let us augment the coffee them with custom styles. I won't go over much about the CSS, save for the fact that you should do something like this at the top to populate the CSS variables so you only have to tweak what you need:

@import url("observablehq:default.css");
@import url("observablehq:theme-coffee.css");

The dev-mode previewer file watcher does not watch the CSS file(s) for changes, so you have to manually reload the page if you tweak the CSS.

Use markdown and HTML as you would with any of these types of systems to make your fancy report. The one big rule you need to remember is that anything in fenced js blocks is going to get executed as if it were a cell in an Observable Notebook. If you need to include JavsScript code examples, use javascript as the language hint or put the js language hint in {}. (View the source of this one to see what I mean.)

Loading Data

While you can use the fetch API to load data, the Framework has a built-in data loader system that will run on build and cache the results. I really like this approach/feature since most dashboard/report idioms are "get data, make report".

You really should read up on loaders, but the TL;DR is that you can pretty much use anything to load data. The only real "requirement" is that whatever it is outputs the return values to stdout and uses a file type hint in the data loader name.

We'll try to keep things simple for this first, example report. I'll re-use some data from the Union of Concerned Scientists (UCS) I've been playing with to remind myself of how evil, dangerous, and daft billionaires are. This UCS dataset holds information about the 🛰️ that are or were in orbit.

I truly dislike data wrangling in JavaScript, so we'll use R to load this data and pull out what we want to provide to this report.

Let's see how horrible this file is (I may be more concerned about these scientists' data handling practices than they are about the satellites):

xdf <- read.csv("https://www.ucsusa.org/media/11493", sep="\t")

colnames(xdf) |> 
  sort() |>
  writeLines()
## Apogee..km.
## Class.of.Orbit
## Comments
## Contractor
## COSPAR.Number
## Country.of.Contractor
## Country.of.Operator.Owner
## Country.Org.of.UN.Registry
## Current.Official.Name.of.Satellite
## Date.of.Launch
## Detailed.Purpose
## Dry.Mass..kg..
## Eccentricity
## Expected.Lifetime..yrs..
## Inclination..degrees.
## Launch.Mass..kg..
## Launch.Site
## Launch.Vehicle
## Longitude.of.GEO..degrees.
## Name.of.Satellite..Alternate.Names
## NORAD.Number
## Operator.Owner
## Perigee..km.
## Period..minutes.
## Power..watts.
## Purpose
## Source
## Source.1
## Source.2
## Source.3
## Source.4
## Source.5
## Source.6
## Source.Used.for.Orbital.Data
## Type.of.Orbit
## Users
## X
## X.1
## X.10
## X.11
## X.12
## X.13
## X.14
## X.15
## X.16
## X.17
## X.18
## X.19
## X.2
## X.20
## X.21
## X.22
## X.23
## X.24
## X.25
## X.26
## X.27
## X.28
## X.29
## X.3
## X.30
## X.31
## X.4
## X.5
## X.6
## X.7
## X.8
## X.9

Yuk.

We don't need to provide all that to this report. We'll just focus on the "date of launch", and "expected lifetime" columns:

xdf <- xdf[,c("Date.of.Launch", "Expected.Lifetime..yrs..")]
colnames(xdf) <- c("launch_date", "expected_lifetime")

str(xdf)
## 'data.frame':	7562 obs. of  2 variables:
##  $ launch_date      : chr  "12/11/19" "1/3/23" "6/23/17" "4/25/16" ...
##  $ expected_lifetime: num  0.5 NA 2 NA 15 15 15 12 15 NA ...

Wow. These folks truly are monsters.

We'll convert the date to something actually useful and then write the shaved down and cleaned up data frame as JSON to stdout:

xdf <- read.csv("https://www.ucsusa.org/media/11493", sep = "\t")

xdf <- xdf[, c("Date.of.Launch", "Expected.Lifetime..yrs..")]
colnames(xdf) <- c("launch_date", "expected_lifetime")

xdf$launch_date <- as.Date(xdf$launch_date, format = "%m/%d/%y")
xdf$year <- as.numeric(format(xdf$launch_date, format = "%Y"))
xdf <- xdf[!is.na(xdf$year),]

xdf |> 
  jsonlite::toJSON() |>
  writeLines()

I saved that in docs as satellites.json.R, and we can load it up and look at it via:

const satellites = await FileAttachment("satellites.json").json()

display(satellites)

That's great for nerds, but what about normal humans?

Now, we'll use it to see how crowded it is up there.

Observable Plot

The Framework comes with all the libraries we have available in Observable-proper, which means we can use Observable Plot to poke at this data.

Please tell me that these new toys will at least last a while…

Build For Production

Playing in the dev sandbox is fun, and all, but we really want to show this report to the world!

To do that, we run npm run build and then use your fav method to put everything in dist to the proper destination (I used rsync to put everything here, where you're reading this from). You can modify package.json to add helper commands to wrap all those tasks up into a bow.

Source Code

The source for this report is on Codeberg.

Observable also has more example you can explore.

Your Turn!

Either modify this project to add more fields to the JSON and use them to show more insights (and play with some of the toys the amazing Observable team included with the Framework) or chart (heh) your own path with a completely new project!

Remember, this is the initial release of Observable Framework, so keep an eye on it, as I suspect they'll be cranking pretty hard on their new creation.