Skip navigation

Author Archives: hrbrmstr

Don't look at me…I do what he does — just slower. #rstats avuncular • ?Resistance Fighter • Cook • Christian • [Master] Chef des Données de Sécurité @ @rapid7

Introducing {weatherkit}: The Eventual Replacement R Package for {darksky}

My {darksky} package has been around for years, now, and the service that powers it was purchased by Apple before the pandemic. The DarkSky API is slated to be shuttered in December of this year and is being replaced by Apple’s WeatherKit xOS Framework and REST API.

I’ve started work on a {weatherkit} package which uses the WeatherKit REST API. You’ll need an Apple Developer account and will also need to setup some items in said account, and locally so you can authenticate to the API. Once all the authentication bits are setup, it’s pretty easy to get the weather data:

wx <- wxkit_weather(43.2683199, -70.8635506)
wx <- wx_tidy(wx)
str(wx)
## List of 4
##  $ currentWeather  :List of 18
##   ..$ name                  : chr "CurrentWeather"
##   ..$ metadata              :List of 8
##   .. ..$ attributionURL: chr "https://weather-data.apple.com/legal-attribution.html"
##   .. ..$ expireTime    : POSIXct[1:1], format: "2022-07-07 12:28:07"
##   .. ..$ latitude      : num 43.3
##   .. ..$ longitude     : num -70.9
##   .. ..$ readTime      : POSIXct[1:1], format: "2022-07-07 12:23:07"
##   .. ..$ reportedTime  : POSIXct[1:1], format: "2022-07-07 10:48:55"
##   .. ..$ units         : chr "m"
##   .. ..$ version       : int 1
##   ..$ asOf                  : POSIXct[1:1], format: "2022-07-07 12:23:07"
##   ..$ cloudCover            : num 0.29
##   ..$ conditionCode         : chr "MostlyClear"
##   ..$ daylight              : logi TRUE
##   ..$ humidity              : num 0.68
##   ..$ precipitationIntensity: num 0
##   ..$ pressure              : num 1018
##   ..$ pressureTrend         : chr "rising"
##   ..$ temperature           : num 19.5
##   ..$ temperatureApparent   : num 19.4
##   ..$ temperatureDewPoint   : num 13.5
##   ..$ uvIndex               : int 2
##   ..$ visibility            : num 29413
##   ..$ windDirection         : int 50
##   ..$ windGust              : num 12
##   ..$ windSpeed             : num 4.42
##  $ forecastDaily   :List of 3
##   ..$ name    : chr "DailyForecast"
##   ..$ metadata:List of 8
##   .. ..$ attributionURL: chr "https://weather-data.apple.com/legal-attribution.html"
##   .. ..$ expireTime    : POSIXct[1:1], format: "2022-07-07 13:23:07"
##   .. ..$ latitude      : num 43.3
##   .. ..$ longitude     : num -70.9
##   .. ..$ readTime      : POSIXct[1:1], format: "2022-07-07 12:23:07"
##   .. ..$ reportedTime  : POSIXct[1:1], format: "2022-07-07 10:48:55"
##   .. ..$ units         : chr "m"
##   .. ..$ version       : int 1
##   ..$ days    :'data.frame': 10 obs. of  26 variables:
##   .. ..$ forecastStart      : POSIXct[1:10], format: "2022-07-07 04:00:00" "2022-07-08 04:00:00" "2022-07-09 04:00:00" "2022-07-10 04:00:00" ...
##   .. ..$ forecastEnd        : POSIXct[1:10], format: "2022-07-08 04:00:00" "2022-07-09 04:00:00" "2022-07-10 04:00:00" "2022-07-11 04:00:00" ...
##   .. ..$ conditionCode      : chr [1:10] "PartlyCloudy" "PartlyCloudy" "MostlyClear" "MostlyClear" ...
##   .. ..$ maxUvIndex         : int [1:10] 7 7 7 8 7 6 7 4 5 4
##   .. ..$ moonPhase          : chr [1:10] "firstQuarter" "firstQuarter" "waxingGibbous" "waxingGibbous" ...
##   .. ..$ moonrise           : POSIXct[1:10], format: "2022-07-07 17:38:12" "2022-07-08 18:50:47" "2022-07-09 20:07:35" "2022-07-10 21:27:35" ...
##   .. ..$ moonset            : POSIXct[1:10], format: "2022-07-07 04:32:48" "2022-07-08 04:54:51" "2022-07-09 05:20:27" "2022-07-10 05:51:50" ...
##   .. ..$ precipitationAmount: num [1:10] 0 0.49 0 0 0 1.32 0.24 3.44 5.07 8.35
##   .. ..$ precipitationChance: num [1:10] 0.01 0.15 0.07 0 0.07 0.39 0.37 0.4 0.47 0.44
##   .. ..$ precipitationType  : chr [1:10] "clear" "rain" "clear" "clear" ...
##   .. ..$ snowfallAmount     : num [1:10] 0 0 0 0 0 0 0 0 0 0
##   .. ..$ solarMidnight      : POSIXct[1:10], format: "2022-07-07 04:48:29" "2022-07-08 04:48:39" "2022-07-09 04:48:49" "2022-07-10 04:48:58" ...
##   .. ..$ solarNoon          : POSIXct[1:10], format: "2022-07-07 16:48:26" "2022-07-08 16:48:35" "2022-07-09 16:48:44" "2022-07-10 16:48:53" ...
##   .. ..$ sunrise            : POSIXct[1:10], format: "2022-07-07 09:10:59" "2022-07-08 09:11:42" "2022-07-09 09:12:26" "2022-07-10 09:13:11" ...
##   .. ..$ sunriseCivil       : POSIXct[1:10], format: "2022-07-07 08:36:06" "2022-07-08 08:36:53" "2022-07-09 08:37:41" "2022-07-10 08:38:31" ...
##   .. ..$ sunriseNautical    : POSIXct[1:10], format: "2022-07-07 07:50:45" "2022-07-08 07:51:39" "2022-07-09 07:52:36" "2022-07-10 07:53:34" ...
##   .. ..$ sunriseAstronomical: POSIXct[1:10], format: "2022-07-07 06:55:17" "2022-07-08 06:56:30" "2022-07-09 06:57:46" "2022-07-10 06:59:04" ...
##   .. ..$ sunset             : POSIXct[1:10], format: "2022-07-08 00:25:50" "2022-07-09 00:25:26" "2022-07-10 00:24:59" "2022-07-11 00:24:30" ...
##   .. ..$ sunsetCivil        : POSIXct[1:10], format: "2022-07-08 01:00:39" "2022-07-09 01:00:10" "2022-07-10 00:59:38" "2022-07-11 00:59:04" ...
##   .. ..$ sunsetNautical     : POSIXct[1:10], format: "2022-07-08 01:46:01" "2022-07-09 01:45:23" "2022-07-10 01:44:42" "2022-07-11 01:43:58" ...
##   .. ..$ sunsetAstronomical : POSIXct[1:10], format: "2022-07-08 02:41:14" "2022-07-09 02:40:16" "2022-07-10 02:39:14" "2022-07-11 02:38:09" ...
##   .. ..$ temperatureMax     : num [1:10] 25.8 28.7 24.9 25.4 28.9 ...
##   .. ..$ temperatureMin     : num [1:10] 13.7 16.3 14.8 12.2 12.4 ...
##   .. ..$ daytimeForecast    :'data.frame':   10 obs. of  11 variables:
##   .. .. ..$ forecastStart      : POSIXct[1:10], format: "2022-07-07 11:00:00" "2022-07-08 11:00:00" "2022-07-09 11:00:00" "2022-07-10 11:00:00" ...
##   .. .. ..$ forecastEnd        : POSIXct[1:10], format: "2022-07-07 23:00:00" "2022-07-08 23:00:00" "2022-07-09 23:00:00" "2022-07-10 23:00:00" ...
##   .. .. ..$ cloudCover         : num [1:10] 0.39 0.45 0.33 0.11 0.42 0.69 0.39 0.95 0.87 0.88
##   .. .. ..$ conditionCode      : chr [1:10] "PartlyCloudy" "PartlyCloudy" "MostlyClear" "Clear" ...
##   .. .. ..$ humidity           : num [1:10] 0.57 0.58 0.54 0.47 0.49 0.63 0.64 0.71 0.7 0.66
##   .. .. ..$ precipitationAmount: num [1:10] 0 0.31 0 0 0 0.26 0.17 3.15 0.22 1.37
##   .. .. ..$ precipitationChance: num [1:10] 0 0.09 0.04 0 0.02 0.29 0.16 0.31 0.33 0.3
##   .. .. ..$ precipitationType  : chr [1:10] "clear" "rain" "clear" "clear" ...
##   .. .. ..$ snowfallAmount     : num [1:10] 0 0 0 0 0 0 0 0 0 0
##   .. .. ..$ windDirection      : int [1:10] 155 263 122 237 231 228 219 98 39 62
##   .. .. ..$ windSpeed          : num [1:10] 8.73 9.42 7.42 6.23 9.75 ...
##   .. ..$ overnightForecast  :'data.frame':   10 obs. of  11 variables:
##   .. .. ..$ forecastStart      : POSIXct[1:10], format: "2022-07-07 23:00:00" "2022-07-08 23:00:00" "2022-07-09 23:00:00" "2022-07-10 23:00:00" ...
##   .. .. ..$ forecastEnd        : POSIXct[1:10], format: "2022-07-08 11:00:00" "2022-07-09 11:00:00" "2022-07-10 11:00:00" "2022-07-11 11:00:00" ...
##   .. .. ..$ cloudCover         : num [1:10] 0.49 0.5 0.15 0.37 0.46 0.4 0.88 0.91 0.8 NA
##   .. .. ..$ conditionCode      : chr [1:10] "PartlyCloudy" "PartlyCloudy" "MostlyClear" "MostlyClear" ...
##   .. .. ..$ humidity           : num [1:10] 0.78 0.78 0.71 0.73 0.69 0.81 0.83 0.85 0.84 NA
##   .. .. ..$ precipitationAmount: num [1:10] 0.06 0.11 0 0 0 1.11 0.04 2.26 6.47 NA
##   .. .. ..$ precipitationChance: num [1:10] 0.05 0.07 0.01 0.02 0.1 0.27 0.24 0.31 0.31 NA
##   .. .. ..$ precipitationType  : chr [1:10] "rain" "rain" "clear" "clear" ...
##   .. .. ..$ snowfallAmount     : num [1:10] 0 0 0 0 0 0 0 0 0 NA
##   .. .. ..$ windDirection      : int [1:10] 192 341 347 223 218 242 276 13 49 NA
##   .. .. ..$ windSpeed          : num [1:10] 9.92 7.15 6.59 5.52 10.95 ...
##   .. ..$ restOfDayForecast  :'data.frame':   10 obs. of  11 variables:
##   .. .. ..$ forecastStart      : POSIXct[1:10], format: "2022-07-07 12:23:07" NA NA NA ...
##   .. .. ..$ forecastEnd        : POSIXct[1:10], format: "2022-07-08 04:00:00" NA NA NA ...
##   .. .. ..$ cloudCover         : num [1:10] 0.47 NA NA NA NA NA NA NA NA NA
##   .. .. ..$ conditionCode      : chr [1:10] "PartlyCloudy" NA NA NA ...
##   .. .. ..$ humidity           : num [1:10] 0.6 NA NA NA NA NA NA NA NA NA
##   .. .. ..$ precipitationAmount: num [1:10] 0 NA NA NA NA NA NA NA NA NA
##   .. .. ..$ precipitationChance: num [1:10] 0.01 NA NA NA NA NA NA NA NA NA
##   .. .. ..$ precipitationType  : chr [1:10] "clear" NA NA NA ...
##   .. .. ..$ snowfallAmount     : num [1:10] 0 NA NA NA NA NA NA NA NA NA
##   .. .. ..$ windDirection      : int [1:10] 163 NA NA NA NA NA NA NA NA NA
##   .. .. ..$ windSpeed          : num [1:10] 9.7 NA NA NA NA NA NA NA NA NA
##  $ forecastHourly  :List of 3
##   ..$ name    : chr "HourlyForecast"
##   ..$ metadata:List of 8
##   .. ..$ attributionURL: chr "https://weather-data.apple.com/legal-attribution.html"
##   .. ..$ expireTime    : POSIXct[1:1], format: "2022-07-07 13:23:07"
##   .. ..$ latitude      : num 43.3
##   .. ..$ longitude     : num -70.9
##   .. ..$ readTime      : POSIXct[1:1], format: "2022-07-07 12:23:07"
##   .. ..$ reportedTime  : POSIXct[1:1], format: "2022-07-07 10:48:55"
##   .. ..$ units         : chr "m"
##   .. ..$ version       : int 1
##   ..$ hours   :'data.frame': 243 obs. of  20 variables:
##   .. ..$ forecastStart         : POSIXct[1:243], format: "2022-07-07 02:00:00" "2022-07-07 03:00:00" "2022-07-07 04:00:00" "2022-07-07 05:00:00" ...
##   .. ..$ cloudCover            : num [1:243] 0.02 0.01 0.02 0.31 0.44 0.74 0.3 1 0.96 0.32 ...
##   .. ..$ conditionCode         : chr [1:243] "Clear" "Clear" "Clear" "MostlyClear" ...
##   .. ..$ daylight              : logi [1:243] FALSE FALSE FALSE FALSE FALSE FALSE ...
##   .. ..$ humidity              : num [1:243] 0.74 0.78 0.81 0.83 0.86 0.88 0.9 0.92 0.88 0.83 ...
##   .. ..$ precipitationAmount   : num [1:243] 0 0 0 0 0 0 0 0 0 0 ...
##   .. ..$ precipitationIntensity: num [1:243] 0 0 0 0 0 0 0 0 0 0 ...
##   .. ..$ precipitationChance   : num [1:243] 0 0 0 0 0 0 0 0 0 0 ...
##   .. ..$ precipitationType     : chr [1:243] "clear" "clear" "clear" "clear" ...
##   .. ..$ pressure              : num [1:243] 1014 1015 1016 1016 1016 ...
##   .. ..$ pressureTrend         : chr [1:243] "rising" "rising" "rising" "rising" ...
##   .. ..$ snowfallIntensity     : num [1:243] 0 0 0 0 0 0 0 0 0 0 ...
##   .. ..$ temperature           : num [1:243] 18.3 17 16.3 15.7 14.9 ...
##   .. ..$ temperatureApparent   : num [1:243] 18.2 16.9 16.1 15.5 14.7 ...
##   .. ..$ temperatureDewPoint   : num [1:243] 13.6 13.1 12.9 12.8 12.6 ...
##   .. ..$ uvIndex               : int [1:243] 0 0 0 0 0 0 0 0 0 1 ...
##   .. ..$ visibility            : num [1:243] 28105 26514 24730 23883 23669 ...
##   .. ..$ windDirection         : int [1:243] 315 302 315 308 310 298 307 316 319 6 ...
##   .. ..$ windGust              : num [1:243] 2.93 2.56 2.92 3.25 3.35 ...
##   .. ..$ windSpeed             : num [1:243] 2.93 2.56 2.92 3.25 3.35 3.09 3.51 2.91 2.36 4.55 ...
##  $ forecastNextHour:List of 6
##   ..$ name         : chr "NextHourForecast"
##   ..$ metadata     :List of 9
##   .. ..$ attributionURL: chr "https://weather-data.apple.com/legal-attribution.html"
##   .. ..$ expireTime    : POSIXct[1:1], format: "2022-07-07 13:23:07"
##   .. ..$ language      : chr "en-US"
##   .. ..$ latitude      : num 43.3
##   .. ..$ longitude     : num -70.9
##   .. ..$ providerName  : chr "US National Weather Service"
##   .. ..$ readTime      : POSIXct[1:1], format: "2022-07-07 12:23:07"
##   .. ..$ units         : chr "m"
##   .. ..$ version       : int 1
##   ..$ summary      :'data.frame':    1 obs. of  4 variables:
##   .. ..$ startTime             : POSIXct[1:1], format: "2022-07-07 12:24:00"
##   .. ..$ condition             : chr "clear"
##   .. ..$ precipitationChance   : num 0
##   .. ..$ precipitationIntensity: num 0
##   ..$ forecastStart: POSIXct[1:1], format: "2022-07-07 12:24:00"
##   ..$ forecastEnd  : POSIXct[1:1], format: "2022-07-07 13:45:00"
##   ..$ minutes      :'data.frame':    81 obs. of  3 variables:
##   .. ..$ startTime             : POSIXct[1:81], format: "2022-07-07 12:24:00" "2022-07-07 12:25:00" "2022-07-07 12:26:00" "2022-07-07 12:27:00" ...
##   .. ..$ precipitationChance   : num [1:81] 0 0 0 0 0 0 0 0 0 0 ...
##   .. ..$ precipitationIntensity: num [1:81] 0 0 0 0 0 0 0 0 0 0 ...

The wx_tidy() function, for now, only does date-time string conversion to POSIXct objects, but it may do more in the future.

It doesn’t appear that historical weather data is available, yet, so you’re limited to using the API to get daily and hourly conditions and forecasts for the present plus some days. As such, I’ve focused a bit on some helper functions to show current conditions and forecasts in the R console/stdout:

current_conditions(wx)

##  Weather for (43.268, -70.864) as of 2022-07-07 08:23:07
## 
##  Conditions: Mostly Clear
## Temperature: 67.08°F
##  Feels like: 66.92°F
##   Dew Point: 56.28°F
##        Wind: 2.7 mph (NE)
##    Pressure: 1017.68 mb (rising)
##  Visibility: 18 miles
##    UV Index: 🟩 2 (Low)
## 
## https://weather-data.apple.com/legal-attribution.html

hourly_forecast(wx)

##  Weather forecast for (43.268, -70.864) as of 2022-07-07 08:23:07
## 
## Today @ 09:00 │ 🌡 69°F │ 💦 63% │ 1018 mb — │ 😎 │ Mostly Clear  │ 🟨
##       @ 10:00 │ 🌡 71°F │ 💦 58% │ 1018 mb — │ 😎 │ Mostly Clear  │ 🟨
##       @ 11:00 │ 🌡 74°F │ 💦 55% │ 1018 mb — │ 😎 │ Mostly Clear  │ 🟧
##       @ 12:00 │ 🌡 75°F │ 💦 53% │ 1017 mb — │ 😎 │ Partly Cloudy │ 🟧
##       @ 13:00 │ 🌡 77°F │ 💦 51% │ 1017 mb ↓ │ 😎 │ Partly Cloudy │ 🟧
##       @ 14:00 │ 🌡 78°F │ 💦 50% │ 1016 mb ↓ │ 😎 │ Partly Cloudy │ 🟧
##       @ 15:00 │ 🌡 78°F │ 💦 50% │ 1016 mb ↓ │ 😎 │ Partly Cloudy │ 🟨
##       @ 16:00 │ 🌡 77°F │ 💦 52% │ 1016 mb ↓ │ 😎 │ Partly Cloudy │ 🟨
##       @ 17:00 │ 🌡 76°F │ 💦 55% │ 1015 mb ↓ │ 😎 │ Partly Cloudy │ 🟩
##       @ 18:00 │ 🌡 75°F │ 💦 58% │ 1015 mb — │ 😎 │ Partly Cloudy │ 🟩
##       @ 19:00 │ 🌡 73°F │ 💦 62% │ 1015 mb — │ 😎 │ Mostly Clear  │ 🟩
##       @ 20:00 │ 🌡 70°F │ 💦 67% │ 1015 mb — │ 😎 │ Partly Cloudy │ 🟩
##       @ 21:00 │ 🌡 68°F │ 💦 71% │ 1015 mb — │ 🌕 │ Mostly Cloudy │ 🟩
##       @ 22:00 │ 🌡 67°F │ 💦 74% │ 1015 mb — │ 🌕 │ Mostly Cloudy │ 🟩
##       @ 23:00 │ 🌡 67°F │ 💦 74% │ 1015 mb — │ 🌕 │ Mostly Cloudy │ 🟩
##   Fri @ 00:00 │ 🌡 66°F │ 💦 74% │ 1015 mb — │ 🌕 │ Partly Cloudy │ 🟩
##       @ 01:00 │ 🌡 65°F │ 💦 78% │ 1015 mb — │ 🌕 │ Partly Cloudy │ 🟩
##       @ 02:00 │ 🌡 64°F │ 💦 81% │ 1015 mb — │ 🌕 │ Mostly Clear  │ 🟩
##       @ 03:00 │ 🌡 63°F │ 💦 83% │ 1015 mb — │ 🌕 │ Partly Cloudy │ 🟩
##       @ 04:00 │ 🌡 62°F │ 💦 85% │ 1015 mb — │ 🌕 │ Partly Cloudy │ 🟩
## 
## https://weather-data.apple.com/legal-attribution.html

Note that the attribution is required by Apple.

I’ll likely add a daily forecast console printer soon.

There are a few helper functions in the package for value conversion between unit systems, iconifying some values, and working with time zones. {weatherkit} uses lutz::tz_lookup_coords() in places to auto-determine the time zone from lat/lng pairs, and also includes a function to intuit lat/lng from an IP address using ipapi (they have a generous free tier).

As of the timestamp on this blog post, Apple’s WeatherKit provides up to 500,000 API calls a month per Apple Developer Program membership. If you need additional API calls, monthly subscription plans will be available for purchase sometime after the beta is officially over. This is the expected pricing:

  • 500,000 calls/month: Included with membership
  • 1 million calls/month: US$ 49.99
  • 2 million calls/month: US$ 99.99
  • 5 million calls/month: US$ 249.99
  • 10 million calls/month: US$ 499.99
  • 20 million calls/month: US$ 999.99

Apple’s WeatherKit documentation consistently says “Apple Developer Program membership”, which seems to indicate you need to give them money every year to use the REST API. We’ll see if that’s truly the case after the service leaves beta status.

FIN

Kick the tyres & drop issues/PRs as one may be wont to do.

Saying The Quiet Part Out Loud On Juneteenth

I realize you have to be living under a rock in the U.S. to not know that yesterday, was Juneteenth (a portmanteau of “June Nineteenth”). Still, I feel compelled to explain that said date marks the day when federal troops arrived in Galveston, Texas in 1865 to take control of the state and ensure that all enslaved people be freed. (See, Texas has always been kinda horrible when it comes to basic human decency). The arrival of said troops came ~2.5 years after the signing of the Emancipation Proclamation. The day honors the end to slavery in the United States.

GreyNoise (my employer) — like an increasing number of organizations in the U.S. — observes Juneteenth as a company-wide holiday, and I’ll be spending part of today pondering that word, “end”, in the last sentence of the previous paragraph.

There are two major political parties in the U.S. and one just decided, this past Juneteenth weekend, we no longer need the Voting Rights Act of 1965. March 7th of that year was a Sunday, known today as “Bloody Sunday” (it turns out there have been far too many of those kinds of Sundays). It was a day when Alabama state troopers beat & whipped voting rights advocates with nightsticks, and also used chemical weapons on them, in an effort to keep in place discriminatory practices that were extensively used to prevent African Americans from exercising their right to vote.

The act banned literacy tests for voting, required federal oversight over voter registration in areas where less than 50 percent of the non-white population had registered to vote, and authorized the U.S. attorney general to investigate the use of poll taxes in state and local elections. (In 1964, the 24th Amendment made poll taxes illegal in federal elections; poll taxes in state elections were banned in 1966 by the U.S. Supreme Court.)

I’m fairly certain (~95%) — regardless of the party in power — we’ll see the current Supreme Court overturn the state poll tax decision within the next 5-10 years (thankfully, constitutional amendments are a bit harder to wipe way). You can definitely say goodbye to the voting rights act if Republicans gain control of Congress and especially if they regain Congress and the POTUS seat.

Both of those events will make it possible for ~21 states to become even more evil than they already are, and set the stage for a few, awful decades.

I know gas is expensive.

I know food is more expensive than ever.

I know some shelves are bare.

I know lines are longer.

I know we’re still in a pandemic.

I also know that I don’t want to see the devolution of America back to when it was so “great” that we cheered on police when they were treating innocent, peaceful citizens like armed, enemy combatants, and ensured the reign of evil men by making it impossible for large swaths of Americans to vote.

In the fall of 2022 and fall of 2024, we’ll know if we, as a nation, care more about convenience than conscience.

Which side of Juneteenth will you be on when the time comes to choose the path forward?

Using Leonardo SVG Palettes in R

In today’s newsletter Leonardo, an open source project and free online too from Adobe that lets you make great and accessible color palettes for use in UX/UI design and data visualizations! You can read the one newsletter section to get a feel for Leonardo, then go play with it a bit.

The app lets you download the palettes in many forms, as well as just copy the values from the site. Two of the formats are SVG: one for discrete mappings (so, a small, finite number of colors) and another for continuous mappings (so, a gradient). I’ll eventually add the following to my {swatches} package, but, for now, you can tuck these away into a snippet if you do end up working with Leonardo on-the-regular.

Read a qualitative leonardo SVG palette

This is a pretty straightforward format to read and transform into something usable in R:

<svg xmlns="http://www.w3.org/2000/svg" version="1.1" width="616px" height="80px" aria-hidden="true" id="svg">
    <rect x="0" y="0" width="80" height="80" rx="8" fill="#580000"></rect>
    <rect x="88" y="0" width="80" height="80" rx="8" fill="#a54d15"></rect>
    <rect x="176" y="0" width="80" height="80" rx="8" fill="#edc58d"></rect>
    <rect x="264" y="0" width="80" height="80" rx="8" fill="#ffffe0"></rect>
    <rect x="352" y="0" width="80" height="80" rx="8" fill="#b9d6c7"></rect>
    <rect x="440" y="0" width="80" height="80" rx="8" fill="#297878"></rect>
    <rect x="528" y="0" width="80" height="80" rx="8" fill="#003233"></rect>
</svg>

which means {xml2} can make quick work of it:

read_svg_palette <- \(path) {
  xml2::read_xml(path) |> 
    xml2::xml_find_all(".//d1:rect") |> 
    xml2::xml_attr("fill")
}

pal <- read_svg_palette("https://rud.is/dl/diverging.svg")

scales::show_col(pal)

Read a gradient leonardo SVG palette

The continuous one is only slightly more complex:

<svg xmlns="http://www.w3.org/2000/svg" version="1.1" width="800px" height="80px" aria-hidden="true" id="gradientSvg">
    <rect id="gradientRect" width="800" height="80" fill="url(#gradientLinearGrad)" rx="8"></rect>
    <defs id="gradientDefs">
        <linearGradient id="gradientLinearGrad" x1="0" y1="0" x2="800" y2="0" gradientUnits="userSpaceOnUse">
            <stop offset="0" stop-color="rgb(88, 0, 0)"></stop>
            <stop offset="0.04081632653061224" stop-color="rgb(123, 37, 6)"></stop>
            <stop offset="0.08163265306122448" stop-color="rgb(153, 65, 16)"></stop>
            <stop offset="0.12244897959183673" stop-color="rgb(179, 90, 25)"></stop>
            <stop offset="0.16326530612244897" stop-color="rgb(203, 115, 34)"></stop>
            <stop offset="0.20408163265306123" stop-color="rgb(222, 139, 51)"></stop>
            <stop offset="0.24489795918367346" stop-color="rgb(230, 166, 94)"></stop>
            <stop offset="0.2857142857142857" stop-color="rgb(236, 190, 130)"></stop>
            <stop offset="0.32653061224489793" stop-color="rgb(240, 210, 160)"></stop>
            <stop offset="0.3673469387755102" stop-color="rgb(245, 227, 184)"></stop>
            <stop offset="0.40816326530612246" stop-color="rgb(249, 241, 204)"></stop>
            <stop offset="0.4489795918367347" stop-color="rgb(252, 250, 217)"></stop>
            <stop offset="0.4897959183673469" stop-color="rgb(254, 254, 222)"></stop>
            <stop offset="0.5306122448979592" stop-color="rgb(251, 252, 222)"></stop>
            <stop offset="0.5714285714285714" stop-color="rgb(242, 248, 220)"></stop>
            <stop offset="0.6122448979591837" stop-color="rgb(229, 240, 216)"></stop>
            <stop offset="0.6530612244897959" stop-color="rgb(210, 229, 209)"></stop>
            <stop offset="0.6938775510204082" stop-color="rgb(188, 216, 201)"></stop>
            <stop offset="0.7346938775510204" stop-color="rgb(160, 202, 189)"></stop>
            <stop offset="0.7755102040816326" stop-color="rgb(126, 186, 178)"></stop>
            <stop offset="0.8163265306122449" stop-color="rgb(74, 170, 167)"></stop>
            <stop offset="0.8571428571428571" stop-color="rgb(53, 147, 146)"></stop>
            <stop offset="0.8979591836734694" stop-color="rgb(42, 122, 121)"></stop>
            <stop offset="0.9387755102040817" stop-color="rgb(28, 94, 95)"></stop>
            <stop offset="0.9795918367346939" stop-color="rgb(9, 65, 66)"></stop>
        </linearGradient>
    </defs>
</svg>

Which means we have to do a tad bit more work in R:

read_svg_gradient <- \(path) {

  xml2::read_xml(path) |> 
    xml2::xml_find_all(".//d1:stop") -> stops

  stringi::stri_replace_last_fixed(
    str = xml2::xml_attr(stops, "stop-color"),
    pattern = ")",
    replacement = ", alpha = 255, maxColorValue = 255)"
  ) -> rgbs

  list(
    colours = lapply(rgbs, \(rgb) parse(text = rgb)) |> 
      sapply(eval) |> 
      stringi::stri_replace_last_regex("FF$", ""),
    values = as.numeric(xml2::xml_attr(stops, "offset"))
  )

}

svg_grad <- read_svg_gradient("https://rud.is/dl/diverging-gradient.svg")

scales::show_col(svg_grad$colours)

We can use the continuous palette with ggplot2::scale_color_gradientn():

df <- data.frame(
  x = runif(100),
  y = runif(100),
  z1 = rnorm(100),
  z2 = abs(rnorm(100))
)

ggplot2::ggplot(df, ggplot2::aes(x, y)) +
  ggplot2::geom_point(ggplot2::aes(colour = z1)) +
  ggplot2::scale_color_gradientn(
    colours = svg_grad$colours,
    values = svg_grad$values
  ) +
  hrbrthemes::theme_ft_rc(grid="XY")

FIN

Short post, but hopefully a few folks are inspired to try Leonardo out.

Escaping Groundhog Day

Before digging into this post, I need to set some context.

Friday, May 13, 2022 was my last day at my, now, former employer of nearly seven years. I’m not mentioning the company name1 because this post is not about them2.

This post is about burnout and the importance of continuous monitoring and maintenance of you.

Occasionally, I mention3 that I’m one of those Peloton cult members. Each instructor has a pull-list of inspirational quotes that they interject in sessions4, and I’ve worked pretty hard across many decades curating mental firewall rules for such things, as words can have real power and should not be consumed lightly.

Like any firewall, some unintended packets get through, and one of Jess King’s mantras kept coming back to me recently as I was post-processing my decision to quit.

My biggest fear is waking up tomorrow and repeating today.

Many events ensued, both over the years and very recently, prior to giving notice, which was three weeks before my last day. Anyone who has built a fire by hand, by which I mean use a technique such as a bow drill vs strike a match, knows that it can take a while for the pile of kindling to finally go from docile carbon to roaring flame. For those more inclined to books5 than bivouacs, it’s also a bit like bankruptcy:

“How did you go bankrupt?” Bill asked.
“Two ways,” Mike said. “Gradually and then suddenly.”

That’s how I’d describe finally making the decision.

Personal Observability Failures

Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.6 I’m using that term as many folks reading this will have come from similar technical backgrounds and it has been my (heh) observation that technically inclined folks seem to have a harder time with emotional language than they do with technical language. I certainly do.

The day after officially giving notice, I went — as usual — to the DatCave to begin the day’s work after getting #4 and $SPOUSE ready for school(s). After about an hour, I looked down and noticed I wasn’t using my wrist braces.

I should probably describe why that was a Big Deal™.

For the past ~2.5 years I’ve had to wear wrist braces when doing any keyboard typing at all. I’ve had a specific RSI7 condition since high school that has, on occasion, required surgery to correct. Until this flare-up started, I had not needed any braces, or had any RSI pain, for ages8.

But, ~2.5 years ago I started to have severe pain when typing to the point where, even with braces, there were days I really couldn’t type at all. Even with braces, this bout of RSI also impacted finger coordination to the extent that I had to reconfigure text editors to not do what they usually would for certain key combinations, and craft scripts to fix some of the more common errors said lack of coordination caused. I could tell surgery could have helped this flare-up, but there’s no way I was going for elective surgery during a pandemic.

Seeing full-speed, error-free, painless typing sans-braces was a pretty emotional event. It was shortly thereafter when I realized that I had pretty much stopped reading my logs (what normal folks would might say as “checking in with myself”) ~3 years ago.

Fans of observability know that a failing complex system may continue to regularly send critical event logs, but if nothing is reading and taking action9 on those logs, then the system will just continue to degrade or fail completely over time, often in unpredictable ways.

After a bit more reflection, I realized that, at some point, I became Bill Murray10, waking up each day and just repeating the last day, at least when it came to work. I think I can safely say Jess’ (and Phil11‘s) biggest fear is now at least in my own top five.

Burnout, general stress, the Trump years, the rise of Christian nationalism, the pandemic, and the work situation all contributed to this personal, Academy Award-winning performance of Groundhog Day and I’m hoping a small peek into what I saw and what I’m doing now will help at least one other person out there.

Personal Failure Mode Effects And Mitigations

There’s a process in manufacturing called “failure mode and effects analysis”12 that can be applied to any complex system, including one’s self. It’s the structured act of reviewing as many components, assemblies, and subsystems as possible to identify potential failure modes in a complex system and their causes and effects.

Normal folks would likely just call this “self-regulation, recovery, and stress management”13,14.

My human complex system was literally injuring itself (my particular RSI is caused by ganglia sac growth; the one in my left wrist is now gone and the right wrist is reducing, both without medical intervention, ever since quitting), but rather than examine the causes, I just attributed it to “getting old”, and kept on doing the same thing every day.

I’ll have some more time for self-reflection during this week of funemployment, but I’ve been assessing the failure modes, reading new recovery and management resources, and wanted to share a bit of what I learned.

Some new resources linked-to in the footnotes, and found in annotated excerpts below, that I have found helpful in understanding and designing corrective systems for my personal failure modes are from Cornell.

  • Don’t be afraid of change: For someone who is always looking to the future and who groks “risk management”, I’m likely one of the most fundamentally risk-averse folks you’ve encountered.

    I let myself get stuck in a pretty unhealthy situation mostly due to fear of change and being surface-level comfortable. If I may show my red cult colors once again, “allow yourself the opportunity to get uncomfortable” should apply equally to work as it does to watts.

    Please do not let risk aversion and surface-level comfort keep you in a bad situation. My next adventure is bolder than any previous one, and is, in truth, a bit daunting. It is far from comfortable, and that’s O.K.

  • Take care of your physical needs: Getting a good night’s rest, eating well, and exercising are all essential to being able to feel satisfaction in life. They’re also three things that have been in scarce supply for many folks during the pandemic.

    I like to measure things, but I finally found the Apple Watch lacking in quantified self utility and dropped some coin on a Whoop band, and it was one of the better investments I’ve made. I started to double-down on working out when I learned I was going to be a pampa15, as I really want to be around to see him grow up and keep up with him. I’ve read a ton about exercise, diet, etc. over the years, but the Whoop (and Peloton + Supernatural coaches) really made me understand the importance of recovery.

    Please make daily time to check in with your mental and physical stress levels and build recovery paths into your daily routines. A good starting point is to regularly ask yourself something like “When I listen to my body, what does it need? A deep breath? Movement? Nourishment? Rest?”

  • Engage in activities that build a sense of achievement: The RSI made it nigh impossible to engage with the R and data science communities, something which I truly love doing, but now realize I was also using as a coping mechanism for the fact that a large chunk of pay-the-bills daily work was offering almost no sense of achievement16. I’m slowly getting back into engaging with the communities again, and I know for a fact that the it will be 100% on me if I do not have a daily sense of achievement at the new pay-the-bills daily workplace.

    It’d be easy for me to say “please be in a job that gives you this sense of daily achievement”, but, that would be showing my privilege. As long as you can find something outside of an achievement-challenged job to give you that sense of achievement (without falling into the similar trap I did) then that may be sufficient. The next bullet may also help for both kinds of work situations.

    You can also be less hard on yourself outside of work/communities and let yourself feel achieved for working out, taking a walk, or even just doing other things from the first bullet.

  • Changing thoughts is easier than changing feelings: Thoughts play a critical role in how we experience a situation. When you notice yourself first becoming frustrated or upset, try to evaluate what you are thinking that is causing that emotion.

    This is also known as cognitive re-framing/restructuring17. That footnote goes to a paper series, but a less-heady read is Framers, which is fundamentally about the power of mental models to make better decisions. I’d note that you cannot just “stop caring” to dig yourself out of a bad situation. You will just continue to harm yourself.

    Note that this last bullet can be super-hard for those of us who have a strong sense of “justice”, but hang in there and don’t stop working on re-framing.

FIN

I let myself get into a situation that I never should have.

Hindsight tells me that I should have made significant changes about four years ago, and I hope I can remember this lesson moving forward since there are fewer opportunities for “four year mistakes” ahead of me than there are behind me.

Burnout — which is an underlying component of above — takes years to recover from. Not minutes. Not hours. Not days. Not weeks. Not months. Years.

I’m slowly back to trying to catch up to mikefc when it comes to crazy R packages. I have more mental space available than I did a few years ago, and I’m healthier and more fit than I have been in a long time. I am nowhere near recovered, though.

If you, too, lapsed when it comes to checking in with yourself, there’s no time like the present to restart that practice. The resources I posted here may not work for you, but there are plenty of good ones out there.

If you’ve been doing a good job on self-care, make sure to reach out to others you may sense aren’t in the same place you are. You could be a catalyst for great change.

  1. I mean, you do have LinkedIn for discovering things like that ↩
  2. Though you’d be hard-pressed to not think some folks there only listen to Carly Simon ↩
  3. Usually on Twitter (b/c ofc) ↩
  4. Which is part of what makes it a bona fide cult ↩
  5. The Sun Also Rises ↩
  6. Observability ↩
  7. RSI ↩
  8. RSI wasn’t the only negative physical manifestation, but listing out all the things that manifest and got better isn’t truly necessary. ↩
  9. Software systems observability ↩
  10. Groundhog Day ↩
  11. I’m a billion years old, have seen Groundhog Day far more than a few times, and just got the joke (as I was writing this post) that Murray’s character was named “Phil”. ↩
  12. FMEA ↩
  13. Cornell: Emotional regulation [PDF] ↩
  14. Cornell: Stress management strategies [PDF] ↩
  15. Belter creole for granddad (et al) ↩
  16. I feel compelled to note that I was able to perform many, many work activities over the course of nearly seven years that brought a great sense of achievement. For a host of reasons, they went from a stream to a trickle. ↩
  17. Cognitive Restructuring ↩

Rusty HTML Minifying In R with {rextendr}

In my M-F newsletter today I mentioned an awesome Rust-based HTML/JS/CSS minifier library that also include batteries for a few other languages.

There was no R port, so I made one using {rextendr}. The {rextendr} package makes is as easy to use Rust code in R packages as {Rcpp} does C/C++ code.

It was as simple as adding some dependencies to the Rust Cargo.toml file and then adding one Rust function to the main lib.rs file, and writing a thin wrapper function ({rextendr} can do that, too, but I wanted some default function parameters) for the shim. It took almost no time, and now you, too, can use the utility:

library(minifyhtml)

'
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta charset="UTF-8"/>
  <meta name="viewport" content="width=device-width, initial-scale=1"/>
  <!-- COMMENT -->
  <style>
    * { color: black; }
  </style>
  <title>TITTLE</title>
  </head>
  <body>
    <p>
       Some text
    </p>
    <script>
      console.log("This is a console log message.");
    </script>
  </body>
</html>
' -> src

cat(minify(src))
## <html xmlns=http://www.w3.org/1999/xhtml><meta charset=UTF-8><meta content=width=device-width,initial-scale=1 name=viewport><style>* { color: black; }</style><title>TITTLE</title><body><p>Some text</p><script>console.log("This is a console log message.");</script>

FIN

I have to work out one kink (due to developing on arm64 macOS) and the utility will also be able to minify CSS and JS embedded in HTML.

You can find {minifyhtml} on GitHub.

Announcement: New Substack Newsletter

Just a quick post to note that I’ll still be doing long-form, single-topic posts here on the blog, but I’ve also started an M-F daily Substack newsletter [free], that introduces 1-3(ish) topics per-issue on stuff I find during my daily RSS trawls.

It’s a mix of tech, Mac, linux, data science, science-science, food, and some odds-and-ends that folks seem to find interesting so far.

There are a few issues up in the archive, so you can get a feel for what it’s all aboot.

I may drop some podcasts into there as well.

Keeping Those SSH Keys Safe

I came across a neat site that uses a Golang wasm function called from javascript on the page to help you see if your GitHub public SSH keys are “safe”. What does “safe” mean? This is what the function checks for (via that site):

Recommended key sizes are as follows:

  • For the RSA algorithm at least 2048, recommended 4096
  • The DSA algorithm should not be used
  • For the ECDSA algorithm, it should be 521
  • For the ED25519, the key size should be 256 or larger

The site also provides links to standards and guides to support the need for stronger keys.

I threw together a small R package — {pubcheck} — to check local keys, keys in a character vector, and keys residing in GitHub. One function will even check the GitHub keys of all the GitHub users a given account is following:

Local file

library(pubcheck)
library(tidyverse)

check_ssh_pub_key("~/.ssh/id_rsa.pub") |> 
  mutate(key = ifelse(is.na(key), NA_character_, sprintf("%s…", substr(key, 1, 30)))) |> 
  knitr::kable()
key algo len status
ssh-rsa AAAAB3NzaC1yc2EAAAADAQ… rsa 2048 ✅ Key is safe; For the RSA algorithm at least 2048, recommended 4096

A GitHub user

check_gh_user_keys(c("hrbrmstr", "mikemahoney218")) |> 
  mutate(key = ifelse(is.na(key), NA_character_, sprintf("%s…", substr(key, 1, 30)))) |> 
  knitr::kable()
user key algo len status
hrbrmstr ssh-rsa AAAAB3NzaC1yc2EAAAADAQ… rsa 2048 ✅ Key is safe; For the RSA algorithm at least 2048, recommended 4096
hrbrmstr ssh-rsa AAAAB3NzaC1yc2EAAAADAQ… rsa 2048 ✅ Key is safe; For the RSA algorithm at least 2048, recommended 4096
hrbrmstr ssh-rsa AAAAB3NzaC1yc2EAAAADAQ… rsa 2048 ✅ Key is safe; For the RSA algorithm at least 2048, recommended 4096
mikemahoney218 ssh-rsa AAAAB3NzaC1yc2EAAAADAQ… rsa 4096 ✅ Key is safe
mikemahoney218 ssh-ed25519 AAAAC3NzaC1lZDI1NT… ed25519 256 ✅ Key is safe
mikemahoney218 ssh-ed25519 AAAAC3NzaC1lZDI1NT… ed25519 256 ✅ Key is safe
mikemahoney218 ssh-ed25519 AAAAC3NzaC1lZDI1NT… ed25519 256 ✅ Key is safe

Keys of all the users a GitHub account is following

check_gh_following("koenrh") |> 
  mutate(key = ifelse(is.na(key), NA_character_, sprintf("%s…", substr(key, 1, 30)))) |> 
  knitr::kable()
user key algo len status
framer NA NA NA NA
jurre ssh-rsa AAAAB3NzaC1yc2EAAAADAQ… rsa 2048 ✅ Key is safe; For the RSA algorithm at least 2048, recommended 4096

What’s it like out there?

I processed my followers list and had some interesting results:

library(pubcheck)
library(hrbragg)
library(tidyverse)

# this takes a while as the # of users is > 500!
res <- check_gh_following("hrbrmstr")

res |> 
  count(user) |> 
  arrange(n) |> 
  count(n, name = "n_users") |> 
  mutate(csum = cumsum(n_users)) |> 
  ggplot() +
  geom_line(
    aes(n, csum)
  ) +
  geom_point(
    aes(n, csum)
  ) + 
  scale_x_continuous(breaks = 1:21) +
  scale_y_comma() +
  labs(
    x = "# Keys In User GH", y = "# Users",
    title = "Cumulative Plot Of User/Key Counts [n=522 users]",
    subtitle = "A handful of users have >=10 keys configured in GitHub; one has 21!!"
  ) +
  theme_cs(grid="XY")


res |> 
  count(algo, len, status) |> 
  mutate(kind = ifelse(is.na(status), "No SSH keys in account", sprintf("%s:%s\n%s", algo, len, status))) |> 
  mutate(kind = fct_reorder(gsub("[;,]", "\n", kind), n, identity)) |> 
  ggplot() +
  geom_col(
    aes(n, kind),
    width = 0.65, 
    fill = "steelblue", 
    color = NA
  ) +
  scale_x_comma(position = "top") +
  labs(
    x = NULL, y = NULL,
    title = "SSH Key Summary For GH Users hrbrmstr Is Following"
  ) +
  theme_cs(grid="X") +
  theme(plot.title.position = "plot")

FIN

Whether you use the website or the R package, it’d be a good idea to check on your SSH keys at least annually.

Update On ESC/POS Package Work

The morning before work was super productive and there is a nigh-complete DSL for ESC/POS commands along with the ability to just print {ggplot2}/{grid} object.

I changed the package name to {escpos} since it is no longer just plot object focused, and the DSL looks a bit like this:

library(stringi)
library(hrbrthemes)
library(ggplot2)
library(escpos)

ggplot() +
  geom_point(
    data = mtcars,
    aes(wt, mpg),
    color = "red"
  ) +
  labs(
    title = "A good title"
  ) +
  theme_ipsum_es(grid="XY") -> gg

epson_ip = "HOSTNAME_OR_IP_OF_YOUR_PRINTER"

escpos(epson_ip) |>
  pos_bold("on") %>%
  pos_align("center") %>%
  pos_size("2x") %>%
  pos_underline("2dot") %>%
  pos_plaintext("This Is A Title") %>%
  pos_lf(2) |>
  pos_underline("off") %>%
  pos_size("normal") %>%
  pos_align("left") %>%
  pos_bold("off") %>%
  pos_font("b") %>%
  pos_plaintext(
    stringi::stri_rand_lipsum(1)
  ) |>
  pos_lf(2) |>
  pos_font("a") %>%
  pos_plaintext(
    paste0(capture.output(
      str(mtcars, width = 40, strict.width = "cut")
    ), collapse = "\n")
  ) |>
  pos_lf(2L) |>
  pos_plot(gg, color = TRUE) %>%
  pos_lf(2L) |>
  pos_font("c") %>%
  pos_plaintext(
    stringi::stri_rand_lipsum(1, start_lipsum = FALSE)
  ) |>
  pos_lf(3) |>
  pos_cut() %>%
  pos_print()

full capabilities ESC/POS printing

FIN

I still need to make a more generic options “setter” (i.e. so one can set multiple modes in one function call), and I think supporting some kind of markdown/HTML subset to make it easier just to specify that without using the full DSL would be helpful. More updates over the coming weeks!