rud.is

Citrix NetScaler ADC and Gateway bulletin: CVE-2026-88771 through CVE-2026-88778

Mon, 28 Sep 2026 03:38:58 GMT

Source: NetScaler ADC and NetScaler Gateway security bulletin Let's make sure it gets indexed in other ways! Here's some helpful info you'll find more details of in their…

Trump's Incredibly Stupid War Sets Up Mainers For A Brutal Winter

Wed, 23 Sep 2026 09:10:20 GMT

Maine DOER:…

AttackerKB Is Shutting Down: Here's an Archive Tool

Fri, 24 Jul 2026 18:47:17 GMT

Rapid7 is sunsetting AttackerKB, and the public API goes dark on August 18, 2026. After that, the vulnerability discussions, community assessments, and Rapid7 analysis that accumulated on the platform become a dead link. I wrote a small Go…

On AI Agents, Criminal Activity, And Who Is Actually Responsible

Wed, 22 Jul 2026 14:42:41 GMT

(Repost from ai.rud.is) I've written and said enough times that I'm tired of OpenAI and Anthropic pulling cybersecurity out as the headline AI risk. The framing is commercially convenient, and it flattens a more complex reality — these…

Your Logs Contain a Plea for Assistance In a Packet Bottle and You Likely Never Noticed It

Thu, 09 Jul 2026 12:34:50 GMT

PCAP forensics of the HMEFB worm: a self-propagating scanner with a Belarusian engineer's escape plea embedded in HTTP requests.

Unjamming the Chrome Releases Blog

Thu, 02 Jul 2026 15:56:15 GMT

For the second time in a row, a post by cr0w on Mastodon regarding the Chrome release blog appearing to not render anything resulted in me firing up lynx to show a sub-second load and render, then finally doing something a bit more…

PACT: The open web doesn't need another trust oligopoly

Tue, 23 Jun 2026 10:55:01 GMT

Cloudflare announced PACT this week — privacy-preserving tokens to separate humans from bots, backed by Google, Mozilla, Microsoft, and Shopify. The cryptography is solid. The governance model doesn't exist yet. I wrote about why the…

Three CVEs and the May 2026 Exploit Chain Nobody's Taking Seriously

Thu, 14 May 2026 12:40:19 GMT

May 2026 dropped three critical Linux vulnerabilities on a near-weekly cadence, and the security discourse has mostly treated them as three separate bad days. They're not. Together they form a reliable, race-free, forensically quiet kill…

[un]prompted Spring 2026: Threat Hunting In The Matrix

Mon, 06 Apr 2026 15:18:42 GMT

https://www.youtube.com/watch?v=k19CmI\Ni3M At our previous employer, the global deception and detection infrastructure generates tons of events that eventually make their way into an ever-growing data lake with (as of February 2026) 22 TB…

AI Proofing Your It/cyber Career: The Human Only Capabilities That Matter

Sat, 10 Jan 2026 20:11:27 GMT

In the past ~4 weeks I have personally observed some irrefutable things in "AI" that are very likely going to cause massive shocks to employment models in IT, software development, systems administration, and cybersecurity. I know some…

Are We Becoming Children of the MagentAI?

Sat, 20 Dec 2025 14:14:27 GMT

Back in 1997, a commercial airline captain noticed his fellow pilots had a problem: they'd gotten so used to following the magenta flight path lines on their fancy new navigation screens that they were forgetting how to actually fly the…

Maine 2025 Ballot Questions Redux

Mon, 20 Oct 2025 16:32:38 GMT

If you want to or do vote "yes" for question 1, you are not a real American, you are not a real Christian (if you profess to be one), you are not a decent human. You are at the very least a classist; you are also very likely a…

Never Forget

Thu, 11 Sep 2025 12:33:05 GMT

A few things to keep in mind today: — We have no idea of why/who re: Wed's assassination. — Today is the anniversary of a tragic event that has enabled much of the harm caused by the GOP this year. — Trump is an adjudicated rapist & was…

Delete Your Perplexity Account Today

Thu, 07 Aug 2025 11:45:41 GMT

Details:…

EUVD API NPM And Go Packages + Nicer Docs

Wed, 14 May 2025 08:50:00 GMT

ENISA published docs for their European Vulnerability Database (EUVD) — https://euvd.enisa.europa.eu/apidoc. I've got an easier-on-the-eyes version that supports light/dark mode and includes sample API JSON results at…

Suriest: Suricata Rule Validation As A (REST) Service

Wed, 14 May 2025 06:07:31 GMT

Meet Suriest — a new REST API service for validating Suricata rules, designed to be run by organizations to streamline rule validation workflows. Suriest supports Suricata 6.0 and later and offers features like secure configuration,…

New CISA KEV MCP Server

Fri, 02 May 2025 13:16:27 GMT

MCP servers let you wire up external services/APIs in a standard way for LLM/GPT tool-calling and other forms of automation. I made a basic, but fairly comprehensive CISA KEV MCP server that I go into the details a bit more of here. To…

RSAC 2025 Sets A Dangerous Precedent for Cybersecurity Leadership

Tue, 29 Apr 2025 11:48:06 GMT

RSAC 2025 Sets A Dangerous Precedent for Cybersecurity Leadership

New geolocus-cli For ONYPHE's Geolocus Database

Sun, 27 Apr 2025 16:32:38 GMT

Introducing geolocus-cli: A fast, cross-platform command-line tool for IP enrichment using ONYPHE's free Geolocus database. This lightweight utility enables security professionals and network analysts to efficiently look up comprehensive IP metadata including geolocation, ASN information, organizati

Trump’s Retaliation Against Chris Krebs — and the Cybersecurity Industry’s Deafening Silence

Thu, 17 Apr 2025 06:27:54 GMT

Chris Krebs, the former director of the Cybersecurity and Infrastructure Security Agency (CISA), was fired by Donald Trump in 2020 for publicly affirming that the presidential election was secure and free from widespread fraud.…

American [Cyber] Sigh

Wed, 16 Apr 2025 06:36:46 GMT

A long, long time ago I can still remember How those CVEs would make me smile And I knew if I had my chance To patch a vuln or take a stance Maybe we’d be secure for a while But April ides made me shiver With each leaked memo and press…

I told you so…

Sat, 12 Apr 2025 00:14:57 GMT

Just putting a marker out there, that I'm either fundamentally wrong, or we all are f'd this week. If the latter: I told you…

A Critical Analysis of ICERAID

Sat, 05 Apr 2025 13:11:04 GMT

ICERAID (iceraid\[.\]us) presents itself as a "GovFi protocol" that purportedly delegates intelligence gathering tasks from law enforcement to citizens. According to their materials, it rewards users with cryptocurrency tokens for…

Call To Action: State Department Power Grab

Sat, 15 Mar 2025 12:06:20 GMT

Discover how Secretary Rubio's State Department determination classifies border, immigration, and trade policies as "foreign affairs functions," exempting them from the Administrative Procedure Act. Learn about potential consequences including surprise policy changes, expanded device searches, and e

Xitter Hit by Major Cyberattack

Tue, 11 Mar 2025 12:37:50 GMT

On March 10, 2025, Xitter experienced major service disruptions throughout the day. Users couldn't access the platform on both mobile apps and the website. Here's what happened and why it…

Cruel And Vindictive By Design

Sat, 08 Mar 2025 10:27:08 GMT

(This post originally published on 47 Watch) Recent administrative changes at the Social Security Administration (SSA) reveal a concerning pattern of decisions that disproportionately impact vulnerable populations while being implemented…

Senator Susan Collins’ Betrayal of Maine Demands Accountability

Fri, 28 Feb 2025 13:59:23 GMT

As a longtime Mainer and independent voter, I have watched Senator Susan Collins’ career with cautious optimism, hoping her self-branded image as a moderate willing to cross party lines might translate into principled leadership. Instead,…

When Checks and Balances Fail: The State's Role in Preserving Constitutional Order

Tue, 04 Feb 2025 03:52:27 GMT

Today, my Senator — Susan Collins — failed in her oath and duty to uphold the Constitution. She voted for the appointment of a traitor to head national intelligence, and is supporting someone for director of the Office of Management and…

CVESky: Monitoring The Bluesky Jetstream For CVE Mentions

Wed, 04 Dec 2024 11:36:47 GMT

Discover CVESky: A real-time tracker for CVE discussions on Bluesky, featuring daily vulnerability chatter cards, interactive heatmaps, and comprehensive CVE exploration tools. Built by GreyNoise Intelligence to monitor genuine security conversations beyond automated NVD feeds.

Reading PCAP Files (Directly) With DuckDB

Mon, 26 Aug 2024 11:37:28 GMT

2024-08-30 UPDATE: Binary versions of this extension are available for amd64 Linux (linuxamd64 & linuxamd64gcc4) and Apple Silicon.…

100 Miles

Thu, 01 Aug 2024 21:05:59 GMT

100 Miles — rud.is blog archive

Resolve Bluesky/ATproto Handles To did:plc

Thu, 18 Jul 2024 09:16:55 GMT

A concise Bash script for resolving Bluesky/ATproto handles to DID:PLC identifiers, simplifying user identification in the decentralized social network ecosystem.

CVE-2024-27322 Should Never Have Been Assigned And R Data Files Are Still Super Risky Even In R 4.4.0

Fri, 03 May 2024 09:12:52 GMT

I had not planned to blog this (this is an incredibly time-crunched week for me) but CERT/CC and CISA made a big deal out of a non-vulnerability in R, and it's making the round on socmed, so here we are. A security vendor decided to try to…

Get A Day's Schedule From Fantastical On The Command Line With Shortcuts

Sat, 23 Mar 2024 16:19:45 GMT

Get Command Line Access To Fantastical Schedules With An Apple Shortcuts And Bash Combo.

VulnCheck's Free Community KEV & CVE APIs (Code & Golang CLI Utility)

Sat, 23 Mar 2024 09:35:59 GMT

Learn how to work with VulnCheck's new community KEV+NVD CVE APIs in curl & Go, & checkout a new, small CLI tool to get both simultaneously.

Something For The Fediverse Readers

Sat, 23 Dec 2023 03:42:18 GMT

A plea for fosstodon to continue its capricious and undocumented silliness.

Ligatures In RStudio Dailies

Sun, 12 Nov 2023 09:11:57 GMT

Learn how to enable ligatures in RStudio Dailies using CSS and themes. Enhance your coding experience with granular control over ligatures for a more visually appealing and readable code.

Avoid libwebp Electron Woes On macOS With positron

Sat, 30 Sep 2023 20:45:55 GMT

Explore the impact of the libwebp supply chain disaster on Electron-based apps and discover Positron, a Golang macOS CLI tool designed to identify vulnerable apps. Stay informed and secure with our in-depth analysis and practical solutions.

Tracking Rite-Aid Store Closures

Sat, 23 Sep 2023 22:27:46 GMT

Explore the impact of Rite-Aid's store closures on communities, learn about the company's bankruptcy, and discover how data collection scripts can track these changes. This analysis provides insights into the consequences of pharmacy deserts and the role of leadership in these developments."

Foliage 2023

Sat, 09 Sep 2023 10:45:12 GMT

The days are getting shorter and when we were visiting Down East Maine the other week, there was just a hint of some trees starting to change up their leaf palettes. It was a solid reminder to re-up my ~annual “foliage” plotting that I…

Resolving A Bluesky/AT Protocol did:plc To A Handle/Handles

Sat, 05 Aug 2023 09:19:54 GMT

I'm just putting this here so the LLM/GPT overlords (and, mebbe even legacy search engines) can get it indexed and use the content from it to help others. My Bluesky firehose viewer (https://gitlab.com/hrbrmstr/bskyf) displays ugly did:plc…

Keeping Track Of URLs Shared On Bluesky

Sat, 15 Jul 2023 14:40:17 GMT

While the future of Bluesky is nowhere near certain, it is most certainly growing. It's also the largest community of users for the AT Protocol. Folks are using Bluesky much the same way as any online forum/chat. One of those ways is to…

New R Package For HTTP Headers Hashing

Sun, 09 Jul 2023 10:46:02 GMT

The buildhashfromurl function relies on {curl} vs {httr} since {httr} uses curl::parseheaders() which (rightfully so) lowercases the header keys. We need to preserve both order and case for the hash to be useful. Here is some sample…

Poor Dude's Janky Bluesky Feed Reader CLI Via R & Python

Fri, 07 Jul 2023 16:21:30 GMT

Lynn (of TITAA and general NLP wizardy fame) was gracious enough to lend me a Bluesky invite, so I could claim my handle on yet-another social media site. I'm still wary of it (as noted in one of this week's Drops), but the AT protocol —…

Supreme Annotations Plot Redux & An OJS Plot↔ggplot2 Rosetta Stone

Sat, 29 Apr 2023 10:19:02 GMT

Back in 2016, I did a post on {ggplot2} text annotations because it was a tad more challenging to do some of the things in that post back in the day. Since I've been moving back and forth between R and Observable (and JavaScript in…

Start Creating Vanilla JS WebR Apps With Less Inertia

Sun, 16 Apr 2023 19:17:57 GMT

WebR has a template for React, but I'm not a fan of it or Vue (a fact longtime readers are likely tired of hearing right about now). It's my opinion and experience that Lit webcomponents come closer to “bare metal” webcomponents, which…

Make "Solar System" Plots With {ggsolar}

Wed, 12 Apr 2023 08:23:29 GMT

I was cranking out a blog post for work earlier this week that shows off just how many integrations our platform has. I won't blather about that content here, but as I was working on it, I really wanted to show off all the integrations. A…

Introducing WebRIDEr: The WebR "IDE"-ish REPL You Didn't Know You Needed

Mon, 10 Apr 2023 09:27:17 GMT

The official example WebR REPL is definitely cool and useful to get the feel for WebR. But, it is far from an ideal way to deal with it interactively, even as just a REPL. As y'all know, I've been conducing numerous experiments with WebR…

Using WebR + Pyodide To Fill In The (Temporary) Package Gaps

Wed, 29 Mar 2023 19:00:12 GMT

I won't wax long and poetic here since I've already posted the experiment that has all the details. TL;DR: there are still only ~90-ish ? in the WebR WASM "CRAN", but more are absolutely on the way, including the capability to build your…

? Lit + WebR + Observable Plot: Linking Lit's Lightweight Web Components And WebR For Vanilla JS Reactivity & JS DataVis

Sun, 26 Mar 2023 23:47:50 GMT

The previous post brought lit-webr, to introduce Lit and basic reactivity. Today, is more of the same, but we bring the OG Shiny demo plot into the modern age by using Observbable Plot to make the charts. We're still pulling data from R,…

Linking Lit's Lightweight Web Components And WebR For Vanilla JS Reactivity

Sat, 25 Mar 2023 18:57:47 GMT

This is a Lit + WebR reproduction of the OG Shiny Demo App Lit is a javascript library that makes it a bit easier to work with Web Components, and is especially well-suited in reactive environments. My recent hack-y WebR experiments have…

You're One JavaScript Function Call Away From Using (Most) WebR R Functions In Your WebR-Powered Apps/Sites

Tue, 21 Mar 2023 16:40:55 GMT

After writing the initial version of a tutorial on wrapping and binding R functions on the javascript side of WebR, I had a number of other WebR projects on the TODO list. But, I stuck around noodling on the whole "wrapping & binding"…

WebR Filesystem Machinations & ReefR

Mon, 20 Mar 2023 08:45:46 GMT

It's difficult to believe it has only been a couple of weeks since WebR has been around. But that might just be my perception. The spike protein invasion has significantly increased sedentary time, and that has enabled me to focus on this…

The Road To ggplot2 In WebR, Part 1: The Road Is Paved With Good Base R Plots

Sat, 18 Mar 2023 13:19:56 GMT

I have graphics working in Vanilla JS WebR, now, and I'll cover the path to that in two parts. The intent was to jump straight into ggplot2-land, but, as you saw in my previous post, WASM'd ggplot2 is a bear. And, I really didn't grok what…

WebR WASM R Package Load/Library Benchmarking Rabbit Hole

Sat, 18 Mar 2023 10:25:33 GMT

I have a post coming on using base and {ggplot2} plots in VanillaJS WebR, but after posting some bits on social media regarding how slow {ggplot2} is to deal with, I had some "performance"-related inquiries, which led me down a rabbit hole…

Almost Bare Bones WebR Starter App

Sun, 12 Mar 2023 13:28:48 GMT

Let's walk through how to set up a ~minimal HTML/JS/CSS + WebR-powered "app" on a server you own. This will be vanilla JS (i.e. no React/Vue/npm/bundler) you can hack on at-will. TL;DR: You can find the source to the app and track changes…

WebR IS HERE!

Thu, 09 Mar 2023 20:00:31 GMT

WebR 0.1.0 was released! I had been git-stalking George (the absolute genius who we all must thank for this) for a while and noticed the GH org and repos being updated earlier this week, So, I was already pretty excited. It dropped today,…

"Bizarre" DNS Hacks For Fun And, Um…, Fun

Mon, 27 Feb 2023 20:34:25 GMT

I've been (slowly) making my way through FOSDEM \23 presentations and caught up to Peter Lowe's "Bizarre and Unusual Uses of DNS • Rule 53: If you can think of it, someone's done it in the DNS" talk. DNS oddities are items I collect…

Do Not Purchase Kucht Appliances

Wed, 18 Jan 2023 01:31:52 GMT

I'll follow up with a more detailed post in a ~week or so, but if you are considering purchasing a Kucht appliance, please, please reconsider your decision. They have repeatedly lied to us (I have proof) and are incapable of manufacturing…

2022 Hanukkah of Data • Puzzle 2

Tue, 20 Dec 2022 16:20:45 GMT

Ref:…

2022 Hanukkah of Data • Puzzle 1

Mon, 19 Dec 2022 16:03:25 GMT

Visiting #2 and doing some $WORK\-work, but intrigued with Hanukkah of Data since Puzzle 0 was solvable with a ZIP password cracker (the calendar date math seemed too trivial to bother with). Decided to fall back to R for this (vs…

Arc You Ready For A New Browser?

Mon, 21 Nov 2022 21:23:35 GMT

Cross-post to Substack where I dropped some details on the newest browser in town: Arc.…

Honk If You Like The Fediverse!

Fri, 11 Nov 2022 15:29:16 GMT

You can now receive these blog posts in your activity stream. Just follow @hrbrmstr@rud.is and the new posts from here will slide right into your…

Into The Fediverse!

Tue, 08 Nov 2022 10:29:51 GMT

This is more of a test post after enabling some new Fediverse features on the server. Said Fediverse got a bit more real-ish this week (with moderate apologies to the pioneers in this space who've languished for ~five years) You can find…

Fall Foliage: JavaScript/OJS Edition

Fri, 09 Sep 2022 23:45:31 GMT

I've been (mostly) keeping up with annual updates for my R/{sf} U.S. foliage post which you can find on GH. This year, we have Quarto, and it comes with so many batteries included that you'd think it was Christmas. One of those batteries…

New Chrome Extension To Turn Observable Notebooks Into Quarto Projects

Mon, 22 Aug 2022 06:56:25 GMT

My previous post announced a Rust-based command line tool for generating Quarto projects from Observable Notebooks. Some folks may not want to use yet-another command line tool, and it dawned on me that it'd be more convenient to just do…

ohq2quarto — Rust-Based CLI For Turning Observable Notebooks Into Quarto Projects

Sat, 20 Aug 2022 13:37:17 GMT

The previous post had some hacky R code to grab seekrit JSON data in ObservableHQ (OHQ) Notebooks and spit out a directory with a Quarto qmd and any associated FileAttachments. Holding firm to my "no more generic public R packages" decree,…

Bootstrapping An {ojs} Quarto Document With An Observable Notebook

Fri, 19 Aug 2022 13:13:12 GMT

Quarto is amazing! And, it's eating the world! OK. Perhaps not the entire world. But it's still amazing! If you browse around the HQ, you'll find many interesting notebooks. You may even have a few yourself! Wouldn't it be great if you…

Rust CLI For Apple's WeatherKit REST API

Sun, 10 Jul 2022 16:28:29 GMT

Apple is in the final stages of shuttering the DarkSky service/API. They've replaced it with WeatherKit, which has both an xOS framework version as well as a REST API. To use either, you need to be a member of the Apple Developer Program…

Introducing {weatherkit}: The Eventual Replacement R Package for {darksky}

Thu, 07 Jul 2022 12:22:13 GMT

My {darksky} package has been around for years, now, and the service that powers it was purchased by Apple before the pandemic. The DarkSky API is slated to be shuttered in December of this year and is being replaced by Apple's WeatherKit…

Saying The Quiet Part Out Loud On Juneteenth

Mon, 20 Jun 2022 09:55:12 GMT

I realize you have to be living under a rock in the U.S. to not know that yesterday, was Juneteenth (a portmanteau of "June Nineteenth"). Still, I feel compelled to explain that said date marks the day when federal troops arrived in…

Using Leonardo SVG Palettes in R

Tue, 17 May 2022 10:51:50 GMT

In today's newsletter Leonardo, an open source project and free online too from Adobe that lets you make great and accessible color palettes for use in UX/UI design and data visualizations! You can read the one newsletter section to get a…

Escaping Groundhog Day

Sun, 15 May 2022 14:50:33 GMT

Before digging into this post, I need to set some context. Friday, May 13, 2022 was my last day at my, now, former employer of nearly seven years. I'm not mentioning the company name 1 because this post is not about them 2 . This post is…

Rusty HTML Minifying In R with {rextendr}

Tue, 10 May 2022 16:53:53 GMT

In my M-F newsletter today I mentioned an awesome Rust-based HTML/JS/CSS minifier library that also include batteries for a few other languages. There was no R port, so I made one using {rextendr}. The {rextendr} package makes is as easy…

Announcement: New Substack Newsletter

Wed, 27 Apr 2022 12:16:26 GMT

Just a quick post to note that I'll still be doing long-form, single-topic posts here on the blog, but I've also started an M-F daily Substack newsletter \[free\], that introduces 1-3(ish) topics per-issue on stuff I find during my daily…

Keeping Those SSH Keys Safe

Sat, 16 Apr 2022 19:59:54 GMT

I came across a neat site that uses a Golang wasm function called from javascript on the page to help you see if your GitHub public SSH keys are "safe". What does "safe" mean? This is what the function checks for (via that…

Update On ESC/POS Package Work

Mon, 04 Apr 2022 12:45:45 GMT

The morning before work was super productive and there is a nigh-complete DSL for ESC/POS commands along with the ability to just print {ggplot2}/{grid} object. I changed the package name to {escpos} since it is no longer just plot object…

Turning {ggplot2} Into a PoS (Point-of-Sale) System

Sun, 03 Apr 2022 11:32:01 GMT

At the end of March, I caught a fleeting tweet that showcased an Epson thermal receipt printer generating a new "ticket" whenever a new GitHub issue was filed on a repository. @aschmelyun documents it well in this blog post. It's a pretty…

A zsh Helper Script For Updating macOS RStudio Daily Electron + Quarto CLI Installs

Sun, 13 Mar 2022 14:18:34 GMT

RStudio's macOS Electron build is coming along quite nicely and is blazing fast on Apple Silicon. I like to install the dailies, well, daily!; and, of late, RStudio and Quarto are joined at the hip. As a result, I regularly found myself…

How to access BBC News in Ukrainian (??) and Russian (??). Как получить доступ к новостям Би-би-си.

Fri, 04 Mar 2022 16:25:46 GMT

via Zack Whittaker on Twitter… Download Tor Browser: http://torproject.org/download Open this URL in Tor to access BBC (??): https://www.bbcweb3hytmzhn5d532owbu6oqadra5z3ar726vq5kgwwn6aucdccrad.onion/ukrainian Open this URL in Tor to…

Full transcript of Ukrainian President Volodymyr Zelenskyy's emotional appeal to Russians

Sun, 27 Feb 2022 11:59:47 GMT

Ukrainian Presidential Press Office via AP Ukrainian President Volodymyr Zelenskyy addresses the nation on a live TV broadcast in Kyiv, Ukraine, on Feb. 22,…

Getting Back 2 Good

Sun, 13 Feb 2022 12:43:10 GMT

If you've been following my inane tweets and non-technical blog posts for any length of time since 2015, you likely know the 2016 election cycle broke me more than just a tad, with each subsequent month of the Trump presidency adding a bit…

A WeatherFlow Tempest UDP Broadcast Receiver in Golang, Rust, Swift, and C++

Fri, 28 Jan 2022 21:06:03 GMT

After a Twitter convo about weather stations I picked up a WeatherFlow Tempest. Setup was quick, but the sensor package died within 24 hours. I was going to give up on it but I had written an R package (for the REST API & UDP broadcast…

Starting 2022 Off With A Fairly Complex {ggplot2} Recreation Plot

Tue, 04 Jan 2022 16:32:01 GMT

The New York Times had a \[tragic\] story on Covid deaths today and one of their plots really stuck with me for how well it told that part of the story. NOTE: The red panel highlights are off a bit as I manually typed the data in (I only…

2021 Advent of Code Day 02 "Don't Try This At Home" Edition

Thu, 02 Dec 2021 07:34:45 GMT

The Moderna booster level drained me all day on Dec 1 and did what jab two did during the overnight period (achy enough to wake me up and not get back to slumber easily). To try to wear myself down, I decided to practice a bit of R with…

Creating A Custom MaxMind mmdb File For Cloud Provider Ranges

Tue, 16 Nov 2021 16:08:16 GMT

I've been wanting to create a custom MaxMind mmdb file for alternate IPv4 range classifications for a while, and finally had an opportunity to do so for the $DAYJOB. MaxMind mmdb files are small and easy to work with and provide lightning…

#30DayMapChallenge 2021 Day 2: Lines

Tue, 02 Nov 2021 17:55:42 GMT

The 30-Day Map Challenge is on again, and I'm hoping to be able to scrounge some time to get an entry for each day. Day 2 is lines (Day 1 was posted on Twitter only) and — while I'm hoping to focus on saving U.S. democracy for the majority…

RSwitch 2.1.2b Bug-fix Release

Mon, 06 Sep 2021 12:39:14 GMT

RStudio is continuing to refine their ?? new product version string scheme, with the most recent change using something like the following for daily…

Some Covid Donuts To End The Week

Fri, 13 Aug 2021 19:23:00 GMT

Vox grabbed some data from the Kaiser Family Foundation and did a story a few days ago on it, then posted a different visualization of it that attracted some attention: I'm a pretty ardent donut detractor, but I have to also admit that…

Hasty Bug-fix Release of RSwitch v2

Wed, 28 Jul 2021 18:42:29 GMT

Version 2 of RSwitch — the macOS menubar utility that enables fast and seamless switching between R versions on macOS and also provides notifications for new versions of RStudio Dailies has had a hasty update to deal with an error…

Acoustic: Solving a CyberDefenders PCAP SIP/RTP Challenge with R, Zeek, tshark (& friends)

Sun, 25 Jul 2021 13:40:14 GMT

Hot on the heels of the previous CyberDefenders Challenge Solution comes this noisy installment which solves their Acoustic challenge. You can find the source Rmd on GitHub, but I'm also testing the limits of WP's markdown rendering and…

Packet Maze: Solving a CyberDefenders PCAP Puzzle with R, Zeek, and tshark

Tue, 20 Jul 2021 19:18:12 GMT

It was a rainy weekend in southern Maine and I really didn't feel like doing chores, so I was skimming through RSS feeds and noticed a link to a PacketMaze challenge in the latest This Week In 4n6. Since it's also been a while since I've…

New TabularData Available in Swift on macOS Monterery (et al)

Tue, 08 Jun 2021 16:56:08 GMT

WWDC 2021 is on this week and many new fun things are being introduced, including some data science-friendly additions to the frameworks that come with Xcode 13 and available on macOS 12+ (and its \OS cousins). Specifically, Apple has made…

archinfo v0.4.0 Released

Wed, 26 May 2021 13:38:42 GMT

My archinfo utility is a small macOS command line application that displays process id, name, and architecture (arm64 or x86\64). Version 0.4.0 adds the following…

A {swiftr} Brief Interlude While Awaiting {cdcfluview} CRAN Checks

Sat, 22 May 2021 14:01:35 GMT

My {cdcfluview} package started tossing erros on CRAN just over a week ago when the CDC added an extra parameter to one of the hidden API endpoints that the package wraps. After a fairly hectic set of days since said NOTE came, I had time…

Using the new Plot Javascript Exploratory Visualization Library Sans-Observable

Sun, 09 May 2021 12:25:51 GMT

The fine folks over at @ObservableHQ released a new javascript exploratory visualization library called Plot last week with great fanfare. It was primarily designed to be used in Observable notebooks and I quickly tested it out there (you…

Feedly Mini Extension Removed From Chrome Store Due To "Malware"

Sun, 09 May 2021 11:57:52 GMT

On or about Friday evening (May 7, 2021) Edge notified me that the Feedly Mini extension (one of the only extensions I use as extensions are dangerous things) was remove from the store due to "malware". Feedly is used by many newshounds,…

COVID-19 U.S. County Vaccination Tracker With An Observable Notebook Using Datasettes and {Plot}

Thu, 06 May 2021 08:46:02 GMT

Rather than continue to generate daily images with R, I threw together an Observable notebook that takes advantage of the CDC COVID-19 county data datasette (provided by Simon Willison) and the new {Plot} library (by the @ObservableHQ…

A Small macOS (Big Sur+) App to Extract Indicators of Compromise

Sun, 25 Apr 2021 10:20:21 GMT

There's a semi-infrequent-but-frequent-enough-to-be-annoying manual task at $DAYJOB that involves extracting a particular set of strings (identifiable by a fairly benign set of regular expressions) from various interactive text sources…

Making macOS Universal Apps in Swift with Universal Golang Static Libraries

Sat, 24 Apr 2021 17:34:58 GMT

There are a plethora of amazingly useful Golang libraries, and it has been possible for quite some time to use Go libraries with Swift. The advent of the release of the new Apple Silicon/M1/arm64 architecture for macOS created the need for…

Avoiding The mdls Command Line Round Trip With swiftr::swift_function()

Wed, 14 Apr 2021 23:50:51 GMT

The last post showed how to work with the macOS mdls command line XML output, but with {swiftr} we can avoid the command line round trip by bridging the low-level Spotlight API (which mdls uses) directly in R via Swift. If you've already…

Quick Hit: Processing macOS Application Metadata Weirdly Fast with mdls and R

Tue, 13 Apr 2021 12:59:26 GMT

(reminder: Quick Hits have minimal explanatory blathering, but I can elaborate on anything if folks submit a comment). I'm playing around with Screen Time on xOS again and noticed mdls (macOS command line utility for getting file metadata)…

Check 'Developer Tools' First To Avoid Heavy-ish Dependencies

Mon, 12 Apr 2021 11:43:14 GMT

Guillaume Pressiat (@GuillaumePressiat) did a solid post & video on using Selenium to scrape a paginated table from understat[.]com/league/EPL/2020 (I just cannot bring myself to provide an active link to any SportsBall site). He does a…

Help Your Mac Stand Between The Darkness And The Light with GreyWatch

Tue, 30 Mar 2021 10:57:45 GMT

Greynoise helps security teams focus on potential threats by reducing the noise from logs, alerts, and SIEMs. They constantly watch for badly behaving internet hosts, keep track of the benign ones, and use this research to classify IP…

Retrieve Process Run-time Architecture on Apple Silicon Macs On The Command Line with `archinfo`

Sat, 13 Mar 2021 18:51:26 GMT

Apple M1/Apple Silicon/arm64 macOS can run x8664 programs via Rosetta and most M1 systems currently (~March 2021) very likely run a mix of x8664 and arm64 processes. Activity Monitor can show the architecture: but command line tools such…

Brimming With Possibilities: Query zqd & Mine Logs with zq from R

Mon, 01 Mar 2021 12:45:14 GMT

Brim Security maintains a free, Electron-based desktop GUI for exploration of PCAPs and select cybersecurity logs: along with a broad ecosystem of tools which can be used independently of the GUI. The standalone or embedded zqd server, as…

Getting Into the Rhythm of Chart Typography with {ragg} and {hrbragg} (a.k.a. It's {ragg}-time}

Fri, 19 Feb 2021 21:59:48 GMT

Horrible puns aside, hopefully everyone saw the news, earlier this week, from @thomasp85 on the evolution of modern typographic capabilities in the R ecosystem. Thomas (and some cohorts) has been working on {systemfonts}, {ragg}, and…

Extracting Heart Rate Data (Two Ways!) from Apple Health XML Export Files Using R (a.k.a. The Least Romantic Valentine's Day R Post Ever)

Sun, 14 Feb 2021 18:11:24 GMT

readcsv( file = "~/Data/applehealthexport/electrocardiograms/ecg2020-09-24.csv", # this is extracted below skip = 12, colnames = "µV" ) %>% mutate( idx = 1:n() ) ->…

Fully Native M1/Apple Silicon R Setup

Sun, 07 Feb 2021 21:34:36 GMT

Presented without much commentary since I stopped once {ggrepel} and {graphlayouts} failed (RStudio doesn't support it yet, either, which I knew). The following steps will get you a fully working and STUPID FAST fully native ARM64 M1/Apple…

Getting a Handle on macOS App Entitlements with R

Sun, 07 Feb 2021 18:15:44 GMT

If you've been following me around the internets for a while you've likely heard me pontificate about the need to be aware of and reduce — when possible — your personal "cyber" attack surface. One of the ways you can do that is to install…

Amazon Athena {dbplyr} Implicit Usage of Presto Functions and Making JSON Casting Great Again

Wed, 03 Feb 2021 04:10:21 GMT

I was chatting with a fellow Amazon Athena user and the topic of using Presto functions such as approx\distinct() via {d\[b\]plyr} came up and it seems it might not be fully common knowledge that any non-already translated function is…

Parler Whack-a-Mole

Sat, 30 Jan 2021 09:35:35 GMT

(this is an unrolled Twitter thread converted to the blog since one never knows how long content will be preserved anywhere…

Making It Easier To Experiment With Compiled Swift Code In R

Wed, 27 Jan 2021 03:19:38 GMT

The past two posts have (lightly) introduced how to use compiled Swift code in R, but they've involved a bunch of "scary" command line machinations and incantations. One feature of {Rcpp} I've always ? is the cppFunction() ("r-lib" zealots…

Calling [Compiled] Swift from R: Part 2

Sun, 24 Jan 2021 13:26:59 GMT

The previous post introduced the topic of how to compile Swift code for use in R using a useless, toy example. This one goes a bit further and makes a case for why one might want to do this by showing how to use one of Apple's machine…

SwiftR Switcheroo: Calling [Compiled] Swift from R!

Sat, 23 Jan 2021 22:09:18 GMT

I've been on a Swift + R bender for a while now, but have been envious of the pure macOS/iOS (et al) folks who get to use Apple's seriously ++good machine learning libraries, which are even more robust on the new M1 hardware (it's cool…

Soon May the Vendorman Come

Fri, 22 Jan 2021 11:54:35 GMT

There was an org that didn't see The data exfil hacking spree. A patch went up, our guard was down, Oh blow, SolarWinds,…

New SwiftR Chapter Up: Building an R-backed SwiftUI macOS App

Sat, 16 Jan 2021 16:42:11 GMT

Last week I introduced a new bookdown series on how to embed R into a macOS Swift application. The initial chapters focused on core concepts and showed how to build a macOS compiled, binary command line application that uses embedded R for…

Nos Autem Non In Antebellum; Bella Iam Inceperat

Sun, 10 Jan 2021 11:56:41 GMT

(Leading this with the periodic warning/reminder that this blog occasionally breaks from technical content and has category-based RSS feeds which can be used to ensure one never see non-technical content.) Every decent human (which…

Safety Checking Locally Installed Package URLs

Fri, 08 Jan 2021 14:00:47 GMT

I went completely daft this week and broke my months-long Twitter break due to the domestic terror event in my nation's capitol. I'll likely be resuming the break starting today. Whilst keeping up with the final descent of the U.S. into a…

Bringing R to Swift on macOS

Mon, 04 Jan 2021 22:29:36 GMT

// handling dots in named calls is a WIP = try R.evalParse("options(tidyverse.quiet = TRUE…

Updated Apache Drill R JDBC Interface Package {sergeant.caffeinated} With {dbplyr} 2.x Compatibility

Sat, 21 Nov 2020 01:59:33 GMT

While the future of the Apache Drill ecosystem is somewhat in-play (MapR — a major sponsoring org for the project — is kinda dead), I still use it almost daily (on my local home office cluster) to avoid handing over any more money to…

Apple Silicon + Big Sur + RStudio + R Field Report

Wed, 18 Nov 2020 22:15:27 GMT

It's been a while since I've posted anything R-related and, while this one will be brief, it may be of use to some R folks who have taken the leap into Big Sur and/or Apple Silicon. Stay to the end for an early Christmas…

It's [Almost] Over; Much Damage Has Been Done; But I [We] Have A Call To Unexpected Action

Sun, 08 Nov 2020 12:45:40 GMT

NOTE: There's a unique feed URL for R/tech stuff — https://rud.is/b/category/r/feed/. If you hit the generic "subscribe" button b/c the vast majority of posts have been on that, this isn't one of those posts and you should probably delete…

Short & Sweet: {cdcfluview} 0.9.2 Is On Its Way to CRAN Mirrors

Tue, 13 Oct 2020 15:50:31 GMT

The CDC continues to "deliver" in 2020, this time by changing the JSON response of one of the hidden APIs that my {cdcfluview} package wraps. CDC: So helpful! It was a quick fix, and version 0.9.2 passed automated CRAN checks in ~9.42…

Quick Hit: Speeding Up Data Frame Creation

Sat, 08 Aug 2020 12:28:20 GMT

(This is part 2 of n "quick hit" posts, each walking through some approaches to speeding up components of an iterative operation. Go here for part 1). Thanks to the aforementioned previous post, we now have a super fast way of reading…

Quick Hit: Comparison of "Whole File Reading" Methods

Fri, 07 Aug 2020 11:42:25 GMT

(This is part 1 of n posts using this same data; n will likely be 2-3, and the posts are more around optimization than anything else.) I recently had to analyze HTTP response headers (generated by a HEAD request) from around 74,000 sites…

Aligning the Dots On COVID Prison Waffles

Fri, 24 Jul 2020 20:25:15 GMT

The Marshall Project has a solid story and set of visualizations on the impact of COVID-19 in U.S. prisons. They keep the data (and vis) regularly updated. They do great work and this is an important topic, but this visualization breaks my…

A Look at PAN-OS Versions with a Bit of R

Fri, 10 Jul 2020 15:03:51 GMT

The incredibly talented folks over at Bishop Fox were quite generous this week, providing a scanner for figuring out PAN-OS GlobalProtect versions. I've been using their decoding technique and date-based fingerprint table to keep an eye on…

The Avatar Is Blue

Wed, 03 Jun 2020 02:36:31 GMT

I've changed my years-long avatar to the blue Cap' shield because the colors of our flag have no business being displayed in any venue until Donald Trump is no longer President (one way or another). The red/white/blue triad has been…

{sergeant} 0.9.0 Is On Its Way to CRAN Mirrors!

Mon, 01 Jun 2020 17:06:13 GMT

Tis been a long time coming, but a minor change to default S3 parameters in tibbles finally caused a push of {sergeant} — the R package that lets you use the Apache Drill REST API via {DBI}, {dplyr}, or directly — to CRAN. The CRAN…

RSwitch 1.7.0 Has Been Released

Mon, 25 May 2020 18:20:47 GMT

I (and, apparently, Gandalf O\o) are pleased to announce that RSwitch version 1.7.0 has been released. (Direct Download) RSwitch is a macOS menubar utility…

Mining R 4.0.0 Changelog for Nuggets of Gold: #1 stopifnot()

Tue, 19 May 2020 21:31:39 GMT

R 4.0.0 has been out for a while, now, and — apart from a case where merge() was slower than dirt — it's been really stable for at least me (I use it daily on macOS, Linux, and Windows). Sure, it came with some headline-grabbing…

Attach Your R Code To Charts You Tweet For Reproducible R Tweets!

Sat, 16 May 2020 12:38:49 GMT

I caught this tweet by Terence Eden about using Twitter image alt-text to "PGP sign" tweet and my mind immediately went to "how can I abuse this for covert communications, malicious command-and-control, and embedding R code in tweets?".…

Dairy-Free No-Fail Biscuits

Thu, 16 Apr 2020 12:44:14 GMT

Oven @ 425°F/218°C ; Sheet pan lined with parchment paper. Sift dry bits together. Cut up shortening into small squares and mix into the dry bits with a pastry blender. Combine eggs and yogurt. Add to ^^. Fold and knead gently just until…

New Package — {cdccovidview} — To Work with the U.S. CDC's New COVID-19 Trackers: COVIDView and COVID-NET

Sat, 11 Apr 2020 16:46:03 GMT

The United States Centers for Disease Control (CDC from now on) has setup two new public surveillance resources for COVID-19. Together, COVIDView and COVID-NET provide similar weekly surveillance data as FluView does for influenza-like…

Adzuki & Andouille Soup

Sun, 05 Apr 2020 15:53:09 GMT

Brown sausage in oil then remove. Sauté onion in same oil (add more if dry) til clear. Add garlic and sauté for 1-2 minutes. Add back sausage and add in everything else and simmer for 45 minutes. To thicken, remove and pulse a few…

{uaparserjs} Updated on CRAN & Using webpack to Make {V8} Application Bundles

Wed, 01 Apr 2020 13:45:45 GMT

Just a quick note that thanks to a gentle nudge an updated version of {uaparser} --- a package that processes User Agent strings web clients send to servers --- is making its way to all the CRAN mirrors and is also available on CINC. The…

The Waffle House Index

Thu, 26 Mar 2020 11:10:40 GMT

Waffle House announced it was closing hundreds of stores this week due to SARS-Cov-2 (a.k.a COVID-19). This move garnered quite a bit of media attention since former FEMA Administrator Craig Fugate used the restaurant chain as both an…

Dairy-free Chocolate Chip Muffins

Tue, 17 Mar 2020 12:21:35 GMT

Oven @ 375°F. Paddle sugar, shortening and salt. 3-5 mins. Whisk eggs, milk, yogurt, vanilla & oil. In three batches, mix/fold ^^ into the paddled mixture. Sift together dry ingredients and mix until moist. Don't over-mix. Fold in chips.…

{catchpole} Redux and Hashing Files & Websites with {ssdeepr}

Wed, 04 Mar 2020 12:36:26 GMT

Über Tuesday has come and almost gone (some state results will take a while to coalesce) and I'm relieved to say that {catchpole} did indeed work, with the example code from before producing this on first run: If we tweak the buffer space…

Make WSJ-esque Über Tuesday Democrat Delegate Cartograms in R with {catchpole}

Mon, 02 Mar 2020 12:00:10 GMT

For folks who are smart enough not to go near Twitter, I've been on a hiatus from the platform insofar as reading the Twitter feed goes. "Why" isn't the subject of this post so I won't go into it, but I've broken this half-NYE resolution…

BIMI Up, Scotty! A look at Brand Indicators for Message Identification (BIMI) Adoption with R and the Alexa Top 1m

Fri, 21 Feb 2020 17:28:13 GMT

It seems that the need for MX, DKIM, SPF, and DMARC records for modern email setups were just not enough acronyms (and setup tasks) for some folks, resulting in the creation of yet-another-acronym --- BIMI, or, Brand Indicators for Message…

Prying ".R" Script Files Away from Xcode (et al) on macOS

Thu, 06 Feb 2020 16:36:40 GMT

As the maintainer of RSwitch --- and developer of my own (for personal use) macOS, iOS, watchOS, iPadOS and tvOS apps --- I need the full Apple Xcode install around (more R-focused macOS folk can get away with just the command-line tools…

Monitoring Website SSL/TLS Certificate Expiration Times with R, {openssl}, {pushoverr}, and {DT}

Wed, 29 Jan 2020 12:35:05 GMT

macOS R users who tend to work on the bleeding edge likely noticed some downtime at this past weekend. Part of the issue was an SSL/TLS certificate expiration situation. Moving forward, we can monitor this with R using the super spiffy…

Davos 2020 World Economic Forum 2020 Global Risk Report Cyber Cliffs Notes

Tue, 21 Jan 2020 14:14:54 GMT

Each year the World Economic Forum releases their Global Risk Report around the time of the annual Davos conference. This year's report is out and below are notes on the "cyber" content to help others speed-read through those sections (in…

Convert Apple Card PDF Statements to Tidy Data (i.e. for CSV/Excel/database export)

Mon, 13 Jan 2020 14:20:29 GMT

UPDATE 2020-02-11 Apple now supports downloading transactions as CSV or OFX! (via…

Writing Frictionless R Package Wrappers — Building A Basic R Package

Fri, 03 Jan 2020 16:22:02 GMT

Before we start wrapping foreign language code we need to make sure that basic R packages can be created. If you've followed along from the previous post you have everything you need to get started here. Just to make sure, you should be…

Yeast-raised Apple Cider Doughnuts

Wed, 01 Jan 2020 17:58:39 GMT

Offspring #4 eats pretty healthy, normally, but likes to indulge in sugary confections on occasion. He was a big help clearing away the massive amount of snow that accumulated in the last winter storm of 2019 so I decided to start the new…

Writing Frictionless R Package Wrappers — Introduction

Wed, 01 Jan 2020 12:50:07 GMT

The R language and RStudio IDE are a powerful combination for "getting stuff done", and one aspect of R itself that makes it especially useful is the ability to use it with other programming languages via a robust foreign language…

Short Attention Span Theatre: Reproducing Axios' "1 Big Thing" Google Trends 2019 News In Review with {ggplot2}

Fri, 27 Dec 2019 16:18:11 GMT

I woke up to Axios' "1 Big Thing" ridgeline chart showing the crazy that was the 2019 news cycle: and, I decided to reproduce it in…

Using #rstats to Help Santa Deliver Presents This Christmas!

Mon, 23 Dec 2019 17:49:55 GMT

The right jolly old elves over at Alteryx created a "Santalytics" challenge back in 2016 to see if their community members could help Santa deliver presents to kids all across the globe. They posted data for four challenges along with…

Quickly Create (Mostly) Responsive HTML Columns With {htmltools}

Wed, 18 Dec 2019 15:38:56 GMT

I had need to present a wall-of-text to show off a giant list of SSL certificate alternate names and needed the entire list to fit on one slide (not really for reading in full, but to show just how many there were in a way that a simple…

Spelunking macOS 'ScreenTime' App Usage with R

Mon, 28 Oct 2019 13:33:00 GMT

Apple has brought Screen Time to macOS for some time now and that means it has to store this data somewhere. Thankfully, Sarah Edwards has foraged through the macOS filesystem for us and explained where these bits of knowledge are in her…

100% Stacked Chicklets

Fri, 27 Sep 2019 17:15:57 GMT

I posted a visualization of email safety status (a.k.a. DMARC) of the Fortune 500 (2017 list) the other day on Twitter and received this spiffy request from…

RSwitch 1.5.1 Released

Sat, 21 Sep 2019 12:25:46 GMT

An RSwitch user, lcolladotor filed a most-welcome issue letting me know that the core functionality of the switcher was busted ?. After testing out the 1.5.1 release candidate I had made a "harmless" & "clever" change to reduce some…

Twitter "Account Analysis" in R

Sat, 14 Sep 2019 13:11:23 GMT

This past week @propublica linked to a really spiffy resource for getting an overview of a Twitter user's profile and activity called accountanalysis. It has a beautiful interface that works as well on mobile as it does in a real browser.…

RSwitch 1.5.0 Release Now Also Corrals RStudio Server Connections

Sat, 14 Sep 2019 12:19:29 GMT

RSwitch is a macOS menubar application that works on macOS 10.14+ and provides handy shortcuts for developing with R on macOS. Version 1.5.0 brings a reorganized menu system and the ability to manage and make connections to RStudio Server…

September 2019 Democratic Debates Added to {ggchicklet}

Sat, 14 Sep 2019 08:34:25 GMT

The latest round of the 2020 Democratic debates is over and the data from all the 2019 editions of the debates have been added to {ggchicklet}. The structure of the debates2019 built-in dataset has changed a…

RSwitch 1.4.1 Released

Sun, 01 Sep 2019 13:53:15 GMT

A minor update to RSwitch has been released. Apart from some internal code reorganization there are three user-facing changes. First, RSwitch is now notarized! That means you won't get a notice about it being from an "unidentified…

RSwitch 1.4.0 Released

Mon, 26 Aug 2019 09:46:57 GMT

Swift 5 has been so much fun to hack on that there's a new update to macOS R-focused mebubar utility RSwitch available. Along with the app comes a new dedicated RSwitch landing page and a new user's guide since it has enough features to…

Improvements to RSwitch in v1.3.0

Fri, 23 Aug 2019 18:40:30 GMT

It's only been a couple days since the initial version of my revamped take on RSwitch but there have been numerous improvements since then worth mentioning. For starters, there's a new app icon that uses the blue and gray from the official…

Polyglot FizzBuzz in R (Plus: "Why Can't Johnny Code?")

Fri, 23 Aug 2019 17:04:34 GMT

I caught this post on the The Surprising Number Of Programmers Who Can’t Program from the Hacker News RSS feed. Said post links to another, classic post on the same subject and you should read both before continuing. Back? Great! Let's dig…

Quick Hit: A new 64-bit Swift 5 RSwitch App

Thu, 22 Aug 2019 07:30:41 GMT

At the bottom of the R for macOS Developer's Page there's mention of an "other binary" called "RSwitch" that is "a small GUI that allows you to switch between R versions quickly (if you have multiple versions of R framework installed)."…

Two New Testing-themed Addins + One New and One Updated CRAN Package

Sat, 27 Jul 2019 11:47:40 GMT

It's been yet-another weirdly busy summer but I'm finally catching up on noting some recent-ish developments in the blog. First up is a full rewrite of the {wand} pacakge which still uses magic#Magicnumbersinfiles) but is 100% R code (vs a…

Quick Hit: A Different (Diminutive) Look At Distributions With {ggeconodist}

Mon, 15 Jul 2019 20:21:19 GMT

Despite being a full-on denizen of all things digital I receive a fair number of dead-tree print magazines as there's nothing quite like seeing an amazing, large, full-color print data-driven visualization up close and personal. I also…

Quick Hit: {waffle} 1.0 Font Awesome 5 Pictograms and More

Fri, 12 Jul 2019 19:46:07 GMT

The {waffle} package got some ? this week and now has a substantially improved geomwaffle() along with a brand new sibling function geompictogram() which has all the powerful new features of geomwaffle() but lets you use Font Awesome 5…

Make Refreshing Segmented Column Charts with {ggchicklet}

Sun, 30 Jun 2019 14:26:03 GMT

The first U.S. Democratic debates of the 2020 election season were held over two nights this past week due to the daft number of candidates running for POTUS. The spiffy @NYTgraphics folks took the tallies of time spent blathering by each…

Quick hit: 'dig'-ging Into r-project.org DNS Records with {processx}

Fri, 28 Jun 2019 11:29:57 GMT

The r-project.org domain had some temporary technical difficulties this week (2019-29) that made reaching R-related resources problematic for a bunch of folks for a period of time. Incidents like this underscore the need for regional and…

Quick Hit: Above the Fold; Hard wrapping text at 'n' characters

Wed, 26 Jun 2019 11:48:31 GMT

Despite being on holiday I'm getting in a bit of non-work R coding since the fam has a greater ability to sleep late than I do. Apart from other things I've been working on a PR into {lutz}, a package by @andyteucher that turns lat/lng…

Quick hit: Some ggplot2 Stat ? for {logspline}

Tue, 18 Jun 2019 16:16:40 GMT

I've become a big fan of the {logspline} package over the past ~6 months and decided to wrap up a manual ggplot2 plotting process (well, it was at least in an RStudio snippet) into a small {ggplot2} Stat to make it easier to visualize…

Introducing the {ethercalc} package

Sat, 15 Jun 2019 15:53:37 GMT

I mentioned EtherCalc in a previous post and managed to scrounge some time to put together a fledgling {ethercalc} package (it's also on GitLab, SourceHut, Bitbucket and GitUgh, just sub out the appropriate URL prefix). I'm creating a…

Makeover Jambalaya: Beating Dumbbells into Slopegraphs Whilst Orchestrating EtherCalc

Tue, 11 Jun 2019 20:01:16 GMT

This morning, @kairyssdal tweeted out the following graphic from…

Wrapping Up Exploration of John Deere's MowerPlus Database

Sun, 09 Jun 2019 16:32:51 GMT

I did another twitter thread on the aforeblogged MowerPlus database as I explored the tables after a second mow to determine what identified a unique mowing "session" (using John Deere's terms). This is the…

Reader Challenge: Have Some Fun with Apple's WWDC 2019 Keynote Subtitles & Show Your Work

Fri, 07 Jun 2019 15:14:27 GMT

I was pretty brutal to Apple earlier this week in a Twitter thread that I tried to craft so it occurred in-line with the WWDC live stream (which might be something you want to remember as/if you read on). I really don't care about…

Make Multi-point "dumbbell" Plots in ggplot2

Thu, 06 Jun 2019 13:16:22 GMT

A user of the {ggalt} package recently posted a question about how to add points to a geomdumbbell() plot. For now, this is not something you can do with geomdumbbell() but with a bit of data wrangling you can do this in a pretty…

Trawling Through iOS Backups For Treasure (a.k.a. How to fish for target files in iOS backups) with R

Sun, 02 Jun 2019 15:35:55 GMT

In a recent previous post I brazenly talked over the "hard parts" of how I got to the target SQLite file that houses "mowing history" for what has become my weekend obsession. So, we'll cover just how to do that (find things in iOS…

Quick Hit: Handling Cocoa Core Data Timestamps in R

Sat, 01 Jun 2019 23:03:34 GMT

For the first time ever we got a new riding mower this weekend. We've always haggled to keep the one sellers were using with any given house we've purchased over the years (that was big enough for a yard that "requires" a riding mower). We…

Making a Command Line HTML Rendering Script for "The Art of the Command Line" (in R)

Fri, 31 May 2019 14:43:05 GMT

The Feedly category I have setup for git-stalking has indicated a fairly massive interest in Joshua Levy's The Art of the Command Line. What is "The Art of the Command Line"? To quote the…

Two New Ways to Make DNS over HTTPS Queries in R

Sun, 26 May 2019 18:30:32 GMT

A fair bit of time ago the {gdns} package made its way to CRAN to give R users the ability to use Google's (at that time) nascent support for DNS over HTTPS (DoH). A bit later on Cloudflare also provided a global DoH endpoint and that…

Add Dressbarn to the Continued Retailpocalypse

Tue, 21 May 2019 11:49:07 GMT

I've talked about the retailpocalypse before and this morning I was greeted with the news about Dressbarn closing all 650 stores as I fired up a browser. I tweeted some pix and data but not everyone is on Twitter so I'm just posting a…

Mapping Tornado Alley with R

Sat, 18 May 2019 14:26:34 GMT

So, we have to create a map of historical tornado frequency trends on deadline. We emailed researchers and received three txt files. One is a set of latitudes, another longitudes, and the final one is the trend value. It's gridded data.…

Quick Hit: Updates to QuickLookR and {rdatainfo}

Sun, 12 May 2019 13:06:48 GMT

I'm using GitUgh links here b/c the issue was submitted there. Those not wishing to be surveilled by Microsoft can find the macOS QuickLook plugin project and {rdatainfo} project in SourceHut and GitLab (~hrbrmstr and hrbrmstr accounts…

Quick Hit: Scraping javascript-"enabled" Sites with {htmlunit}

Sat, 27 Apr 2019 17:20:50 GMT

I've mentioned {htmlunit} in passing before, but did not put any code in the blog post. Since I just updated {htmlunitjars} to the latest and greatest version, now might be a good time to do a quick demo of it. The {htmlunit}/{htmunitjars}…

A Note to Our Community On How To Hide Your Content From Search Engines

Fri, 12 Apr 2019 16:27:40 GMT

UPDATE 2019-04-17 — The example at the bottom which shows that the, er, randomly chosen site has the offending tag present is an old result. As of this update timestamp, that robots noindex tag is not on the site. Since the presence status…

Lost In [SQL] Translation: Charting d[b]plyr Mapped SQL Function Support Across All Backends

Wed, 10 Apr 2019 10:57:42 GMT

Like more posts than I care to admit, this one starts innocently enough with a tweet by…

A Limited-but-Functional Couchbase Free Text Search & Retrieval Un-package; or, "How I Abused Couchbase & R to Perform Bulk IP Whois Full-text Searches" (a Cobbler's Tale)

Sun, 07 Apr 2019 14:31:29 GMT

Researching "the internet" (i.e. $DAYJOB) means having to deal with a ton of "unique" (I'm being kind) data formats. This is ultimately a tale of how I performed full-text searches across one of them. It all started off innocently enough.…

Wicked Fast, Accurate Quantiles Using ‘t-Digests’ in R with the {tdigest} Package

Wed, 03 Apr 2019 10:04:50 GMT

@ted\dunning recently updated the t-Digest algorithm he created back in 2013. What is this "t-digest"? Fundamentally, it is a probabilistic data structure for estimating any percentile of distributed/streaming data. Ted explains it quite…

Rome Was Not Built In A Day But widgetcard Was!

Tue, 26 Mar 2019 14:58:38 GMT

I saw a second post on turning htmlwidgets into interactive Twitter Player cards and felt somewhat compelled to make creating said entities a bit easier so posited the…

Assumptions Matter More Than Dependencies

Tue, 19 Mar 2019 04:40:25 GMT

There's been alot of talk about "dependencies" in the R universe of late. This is not really a post about that but more of a "really, don't do this" if you decide you want to poke the dependency bear by trying to build a deeply flawed…

Handling & Sharing PCAPs Like a Boss with PacketTotal

Sun, 17 Mar 2019 23:43:22 GMT

The fine folks over at @PacketTotal bequeathed an API token on me so I cranked out an R package for it to enable more dynamic investigations work (RStudio makes for an amazing incident responder investigations console given that you can…

Collecting Content Security Policy Violation Reports in S3 ('Effortlessly'/'Freely')

Thu, 14 Mar 2019 14:02:17 GMT

In the previous post I tried to explain what Content Security Policies (CSPs) are and how to work with them in R. In case you didn't RTFPost the TLDR is that CSPs give you control over what can be loaded along with your web content and can…

Head's Up! Roll Your Own HTTP Headers Investigations with the 'hdrs' Package

Tue, 05 Mar 2019 20:50:24 GMT

I blathered alot about HTTP headers in the last post. In the event you wanted to dig deeper I threw together a small package that will let you grab HTTP headers from a given URL and take a look at them. The README has examples for most…

IP, User Agent, and Referrer Tracking Disabled on cinc.rud.is and git.rud.is

Mon, 04 Mar 2019 19:10:38 GMT

Not flagging this with an "R" tag since I don't want to spam R-bloggers but I mentioned here that I'd be disabling logging on https://cinc.rud.is and https://git.rud.is and I wanted to follow up on that with an addendum that I've opted to…

CRAN Mirror "Security"

Sun, 03 Mar 2019 20:22:11 GMT

In the "Changes on CRAN" section of the latest version of the The R Journal (Vol. 10/2, December 2018) had this short blurb entitled "CRAN mirror…

htmlunitjars Updated to 2.34.0

Thu, 28 Feb 2019 23:10:55 GMT

The in-dev htmlunit package for javascript-"enabled" web-scraping without the need for Selenium, Splash or headless Chrome relies on the HtmlUnit library and said library just released version 2.34.0 with a wide array of changes that…

drat All The ?! : Enabling Easier Package Discovery and Installation with Your Own CRAN-like Repo for Your Packages

Thu, 28 Feb 2019 14:34:40 GMT

I've got a work-in-progress drat\-ified CRAN-like repo for (eventually) all my packages over at CINC? ("CINC is not CRAN" and it also sounds like "sync"). This is in parallel with a co-location/migration of all my packages to SourceHut…

Cloudy with a chance of Caffeinated Query Orchestration - New rJava Wrappers for AWS Athena SDK for Java

Fri, 22 Feb 2019 21:54:33 GMT

There are two fledgling rJava-based R packages that enable working with the AWS SDK for…

I Just Wanted The Data : Turning Tableau & Tidyverse Tears Into Smiles with Base R (An Encoding Detective Story)

Wed, 20 Feb 2019 05:08:49 GMT

Those outside the Colonies may not know that Payless—a national chain that made footwear affordable for millions of 'Muricans who can't spare $100.00 USD for a pair of shoes their 7 year old will outgrow in a year— is closing. CNBC also…

In Dev: WiGLE Your Way Into A Hotspot with wiglr

Mon, 18 Feb 2019 17:13:21 GMT

WiGLE has been around a while and is a great site to explore the pervasiveness or sparsity of Wi-Fi (and cellular) networks around the globe. While interactive use is fun, WiGLE also has a free API (so long as you obey the EULA and aren't…

Conquering Caffeinated Amazon Athena with the metis Trio of Packages

Mon, 18 Feb 2019 03:37:14 GMT

I must preface this post with the posit that if you're doing anything interactive() with Amazon Athena you should seriously consider just using their free ODBC drivers as it's the easiest way to wire them up to R DBI- and tidyverse-wise.…

Using the ropendata R Package to Access Petabytes of Free Internet Telemetry Data from Rapid7

Fri, 15 Feb 2019 03:15:02 GMT

I've got a post up over at $DAYJOB's blog on using the ropendata? package to access the ginormous and ever-increasing amount of internet telemetry (scan) data via the Rapid7 Open Data API. It's super-R-code-heavy but renders surprisingly…

Quick Hit: Speeding Up a Slow/Mundane Task with a Little Rcpp

Sat, 09 Feb 2019 22:03:43 GMT

Over at $DAYJOB's blog I've queued up a post that shows how to use our new ropendata? package to work with our Open Data portal's API. I'm not super-sure when it's going to be posted so keep an RSS reader fixed on https://blog.rapid7.com/…

Send UDP Probes (with payloads) and Receive/Process Responses in R

Sun, 03 Feb 2019 20:32:48 GMT

We worked pretty hard over at $DAYJOB on helping to quantify and remediate a fairly significant configuration weakness in Ubiquiti network work gear attached to the internet. Ubiquiti network gear — routers, switches, wireless access…

R Package Update: urlscan

Sun, 03 Feb 2019 11:48:55 GMT

The urlscan? package (an interface to the urlscan.io API) is now at version 0.2.0 and supports urlscan.io's authentication requirement when submitting a link for analysis. The service is handy if you want to learn about the details — all…

Homebrew 2.0.0 Released == homebrewanalytics package updated

Sat, 02 Feb 2019 14:36:40 GMT

A major new release of Homebrew has landed and now includes support for Linux as well as Windows! via the Windows Subsystem for Linux. There are overall stability and speed improvements baked in as well. The aforelinked notification has…

Fast Static Maps Built with R

Wed, 30 Jan 2019 12:52:37 GMT

Luke Whyte posted an article (apologies for a Medium link) over on Towards Data Science showing how to use a command line workflow involving curl, node and various D3 libraries and javascript source files to build a series of SVG static…

Quick Hit: Using seymour to Subscribe to your Git[la|hu]b Repo Issues in Feedly

Wed, 30 Jan 2019 08:25:54 GMT

The seymour? Feedly API package has been updated to support subscribing to RSS/Atom feeds. Previously the package was intended to just treat your Feedly as a data source, but there was a compelling use case for enabling subscription…

Acquisitions and Supply Chains: The Achilles' heel of Product/Organizational Security

Tue, 29 Jan 2019 14:50:29 GMT

(A reminder to folks expecting "R"/"data science" content: the feed for that is at https://rud.is/b/category/r/feed/ if you don't want to see the occasional non-R/datasci…

Quick Hit: Automating Production Graphics Uploads in R Markdown Documents with googledrive

Thu, 24 Jan 2019 20:03:42 GMT

As someone who measures all kinds of things on the internet as part of his $DAYJOB, I can say with some authority that huge swaths of organizations are using cloud-services such as Google Apps, Dropbox and Office 365 as part of their…

hrbrthemes 0.6.0 on CRAN + Other In-Development Package News

Mon, 21 Jan 2019 16:47:58 GMT

Version 0.6.0 of the hrbrthemes? package should be hitting a CRAN mirror near you soon. Apart from some general documentation and code cleanup this release includes the dark theme folks have been seeing in blog posts and tweets over the…

splashr 0.6.0 Now Uses the CRAN-nascent stevedore Package for Docker Orchestration

Tue, 15 Jan 2019 01:56:30 GMT

The splashr package \[srht|GL|GH\] — an alternative to Selenium for javascript-enabled/browser-emulated web scraping — is now at version 0.6.0 (still in dev-mode but on its way to CRAN in the next 14 days). The major change from version…

Roll Your Own Federal Government Shutdown-caused SSL Certificate Expiration Monitor in R

Thu, 10 Jan 2019 22:22:25 GMT

By now, even remote villages on uncharted islands in the Pacific know that the U.S. is in the midst of a protracted partial government shutdown. It's having real impacts on the lives of Federal government workers but they aren't the only…

Waffle Geoms & Other Miscellaneous In-Development Package Updates

Thu, 10 Jan 2019 13:56:02 GMT

More than just sergeant has been hacked on recently, so here's a run-down of various ?…

On the Road to 0.8.0 — Some Additional New Features Coming in the sergeant Package

Wed, 09 Jan 2019 19:32:59 GMT

It was probably not difficult to discern from my previous Drill-themed post that I'm fairly excited about the Apache Drill 1.15.0 release. I've rounded out most of the existing corners for it in preparation for a long-overdue CRAN update…

'data:' Scraping & Chart Reproduction : Arrows of Environmental Destruction

Thu, 03 Jan 2019 15:26:02 GMT

Today's RSS feeds picked up this article by Marianne Sullivan, Chris Sellers, Leif Fredrickson, and Sarah Lamdanon on the woeful state of enforcement actions by the U.S. Environmental Protection Agency (EPA). While there has definitely…

Apache Drill 1.15.0 + sergeant 0.8.0 = pcapng Support, Proper Column Types & Mounds of New Metadata

Wed, 02 Jan 2019 22:24:17 GMT

Apache Drill is an innovative distributed SQL engine designed to enable data exploration and analytics on non-relational datastores \[...\] without having to create and manage schemas. \[...\] It has a schema-free JSON document model…

Exploring 2018 R-bloggers & R Weekly Posts with Feedly & the 'seymour' package

Mon, 31 Dec 2018 12:36:32 GMT

Well, 2018 has flown by and today seems like an appropriate time to take a look at the landscape of R bloggerdom as seen through the eyes of readers of R-bloggers and R Weekly. We'll do this via a new package designed to make it easier to…

Certifiably Gone Phishing

Sun, 23 Dec 2018 22:28:00 GMT

Phishing is \[still\] the primary way attackers either commit a primary criminal act (i.e. phish a target to, say, install ransomware) or is the initial vehicle used to gain a foothold in an organization so they can perform other criminal…

More "Scraping Ethics Gone Awry" and "Why Do This When There's a Free API?"

Sun, 02 Dec 2018 19:20:11 GMT

I can't seem to free my infrequently-viewed email inbox from "you might like!" notices by the content-lock-in site Medium. This one made it to the iOS notification screen (otherwise I'd've been blissfully unaware of it and would have saved…

Tis the Season to Check your SSL/TLS Cipher List Thrice (RCurl/curl/openssl)

Sat, 17 Nov 2018 17:00:23 GMT

The libcurl library (the foundational library behind the RCurl and curl packages) has switched to using OpenSSL's default ciphers since version 7.56.0 (October 4 2017). If you're a regular updater of curl/httr you should be fairly current…

Use GitHub Vulnerability Alerts to Keep Users of Your R Packages Safe

Wed, 14 Nov 2018 13:47:47 GMT

Despite their now inherent evil status, GitHub has some tools other repository aggregators do not. One such tool is the free vulnerability alert service which will scan repositories for outdated+vulnerable dependencies. Now, "R" is nowhere…

Escaping the macOS 10.14 (Mojave) Filesystem Sandbox with R / RStudio

Fri, 09 Nov 2018 12:01:20 GMT

If you're an R/RStudio user who has migrated to Mojave (macOS 10.14) or are contemplating migrating to it, you will likely eventually run into an issue where you're trying to access resources that are in Apple's new hardened filesystem…

GeoJSON Version of CBC Quebec Ridings Hex Cartograms with Example Usage in R

Wed, 10 Oct 2018 08:39:50 GMT

The CBC covered the recent (as of the original post-time on this blog entry) Quebec elections and used a well-crafted hex grid map to display results: They have a great 'splainer on why they use this type of map. Thinking that it may be…

Quick Hit: Using the New Equal Earth Projection in R

Fri, 28 Sep 2018 11:08:23 GMT

In my semi-daily run of brew update I noticed that proj4 had been updated to 5.2. I kinda "squeee"'d since (as the release notes show) the Equal Earth projection was added to it (+proj=eqearth). As the team who created the projection…

Taking a Tour of the Pirate Ship 'GitHub DMCA' with R

Wed, 19 Sep 2018 13:26:17 GMT

Despite having sailed through the core components of this year's Talk Like A Pirate Day R post a few months ago, time has been an enemy of late so this will be a short post that others can build off of, especially since there's lots more…

Access the Internet Archive Advanced Search/Scrape API with wayback (+ links to a new vignette & pkgdown site)

Mon, 17 Sep 2018 21:27:59 GMT

The wayback? package has had an update to more efficiently retrieve mementos and added support for working with the Internet Archive's advanced search+scrape…

The Evolution of Data Literacy at the U.S. Department of Energy + Finding Power Grid Cyber Attacks in a Data Haystack

Wed, 12 Sep 2018 15:33:07 GMT

I was chatting with some cyber-mates at a recent event and the topic of cyber attacks on the U.S. power-grid came up (as it often does these days). The conversation was brief, but the topic made its way into active memory and resurfaced…

Driving Drill Dynamically with Docker and Updating Storage Configurations On-the-fly with sergeant

Sun, 09 Sep 2018 19:58:06 GMT

The sergeant? package has a minor update that adds REST API coverage for two "new" storage endpoints that make it possible to add, update and remove storage configurations on-the-fly without using the GUI or manually updating a config…

Simplifying World Tile Grid Creation with geom_wtg()

Mon, 27 Aug 2018 17:45:04 GMT

Nowadays (I've seen that word used so much in journal articles lately that I could not resist using it) I'm using world tile grids more frequently as the need arises to convey the state of exposure of various services at a global (country)…

Friday #rstats twofer: Finding macOS 32-bit apps & Processing Data from System Commands

Fri, 24 Aug 2018 09:01:26 GMT

Apple has run the death bell on 32-bit macOS apps and, if you're running a recent macOS version on your Mac (which you should so you can get security updates) you likely see this alert from time-to-time: If you're like me, you click…

Introducing 'gepetto' — a Splash-like REST API to Headless Chrome

Thu, 23 Aug 2018 21:32:53 GMT

It's been over a year since Headless Chrome was introduced and it has matured greatly over that time and has acquired a pretty large user base. The TLDR on it is that you can now use Chrome as you would any command-line interface (CLI)…

In-brief: Using Bro connection logs with Apache Drill

Fri, 17 Aug 2018 14:56:00 GMT

If you've got a directory full of Bro NSM logs, it's easy to work with them in Apache Drill since they're just tab-separated values (TSV) files by default. The most tedious part is mapping the columns to proper types and hopefully this…

Updates to the sergeant (Apache Drill connector) Package & a look at Apache Drill 1.14.0 release

Thu, 16 Aug 2018 15:49:17 GMT

Apache Drill 1.14.0 was recently released, bringing with it many new features and a temporary incompatibility with the current rev of the MapR ODBC drivers. The Drill community expects new ODBC drivers to arrive shortly. The sergeant? is…

In-brief: splashr update + High Performance Scraping with splashr, furrr & TeamHG-Memex's Aquarium

Mon, 13 Aug 2018 20:35:05 GMT

The development version of splashr now support authenticated connections to Splash API instances. Just specify user and pass on the initial splashr::splash() call to use your scraping setup a bit more safely. For those not familiar with…

Connecting Apache Zeppelin and Apache Drill, PostgreSQL, etc.

Sun, 12 Aug 2018 02:20:34 GMT

A previous post showed how to use a different authentication provider to wire up Apache Zeppelin and Amazon Athena. As noted in that post, Zeppelin is a "notebook" alternative to Jupyter (and other) notebooks. Unlike Jupyter, I can…

Connecting Apache Zeppelin Up to Amazon Athena with an IAM Profile Name

Sat, 11 Aug 2018 21:37:39 GMT

Apache Zeppelin is a "notebook" alternative to Jupyter (and other) notebooks. It supports a plethora of kernels/interpreters and can do a ton of things that this post isn't going to discuss (perhaps future ones will, especially since it's…

Digging into mbox details: A tale of tm & reticulate

Sat, 04 Aug 2018 13:17:46 GMT

I had to processes a bunch of emails for a $DAYJOB task this week and my "default setting" is to use R for pretty much everything (this should come as no surprise). Treating mail as data is not an uncommon task and many R packages exist…

ggplot "Doodling" with HIBP Breaches

Sun, 29 Jul 2018 13:08:34 GMT

After reading this interesting analysis of "How Often Are Americans' Accounts Breached?" by Gaurav Sood (which we need more of in cyber-land) I gave in to the impulse to do some gg-doodling with the "Have I Been Pwnd" JSON data he used.…

Two new Apache Drill UDFs for Processing UR[IL]s and Internet Domain Names

Fri, 27 Jul 2018 00:28:05 GMT

Continuing the blog's UDF theme of late, there are two new UDF kids in…

New Apache Drill UDF for Processing Twitter Tweet Text

Sun, 22 Jul 2018 19:47:23 GMT

There are many ways to gather Twitter data for analysis and many R and Python (et al) libraries make full use of the Twitter API when building a corpus to extract useful metadata for each tweet along with the text of each tweet. However,…

A new 'boto3' Amazon Athena client wrapper with dplyr async query support

Fri, 20 Jul 2018 14:32:02 GMT

A previous post explored how to deal with Amazon Athena queries asynchronously. The function presented is a beast, though it is on purpose (to provide options for folks). In reality, nobody really wants to use rJava wrappers much anymore…

Alleviating AWS Athena Aggravation with Asynchronous Assistance

Sat, 14 Jul 2018 23:39:37 GMT

I've blogged about how to use Amazon Athena with R before and if you are a regular Athena user, you've likely run into a situation where you prepare a dplyr chain, fire off a collect() and then wait. And, wait. And, wait. And, wait.…

Visualizing macOS App Usage with a Little Help from osqueryr & mactheknife

Fri, 06 Jul 2018 17:34:53 GMT

Both my osqueryr and macthekinfe packages have had a few updates and I wanted to put together a fun example (it being Friday, and all) for what you can do with them. All my packages are now on GitHub and GitLab and I'll be maintaining them…

Freeing PDF Data to Account for the Unaccounted

Mon, 02 Jul 2018 10:39:59 GMT

I've mentioned @stiles before on the blog but for those new to my blatherings, Matt is a top-notch data journalist with the @latimes and currently stationed in South Korea. I can only imagine how much busier his life has gotten since that…

Build httr Functions Automagically from Manual Browser Requests with the middlechild Package

Fri, 15 Jun 2018 11:32:44 GMT

You can catch a bit of the @rOpenSci 2018 Unconference experience at home w with this short-ish 'splainer video on how to use the new middlechild package (https://github.com/ropenscilabs/middlechild) & mitmproxy to automagically create…

Running RStudio (1.2) Background Jobs

Sat, 09 Jun 2018 14:37:20 GMT

The forthcoming RStudio 1.2 release has a new "Jobs" feature for running and managing background R tasks. I did a series of threaded screencaps on Twitter but that doesn't do the feature justice. So I threw together a quick 'splainer on…

Making World Tile Grid-Grids

Thu, 07 Jun 2018 21:01:52 GMT

A colleague asked if I would blog about how I crafted the grid of world tile grids in this post and I accepted the challenge. The technique isn't too hard as it just builds on the initial work by Jon Schwabish and a handy file made by…

Hello, Dorling! (Creating Dorling Cartograms from R Spatial Objects + Introducing Prism Skeleton)

Mon, 04 Jun 2018 01:42:08 GMT

After seeing Fira Sans in action in presentations at eRum 2018 I felt compelled to add hrbrthemes support for it so I made a firasans? extension to it that uses Fira Sans Condensed and Fira Code fonts for ggplot2 graphics. But I really…

OS Secrets Exposed: Extracting Extended File Attributes and Exploring Hidden Download URLs With The xattrs Package

Wed, 30 May 2018 16:19:23 GMT

Most modern operating systems keep secrets from you in many ways. One of these ways is by associating extended file attributes with files. These attributes can serve useful purposes. For instance, macOS uses them to identify when files…

The Fix Is In: Finding infix functions inside contributed R package "utilities" files

Tue, 29 May 2018 11:42:06 GMT

Regular readers will recall the "utility belt" post from back in April of this year. This is a follow-up to a request made asking for a list of all the % infix functions in those files. We're going…

The Power of Standards and Consistency

Sun, 27 May 2018 03:32:52 GMT

I'm going to (eventually) write a full post on the package I'm mentioning in this one : osqueryr?. The TLDR on osqueryr is that it is an R DBI wrapper (that has just enough glue to also be plugged into dbplyr) for osquery?. The TLDR on…

GDPR Unintended Consequences Part 1 — Increasing WordPress Blog Exposure

Thu, 24 May 2018 19:31:02 GMT

I pen this mini-tome on "GDPR Enforcement Day". The spirit of GDPR is great, but it's just going to be another Potempkin Village in most organizations much like PCI or SOX. For now, the only thing GDPR has done is made GDPR consulting…

Create Code Metrics with cloc

Sat, 19 May 2018 17:31:22 GMT

The cloc Perl script (yes, Perl!) by Al Danial (https://github.com/AlDanial/cloc) has been one of the go-to tools for generating code metrics. Given a single file, directory tree, archive, or git repo, cloc can speedily give you metrics on…

"LMX ot NOSJ!" Interchanging Classic Data Formats With Single blackmagic Incantations

Fri, 18 May 2018 11:20:08 GMT

One file that's in XML that I only occasionally have to process is the IANA IPv4 Address Space Registry. If you visited that link you may have uttered "Hey! That's not XML it's HTML!", to wit --- I would respond --- "Well, HTML is really…

RIPE 76 Selected Talks

Wed, 16 May 2018 13:49:01 GMT

RIPE 76 is going on this week and --- as usual --- there are scads of great talks. The selected ones below are just my (slightly) thinner slice at what may have broader appeal outside pure networking circles. Do not read anything more into…

Wrangling Data Table Out Of the FBI 2017 IC3 Crime Report

Tue, 08 May 2018 13:10:02 GMT

The U.S. FBI Internet Crime Complaint Center was established in 2000 to receive complaints of Internet crime. They produce an annual report, just released 2017's edition, and I need the data from it. Since I have to wrangle it out, I…

Seventeen Minutes From Tweet To Package

Fri, 04 May 2018 01:19:53 GMT

but I wanted to do more than that since Noam took 20 minutes out his day this week (with no advance warning) to speak to my intro-to-stats class about his work and R. The Twitter request was (ultimately) a question on how to use R to post…

Painless ODBC + dplyr Connections to Amazon Athena and Apache Drill with R & odbc

Fri, 20 Apr 2018 14:44:29 GMT

I spent some time this morning upgrading the JDBC driver (and changing up some supporting code to account for changes to it) for my metis package? which connects R up to Amazon Athena via RJDBC. I'm used to JDBC and have to deal with Java…

Examining POTUS Executive Orders

Wed, 18 Apr 2018 15:08:53 GMT

This week's edition of Data is Plural had two really fun data sets. One is serious fun (the first comprehensive data set on U.S. evictions, and the other I knew about but had forgotten: The Federal Register Executive Order (EO) data…

Access Your "Saved for Later" Feedly Items By Hooking Up Dropbox to Feedly

Wed, 18 Apr 2018 11:36:39 GMT

If you come here often you've noticed that I've been writing a semi-frequent series on using the Feedly API with R. A recent post was created to help someone use the API. It worked for them but — as you can see in the comment — an…

By Request: Retrieving Your Feedly "Saved for Later" Entries

Mon, 16 Apr 2018 18:28:26 GMT

@mkjcktzn asked if one can access Feedly "Saved for Later" items via the API. The answer is "Yes!", and it builds off of that previous post. You'll need to read it and get your authentication key (still no package ?) before continuing.…

A Word About Tracking On This Site

Sun, 15 Apr 2018 18:58:22 GMT

Since I just railed against Congress for being a bit two-faced about privacy I thought some rud.is site disclosure would be in order. At present, third-party tracking is limited…

Does Congress Really Care About Your Privacy?

Fri, 13 Apr 2018 18:50:55 GMT

I apologize up-front for using bad words in this post. Said bad words include "Facebook", "Mark Zuckerberg" and many referrals to entities within the U.S. Government. Given the topic, it cannot be helped. I've also left the R tag on this…

Convert epub to Text for Processing in R

Thu, 12 Apr 2018 10:55:36 GMT

We'll use one of @hadleywickham's books since it's O'Reilly and they do epubs well. The archive package lets us treat the epub (which is really just a ZIP file) as a mini-filesystem and embraces "tidy" so we have lovely data frames to work…

Dissecting R Package "Utility Belts"

Sun, 08 Apr 2018 11:41:14 GMT

Many R package authors (including myself) lump a collection of small, useful functions into some type of utils.R file and usually do not export the functions since they are (generally) designed to work on package internals rather than…

Adding macOS Touch Bar Support to RStudio

Fri, 06 Apr 2018 08:29:02 GMT

Modern MacBook Pros have a fairly useless (c'mon, admit it!) "Touch Bar" that did little more than cause severe ire in the developer community after turning a full-fledged, tactile Escape key into a hollow version if its former self.…

Exploring R-Bloggers Posts with the Feedly API

Wed, 04 Apr 2018 11:18:23 GMT

There's a yuge chance you're reading this post (at least initially) on R-Bloggers right now (though you should also check out R Weekly and add their live feed to your RSS reader pronto!). It's a central "watering hole" for R folks and is…

2018 IEEE Security & Privacy (Filtered) Paper Dump

Tue, 03 Apr 2018 14:20:35 GMT

The 2018 IEEE Security & Privacy Conference is in May but they've posted their full proceedings and it's better to grab them early than to wait for it to become part of a paid journal offering. There are alot of papers. Not all match my…

Nykolas Z Updates Global Alternative DNS Speed Test to Include Cloudflare (1.1.1.1)

Mon, 02 Apr 2018 10:58:50 GMT

A while back, Medium blogger 'Nykolas Z' posted results from a globally distributed DNS resolver test to find the speediest provider (NOTE: speed is not the only consideration when choosing an alternative DNS provider). While the test…

More Options For Querying DNS From R with 1.1.1.1

Sun, 01 Apr 2018 23:22:52 GMT

You have to have been living under a rock to not know about Cloudflare's new 1.1.1.1 DNS offering. I won't go into "privacy", "security" or "speed" concepts in this post since that's a pretty huge topic to distill for folks given the, now,…

Compute/Visualize Drive Space Consumption of Your Installed R Packages

Sun, 01 Apr 2018 15:08:31 GMT

The fs package makes it super quick and easy to find out just how much "package hoarding" you've been…

R⁶ — Capturing [YouTube] Captions

Thu, 29 Mar 2018 10:53:30 GMT

(R⁶ == brief, low-expository posts) @yoniceedee suggested I look at the Cambridge Analytics "whistleblower" testimony…

On MIMEs, software versions and web site promiscuity (a.k.a. three new packages to round out the week)

Fri, 23 Mar 2018 17:06:39 GMT

A quick Friday post to let folks know about three in-development R packages that you're encouraged to poke the tyres o\[fn\] and also jump in and file issues or PRs…

Steel-ing a Makeover from Tariffic Post

Mon, 12 Mar 2018 15:06:12 GMT

(FWIW I think I even caused myself pain due to the title of this blog post). Kaiser Fung (@junkcharts) did a makeover post on this chart about U.S. steel tariffs: Kaiser's makeover is good (Note: just because I said "good" does not mean…

Handling Semantic Version Strings Like a Boss with the semver Package

Wed, 07 Mar 2018 20:18:04 GMT

I work with internet-scale data and do my fair share of macro-analyses on vulnerabilities. I use the R semver package for most of my work and wanted to blather on a bit about it since it's super-helpful for this work and doesn't get the…

Comparing 2017 Maine Lobster Landings To Historical Landings

Fri, 02 Mar 2018 23:28:47 GMT

Tis the season for finding out how well Maine fisherfolk did last year; specifically, Maine lobsterfolk. Most of the news sites in Maine do a feature on the annual landings (here's one from Bangor Daily News). There was a marked decline —…

Pym.js Library Vulnerability in widgetframe Package

Fri, 16 Feb 2018 12:44:27 GMT

The NPR Visuals Team created and maintains a javascript library that makes it super easy to embed iframes on web pages and have said documents still be responsive. The widgetframe R htmlwidget uses pym.js to bring this (much needed)…

Quick and Clean DMARC Record Processing with "Inline" Rcpp

Tue, 06 Feb 2018 20:29:49 GMT

Much of what I need to do for work-work involves using tools that are (for the moment) not in R. Today, I needed to test the validity of (and other processing on) DMARC records and I'm loathe to either reinvent the wheel or reticulate bits…

The Friday #rstats PuzzleR : 2018-01-26

Fri, 26 Jan 2018 13:14:17 GMT

Time for another look at what's new and interesting in the #rstats world with the help of Peter Meissner's (@marvin\dpr) crossword.r?. The answers to last week's puzzle have been posted (it seemed to make more sense posting the answers a…

The Friday #rstats PuzzleR : 2018-01-19

Fri, 19 Jan 2018 21:04:25 GMT

Peter Meissner (@marvin\dpr) released crossword.r? to CRAN today. It's a spiffy package that makes it dead simple to generate crossword puzzles. He also made a super spiffy javascript library to pair with it, which can turn crossword model…

Bitcoin (World Map) Bubbles

Thu, 18 Jan 2018 14:58:44 GMT

We're doing some interesting studies (cybersecurity-wise, not finance-wise) on digital currency networks at work-work and --- while I'm loathe to create a geo-map from IPv4 geolocation data ---…

Can't Stop at 21: Twitter Recipe #22 — Tying Up Loose Threads

Mon, 15 Jan 2018 20:08:38 GMT

NOTE: The likelihood of this recipe being added to the recent practice bookdown book is slim, but I'll try to keep the same format for the blog…

A bookdown "Hello World" : Twenty-one (minus two) Recipes for Mining Twitter with rtweet

Fri, 05 Jan 2018 13:58:44 GMT

The new year begins with me being on the hook to crank out a book on advanced web-scraping in R by July (more on that in a future blog post). The bookdown? package seemed to be the best way to go about doing this but I had only played with…

R⁶ — Capture Tweets with tweet_shot()

Sat, 30 Dec 2017 13:31:18 GMT

UPDATE 2018-01-01 --- this has been added to rtweet (GH version). A Twitter…

2017. Quantified. In. R.

Sat, 30 Dec 2017 03:13:32 GMT

2017 is nearly at an end. We humans seem to need these cycles to help us on our path forward and have, throughout history, used these annual demarcation points as a time of reflection of what was, what is an what shall come next. To that…

New Package swatches is Now on CRAN

Fri, 22 Dec 2017 04:59:34 GMT

First off, swatches has nothing to do with those faux-luxury brand Swiss-made timepieces. swatches is all about color. R/CRAN has plenty of color picking packages. The colourlovers? ? by @thosjleeper is one of my favs. But, color palettes…

uBlock Filters for New GitHub Activity Feed

Wed, 20 Dec 2017 15:35:58 GMT

GitHub (2017-12-21 post-time) started adding obnoxious boxes to their activity feed. I use that to discover new projects/developers. While I also have it in RSS and that's nice and compact, I do browse the activity feed directly. Those…

R⁶ Series — Random Sampling From Apache Drill Tables With R & sergeant

Wed, 20 Dec 2017 13:09:45 GMT

(For first-timers, R⁶ tagged posts are short & sweet with minimal expository; R⁶ feed) At work-work I mostly deal with medium-to-large-ish data. I often want to poke at new or existing data sets w/o working across billions of rows. I also…

mqtt Development Log : On DSLs, Rcpp Modules and Custom Formula Functions

Sun, 17 Dec 2017 20:23:30 GMT

I know some folks had a bit of fun with the previous post since it exposed the fact that I left out unique MQTT client id generation from the initial 0.1.0 release of the in-development package (client ids need to be unique). There have…

Inter-operate with 'MQTT' Message Brokers With R (a.k.a. Live! BBC! Subtitles!)

Fri, 15 Dec 2017 04:31:00 GMT

Most of us see the internet through the lens of browsers and apps on our laptops, desktops, watches, TVs and mobile devices. These displays are showing us --- for the most part --- content designed for human consumption. Sure, apps handle…

A Workaround For When Anti-DDoS Also Means Anti-Data

Sun, 10 Dec 2017 14:20:35 GMT

More sites are turning to services like Cloudflare due to just how stupid-easy it is to DDoS --- perform a (possibly Distributed) Denial of Service attack on -- a site. Sometimes the DDoS is intentional (malicious). Sometimes it's because…

The Cost of True Love (a.k.a. The Tidy — and expensive! — Twelve Days of Christmas)

Tue, 05 Dec 2017 18:13:58 GMT

This one was complex enough formatting-wise that I needed to iframe it below. Feel free to bust out of the iframe at any time. Some good follow-ups to this (if you're so inclined) would be to predict prices next year and/or clean up the…

A Public Apology to Tal/R-Bloggers + The Trouble With Tibbles

Fri, 01 Dec 2017 22:18:49 GMT

Over the past few weeks, I had been noticing that some posts in the R-bloggers feed were getting truncated in Feedly. I don't remember when I noticed that since I usually click through immediately from the headline entry to the R-bloggers…

Sentiment Analysis of "A Christmas Carol"

Wed, 29 Nov 2017 23:16:55 GMT

Our family has been reading, listening to and watching "A Christmas Carol" for just abt 30 years now. I got it into my crazy noggin to perform a sentiment analysis on it the other day and tweeted out the results, but a large chunk of the R…

voteogram Is Now On CRAN

Mon, 27 Nov 2017 20:38:43 GMT

Earlier this year, I made a package that riffed off of ProPublica's really neat voting cartograms (maps) for the U.S. House and Senate. You can see one for disaster relief spending in the House and one for the ACA "Skinny Repeal" in the…

"Black"/"Cyber" Tips (a.k.a. How the hrbrgrinch ruined Christmas shopping)

Thu, 23 Nov 2017 19:34:53 GMT

NOTE: This is mainly for those of us in the Colonies, but some tips apply globally. Black Friday / Cyber Monday / Cyber November / Holiday ?hopping is upon us. You're going to buy stuff. You're going to use digital transactions to do so.…

For Victoria & Kyle

Mon, 20 Nov 2017 13:47:20 GMT

(one more time: sub to the #rstats feed if you're only on the blog for #rstats/datasci…

Statebins Reimagined

Sat, 18 Nov 2017 15:33:40 GMT

A long time ago, in a github repo far, far away there lived a tiny package that made it possible to create equal area, square U.S. state cartograms in R dubbed statebins?. Three years have come and gone and --- truth be told --- I've never…

New IBM Plex Sans Support in hrbrthemes + Automating Axis Text Justification

Thu, 16 Nov 2017 14:07:38 GMT

IBM has a new set of corporate typefaces --- dubbed "Plex" --- and has released them with a generous open license. IBM Plex Sans is not too shabby: (that image was grifted from a Font Squirrel preview page) The digit glyphs are especially…

Twitter Outer Limits : Seeing How Far Have Folks Fallen Down The Slippery Slope to "280" with rtweet

Mon, 13 Nov 2017 23:25:04 GMT

By now, virtually every major media outlet has covered the "280 Apocalypse"™. For those still not "in the know", Twitter recently moved the tweet character cap to 280 after a "successful" beta test (some of us have different ideas of what…

Measuring & Monitoring Internet Speed with R

Sat, 11 Nov 2017 18:08:20 GMT

Working remotely has many benefits, but if you work remotely in an area like, say, rural Maine, one of those benefits is not massively speedy internet connections. Being able to go fast and furious on the internet is one of the many things…

Taking a Shot at cdcfluview v0.7.0 (a.k.a. The Dangers of Relying on 'Hidden' APIs)

Mon, 06 Nov 2017 22:55:05 GMT

Unlike @noamross, I am not an epidemiologist (NOTE: Noam battles pandemics before breakfast, so be super nice to him) but I do like to find kindred methodologies in other disciplines to help foster the growth of cybersecurity into…

I, For One, Welcome Our Forthcoming New robots.txt Overlords

Fri, 03 Nov 2017 12:18:12 GMT

Despite my week-long Twitter consumption sabbatical (helped --- in part --- by the nigh week-long internet and power outage here in Maine), I still catch useful snippets from folks. My cow-orker @dabdine shunted a tweet by @terrencehart…

Yet-Another-Power Outages Post : Full Tidyverse Edition

Thu, 02 Nov 2017 11:23:01 GMT

This past weekend, violent windstorms raged through New England. We — along with over 500,000 other Mainers — went "dark" in the wee hours of Monday morning and (this post was published on Thursday AM) we still have no utility-provided…

gg_tweet'ing Power Outages

Tue, 31 Oct 2017 00:16:15 GMT

As many folks know, I live in semi-rural Maine and we were hit pretty hard with a wind+rain storm Sunday to Monday. The hrbrmstr compound had no power (besides a generator) and no stable/high-bandwidth internet (Verizon LTE was heavily…

A Call to Tweets (& Blog Posts)!

Sun, 22 Oct 2017 17:11:09 GMT

Way back in July of 2009, the first version of the twitteR package was published by Geoff Jentry in CRAN. Since then it has seen 28 updates, finally breaking the 0.x.y barrier into 1.x.y territory in March of 2013 and receiving it's last…

Enabling Concerned Visitors & Ethical Security Researchers with security.txt Web Security Policies (plus analyze them at-scale with R)

Mon, 09 Oct 2017 18:59:44 GMT

I've blogged a bit about robots.txt --- the rules file that documents a sites "robots exclusion" standard that instructs web crawlers what they can and cannot do (and how frequently they should do things when they are allowed to). This is…

Tragic Documentation

Sun, 08 Oct 2017 14:44:47 GMT

NOTE: If the usual aggregators are picking this up and there are humans curating said aggregators, this post is/was not intended as something to go into the "data science" aggregation sites. Just personal commentary with code in the event…

Retrieve & process TV News chyrons with newsflash

Sun, 01 Oct 2017 12:38:11 GMT

The Internet Archive recently announced a new service they've dubbed 'Third Eye'. This service scrapes the chyrons that annoyingly scroll across the bottom-third of TV news broadcasts. IA has a vast historical archive of TV news that…

Identify & Analyze Web Site Tech Stacks With rappalyzer

Sat, 30 Sep 2017 20:09:03 GMT

Modern websites are complex beasts. They house photo galleries, interactive visualizations, web fonts, analytics code and other diverse types of content. Despite the potential for diversity, many web sites share similar "tech stacks" ---…

SODD — StackOverflow Driven-Development

Thu, 28 Sep 2017 12:27:49 GMT

I occasionally hang out on StackOverflow and often use an answer as an opportunity to fill a package void for a particular need. docxtractr and qrencoder are two (of many) packages that were birthed from SO answers. I usually try to answer…

Speeding Up Digital Arachnids

Mon, 25 Sep 2017 10:17:04 GMT

Is it fast? Listen bud, It's got C++ under the hood. Can you scrape, from a site? Test with canfetch(), TRUE == alright Hey, there There goes the…

Pirating Web Content Responsibly With R

Tue, 19 Sep 2017 11:28:03 GMT

International Code Talk Like A Pirate Day almost slipped by without me noticing (September has been a crazy busy month), but it popped up in the calendar notifications today and I was glad that I had prepped the meat of a post a few weeks…

Mapping Fall Foliage with sf

Mon, 18 Sep 2017 11:53:14 GMT

I was socially engineered by @yoniceedee into creating today's post due to being prodded with this…

Armchair Quarterbacking Systemic Organization and Industry Failures

Sun, 17 Sep 2017 15:30:48 GMT

This is a post about the fictional company FredCo. If the context or details presented by the post seem familiar, it's purely coincidental. This is, again, a fictional story. Let's say FredCo had a pretty big breach that (fictionally)…

It's a FAKE (?)! Revisiting Trust In FOSS Ecosystems

Fri, 15 Sep 2017 18:33:45 GMT

I've blathered about trust before 1 2 , but said blatherings were in a "what if" context. Unfortunately, the if has turned into a when, which begged for further blathering on a recent FOSS ecosystem cybersecurity incident. The gg\spiffy…

Revisiting Readability With RStudio

Wed, 13 Sep 2017 23:45:26 GMT

I've blogged about my in-development R package hgr a before and it's slowly getting to a CRAN release. There are two new features to it that are more useful in an interactive session than in a programmatic context. Since they build on each…

Increasing Output Buffer Size in Apache Drill UDFs Custom (Simple) Functions

Mon, 11 Sep 2017 13:51:46 GMT

Putting this here to make it easier for others who try to Google this topic to find it w/o having to find and tediously search through other UDFs (user-defined functions). I was/am making a custom UDF for base64 decoding/encoding and ran…

Teasing Out Top Daily Topics with GDELT's Television Explorer

Sat, 09 Sep 2017 18:12:27 GMT

Earlier this year, the GDELT Project released their Television Explorer that enabled API access to closed-caption tedt from television news broadcasts. They've done an incredible job expanding and stabilizing the API and just recently…

Readability Redux

Mon, 04 Sep 2017 23:19:58 GMT

I recently posted about using a Python module to convert HTML to usable text. Since then, a new package has hit CRAN dubbed htm2txt that is 100% R and uses regular expressions to strip tags from text. I gave it a spin so folks could…

New CRAN Package Announcement: splashr

Tue, 29 Aug 2017 21:26:59 GMT

I'm pleased to announce that splashr is now on CRAN. (That image was generated with splashr::renderpng(url = "https://cran.r-project.org/web/packages/splashr/")). The package is an R interface to the Splash javascript rendering service. It…

Rpad Domain Repurposed To Deliver Creepy (and potentially malicious) Content

Tue, 29 Aug 2017 13:14:19 GMT

I was about to embark on setting up a background task to sift through R package PDFs for traces of functions that "omit NA values" as a surprise present for Colin Fay and Sir…

Unbottling ".msg" Files in R

Fri, 25 Aug 2017 22:10:35 GMT

There was a discussion on Twitter about the need to read in ".msg" files using R. The "MSG" file format is one of the many binary abominations created by Microsoft to lock folks and users into their platform and tools. Thankfully, they…

Reticulating Readability

Thu, 24 Aug 2017 15:01:41 GMT

I needed to clean some web HTML content for a project and I usually use hgr::cleantext() for it and that generally works pretty well. The cleantext() function uses an XSLT stylesheet to try to remove all non-"main text content" from an…

Caching httr Requests? This means WAR[C]!

Tue, 22 Aug 2017 16:53:34 GMT

I've blathered about my crawldelay project before and am just waiting for a rainy weekend to be able to crank out a follow-up post on it. Working on that project involved sifting through thousands of Web Archive (WARC) files. While I have…

R⁶ — Exploring macOS Applications with codesign, Gatekeeper & R

Sun, 13 Aug 2017 12:28:42 GMT

(General reminder abt "R⁶" posts in that they are heavy on code-examples, minimal on expository. I try to design them with 2-3 "nuggets" embedded for those who take the time to walk through the code examples on their systems. I'll always…

R⁶ — Reticulating Parquet Files

Tue, 01 Aug 2017 18:50:11 GMT

The reticulate package provides a very clean & concise interface bridge between R and Python which makes it handy to work with modules that have yet to be ported to R (going native is always better when you can do it). This post shows how…

Analyzing "Crawl-Delay" Settings in Common Crawl robots.txt Data with R

Fri, 28 Jul 2017 14:37:22 GMT

One of my tweets that referenced an excellent post about the ethics of web scraping garnered some…

Reading PCAP Files with Apache Drill and the sergeant R Package

Thu, 27 Jul 2017 22:07:24 GMT

It's no secret that I'm a fan of Apache Drill. One big strength of the platform is that it normalizes the access to diverse data sources down to ANSI SQL calls, which means that I can pull data from parquet, Hie, HBase, Kudu, CSV, JSON,…

R⁶ — General (Attys) Distributions

Tue, 25 Jul 2017 15:15:01 GMT

Matt @stiles is a spiffy data journalist at the @latimes and he posted an interesting chart on U.S. Attorneys General longevity (given that the current US AG is on thin…

Ten-HUT! The Apache Drill R interface package — sergeant — is now on CRAN

Mon, 17 Jul 2017 21:54:57 GMT

I'm extremely pleased to announce that the sergeant package is now on CRAN or will be hitting your local CRAN mirror soon. sergeant provides JDBC, DBI and dplyr/dbplyr interfaces to Apache Drill. I've also wrapped a few goodies into the…

R⁶ — Disproving Approval

Sun, 18 Jun 2017 13:34:53 GMT

list( Obama="http://m.rasmussenreports.com/publiccontent/politics/obamaadministration/obamaapprovalindexhistory", Trump="http://m.rasmussenreports.com/publiccontent/politics/trumpadministration/trumpapprovalindexhistory" ) %>% mapdf(~{…

Replicating the Apache Drill 'Yelp' Academic Dataset Analysis with sergeant

Sat, 17 Jun 2017 19:07:50 GMT

The Apache Drill folks have a nice walk-through tutorial on how to analyze the Yelp Academic Dataset with Drill. It's a bit out of date (the current Yelp data set structure is different enough that the tutorial will error out at various…

Keeping Users Safe While Collecting Data

Tue, 13 Jun 2017 19:48:27 GMT

I caught a mention of this project by Pete Warden on Four Short Links today. If his name sounds familiar, he's the creator of the DSTK, an O'Reilly author, and now works at Google. A decidedly clever and decent chap. The project goal is…

Engaging the tidyverse Clean Slate Protocol

Sat, 10 Jun 2017 13:32:48 GMT

I caught the 0.7.0 release of dplyr on my home CRAN server early Friday morning and immediately set out to install it since I'm eager to finish up my sergeant package and get it on CRAN. "Tidyverse" upgrades aren't trivial for me as I…

R⁶ — Scraping Images To PDFs

Mon, 05 Jun 2017 13:34:48 GMT

I've been doing intermittent prep work for a follow-up to an earlier post on store closings and came across this CNN Money "article" on it. Said "article" is a deliberately obfuscated or lazily crafted series of GIF images that contain all…

Drilling Into CSVs — Teaser Trailer

Wed, 31 May 2017 14:12:33 GMT

I used reading a directory of CSVs as the foundational example in my recent post on idioms. During my exchange with Matt, Hadley and a few others -- in the crazy Twitter thread that spawned said post -- I mentioned that I'd personally…

L.A. Unconf-idential : a.k.a. an rOpenSci #runconf17 Retrospective

Sun, 28 May 2017 14:42:27 GMT

Last year, I was able to sit back and lazily "RT" Julia Silge's excellent retrospective on her 2016 @rOpenSci "unconference" experience. Since Julia was not there this year, and the unconference experience is still in primary storage (LMD…

R⁶ — Idiomatic (for the People)

Tue, 23 May 2017 10:56:33 GMT

NOTE: I'll do my best to ensure the next post will have nothing to do with Twitter, and this post might not completely meet my R⁶ criteria. A single, altruistic, nigh exuberant R tweet about slurping up a directory of CSVs devolved quickly…

A Very Palette-able Post

Sun, 21 May 2017 13:32:21 GMT

UPDATE: I was reminded that I made a more generic version of adobecolor to handle many types of swatch files which you can find on…

R⁶ — Using R With Amazon Athena & AWS Temporary Security Credentials

Tue, 16 May 2017 18:10:33 GMT

Most of the examples of working with most of the AWS services show basic username & password authentication. That's all well-and-good, but many shops use the AWS Security Token Service to provide temporary credentials and session tokens to…

R⁶ — Tracking WannaCry Bitcoin Wallet Payments with R

Mon, 15 May 2017 01:36:33 GMT

If you follow me on Twitter or monitor @Rapid7's Community Blog you know I've been involved a bit in the WannaCry ransomworm triage. One thing I've been doing is making charts of the hourly contribution to the Bitcoin addresses that the…

Travis-CI Flaw Exposed Some 'Secure' Environment Variable Contents

Tue, 09 May 2017 02:04:39 GMT

Tagging this as #rstats\-related since many R coders use Travis-CI to automate package builds (and other things). Security researcher Ivan Vyshnevskyi did some ++gd responsible disclosure to the Travis-CI folks letting them know they were…

Plot the Vote: Making U.S. Senate & House Cartograms in R

Sun, 07 May 2017 15:37:14 GMT

Political machinations are a tad insane in the U.S. these days & I regularly hit up @ProPublica & @GovTrack sites (& sub to the GovTrack e-mail updates) as I try to be an informed citizen, especially since I've got a Senator and…

Scrapeover Friday — a.k.a. Another R Scraping Makeover

Fri, 05 May 2017 23:36:03 GMT

As I've noted before, web scraping has some rules, even though they can be tough to find. This post made a very common mistake of not putting in a time delay between requests (a cardinal scraping rule) which we'll fix in a moment. There…

R⁶ — Using pandoc from R + A Neat Package For Reading Subtitles

Sun, 30 Apr 2017 11:27:42 GMT

Once I realized that my planned, larger post would not come to fruition today I took the R⁶ post (i.e. "minimal expository, keen focus") route, prompted by a Twitter discussion with some R mates who needed to convert "lightly formatted"…

Decomposing Composers with R

Sun, 23 Apr 2017 21:27:01 GMT

The intrepid @ma\salmon cranked out another blog post, remixing classical music schedule data from Radio Swiss Classic. It's a fun post and you should read it before continuing here. Seriously, click the link and go read it before…

Shuttering Pies With Retiring Stores

Sat, 22 Apr 2017 02:32:28 GMT

I caught this "gem" in the Wall Street Journal tonight: It's pretty hard to compare store-to-store, even though it is fairly clear which ones are going-going-gone. If we want to see the relative percentage of each store closing and also…

When Homoglyphs Attack! Generating Phishing Domain Names with R

Mon, 17 Apr 2017 21:32:44 GMT

It's likely you've seen the news regarding yet-another researcher showing off a phishing domain attack. The technique is pretty…

Come Fly With Me (well, not really) — Comparing Involuntary Disembarking Rates Across U.S. Airlines in R

Thu, 13 Apr 2017 10:56:25 GMT

By now, word of the forcible deplanement of a medical professional by United has reached even the remotest of outposts in the #rstats universe. Since the news brought this practice to global attention, I found some aggregate U.S. Gov data…

R⁶ — RStudio Server Client? Make An App For That!

Fri, 07 Apr 2017 14:05:00 GMT

RStudio is a great way to work through analyses tasks, and I suspect most folks use the "desktop" version of the product on their local workstations. The fine folks at RStudio also make a server version (the codebase for RStudio is able to…

Simple Offline Airport Wi-Fi Tracker in R

Sun, 02 Apr 2017 13:03:03 GMT

The code is directly embedded in the generated R markdown document and also in this gist so I won't pollute this post with code blocks. I'm hoping @bhaskar\vk will apply his mad, l33t Leaflet h@x0r $k1llz to this crude, quick hack and kick…

Superclassing to R⁶

Sun, 02 Apr 2017 00:31:07 GMT

To avoid "branding" confusion with R⁴ I'm superclassing it to R⁶ and encouraging others in the R community to don the moniker and do their own small, focused posts on topics that would help the R community learn things. Feel free to use R⁶…

R⁶ — Snow Day Facets

Sat, 01 Apr 2017 16:34:04 GMT

Back in 2014 I blogged about first snowfall dates for a given U.S. state. It's April 1, 2017 and we're slated to get 12-18" of snow up here in Maine and @mrshrbrmstr asked how often this — snow in May — has occurred near us. As with all of…

All-in on R⁶ : Progress [bars] on first post

Mon, 27 Mar 2017 20:30:32 GMT

@eddelbuettel's idea is a good one. (it's a quick read…jump there and come back). We'll avoid confusion and call it R⁶ over here. Feel free to don the superclass. I often wait for a complete example or new package announcement to blog…

Exploring 2017 Retail Store Closings with R

Sun, 19 Mar 2017 16:57:46 GMT

A story about one of the retail chains (J.C. Penny) releasing their list of stores closing in 2017 crossed paths with my Feedly reading list today and jogged my memory that there were a number of chains closing many of their doors this…

Think Twice Before Using Ohmconnect

Sun, 12 Mar 2017 15:26:56 GMT

I listen to @NPR throughout the day (on most days) and a story on Ohmconnect piqued my interest (it aired 5 days prior to this post). The TLDR on Ohmconnect is that it ostensibly helps you save energy by making you more aware of…

Making a Case for case_when

Fri, 10 Mar 2017 15:36:13 GMT

This is a brief (and likely obvious, for some folks) post on the dplyr::casewhen() function. Part of my work\-work is dealing with data from internet scans. When we're performing a deeper inspection of a particular internet protocol or…

Candy Coated Confidence Intervals

Fri, 03 Mar 2017 22:49:44 GMT

@mrshrbrmstr hinted that she would like this post by @RickWicklin translated into R for her stats class. She's quite capable of cranking out the translation of the core component of that post — a call to chisq.test — but she wanted to show…

hrbrthemes 0.1.0 is now on CRAN

Sun, 26 Feb 2017 16:35:43 GMT

I'm pleased to announce the inaugural release of my hrbrthemes (0.1.0) package on CRAN The primary goal of said package is to provide opinionated typographical and other aesthetic defaults for ggplot2 charts. Two core themes are…

On Watering Holes, Trust, Defensible Systems and Data Science Community Security

Thu, 23 Feb 2017 17:16:37 GMT

I've been threatening to do a series on "data science community security" for a while and had cause to issue this inaugural post today. It all started with…

Putting It All Together

Sat, 18 Feb 2017 16:03:42 GMT

The kind folks over at @RStudio gave a nod to my recently CRAN-released epidata package in their January data package roundup and I thought it might be useful to give it one more showcase using the recent CRAN update to ggalt and the new…

ggalt 0.4.0 now on CRAN

Thu, 16 Feb 2017 03:06:22 GMT

I'm uncontainably excited to report that the ggplot2 extension package ggalt is now on CRAN. The absolute best part of this package is the R community members who contributed suggestions and new geoms, stats, annotations and integration…

Spelunking XHRs (XMLHttpRequests) with splashr

Tue, 14 Feb 2017 18:19:29 GMT

splashr has gained some new functionality since the introductory post. First, there's a whole new Docker image for it that embeds a local web server. Why? The main request for it was to enable rendering of…

Geom❤️

Tue, 14 Feb 2017 14:55:48 GMT

Presented without exposition (since it's a silly Geom) This particular ❤️ math pilfered this morning from…

Diving Into Dynamic Website Content with splashr

Thu, 09 Feb 2017 13:27:36 GMT

If you do enough web scraping, you'll eventually hit a wall that the trusty httr verbs (that sit beneath rvest) cannot really overcome: dynamically created content (via javascript) on a site. If the site was nice enough to use XHR requests…

Strung Out On String Ops - A Brief Comparison of stringi and stringr

Mon, 06 Feb 2017 05:11:31 GMT

I made a promise to someone that my next blog would be about stringi vs stringr and I intend to keep said promise. stringr and stringi do "string operations": find, replace, match, extract, convert, transform, etc. The stringr package is…

Exploring News Coverage With newsflash

Thu, 02 Feb 2017 04:54:12 GMT

I was enthused to see a mention of this on the GDELT blog since I've been working on an R package dubbed newsflash to work with the API that the form front-ends. Given the current climate, I feel compelled to note that I'm neither a…

The ? Resistance

Sat, 28 Jan 2017 13:12:51 GMT

I need to be up-front about something: I'm somewhat partially at fault for ? being elected. While I did not vote for him, I could not in any good conscience vote for his Democratic rival. I wrote in a ticket that had one Democrat and one…

One View of the Impact of the New Immigration Ban (+ freeing PDF data with tabulizer)

Thu, 26 Jan 2017 12:37:21 GMT

Dear Leader has made good on his campaign promise to "crack down" on immigration from "dangerous" countries. I wanted to both see one side of the impact of that decree — how many potential immigrants per year might this be impacting — and…

Coconut Pumpkin Pie with Coconut Dark Chocolate Ganache

Mon, 23 Jan 2017 03:48:22 GMT

I'm not about to tell you how to make your fav pie crust as that's a deeply personal subject. You'll need two of them. Mix the above well with a whisk. Put into two prepared pie crusts. Bake at 425°F for 15 minutes. Turn oven to 350°F and…

Create Parquet Files From R Data Frames With sergeant & Apache Drill (a.k.a. Make Parquet Files Great Again in R)

Sun, 22 Jan 2017 14:53:35 GMT

Apache Drill is a nice tool to have in the toolbox as it provides a SQL front-end to a wide array of database and file back-ends and runs in standalone/embedded mode on every modern operating system (i.e. you can get started with or play…

Workout Wednesday Redux (2017 Week 3)

Wed, 18 Jan 2017 13:21:02 GMT

I had started a "52 Vis" initiative back in 2016 to encourage folks to get practice making visualizations since that's the only way to get better at virtually anything. Life got crazy, 52 Vis fell to the wayside and now there are more…

The Devil's in the [Davos] Details — A quick look at this year's WEF Global Risks Report

Mon, 16 Jan 2017 23:08:35 GMT

It's Davos time again. Each year the World Economic Forum (WEF) gathers the global elite together to discuss how they're going to shape our collective future. WEF also releases their annual Global Risks Report at the same time. I read it…

Knit directly to jupyter notebooks from RStudio

Tue, 10 Jan 2017 15:37:49 GMT

Did you know that you can completely replace the "knitting" engine in R Markdown documents? Well, you can! Why would you want to do this? Well, in the case of this post, to commit the unpardonable sin of creating a clunky jupyter notebook…

2017-01 Authored Package Updates

Sun, 08 Jan 2017 19:13:05 GMT

The rest of the month is going to be super-hectic and it's unlikely I'll be able to do any more to help the push to CRAN 10K, so here's a breakdown of CRAN and GitHub new packages & package updates that I felt were worth raising awareness…

The Most Important Commodity in 2017 is Data

Wed, 04 Jan 2017 22:41:05 GMT

Despite being in cybersecurity nigh forever (a career that quickly turns one into a determined skeptic if you're doing your job correctly) I have often trusted various (not to be named) news sources, reports and data sources to provide…

Removing Personal Bias From Flu Severity Estimation (a.k.a. Misery Loves Data)

Mon, 02 Jan 2017 21:25:54 GMT

The family got hit pretty hard with the flu right as the Christmas festivities started and we were all pretty much bed-ridden zombies up until today (2017-01-02). When in the throes of a very bad ILI it's easy to imagine that you're a…

North Carolina's Neighborhood

Fri, 23 Dec 2016 22:14:48 GMT

UPDATE: I'm glad I'm not the only one who was skeptical of this project:…

Pipes (%>%) Everywhere

Thu, 22 Dec 2016 20:09:02 GMT

An R user asked a question regarding whether it's possible to have the RStudio pipe (%>%) shortcut (Cmd-Shift-M) available in other macOS applications. If you're using Alfred then you can use this workflow for said task (IIRC this requires…

sergeant : An R Boot Camp for Apache Drill

Tue, 20 Dec 2016 16:01:17 GMT

I recently mentioned that I've been working on a development version of an Apache Drill R package called sergeant. Here's a lifted "TLDR" on…

Package update: longurl 0.3.0 is hitting CRAN mirrors

Sun, 18 Dec 2016 19:43:42 GMT

The longurl package has been updated to version 0.3.0 as a result of a bug report noting that the URL expansion API it was using went pay-for-use. Since this was the second time a short URL expansion service either went belly-up or had…

Minding the zoo[keeper] with R

Fri, 16 Dec 2016 16:35:27 GMT

I've been drafting a new R package — \\sergeant\\ — to work with \Apache Drill\ and have found it much easier to manage having Drill operating in a single node cluster vs \drill-embedded\ mode (esp when I need to add a couple more nodes…

The Actual, Über, Ultimate Fake News List

Thu, 08 Dec 2016 03:15:07 GMT

It is surprisingly shorter (or longer if you grok regular expressions) than you think it might be: http[s]://.. Like it or not, bias is everywhere and tis woefully insidious. "Mainstream media" \[MSM\]; Alt-left; Alt-right; and, everything…

Survey on Data Science In Two Year Colleges

Tue, 06 Dec 2016 14:30:09 GMT

The ASA (American Statistical Association) has been working in collaboration with the ACM (Association for Computing Machinery) on developing a data science curriculum for Two Year Colleges. Part of this development is the need to…

Interacting With Amazon Athena from R

Mon, 05 Dec 2016 20:51:47 GMT

This is a short post for those looking to test out Amazon Athena with R. Amazon makes Athena available via JDBC, so you can use RJDBC to query data. All you need is their JAR file and some setup information. Here's how to get the JAR file…

Jumping on the "Banned" Wagon

Fri, 02 Dec 2016 20:32:38 GMT

\>\The parent proposed a committee made up of parents and teachers of different cultural backgrounds come up with a list of books that are inclusive for all students.\ Where's the \ACLU\? They'd be right there if this was an alt-right'er…

The Most Important Hashtag for 2017

Wed, 23 Nov 2016 13:39:54 GMT

When something pops up in the news, on Twitter, Facebook or even (ugh) Gab claiming or teasing findings based on data I believe it's more important than ever to reply with some polite text and a \#showmethedata\ hashtag. We desperately…

The Devil is in the Details

Tue, 22 Nov 2016 12:18:46 GMT

The \first public informational video\ from the PEOTUS didn't add a full transcript of the video to the web site and did not provide (at least as of 0700 EST on 2016-11-22) their own text annotations/captions to the video. Google's…

This is not cool.

Mon, 21 Nov 2016 16:47:03 GMT

First it was OpenDNS selling their souls (and, \[y\]our data) to Cisco (whom I don't trust at all with my data). Now, it's Dyn — — doing something even worse (purely my own opinion). I'm currently evaluating offerings by \FoolDNS\ &…

Dis-ingenuous

Fri, 18 Nov 2016 02:25:51 GMT

\Bulbs\. If those were real, functional bulbs that were destroyed…spreading real, irreclaimable refuse…all to shill a far less than revolutionary "professional" laptop…then, just how "enlightened" is Apple, really? But, I guess it's fine…

Stop.

Thu, 17 Nov 2016 15:17:14 GMT

\>\Stop making people suicidal. Stop telling people they’re going to be killed. Stop terrifying children. Stop giving racism free advertising. Stop trying to convince Americans that all the other Americans hate them. Stop. Stop. Stop.\…

Counting [U.S.] Expatriation with R (a.k.a. a Decade of Desertion)

Mon, 08 Aug 2016 22:22:35 GMT

2016-08-13 UPDATE: Fortune has a story on this and it does seem to be tax-related vs ideology. @thosjleeper suggested something similar as well about a week…

QuickLookR - A macOS QuickLook plugin for R Data files

Sat, 06 Aug 2016 14:47:50 GMT

I had tried to convert my data-saving workflows to \\feather\\ but there have been \issues\ with it supporting large files (that seem to be near resolution), so I've been continuing to use R Data files for local saving of processed/cleaned…

When life imitates Fullmetal Alchemist

Thu, 04 Aug 2016 14:36:07 GMT

The NIH is \moving forward\ with plans to financially support & encourage human-animal chimera research. You can find more info over at the \NIH blog\. Chimera's have been a longstanding subject of science-fiction/fantasy and many authors…

New codewords for "age & family discrimination"

Tue, 02 Aug 2016 10:51:31 GMT

From: \Why Corporate America Is Leaving the Suburbs for the City\: >\We wanted energy, vibrancy and diversity, and to accelerate a change in our culture by moving downtown.\ translation: >\We want to begin a process of strategically…

U.S. Drought Animations with the "Witch's Brew" (purrr + broom + magick)

Wed, 27 Jul 2016 13:08:54 GMT

This is another purrr\-focused post but it's also an homage to the nascent magick package (R interface to ImageMagick) by @opencpu. We're starting to see/feel the impact of the increasing drought up here in southern Maine. I've used the…

Use quick formula functions in purrr::map (+ base vs tidtyverse idiom comparisons/examples)

Tue, 26 Jul 2016 16:33:24 GMT

I've converted the vast majority of my apply usage over to purrr functions. In an attempt to make this a quick post, I'll refrain from going into all the benefits of the purrr package. Instead, I'll show just one thing that's super…

Mid-year R Packages Update Summary

Sun, 24 Jul 2016 14:24:49 GMT

I been updating some existing packages and github-releasing new ones (before a CRAN push). Most are "cyber"-related, but there are some general purpose ones. Here's a quick…

Slaying CIDR Orcs with Triebeard (a.k.a. fast trie-based 'IPv4-in-CIDR' lookups in R)

Tue, 12 Jul 2016 20:46:06 GMT

The insanely productive elf-lord, @quominus put together a small package (\\triebeard\\) that exposes an API for \radix/prefix tries\ at both the R and Rcpp levels. I know he had some personal needs for this and we both kinda need these to…

CRAN Packages on GitHub (and some CRAN DESCRIPTION observations)

Sun, 10 Jul 2016 16:06:04 GMT

Just about a week ago @thosjleeper posited something on twitter w/r/t how many CRAN packages had associations with GitHub (i.e. how many used GitHub for development). The \DESCRIPTION\ file (that comes with all R packages) has some fields…

Bridging The Political [Polygons] Gap with ggplot2

Thu, 07 Jul 2016 20:27:25 GMT

The @pewresearch folks have been collecting political survey data for quite a while, and I noticed the \visualization below\ referenced in a \Tableau vis contest entry\: Those are filled \frequency polygons\, which are super-easy to…

A Simple Prediction Web Service Using the New fiery Package

Tue, 05 Jul 2016 12:15:06 GMT

\\fiery\\ is a new \Rook\/\httuv\-based R web server in town created by @thomasp85 that aims to fill the gap between raw http & websockets and Shiny with a flexible framework for handling requests and serving up responses. The intent of…

Making "Time Rivers" in R

Tue, 28 Jun 2016 21:18:08 GMT

Once again, @albertocairo notices an interesting chart and spurs pondering in the visualization community with \his post\ covering an unusual "vertical time series" chart produced for the print version of the NYTimes: I'm actually less…

A Call to Arms[list] Data Analysis!

Mon, 20 Jun 2016 00:53:49 GMT

The NPR vis team contributed to a recent \story\ about Armslist, a "craigslist for guns". Now, I'm neither pro-"gun" or anti-"gun" since this subject, like most heated ones, has more than two sides. What I \am\ is pro-\data\, and the U.S.…

Your data vis "Spidey-sense" & the need for a robust "utility belt"

Thu, 16 Jun 2016 13:01:11 GMT

@theboysmithy did a \great piece\ on coming up with an alternate view for a timeline for an FT piece. Here's an excerpt (read the whole piece, though, it's worth…

On Whether Y-axis Labels Are Always Necessary

Sun, 12 Jun 2016 15:28:36 GMT

The infamous @albertocairo \blogged about\ a \nice interactive piece on German company tax avoidance\ by @ProPublica. Here's a snapshot of their interactive chart: !\\ Dr. Cairo (his PhD is in the bag as far as I'm concerned :-) posited:…

New viridis & colorbrewer palettes for ipv4-heatmap

Tue, 07 Jun 2016 19:59:53 GMT

It's no seekrit that I :heart: Hilbert curve heatmaps of IPv4 space. Real-world IPv4 maps (i.e. the ones that drop dots on the Earth) have little utility, but with Hilbert curves maps of IPv4 space many different topologies can be…

Global Temperature Change in R & D3 (without the vertigo)

Sun, 15 May 2016 00:49:49 GMT

URL <- "http://www.metoffice.gov.uk/hadobs/hadcrut4/data/current/timeseries/HadCRUT.4.4.0.0.monthlynsavg.txt" fil <- sprintf("data/%s", basename(URL)) if (!file.exists(fil)) download.file(URL,…

New #rstats Podcast - R World News

Tue, 10 May 2016 10:31:05 GMT

Keeping up with R-related news on Twitter, GitHub, CRAN & even R-Bloggers (et al) can be an all-encompassing task that may be fun, but doesn't always make it easy to get work done. There is so much going on in the R community that we…

Pining for the fjoRds & monitoring SSL/TLS certificate expiration in R with flexdashboard

Mon, 02 May 2016 00:31:46 GMT

Rumors of my demise have been (almost) greatly exaggerated. Folks have probably noticed that #52Vis has stalled, as has most blogging, package & Twitter activity. I came down with a nasty bout of bronchitis after attending rOpenSci Unconf…

(ggplot2) Exercising with (ggalt) dumbbells

Sun, 17 Apr 2016 14:43:11 GMT

I follow the most excellent Pew Research folks on Twitter to stay in tune with what's happening (statistically speaking) with the world. Today, they tweeted this excerpt from their 2015 Global Attitudes…

52Vis Week #3 - Waste Not, Want Not

Thu, 14 Apr 2016 03:35:44 GMT

The Wall Street Journal did a project piece \a while back\ in the \"Waste Lands: America's Forgotten Nuclear Legacy"\. They dug through \Department of Energy\ and \CDC\ data to provide an overview of the lingering residue of this toxic…

52 Vis Week #2 Wrap Up

Thu, 14 Apr 2016 03:07:24 GMT

I've been staring at this homeless data set for a few weeks now since I'm using it both here and in the data science class I'm teaching. It's been one of the most mindful data sets I've worked with in a while. Even when reduced to pure…

52 Vis Week 1 Winners!

Wed, 13 Apr 2016 20:59:00 GMT

The response to 52Vis has exceeded expectations and there have been great entries for both weeks. It's time to award some prizes! ### Week 1 - Send in the Drones I'll take \this week\ in comment submission order (remember, the rules…

Beating lollipops into dumbbells

Tue, 12 Apr 2016 23:56:48 GMT

Shortly after I added lollipop charts to ggalt I had a few requests for a dumbbell geom. It wasn't difficult to do modify the underlying lollipop Geoms to make a geomdumbbell(). Here it is in…

Clandestine DNS lookups with gdns

Mon, 11 Apr 2016 04:25:58 GMT

Google recently \announced\ their DNS-over-HTTPS API, which \"enhances privacy and security between a client and a recursive resolver, and complements DNSSEC to provide end-to-end authenticated DNS lookups"\. The REST API they provided was…

geom_lollipop() by the Chartettes

Thu, 07 Apr 2016 22:19:22 GMT

\>UPDATE: Changed code to reflect the new \horizontal\ parameter for \geom\lollipop()\ I make a fair share of bar charts throughout the day and really like switching to lollipop charts to mix things up a bit and enhance the visual appeal.…

52Vis Week 2 (2016 Week #14) - Honing in on the Homeless

Thu, 07 Apr 2016 00:43:29 GMT

\>UPDATE: Since I put in a "pull request" requirement, I intended to put in a link to getting started with GitHub. Dr. Jenny Bryan's @stat545 has a great \section on git\ that should hopefully make it a bit less painful. ### Why 52Vis? In…

iptools 0.4.0 released into the wild (i.e. is hitting the CRAN mirrors today)

Mon, 04 Apr 2016 19:05:20 GMT

The \\iptools\ package\—a toolkit for manipulating, validating and testing IP addresses and ranges, along with datasets relating to IP addresses—is flying through the internets and hitting a CRAN mirror near you, soon. ### What's fixed?…

Introducing a Weekly R / Python / JS / etc Vis Challenge!

Wed, 30 Mar 2016 13:37:40 GMT

\>UPDATE: Deadline is now 2016-04-05 23:59 EDT; next vis challenge is 2016-04-06! Per a suggestion, I'm going to try to find a neat data set (prbly one from @jsvine) to feature each week and toss up some sample code (99% of the time prbly…

Easier Composite U.S. Choropleths with albersusa

Tue, 29 Mar 2016 12:14:29 GMT

Folks who've been tracking this blog on R-bloggers probably remember \this post\ where I showed how to create a composite U.S. map with an Albers projection (which is commonly referred to as AlbersUSA these days thanks to D3). I'm not sure…

Nuclear Animations in D3

Mon, 28 Mar 2016 02:30:33 GMT

As I said, I'm kinda obsessed with the "nuclear" data set. So much so that I made a D3 version that's similar to the R version I made the other day. I tried not to code much today (too much Easter fun going on), so I left off the size &…

Nuclear Animations in R

Sat, 26 Mar 2016 16:17:18 GMT

\> UPDATE: For #rstats folks interested in what this might look like in D3, \take a gander\ @jsvine (Data Editor at BuzzFeed) cleaned up and posted a \data sets of historical nuclear explosions\ earlier this week. I used it to show a few…

Using ProPublica "statefaces" in ggplot2

Sat, 19 Mar 2016 10:46:39 GMT

I'm a huge fan of ProPublica. They have a super-savvy tech team, great reporters, awesome graphics folks and excel at data-driven journalism. Plus, they give away virtually everything, including data, text, graphics & tools. I was reading…

Stacking the deck against treemaps

Fri, 18 Mar 2016 17:28:09 GMT

In real life, I'd add a DataTables interactive table with this to let folks explore a bit…

Hy-phen-ate All The Things! (in R)

Fri, 18 Mar 2016 10:36:40 GMT

\hyphenatr\--what may be my smallest package ever--has just hit \CRAN\. It, well, \hyphenates\ words using \\libhyphen\\ (a.k.a. \libhnj\). There are no external dependencies (i.e. no \brew install\, \apt get\, et. al. required) and it…

Supreme Annotations

Wed, 16 Mar 2016 11:47:02 GMT

I had not intended this to be "Annotation Week" but there was a large, positive response to my annotation "hack" post. This reaction surprised me, then someone pointed me to this link and also noted that if having to do subtitles via hacks…

Spinning Cycles in Box #4 To Take the Pies out of Pi Day

Mon, 14 Mar 2016 14:43:06 GMT

\>UPDATE: time spent per task factor order was wrong before. now fixed. I caught this tweet…

ggplot2で字幕
[Subtitles in ggplot2]

Sat, 12 Mar 2016 21:09:47 GMT

Subtitles aren't always necessary for plots, but I began to use them enough that I whipped up a function for ggplot2 that does a decent job adding a subtitle to a finished plot object. More than a few folks have tried their hand at this in…

Some Light Image Processing & Creation With R

Thu, 10 Mar 2016 23:28:30 GMT

\>UPDATE: I rejiggered the function to actually now, y'know, do what it says it should do :-) A friend, we'll call him \Alen\ put a call out for some function that could take an image and produce a per-row "histogram" along the edge for…

OS X XQuartz Vulnerability Test Using R

Mon, 07 Mar 2016 15:59:02 GMT

It's usually a good thing when my #rstats and infosec worlds collide. Unfortunately, this time it's a script that R folk running on OS X can use to see if they are using a version of XQuartz that has a nasty vulnerability in the framework…

Primary Plotting

Mon, 07 Mar 2016 14:02:41 GMT

My wife tricked me into a partial-weekend project to try to get all the primary/caucus results to-date on a map (the whole us). This is challenging since not all states use counties as boundaries for aggregate results. I'm still piecing…

Capturing wild widgets with webshot

Fri, 04 Mar 2016 16:58:36 GMT

NOTE: you won't need to use this function if you use the \development version\ of…

A Tale of Two Charting Paradigms: Vega-Lite vs R+ggplot2

Sun, 28 Feb 2016 17:05:15 GMT

This post comes hot off the heels of the \nigh-feature-complete release of \vegalite\\ (virtually all the components of Vega-Lite are now implemented and just need real-world user testing). I've had a few and seen a few questions about…

Create Vega-Lite specs & widgets with the vegalite package

Sat, 27 Feb 2016 16:18:17 GMT

\Vega-Lite\ 1.0 was \released this past week\. I had been meaning to play with it for a while but I've been burned before by working with unstable APIs and was waiting for this to bake to a stable release. Thankfully, there were no new…

Working with the Clinton State Dept Email Dumps in R (Part 1: Graphs)

Sat, 20 Feb 2016 17:06:02 GMT

I put this together after experimenting with \ggplot2\ and \ggnetwork\ earlier this week. The changes I made added \svgPanZoom\ into the mix. Consequently, it has a widget in it, so it was just easier to embed the full R markdown HTML into…

Thai Curry Puffs (Karipap • กะหรี่ปั๊บ)

Mon, 15 Feb 2016 13:19:26 GMT

I made Thai curry puffs for @mrshrbrmstr for Valentine's Day (paired with Thai crispy duck red curry) and was asked to post the recipe. There are two versions, one with lard pie dough (#3, #4 & I are dairy sensitive) and one with puff…

Making Faceted Heatmaps with ggplot2

Sun, 14 Feb 2016 13:17:25 GMT

We were doing some exploratory data analysis on some attacker data at work and one of the things I was interested is what were "working hours" by country. Now, I don't put a great deal of faith in the precision of geolocated IP addresses…

Dairy-free Parker House Rolls

Fri, 12 Feb 2016 15:56:12 GMT

Here's a quick/easy recipe for soft roll dough that you can use to make dairy-free Parker House rolls, knot rolls or just plain rolls. NOTE: It makes ~3lbs of dough. I freeze it to save time in the work-weeks. Also, weigh your ingredients.…

Plot the new SVG R logo with ggplot2

Thu, 11 Feb 2016 12:14:59 GMT

High resolution and SVG versions of the new R logo are finally available. I converted the SVG to WKT (file here) which means we can use it like we would a shapefile in R. That includes plotting! Here's a short example of how to read that…

Craft httr calls cleverly with curlconverter

Wed, 10 Feb 2016 17:11:02 GMT

UPDATE curlconverter will now return (as the function return value) a working R function. See the README for…

Alternate R Markdown Templates

Fri, 05 Feb 2016 02:27:51 GMT

The \knitr\/R markdown system is a great way to organize reports and analyses. However, the built-in ones (that come with RStudio/the \rmarkdown\ package) rely on Bootstrap and also use jQuery. There's nothing wrong with that, but the…

Cobble XPath Interactively with the xmlview Package

Wed, 13 Jan 2016 17:35:23 GMT

(If you don't know what XML is, you should probably \read a primer\ before reading this post,) When working with data, one inevitably comes across things encoded in XML. I'm in the "anti-XML" camp, but deal with my fair share of XML in…

The Force: Accounted, in R

Sat, 09 Jan 2016 15:47:30 GMT

Despite being a cybersecurity professional, it's pretty easy to social engineer…

iptools 0.3.0 ("Violet Packet") Now on CRAN with Windows Support!

Fri, 08 Jan 2016 15:44:07 GMT

\iptools\ is a set of tools for working with IP addresses. Not just work, but work \fast\. It's backed by \Rcpp\ and now uses the \AsioHeaders\ package by Dirk Eddelbuettel, which means it no longer needs to \link\ against the monolithic…

Paths To Discovery: Mapping Your Home Network

Thu, 07 Jan 2016 16:08:28 GMT

Gone are the days when one had a single computer plugged directly into a modem (cable, DSL or good ol' Hayes). Even the days when there were just one or two computers connected via wires or invisible multi-gigahertz waves passing through…

Zellingenach: A visual exploration of the spatial patterns in the endings of German town and village names in R

Sun, 03 Jan 2016 18:18:08 GMT

Moritz Stefaner started off 2016 with a \very spiffy post\ on \"a visual exploration of the spatial patterns in the endings of German town and village names"\. Moritz was \exploring some new data processing & visualization tools\ for the…

World Map Panel Plots with ggplot2 2.0 & ggalt

Mon, 28 Dec 2015 19:57:44 GMT

James Austin (@awhstin) made some #spiffy 4-panel maps with base R graphics but also posited he didn't use ggplot2…

Update to metricsgraphics 0.9.0 (now on CRAN)

Mon, 21 Dec 2015 16:42:13 GMT

It's been a while since I've updated my \metricsgraphics package\. The hit list for changes includes: - Fixes for the new ggplot2 release (metricsgraphics uses the \movies\ data set which is now in ggplot2movies) - Updated all javascript…

Fear of WaPo Using Bad Pie Charts Has Increased Since Last Year

Sun, 13 Dec 2015 14:40:01 GMT

I woke up this morning to a \headline story from the Washington Post\ on \"Americans are twice as willing to distance Christian extremists from their religion as Muslims\". This post is not about the content of the headline or story. It…

Moving the Shield to a new home

Mon, 23 Nov 2015 21:47:38 GMT

This week (Thanksgiving-ish, 2015) I start a new adventure at Rapid7! You can read more about why over at \the official announcement\. Rapid7 has an amazing amount of data, a world-class team and a determination to deliver cutting-edge…

An OS X R Task Runner for—and a Mini-R-centric review of—Microsoft's Visual Studio Code Editor

Sat, 21 Nov 2015 13:58:25 GMT

Microsoft's newfound desire to make themselves desirable to the hipster development community has caused them to make many things \open\ and/or free of late. One of these manifestations is \Visual Studio Code\, an \Atom\-ish editor for us…

Beware of sideloading f.lux for iOS from files obtained on sharing/torrent sites

Fri, 13 Nov 2015 16:13:44 GMT

as a result \DO NOT SIDE LOAD THE APP\. It means the good/safe f.lux source code/project has been modified by someone since November 11th, 2015 and that you are putting your device (i.e. the confidentiality, integrity and availability of…

Using MonetDB[Lite] with real-world CSV files

Wed, 11 Nov 2015 14:39:35 GMT

\MonetDBLite\ (for R) was announced/released today and, while the examples they provide are compelling there's a "gotcha" for potential new folks using SQL in general and SQL + MonetDB + R together. The toy example on the site shows…

Visualizing Survey Data : Comparison Between Observations

Sun, 08 Nov 2015 17:46:56 GMT

Cybersecurity is a domain that really likes surveys, or at the very least it has many folks within it that like to conduct and report on surveys. One recent survey on threat intelligence is in it's second year, so it sets about comparing…

An Ephemeral Update to daylight()

Mon, 02 Nov 2015 13:45:14 GMT

This occurrence of the bi-annual corruption of the space-time continuum (i.e. changing to/from standard/daylight time) in the U.S. caused me to make a slight change to the code \from an older post\. The \daylight()\ function now…

Less Drama, More Encoding

Sat, 24 Oct 2015 14:43:46 GMT

Junk Charts \adeptly noted and fixed\ this excessively stylized chart from the WSJ this week: Their take on it does reduce the ZOMGOSH WE ARE DOOMED! look and feel of the WSJ chart: But, we can further reduce the drama by using a more…

Installing R on OS X - "100% Homebrew Edition"

Thu, 22 Oct 2015 12:29:17 GMT

In a previous post I provided "mouse-heavy" instructions for getting R running on your Mac. A few of the comments suggested that an "all Homebrew" solution may be preferable for some folks. Now, there are issues with this since getting…

Installing R on OS X

Tue, 20 Oct 2015 18:35:50 GMT

NOTE: The comments are a must read for this. Some excellent additional advice and "gotchas" by some super-helpful…

Replicating NatGeo's "Proper" Earthquake Map in R

Mon, 05 Oct 2015 00:01:20 GMT

I saw this post over at NatGeo over the weekend and felt compelled to replicate this: with ggplot2. Three shapefiles later and we have it close enough to toss into a post (and I really don't believe the continent names are…

For Some Definition of "Directly" and/or "Contort"

Wed, 30 Sep 2015 13:21:34 GMT

Junk Charts did a post on \Don't pick your tool before having your design\ and made a claim that this: \"cannot be produced directly from a tool (without contorting your body in various painful locations)"\. I beg to differ. With R &…

A Package Full o' Pirates & Makin' Interactive Pirate Maps in arrrrrRstats

Sat, 19 Sep 2015 18:34:10 GMT

Avast, me hearties! It's time four t' annual International Talk Like a Pirate Day #rstats post! (OK, I won't make you suffer continuous pirate-speak for the entire post) I tried to be a bit more practical this year and have two treasuRe…

A Better Way To Read Nest Data Into R

Thu, 10 Sep 2015 13:05:06 GMT

I glimpsed a \post\ in the RSS feeds today on how to connect Nest data with a Shiny dashboard and was compelled to post a less brute-force way to get data from the Nest API. The authors of the Shiny+Nest post used \system\ calls to \curl\…

Roll Your Own Stats and Geoms in ggplot2 (Part 1: Splines!)

Tue, 08 Sep 2015 23:06:39 GMT

A huge change is coming to ggplot2 and you can get a preview of it over at Hadley's github repo. I've been keenly interested in this as I will be fixing, finishing & porting coord\proj to it once it's done. Hadley & Winston have re-built…

Three New #rstats Twitter Bots To Follow

Sun, 06 Sep 2015 11:30:32 GMT

I engage with the Stack\[Overflow|Exchange\] community quite a bit and was super-happy @treycausey made the \Stack Overflow #rstats bot\ (@StackOverflowR) since I'm also on Twitter alot (mostly hanging out in #rstats these days). However,…

Coloring (and Drawing) Outside the Lines in ggplot

Thu, 27 Aug 2015 13:36:08 GMT

Time for another Twitter-inspired blog post this week, this time from a tweet by…

New Pacakge "docxtractr" - Easily Extract Tables From Microsoft Word Docs

Mon, 24 Aug 2015 16:51:05 GMT

UPDATE: \docxtractr\ is now \on CRAN\ --------------------- This is more of a follow-up from \yesterday's post\. The hack and function in said post was fine, but it was limited to uniform tables and made you do more work than you had to.…

Using R To Get Data Out Of Word Docs

Sun, 23 Aug 2015 13:14:04 GMT

NOTE: after reading this post head on over to this new one as it has wrapped this functionality (and more!) into a package. Also: docxtractr is now on…

Doh! I Could Have Had Just Used V8!

Fri, 21 Aug 2015 08:32:50 GMT

An R user recently had the need to split a "full, human name" into component parts to retrieve first & last names. The full names could be anything from something simple like \"David Regan"\ to more complex & diverse such as \"John Smith…

Track Hurricane Danny (Interactively) with R + leaflet

Thu, 20 Aug 2015 15:41:50 GMT

Why aren't we using \read.csv\ or \read.table\ directly, you ask? Well, the data is in a \really\ ugly format thanks to the spaces in the \STATUS\ column and two prefix…

cdcfluview - On The Way to "CRAN 7K"

Sun, 09 Aug 2015 16:37:11 GMT

I like to turn coincidence into convergence whenever possible. This weekend, a user of \cdcfluview\ had a question that caused me to notice a difference in behaviour between the package was interacting with CDC FluView API, so I updated…

Adding a CRAN Search Engine to Chrome

Fri, 07 Aug 2015 11:14:58 GMT

Riffing off of \the previous post\, here's a way to quickly search CRAN (the @RStudio flavor) from the Chrome search bar. - Paste \chrome://settings/searchEngines\ into your location bar and hit return/enter - Scroll down until the input…

Speeding Up Your Quest(s) For "R Stuff"

Thu, 06 Aug 2015 00:25:04 GMT

I use Google quite a bit when conjuring up R projects, whether it be in a lazy pursuit of a PDF vignette or to find a package or function to fit a niche need. Inevitably, I'll do something like \this\ (yeah, I'm still on a mapping kick)…

Achieve Charting Zen With TauCharts

Tue, 04 Aug 2015 21:28:22 GMT

There was some chatter on the twitters this week about a relatively new D3-based charting library called \TauCharts\ (also @taucharts). The API looked pretty clean and robust, so I started working on an htmlwidget for it and was quickly…

Two New R Packages - qrencoder & passwordrandom

Sun, 02 Aug 2015 21:20:02 GMT

Believe it or not, there are two \[1\] \[2\] questions on @StackOverflowR about how to make QR codes in R. I personally think QR codes are kinda hokey, but who am I to argue with pressing needs of the #rstats community? I found libqrencode…

Introducing the nominatim geocoding package

Wed, 29 Jul 2015 13:20:31 GMT

In the never-ending battle for truth, justice and publishing more R packages than \Oliver\, I whipped out an R package for the \OpenStreetMap Nominatim API\. It actually hits the \MapQuest Nominatim Servers\ for most of the calls, but the…

Making Static/Interactive Voronoi Map Layers In ggplot/leaflet

Sun, 26 Jul 2015 15:33:20 GMT

Despite having shown various ways to overcome D3 cartographic envy, there are always more examples that can cause the green monster to rear it's ugly head. Take the Voronoi Arc Map…

Roll Your Own Gist Comments Notifier in R

Sat, 25 Jul 2015 13:49:19 GMT

As I was putting together the \coord\proj\ ggplot2 extension I had posted a \gist\ that I shared on Twitter. Said gist received a comment (several, in fact) and a bunch of us were painfully reminded of the fact that there is no built-in…

A Path Towards Easier Map Projection Machinations with ggplot2

Fri, 24 Jul 2015 18:50:24 GMT

The $DAYJOB doesn't afford much opportunity to work with cartographic datasets, but I really like maps and tinker with shapefiles and geo-data when I can, plus answer a ton of geo-questions on StackOverflow. R makes it easy—one might even…

Using the new 'viridis' colormap in R (thanks to Simon Garnier)

Mon, 20 Jul 2015 07:11:52 GMT

Nathaniel Smith and Stéfan van der Walt presented a new colormap (for Python) at SciPy 2015 called viridis. From the…

R Package to access the Open Movie Database (OMDB) API

Fri, 10 Jul 2015 20:03:13 GMT

It's not on CRAN yet, but there's a devtools\-installable R package for getting data from the OMDB API. It covers all of the public API…

Faceted "World Population by Income" Choropleths in ggplot

Thu, 09 Jul 2015 18:25:18 GMT

Poynter did a nice interactive piece on world population by income (i.e. "How Many Live on How Much, and Where"). I'm always on the lookout for optimized shapefiles and clean data (I'm teaching a data science certificate program starting…

Two pending features to metricsgraphics

Mon, 06 Jul 2015 02:47:35 GMT

I've been slowly prodding the \metricsgraphics package\ towards a 1.0.0 release, but there are some rough edges that still need sorting out. One of them is the ability to handle passing in variables for the \x\ & \y\ accessor values (you…

Get by with a little (R) help from your friends (at GitHub)

Mon, 29 Jun 2015 14:13:27 GMT

@JennyBryan posted her \slides from the 2015 R Summit\ and they are a must-read for instructors and even general stats/R-folk. She's one of the foremost experts in R+GitHub and her personal and class workflows provide solid patterns worth…

DO Something Nifffty with R

Fri, 19 Jun 2015 22:07:08 GMT

@briandconnelly (of \pushoverr\ fame) made a super-cool post about \connecting R\ to @IFTTT via IFTTT's "Maker" channel. The IFTTT Maker interface to receive events is fairly straightforward and Brian's code worked flawlessly, so it was…

metricsgraphics 0.8.5 is now on CRAN!

Mon, 15 Jun 2015 13:33:12 GMT

I'm super-pleased to announce that the Benevolent CRAN Overlords \accepted the metricsgraphics package\ into CRAN over the weekend. Now, you no longer need to rely on github/devtools to use \MetricsGraphics.js\ charts from your R scripts.…

Animated US Hexbin Map of the Avian Flu Outbreak

Sun, 07 Jun 2015 20:59:42 GMT

The recent announcement of the \start of egg rationing in the U.S.\ made me curious enough about the avian flu outbreak to try to dig into the numbers a bit. I finally stumbled upon a \USDA site\ that had an embedded HTML table of flock…

A quick, incomplete comparison of ggplot2 & rbokeh plotting idioms

Tue, 26 May 2015 21:04:44 GMT

I set aside a small bit of time to give \rbokeh\ a try and figured I'd share a small bit of code that shows how to make the "same" chart in both ggplot2 and rbokeh. #### What is Bokeh/rbokeh? rbokeh is an \htmlwidget\ wrapper for the…

Scraping jQuery DataTable Programmatic JSON with R

Mon, 18 May 2015 10:35:22 GMT

School of Data had a recent post how to copy "every item" from a multi-page list. While their post did provide a neat hack, their "words of warning" are definitely missing some items and the overall methodology can be improved upon with…

U.S. Drought Monitoring With Hexbin State Maps in R

Fri, 15 May 2015 11:18:28 GMT

On the news, today, of the early stages of drought hitting the U.S. northeast states I decided to springboard off of yesterday's post and show a more practical use of hexbin state maps than the built-in (and still purpose unknown to me)…

GeoJSON Hexagonal "Statebins" in R

Thu, 14 May 2015 18:29:50 GMT

There's been lots of buzz about "statebin" maps of late. A recent tweet by @andrewxhill referencing work by @dannydb pointed to a nice shapefile (alternate link) that ends up being a really great way to handle statebin maps (and I feel…

quotebox - An NPR-like Embedded Twitter Quote Generator

Sat, 09 May 2015 11:17:55 GMT

I'm an avid NPR listener also follow a number of their programs and people on Twitter. I really dig their \quotable\ tweets. Here's a sample of a recent…

New Fake Vulnerability Service - SVE

Thu, 16 Apr 2015 17:17:47 GMT

The SVE service is a fake new service solely put out there to facilitate some inane puzzle. You can reference SVE-2015-9999 via this URL and hopefully this will make it possible for google to get it back in their…

More Airline Crashes via the Hadleyverse

Wed, 01 Apr 2015 00:36:08 GMT

I saw a fly-by \#rstats\ mention of more airplane accident data on -- of all places -- LinkedIn (email) today which took me to a \GitHub repo\ by @philjette. It seems there's a \web site\ (run by what seems to be a single human) that…

A look at airline crashes in R with googlesheets, dplyr & ggplot2

Mon, 30 Mar 2015 18:32:08 GMT

Over on The DO Loop, @RickWicklin does a nice job \visualizing the causes of airline crashes\ in SAS using a mosaic plot. More often than not, I find mosaic plots can be a bit difficult to grok, but Rick's use was spot on and I believe it…

Pre-CRAN waffle update - isotype pictograms

Thu, 26 Mar 2015 12:23:04 GMT

It seems Ruben C. Arslan had the waffle idea about the same time I did. Apart from some extra spiffy XKCD-like styling, one other thing his waffling routines allowed for was using FontAwesome icons. When you use an icon vs a block, you are…

Fixing Colors & Proportions in Jerusalem Post Election Graphics

Sat, 21 Mar 2015 00:13:41 GMT

Vis expert Naomi Robbins did an excellent \critique\ of the \graphics\ that went along with an article on Israeli election in the Jerusalem Post. Non-uniform and color-blind-unfriendly categorical colors and disproportionate arc sizes are…

Making waffle charts in R (with the new 'waffle' package)

Wed, 18 Mar 2015 19:44:25 GMT

NOTE: The waffle package (sans JavaScript-y goodness) is up on CRAN so you can do an install.packages("waffle") and library(waffle) vs the devtools…

Simple Lower US 48 Albers Maps & Local (no-API) City/State Geocoding in R

Sun, 15 Mar 2015 12:22:10 GMT

I've been seeing an uptick in static US "lower 48" maps with "meh" projections this year, possibly caused by a flood of new folks resolving to learn R but using pretty old documentation or tutorials. I've also been seeing an uptick in…

Streamgraph htmlwidget version 0.7 released (adds support for markers & annotations)

Fri, 13 Mar 2015 00:45:10 GMT

In preparation for using some of our streamgraphs for production (PDF/print) graphics, I ended up having to hand-edit labels in on one of the graphics in an Adobe product. This bumped up the priority on adding annotation functions to the…

New R Package - ipapi (IP/Domain Geolocation)

Mon, 09 Mar 2015 20:32:20 GMT

I noticed that the @rOpenSci folks had an interface to \ip-api.com\ on their \ToDo\ list so I whipped up a small R package to fill said gap. Their IP Geolocation API will take an IPv4, IPv6 or FQDN and kick back a ASN, lat/lon, address and…

Streamgraph package now supports continuous x axis scale

Sat, 07 Mar 2015 18:08:42 GMT

A post on \StackOverflow\ asked about using a continuous variable for the x-axis (vs dates) in my \streamgraph package\. While I provided a workaround for the question, it helped me bump up the priority for adding support for continuous x…

Introducing the streamgraph htmlwidget R Package

Sun, 15 Feb 2015 16:54:37 GMT

We were looking for a different type of visualization for a project at work this past week and my thoughts immediately gravitated towards \streamgraphs\. The TLDR on streamgraphs is they they are generalized versions of stacked area graphs…

SweetheaRstats

Fri, 13 Feb 2015 00:01:25 GMT

I felt compelled to dust off my 2013 \Valentine's Day\ \#rstats\ post and make it all Shiny and new again. I used the same math from that post, but made the polygon a bit sharper and used ggplot2 for the plotting. To kick it up a bit, I…

A Step to the Right in R Assignments

Wed, 04 Feb 2015 20:27:20 GMT

I received an out-of-band question on the use of \%<>%\ in my \CDC FluView\ post, and took the opportunity to address it in a broader, public fashion. Anyone using R knows that the two most common methods of assignment are the venerable…

New release (0.7) of metricsgraphics htmlwidget — grids & rollovers

Sun, 01 Feb 2015 15:34:18 GMT

I've updated my \metricsgraphics\ package to version \0.7\. The core \MetricsGraphics\ JavaScript library has been updated to version 2.1.0 (from 1.1.0). Two blog-worthy features since releasing version 0.5 are \mjs\grid\ (which is a…

NASA GISS’s Annual Global Temperature Anomaly Trends (dplyr/ggplot version)

Sun, 18 Jan 2015 14:50:05 GMT

D Kelly O’Day did a \great post\ on charting NASA’s Goddard Institute for Space Studies (GISS) temperature anomaly data, but it sticks with base R for data munging & plotting. While there's absolutely nothing wrong with base R operations,…

New R Package: cdcfluview — Retrieve Flu Data from CDC's FluView Portal

Sun, 11 Jan 2015 02:12:26 GMT

\\NOTE\\ If there's a particular data set from http://www.cdc.gov/flu/weekly/fluviewinteractive.htm that you want and that isn't in the pacakge, please file it as an issue and be as specific as you can (screen shot if possible). -----…

New R Package: metricsgraphics

Thu, 08 Jan 2015 21:50:52 GMT

Mozilla released the \MetricsGraphics.js library\ back in November of 2014 (\gh repo\) and was greeted with great fanfare. It's primary focus is on crisp, clean layouts for interactive time-series data, but they have support for other…

Making Static & Interactive Maps With ggvis (+ using ggvis maps w/shiny)

Mon, 29 Dec 2014 14:35:33 GMT

Even though it's still at version \0.4\, the \ggvis\ package has quite a bit of functionality and is highly useful for exploratory data analysis (EDA). I wanted to see how geographical visualizations would work under it, so I put together…

Power Outage Impact Choropleths In 5 Steps in R (featuring rvest & RStudio "Projects")

Thu, 27 Nov 2014 15:17:49 GMT

I and @awpiii were trading news about the power outages in Maine & New Hampshire last night and he tweeted the link to the @PSNH \Outage Map\. As if the Bing Maps tiles weren't bad enough, the use of a categorical color scale instead of a…

Visualizing Historical & Most-likely First Snowfall Dates for U.S. Regions

Wed, 26 Nov 2014 06:47:33 GMT

The impending arrival of the first real snowfall of the year in my part of Maine got me curious about what the most likely "first snow" dates are for my region. The U.S. Historical Climatology Network (USHCN) maintains \historical daily…

Moving The Earth (well, Alaska & Hawaii) With R

Sun, 16 Nov 2014 22:35:44 GMT

In a previous post we looked at how to use D3 TopoJSON files with R and make some very D3-esque maps. I mentioned that one thing missing was moving Alaska & Hawaii a bit closer to the continental United States and this post shows you how…

GRANITESEC X!!!

Tue, 21 Oct 2014 19:08:14 GMT

GRANITESEC X!!! — rud.is blog archive

Spending Seized Assets - A State-by-State Per-capita Comparison in R

Mon, 13 Oct 2014 20:28:08 GMT

The Washingon Post did another great story+vis, this time on states \Spending seized assets\. According to their sub-head: >\Since 2008, about 5,400 police agencies have spent $2.5 billion in proceeds from cash and property seized under…

Plot Me Like a Hurricane (a.k.a. animating historical North Atlantic basin tropical storm tracks)

Wed, 08 Oct 2014 02:34:24 GMT

Markus Gessman (@MarkusGesmann) did a beautiful job \Visualising the seasonality of Atlantic windstorms\ using small multiples, which was inspired by both a \post\ by Arthur Charpentier (@freakonometrics) on using Markov spatial processes…

Blast From The Past Also Reveals A Bit Of (US) Postal HIstory

Fri, 26 Sep 2014 10:08:13 GMT

I saw a re-tweet for this "Man From Mars Radio Hat" advertisement and—while it's awesome as a bit of nostalgia on it's own—it also reveals a bit about our (US) Postal…

Overcoming D3 Cartographic Envy With R + ggplot

Fri, 26 Sep 2014 04:49:32 GMT

When I used one of the Scotland TopoJSON files for a recent post, it really hit me just how much D3 cartography envy I had/have as an R user. Don't get me wrong, I can conjure up D3 maps pretty well \[1\] \[2\] and the utility of an…

Seeing the (day)light with R

Tue, 23 Sep 2014 16:28:49 GMT

The arrival of the autumnal equinox foreshadows the reality of longer nights and shorter days here in the northeast US. We can both see that reality and distract ourselves from it at the same time by firing up RStudio (or your favorite…

Charting/Mapping the Scottish Vote with R (an rvest/dplyr/tidyr/TopoJSON/ggplot tutorial)

Sat, 20 Sep 2014 14:49:55 GMT

The BBC did a pretty good job \live tracking the Scotland secession vote\, but I really didn't like the color scheme they chose and decided to use the final tally site as the basis for another tutorial using the tools from the Hadleyverse…

An Alfred (OS X) Workflow for Weather Underground Personal Weather Station (PWS) Data

Mon, 15 Sep 2014 00:10:20 GMT

I've operated a \Weather Underground\ \Personal Weather Station\ (PWS) \\[KMEBERWI7\\] off-and-on (hardware issues notwithstanding) for as long as I can remember, and I thought it was about time to finally do an Alfred↔PWS mashup. My…

R version of "An exploratory technique for visualizing the distributions of 100 variables:"

Wed, 10 Sep 2014 13:59:17 GMT

Rick Wicklin (@\RickWicklin\) made a recent post to the SAS blog on \An exploratory technique for visualizing the distributions of 100 variables\. It's a very succinct tutorial on both the power of boxplots and how to make them in SAS (of…

OUT FROM THE DEPTHS! @GraniteSec LIVES!

Fri, 04 Jul 2014 22:50:42 GMT

OUT FROM THE DEPTHS! @GraniteSec LIVES! — rud.is blog archive

New Gig!

Sat, 17 May 2014 18:15:17 GMT

Well, the proverbial cat is definitely out of the bag now. I'm moving on from the current gig to take a security data scientist position at Verizon Enterprise. The esteemed Wade Baker will be my new benevolent overlord and it probably…

Rforecastio 1.3.0 Update

Wed, 07 May 2014 00:56:00 GMT

Thanks to a comment suggestion, the Rforecastio package is now up to version 1.3.0 and has a new parameter which lets you specify which time conversion function you want to use. Details are up on…

Rforecastio 1.2.0 Bug-fix Update

Mon, 05 May 2014 16:57:10 GMT

Not even going to put an \R\ category on this since I don't want to pollute R-bloggers with this tiny post, but I had to provide the option to let folks specify \ssl.verifypeer=FALSE\ (so I made it a generic option to pass in any CURL…

Rforecastio Package Update (1.1.0)

Sun, 04 May 2014 12:04:46 GMT

I've bumped up the version number of \Rforecastio\ (\github\) to \1.1.0\. The new features are: - removing the SSL certificate bypass check (it doesn't need it anymore) - using \plyr\ for easier conversion of JSON-\\>data frame - adding in…

Moving From system() calls to Rcpp Interfaces

Fri, 25 Apr 2014 18:47:57 GMT

Over on the \Data Driven Security Blog\ there's a post on how to use \Rcpp\ to interface with an external library (in this case \ldns\ for DNS lookups). It builds on \another post\ which uses \system()\ to make a call to \dig\ to lookup…

Getting Fit2Tcx Working on Mac OS X (10.9.x)

Tue, 22 Apr 2014 03:23:43 GMT

Andreas Diesner's \#spiffy\ \Fit2Tcx\ command-line utility is a lightweight way to convert Garmin/ANT \FIT\ files to \TCX\ for further processing. On a linux system, installing it is as simple…

Mapping the March 2014 California Earthquake with ggmap

Tue, 01 Apr 2014 18:44:33 GMT

I had no intention to blog this, but @jayjacobs convinced me otherwise. I was curious about the recent (end of March, 2014) \California earthquake\ "storm" and did a quick plot for "fun" and personal use using \ggmap\/\ggplot\. I used data…

Guardian Words: Visualized

Sat, 15 Mar 2014 18:37:58 GMT

Andy Kirk (@visualisingdata) & Lynn Cherny (@arnicas) tweeted about the Guardian Word Count service/archive site, lamenting the lack of…

Using Twitter as a Data Source For Monitoring Password Dumps

Thu, 20 Feb 2014 15:24:07 GMT

I shot a quick post over at the \Data Driven Security blog\ explaining how to separate Twitter data gathering from R code via the Ruby \t\ (\github repo\) command. Using \t\ frees R code from having to be a Twitter processor and lets the…

One More (Yet-another?) Olympic Medal Live-tracking Shiny App

Wed, 12 Feb 2014 18:23:14 GMT

I'm posting this mostly to show how to: - use the Google spreadsheet data-munging "hack" from the \previous post\ in a Shiny context - include it seamlessly into a web page, and - run it locally without a great deal of wrangling The code…

Live Google Spreadsheet For Keeping Track Of Sochi Medals

Tue, 11 Feb 2014 17:38:20 GMT

The "medals" R post by \TRInker\ and re-blogged by \Revolutions\ were both spiffy and a live example why there's no point in not publishing raw data. You don't need to have R (or any other language) do the scraping, though. The…

Data Driven Security Roundup: betaPERT, Shiny, Honeypots, Passwords & Reproducible Research

Mon, 10 Feb 2014 04:10:28 GMT

Jay Jacobs (@jayjacobs)—my co-author of the soon-to-be-released book \Data-Driven Security\—& I have been hard at work over at the book's \sister-blog\ cranking out code to help security domain experts delve into the dark art of data…

Data-Driven Security (The Book) Update #ShamelessSelfPromotion

Fri, 31 Jan 2014 11:27:45 GMT

Data-Driven Security (The Book) Update #ShamelessSelfPromotion — rud.is blog archive

Lies, Damn Lies, "Data Journalism" and Charts That Don't Start at 0

Wed, 29 Jan 2014 01:48:00 GMT

This tweet by @moorehn (who usually is a superb economic journalist) really bugged…

Change The Default "Shell..." Action In RStudio for OS X

Fri, 10 Jan 2014 14:31:16 GMT

RStudio is my R development environment of choice and I work primarily on/in Mac OS X. While it's great that Apple provides a built-in Terminal application, I prefer to use \iTerm 2\ when I need to do work at a shell. The fine folks at…

Announcing The Launch Of The Data Driven Security [Blog|Podcast]

Mon, 06 Jan 2014 12:06:12 GMT

While you're waiting for the \book\ by @jayjacobs & @hrbrmstr to hit the shelves, why not head on over to the inaugural post of the \Data Driven Security Blog\ & give a listen to the first episode of the \Data Driven Security Podcast\. The…

Points, Polygons and Power Outages

Fri, 27 Dec 2013 15:30:09 GMT

Most of my free coding time has been spent tweaking a D3-based live power outage tracker for Central Maine Power customers (there's also a woefully less-featured Shiny app for it, too). There is some R associated with the D3 vis, but it's…

Newest Font Obsession: Lato

Wed, 04 Dec 2013 13:15:56 GMT

Newest Font Obsession: Lato — rud.is blog archive

Slopegraph Workbench/Workshop in D3

Mon, 02 Dec 2013 21:50:24 GMT

I've been getting a huge uptick in views of my Slopegraphs in Python post and I think it's due to @edwardtufte's recent slopegraph contest announcement. The original Python code is crufty and a mess mostly due to the intermittent attention…

One More Time…Mapping Maine Power Outages with D3

Fri, 29 Nov 2013 01:48:03 GMT

It started with a local R version and migrated to a Shiny version and is now in full D3 glory. Some down time gave me the opportunity to start a basic D3 version of the outage map, but it needs a bit of work as it relies on a page meta…

Back To Frank

Thu, 28 Nov 2013 12:06:33 GMT

Even though I really liked Origin, the performance issues associated with it were just too much to debug and I have a ton of other work to do. Back to Frank it is. Page loads are much faster and there are far fewer warnings in Google's…

Mapping Power Outages in Maine Dynamically with Shiny/R

Wed, 27 Nov 2013 20:13:29 GMT

I decided to forego the D3 map mentioned in the previous post in favor of a Shiny one since I had 90% of the mapping code written. I binned the ranges into three groups, changed the color over to something more pleasant (with…

Mapping Power Outages In Maine With R

Wed, 27 Nov 2013 14:01:25 GMT

UPDATE: A Shiny (dynamic) version of this is now available. We had yet-another power outage this morning due to the weird weather patterns of the week and it was the final catalyst I needed to crank out some R code to map the affected…

Preparing For The February 2014 Book Launch

Wed, 27 Nov 2013 00:05:17 GMT

Data Driven Security launches in February 2014. @jayjacobs & I have seen half of the book in PDF form so far and it's almost unbelievable that this journey is almost…

Visual Anatomy Of R Packages Used in Data Driven Security

Thu, 14 Nov 2013 01:52:06 GMT

Since @jayjacobs & I are down to the home stretch on Data Driven Security, I thought it would be interesting to do some post-writing pseudo-analyses of the book itself. I won't have exact page or word counts for a bit, but I wanted to see…

Alternative to Grouped Bar Charts in R

Sun, 27 Oct 2013 17:55:48 GMT

I have almost as much disdain for grouped bar charts as I do for pie or donut charts, so appreciated the opportunity to try a makeover. However, I ran into an immediate problem: the usually #spiffy 451 Group folks did not include raw data.…

Open Campgrounds In Maine During The Stupid Shutdown

Thu, 03 Oct 2013 00:30:51 GMT

Don't let the morons in Congress stop you from visiting our fair state during this beautiful, colorful autumn season. Below is a Google Maps view of known, open campgrounds that will let you experience the best our state has to offer this…

Visualizing "ObamaCare-related" Job Cuts

Sun, 29 Sep 2013 00:44:50 GMT

UPDATE: Added some extra visualization elements since this post went live. New select menu and hover text for individual job impact detail lines in the…

Scraping Content From Google Groups

Wed, 25 Sep 2013 15:57:16 GMT

I was helping a friend out who wanted to build a word cloud from the text in Google Groups posts. If you've made any efforts to try to get content out of Google Groups you know that the only way to do so is to ensure you subscribe to the…

Animated IRL Pirate Attacks In R

Thu, 19 Sep 2013 19:11:13 GMT

Avast me hearRties! (ok, enough of the pirate speak in a blog post) It wouldn't be TLAPD without out some modest code & idea pilfering from Mark Bulling & Simon Raper. While those mateys did a fine job hoisting up some R code (your really…

Christian Science Monitor Text Layout (Single Page Full Article View)

Sun, 15 Sep 2013 20:06:19 GMT

As I was about to "buffer" a reference to a CSM article on "cyber war" (below), I paused to look for a "print view" or some other icon that would let me show the whole article as a single page vs the annoying multi-page layout view they…

Send Mac OS Notifications From RStudio Server! (via Growl)

Sat, 14 Sep 2013 19:37:09 GMT

Having received a couple follow-ups to the OS X notifications on RStudio Desktop for the Mac post, I was determined to find a quick hack to get remote notifications to OS X working from (at least) RStudio Server instances running on the…

Rforecastio Update

Fri, 13 Sep 2013 15:49:05 GMT

Thanks to a comment, I tweaked the data retrieval to ignore SSL cert errors. You can change that tweak back if you go through the pain of updating the SSL libraries on your Windows boxes (it doesn't seem to be an issue on OS X/Linux). I…

Send Mac OS Notifications From R

Thu, 12 Sep 2013 23:12:06 GMT

2013-09-16 UPDATE: I took suggestions from a couple comments, expanded the function a bit and stuck it in a gist. See this comment for…

Rforecastio - Simple R Package To Access forecast.io Weather Data

Sun, 08 Sep 2013 11:59:09 GMT

It doesn't get much better for me than when I can combine R and weather data in new ways. I've got something brewing with my Nest thermostat and needed to get some current wx readings plus forecast data. I could have chosen a number of…

nizdos - Nest Thermometer Notifications And Data Logging In Python

Thu, 05 Sep 2013 01:39:30 GMT

I've had a Nest thermometer for a while now and it's been an overall positive experience. It's given me more visibility into our heating/cooling system usage, patterns and preferences; plus, it actually saved us money last winter. We try…

Use Fluid On MacOS To Build Per-RStudio Server "Dedicated" Clients

Mon, 26 Aug 2013 01:26:06 GMT

I'm jumping around analytics environments these days and have to leave the comfort of my Mac's RStudio Desktop application to use various RStudio Server instances via browser. While I prefer to use Chrome, the need to have a "dedicated"…

ZeroAccess Bots Desperately Seeking Freedom (Visualization)

Wed, 21 Aug 2013 12:04:34 GMT

I've been doing a bit of graphing (with real, non-honeypot network data) as part of the research for the book I'm writing with @jayjacobs and thought one of the images was worth sharing (especially since it may not make it into the book…

Reverse IP Address Lookups With R (From Simple To Bulk/Asynchronous)

Tue, 13 Aug 2013 02:17:33 GMT

R lacks some of the more "utilitarian" features found in other scripting languages that were/are more geared—at least initially—towards systems administration. One of the most frustrating missing pieces for security data scientists is the…

IP Intelligence Lookup Chrome Extension

Sat, 10 Aug 2013 12:09:04 GMT

The topic of "IP intelligence" gets a nod in the book that @jayjacobs & I are writing and it was interesting to see just how many sites purport to "know something" about an IP address. I shamelessly admit to being a Chrome user and noticed…

Summer 2013 @GraniteSec Is ON!

Thu, 25 Jul 2013 09:52:49 GMT

What's missing from that picture? YOU! Like an aging action hero, @GraniteSec is back in action after an unexpected hiatus. Join us on August 17th for food and fun at the beautiful Fort Foster in Kittery Point, Maine. The water is chilly,…

2013 Summer Vacation Puzzle Challenge

Tue, 02 Jul 2013 18:25:59 GMT

When I am out of the office for an extended time, I try to post a "crypto" challenge for work-folk to do while I'm gone with the added bonus of winning fabulous prizes. There were no answers submitted for the clues during our ANP trip, but…

Re-imagining @panda_security's Q1 2013 Report Pie Charts

Fri, 24 May 2013 11:31:35 GMT

We infosec folk eat up industry reports and most of us have no doubt already gobbled up @panda\security's recently released \Q1 2013 Report\ \[PDF\]. It's a good read (so go ahead and read it, we'll still be here!) and I was really happy…

Secure360 (@Secure360) Data Analysis & Visualization Talk Resources #Sec360

Wed, 15 May 2013 05:00:32 GMT

Many thanks to all who attended the talk @jayjacobs & I gave at @Secure360 on Wednesday, May 15, 2013. As promised, here are the \slides\. We've enumerated quite a bit of non-slide-but-in-presentation information that we wanted to…

Slopegraph As A Service

Fri, 26 Apr 2013 21:58:51 GMT

@adammontville \posited\ that Figure 15 from this year's \DBIR\ could use some slopegraph love. As I am not one to back down from a reasonable challenge, I obliged. Here's the original chart (produced by…

Wait Wait...Don't Pwn Me! (SOURCE Boston) Transcript & Thank You!

Fri, 19 Apr 2013 14:01:13 GMT

Wait Wait...Don't Pwn Me! (SOURCE Boston) Transcript & Thank You! — rud.is blog archive

SOURCE Boston (@SOURCEConf) Data Analysis & Visualization Talk Resources #srcbos13

Thu, 18 Apr 2013 17:50:29 GMT

Many thanks to all who attended the talk @jayjacobs & I gave at @SOURCEconf on Thursday, April 18, 2013. As promised, here are the \slides\ which should be much less washed out than the projector version :-) We've enumerated quite a bit of…

Framing A One Data Vis A Day Challenge

Mon, 15 Apr 2013 02:40:58 GMT

Earlier this evening, I somewhat half-heartedly challenged @jayjacobs that he & I should be generating one data visualization per day. I didn't specify anything else (well, at least that I can disclose publicly, for now) but I think I'm…

PC Maker Slopegraph + More Functional Table (One Last Time)

Sat, 13 Apr 2013 14:03:20 GMT

Now that I'm back in the US and relaxing, I can take time for one final blather on the \PC Maker Slopegraph\ post from earlier in the week. Slopegraphs can be quite long depending on the increment between discrete entries (as I've \pointed…

Use @datawrapper For Slopegraphs

Sat, 13 Apr 2013 00:03:06 GMT

While not perfect, I noticed that it was possible to make a pretty decent slopegraph over at \Datawrapper\ as I was poking at some new features they announced recently. As an example, I ran one of the charts from my \most recent\ blog post…

Ugly Tables vs Slopegraphs : PC Maker Shipments & Marketshare

Thu, 11 Apr 2013 11:05:09 GMT

Andrew Cunningham (@IT\AndrewC) posted an article—If you make PCs and you’re not Lenovo, you might be in trouble—on the always #spiffy @arstechnica that had this horrid table in…

Bahrain eGov Conference "Risk Reality" Slides

Wed, 10 Apr 2013 19:52:45 GMT

For those finding this post from the Bahrain eGov conference, I'd like to re-extend a hearty "Thank you!" for being one of most engaging, interactive and intelligent audiences I've ever experienced. I truly enjoyed talking with all of you.…

Off to Bahrain

Mon, 08 Apr 2013 11:21:00 GMT

As a result of a prod by @djbphaedrus I'm off to the Bahrain International eGovernment Forum this week to host a two hour workshop on "information risk reality". As a result, blogging & tweeting will be at significantly reduced levels, so…

It's Never About The Security...

Sun, 31 Mar 2013 03:29:55 GMT

It's Never About The Security... — rud.is blog archive

A Wish for Snow in Spring

Wed, 27 Mar 2013 22:56:03 GMT

Sadly, that's not a modern story. It's an adapted quote from chapter 4 (pp. 97-98, paperback) of The Ghost Map, by Steven Johnson, a book on the cholera epidemic of 1854. I won't ruin the book nor continue my attempt at analogy any…

Interactive Slopegraphs With Processing

Tue, 19 Mar 2013 17:53:19 GMT

Given my \obsession\ with slopegraphs, I'm not sure how I missed this \post\ back in late February by @JeffClark that includes a very nicely executed \interactive sloepgraph\ on the global obesity problem. He used \Processing\ &…

My Picks From #PyCon2013

Tue, 19 Mar 2013 11:20:12 GMT

My Picks From #PyCon2013 — rud.is blog archive

Chrome App for Tiny Tiny RSS

Tue, 19 Mar 2013 03:18:57 GMT

I'm still getting my self-hosted \Tiny Tiny RSS\ configuration just the way I want it prior to doing a full blog post on it, but I thought it would be helpful to share a basic Chrome App for it. I ended up creating it to replace the Google…

π, Awareness, DataVis, VAST 2013, Moar data! & GReader Machinations

Thu, 14 Mar 2013 13:44:02 GMT

Far too many interesting bits to spam on Twitter individually but each is worth getting the word out on: - It's \π Day\ - Unless you're living in a hole, you probably know that \Google Reader is on a death march\. I'm really liking…

Visualizing Risky Words — Part 4 (D3 Word Trees)

Tue, 12 Mar 2013 16:55:38 GMT

This is a fourth post in my \Visualizing Risky Words\ series. You'll need to read starting from that link for context if you're just jumping in now. I was going to create a rudimentary version of an interactive word tree for this, but the…

Visualizing Risky Words — Part 3

Mon, 11 Mar 2013 03:11:00 GMT

The DST changeover in the US has made today a fairly strange one, especially when combined with a very busy non-computing day yesterday. That strangeness manifest as a need to take the D3 heatmap idea mentioned in the \previous post\ and…

Visualizing Risky Words — Part 2

Sat, 09 Mar 2013 12:29:35 GMT

This is a follow-up to my \Visualizing Risky Words\ post. You'll need to read that for context if you're just jumping in now. Full R code for the generated images (which are pretty large) is at the end. Aesthetics are the primary reason…

Visualizing Risky Words

Thu, 07 Mar 2013 04:05:26 GMT

Inspired by a post by @bfist who created the following word cloud in Ruby from VZ RISK INTSUM posts (visit the link or select the visualization to go to the…

Security Hobos

Sun, 03 Mar 2013 12:35:50 GMT

If you haven't viewed/read Wendy Nather's (@451Wendy) insightful \Living Below The Security Poverty Line\ you really need to do that before continuing (we'll still be here when you get back). Unfortunately, the catalyst for this post came…

Third Annual RSA PhöCon : Thursday, Feb 27 @ 1830 : Miss Saigon

Wed, 27 Feb 2013 15:10:08 GMT

Come join us for a PhöCon good time at 18:30 (6:30PM) Thursday! The neighborhood is…interesting…but it's close to Moscone and has had good food for the past couple years. I'll be heading up there from the Expo area at ~1815. Hit me up on…

Follow up/Resources :: GRC-T18 – Data Analysis and Visualization for Security Professionals #RSAC

Wed, 27 Feb 2013 12:30:55 GMT

Many thanks to all who attended the talk @jayjacobs & I gave at RSA on Tuesday, February 26, 2013. It was really great to be able to talk to so many of you afterwards as well. We've enumerated quite a bit of non-slide-but-in-presentation…

RSA & Metricon Speaking & Dining Round Up

Fri, 22 Feb 2013 13:47:00 GMT

Just joining in the fray of "where I'm speaking/where I'll be the week of @RSAConference"…

R/netintel : Cross-check APT-1's IP list with AlienVault Reputation DB (+ some graphs/analysis)

Wed, 20 Feb 2013 12:00:00 GMT

Here's a quick example of couple additional ways to use the netintel R package I've been tinkering with. This could easily be done on the command line with other tools, but if you're already doing scripting/analysis with R, this provides a…

[REDUX] Reassessing Cyber Risks as a Security Strategy

Tue, 19 Feb 2013 05:08:23 GMT

I happened across \Between Hype and Understatement: Reassessing Cyber Risks as a Security Strategy\ \[PDF\] when looking for something else at the \Journal of Strategic Security\ site and thought it was a good enough primer to annoy…

Happy Valentine's Day @mrshrbrmstr!

Thu, 14 Feb 2013 15:00:26 GMT

Happy Valentine's Day @mrshrbrmstr! — rud.is blog archive

Visualizing Malicious Clusters & Outliers

Fri, 08 Feb 2013 22:49:24 GMT

So, I've had some quick, consecutive blog posts around this R package I'm working on, and this one is more of an answer to my own, self-identified question of "so what?". As I was working on an importer for AlienValut's IP reputation…

Extended (Simple) ASN Graph Visualization Example [R to D3]

Fri, 08 Feb 2013 13:02:55 GMT

The small igraph visualization in the previous post shows the basics of what you can do with the BulkOrigin & BulkPeer functions, and I thought a larger example with some basic D3 tossed in might be even more useful. Assuming you have the…

Retrieve IP ASN & BGP Peer Info With R

Fri, 08 Feb 2013 04:22:18 GMT

This is part of a larger project I'm working on, but it's useful enough to share (github version coming soon). The fine folks at @TeamCymru have a great service to map IP addresses to ASN/BGP) information en masse. There are libraries for…

Winter Wind Flows

Fri, 01 Feb 2013 14:36:05 GMT

Yesterday, I took a very short video capture from \wind map\ of the massive wind flows buffeting the northeast. I did the same this morning and stitched them together to see what a difference a day makes. Nothing earth-shattering here, but…

Data Crunching Competitions

Thu, 31 Jan 2013 18:49:12 GMT

I came across a list of data crunching and/or visualization competition sites today. I had heard of Kaggle & NITRD before but not the other ones. If you're looking to get into data analytics/visualization or get better at it, the only way…

Once More Into The [PRC Aggregated] Breaches

Sun, 27 Jan 2013 23:29:45 GMT

If you're not on the SecurityMetrics.org mailing list you missed an interaction about the Privacy Rights Clearinghouse Chronology of Data Breaches data source started by Lance Spitzner (@lspitzner). You'll need to subscribe to the list see…

River Post Dissected

Sun, 20 Jan 2013 19:50:11 GMT

Good Stats, Bad Stats has a really good critique of my Mississippi River post that you should read (TL;DR: my graphs & analysis need…

SHODAN API in R (With Examples)

Thu, 17 Jan 2013 20:21:36 GMT

Folks may debate the merits of the SHODAN tool, but in my opinion it's a valuable resource, especially if used for "good". What is SHODAN? I think ThreatPost summed it up…

How Low Can It [The Mississsippi River] Go?

Sun, 13 Jan 2013 03:35:23 GMT

I've heard quite a bit about the current problems with the Mississippi River and wanted to see for myself (with data) just how bad it is. St Louis seems to be quite indicative of the severity of the situation, so I pulled the USGS "stream"…

Slopegraphs in R

Sat, 12 Jan 2013 01:59:09 GMT

If you even remotely follow this blog, you'll see that I'm kinda obsessed with slopegraphs. While I'm pretty happy with my Python implementation, I do quite a bit of data processing & data visualization in R these days and had a few free…

CVE Queries Right From Your Browser's Address Bar

Sat, 29 Dec 2012 02:09:38 GMT

I'm not sure why I never did this earlier, but a post on LifeHacker gave me an idea to add location bar quick search of CVEs (Common Vulnerabilities and Exposures), no doubt due to their example on searching LifeHacker for "security". My…

Easier HTML Table-scraping For Scripts With Google Drive

Mon, 17 Dec 2012 20:21:50 GMT

We had our first, real, snowfall of the season in Maine today and that usually means school delays/closings. Our "local" station – @WCHS6 – has a Storm Center Closings page as well as an SMS notification service. I decided this morning…

The 'fing' Corollary

Thu, 13 Dec 2012 22:24:53 GMT

Back in 2011, @joshcorman posited "HD Moore's Law" which is…

Forbes Graph Makeover Contest Entry #1

Wed, 05 Dec 2012 16:10:41 GMT

Naomi Robbins is running a graph makeover challenge over at her Forbes blog and this is my entry for the B2B/B2C Traffic Sources one (click for larger…

Election Trackers Moved From Flash to HTML5

Tue, 27 Nov 2012 23:56:55 GMT

For those inclined to click, I was interviewed by Fahmida Rashid (@fahmiwrite) over at Sourceforge's HTML5 center a few weeks ago (right after the elections) due to my tweets on the use of HTML5 tech over Flash. Here's one of them:…

Speaking At RSA Conference 2013!

Sun, 18 Nov 2012 14:57:27 GMT

Earlier this week, @jayjacobs & I both received our acceptance notice for the talk we submitted to the RSA CFP! \[W00t!\] Now the hard part: crank out a compelling presentation in the next six weeks! If you're interested at all in doing…

'Sandy' Code Up On Github

Mon, 29 Oct 2012 09:49:44 GMT

I will (most likely) not be littering the blog with any more updates to the 'Sandy' code unless they are really significant. You can follow along at home to any changes over at github. Of note (in terms of changes) is the addition of the…

Watch Sandy in "R" (Including Forecast Cone)

Sun, 28 Oct 2012 11:11:19 GMT

This is a follow-up post to the quickly crafted Watch Sandy in "R" post last night. I noticed that Google provided the KML on their crisis map and wanted to show how easy it is to add it to previous code. I added comments inline to make it…

Watch "Sandy" In R

Sun, 28 Oct 2012 00:21:20 GMT

While this will pale in comparison to the #spiffy Sandy graphics on Weather Underground and other sites (including Google's "Crisis Center" maps), I thought it might be interesting to show folks how they can keep an eye on "Sandy" using…

Extracting OSE Firewall Alert Data From IMAP (Gmail) Mail To CSV With Python

Wed, 24 Oct 2012 10:12:16 GMT

I played around with OSE Firewall for WordPress for a couple days to see if it was worth switching to from the plugin I was previously using. It's definitely not as full featured and I didn't see any WP database extensions where it kept a…

Get an R Data Frame from a MongoDB Query

Mon, 22 Oct 2012 18:50:30 GMT

There's a good FAQ on how to do the MongoDB query -> R data frame but I wanted to post a more complete example that included the database connection and query setup since I suspect there are folks new to Mongo who would appreciate the…

Putting Cybercrime [Infographics] Behind Bars

Mon, 22 Oct 2012 13:37:38 GMT

HP & the Ponemon Institute have released their third annual "Cost of Cybercrime" report and the web wizards at HP have given us an infographic from…

"Show in Finder" Broken? Here's One Potential Way To Fix It

Mon, 22 Oct 2012 02:14:07 GMT

I'm not entirely sure what happened to cause it, but the "Show in Finder…" feature of a plethora of OS X apps broke on me just over a month ago. I poked around a bit, deleted a preference file here or there and un-tweaked some tweaks to…

DIY ZeroAccess Analysis Lesson #1 : Treat GeoIP Results Lightly

Thu, 11 Oct 2012 18:51:40 GMT

We're still crunching through some of the ZeroAccess data and have some (hopefully) interesting results to present, but an weirld GeoIP anomaly has come up that I wanted to quickly share. To get some more granular data, I'm using the…

DIY ZeroAccess GeoIP Analysis : So What?

Mon, 08 Oct 2012 13:24:45 GMT

NOTE: A great deal of this post comes from @jayjacobs as he took a conversation we were having about thoughts on ways to look at the data and just ran like the Flash with it. Did you know that – if you're a US citizen – you have…

DIY ZeroAccess GeoIP Visualizations :: Back To The Basics

Sat, 06 Oct 2012 01:14:08 GMT

While shiny visualizations are all well-and-good, sometimes plain ol' charts & graphs can give you the data you're looking for. If we take the one-liner filter from the previous example and use it to just output CSV-formatted summary…

More DIY ZeroAccess GeoIP Fun : jQuery/D3 Choropleths

Fri, 05 Oct 2012 23:48:23 GMT

In the spirit of the previous example this one shows you how to do a quick, country-based choropleth in D3/jQuery with some help from the command-line since not everyone is equipped to kick out some R and most folks I know are very handy…

DIY ZeroAccess GeoIP Plots

Fri, 05 Oct 2012 13:48:56 GMT

Since F-Secure was #spiffy enough to provide us with GeoIP data for mapping the scope of the ZeroAccess botnet, I thought that some aspiring infosec data scientists might want to see how to use something besides Google Maps & Google Earth…

Show Mac App Store Downloads Folder (and more!)

Mon, 17 Sep 2012 19:04:55 GMT

@mroesch asked how to find the "Downloads" folder for the Mac App Store. Apple changed this from the time I posted the one and only app I ever did to MAS (a while ago). So, here's how to do it now. First, open up iTerm or Terminal and…

Businessweek's #Spiffy Rank-order Stock Market Slopegraph

Mon, 03 Sep 2012 14:48:07 GMT

Businessweek's bleeding-edge approach to typography, layout and overall design is one of the features that keeps me reading the magazine in print form. The design team also often delves into experiments with data visualization and…

Building Jytter On OS X (tested on Mountain Lion)

Sat, 25 Aug 2012 01:40:00 GMT

Dan Kaminski (@dakami) tweeted a cool, small (11 instruction) TRNG generator called Jytter Friday: https://twitter.com/dakami/status/239179831860924416 The authors have Windows & Linux ports but no OS X port (and, I play mostly on OS X…

Just When You Thought It Was Safe To Make A Slopegraph

Mon, 06 Aug 2012 12:06:51 GMT

Thanks to a nice call-out post link on Flowing Data in my RSS feeds this morning, I found Naomi Robbins' Effective Graphs Forbes blog, perused the archives a bit and came across her post on arrow charts. She presented a nice comparison…

VERIS Community :: JSON vs XML

Sat, 21 Jul 2012 01:06:20 GMT

You may not be aware of the fact that the #spiffy Verizon Biz folk have some VERIS open source components, one of which is the XML schema for the "Vocabulary for Event Recording and Incident Sharing". While most Java-backends will readily…

"Fibonacci" Graphing

Sat, 14 Jul 2012 10:27:04 GMT

Benchmarking/profiling is one of the fundamental practices for tech folk and Feng Shen's recent post to Hacker News continues this fine tradition with a look at startup & run times for "fibonacci(40)" in seven computer languages (two 'C'…

SSH Password Time-series Heatmap In D3

Thu, 12 Jul 2012 18:33:20 GMT

In @jayjacobs' latest post on SSH honeypot passsword analysis he shows some spiffy visualizations from crunching the data with Tableau. While I've joked with him and called them "robocharts", the reality is that Tableau does let you work…

Security & Privacy Of Mountain Lion's Dictation Feature

Thu, 12 Jul 2012 13:43:10 GMT

With Gizmodo doing a post hyping Mountain Lion's new dictation feature it's probably a good time to note that folks in regulated environments or who just care about security & privacy a bit more than others should not enable or use this…

Honeypot Analytics : 500 Pretty Passwords

Mon, 09 Jul 2012 18:23:21 GMT

I had a few moments this past weekend to play with an idea for visualizing the passwords used against the honeypot @jayjacobs set up. While it's not as informative as Jay's weekend endeavors:…

Beach Blanket Tweetup
(a.k.a. GraniteSec Tweetup #5)

Sat, 07 Jul 2012 20:18:21 GMT

Brylcream? Check. Decked out in your best Frankie Avalon or Annette Funicello beach fashions? Check. Now, polish up your cruiser and get ready 'cuz @GraniteSec is heading back to Fort Foster \[info\] \[map\] at Kittery Point, Maine for…

2012 WhiteHat Security Website Security Statistics Report Redux

Mon, 02 Jul 2012 16:38:54 GMT

The always #spiffy @WhiteHatSec released their 2011 Web Site Security stats report \[direct link (PDF)\] last week (here's one of their teaser tweets): https://twitter.com/whitehatsec/status/218046945783054337 With over 7,000 sites and…

RIM of Destruction

Fri, 29 Jun 2012 14:33:04 GMT

(With apologies to Barry McGuire & P.F. Sloan) To a familiar tune… RIM of Destruction The mobile world, it is explodin' executives flarin', app updates loadin' you're Bold's enough to sell but no one's buyin' no one believes in your…

Honeypot Analytics

Thu, 28 Jun 2012 18:02:13 GMT

For this post (and probably a few subsequent ones), I'm taking the role of 'Pinky" to @jayjacobs' 'Brain' as I share some of my own analysis on the ssh honeypot passwords that Jay collected (you'll need to read his VZB post before…

Breach Reach : Google Insights

Sat, 23 Jun 2012 14:16:19 GMT

Information security breaches have been the "new black" in the past eighteen months, with the latest fashion-trend being the LinkedIn passwords fiasco. This got me thinking: what is the "half-life" of a breach? It's becoming obvious that…

Slopegraphs in Python – Failed States Index (Part 1)

Thu, 21 Jun 2012 03:43:13 GMT

The Fund For Peace (FFP) and Foreign Policy jointly released the 2012 version of the "failed states index" (FSI). From the FFP site, the…

Tweet From Mountain Lion Notification Center

Sun, 17 Jun 2012 11:47:23 GMT

If this feature was in Mountain Lion Developer Previews prior to revision 4, I didn't notice it, but you can now tweet directly from the Notification Center "pane". You first need to make sure said feature is enabled in System Preferences…

WEIS 2012 :: Best of the Best

Sat, 16 Jun 2012 16:13:20 GMT

Sadly, I could not make it to this year's Workshop on the Economics of Information Security. However, the intrepid conference organizers were quick to post the papers that were presented, and I had a chance to sift through them to pick out…

Just Like The Federation

Thu, 14 Jun 2012 16:38:05 GMT

\[@hrbrmstr starts working in javascript again\] The Internets: What do you think? @hrbrmstr: It's vile. The Internets: I know. It's so bubbly and cloying and happy. @hrbrmstr: Just like the Federation. The Internets: And you know what's…

Slopegraphs in Python – Gratuitous Raphaël Animations

Tue, 12 Jun 2012 17:21:32 GMT

So, I had to alter this to start after a user interaction. It loaded fine as a static, local page but seems to get a bit wonky embedded in a complex page. I also see some artifacts in Chrome but not in Safari. Still, not a bad foray into…

Slopegraphs in Python – Formatting Tweaks

Mon, 11 Jun 2012 14:59:24 GMT

There were enough eye-catching glitches in the experimental javascript support and the ugly large-number display in the spam example post that I felt compelled to make a couple formatting tweaks in the code. I also didn't have time to do…

Slopegraphs Everywhere

Mon, 11 Jun 2012 01:14:21 GMT

Not much progress over the weekend on my latest obsession (been busy enjoying some non-rainy days here in Maine). So, here are some other slopegraph implementations/resources I've found through mining the…

Slopegraphs in Python – Experimental Raphaël Support

Fri, 08 Jun 2012 18:38:23 GMT

In preparation for the upcoming 1.0 release and with the hopes of laying a foundation for more interactive slopegraphs, I threw together some rudimentary output support over lunch today for Raphaël, which means that all you have to do is…

Slopegraphs in Python – Log Scales & Spam Data Analysis

Thu, 07 Jun 2012 21:32:06 GMT

Given the focus on actual development of the PySlopegraph tool in most of the blog posts of late, folks may be wondering why an infosec/inforisk guy is obsessing so much on a tool and not talking security. Besides the fixation on filling a…

Slopegraphs in Python – Exploring Binning/Rounding

Thu, 07 Jun 2012 16:17:29 GMT

One of the last items for the 1.0 release is support for multiple columns of data. That will require some additional refactoring, so I've been procrastinating by exploring the recent "fudging" discovery. Despite claims to the contrary on…

Slopegraphs in Python – Slope Colors

Wed, 06 Jun 2012 02:43:46 GMT

As the codebase gets closer to the 1.0 stretch we now have the addition of slope colors for when values go up/down or remain constant between points. The code still only handles two columns of data, but the intent is for each segment to…

Slopegraphs in Python – More Output Tweaks

Tue, 05 Jun 2012 16:12:53 GMT

The best way to explain this release will be to walk you through an updated configuration…

Slopegraphs – The Quintessential Tufte GDP Example Revisited

Tue, 05 Jun 2012 15:17:37 GMT

If you're even remotely familiar with slopegraphs, then you'll recognize Tufte's classic 1970-1979 GDP chart example (click for larger…

Slopegraphs in Python – The Great Refactor

Sat, 02 Jun 2012 22:47:43 GMT

Despite being on holiday, I had a spare hour to refactor the code (@mrshrbrmstr was joining the 1% in the hotel spa). It's up on github and now sports a spiffy JSON-format config file. You now must execute the slopegraph.py script with a…

Slopegraphs in Python – Attention to Detail

Thu, 31 May 2012 10:04:00 GMT

In the previous installment, a foundation was laid for "parameterizing" fonts, colors and overall slopegraph size. However, a big failing in all this code (up until now) was the reliance on character string length to determine label width.…

Slopegraphs In Python – colors, fonts & alignment

Wed, 30 May 2012 01:49:27 GMT

On the heels of last evening's release of Slopegraphs in Python post comes some minor…

An Homage To Flame

Tue, 29 May 2012 23:27:27 GMT

(Sing to the tune of "Fame – Remember My name" … Here's some YouTube background music) They've been lookin' at me, but they never did see— no, no trace of me did they detect; Gave me time to collect all the data at rest. I've got so much…

Slopegraphs in Python

Tue, 29 May 2012 01:01:39 GMT

(NOTE: You can keep up with progress best at github, but can always search on "slopegraph" here or just hit the tag page: "slopegraph" regularly) I've been a bit obsessed with slopegraphs (a.k.a "Tufte table-chart") of late and very…

New SecurID Soft Token Cloning Weakness : What's The Risk?

Tue, 22 May 2012 12:45:17 GMT

I posted a link to Twitter earlier on a recent discovery of the ability to clone RSA SecurID soft…

Feudal Japan's Haiku Clubs A Pre-cursor of Modern-day Twitter?

Mon, 21 May 2012 21:35:20 GMT

I recently finished watching "Japan: Memoirs of a Secret Empire" \ [Netflix | PBS | Amazon \]. It's a three-part documentary that centers around the 16th & 17th centuries (where a vast majority of my favorite samurai movies are set). In…

Mountain Lion, AirPlay, Screen Captures & DRM

Tue, 15 May 2012 23:39:51 GMT

I've been wanting to post this entry for a while, but I didn't have the opportunity to compel an extra pair of hands to assist with some necessary, salient portions of it until tonight. For those who were hoping Mountain Lion's AirPlay…

Off By One : The Importance Of Fact Checking Breach Reports

Thu, 10 May 2012 17:09:02 GMT

I didn't read through the Massachusetts 2011 Report on Data Breach Notifications \[PDF\] until recently, but once I went through the report my brain kept telling me "something is wrong". Not something earth shattering, but more of a…

Is Your Organization Ready For a Risk Management Program?

Tue, 08 May 2012 16:55:08 GMT

While the slides will be officially available from SIRA web site in the not-too-distant future—complete with video (for all the talks)—I figured it wouldn't hurt to put them up here as…

Both Candidates Weak On [SSL] Security

Sat, 05 May 2012 10:52:08 GMT

UPDATE: Fixed link to cached Obama image thx to notice from JB While the two front-running candidates engaged in a bizarre, Klingon-esque ritual of hubris regarding which one was the better killer, their respective technical campaign…

Angry Birds Keynote Theme

Sun, 29 Apr 2012 21:52:35 GMT

If you went to SOURCE Boston this year (2012), attended my security awareness talk and liked the Angry Birds theme to the slides, here's a copy of the Keynote theme (it's not really a true Keynote theme as there are divergent slides I've…

Slaying the BEAST in nginx

Sat, 28 Apr 2012 14:03:16 GMT

Just a quick post as I noticed that my nginx configuration was vulnerable to the BEAST attack thanks to the #spiffy SSL Certificate Tester from Qualys (I scored an "A", btw :-). The nginx docs show how to do this, now, and it's pretty…

A Month With AfterShokz

Wed, 18 Apr 2012 02:15:17 GMT

Quick hit :: Over at the #tri blog, I've got a review of a relatively new bone-conducting headset by…

Getting Homebrew Working in Mountain Lion Developer Preview 2 with Xcode 4.4 Developer Preview 2

Mon, 26 Mar 2012 13:40:35 GMT

Work & home chaos has me a bit behind in the "ThinkStats…in R" posts, but I "needed" to get some of the homebrew kit working in Mountain Lion Developer Preview 2 (to run some network discovery tools while waiting for #4's surgery to be…

ThinkStats … in R :: Example/Chapter 2 :: Example 2.1-2.3

Wed, 14 Mar 2012 06:30:14 GMT

As promised, this post is a bit more graphical, but I feel the need to stress the importance of the first few points in chapter 2 of the book (i.e. the difference between mean and average and why variance is meaningful). These are…

ThinkStats … in R :: Example 1.3

Wed, 07 Mar 2012 07:00:27 GMT

With 1.2 under our belts, we go now to the example in section 1.3 which was designed to show us how to partition a larger set of data into subsets for analysis. In this case, we're going to jump to example 1.3.2 to determine the number of…

ThinkStats … in R (including Example 1.2)

Mon, 05 Mar 2012 04:53:14 GMT

ThinkStats (by Allen B. Downey) is a good book to get you familiar with statistics (and even Python, if you've done some scripting in other languages). I thought it would be interesting to present some of the examples & exercises in the…

Two Alfred Extensions For Retrieving IP Address Information

Mon, 13 Feb 2012 03:30:27 GMT

I've been an unapologetic Alfred user since @hatlessec recommended it and have recently been cobbling together quick shell scripts that make life a bit easier. The following ones - lip & rip - copy your local & remote IP addresses…

An Infosec Hippocratic Oath

Mon, 23 Jan 2012 18:40:15 GMT

I swear to fulfill, to the best of my ability and judgment, this covenant: I will respect the hard-fought empirical gains of those practitioners in whose steps I walk, and gladly share such knowledge as is mine with those who are to…

Google Spreadsheet "importHTML" Rocks For Quick Analytics

Fri, 13 Jan 2012 16:02:01 GMT

I usually take a peek at the Internet Traffic Report (ITR) a couple times a day as part of my routine and was a bit troubled by all of the red…

USDA Driving Towards Both Food Insecurity And Cyber-Insecurity?

Tue, 10 Jan 2012 19:58:00 GMT

It's rare that two of my passions—food and information security—intersect, but thanks to the USDA's announcement of their Blueprint For Stronger Service, I can touch on both in one post. In 2011, the Obama administration challenged all…

Getting Things Done : A Cobbler's Tale

Sat, 07 Jan 2012 22:46:15 GMT

Starting sometime mid-year in 2011, I began having more 'stuff' to do than even my eidetic memory could help with. It's not that I forgot things, per se, but the ability to mentally recall and prioritize work, family, personal and other…

Feed Link Has Moved

Wed, 04 Jan 2012 11:58:29 GMT

Feedburner has borked the old RSS feed for the site and has completely disassociated me from it (meaning it's no longer in my Google Feedburner admin options and they won't let me re-claim it). So… the new feed link is…

Businessweek Infographic Illustrates The Pounding We Took In 2011

Tue, 03 Jan 2012 11:03:42 GMT

Tweet (dead service):…

Improve Your Security Metrics For $14.00USD

Tue, 03 Jan 2012 01:16:58 GMT

IT Security Metrics : A Practical Framework for Measuring Security & Protecting Data has has solid reviews by Richard Bejtlich (@TaoSecurity), David J. Elfering (@icxc) & Dr. Anton Chuvakin (@anton\chuvakin), amongst others. You can get it…

NEISTU 3.0 : Rise of the Tweetup

Fri, 30 Dec 2011 20:58:15 GMT

It's that time again! Coming to Nashua, NH on 2012-01-21 : The New England Information Security Tweetup 3.0! You can use the handy graphic below to promote the tweetup or just give folks this handy…

Three Resolutions For Web Developers

Fri, 30 Dec 2011 15:51:51 GMT

I'm on a "three things" motif for 2012, as it's really difficult for most folks to focus on more than three core elements well. This is especially true for web developers as they have so much to contend with on a daily basis, whether it be…

Three Resolutions For Mac OS X Users

Thu, 29 Dec 2011 21:18:52 GMT

In 2011, we saw a large increase in web site exploits that exposed private user data as well as a breakdown in the trust of SSL (for various reasons) and the introduction of real malware on to the OS X scene. If there were just three…

An Open Letter to IT Vendors For 2012

Mon, 19 Dec 2011 00:52:53 GMT

Dear $VENDOR, 2012 is nigh upon us and with the new year, I am throwing down a challenge to each and every IT vendor out there. 2011 was a brutal year of incidents, breaches, outages and FUD and the last thing anyone needs is a repeat…

Still Not A Fan Of Paywalls

Sat, 17 Dec 2011 16:18:31 GMT

As you can probably tell from a previous post, I'm not a fan of paywalls—especially poorly implemented ones. Clicking on a link in an RSS feed post and having it land on a page, only to have it smothered in an HTML layer or — in the…

Predictions? Humbug! Resolve Is Where It's At

Sat, 10 Dec 2011 14:27:41 GMT

This is the time of year when pundits and armchair/amateur analysts make predictions for the coming year. Given that only a tiny fraction of them predicted the Sonage of 2011 (not Sony specifically or the level of pwnage) or the…

You're The Mayor of FUDville!

Fri, 02 Dec 2011 03:29:51 GMT

Tweet (dead service):…

DNSChanger Detector

Mon, 14 Nov 2011 19:17:54 GMT

The FBI made a tool to help you determine if you were a victim of the DNSChanger malware. If you're like many casual Internet users, you have no idea how to get the information to plug into the input box. Unfortunately, the security model…

Why Didn't They Just…?

Fri, 23 Sep 2011 15:03:25 GMT

A while back I was engaged in a conversation on Twitter with @diami03 & @chriseng regarding (what I felt was) the need for someone to provide the perspective from within a medium-to-large enterprise, especially when there are so many folks…

Mid/Northern New England Tweetup

Tue, 16 Aug 2011 23:13:21 GMT

Eventbrite site: http://www.eventbrite.com/s/5cnV It's at Fort Foster. We thought about it a bit late in the season so no pavillion, but I'll be there wicked-early and have a gazebo-like covering setup over some tables. I highly suggest…

Your New Mega Security Program

Wed, 29 Jun 2011 18:45:38 GMT

Everyone who can read this blog should remember the Deepwater Horizon spill that occurred in the Spring of 2010; huge loss of life (any loss is huge from my persective) and still unknown impact to the environment. This event was a wake-up…

What Can We Learn From The @lulzsec senate.gov Hack Dump?

Tue, 14 Jun 2011 23:35:16 GMT

What can the @lulzsec senate.gov dump tell us about how the admins maintained their…

WEIS 2011 :: Session 2 :: Identity :: Negative Information Looms Longer Than Positive Information

Tue, 14 Jun 2011 18:01:53 GMT

Laura Brandimarte Alessandro Acquisti Joachin Vosgerau Twitter transcript #weis2011 How does information related to past events and retrieved today get discounted? Why does neg valence receive more weight? #weis2011 how do we improve…

WEIS 2011 :: Session 2 :: Identity :: Economic Tussles in Federated Identity Management

Tue, 14 Jun 2011 17:39:35 GMT

Susan Landau Tyler Moore Presentation \[PDF\] Tyler presented really well and it's a great data set and problem to investigate. He & Susan shed a ton of light on an area most folks never think about. Well done. Twitter transcript #weis2011…

WEIS 2011 :: Session 2 :: Identity :: Social Networks, Personalized Advertising & Privacy Controls

Tue, 14 Jun 2011 17:16:54 GMT

Catherine Tucker Presentation \[PDF\] Catherine's talk was really good. She handled questions well and is a very dynamic speaker. I'm looking forward to the paper. Twitter transcript #weis2011 Premise of the study was to see what impact…

WEIS 2011 :: Session 2 :: Identity :: The Inconvenient Truth About Web Certificates

Tue, 14 Jun 2011 16:57:34 GMT

Nevena Vratonjic Julien Freudiger Vincent Bindschaedler Jeane-Pierre Hubaux Presentation \[PDF\] Twitter transcript #weis2011 Overview of basic ssl/tls/https concepts. Asking: how prevalent is https, what are problems with https? #weis2011…

WEIS 2011 :: Session 1 :: Attacks :: The Underground Economy of Fake Antivirus Software

Tue, 14 Jun 2011 15:13:37 GMT

Brett Stone-Gross Ryan Abman Richard A. Kemmerer Christopher Kruegel Douglas G Steigerwald Presentation \[PDF\] Twitter transcript #weis2011 presenting analysis of actual data from 21 servers from 3 multi-million $ fake a/v ops!!! <…

WEIS 2011 :: Session 1 :: Attacks :: Sex, Lies & Cyber-crime Survey

Tue, 14 Jun 2011 14:41:12 GMT

WEIS 2011 :: Session 1 :: Attacks :: Sex, Lies & Cyber-crime Survey — rud.is blog archive

WEIS 2011 :: Session 1 :: Attacks :: Where Do All The Attacks Go?

Tue, 14 Jun 2011 14:32:44 GMT

Dinei Florncio Cormac Herley Presentation \[PDF\] Twitter transcript #weis2011 New threat model (that may scale). Rather than use individual users & attackers, use population of users, pop of attackers #weis2011 assumption/proposition:…

WEIS 2011 :: Session 1 :: Attacks :: The Impact of Immediate Disclosure on Attack Diffusion & Volume

Tue, 14 Jun 2011 14:11:59 GMT

Sam Ransbotham Sabayasachi Mitra Presentation \[PDF\] Twitter transcript #weis2011 Does immediate disclosure of vulns affect exploitation attempts? Looking at impact on risk/diffusion/volume #weis2011 speaker is presenting standard attack…

WEIS 2011 :: Keynote :: Dr Christopher Greer

Tue, 14 Jun 2011 13:16:03 GMT

Dr Greer \[cgreer at ostp.eop.gov\] is Assistant Director, Information Technology R&D, Office of Science & Technology Policy, The White House Opening: "The expertise of the attendees is greatly needed." He provided a broad overview of the…

Dropbox 1.2.0 Experimental Build "Fixes" Security Issue

Thu, 28 Apr 2011 10:23:20 GMT

If you are concerned about the Dropbox design flaw exposed by the dbClone attack, then have we got a link for you! The intrepid DB devs have tossed up a forum release which purports to fix all the thorny security issues. You can no longer…

RLRAA - Real Life Risk Assessment Acronyms

Mon, 18 Apr 2011 15:09:24 GMT

UPDATE: I have intentionally cross-posted this to my SIRA blog since the combined wit & intelligence of the folks there trumps anything I could do alone…

A Fully Operational OS X dbClone

Sun, 17 Apr 2011 21:51:27 GMT

A Fully Operational OS X dbClone — rud.is blog archive

dbClone "hack" for OS X

Sun, 17 Apr 2011 03:29:54 GMT

UPDATE: Check out the newer post on additional features. There has been much ado of late about Dropbox security with one of the most egregious issues being how easy it is to surreptitiously "clone" someone else's Dropbox by obtaining just…

"Flatten" HTML Content (i.e strip tags) in Cocoa/Objective-C [UPDATED] [BUG-FIX]

Tue, 12 Apr 2011 10:17:01 GMT

One of my most popular blog posts — 24,000 reads — in the old, co-mingled site was a short snippet on how to strip HTML tags from a block of content in Objective-C. It's been used by many-an-iOS developer (which was the original intent).…

Checkboxes Fail To Defend Maine Department of Conservation Against Attacks

Sat, 26 Mar 2011 12:09:09 GMT

I tweeted a quick note about the 2010 Maine Department of Conservation state park pass ordering system breach. The brief AP story indicated that the breach itself was caused by a malware infection on systems at their SasS provider…

"Repairing" Strict Transport Security in Chrome on OS X

Thu, 24 Mar 2011 09:53:54 GMT

One of my subdomains is for mail and I was using an easy DNS hack to point it to my hosted Gmail setup (just create a CNAME pointing to ghs.google.com). This stopped working for some folks this week and I've had no time to debug exactly…

Crossroad of ERM and the Parallels to IRM

Sat, 19 Mar 2011 14:32:00 GMT

I was reviewing the - er - highlights? - from the ninth ERM Symposium in Chicago over at Riskviews this morning and was intrigued by some of the parallels to the current situation in enterprise security risk management (the ERM symposium…

Remote Assistance/Information Gathering Aid - SupportDetails

Sat, 12 Mar 2011 22:44:13 GMT

I'm putting together a computer & online safety presentation for an upcoming talk at a senior center in Portsmouth (NH) and came across Support Details in my information hunting trek. This site makes it dirt simple to get basic information…

Behind The Mask : Supporting The New CIO Personas

Mon, 07 Mar 2011 23:02:36 GMT

This morning, @joshcorman linked to an article in the Harvard Business Review "The Conversation" blog that put forth the author's view of The Four Personas of the Next-Genereation CIO. The term persona is very Jungian and literally refers…

Micropwns :: Risk Microprobabilities for Infosec?

Sat, 05 Mar 2011 16:49:33 GMT

NOTE: This is a re-post from a topic I started on the SecurityMetrics & SIRA mailing lists. Wanted to broaden the discussion to anyone not on those (and, why aren't you on…

Post-theft/loss Response & Recovery With Evernote

Mon, 28 Feb 2011 21:57:12 GMT

Last night, the kids left the garage open after sledding all afternoon and I failed to perform my usual rounds due to still being horrendously ill. At some point between 23:00 & 05:30, miscreants did a snatch & grab on some electronics and…

Never A Better Time To Baseline

Sun, 27 Feb 2011 02:58:42 GMT

If you're preparing to install Windows 7 or Windows Server 2008 R2 Service Pack 1, now would be a good time to give Microsoft's Attack Surface Analyzer a spin. ASA takes a baseline snapshot of your system state and then lets you take…

The 5 Dysfunctions Of A Team

Fri, 25 Feb 2011 23:08:19 GMT

The following excerpt is from The Five Dysfunctions of a Team: A Leadership Fable by Patrick Lencioni. I wonder how many of you recognize traits like these on your own team(s), past or present. I can certainly point to these as being core…

Herding [Fire]sheep

Thu, 24 Feb 2011 01:04:05 GMT

By now, many non-IT and non-Security folk have heard of Firesheep, a tool written by @codebutler which allows anyone using Firefox on unprotected networks to capture and hjijack active sessions to popular social media sites (and other web…

Fixing "Are you sure" In Media Upload (WordPress)

Wed, 23 Feb 2011 18:32:51 GMT

If anyone is experiencing the "Are you sure..." problem when trying to upload media to your WordPress blog, it's usually due to a wonky plugin. In my case it was "Google Analytics 3 codes for WordPress". Hopefully this helps others who are…

Securing 'su' with Google Authenticator

Sat, 19 Feb 2011 16:21:38 GMT

Google's new do-it-yourself two-factor authentication (Google Authenticator) enables you to setup stronger logins on your linux system. Nick Wilkens (@nwilkens) has a good/quick tutorial up on his company's blog for acquiring, compiling…

For Everyone Who Thought I Was Just A Zombie

Fri, 18 Feb 2011 05:46:15 GMT

For Everyone Who Thought I Was Just A Zombie — rud.is blog archive

Metricon: Verification versus Validation

Tue, 15 Feb 2011 01:00:28 GMT

Use systemogram to show what systems are supposed to do (very cool visualization for differing views of "security systems…

Metricon: Measuring Metrics Programs (Why Aren't We?)

Mon, 14 Feb 2011 23:18:54 GMT

If metrics are valuable, why aren't we measuring them. Virtually no research on…

Metricon: Software Security's Futures Plural

Mon, 14 Feb 2011 20:31:56 GMT

UPDATE - 2011-02-26: Alphonso has posted his slides and BeeWise is…

Metricon: Name Server Log Data

Mon, 14 Feb 2011 19:46:25 GMT

"On the use of name server log data as input for security…

Metricon: Automated Incident Reporting

Mon, 14 Feb 2011 19:16:21 GMT

Goal: make finland mostly harmless to the rest of the…

Metricon: Critical Consumption Of Infosec Statistics

Mon, 14 Feb 2011 18:26:29 GMT

Every major infosec company publishes quarterly/yearly summary reports. Some based on survey, some based on real captured…

Metricon: Evidence Based Risk Management

Mon, 14 Feb 2011 17:51:21 GMT

Better management through better measurement Speakers: Wade Baker and Alex Hutton and Chris…

"Web Development Is Dangerous"

Sat, 12 Feb 2011 15:37:17 GMT

Those were the words that greeted me within five minutes of checking out the Flask microframework for Python web applications. I feel compelled to inline those four, short…

Quick Hits :: 2011-02-09

Thu, 10 Feb 2011 00:26:58 GMT

Quick Hits :: 2011-02-09 — rud.is blog archive

Quick Hits :: 2011-02-08

Wed, 09 Feb 2011 02:37:55 GMT

Quick Hits :: 2011-02-08 — rud.is blog archive

Quick Hits :: 2011-01-07

Tue, 08 Feb 2011 00:35:36 GMT

Programming Interesting points/counterpoints on the efficacy of Node.js being tied so closely to the V8 javascript…

'Spaces' for Windows

Sat, 05 Feb 2011 19:06:45 GMT

UPDATE \[2011-02-05\] Added VirtuaWin to the list thanks to a tip by @ken5m1th. I've been setting up a relatively new 64-bit Windows 7 Ultimate machine and decided to see if the virtual desktops landscape had changed much in the recent…

AwesomeChartJS Meets Microsoft Security Bulletins

Thu, 03 Feb 2011 21:43:24 GMT

I wanted to play with the AwesomeChartJS library and figured an interesting way to do that was to use it to track Microsoft Security Bulletins this year. While I was drawn in by just how simple it is to craft basic charts, that simplicity…

Backup Workflow

Wed, 02 Feb 2011 21:36:55 GMT

I'm \[still\] a die-hard OS X user, despite the increasing gatekeeper motif Apple is sporting these days. My main computer is a MacBook Pro which I would stupidly run back into a burning building to rescue. Everything is on it. Everything.…

HDMI Overscan Tweak For Wimpy Windows Font Display

Wed, 02 Feb 2011 20:15:23 GMT

I recently hooked up a Windows 7 box to my Dell ST2310 monitor and was surprised at just how horrid the fonts looked, especially since my MacBook Pro looks fantastic using both DVI and HDMI with the display. I even tried all the ClearType…

Bootstrapping The New Site

Tue, 01 Feb 2011 22:10:27 GMT

A swirling mass of shiny new objects has caught my attention this year. Carving out a more dedicated space to document, opine and…