{"id":12120,"date":"2019-04-03T06:04:50","date_gmt":"2019-04-03T11:04:50","guid":{"rendered":"https:\/\/rud.is\/b\/?p=12120"},"modified":"2019-04-03T06:04:50","modified_gmt":"2019-04-03T11:04:50","slug":"wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package","status":"publish","type":"post","link":"https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/","title":{"rendered":"Wicked Fast, Accurate Quantiles Using \u2018t-Digests\u2019 in R with the {tdigest} Package"},"content":{"rendered":"<p>@ted_dunning <a href=\"https:\/\/twitter.com\/ted_dunning\/status\/1073802202639036416?s=20\">recently updated<\/a> the t-Digest algorithm he created back in 2013. What is this &#8220;t-digest&#8221;? Fundamentally, it is a probabilistic data structure for estimating any percentile of distributed\/streaming data. Ted explains it quite elegantly in this short video:<\/p>\n<p><center><\/p>\n<p><iframe loading=\"lazy\" width=\"560\" height=\"315\" src=\"https:\/\/www.youtube.com\/embed\/B0dMc0t7K1g\" frameborder=\"0\" allow=\"accelerometer; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<p><\/center><\/p>\n<p>Said video has a <a href=\"https:\/\/mapr.com\/blog\/better-anomaly-detection-t-digest-whiteboard-walkthrough\/\">full transcript<\/a> as well.<\/p>\n<p>T-digests have been baked into many &#8220;big data&#8221; analytics ecosystems for a while but I hadn&#8217;t seen any R packages for them (ref any in a comment if you do know of some) so I wrapped one of <a href=\"https:\/\/github.com\/ajwerner\/tdigestc\">the low-level implementation libraries by ajwerner<\/a> into a diminutive R package boringly, but appropriately named <code>tdigest<\/code>:<\/p>\n<ul>\n<li><a href=\"https:\/\/git.sr.ht\/~hrbrmstr\/tdigest\">SourceHut<\/a><\/li>\n<li><a href=\"https:\/\/gitlab.com\/hrbrmstr\/tdigest\">GitLab<\/a><\/li>\n<li><a href=\"https:\/\/cinc.rud.is\/web\/packages\/tdigest\/\">CINC<\/a> (The CINC repo has R 3.5 package binaries for Windows and macOS.)<\/li>\n<li><a href=\"https:\/\/github.com\/hrbrmstr\/tdigest\">GitUgh<\/a><\/li>\n<\/ul>\n<p>There are wrappers for the low-level accumulators and quantile\/value extractors along with vectorised functions for creating t-digest objects and retrieving quantiles from them (including a <code>tdigest<\/code> S3 method for <code>stats::quantile()<\/code>).<\/p>\n<p>This:<\/p>\n<pre><code class=\"language-r\">install.packages(\"tdigest\", repos=\"https:\/\/cinc.rud.is\/\")\n<\/code><\/pre>\n<p>will install from source or binaries onto your system(s).<\/p>\n<h3>Basic Ops<\/h3>\n<p>The low-level interface is more useful in &#8220;streaming&#8221; operations (i.e. accumulating input over time):<\/p>\n<pre><code class=\"language-r\">set.seed(2019-04-03)\n\ntd &lt;- td_create()\n\nfor (i in 1:100000) {\n  td_add(td, sample(100, 1), 1)\n}\n\nquantile(td)\n## [1]   1.00000  25.62222  53.09883  74.75522 100.00000\n<\/code><\/pre>\n<h3>More R-like Ops<\/h3>\n<p>Vectorisation is the name of the game in R and we can use <code>tdigest()<\/code> to work in a vectorised manner:<\/p>\n<pre><code class=\"language-r\">set.seed(2019-04-03)\n\nx &lt;- sample(100, 1000000, replace=TRUE)\n\ntd &lt;- tdigest(x)\n\nquantile(td)\n## [1]   1.00000  25.91914  50.79468  74.76439 100.00000\n<\/code><\/pre>\n<h3>Need for Speed<\/h3>\n<p>The t-digest algorithm was designed for both streaming operations and speed. It&#8217;s pretty, darned fast:<\/p>\n<pre><code class=\"language-r\">microbenchmark::microbenchmark(\n  tdigest = tquantile(td, c(0, 0.01, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.99, 1)),\n  r_quantile = quantile(x, c(0, 0.01, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.99, 1))\n)\n## Unit: microseconds\n##        expr      min         lq        mean    median       uq       max neval\n##     tdigest    22.81    26.6525    48.70123    53.355    63.31    151.29   100\n##  r_quantile 57675.34 59118.4070 62992.56817 60488.932 64731.23 160130.50   100\n<\/code><\/pre>\n<p>Note that &#8220;accurate&#8221; is not the same thing as &#8220;precise&#8221;, so regular quantile ops in R will be close to what t-digest computes, but not always exactly the same.<\/p>\n<h3>FIN<\/h3>\n<p>This was a quick (but, complete) wrapper and could use some tyre kicking. I&#8217;ve a mind to add serialization to the C implementation so I can then enable [de]serialization on the R-side since that would (IMO) make t-digest ops more useful in an R-context, especially since you can merge two different t-digests.<\/p>\n<p>As always, code\/PR where you want to and file issues with any desired functionality\/enhancements.<\/p>\n<p>Also, whomever started the braces notation for package names (e.g. {ggplot2}): brilliant!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>@ted_dunning recently updated the t-Digest algorithm he created back in 2013. What is this &#8220;t-digest&#8221;? Fundamentally, it is a probabilistic data structure for estimating any percentile of distributed\/streaming data. Ted explains it quite elegantly in this short video: Said video has a full transcript as well. T-digests have been baked into many &#8220;big data&#8221; analytics [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":3,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":""},"categories":[91],"tags":[],"class_list":["post-12120","post","type-post","status-publish","format-standard","hentry","category-r"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.3 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Wicked Fast, Accurate Quantiles Using \u2018t-Digests\u2019 in R with the {tdigest} Package - rud.is<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Wicked Fast, Accurate Quantiles Using \u2018t-Digests\u2019 in R with the {tdigest} Package - rud.is\" \/>\n<meta property=\"og:description\" content=\"@ted_dunning recently updated the t-Digest algorithm he created back in 2013. What is this &#8220;t-digest&#8221;? Fundamentally, it is a probabilistic data structure for estimating any percentile of distributed\/streaming data. Ted explains it quite elegantly in this short video: Said video has a full transcript as well. T-digests have been baked into many &#8220;big data&#8221; analytics [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/\" \/>\n<meta property=\"og:site_name\" content=\"rud.is\" \/>\n<meta property=\"article:published_time\" content=\"2019-04-03T11:04:50+00:00\" \/>\n<meta name=\"author\" content=\"hrbrmstr\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"hrbrmstr\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2019\\\/04\\\/03\\\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2019\\\/04\\\/03\\\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\\\/\"},\"author\":{\"name\":\"hrbrmstr\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\"},\"headline\":\"Wicked Fast, Accurate Quantiles Using \u2018t-Digests\u2019 in R with the {tdigest} Package\",\"datePublished\":\"2019-04-03T11:04:50+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2019\\\/04\\\/03\\\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\\\/\"},\"wordCount\":339,\"commentCount\":6,\"publisher\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\"},\"articleSection\":[\"R\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/rud.is\\\/b\\\/2019\\\/04\\\/03\\\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2019\\\/04\\\/03\\\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\\\/\",\"url\":\"https:\\\/\\\/rud.is\\\/b\\\/2019\\\/04\\\/03\\\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\\\/\",\"name\":\"Wicked Fast, Accurate Quantiles Using \u2018t-Digests\u2019 in R with the {tdigest} Package - rud.is\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#website\"},\"datePublished\":\"2019-04-03T11:04:50+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2019\\\/04\\\/03\\\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/rud.is\\\/b\\\/2019\\\/04\\\/03\\\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2019\\\/04\\\/03\\\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/rud.is\\\/b\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Wicked Fast, Accurate Quantiles Using \u2018t-Digests\u2019 in R with the {tdigest} Package\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#website\",\"url\":\"https:\\\/\\\/rud.is\\\/b\\\/\",\"name\":\"rud.is\",\"description\":\"&quot;In God we trust. All others must bring data&quot;\",\"publisher\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/rud.is\\\/b\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\",\"name\":\"hrbrmstr\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\",\"width\":460,\"height\":460,\"caption\":\"hrbrmstr\"},\"logo\":{\"@id\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\"},\"description\":\"Don't look at me\u2026I do what he does \u2014 just slower. #rstats avuncular \u2022 ?Resistance Fighter \u2022 Cook \u2022 Christian \u2022 [Master] Chef des Donn\u00e9es de S\u00e9curit\u00e9 @ @rapid7\",\"sameAs\":[\"http:\\\/\\\/rud.is\"],\"url\":\"https:\\\/\\\/rud.is\\\/b\\\/author\\\/hrbrmstr\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Wicked Fast, Accurate Quantiles Using \u2018t-Digests\u2019 in R with the {tdigest} Package - rud.is","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/","og_locale":"en_US","og_type":"article","og_title":"Wicked Fast, Accurate Quantiles Using \u2018t-Digests\u2019 in R with the {tdigest} Package - rud.is","og_description":"@ted_dunning recently updated the t-Digest algorithm he created back in 2013. What is this &#8220;t-digest&#8221;? Fundamentally, it is a probabilistic data structure for estimating any percentile of distributed\/streaming data. Ted explains it quite elegantly in this short video: Said video has a full transcript as well. T-digests have been baked into many &#8220;big data&#8221; analytics [&hellip;]","og_url":"https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/","og_site_name":"rud.is","article_published_time":"2019-04-03T11:04:50+00:00","author":"hrbrmstr","twitter_card":"summary_large_image","twitter_misc":{"Written by":"hrbrmstr","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/#article","isPartOf":{"@id":"https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/"},"author":{"name":"hrbrmstr","@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"headline":"Wicked Fast, Accurate Quantiles Using \u2018t-Digests\u2019 in R with the {tdigest} Package","datePublished":"2019-04-03T11:04:50+00:00","mainEntityOfPage":{"@id":"https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/"},"wordCount":339,"commentCount":6,"publisher":{"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"articleSection":["R"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/","url":"https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/","name":"Wicked Fast, Accurate Quantiles Using \u2018t-Digests\u2019 in R with the {tdigest} Package - rud.is","isPartOf":{"@id":"https:\/\/rud.is\/b\/#website"},"datePublished":"2019-04-03T11:04:50+00:00","breadcrumb":{"@id":"https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/rud.is\/b\/"},{"@type":"ListItem","position":2,"name":"Wicked Fast, Accurate Quantiles Using \u2018t-Digests\u2019 in R with the {tdigest} Package"}]},{"@type":"WebSite","@id":"https:\/\/rud.is\/b\/#website","url":"https:\/\/rud.is\/b\/","name":"rud.is","description":"&quot;In God we trust. All others must bring data&quot;","publisher":{"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/rud.is\/b\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886","name":"hrbrmstr","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","url":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","contentUrl":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","width":460,"height":460,"caption":"hrbrmstr"},"logo":{"@id":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1"},"description":"Don't look at me\u2026I do what he does \u2014 just slower. #rstats avuncular \u2022 ?Resistance Fighter \u2022 Cook \u2022 Christian \u2022 [Master] Chef des Donn\u00e9es de S\u00e9curit\u00e9 @ @rapid7","sameAs":["http:\/\/rud.is"],"url":"https:\/\/rud.is\/b\/author\/hrbrmstr\/"}]}},"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p23idr-39u","jetpack_likes_enabled":true,"jetpack-related-posts":[{"id":3679,"url":"https:\/\/rud.is\/b\/2015\/09\/19\/a-package-full-o-pirates-makin-interactive-pirate-maps-in-arrrrrrstats\/","url_meta":{"origin":12120,"position":0},"title":"A Package Full o&#8217; Pirates &#038; Makin&#8217; Interactive Pirate Maps in arrrrrRstats","author":"hrbrmstr","date":"2015-09-19","format":false,"excerpt":"Avast, me hearties! It's time four t' annual International Talk Like a Pirate Day #rstats post! (OK, I won't make you suffer continuous pirate-speak for the entire post) I tried to be a bit more practical this year and have two treasuRe chests for you to (hopefully) enjoy. A Package\u2026","rel":"","context":"In &quot;cartography&quot;","block_context":{"text":"cartography","link":"https:\/\/rud.is\/b\/category\/cartography\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3460,"url":"https:\/\/rud.is\/b\/2015\/06\/15\/metricsgraphics-0-8-5-is-now-on-cran\/","url_meta":{"origin":12120,"position":1},"title":"metricsgraphics 0.8.5 is now on CRAN!","author":"hrbrmstr","date":"2015-06-15","format":false,"excerpt":"I'm super-pleased to announce that the Benevolent CRAN Overlords [accepted the metricsgraphics package](http:\/\/cran.r-project.org\/web\/packages\/metricsgraphics\/index.html) into CRAN over the weekend. Now, you no longer need to rely on github\/devtools to use [MetricsGraphics.js](http:\/\/metricsgraphicsjs.org\/) charts from your R scripts. If you're not familiar with `htmlwidgets`, take a look at [the official site for them](http:\/\/www.htmlwidgets.org\/).\u2026","rel":"","context":"In &quot;d3&quot;","block_context":{"text":"d3","link":"https:\/\/rud.is\/b\/category\/d3\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2637,"url":"https:\/\/rud.is\/b\/2013\/09\/08\/rforecastio-simple-r-package-to-access-forecast-io-weather-data\/","url_meta":{"origin":12120,"position":2},"title":"Rforecastio &#8211; Simple R Package To Access forecast.io Weather Data","author":"hrbrmstr","date":"2013-09-08","format":false,"excerpt":"It doesn't get much better for me than when I can combine R and weather data in new ways. I've got something brewing with my Nest thermostat and needed to get some current wx readings plus forecast data. I could have chosen a number of different sources or API's but\u2026","rel":"","context":"In &quot;Development&quot;","block_context":{"text":"Development","link":"https:\/\/rud.is\/b\/category\/development\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2933,"url":"https:\/\/rud.is\/b\/2014\/02\/20\/using-twitter-as-a-data-source-for-monitoring-password-dumps\/","url_meta":{"origin":12120,"position":3},"title":"Using Twitter as a Data Source For Monitoring Password Dumps","author":"hrbrmstr","date":"2014-02-20","format":false,"excerpt":"I shot a quick post over at the [Data Driven Security blog](http:\/\/bit.ly\/1hyqJiT) explaining how to separate Twitter data gathering from R code via the Ruby `t` ([github repo](https:\/\/github.com\/sferik\/t)) command. Using `t` frees R code from having to be a Twitter processor and lets the analyst focus on analysis and visualization,\u2026","rel":"","context":"In &quot;Data Analysis&quot;","block_context":{"text":"Data Analysis","link":"https:\/\/rud.is\/b\/category\/data-analysis-2\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":11911,"url":"https:\/\/rud.is\/b\/2019\/02\/14\/using-the-ropendata-r-package-to-access-petabytes-of-free-internet-telemetry-data-from-rapid7\/","url_meta":{"origin":12120,"position":4},"title":"Using the ropendata R Package to Access Petabytes of Free Internet Telemetry Data from Rapid7","author":"hrbrmstr","date":"2019-02-14","format":false,"excerpt":"I've got a post up over at $DAYJOB's blog on using the ropendata? package to access the ginormous and ever-increasing amount of internet telemetry (scan) data via the Rapid7 Open Data API. It's super-R-code-heavy but renders surprisingly well in Ghost (the blogging platform we use at work) and covers everything\u2026","rel":"","context":"In &quot;R&quot;","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":12430,"url":"https:\/\/rud.is\/b\/2019\/07\/27\/two-new-testing-themed-addins-one-new-and-one-updated-cran-package\/","url_meta":{"origin":12120,"position":5},"title":"Two New Testing-themed Addins + One New and One Updated CRAN Package","author":"hrbrmstr","date":"2019-07-27","format":false,"excerpt":"It's been yet-another weirdly busy summer but I'm finally catching up on noting some recent-ish developments in the blog. First up is a full rewrite of the {wand} pacakge which still uses magic but is 100% R code (vs a mix of compiled C and R code) and now works\u2026","rel":"","context":"In &quot;R&quot;","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts\/12120","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/comments?post=12120"}],"version-history":[{"count":0,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts\/12120\/revisions"}],"wp:attachment":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/media?parent=12120"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/categories?post=12120"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/tags?post=12120"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}