{"id":5841,"date":"2017-04-23T17:27:01","date_gmt":"2017-04-23T22:27:01","guid":{"rendered":"https:\/\/rud.is\/b\/?p=5841"},"modified":"2018-03-07T17:19:23","modified_gmt":"2018-03-07T22:19:23","slug":"decomposing-composers-with-r","status":"publish","type":"post","link":"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/","title":{"rendered":"Decomposing Composers with R"},"content":{"rendered":"

The intrepid @ma_salmon cranked out another blog post<\/a>, remixing classical music schedule data from Radio Swiss Classic<\/a>. It’s a fun post and you should read it before continuing here.<\/p>\n

Seriously, click the link and go read it before continuing.<\/p>\n

No, I mean it. Click the link or the rest of this makes no sense ;-)<\/p>\n

OK, good. You finally read her ? post.<\/p>\n

Now, I’m riffing off of said post here for four reasons. Three of the reasons are short, one is longer.<\/p>\n

The first, short one is: be kind to web servers when scraping<\/em>. If you ran a site and suddenly got hit with 3,000+ immediately sequential requests you might not be able to handle it depending on your server config. At a minimum<\/em> add a Sys.sleep(sample(seq(0,1,0.25), 1))<\/code> before each sequential scrape and \u2014 if you can spare the time \u2014 sample(5,1)<\/code> would be even better for a delay.<\/p>\n

The second, short one is: purrr::safely()<\/code> is your bff when it comes to xml2::read_html()<\/code> and other network-ops. The internet is fundamentally broken. Nodes die. Pages get lost. Links rot. You have to be able to handle exceptions and if you define something like s_read_html <- safely(read_html)<\/code> then when you do s_read_html(\"https:\/\/example.com\/\")<\/code> the $result<\/code> component will be NULL<\/code> if the network request failed but will contain valid, parsed HTML if it succeeds. It is silent by default and works quite well (as we’ll see below).<\/p>\n

The third, short one is: MPGA (Make Progress-bars Great Again). dplyr::progress_estimated()<\/code> can really simplify the usage of progress bars in purrr<\/code> calls (drop a note in the comments if the code is confusing and I’ll add some expository).<\/p>\n

The last requires the code example for context:<\/p>\n

library(rvest)\r\nlibrary(stringi)\r\nlibrary(lubridate)\r\nlibrary(tidyverse)\r\n\r\ns_read_html <- safely(read_html)\r\n\r\n# helper for brevity\r\nxtract_nodes <- function(node, css) {\r\n  html_nodes(node, css) %>% html_text(trim = TRUE)\r\n}\r\n\r\nget_one_day_program <- function(date=Sys.Date(),\r\n                                base_url="http:\/\/www.radioswissclassic.ch\/en\/music-programme\/search\/%s",\r\n                                pb=NULL) {\r\n\r\n  if (!is.null(pb)) pb$tick()$print()\r\n\r\n  Sys.sleep(sample(seq(0,1,0.25), 1)) # ideally, make this sample(5,1)\r\n\r\n  date <- ymd(date) # handles case where input is character ISO date\r\n\r\n  pg <- s_read_html(sprintf(base_url, format(date, "%Y%m%d")))\r\n\r\n  if (!is.null(pg$result)) {\r\n\r\n    data_frame(\r\n\r\n      date = date,\r\n      duration = xtract_nodes(pg$result, 'div[class="playlist"] *\r\n                                            span[class="time hidden-xs"]') %>% hm() %>% as.numeric(),\r\n      artist = xtract_nodes(pg$result, 'div[class="playlist"] * span[class="titletag"]'),\r\n      title = xtract_nodes(pg$result, 'div[class="playlist"] * span[class="artist"]'),\r\n\r\n      hour = purrr::map(0:23, ~{\r\n        if (.x<23) {\r\n          nod <- html_nodes(pg$result,\r\n                             xpath=sprintf(".\/\/div[@id='%02d']\/following-sibling::div[contains(@class, 'item-row')\r\n                                                                 and (following-sibling::div[@id='%02d'])]", .x, .x+1))\r\n        } else {\r\n          nod <- html_nodes(pg$result,\r\n                            xpath=sprintf(".\/\/div[@id='%02d']\/following-sibling::div[contains(@class, 'item-row')]", .x))\r\n        }\r\n        rep(.x, length(nod))\r\n      }) %>%\r\n        flatten_int()\r\n\r\n    )\r\n\r\n  } else {\r\n    closeAllConnections()\r\n    NULL\r\n  }\r\n\r\n}\r\n\r\nsearch_dates <- seq(from = ymd("2008-09-01"), to = ymd("2017-04-22"), by = "1 day")\r\n\r\npb <- progress_estimated(length(search_dates[1:5]))\r\nprograms_df <- map_df(search_dates[1:5], get_one_day_program, pb=pb)\r\nprograms_df\r\n## # A tibble: 825 \u00d7 5\r\n##          date duration                    artist                                                                         title  hour\r\n##        <date>    <dbl>                     <chr>                                                                         <chr> <int>\r\n## 1  2008-09-01       60   Franz Anton Hoffmeister "Andante grazioso" From Flute Quartet In A Major (After Mozart's KV 331) (CH)     0\r\n## 2  2008-09-01      360     Johann Nepomuk Hummel                              "Rondo brillante" Op. 56 For Piano And Orchestra     0\r\n## 3  2008-09-01     1380            Franz Schubert                       "Andante con moto" From Symphony No. 9 In C Major D 944     0\r\n## 4  2008-09-01     2340       Camille Saint-Sa\u00ebns                                       Violin Concerto No. 1 In A Major Op. 20     0\r\n## 5  2008-09-01     3000        Alexander Scriabin                                           Nocturne In A Flat Major Op. posth.     0\r\n## 6  2008-09-01     3180        Alexander Glazunov                                          Valse From "Sc\u00e8nes de ballet" Op. 52     0\r\n## 7  2008-09-01     3540 Carl Philipp Emanuel Bach                                                           Symphony In G Major     0\r\n## 8  2008-09-01     4200            Giuseppe Verdi                      "O Signore, dal tetto natio" From The Opera "I Lombardi"     1\r\n## 9  2008-09-01     4440             Franz Krommer                                 Clarinet Concerto In E Flat Major Op. 36 (CH)     1\r\n## 10 2008-09-01     5820            Georges Onslow             "Andantino molto cantabile" From Symphony No. 4 In G Major Op. 71     1\r\n## # ... with 815 more rows<\/code><\/pre>\n
One of the reasons Ma\u00eblle created her post was to use XPath<\/a>. Now, I was around when XML was defined and I have a sad, long history with the format, so XPath & I are old friends<\/strike> adversaries. However, there are simpler ways to target some of the nodes.<\/p>\n
xpath=\"\/\/span[@class='time hidden-xs']\/\/text()\"<\/code> is ++gd XPath but it doesn’t need to be if we switch to using html_nodes()<\/code> which will automatically translate CSS selectors to XPath for us. That bit of XPath turns into div[class=\"playlist\"] * span[class=\"time hidden-xs\"]<\/code>. Why the extra selector at the beginning? Read on!<\/p>\n
div[class=\"playlist\"] * span[class=\"time hidden-xs\"]<\/code> actually translates to the following XPath:<\/p>\n
selectr::css_to_xpath('div[class="playlist"] * span[class="time hidden-xs"]')\r\n## [1] "descendant-or-self::div[@class = 'playlist']\/descendant::*\/descendant::span[@class = 'time hidden-xs']"<\/code><\/pre>\n
I use the parent playlist<\/code> <div><\/code> because a few of the code bits in Ma\u00eblle’s post have to subtract away the last node because the XPath expression is a bit too greedy and also gets the “now playing” info vs just the “what played that day” info. It’s not strictly necessary for the time-code but it is for the artist & title. You can see that it simplifies the scraping a bit.<\/p>\n
However, we can<\/em> use XPath for to scrape the “hour the song played”  and use it to fill the resultant data frame.<\/p>\n
This .\/\/div[@id='%02d']\/following-sibling::div[contains(@class, 'item-row') and (following-sibling::div[@id='%02d'])]<\/code> is not the most complex XPath but it is pretty gnarly, yet it also shows the power of XPath. What we’re doing in that purrr::map()<\/code> call (which said XPath is in) is:<\/p>\n
    \n
  • if the hour is 0:22<\/code>, then use get all the sibling target nodes between one <div id=\"hh\"><\/code> and the next <div id=\"hh\"><\/code>.<\/li>\n
  • if the hour is 23<\/code>, then get all the target nodes until there are no sibling <\/li>\n
  • for either result, make an integer vector containing the hour repeated n<\/code> times (n<\/code> being the length of the number of songs played in the hour)<\/li>\n
  • flatten it all into one big integer vector<\/li>\n<\/ul>\n
    (also: note that whitespace is your bff as well when it comes to formatting XPath queries)<\/p>\n
    If any read_html<\/code> request is “bad” NULL<\/code> will be returned instead of a data_frame<\/code>, which purrr::map_df()<\/code> will ignore.<\/p>\n
    I only did 5 scrapes since I won’t be using the data, but it’s working well on other random sequences I tried.<\/p>\n
    I tossed in a few more alternative ways to get some of the data, which you can pick up on if you compare the each code bits to each other.<\/p>\n
    Drop any questions, jibes or better XPath queries (once you post an XPath query on the internet the XPath wonks \u2014 like me ? \u2014 come out of hiding to prey on innocent bloggers) in the comments.<\/p>\n","protected":false},"excerpt":{"rendered":"
    The intrepid @ma_salmon cranked out another blog post, remixing classical music schedule data from Radio Swiss Classic. It’s a fun post and you should read it before continuing here. Seriously, click the link and go read it before continuing. No, I mean it. Click the link or the rest of this makes no sense ;-) […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":3,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":""},"categories":[732],"tags":[810],"class_list":["post-5841","post","type-post","status-publish","format-standard","hentry","category-xml","tag-post"],"yoast_head":"\nDecomposing Composers with R - rud.is<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Decomposing Composers with R - rud.is\" \/>\n<meta property=\"og:description\" content=\"The intrepid @ma_salmon cranked out another blog post, remixing classical music schedule data from Radio Swiss Classic. It’s a fun post and you should read it before continuing here. Seriously, click the link and go read it before continuing. No, I mean it. Click the link or the rest of this makes no sense ;-) […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/\" \/>\n<meta property=\"og:site_name\" content=\"rud.is\" \/>\n<meta property=\"article:published_time\" content=\"2017-04-23T22:27:01+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-03-07T22:19:23+00:00\" \/>\n<meta name=\"author\" content=\"hrbrmstr\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"hrbrmstr\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/\"},\"author\":{\"name\":\"hrbrmstr\",\"@id\":\"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886\"},\"headline\":\"Decomposing Composers with R\",\"datePublished\":\"2017-04-23T22:27:01+00:00\",\"dateModified\":\"2018-03-07T22:19:23+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/\"},\"wordCount\":687,\"commentCount\":4,\"publisher\":{\"@id\":\"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886\"},\"keywords\":[\"post\"],\"articleSection\":[\"xml\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/\",\"url\":\"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/\",\"name\":\"Decomposing Composers with R - rud.is\",\"isPartOf\":{\"@id\":\"https:\/\/rud.is\/b\/#website\"},\"datePublished\":\"2017-04-23T22:27:01+00:00\",\"dateModified\":\"2018-03-07T22:19:23+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/rud.is\/b\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Decomposing Composers with R\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/rud.is\/b\/#website\",\"url\":\"https:\/\/rud.is\/b\/\",\"name\":\"rud.is\",\"description\":\""In God we trust. All others must bring data"\",\"publisher\":{\"@id\":\"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/rud.is\/b\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886\",\"name\":\"hrbrmstr\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1\",\"url\":\"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1\",\"width\":460,\"height\":460,\"caption\":\"hrbrmstr\"},\"logo\":{\"@id\":\"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1\"},\"description\":\"Don't look at me\u2026I do what he does \u2014 just slower. #rstats avuncular \u2022 ?Resistance Fighter \u2022 Cook \u2022 Christian \u2022 [Master] Chef des Donn\u00e9es de S\u00e9curit\u00e9 @ @rapid7\",\"sameAs\":[\"http:\/\/rud.is\"],\"url\":\"https:\/\/rud.is\/b\/author\/hrbrmstr\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Decomposing Composers with R - rud.is","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/","og_locale":"en_US","og_type":"article","og_title":"Decomposing Composers with R - rud.is","og_description":"The intrepid @ma_salmon cranked out another blog post, remixing classical music schedule data from Radio Swiss Classic. It’s a fun post and you should read it before continuing here. Seriously, click the link and go read it before continuing. No, I mean it. Click the link or the rest of this makes no sense ;-) […]","og_url":"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/","og_site_name":"rud.is","article_published_time":"2017-04-23T22:27:01+00:00","article_modified_time":"2018-03-07T22:19:23+00:00","author":"hrbrmstr","twitter_card":"summary_large_image","twitter_misc":{"Written by":"hrbrmstr","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/#article","isPartOf":{"@id":"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/"},"author":{"name":"hrbrmstr","@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"headline":"Decomposing Composers with R","datePublished":"2017-04-23T22:27:01+00:00","dateModified":"2018-03-07T22:19:23+00:00","mainEntityOfPage":{"@id":"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/"},"wordCount":687,"commentCount":4,"publisher":{"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"keywords":["post"],"articleSection":["xml"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/","url":"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/","name":"Decomposing Composers with R - rud.is","isPartOf":{"@id":"https:\/\/rud.is\/b\/#website"},"datePublished":"2017-04-23T22:27:01+00:00","dateModified":"2018-03-07T22:19:23+00:00","breadcrumb":{"@id":"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/rud.is\/b\/2017\/04\/23\/decomposing-composers-with-r\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/rud.is\/b\/"},{"@type":"ListItem","position":2,"name":"Decomposing Composers with R"}]},{"@type":"WebSite","@id":"https:\/\/rud.is\/b\/#website","url":"https:\/\/rud.is\/b\/","name":"rud.is","description":""In God we trust. All others must bring data"","publisher":{"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/rud.is\/b\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886","name":"hrbrmstr","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","url":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","contentUrl":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","width":460,"height":460,"caption":"hrbrmstr"},"logo":{"@id":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1"},"description":"Don't look at me\u2026I do what he does \u2014 just slower. #rstats avuncular \u2022 ?Resistance Fighter \u2022 Cook \u2022 Christian \u2022 [Master] Chef des Donn\u00e9es de S\u00e9curit\u00e9 @ @rapid7","sameAs":["http:\/\/rud.is"],"url":"https:\/\/rud.is\/b\/author\/hrbrmstr\/"}]}},"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p23idr-1wd","jetpack_likes_enabled":true,"jetpack-related-posts":[{"id":3141,"url":"https:\/\/rud.is\/b\/2014\/11\/27\/power-outage-impact-choropleths-in-5-steps-in-r-featuring-rvest-rstudio-projects\/","url_meta":{"origin":5841,"position":0},"title":"Power Outage Impact Choropleths In 5 Steps in R (featuring rvest & RStudio “Projects”)","author":"hrbrmstr","date":"2014-11-27","format":false,"excerpt":"I and @awpiii were trading news about the power outages in Maine & New Hampshire last night and he tweeted the link to the @PSNH [Outage Map](http:\/\/www.psnh.com\/outage\/). As if the Bing Maps tiles weren't bad enough, the use of a categorical color scale instead of a sequential one[[1](http:\/\/earthobservatory.nasa.gov\/blogs\/elegantfigures\/2011\/05\/20\/qualitative-vs-sequential-color-scales\/)] caused sufficient\u2026","rel":"","context":"In "Charts & Graphs"","block_context":{"text":"Charts & Graphs","link":"https:\/\/rud.is\/b\/category\/charts-graphs\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":7637,"url":"https:\/\/rud.is\/b\/2017\/12\/20\/r%e2%81%b6-series-random-sampling-from-apache-drill-tables-with-r-sergeant\/","url_meta":{"origin":5841,"position":1},"title":"R\u2076 Series \u2014 Random Sampling From Apache Drill Tables With R & sergeant","author":"hrbrmstr","date":"2017-12-20","format":false,"excerpt":"(For first-timers, R\u2076 tagged posts are short & sweet with minimal expository; R\u2076 feed) At work-work I mostly deal with medium-to-large-ish data. I often want to poke at new or existing data sets w\/o working across billions of rows. I also use Apache Drill for much of my exploratory work.\u2026","rel":"","context":"In "Apache Drill"","block_context":{"text":"Apache Drill","link":"https:\/\/rud.is\/b\/category\/apache-drill\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":12120,"url":"https:\/\/rud.is\/b\/2019\/04\/03\/wicked-fast-accurate-quantiles-using-t-digests-in-r-with-the-tdigest-package\/","url_meta":{"origin":5841,"position":2},"title":"Wicked Fast, Accurate Quantiles Using \u2018t-Digests\u2019 in R with the {tdigest} Package","author":"hrbrmstr","date":"2019-04-03","format":false,"excerpt":"@ted_dunning recently updated the t-Digest algorithm he created back in 2013. What is this \"t-digest\"? Fundamentally, it is a probabilistic data structure for estimating any percentile of distributed\/streaming data. Ted explains it quite elegantly in this short video: Said video has a full transcript as well. T-digests have been baked\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":5131,"url":"https:\/\/rud.is\/b\/2017\/03\/10\/making-a-case-for-case_when\/","url_meta":{"origin":5841,"position":3},"title":"Making a Case for case_when","author":"hrbrmstr","date":"2017-03-10","format":false,"excerpt":"This is a brief (and likely obvious, for some folks) post on the dplyr::case_when() function. Part of my work-work is dealing with data from internet scans. When we're performing a deeper inspection of a particular internet protocol or service we try to capture as much system and service metadata as\u2026","rel":"","context":"In "dplyr"","block_context":{"text":"dplyr","link":"https:\/\/rud.is\/b\/category\/dplyr\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/03\/Cursor_and_RStudio.png?fit=1200%2C464&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/03\/Cursor_and_RStudio.png?fit=1200%2C464&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/03\/Cursor_and_RStudio.png?fit=1200%2C464&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/03\/Cursor_and_RStudio.png?fit=1200%2C464&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/03\/Cursor_and_RStudio.png?fit=1200%2C464&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":12114,"url":"https:\/\/rud.is\/b\/2019\/03\/26\/rome-was-not-built-in-a-day-but-widgetcard-was\/","url_meta":{"origin":5841,"position":4},"title":"Rome Was Not Built In A Day But widgetcard Was!","author":"hrbrmstr","date":"2019-03-26","format":false,"excerpt":"I saw a second post on turning htmlwidgets into interactive Twitter Player cards and felt somewhat compelled to make creating said entities a bit easier so posited the following: Wld this be useful packaged up, #rstats?https:\/\/t.co\/sfqlWnEeJVhttps:\/\/t.co\/troKzmzTNv(TLDR\/V: Single function to turn an HTML widget into a deployable interactive Twitter card) pic.twitter.com\/uahB52YfE2\u2014\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":6496,"url":"https:\/\/rud.is\/b\/2017\/09\/30\/identify-analyze-web-site-tech-stacks-with-rappalyzer\/","url_meta":{"origin":5841,"position":5},"title":"Identify & Analyze Web Site Tech Stacks With rappalyzer","author":"hrbrmstr","date":"2017-09-30","format":false,"excerpt":"Modern websites are complex beasts. They house photo galleries, interactive visualizations, web fonts, analytics code and other diverse types of content. Despite the potential for diversity, many web sites share similar \"tech stacks\" --- the components that come together to make them what they are. These stacks consist of web\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Viewer_Zoom.png?fit=1198%2C1200&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Viewer_Zoom.png?fit=1198%2C1200&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Viewer_Zoom.png?fit=1198%2C1200&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Viewer_Zoom.png?fit=1198%2C1200&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Viewer_Zoom.png?fit=1198%2C1200&ssl=1&resize=1050%2C600 3x"},"classes":[]}],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts\/5841","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/comments?post=5841"}],"version-history":[{"count":0,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts\/5841\/revisions"}],"wp:attachment":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/media?parent=5841"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/categories?post=5841"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/tags?post=5841"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}