{"id":11383,"date":"2018-08-13T16:35:05","date_gmt":"2018-08-13T21:35:05","guid":{"rendered":"https:\/\/rud.is\/b\/?p=11383"},"modified":"2018-08-13T16:35:05","modified_gmt":"2018-08-13T21:35:05","slug":"in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium","status":"publish","type":"post","link":"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/","title":{"rendered":"In-brief: splashr update + High Performance Scraping with splashr, furrr & TeamHG-Memex’s Aquarium"},"content":{"rendered":"

The development version of splashr<\/code><\/a> now support authenticated connections to Splash<\/a> API instances. Just specify user<\/code> and pass<\/code> on the initial splashr::splash()<\/code> call to use your scraping setup a bit more safely. For those not familiar with splashr<\/code> and\/or Splash: the latter is a lightweight alternative to tools like Selenium and the former is an R interface to it. Unlike xml2::read_html()<\/code>, splashr<\/code> renders a URL exactly as a browser does (because it uses a virtual browser) and can return far more than just the HTML from a web page. Splash does need to be running and it’s best to use it in a Docker container.<\/p>\n

If you have a large number of sites to scrape, working with splashr<\/code> and Splash “as-is” can be a bit frustrating since there’s a limit to what a single instance can handle. Sure, it’s possible to setup your own highly available, multi-instance Splash cluster and use it, but that’s work<\/em>. Thankfully, the folks behind TeamHG-Memex created Aquarium<\/a> which uses docker<\/code> and docker-compose<\/code> to stand up a multi-Splash instance behind a pre-configured HAProxy<\/a> instance so you can take advantage of parallel requests the Splash API. As long as you have docker<\/code> and docker-compose<\/code> handy (and Python) following the steps on the aforelinked GitHub page should have you up and running with Aquarium in minutes. You use the same default port (8050<\/code>) to access the Splash API and you get a bonus port of 8036<\/code> to watch in your browser (the HAProxy stats page).<\/p>\n

This works well when combined with furrr<\/code>?<\/a> which is an R package that makes parallel operations very tidy.<\/p>\n

One way to use purrr<\/code>, splashr<\/code> and Aquarium might look like this:<\/p>\n

library(splashr)\nlibrary(HARtools)\nlibrary(urltools)\nlibrary(furrr)\nlibrary(tidyverse)\n\nlist_of_urls_with_unique_urls <- c(\"http:\/\/...\", \"http:\/\/...\", ...)\n\nmake_a_splash <- function(org_url) {\n  splash(\n    host = \"ip\/name of system you started aquarium on\", \n    user = \"your splash api username\", \n    pass = \"your splash api password\"\n  ) %>% \n    splash_response_body(TRUE) %>% # we want to get all the content \n    splash_user_agent(ua_win10_ie11) %>% # splashr has many pre-configured user agents to choose from \n    splash_go(org_url) %>% \n    splash_wait(5) %>% # pick a reasonable timeout; modern web sites with javascript are bloated\n    splash_har()\n}\n\nsafe_splash <- safely(make_a_splash) # splashr\/Splash work well but can throw errors. Let's be safe\n\nplan(multiprocess, workers=5) # don't overwhelm the default setup or your internet connection\n\nfuture_map(sites, ~{\n  \n  org <- safe_splash(.x) # go get it!\n  \n  if (is.null(org$result)) {\n    sprintf(\"Error retrieving %s (%s)\", .x, org$error$message) # this gives us good error messages\n  } else {\n    \n    HARtools::writeHAR( # HAR format saves *everything*. the files are YUGE\n      har = org$result, \n      file = file.path(\"\/place\/to\/store\/stuff\", sprintf(\"%s.har\", domain(.x))) # saved with the base domain; you may want to use a UUID via uuid::UUIDgenerate()\n    )\n    \n    sprintf(\"Successfully retrieved %s\", .x)\n    \n  }\n  \n}) -> results<\/code><\/pre>\n
(Those with a keen eye will grok why splashr<\/code> supports Splash API basic authentication, now)<\/p>\n
The parallel iterator will return a list we can flatten to a character vector (I don’t do that by default since it’s safer to get a list back as it can hold anything and map_chr()<\/code> likes to check for proper objects) to check for errors with something like:<\/p>\n
flatten_chr(results) %>% \n  keep(str_detect, \"Error\")\n## [1] \"Error retrieving www.1.example.com (Service Unavailable (HTTP 503).)\"\n## [2] \"Error retrieving www.100.example.com (Gateway Timeout (HTTP 504).)\"\n## [3] \"Error retrieving www.3000.example.com (Bad Gateway (HTTP 502).)\"\n## [4] \"Error retrieving www.a.example.com (Bad Gateway (HTTP 502).)\"\n## [5] \"Error retrieving www.z.examples.com (Gateway Timeout (HTTP 504).)\"<\/code><\/pre>\n
Timeouts would suggest you may need to up the timeout parameter in your Splash call. Service unavailable or bad gateway errors may suggest you need to tweak the Aquarium configuration to add more workers or reduce your plan(\u2026)<\/code>. It’s not unusual to have to create a scraping process that accounts for errors and retries a certain number of times.<\/p>\n
If you were stuck in the splashr<\/code>\/Splash slow-lane before, give this a try to help save you some time and frustration.<\/p>\n","protected":false},"excerpt":{"rendered":"
The development version of splashr now support authenticated connections to Splash API instances. Just specify user and pass on the initial splashr::splash() call to use your scraping setup a bit more safely. For those not familiar with splashr and\/or Splash: the latter is a lightweight alternative to tools like Selenium and the former is an […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":3,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":""},"categories":[91,725],"tags":[],"class_list":["post-11383","post","type-post","status-publish","format-standard","hentry","category-r","category-web-scraping"],"yoast_head":"\nIn-brief: splashr update + High Performance Scraping with splashr, furrr & TeamHG-Memex's Aquarium - rud.is<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"In-brief: splashr update + High Performance Scraping with splashr, furrr & TeamHG-Memex's Aquarium - rud.is\" \/>\n<meta property=\"og:description\" content=\"The development version of splashr now support authenticated connections to Splash API instances. Just specify user and pass on the initial splashr::splash() call to use your scraping setup a bit more safely. For those not familiar with splashr and\/or Splash: the latter is a lightweight alternative to tools like Selenium and the former is an […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/\" \/>\n<meta property=\"og:site_name\" content=\"rud.is\" \/>\n<meta property=\"article:published_time\" content=\"2018-08-13T21:35:05+00:00\" \/>\n<meta name=\"author\" content=\"hrbrmstr\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"hrbrmstr\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/\"},\"author\":{\"name\":\"hrbrmstr\",\"@id\":\"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886\"},\"headline\":\"In-brief: splashr update + High Performance Scraping with splashr, furrr & TeamHG-Memex’s Aquarium\",\"datePublished\":\"2018-08-13T21:35:05+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/\"},\"wordCount\":420,\"commentCount\":4,\"publisher\":{\"@id\":\"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886\"},\"articleSection\":[\"R\",\"web scraping\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/\",\"url\":\"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/\",\"name\":\"In-brief: splashr update + High Performance Scraping with splashr, furrr & TeamHG-Memex's Aquarium - rud.is\",\"isPartOf\":{\"@id\":\"https:\/\/rud.is\/b\/#website\"},\"datePublished\":\"2018-08-13T21:35:05+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/rud.is\/b\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"In-brief: splashr update + High Performance Scraping with splashr, furrr & TeamHG-Memex’s Aquarium\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/rud.is\/b\/#website\",\"url\":\"https:\/\/rud.is\/b\/\",\"name\":\"rud.is\",\"description\":\""In God we trust. All others must bring data"\",\"publisher\":{\"@id\":\"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/rud.is\/b\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886\",\"name\":\"hrbrmstr\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1\",\"url\":\"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1\",\"width\":460,\"height\":460,\"caption\":\"hrbrmstr\"},\"logo\":{\"@id\":\"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1\"},\"description\":\"Don't look at me\u2026I do what he does \u2014 just slower. #rstats avuncular \u2022 ?Resistance Fighter \u2022 Cook \u2022 Christian \u2022 [Master] Chef des Donn\u00e9es de S\u00e9curit\u00e9 @ @rapid7\",\"sameAs\":[\"http:\/\/rud.is\"],\"url\":\"https:\/\/rud.is\/b\/author\/hrbrmstr\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"In-brief: splashr update + High Performance Scraping with splashr, furrr & TeamHG-Memex's Aquarium - rud.is","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/","og_locale":"en_US","og_type":"article","og_title":"In-brief: splashr update + High Performance Scraping with splashr, furrr & TeamHG-Memex's Aquarium - rud.is","og_description":"The development version of splashr now support authenticated connections to Splash API instances. Just specify user and pass on the initial splashr::splash() call to use your scraping setup a bit more safely. For those not familiar with splashr and\/or Splash: the latter is a lightweight alternative to tools like Selenium and the former is an […]","og_url":"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/","og_site_name":"rud.is","article_published_time":"2018-08-13T21:35:05+00:00","author":"hrbrmstr","twitter_card":"summary_large_image","twitter_misc":{"Written by":"hrbrmstr","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/#article","isPartOf":{"@id":"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/"},"author":{"name":"hrbrmstr","@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"headline":"In-brief: splashr update + High Performance Scraping with splashr, furrr & TeamHG-Memex’s Aquarium","datePublished":"2018-08-13T21:35:05+00:00","mainEntityOfPage":{"@id":"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/"},"wordCount":420,"commentCount":4,"publisher":{"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"articleSection":["R","web scraping"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/","url":"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/","name":"In-brief: splashr update + High Performance Scraping with splashr, furrr & TeamHG-Memex's Aquarium - rud.is","isPartOf":{"@id":"https:\/\/rud.is\/b\/#website"},"datePublished":"2018-08-13T21:35:05+00:00","breadcrumb":{"@id":"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/rud.is\/b\/"},{"@type":"ListItem","position":2,"name":"In-brief: splashr update + High Performance Scraping with splashr, furrr & TeamHG-Memex’s Aquarium"}]},{"@type":"WebSite","@id":"https:\/\/rud.is\/b\/#website","url":"https:\/\/rud.is\/b\/","name":"rud.is","description":""In God we trust. All others must bring data"","publisher":{"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/rud.is\/b\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886","name":"hrbrmstr","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","url":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","contentUrl":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","width":460,"height":460,"caption":"hrbrmstr"},"logo":{"@id":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1"},"description":"Don't look at me\u2026I do what he does \u2014 just slower. #rstats avuncular \u2022 ?Resistance Fighter \u2022 Cook \u2022 Christian \u2022 [Master] Chef des Donn\u00e9es de S\u00e9curit\u00e9 @ @rapid7","sameAs":["http:\/\/rud.is"],"url":"https:\/\/rud.is\/b\/author\/hrbrmstr\/"}]}},"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p23idr-2XB","jetpack_likes_enabled":true,"jetpack-related-posts":[{"id":6206,"url":"https:\/\/rud.is\/b\/2017\/08\/29\/new-cran-package-announcement-splashr\/","url_meta":{"origin":11383,"position":0},"title":"New CRAN Package Announcement: splashr","author":"hrbrmstr","date":"2017-08-29","format":false,"excerpt":"I'm pleased to announce that splashr is now on CRAN. (That image was generated with splashr::render_png(url = \"https:\/\/cran.r-project.org\/web\/packages\/splashr\/\")). The package is an R interface to the Splash javascript rendering service. It works in a similar fashion to Selenium but is fear more geared to web scraping and has quite a\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/08\/splashr.png?fit=1066%2C1108&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/08\/splashr.png?fit=1066%2C1108&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/08\/splashr.png?fit=1066%2C1108&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/08\/splashr.png?fit=1066%2C1108&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/08\/splashr.png?fit=1066%2C1108&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":5004,"url":"https:\/\/rud.is\/b\/2017\/02\/09\/diving-into-dynamic-website-content-with-splashr\/","url_meta":{"origin":11383,"position":1},"title":"Diving Into Dynamic Website Content with splashr","author":"hrbrmstr","date":"2017-02-09","format":false,"excerpt":"If you do enough web scraping, you'll eventually hit a wall that the trusty httr verbs (that sit beneath rvest) cannot really overcome: dynamically created content (via javascript) on a site. If the site was nice enough to use XHR requests to load the dynamic content, you can generally still\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":5031,"url":"https:\/\/rud.is\/b\/2017\/02\/14\/spelunking-xhrs-xmlhttprequests-with-splashr\/","url_meta":{"origin":11383,"position":2},"title":"Spelunking XHRs (XMLHttpRequests) with splashr","author":"hrbrmstr","date":"2017-02-14","format":false,"excerpt":"splashr has gained some new functionality since the introductory post. First, there's a whole new Docker image for it that embeds a local web server. Why? The main request for it was to enable rendering of htmlwidgets: But if you use the new Docker image and the add_tempdir=TRUE parameter it\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":11765,"url":"https:\/\/rud.is\/b\/2019\/01\/14\/splashr-0-6-0-now-uses-the-cran-nascent-stevedore-package-for-docker-orchestration\/","url_meta":{"origin":11383,"position":3},"title":"splashr 0.6.0 Now Uses the CRAN-nascent stevedore Package for Docker Orchestration","author":"hrbrmstr","date":"2019-01-14","format":false,"excerpt":"The splashr package [srht|GL|GH] \u2014 an alternative to Selenium for javascript-enabled\/browser-emulated web scraping \u2014 is now at version 0.6.0 (still in dev-mode but on its way to CRAN in the next 14 days). The major change from version 0.5.x (which never made it to CRAN) is a swap out of\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":11670,"url":"https:\/\/rud.is\/b\/2018\/12\/02\/more-scraping-ethics-gone-awry-and-why-do-this-when-theres-a-free-api\/","url_meta":{"origin":11383,"position":4},"title":"More “Scraping Ethics Gone Awry” and “Why Do This When There’s a Free API?”","author":"hrbrmstr","date":"2018-12-02","format":false,"excerpt":"I can't seem to free my infrequently-viewed email inbox from \"you might like!\" notices by the content-lock-in site Medium. This one made it to the iOS notification screen (otherwise I'd've been blissfully unaware of it and would have saved you the trouble of reading this). Today, they sent me this\u2026","rel":"","context":"In "web scraping"","block_context":{"text":"web scraping","link":"https:\/\/rud.is\/b\/category\/web-scraping\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":6385,"url":"https:\/\/rud.is\/b\/2017\/09\/19\/pirating-web-content-responsibly-with-r\/","url_meta":{"origin":11383,"position":5},"title":"Pirating Web Content Responsibly With R","author":"hrbrmstr","date":"2017-09-19","format":false,"excerpt":"International Code Talk Like A Pirate Day almost slipped by without me noticing (September has been a crazy busy month), but it popped up in the calendar notifications today and I was glad that I had prepped the meat of a post a few weeks back. There will be no\u2026","rel":"","context":"In "data wrangling"","block_context":{"text":"data wrangling","link":"https:\/\/rud.is\/b\/category\/data-wrangling\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Plot_Zoom-2.png?fit=1200%2C917&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Plot_Zoom-2.png?fit=1200%2C917&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Plot_Zoom-2.png?fit=1200%2C917&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Plot_Zoom-2.png?fit=1200%2C917&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/Plot_Zoom-2.png?fit=1200%2C917&ssl=1&resize=1050%2C600 3x"},"classes":[]}],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts\/11383","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/comments?post=11383"}],"version-history":[{"count":0,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts\/11383\/revisions"}],"wp:attachment":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/media?parent=11383"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/categories?post=11383"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/tags?post=11383"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}