{"id":13011,"date":"2021-04-12T07:43:14","date_gmt":"2021-04-12T12:43:14","guid":{"rendered":"https:\/\/rud.is\/b\/?p=13011"},"modified":"2021-04-12T14:13:03","modified_gmt":"2021-04-12T19:13:03","slug":"check-developer-tools-first-to-avoid-heavy-ish-dependencies","status":"publish","type":"post","link":"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/","title":{"rendered":"Check ‘Developer Tools’ First To Avoid Heavy-ish Dependencies"},"content":{"rendered":"

Guillaume Pressiat (@GuillaumePressiat) did a solid post & video<\/a> on using Selenium to scrape a paginated table from understat[.]com\/league\/EPL\/2020<\/code> (I just cannot bring myself to provide an active link to any SportsBall site). He does a great job walking folks through acquiring & orchestrating the heavy dependency that is Selenium.<\/p>\n

I did a quick “look at browser Developer Tools” tweet a few weeks back that included the entire code for retrieving the Forbes billionaires list via the JSON file the Forbes’ site loads via an XHR request<\/a> responding to a similar fine article by another R user on using Selenium to do the same thing.<\/p>\n

If you find yourself thwarted by rvest::read_html()<\/code> not returning “nodes that are clearly there” it is likely due to the page rendering nodes dynamically via javascript. Selenium orchestrates full or headless browsers and lets you scrape the dynamically rendered DOM<\/a>. You can see this yourself if you first view the source of an HTML page (via the browser’s “view source” menu) and then use Developer Tools<\/a> to inspect the browser session. The “view source” view (in Blink-based browsers, at least) will be the raw, unrendered source HTML from the site and the DevTools “Elements” tab will have the rendered DOM elements.<\/p>\n

The “Nework” tab of DevTools has an “XHR” tab of its own, but if you try to use it on this SportsBall site to see the JSON it loads, you’ll be bitterly disappointed because \u2014\u00a0while it does indeed render JSON into HTML DOM nodes dynamically \u2014 that JSON is embedded in the web page:<\/p>\n

\"sportsball<\/a><\/p>\n

We can work in two different ways without the use of Selenium.<\/p>\n

First, we’ll “cheat” and use the {V8} package, which is an R interface to a javascript virtual machine, the type of which browsers use to run javascript on web pages. I say “cheat” because we’re still depending on a chunk of a browser engine.<\/p>\n

Let’s get some boilerplate out of the way:<\/p>\n

library(V8)        # V8 engine\nlibrary(rvest)     # Scraping\nlibrary(stringi)   # String manipulation which we'll use later\nlibrary(tidyverse) # Duh\n\nctx <- v8() # create a new instance of the javascript VM\n\npg <- read_html(\"https:\/\/understat.com\/league\/EPL\/2020\") # read sportsball page\n<\/code><\/pre>\n
If you examine the SportsBall page you’ll see that JSON.parse<\/code> in a few different locations, let’s target them all:<\/p>\n
html_nodes(pg, xpath = \".\/\/script[contains(., 'JSON.parse')]\")\n## {xml_nodeset (4)}\n## [1] <script>\\n\\tvar datesData \\t= JSON.parse('\\\\x5B\\\\x7B\\\\x22id\\\\x22\\\\x3A\\\\x2214086 ...\n## [2] <script>\\n\\tvar teamsData = JSON.parse('\\\\x7B\\\\x2271\\\\x22\\\\x3A\\\\x7B\\\\x22id\\\\x22 ...\n## [3] <script>\\n\\tvar playersData\\t= JSON.parse('\\\\x5B\\\\x7B\\\\x22id\\\\x22\\\\x3A\\\\x22647\\ ...\n## [4] <script>\\n\\t\\tWebFont.load({\\n\\t\\t\\tgoogle: {\\n\\t\\t\\t\\tfamilies: ['Barlow:500', ...\n<\/code><\/pre>\n
We don’t need that last one, so the first three contain all the data we need.<\/p>\n
Turning that into data is pretty straightforward work:<\/p>\n
html_nodes(pg, xpath = \".\/\/script[contains(., 'JSON.parse')]\") %>% \n  .[1:3] %>%         # only want the first three nodes\n  html_text() %>%    # turn the nodes into text\n  walk(ctx$eval)     # tell V8 to evaluate the javascript\n<\/code><\/pre>\n
The VM we created now has those three variables:<\/p>\n
ctx$get(JS(\"Object.keys(global)\"))\n## [1] \"print\"       \"console\"     \"global\"      \"datesData\"   \"_week\"      \n## [6] \"_year\"       \"teamsData\"   \"playersData\"\n<\/code><\/pre>\n
and, we can retrieve them like this:<\/p>\n
as_tibble(ctx$get(\"datesData\"))\n##  A tibble: 380 x 8\n##    id    isResult h$id  $title $short_title a$id  $title $short_title goals$h $a   \n##    <chr> <lgl>    <chr> <chr>  <chr>        <chr> <chr>  <chr>        <chr>   <chr>\n##  1 14086 TRUE     228   Fulham FLH          83    Arsen\u2026 ARS          0       3    \n##  2 14087 TRUE     78    Cryst\u2026 CRY          74    South\u2026 SOU          1       0    \n##  3 14090 TRUE     87    Liver\u2026 LIV          245   Leeds  LED          4       3    \n##  4 14091 TRUE     81    West \u2026 WHU          86    Newca\u2026 NEW          0       2    \n##  5 14092 TRUE     76    West \u2026 WBA          75    Leice\u2026 LEI          0       3    \n##  6 14093 TRUE     82    Totte\u2026 TOT          72    Evert\u2026 EVE          0       1    \n##  7 14094 TRUE     238   Sheff\u2026 SHE          229   Wolve\u2026 WOL          0       2    \n##  8 14095 TRUE     220   Brigh\u2026 BRI          80    Chels\u2026 CHE          1       3    \n##  9 14096 TRUE     72    Evert\u2026 EVE          76    West \u2026 WBA          5       2    \n## 10 14097 TRUE     245   Leeds  LED          228   Fulham FLH          4       3    \n## # \u2026 with 370 more rows, and 6 more variables: xG$h <chr>, $a <chr>, datetime <chr>,\n## #   forecast$w <chr>, $d <chr>, $l <chr>\n<\/code><\/pre>\n
Note that we need to do some extra processing of the second one to make it a bit tidier:<\/p>\n
ctx$get(\"teamsData\") %>% \n  map_df(~{\n    .x$history$id <- .x$id\n    .x$history$title <- .x$title\n    .x$history\n  }) %>% \n  as_tibble()\n## # A tibble: 616 x 21\n##    h_a      xG   xGA  npxG  npxGA ppda$att  $def ppda_allowed$att  $def  deep\n##    <chr> <dbl> <dbl> <dbl>  <dbl>    <int> <int>            <int> <int> <int>\n##  1 h     0.805 0.850 0.805 0.0885       89    20              247    14    17\n##  2 a     2.03  0.535 2.03  0.535       307    33              143    24    10\n##  3 h     3.08  1.66  3.08  1.66        365    25              119    25     7\n##  4 a     0.874 0.672 0.874 0.672       212    23              210    24     7\n##  5 h     1.50  2.38  1.50  2.38        225    17              124    34     7\n##  6 h     2.45  1.00  1.69  1.00        161    23              164    22     5\n##  7 a     1.99  1.39  1.99  1.39        331    24              169    15    16\n##  8 h     1.77  1.50  1.77  1.50        257    14              208    17     6\n##  9 a     2.39  0.572 1.63  0.572       144    11              289    23     8\n## 10 a     1.27  1.14  0.508 1.14        162    28              166    20     5\n## # \u2026 with 606 more rows, and 13 more variables: deep_allowed <int>, scored <int>,\n## #   missed <int>, xpts <dbl>, result <chr>, date <chr>, wins <int>, draws <int>,\n## #   loses <int>, pts <int>, npxGD <dbl>, id <chr>, title <chr>\n<\/code><\/pre>\n
The last one does not need any extra help:<\/p>\n
as_tibble(ctx$get(\"playersData\"))\n## # A tibble: 505 x 18\n##    id    player_name games time  goals xG    assists xA    shots key_passes\n##    <chr> <chr>       <chr> <chr> <chr> <chr> <chr>   <chr> <chr> <chr>     \n##  1 647   Harry Kane  29    2557  19    17.6\u2026 13      6.73\u2026 113   39        \n##  2 1250  Mohamed Sa\u2026 30    2529  19    16.1\u2026 3       4.55\u2026 99    40        \n##  3 1228  Bruno Fern\u2026 31    2659  16    13.4\u2026 11      10.8\u2026 95    87        \n##  4 453   Son Heung-\u2026 30    2509  14    9.35\u2026 9       8.03\u2026 55    56        \n##  5 822   Patrick Ba\u2026 31    2572  14    14.8\u2026 7       3.44\u2026 93    24        \n##  6 5555  Dominic Ca\u2026 26    2248  14    15.7\u2026 0       0.95\u2026 65    13        \n##  7 3277  Alexandre \u2026 27    1818  13    11.8\u2026 2       1.77\u2026 43    21        \n##  8 314   Ilkay G\u00fcnd\u2026 24    1776  12    8.64\u2026 1       3.29\u2026 45    35        \n##  9 755   Jamie Vardy 27    2230  12    16.0\u2026 7       4.26\u2026 64    22        \n## 10 8865  Ollie Watk\u2026 30    2700  12    13.7\u2026 3       4.15\u2026 81    36        \n## # \u2026 with 495 more rows, and 8 more variables: yellow_cards <chr>, red_cards <chr>,\n## #   position <chr>, team_title <chr>, npg <chr>, npxG <chr>, xGChain <chr>,\n## #   xGBuildup <chr>\n<\/code><\/pre>\n
We don’t really<\/em> need {V8} for this, though, if we’re willing to use some regular expressions<\/a>. We have to be a bit careful since some extra, non-JSON data comes along for the ride with that first <script><\/code> tag (see the embedded image above).<\/p>\n
We perform the same initial setup (get the text of the first three <script><\/code> tags), then we erase everthing that isn’t JSON data (so all the var<\/code> and javascript punctuation). By using comments = TRUE<\/code> in call to stri_replace_all_regex<\/code> we can provide documentation along with the (ugly) regex.<\/p>\n
The creator of the SportsBall site did some encoding to make the string easier to shove into a <script><\/code> tag, so we need to undo that by converting the hex-escapes to HTML entity escapes (replace \\x<\/code> with %<\/code>) and then decoding them with curl::curl_unescape()<\/code><\/p>\n
We could have made the regular expression uglier to avoid the other javascript cruft in the first <script><\/code> tag, but it’s just as easy to split them all into lines and pull the first line out.<\/p>\n
Then, it’s just a matter of running each one through jsonlite::fromJSON()<\/code>. I kept it a list and just set the names as the names of the variables above.<\/p>\n
html_nodes(pg, xpath = \".\/\/script[contains(., 'JSON.parse')]\") %>% \n  .[1:3] %>% \n  html_text() %>% \n  stri_replace_all_regex(\"\n^[^\\\\(]+\\\\('            # remove everything from the beginning of the line to the first ('\n|                       # OR\n'\\\\)[;,][[:space:]]*    # remove the last ') and everything after it\n$\n\", \"\", comments = TRUE, multiline = TRUE) %>% \n  stri_replace_all_fixed(\"\\\\x\", \"%\") %>% \n  curl::curl_unescape() %>% \n  stri_split_lines() %>% \n  map_chr(1) %>% \n  map(jsonlite::fromJSON) %>%\n  map(as_tibble) %>% \n  set_names(c(\"datesData\", \"teamsData\", \"playersData\")) %>% \n  str(3)\n## List of 3\n##  $ datesData  : tibble [380 \u00d7 8] (S3: tbl_df\/tbl\/data.frame)\n##   ..$ id      : chr [1:380] \"14086\" \"14087\" \"14090\" \"14091\" ...\n##   ..$ isResult: logi [1:380] TRUE TRUE TRUE TRUE TRUE TRUE ...\n##   ..$ h       :'data.frame':  380 obs. of  3 variables:\n##   ..$ a       :'data.frame':  380 obs. of  3 variables:\n##   ..$ goals   :'data.frame':  380 obs. of  2 variables:\n##   ..$ xG      :'data.frame':  380 obs. of  2 variables:\n##   ..$ datetime: chr [1:380] \"2020-09-12 11:30:00\" \"2020-09-12 14:00:00\" \"2020-09-12 16:30:00\" \"2020-09-12 19:00:00\" ...\n##   ..$ forecast:'data.frame':  380 obs. of  3 variables:\n##  $ teamsData  : tibble [3 \u00d7 20] (S3: tbl_df\/tbl\/data.frame)\n##   ..$ 71 :List of 3\n##   ..$ 72 :List of 3\n##   ..$ 74 :List of 3\n##   ..$ 75 :List of 3\n##   ..$ 76 :List of 3\n##   ..$ 78 :List of 3\n##   ..$ 80 :List of 3\n##   ..$ 81 :List of 3\n##   ..$ 82 :List of 3\n##   ..$ 83 :List of 3\n##   ..$ 86 :List of 3\n##   ..$ 87 :List of 3\n##   ..$ 88 :List of 3\n##   ..$ 89 :List of 3\n##   ..$ 92 :List of 3\n##   ..$ 220:List of 3\n##   ..$ 228:List of 3\n##   ..$ 229:List of 3\n##   ..$ 238:List of 3\n##   ..$ 245:List of 3\n##  $ playersData: tibble [505 \u00d7 18] (S3: tbl_df\/tbl\/data.frame)\n##   ..$ id          : chr [1:505] \"647\" \"1250\" \"1228\" \"453\" ...\n##   ..$ player_name : chr [1:505] \"Harry Kane\" \"Mohamed Salah\" \"Bruno Fernandes\" \"Son Heung-Min\" ...\n##   ..$ games       : chr [1:505] \"29\" \"30\" \"31\" \"30\" ...\n##   ..$ time        : chr [1:505] \"2557\" \"2529\" \"2659\" \"2509\" ...\n##   ..$ goals       : chr [1:505] \"19\" \"19\" \"16\" \"14\" ...\n##   ..$ xG          : chr [1:505] \"17.650331255048513\" \"16.19410896115005\" \"13.438796618022025\" \"9.352356541901827\" ...\n##   ..$ assists     : chr [1:505] \"13\" \"3\" \"11\" \"9\" ...\n##   ..$ xA          : chr [1:505] \"6.7384555246680975\" \"4.557050030678511\" \"10.812157344073057\" \"8.036493374034762\" ...\n##   ..$ shots       : chr [1:505] \"113\" \"99\" \"95\" \"55\" ...\n##   ..$ key_passes  : chr [1:505] \"39\" \"40\" \"87\" \"56\" ...\n##   ..$ yellow_cards: chr [1:505] \"1\" \"0\" \"5\" \"0\" ...\n##   ..$ red_cards   : chr [1:505] \"0\" \"0\" \"0\" \"0\" ...\n##   ..$ position    : chr [1:505] \"F\" \"F S\" \"M S\" \"F M S\" ...\n##   ..$ team_title  : chr [1:505] \"Tottenham\" \"Liverpool\" \"Manchester United\" \"Tottenham\" ...\n##   ..$ npg         : chr [1:505] \"15\" \"13\" \"8\" \"14\" ...\n##   ..$ npxG        : chr [1:505] \"14.605655785650015\" \"11.627095961943269\" \"6.5883138151839375\" \"9.352356541901827\" ...\n##   ..$ xGChain     : chr [1:505] \"20.556765687651932\" \"21.694580920040607\" \"22.04182725213468\" \"17.928756553679705\" ...\n##   ..$ xGBuildup   : chr [1:505] \"3.99019683804363\" \"8.287332298234105\" \"8.843060294166207\" \"5.881684513762593\" ...\n<\/code><\/pre>\n
You can use the cleanup code from the {V8} example to reshape that second element, and readr::type_convert()<\/code> can help you turn the character vectors into something more useful.<\/p>\n
FIN<\/h3>\n
It really always pays to take a look at the DevTools pane before introducing heavy dependencies. More sites are using very straightforward idioms that make the dynamically rendered page JSON source data readily available. Further, sites often add extra fields that you don’t see rendered, but may be useful to have around as you work with the resulting data.<\/p>\n","protected":false},"excerpt":{"rendered":"
Guillaume Pressiat (@GuillaumePressiat) did a solid post & video on using Selenium to scrape a paginated table from understat[.]com\/league\/EPL\/2020 (I just cannot bring myself to provide an active link to any SportsBall site). He does a great job walking folks through acquiring & orchestrating the heavy dependency that is Selenium. I did a quick “look […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":3,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":""},"categories":[91],"tags":[],"class_list":["post-13011","post","type-post","status-publish","format-standard","hentry","category-r"],"yoast_head":"\nCheck 'Developer Tools' First To Avoid Heavy-ish Dependencies - rud.is<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Check 'Developer Tools' First To Avoid Heavy-ish Dependencies - rud.is\" \/>\n<meta property=\"og:description\" content=\"Guillaume Pressiat (@GuillaumePressiat) did a solid post & video on using Selenium to scrape a paginated table from understat[.]com\/league\/EPL\/2020 (I just cannot bring myself to provide an active link to any SportsBall site). He does a great job walking folks through acquiring & orchestrating the heavy dependency that is Selenium. I did a quick “look […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/\" \/>\n<meta property=\"og:site_name\" content=\"rud.is\" \/>\n<meta property=\"article:published_time\" content=\"2021-04-12T12:43:14+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-04-12T19:13:03+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/rud.is\/b\/wp-content\/uploads\/2021\/04\/sportsball-dom-01.png\" \/>\n<meta name=\"author\" content=\"hrbrmstr\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"hrbrmstr\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2021\\\/04\\\/12\\\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2021\\\/04\\\/12\\\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\\\/\"},\"author\":{\"name\":\"hrbrmstr\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\"},\"headline\":\"Check ‘Developer Tools’ First To Avoid Heavy-ish Dependencies\",\"datePublished\":\"2021-04-12T12:43:14+00:00\",\"dateModified\":\"2021-04-12T19:13:03+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2021\\\/04\\\/12\\\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\\\/\"},\"wordCount\":702,\"commentCount\":4,\"publisher\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\"},\"image\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2021\\\/04\\\/12\\\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/sportsball-dom-01.png\",\"articleSection\":[\"R\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/rud.is\\\/b\\\/2021\\\/04\\\/12\\\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2021\\\/04\\\/12\\\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\\\/\",\"url\":\"https:\\\/\\\/rud.is\\\/b\\\/2021\\\/04\\\/12\\\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\\\/\",\"name\":\"Check 'Developer Tools' First To Avoid Heavy-ish Dependencies - rud.is\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2021\\\/04\\\/12\\\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2021\\\/04\\\/12\\\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/sportsball-dom-01.png\",\"datePublished\":\"2021-04-12T12:43:14+00:00\",\"dateModified\":\"2021-04-12T19:13:03+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2021\\\/04\\\/12\\\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/rud.is\\\/b\\\/2021\\\/04\\\/12\\\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2021\\\/04\\\/12\\\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\\\/#primaryimage\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/sportsball-dom-01.png?fit=1172%2C668&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/sportsball-dom-01.png?fit=1172%2C668&ssl=1\",\"width\":1172,\"height\":668,\"caption\":\"sportsball DOM node with JSON\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2021\\\/04\\\/12\\\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/rud.is\\\/b\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Check ‘Developer Tools’ First To Avoid Heavy-ish Dependencies\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#website\",\"url\":\"https:\\\/\\\/rud.is\\\/b\\\/\",\"name\":\"rud.is\",\"description\":\""In God we trust. All others must bring data"\",\"publisher\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/rud.is\\\/b\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\",\"name\":\"hrbrmstr\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\",\"width\":460,\"height\":460,\"caption\":\"hrbrmstr\"},\"logo\":{\"@id\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\"},\"description\":\"Don't look at me\u2026I do what he does \u2014 just slower. #rstats avuncular \u2022 ?Resistance Fighter \u2022 Cook \u2022 Christian \u2022 [Master] Chef des Donn\u00e9es de S\u00e9curit\u00e9 @ @rapid7\",\"sameAs\":[\"http:\\\/\\\/rud.is\"],\"url\":\"https:\\\/\\\/rud.is\\\/b\\\/author\\\/hrbrmstr\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Check 'Developer Tools' First To Avoid Heavy-ish Dependencies - rud.is","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/","og_locale":"en_US","og_type":"article","og_title":"Check 'Developer Tools' First To Avoid Heavy-ish Dependencies - rud.is","og_description":"Guillaume Pressiat (@GuillaumePressiat) did a solid post & video on using Selenium to scrape a paginated table from understat[.]com\/league\/EPL\/2020 (I just cannot bring myself to provide an active link to any SportsBall site). He does a great job walking folks through acquiring & orchestrating the heavy dependency that is Selenium. I did a quick “look […]","og_url":"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/","og_site_name":"rud.is","article_published_time":"2021-04-12T12:43:14+00:00","article_modified_time":"2021-04-12T19:13:03+00:00","og_image":[{"url":"https:\/\/rud.is\/b\/wp-content\/uploads\/2021\/04\/sportsball-dom-01.png","type":"","width":"","height":""}],"author":"hrbrmstr","twitter_card":"summary_large_image","twitter_misc":{"Written by":"hrbrmstr","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/#article","isPartOf":{"@id":"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/"},"author":{"name":"hrbrmstr","@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"headline":"Check ‘Developer Tools’ First To Avoid Heavy-ish Dependencies","datePublished":"2021-04-12T12:43:14+00:00","dateModified":"2021-04-12T19:13:03+00:00","mainEntityOfPage":{"@id":"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/"},"wordCount":702,"commentCount":4,"publisher":{"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"image":{"@id":"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/#primaryimage"},"thumbnailUrl":"https:\/\/rud.is\/b\/wp-content\/uploads\/2021\/04\/sportsball-dom-01.png","articleSection":["R"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/","url":"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/","name":"Check 'Developer Tools' First To Avoid Heavy-ish Dependencies - rud.is","isPartOf":{"@id":"https:\/\/rud.is\/b\/#website"},"primaryImageOfPage":{"@id":"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/#primaryimage"},"image":{"@id":"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/#primaryimage"},"thumbnailUrl":"https:\/\/rud.is\/b\/wp-content\/uploads\/2021\/04\/sportsball-dom-01.png","datePublished":"2021-04-12T12:43:14+00:00","dateModified":"2021-04-12T19:13:03+00:00","breadcrumb":{"@id":"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/#primaryimage","url":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2021\/04\/sportsball-dom-01.png?fit=1172%2C668&ssl=1","contentUrl":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2021\/04\/sportsball-dom-01.png?fit=1172%2C668&ssl=1","width":1172,"height":668,"caption":"sportsball DOM node with JSON"},{"@type":"BreadcrumbList","@id":"https:\/\/rud.is\/b\/2021\/04\/12\/check-developer-tools-first-to-avoid-heavy-ish-dependencies\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/rud.is\/b\/"},{"@type":"ListItem","position":2,"name":"Check ‘Developer Tools’ First To Avoid Heavy-ish Dependencies"}]},{"@type":"WebSite","@id":"https:\/\/rud.is\/b\/#website","url":"https:\/\/rud.is\/b\/","name":"rud.is","description":""In God we trust. All others must bring data"","publisher":{"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/rud.is\/b\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886","name":"hrbrmstr","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","url":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","contentUrl":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","width":460,"height":460,"caption":"hrbrmstr"},"logo":{"@id":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1"},"description":"Don't look at me\u2026I do what he does \u2014 just slower. #rstats avuncular \u2022 ?Resistance Fighter \u2022 Cook \u2022 Christian \u2022 [Master] Chef des Donn\u00e9es de S\u00e9curit\u00e9 @ @rapid7","sameAs":["http:\/\/rud.is"],"url":"https:\/\/rud.is\/b\/author\/hrbrmstr\/"}]}},"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p23idr-3nR","jetpack_likes_enabled":true,"jetpack-related-posts":[{"id":5004,"url":"https:\/\/rud.is\/b\/2017\/02\/09\/diving-into-dynamic-website-content-with-splashr\/","url_meta":{"origin":13011,"position":0},"title":"Diving Into Dynamic Website Content with splashr","author":"hrbrmstr","date":"2017-02-09","format":false,"excerpt":"If you do enough web scraping, you'll eventually hit a wall that the trusty httr verbs (that sit beneath rvest) cannot really overcome: dynamically created content (via javascript) on a site. If the site was nice enough to use XHR requests to load the dynamic content, you can generally still\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":11765,"url":"https:\/\/rud.is\/b\/2019\/01\/14\/splashr-0-6-0-now-uses-the-cran-nascent-stevedore-package-for-docker-orchestration\/","url_meta":{"origin":13011,"position":1},"title":"splashr 0.6.0 Now Uses the CRAN-nascent stevedore Package for Docker Orchestration","author":"hrbrmstr","date":"2019-01-14","format":false,"excerpt":"The splashr package [srht|GL|GH] \u2014 an alternative to Selenium for javascript-enabled\/browser-emulated web scraping \u2014 is now at version 0.6.0 (still in dev-mode but on its way to CRAN in the next 14 days). The major change from version 0.5.x (which never made it to CRAN) is a swap out of\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":6206,"url":"https:\/\/rud.is\/b\/2017\/08\/29\/new-cran-package-announcement-splashr\/","url_meta":{"origin":13011,"position":2},"title":"New CRAN Package Announcement: splashr","author":"hrbrmstr","date":"2017-08-29","format":false,"excerpt":"I'm pleased to announce that splashr is now on CRAN. (That image was generated with splashr::render_png(url = \"https:\/\/cran.r-project.org\/web\/packages\/splashr\/\")). The package is an R interface to the Splash javascript rendering service. It works in a similar fashion to Selenium but is fear more geared to web scraping and has quite a\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/08\/splashr.png?fit=1066%2C1108&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/08\/splashr.png?fit=1066%2C1108&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/08\/splashr.png?fit=1066%2C1108&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/08\/splashr.png?fit=1066%2C1108&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/08\/splashr.png?fit=1066%2C1108&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":12013,"url":"https:\/\/rud.is\/b\/2019\/02\/28\/htmlunitjars-updated-to-2-34-0\/","url_meta":{"origin":13011,"position":3},"title":"htmlunitjars Updated to 2.34.0","author":"hrbrmstr","date":"2019-02-28","format":false,"excerpt":"The in-dev htmlunit package for javascript-\"enabled\" web-scraping without the need for Selenium, Splash or headless Chrome relies on the HtmlUnit library and said library just released version 2.34.0 with a wide array of changes that should make it possible to scrape more gnarly javascript-\"enabled\" sites. The Chrome emulation is now\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":11416,"url":"https:\/\/rud.is\/b\/2018\/08\/23\/introducing-gepetto-a-splash-like-rest-api-to-headless-chrome\/","url_meta":{"origin":13011,"position":4},"title":"Introducing ‘gepetto’ \u2014 a Splash-like REST API to Headless Chrome","author":"hrbrmstr","date":"2018-08-23","format":false,"excerpt":"It's been over a year since Headless Chrome was introduced and it has matured greatly over that time and has acquired a pretty large user base. The TLDR on it is that you can now use Chrome as you would any command-line interface (CLI) program and generate PDFs, images or\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":11383,"url":"https:\/\/rud.is\/b\/2018\/08\/13\/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium\/","url_meta":{"origin":13011,"position":5},"title":"In-brief: splashr update + High Performance Scraping with splashr, furrr & TeamHG-Memex’s Aquarium","author":"hrbrmstr","date":"2018-08-13","format":false,"excerpt":"The development version of splashr now support authenticated connections to Splash API instances. Just specify user and pass on the initial splashr::splash() call to use your scraping setup a bit more safely. For those not familiar with splashr and\/or Splash: the latter is a lightweight alternative to tools like Selenium\u2026","rel":"","context":"In "R"","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts\/13011","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/comments?post=13011"}],"version-history":[{"count":0,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts\/13011\/revisions"}],"wp:attachment":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/media?parent=13011"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/categories?post=13011"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/tags?post=13011"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}