<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/spelunking-xhrs-xmlhttprequests-with-splashr/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/02/14/spelunking-xhrs-xmlhttprequests-with-splashr/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a target="_blank" rel="nofollow" href="https://github.com/hrbrmstr/splashr"><code>splashr</code></a> has gained some new functionality since <a target="_blank" rel="nofollow" href="https://rud.is/b/2017/02/09/diving-into-dynamic-website-content-with-splashr/">the introductory post</a>. First, there’s a whole new Docker image for it that embeds a local web server. Why? The main request for it was to enable rendering of <code>htmlwidgets</code>:</p>
<pre><code>splash_vm &lt;- start_splash(add_tempdir=TRUE)

DiagrammeR(&quot;
  graph LR
    A-->B
    A-->C
    C-->E
    B-->D
    C-->D
    D-->F
    E-->F
&quot;) %>% 
  saveWidget(&quot;/tmp/diag.html&quot;)

splash(&quot;localhost&quot;) %>% 
  render_file(&quot;/tmp/diag.html&quot;, output=&quot;html&quot;)
## {xml_document}
## &lt;html>
## [1] &lt;head>\n&lt;meta http-equiv=&quot;Content-Type&quot; content=&quot;text/html; charset=UTF-8&quot;>\n&lt;meta charset=&quot;utf-8&quot;>\n&lt;script src= ...
## [2] &lt;body style=&quot;background-color: white; margin: 0px; padding: 40px;&quot;>\n&lt;div id=&quot;htmlwidget_container&quot;>\n&lt;div id=&quot;ht ...

splash(&quot;localhost&quot;) %>% 
  render_file(&quot;/tmp/diag.html&quot;, output=&quot;png&quot;, wait=2)</code></pre>
<p><a target="_blank" rel="nofollow" href="https://i1.wp.com/rud.is/b/wp-content/uploads/2017/02/diag.png?ssl=1"><img alt="" class="wp-image-5032" src="https://i1.wp.com/rud.is/b/wp-content/uploads/2017/02/diag.png?resize=384%2C249&ssl=1" data-recalc-dims="1"></a></p>
<p>But if you use the new Docker image and the <code>add_tempdir=TRUE</code> parameter it can render any local HTML file.</p>
<p>The other new bits are helpers to identify content types in the HAR types. Along with <code>get_content_type()</code>:</p>
<pre><code>library(tidyverse)

map_chr(rud_har$log$entries, get_content_type)
##  [1] &quot;text/html&quot;                &quot;text/html&quot;                &quot;application/javascript&quot;   &quot;text/css&quot;                
##  [5] &quot;text/css&quot;                 &quot;text/css&quot;                 &quot;text/css&quot;                 &quot;text/css&quot;                
##  [9] &quot;text/css&quot;                 &quot;application/javascript&quot;   &quot;application/javascript&quot;   &quot;application/javascript&quot;  
## [13] &quot;application/javascript&quot;   &quot;application/javascript&quot;   &quot;application/javascript&quot;   &quot;text/javascript&quot;         
## [17] &quot;text/css&quot;                 &quot;text/css&quot;                 &quot;application/x-javascript&quot; &quot;application/x-javascript&quot;
## [21] &quot;application/x-javascript&quot; &quot;application/x-javascript&quot; &quot;application/x-javascript&quot; NA                        
## [25] &quot;text/css&quot;                 &quot;image/png&quot;                &quot;image/png&quot;                &quot;image/png&quot;               
## [29] &quot;font/ttf&quot;                 &quot;font/ttf&quot;                 &quot;text/html&quot;                &quot;font/ttf&quot;                
## [33] &quot;font/ttf&quot;                 &quot;application/font-woff&quot;    &quot;application/font-woff&quot;    &quot;image/svg+xml&quot;           
## [37] &quot;text/css&quot;                 &quot;text/css&quot;                 &quot;image/gif&quot;                &quot;image/svg+xml&quot;           
## [41] &quot;application/font-woff&quot;    &quot;application/font-woff&quot;    &quot;application/font-woff&quot;    &quot;application/font-woff&quot;   
## [45] &quot;application/font-woff&quot;    &quot;application/font-woff&quot;    &quot;application/font-woff&quot;    &quot;application/font-woff&quot;   
## [49] &quot;text/css&quot;                 &quot;application/x-javascript&quot; &quot;image/gif&quot;                NA                        
## [53] &quot;image/jpeg&quot;               &quot;image/svg+xml&quot;            &quot;image/svg+xml&quot;            &quot;image/svg+xml&quot;           
## [57] &quot;image/svg+xml&quot;            &quot;image/svg+xml&quot;            &quot;image/svg+xml&quot;            &quot;image/gif&quot;               
## [61] NA                         &quot;application/x-javascript&quot; NA                         NA</code></pre>
<p>there are many <code>is_...()</code> functions for logical tests.</p>
<p>But, one of the more interesting <code>is_()</code> functions is <code>is_xhr()</code>. Sites with dynamic content <em>usually</em> load said content via an <code>XMLHttpRequest</code> or XHR for short. Modern web apps <em>usually</em> return JSON in said requests and, for questions like <a target="_blank" rel="nofollow" href="http://stackoverflow.com/q/41435049/1457051">this one on StackOverflow</a> it’s <em>usually</em> better to grab the JSON and use it for data than it is to scrape the table made from JavaScript calls. </p>
<p>Now, it’s not <em>too hard</em> to open Developer Tools and find those XHR requests, but we can also use <code>splashr</code> to programmatically find them. We have to do a bit more work and use the new <code>execute_lua()</code> function since we need to give the page time to load up all the data. (I’ll eventually write a mini-R-DSL around this idiom so you don’t have to grok Lua for non-complex scraping tasks). Here’s how we’d answer that StackOverflow question today…</p>
<p>First, we grab the entire HAR contents (including bodies of the individual requests) after waiting a bit:</p>
<pre><code>splash_local %>%
  execute_lua('
function main(splash)
  splash.response_body_enabled = true
  splash:go(&quot;http://www.childrenshospital.org/directory?state=%7B%22showLandingContent%22%3Afalse%2C%22model%22%3A%7B%22search_specialist%22%3Afalse%2C%22search_type%22%3A%5B%22directoryphysician%22%2C%22directorynurse%22%5D%7D%2C%22customModel%22%3A%7B%22nurses%22%3Atrue%7D%7D&quot;)
  splash:wait(2)
  return splash:har()
end
') -> res

pg &lt;- as_har(res)</code></pre>
<p>then we look for XHRs:</p>
<pre><code>map_lgl(pg$log$entries, is_xhr) %>% which()
## 10</code></pre>
<p>and, finally, we grab the JSON:</p>
<pre><code>pg$log$entries[[10]]$response$content$text %>% 
  openssl::base64_decode() %>% 
  rawToChar() %>% 
  jsonlite::fromJSON() %>% 
  glimpse()
## List of 4
##  $ TotalPages  : int 16
##  $ TotalRecords: int 384
##  $ Records     :'data.frame': 24 obs. of  21 variables:
##   ..$ ID            : chr [1:24] &quot;{5E4B0D96-18D3-4FC6-B1AA-345675F3765C}&quot; &quot;{674EEC8B-062A-4268-9467-5C61030B83C9}&quot; ## &quot;{3E6257FE-67A1-4F13-B377-9EA7CCBD50F2}&quot; &quot;{C28479E6-5458-4010-A005-84E5F35B2FEA}&quot; ...
##   ..$ FirstName     : chr [1:24] &quot;Mirna&quot; &quot;Barbara&quot; &quot;Donald&quot; &quot;Victoria&quot; ...
##   ..$ LastName      : chr [1:24] &quot;Aeschlimann&quot; &quot;Angus&quot; &quot;Annino&quot; &quot;Arthur&quot; ...
##   ..$ Image         : chr [1:24] &quot;&quot; &quot;/~/media/directory/physicians/ppoc/angus_barbara.ashx&quot; &quot;/~/media/directory/physicians/ppoc/## annino_donald.ashx&quot; &quot;/~/media/directory/physicians/ppoc/arthur_victoria.ashx&quot; ...
##   ..$ Suffix        : chr [1:24] &quot;MD&quot; &quot;MD&quot; &quot;MD&quot; &quot;MD&quot; ...
##   ..$ Url           : chr [1:24] &quot;http://www.childrenshospital.org/doctors/mirna-aeschlimann&quot; &quot;http://www.childrenshospital.org/doctors/## barbara-angus&quot; &quot;http://www.childrenshospital.org/doctors/donald-annino&quot; &quot;http://www.childrenshospital.org/doctors/victoria-arthur&quot; ...
##   ..$ Gender        : chr [1:24] &quot;female&quot; &quot;female&quot; &quot;male&quot; &quot;female&quot; ...
##   ..$ Latitude      : chr [1:24] &quot;42.468769&quot; &quot;42.235088&quot; &quot;42.463177&quot; &quot;42.447168&quot; ...
##   ..$ Longitude     : chr [1:24] &quot;-71.100558&quot; &quot;-71.016021&quot; &quot;-71.143169&quot; &quot;-71.229734&quot; ...
##   ..$ Address       : chr [1:24] &quot;{&quot;practice_name&quot;:&quot;Pediatrics, Inc.&quot;, &quot;address_1&quot;:&quot;577 Main ## Street&quot;, &quot;city&quot;:&amp;q&quot;| __truncated__ &quot;{&quot;practice_name&quot;:&quot;Crown Colony Pediatrics&quot;, ## &quot;address_1&quot;:&quot;500 Congress Street, Suite 1F&quot;&quot;| __truncated__ &quot;{&quot;practice_name&quot;:&quot;Pediatricians ## Inc.&quot;, &quot;address_1&quot;:&quot;955 Main Street&quot;, &quot;city&quot;:&quot;| __truncated__ ## &quot;{&quot;practice_name&quot;:&quot;Lexington Pediatrics&quot;, &quot;address_1&quot;:&quot;19 Muzzey Street, Suite 105&quot;, &amp;qu&quot;| ## __truncated__ ...
##   ..$ Distance      : chr [1:24] &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; ...
##   ..$ OtherLocations: chr [1:24] &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; ...
##   ..$ AcademicTitle : chr [1:24] &quot;&quot; &quot;&quot; &quot;&quot; &quot;Clinical Instructor of Pediatrics - Harvard Medical School&quot; ...
##   ..$ HospitalTitle : chr [1:24] &quot;Pediatrician&quot; &quot;Pediatrician&quot; &quot;Pediatrician&quot; &quot;Pediatrician&quot; ...
##   ..$ Specialties   : chr [1:24] &quot;Primary Care, Pediatrics, General Pediatrics&quot; &quot;Primary Care, Pediatrics, General Pediatrics&quot; &quot;General ## Pediatrics, Pediatrics, Primary Care&quot; &quot;Primary Care, Pediatrics, General Pediatrics&quot; ...
##   ..$ Departments   : chr [1:24] &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; ...
##   ..$ Languages     : chr [1:24] &quot;English&quot; &quot;English&quot; &quot;&quot; &quot;&quot; ...
##   ..$ PPOCLink      : chr [1:24] &quot;http://www.childrenshospital.org/patient-resources/provider-glossary&quot; &quot;/patient-resources/## provider-glossary&quot; &quot;http://www.childrenshospital.org/patient-resources/provider-glossary&quot; &quot;http://www.childrenshospital.org/## patient-resources/provider-glossary&quot; ...
##   ..$ Gallery       : chr [1:24] &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; ...
##   ..$ Phone         : chr [1:24] &quot;781-438-7330&quot; &quot;617-471-3411&quot; &quot;781-729-4262&quot; &quot;781-862-4110&quot; ...
##   ..$ Fax           : chr [1:24] &quot;781-279-4046&quot; &quot;(617) 471-3584&quot; &quot;&quot; &quot;(781) 863-2007&quot; ...
##  $ Synonims    : list()</code></pre>
<p><strong>UPDATE</strong> So, I wrote a mini-DSL for this:</p>
<p>It’s unlikely we want to rely on a running Splash instance for our production work, so I’ll be making a helper function to turn HAR XHR requests into a <code>httr</code> function calls, similar to the way <a target="_blank" rel="nofollow" href="https://github.com/hrbrmstr/curlconverter"><code>curlconverter</code></a> works.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2017/02/14/spelunking-xhrs-xmlhttprequests-with-splashr/"> R – rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/qRn5nW1l07A"></a></div>