<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/introducing-gepetto-ae-a-splash-like-rest-api-to-headless-chrome/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2018/08/23/introducing-gepetto-a-splash-like-rest-api-to-headless-chrome/"> R â€“ rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>It’s been over a year since <a rel="nofollow" href="https://developers.google.com/web/updates/2017/04/headless-chrome" target="_blank">Headless Chrome</a> was introduced and it has matured greatly over that time and has acquired a pretty large user base. The TLDR on it is that you can now use Chrome as you would any command-line interface (CLI) program and generate PDFs, images or render javascript-interpreted HTML by supplying some simple parameters. It has a <a rel="nofollow" href="https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop" target="_blank">REPL</a> mode for interactive work and can be instrumented through a custom websockets protocol.</p>
<p>R folks have had the <a rel="nofollow" href="https://gitlab.com/hrbrmstr/decapitated" target="_blank"><code>decapitated</code><img alt="Ã°Å¸â€ - " src="https://i1.wp.com/s.w.org/images/core/emoji/11/72x72/1f517.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"></a> package available almost since the launch day of Headless Chrome. It provides a basic wrapper to the CLI. The package has been updated more recently to enable the downloading of a custom Chromium binary to use instead of the system Chrome installation (which is a highly recommended practice).</p>
<p>However, that nigh-mundane addition is not the only new feature in <code>decapitated</code>.</p>
<h3>Introducing gepetto</h3>
<p>While it would have been possible to create an R wrapper for the Headless Chrome websockets API, the reality is (and this is just my opinion) that it is better to integrate with a more robust and community supported interface to Headless Chrome instrumentation dubbed <a rel="nofollow" href="https://github.com/GoogleChrome/puppeteer" target="_blank"><code>puppeteer</code><img alt="Ã°Å¸â€ - " src="https://i1.wp.com/s.w.org/images/core/emoji/11/72x72/1f517.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"></a>. <code>Puppeteer</code> is a javascript module that adds high level functions on top of the lower-level API and has a massive amount of functionality that can be easily tapped into.</p>
<p>Now, Selenium <a rel="nofollow" href="https://developers.google.com/web/updates/2017/04/headless-chrome#drivers" target="_blank">works</a> <em>really well</em> with Headless Chrome and there’s little point in trying to reinvent that wheel. Rather, I wanted a way to interact with Headless Chrome the way one can with [ScrapingHub’s Splash](https://github.com/scrapinghub/splash service. That is, a simple REST API. To that end, I’ve started a project called <a rel="nofollow" href="https://gitlab.com/hrbrmstr/gepetto" target="_blank"><code>gepetto</code><img alt="Ã°Å¸â€ - " src="https://i1.wp.com/s.w.org/images/core/emoji/11/72x72/1f517.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"></a> which aims to do just that.</p>
<p><code>Gepetto</code> is a Node.js application which uses <code>puppeteer</code> for all the hard work. After seeing that such a REST API interface was possible via the <a rel="nofollow" href="https://github.com/cheeaun/puppetron" target="_blank"><code>puppetron</code> proof of concept</a> I set out to build a framework which will (eventually) provide the same feature set that Splash has, substituting <code>puppeteer</code>-fueled javascript for the Lua interface.</p>
<p>A REST API has a number of advantages over repeated CLI calls. First, each CLI call means more more <code>system()</code> call to start up a new process. You also need to manage Chrome binaries in that mode and are fairly limited in what you can do. With a REST API, Chrome loads once and then pages can be created at-will with no process startup overhead. Plus (once the API is finished) you’ll have far more control over what you can do. Again, this is not going to cover the same ground as Selenium, but should be of sufficient utility to add to your web-scraping toolbox.</p>
<h3>Installing gepetto</h3>
<p>There are instructions over at <a rel="nofollow" href="https://gitlab.com/hrbrmstr/gepetto" target="_blank">the repo</a> on installing <code>gepetto</code> but R users can try a shortcut by grabbing the latest version of <code>decapitated</code> from Git[La|Hu]b and running <code>decapitated::install_gepetto()</code> which should (hopefully) go as smoothly as this provided you have a fairly recent version of Node.js installed along with npm:</p>
<p><img width="780" src="https://i2.wp.com/rud.is/gifs/gepetto/install-gepetto.gif?resize=780%2C439&ssl=1" data-recalc-dims="1" height="439"></p>
<p>The installer provides some guidance should thing go awry. You’ll notice <code>gepetto</code> installs a current version of Chromium for your platform along with it, which helps to ensure smoother sailing than using the version of Chrome you may use for browsing.</p>
<h3>Working with gepetto</h3>
<p>Before showing off the R interface, it’s worth a look at the (still minimal) web interface. Bring up a terminal/command prompt and enter <code>gepetto</code>. You should see something like this:</p>
<pre><code>$ gepetto
🚀 Launch browser!
👍 gepetto running on: http://localhost:3000</code></pre>
<p>NOTE: You can use a different host/port by setting the <code>HOST</code> and <code>PORT</code> environment variables accordingly before startup.</p>
<p>You can then go to <a rel="nofollow" href="http://localhost:3000/" target="_blank">http://localhost:3000</a> in your browser and should see this:</p>
<p><a rel="nofollow" href="https://rud.is/b/2018/08/23/introducing-gepetto-a-splash-like-rest-api-to-headless-chrome/gweb/" target="_blank"><img data-orig-file="https://i0.wp.com/rud.is/b/wp-content/uploads/2018/08/gweb.png?fit=2696%2C1576&amp;ssl=1" data-orig-size="2696,1576" data-medium-file="https://i0.wp.com/rud.is/b/wp-content/uploads/2018/08/gweb.png?fit=300%2C175&amp;ssl=1" data-attachment-id="11419" data-image-meta="{" data-image-description="" src="https://i0.wp.com/rud.is/b/wp-content/uploads/2018/08/gweb.png?resize=780%2C456&ssl=1" data-permalink="https://rud.is/b/2018/08/23/introducing-gepetto-a-splash-like-rest-api-to-headless-chrome/gweb/" data-large-file="https://i0.wp.com/rud.is/b/wp-content/uploads/2018/08/gweb.png?fit=530%2C310&amp;ssl=1" alt="" data-image-title="gweb" width="450" data-comments-opened="1" class="wp-image-11419" data-recalc-dims="1"></a></p>
<p>Enter a URL into the input field and press the buttons! You can do quite a bit just from the web interface.</p>
<p>If you select “API Docs” (<a rel="nofollow" href="http://localhost:3000/documentation" target="_blank">http://localhost:3000/documentation</a>) you’ll get the Swagger-gen’d API documentation for all the API endpoints:</p>
<p><a rel="nofollow" href="https://rud.is/b/2018/08/23/introducing-gepetto-a-splash-like-rest-api-to-headless-chrome/gswag/" target="_blank"><img data-orig-file="https://i1.wp.com/rud.is/b/wp-content/uploads/2018/08/gswag.png?fit=2784%2C1664&amp;ssl=1" data-orig-size="2784,1664" data-medium-file="https://i1.wp.com/rud.is/b/wp-content/uploads/2018/08/gswag.png?fit=300%2C179&amp;ssl=1" data-attachment-id="11420" data-image-meta="{" data-image-description="" src="https://i1.wp.com/rud.is/b/wp-content/uploads/2018/08/gswag.png?resize=780%2C466&ssl=1" data-permalink="https://rud.is/b/2018/08/23/introducing-gepetto-a-splash-like-rest-api-to-headless-chrome/gswag/" data-large-file="https://i1.wp.com/rud.is/b/wp-content/uploads/2018/08/gswag.png?fit=530%2C317&amp;ssl=1" alt="" data-image-title="gswag" width="450" data-comments-opened="1" class="wp-image-11420" data-recalc-dims="1"></a></p>
<p>The Swagger definition JSON is also at <a rel="nofollow" href="http://localhost:3000/swagger.json" target="_blank">http://localhost:3000/swagger.json</a>.</p>
<p>The API documentation will be a bit more robust as the module’s corners are rounded out.</p>
<h3><em>“But, this is supposed to be an R post…”</em></h3>
<p>Yes. Yes it is.</p>
<p>If you followed along in the previous section and started <code>gepetto</code> from a command-line interface, kill the running service and fire up your favourite R environment and let’s scrape some content!</p>
<pre><code>library(rvest)
library(decapitated)
library(tidyverse)

gpid &lt;- start_gepetto()

gpid
## PROCESS 'gepetto', running, pid 60827.

gepetto() %>% 
  gep_active()
## [1] TRUE</code></pre>
<p>Anything other than a “running” response means there’s something wrong and you can use the various <code>processx</code> methods on that <code>gpid</code> object to inspect the error log. If you were able to run <code>gepetto</code> from the command line then it should be fine in R, too. The <code>gep()</code> function build a connection object and <code>gep_active()</code> tests an API endpoint to ensure you can communicate with the server.</p>
<p>Now, let’s try hitting a website that requires javascript. I’ll borrow an <a rel="nofollow" href="http://www.rladiesnyc.org/post/scraping-javascript-websites-in-r/" target="_blank">example from Brooke Watson</a>. The data for <a rel="nofollow" href="http://therapboard.com/" target="_blank">http://therapboard.com/</a> loads via javascript and will not work with <code>xml2::read_html()</code>.</p>
<pre><code>gepetto() %>% 
  gep_render_html(&quot;http://therapboard.com/&quot;) -> doc

html_nodes(doc, xpath=&quot;.//source[contains(@src, 'mp3')]&quot;) %>%  
  html_attr(&quot;src&quot;) %>% 
  head(10)
## [1] &quot;audio/2chainz_4.mp3&quot;        &quot;audio/2chainz_yeah2.mp3&quot;   
## [3] &quot;audio/2chainz_tellem.mp3&quot;   &quot;audio/2chainz_tru.mp3&quot;     
## [5] &quot;audio/2chainz_unh3.mp3&quot;     &quot;audio/2chainz_watchout.mp3&quot;
## [7] &quot;audio/2chainz_whistle.mp3&quot;  &quot;audio/2pac_4.mp3&quot;          
## [9] &quot;audio/2pac_5.mp3&quot;           &quot;audio/2pac_6.mp3&quot;</code></pre>
<p>Even with a Node.js and npm dependency, I think that’s a bit friendlier than interacting with <code>phantomjs</code>.</p>
<p>We can render a screenshot of a site as well. Since we’re not stealing content this way, I’m going to cheat a bit and grab the New York Times front page:</p>
<pre><code>gepetto() %>% 
  gep_render_magick(&quot;https://nytimes.com/&quot;)
##   format width height colorspace matte filesize density
## 1    PNG  1440   6828       sRGB  TRUE        0   72x72</code></pre>
<p><a rel="nofollow" href="https://rud.is/b/2018/08/23/introducing-gepetto-a-splash-like-rest-api-to-headless-chrome/nyt/" target="_blank"><img data-orig-file="https://i0.wp.com/rud.is/b/wp-content/uploads/2018/08/nyt.png?fit=1440%2C990&amp;ssl=1" data-orig-size="1440,990" data-medium-file="https://i0.wp.com/rud.is/b/wp-content/uploads/2018/08/nyt.png?fit=300%2C206&amp;ssl=1" data-attachment-id="11422" data-image-meta="{" data-image-description="" src="https://i0.wp.com/rud.is/b/wp-content/uploads/2018/08/nyt.png?resize=780%2C536&ssl=1" data-permalink="https://rud.is/b/2018/08/23/introducing-gepetto-a-splash-like-rest-api-to-headless-chrome/nyt/" data-large-file="https://i0.wp.com/rud.is/b/wp-content/uploads/2018/08/nyt.png?fit=530%2C364&amp;ssl=1" alt="" data-image-title="nyt" width="450" data-comments-opened="1" class="wp-image-11422" data-recalc-dims="1"></a></p>
<p>Astute readers will notice it returns a <code>magick</code> object so you can work with it immediately.</p>
<p>I’m still working out the interface for image capture and will also be supporting capturing the image of a CSS selector target. I mention that since the <code>gep_render_magick()</code> actually captured <em>the entire page</em> which you can <a rel="nofollow" href="https://rud.is/dl/nyt.png" target="_blank">see for yourself</a> (the thumbnail doesn’t do it justice).</p>
<p>Testing <code>gep_render_pdf()</code> is an exercise left to the reader.</p>
<h3>FIN</h3>
<p>The <code>gepetto</code> REST API is at version 0.1.0 meaning it’s new, raw and likely to change (quickly, too). Jump on board in whatever repo you’re more comfortable with and kick the tyres + file issues or PRs (on either or both projects) as you are wont to do.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2018/08/23/introducing-gepetto-a-splash-like-rest-api-to-headless-chrome/"> R â€“ rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/GIAam_hGpBE" height="1"></a></div>