<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/how-to-scrape-data-from-a-javascript-website-with-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://velaco.github.io/how-to-scrape-data-from-javascript-websites-with-R/"> Curiosity Killed the Cat</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In September 2017, I found myself working on a project that required odds data for football. At the time I didn’t know about resources such as <a rel="nofollow" href="http://www.football-data.co.uk/" target="_blank">Football-Data</a> or the <a rel="nofollow" href="https://the-odds-api.com/" target="_blank">odds-api</a>, so I decided to build a scraper to collect data directly from the bookmakers. However, most of them used JavaScript to display their odds, so I couldn’t collect the data with R and <em>rvest</em> alone. In this article, I’ll demonstrate how PhantomJS can be used with R to scrape JS-rendered content from the web.</p>
<p><span></span></p>
<h2>Requirements</h2>
<p>In addition to R’s base packages, I’ll need the following for this example:</p>
<ul>
<li><a rel="nofollow" href="https://cran.r-project.org/package=rvest" target="_blank">rvest</a> – This package for R is a wrapper for <em>xml2</em> and <em>httr</em> packages. It allows you to download and extract data from HTML and XML.</li>
<li><a rel="nofollow" href="http://phantomjs.org/" target="_blank">PhantomJS</a> – A headless browser that can be used to access webpages and extract information from them, among other things.</li>
<li><a rel="nofollow" href="https://selectorgadget.com/" target="_blank">Selector Gadget</a> – This one is optional, but it makes life easier because it isn’t necessary to go through the source code to find the XPath and/or CSS selectors for extracting the information you need from an HTML document.</li>
</ul>
<h2>Collecting Data</h2>
<p>For this example I’ll stick to the original intent of the project and collect odds from an online bookmaker. <a rel="nofollow" href="https://www.paddypower.com/bet" target="_blank">Paddy Power</a> (no affiliation) uses JavaScript and also doesn’t have an API to my knowledge, so I’ll collect odds from their website.</p>
<h3>Without PhantomJS</h3>
<p>To collect odds data with <em>rvest</em>, we need to:</p>
<ol>
<li>Read the HTML file from a URL.</li>
<li>Extract the HTML elements containing the odds. (The <code><div> elements that contain the data I am interested in have the attribute <code>class=&quot;avb-item&quot;</code>.)
<li>Extract the contents of the elements.</li>
<li>Clean the data if necessary.</li>

<div>
<div>
<pre><code><span>></span><span> </span><span>withoutJS</span><span> </span><span>&lt;-</span><span> </span><span>xml2</span><span>::</span><span>read_html</span><span>(</span><span>&quot;https://www.paddypower.com/football?tab=today&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>+</span><span> </span><span>rvest</span><span>::</span><span>html_nodes</span><span>(</span><span>&quot;.avb-item&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>+</span><span> </span><span>rvest</span><span>::</span><span>html_text</span><span>()</span><span>
</span><span>></span><span> </span><span>withoutJS</span><span>
</span><span>character</span><span>(</span><span>0</span><span>)</span><span>
</span></code></pre>
</div>
</div>
<p>We ended up with an empty character vector because the HTML document contains only JS snippets in the source code instead of the content we see in the browser. That means the CSS selector we can see in the browser does not exist in the document R is trying to extract data from.</p>
<h3>With PhantomJS</h3>
<p>The first step is to save the following script in a separate file:</p>
<div>
<div>
<pre><code><span>// scraper_PaddyPower.js</span>

<span>// Create a webpage object</span>
<span>var</span> <span>page</span> <span>=</span> <span>require</span><span>(</span><span>'webpage'</span><span>).</span><span>create</span><span>();</span>

<span>// Include the File System module for writing to files</span>
<span>var</span> <span>fs</span> <span>=</span> <span>require</span><span>(</span><span>'fs'</span><span>);</span>

<span>// Specify source and path to output file</span>
<span>var</span> <span>url</span>  <span>=</span> <span>'https://www.paddypower.com/football?tab=today'</span>
<span>var</span> <span>path</span> <span>=</span> <span>'odds_PaddyPower.html'</span>

<span>page</span><span>.</span><span>open</span><span>(</span><span>url</span><span>,</span> <span>function</span> <span>(</span><span>status</span><span>)</span> <span>{</span>
  <span>var</span> <span>content</span> <span>=</span> <span>page</span><span>.</span><span>content</span><span>;</span>
  <span>fs</span><span>.</span><span>write</span><span>(</span><span>path</span><span>,</span><span>content</span><span>,</span><span>'w'</span><span>)</span>
  <span>phantom</span><span>.</span><span>exit</span><span>();</span>
<span>});</span>
</code></pre>
</div>
</div>
<p>The purpose of this script is to retrieve the HTML file from the specified URL and store it into a local HTML file, so that R can read contents from that file instead of reading the contents directly from the URL.</p>
<p>We can then use the <code>system()</code> function to invoke the script from R and repeat the steps outlined in the attempt without PhantomJS to extract the data:</p>
<div>
<div>
<pre><code><span>></span><span> </span><span>system</span><span>(</span><span>&quot;/usr/local/phantomjs-2.1.1/bin/phantomjs scraper_PaddyPower.js&quot;</span><span>)</span><span>
</span><span>></span><span> </span><span>withJS</span><span> </span><span>&lt;-</span><span> </span><span>xml2</span><span>::</span><span>read_html</span><span>(</span><span>&quot;odds_PaddyPower.html&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>+</span><span> </span><span>rvest</span><span>::</span><span>html_nodes</span><span>(</span><span>&quot;.avb-item&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>+</span><span> </span><span>rvest</span><span>::</span><span>html_text</span><span>()</span><span>
</span></code></pre>
</div>
</div>
<p><em><strong>Note for Windows users</strong>: The invocation of the script has the same format as the command for Linux, but you may have to add .exe to phantomjs for it to work.</em></p>
<p>The content is currently a mess and the variable is one large chunk of text, i.e. a character vector of length 1. If I remove the white space and split it into individual elements, here’s what I get:</p>
<div>
<div>
<pre><code><span>></span><span> </span><span>pp_odds_data</span><span> </span><span>&lt;-</span><span> </span><span>withJS</span><span> </span><span>%>%</span><span> </span><span>+</span><span> </span><span>gsub</span><span>(</span><span>&quot;[\n\t]&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>.</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>+</span><span> </span><span>stringr</span><span>::</span><span>str_trim</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>+</span><span> </span><span>gsub</span><span>(</span><span>&quot;\\s+&quot;</span><span>,</span><span> </span><span>&quot; &quot;</span><span>,</span><span> </span><span>.</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>+</span><span> </span><span>strsplit</span><span>(</span><span>&quot; &quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>+</span><span> </span><span>unlist</span><span>()</span><span>
</span><span>></span><span> </span><span>pp_odds_data</span><span>[</span><span>1</span><span>:</span><span>40</span><span>]</span><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>&quot;Bahrain&quot;</span><span> </span><span>&quot;Tajikistan&quot;</span><span> </span><span>&quot;4/7&quot;</span><span> </span><span>&quot;13/5&quot;</span><span> </span><span>&quot;5/1&quot;</span><span> </span><span>&quot;17:00&quot;</span><span> </span><span>&quot;More&quot;</span><span>
</span><span>[</span><span>8</span><span>]</span><span> </span><span>&quot;Bets&quot;</span><span> </span><span>&quot;>&quot;</span><span> </span><span>&quot;Hatayspor&quot;</span><span> </span><span>&quot;Genclerbirligi&quot;</span><span> </span><span>&quot;1/14&quot;</span><span> </span><span>&quot;13/2&quot;</span><span> </span><span>&quot;19/1&quot;</span><span>
</span><span>[</span><span>15</span><span>]</span><span> </span><span>&quot;More&quot;</span><span> </span><span>&quot;Bets&quot;</span><span> </span><span>&quot;>&quot;</span><span> </span><span>&quot;Darica&quot;</span><span> </span><span>&quot;Genclerbirligi&quot;</span><span> </span><span>&quot;Antalyaspor&quot;</span><span> </span><span>&quot;4/1&quot;</span><span>
</span><span>[</span><span>22</span><span>]</span><span> </span><span>&quot;23/10&quot;</span><span> </span><span>&quot;4/7&quot;</span><span> </span><span>&quot;More&quot;</span><span> </span><span>&quot;Bets&quot;</span><span> </span><span>&quot;>&quot;</span><span> </span><span>&quot;Akhisar&quot;</span><span> </span><span>&quot;Belediye&quot;</span><span>
</span><span>[</span><span>29</span><span>]</span><span> </span><span>&quot;Fatih&quot;</span><span> </span><span>&quot;Karagumrukspor&quot;</span><span> </span><span>&quot;1/3&quot;</span><span> </span><span>&quot;10/3&quot;</span><span> </span><span>&quot;6/1&quot;</span><span> </span><span>&quot;16:00&quot;</span><span> </span><span>&quot;More&quot;</span><span>
</span><span>[</span><span>36</span><span>]</span><span> </span><span>&quot;Bets&quot;</span><span> </span><span>&quot;>&quot;</span><span> </span><span>&quot;Giresunspor&quot;</span><span> </span><span>&quot;Fenerbahce&quot;</span><span> </span><span>&quot;9/2&quot;</span><span>
</span></code></pre>
</div>
</div>
<p>There’s still some useless data mixed in, and there is no guarantee that I can turn this vector into a data frame to store and analyze the data. In this particular case, team names can have more than two words, so I should extract information about matches and odds separately:</p>
<div>
<div>
<pre><code><span>></span><span> </span><span>matches</span><span> </span><span>&lt;-</span><span> </span><span>xml2</span><span>::</span><span>read_html</span><span>(</span><span>&quot;odds_PaddyPower.html&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>+</span><span> </span><span>rvest</span><span>::</span><span>html_nodes</span><span>(</span><span>&quot;.avb-item .ui-scoreboard-coupon&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>+</span><span> </span><span>rvest</span><span>::</span><span>html_text</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>+</span><span> </span><span>stringr</span><span>::</span><span>str_trim</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>+</span><span> </span><span>gsub</span><span>(</span><span>&quot;\\s\\s+&quot;</span><span>,</span><span>&quot; vs &quot;</span><span>,</span><span> </span><span>.</span><span>)</span><span>
</span><span>></span><span> </span><span>odds</span><span> </span><span>&lt;-</span><span> </span><span>xml2</span><span>::</span><span>read_html</span><span>(</span><span>&quot;odds_PaddyPower.html&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>+</span><span> </span><span>rvest</span><span>::</span><span>html_nodes</span><span>(</span><span>&quot;.avb-item .btn-odds&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>+</span><span> </span><span>rvest</span><span>::</span><span>html_text</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>+</span><span> </span><span>stringr</span><span>::</span><span>str_trim</span><span>()</span><span>
</span><span>></span><span> </span><span>odds_selector</span><span> </span><span>&lt;-</span><span> </span><span>seq</span><span>(</span><span>1</span><span>,</span><span> </span><span>length</span><span>(</span><span>odds</span><span>),</span><span> </span><span>3</span><span>)</span><span>
</span><span>></span><span> </span><span>pp_odds_df</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>Match</span><span> </span><span>=</span><span> </span><span>matches</span><span>,</span><span> </span><span>Home</span><span> </span><span>=</span><span> </span><span>odds</span><span>[</span><span>odds_selector</span><span>],</span><span> </span><span>Draw</span><span> </span><span>=</span><span> </span><span>odds</span><span>[(</span><span>odds_selector</span><span> </span><span>+</span><span> </span><span>1</span><span>)],</span><span> </span><span>Away</span><span> </span><span>=</span><span> </span><span>odds</span><span>[(</span><span>odds_selector</span><span> </span><span>+</span><span> </span><span>2</span><span>)])</span><span>
</span><span>></span><span> </span><span>head</span><span>(</span><span>pp_odds_df</span><span>)</span><span>
</span><span>Match</span><span> </span><span>Home</span><span> </span><span>Draw</span><span> </span><span>Away</span><span>
</span><span>1</span><span> </span><span>Bahrain</span><span> </span><span>vs</span><span> </span><span>Tajikistan</span><span> </span><span>4</span><span>/</span><span>7</span><span> </span><span>13</span><span>/</span><span>5</span><span> </span><span>5</span><span>/</span><span>1</span><span>
</span><span>2</span><span> </span><span>Hatayspor</span><span> </span><span>vs</span><span> </span><span>Genclerbirligi</span><span> </span><span>1</span><span>/</span><span>14</span><span> </span><span>13</span><span>/</span><span>2</span><span> </span><span>19</span><span>/</span><span>1</span><span>
</span><span>3</span><span> </span><span>Darica</span><span> </span><span>Genclerbirligi</span><span> </span><span>vs</span><span> </span><span>Antalyaspor</span><span> </span><span>4</span><span>/</span><span>1</span><span> </span><span>23</span><span>/</span><span>10</span><span> </span><span>4</span><span>/</span><span>7</span><span>
</span><span>4</span><span> </span><span>Akhisar</span><span> </span><span>Belediye</span><span> </span><span>vs</span><span> </span><span>Fatih</span><span> </span><span>Karagumrukspor</span><span> </span><span>1</span><span>/</span><span>3</span><span> </span><span>10</span><span>/</span><span>3</span><span> </span><span>6</span><span>/</span><span>1</span><span>
</span><span>5</span><span> </span><span>Giresunspor</span><span> </span><span>vs</span><span> </span><span>Fenerbahce</span><span> </span><span>9</span><span>/</span><span>2</span><span> </span><span>11</span><span>/</span><span>4</span><span> </span><span>1</span><span>/</span><span>2</span><span>
</span><span>6</span><span> </span><span>Botosani</span><span> </span><span>vs</span><span> </span><span>Dunarea</span><span> </span><span>Calarasi</span><span> </span><span>8</span><span>/</span><span>13</span><span> </span><span>5</span><span>/</span><span>2</span><span> </span><span>9</span><span>/</span><span>2</span><span>
</span></code></pre>
</div>
</div>
<p>Now I can further transform the data before storing and analyzing it, e.g. convert fractional odds into decimal, split the Match column into “Home Team” and “Away Team” columns, and so on.</p>
<h2>Final Comments</h2>
<p>Although the script I used still works today, I should note that the development of PhantomJS is currently suspended and its <a rel="nofollow" href="https://github.com/ariya/phantomjs" target="_blank">git repository</a> is archived. However, there are many ways to collect data from a JavaScript-rendered website, so you can also try using <a rel="nofollow" href="https://datascienceplus.com/scraping-javascript-rendered-web-content-using-r/" target="_blank"><em>rvest</em> and <em>V8</em> from R</a> or <a rel="nofollow" href="https://medium.com/@hoppy/how-to-test-or-scrape-javascript-rendered-websites-with-python-selenium-a-beginner-step-by-c137892216aa" target="_blank">Python and Selenium</a> to collect the data you need. Good luck!</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://velaco.github.io/how-to-scrape-data-from-javascript-websites-with-R/"> Curiosity Killed the Cat</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/77vaUcLm1kw" height="1"></a></div></div></code></ol>