<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/scrapeover-friday-a-k-a-another-r-scraping-makeover/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/05/05/scrapeover-friday-a-k-a-another-r-scraping-makeover/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I caught a glimpse of a tweet by <a target="_blank" rel="nofollow" href="http://twitter.com/dataandme">@dataandme</a> on Friday:</p>
<blockquote data-lang="en" class="twitter-tweet"><p dir="ltr" lang="en">Using R &amp; rvest to explore Malaysian property mkt: &quot;Web Scraping: The Sequel, Propwall.my&quot; <a target="_blank" rel="nofollow" href="https://t.co/daZOOJJfPN">https://t.co/daZOOJJfPN</a> <a target="_blank" rel="nofollow" href="https://twitter.com/hashtag/rstats?src=hash">#rstats</a> <a target="_blank" rel="nofollow" href="https://twitter.com/hashtag/rvest?src=hash">#rvest</a> <a target="_blank" rel="nofollow" href="https://t.co/u6QMhm4M3e">pic.twitter.com/u6QMhm4M3e</a></p>
<p>— Mara Averick (<a target="_blank" rel="nofollow" href="http://twitter.com/dataandme">@dataandme</a>) <a target="_blank" rel="nofollow" href="https://twitter.com/dataandme/status/860604292738285568">May 5, 2017</a></p></blockquote>
<p></p>
<p>Mara is — without a doubt — the best data science promoter in the Twitterverse. She seems to have her finger on the pulse of everything that’s happening in the data science world and is one of the most ardent amplifiers there is.</p>
<p>The post she linked to was a bit older (2015) and had a very “stream of consciousness” feel to it. I actually wish more R folks took to their blogs like this to post their explorations into various topics. The code in this post likely worked at the time it was posted and accomplished the desired goal (which means it was ultimately decent code). Said practice will ultimately help both you and others.</p>
<h3>Makeover Time</h3>
<p>As I’ve noted before, web scraping has some rules, even though they can be tough to find. This post made a very common mistake of not putting in a time delay between requests (a cardinal scraping rule) which we’ll fix in a moment.</p>
<p>There are a few other optimizations we can make. The first is moving from a <code>for</code> loop to something a bit more vectorized. Another is to figure out how many pages we need to scrape from information in the first set of results. </p>
<p>However, an even bigger one is to take advantage of the underlying XHR <code>POST</code> request that the new version of the site ultimately calls (it appears this site has undergone some changes since the blog post and it’s unlikely the code in the post actually works now).</p>
<p>Let’s start by setting up a function to grab individual pages:</p>
<pre><code>library(httr)
library(rvest)
library(stringi)
library(tidyverse)

get_page &lt;- function(i=1, pb=NULL) {
  
  if (!is.null(pb)) pb$tick()$print()
  
  POST(url = &quot;http://www.propwall.my/wp-admin/admin-ajax.php&quot;, 
       body = list(action = &quot;star_property_classified_list_change_ajax&quot;, 
                   tab = &quot;Most Relevance&quot;, 
                   page = as.integer(i), location = &quot;Mont Kiara&quot;, 
                   category = &quot;&quot;, listing = &quot;For Sale&quot;, 
                   price = &quot;&quot;, keywords = &quot;Mont Kiara, Kuala Lumpur&quot;, 
                   filter_id = &quot;17&quot;, filter_type = &quot;Location&quot;, 
                   furnishing = &quot;&quot;, builtup = &quot;&quot;, 
                   tenure = &quot;&quot;, view = &quot;list&quot;, 
                   map = &quot;on&quot;, blurb = &quot;0&quot;), 
       encode = &quot;form&quot;) -> res
  
  stop_for_status(res)
  
  res &lt;- content(res, as=&quot;parsed&quot;) 
  
  Sys.sleep(sample(seq(0,2,0.5), 1))
  
  res
  
}</code></pre>
<p>The <code>i</code> parameter gets passed into the body of the <code>POST</code> request. You can find that XHR <code>POST</code> request via the Network tab of your browser Developer Tools view. You can either transcribe it by hand or use the <code>curlconverter</code> package (which is temporarily off CRAN so you’ll need to get it from <a target="_blank" rel="nofollow" href="https://github.com/hrbrmstr/curlconverter">github</a>) to auto-convert it to an <code>httr::VERB</code> request.</p>
<p>We also add a parameter (default to <code>NULL</code>) to support the use of a progress bar (so we can see what’s going on). If we pass in a populated <code>dplyr</code> progress bar, this will tick it down for us.</p>
<p>Now, we can use that to get the total number of listings.</p>
<pre><code>get_page(1) %>% 
  html_node(xpath=&quot;.//a[contains(., 'Classifieds:')]&quot;) %>% 
  html_text() %>% 
  stri_match_last_regex(&quot;([[:digit:],]+)$&quot;) %>% 
  .[,2] %>% 
  stri_replace_all_fixed(&quot;,&quot;, &quot;&quot;) %>% 
  as.numeric() -> classified_ct

total_pages &lt;- 1 + (classified_ct %/% 20)</code></pre>
<p>We’ll setup another function to extract the listing URLs and titles:</p>
<pre><code>get_listings &lt;- function(pg) {
  data_frame(
    link = html_nodes(pg, &quot;div#list-content > div.media * h4.media-heading > a:nth-of-type(1)&quot; ) %>%  html_attr(&quot;href&quot;),
    description = html_nodes(pg, &quot;div#list-content > div.media * h4.media-heading > a:nth-of-type(1)&quot; ) %>% html_text(trim = TRUE)  
  )
}</code></pre>
<p>Rather than chain calls to <code>html_nodes()</code> we take advantage of well-formed CSS selectors (which ultimately gets auto-translated to XPath strings). This has the advantage of speed (though that’s not necessarily an issue when web scraping) as well as brevity. </p>
<p>Now, we’ll scrape all the listings:</p>
<pre><code>pb &lt;- progress_estimated(total_pages)
listings_df &lt;- map_df(1:total_pages, ~get_listings(get_page(.x, pb)))</code></pre>
<p>Yep. That’s it. Everything’s been neatly abstracted into functions and we’ve taken advantage of some modern R idioms to accomplish our first task.</p>
<h2>FIN</h2>
<p>With the above code you should be able to do your own makeover of the remaining code in the original post. Remember to:</p>
<ul>
<li>add a delay when you sequentially scrape pages from a site</li>
<li>abstract out common operations into functions</li>
<li>take advantage of <code>purrr</code> functions (or built-in <code>*apply</code> functions) to avoid <code>for</code> loops</li>
</ul>
<p>I’ll close with a note about adhering to site terms of service / terms and conditions. Nothing I found when searching for ToS/ToC on the site suggested that scraping, automated grabbing or use of the underlying data in bulk was prohibited. Many sites have such restrictions — like IMDB (I mention that as it’s been used alot lately by R folks and it really shouldn’t be). LinkedIn recently sued scrapers for ToS such violations. </p>
<p>I fundamentally believe violating ToS is unethical behavior and should be avoided just on those grounds. When I come across sites I need information from that have restrictive ToS I contact the site owner (when I can find them) and ask them for permission and have only been refused a small handful of times. Given those recent legal actions, it’s also to better be safe than sorry. </p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2017/05/05/scrapeover-friday-a-k-a-another-r-scraping-makeover/"> R – rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/3ODcKjq--9g"></a></div>