<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/building-a-hacker-news-scraper-with-8-lines-of-r-code-using-rvest-library/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://datascienceplus.com/building-a-hacker-news-scraper-with-8-lines-of-r-code-using-rvest-library/"> R Programming – DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>There was a time when Web scraping was quite a difficult task requiring knowledge of XML Tree parsing and HTTP Requests. But with the likes of libraries like beautifulsoup (for Python) and rvest (for R), Web scraping has become a toy for any beginner to play with. </p>
<p>This post aims to explain how insanely simple it is <a rel="nofollow" href="https://datascienceplus.com/building-a-telecom-dictionary-scraping-web-using-rvest-in-r/" target="_blank">to build a scraper in R using rvest</a> and the site we have decided to scrape content from is <a rel="nofollow" href="https://news.ycombinator.com/" target="_blank">Hacker News Front Page</a>. </p>
<h3> Package Installation and Loading </h3>
<p><code>rvest</code> can be installed from <a rel="nofollow" href="https://cran.r-project.org/web/packages/rvest/index.html" target="_blank">CRAN</a> and loaded into R like below:</p>
<pre>
install.packages('rvest')
library(rvest)
</pre>
<p><code>read_html()</code> function of <code>rvest</code> can be used to extract the html content of the url given as the arguemnt for read_html function.</p>
<pre>
content &lt;- read_html('https://news.ycombinator.com/&amp;#x27)
</pre>
<p>For this above <code>read_html()</code> to work without any concern, Please make sure you are not behind any organization firewall and if so, configure your RStudio with a proxy to bypass the firewall otherwise you might face <code>connection timed out error</code>.</p>
<p>Below is the screenshot of HN front page layout (with key elements highlighted):</p>
<p><a rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/12/hn_page_layout.png?ssl=1" target="_blank"><img sizes="(max-width: 1301px) 100vw, 1301px" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/12/hn_page_layout.png?w=450&ssl=1" alt="" class="wp-image-10084" data-recalc-dims="1"></a></p>
<p>Now, with the html content of the Hacker News front page loaded into the R object <i>content</i>, Let us extract the data that we need – starting from the Title. The most important aspect of making any web scraping assignment successful is to identify the right css selector or xpath values of the html elements whose values are supposed to be scraped and the easiest way to get the right element value is to use <code>the inspect tool </code> in Developer Tools of any browser. </p>
<p>Here’s the screenshot of the css selector value highlighted by Chrome Inspect Tool when hovered over Title of the links present in Hacker News Frontpage. </p>
<p><a rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/12/hn_page_title.png?ssl=1" target="_blank"><img sizes="(max-width: 1366px) 100vw, 1366px" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/12/hn_page_title.png?w=450&ssl=1" alt="" class="wp-image-10085" data-recalc-dims="1"></a></p>
<pre>
title &lt;- content %>% html_nodes('a.storylink') %>% html_text()
<em>title
[1] &quot;Magic Leap One&quot; [2] &quot;Show HN: Termipal – native micro-GUIs for shell scripts and command line apps&quot; [3] &quot;Tokio internals: Understanding Rust's async I/O framework&quot; [4] &quot;Funding Yourself as a Free Software Developer&quot; [5] &quot;US Federal Ban on Making Lethal Viruses Is Lifted&quot; [6] &quot;Pass-Thru Income Deduction&quot; [7] &quot;Orson Welles' first attempt at movie-making&quot; [8] &quot;D’s Newfangled Name Mangling&quot; [9] &quot;Apple Plans Combined iPhone, iPad and Mac Apps to Create One User Experience&quot; [10] &quot;LiteDB – A .NET NoSQL Document Store in a Single Data File&quot; [11] &quot;Taking a break from Adblock Plus development&quot; [12] &quot;SpaceX’s Falcon Heavy rocket sets up at Cape Canaveral ahead of launch&quot; [13] &quot;This is not a new year’s resolution&quot; [14] &quot;Artists and writers whose works enter the public domain in 2018&quot; [15] &quot;Open Beta of Texpad 1.8, macOS LaTeX editor with integrated realtime typesetting&quot;
[16] &quot;The triumph and near-tragedy of the first Moon landing&quot; [17] &quot;Retrotechnology – PC desktop screenshots from 1983-2005&quot; [18] &quot;Google Maps' Moat&quot; [19] &quot;Regex Parser in C Using Continuation Passing&quot; [20] &quot;AT&amp;T giving $1000 bonus to all its employees because of tax reform&quot; [21] &quot;How a PR Agency Stole Our Kickstarter Money&quot; [22] &quot;Google Hangouts now on Firefox without plugins via WebRTC&quot; [23] &quot;Ubuntu 17.10 corrupting BIOS of many Lenovo laptop models&quot; [24] &quot;I Know What You Download on BitTorrent&quot; [25] &quot;Carrie Fisher’s Private Philosophy Coach&quot; [26] &quot;Show HN: Library of API collections for Postman&quot; [27] &quot;Uber is officially a cab firm, says European court&quot; [28] &quot;The end of the Iceweasel Age (2016)&quot; [29] &quot;Google will turn on native ad-blocking in Chrome on February 15&quot; [30] &quot;Bitcoin Cash deals frozen as insider trading is probed&quot;
</em>
</pre>
<p>Since rvest package supports pipe %>% operator, content (the R object containing the content of the html page read with read_html) can be piped with <code>html_nodes()</code> that takes css selector or xpath as its arugment and then extract respective xml tree (or html node value) whose text value could be extracted with <code>html_text()</code> function. The beauty of rvest is that it abstracts the entire xml parsing operation under the hood of functions like html_nodes() and html_text() making it easier for us to achieve our scraping goal with minimal code. </p>
<p>Like Title, The css selector value of other required elements of the web page can be identified with the Chrome Inspect tool and passed as an argument to html_nodes() function and respective values can be extracted and stored in R objects. </p>
<pre>
link_domain &lt;- content %>% html_nodes('span.sitestr') %>% html_text()
score &lt;- content %>% html_nodes('span.score') %>% html_text()
age &lt;- content %>% html_nodes('span.age') %>% html_text()
</pre>
<p>With all the essential pieces of information extracted from the page, an R data frame can be made with the extracted elements to put the extracted data into a structured format. </p>
<pre>
df &lt;- data.frame(title = title, link_domain = link_domain, score = score, age = age)
</pre>
<p>Below is the screenshot of the final dataframe in RStudio viewer:<br>
<a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/12/hn_scraped_dataframe.png?ssl=1" target="_blank"><img sizes="(max-width: 904px) 100vw, 904px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/12/hn_scraped_dataframe.png?w=450&ssl=1" alt="" class="wp-image-10086" data-recalc-dims="1"></a></p>
<p>Thus in just 8 lines of code, We have successfully built a Hacker News Scraper in R using rvest package and this scraper could be for a variety of purposes like News Reader, Summarizer, Text Analytics and much more. While a lot more things could be done with rvest, this post is kept simple to explain how easily a web scraper could be built with rvest. The code used here is available <a rel="nofollow" href="https://github.com/amrrs/HN_scraper_in_R" target="_blank">on my github</a>.</p>
<ul></ul>
<p><strong>
<p>Related Post</p>
<p></p></strong>
<ol>
<li><a rel="nofollow" href="https://datascienceplus.com/building-a-telecom-dictionary-scraping-web-using-rvest-in-r/" title="Permanent Link toBuilding a Telecom Dictionary scraping web using rvest in R" target="_blank">Building a Telecom Dictionary scraping web using rvest in R</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/scraping-javascript-rendered-web-content-using-r/" title="Permanent Link toScraping Javascript-rendered web content using R" target="_blank">Scraping Javascript-rendered web content using R</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/analysing-cryptocurrency-market-in-r/" title="Permanent Link toAnalysing Cryptocurrency Market in R" target="_blank">Analysing Cryptocurrency Market in R</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/time-series-analysis-in-r-part-3-getting-data-from-quandl/" title="Permanent Link toTime Series Analysis in R Part 3: Getting Data from Quandl" target="_blank">Time Series Analysis in R Part 3: Getting Data from Quandl</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/pulling-data-out-of-census-spreadsheets-using-r/" title="Permanent Link toPulling Data Out of Census Spreadsheets Using R" target="_blank">Pulling Data Out of Census Spreadsheets Using R</a></li>
</ol>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://datascienceplus.com/building-a-hacker-news-scraper-with-8-lines-of-r-code-using-rvest-library/"> R Programming – DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/ssWySeGHBYs" height="1"></a></div>