<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/building-a-telecom-dictionary-scraping-web-using-rvest-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://datascienceplus.com/building-a-telecom-dictionary-scraping-web-using-rvest-in-r/"> R Programming – DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>One of the biggest problems in Business to carry out any analysis is the availability of Data. That is where in many cases, Web Scraping comes very handy in creating that data that’s required. Consider the following case: To perform text analysis on Textual Data collected in a Telecom Company as part of Customer Feedback or Reviews, primarily requires a dictionary of Telecom Keywords. But such a dictionary is hard to find out-of-box. Hence as an Analyst, the most obvious thing to do when such dictionary doesn’t exist is to build one. Hence this article aims to help beginners get started with web scraping with <code>rvest</code> in R and at the same time, building a Telecom Dictionary by the end of this exercise. </p>
<h3>Disclaimer</h3>
<p>Web Scraping is not allowed by some websites. Kindly check the site’s Terms of Service before Scraping. This post is only for Educational-purpose. </p>
<p>A simple Google search for Telecom Glossary results in this URL:<br>
<a rel="nofollow" href="http://www.atis.org/glossary/" target="_blank">Atis.org</a> of which the required Telecom Dictionary could be built. </p>
<p>Let us start by loading the required libraries:</p>
<pre>
#load rvest and stringr library
library(rvest)
library(stringr)
library(dplyr)
</pre>
<p><code>rvest</code> is a web scraping library in R that makes it easier to write common scraping tasks (to scrape useful information from web pages) without getting our head into xml parsing. rvest can be downloaded <a rel="nofollow" href="https://cran.r-project.org/web/packages/rvest/index.html" target="_blank">from CRAN</a> and the development version is also available <a rel="nofollow" href="https://github.com/hadley/rvest" target="_blank">on Github</a>. </p>
<p>It could be seen in the above-mentioned URL that the Glossary words are listed alphabetically from A to Z with a separate link / URL for every (starting) letter. Clicking ‘A’ takes us to this link: <a rel="nofollow" href="http://www.atis.org/glossary/definitionsList.aspx?find=A&kw=0" target="_blank">atis.org</a> that lists all the keywords with starting letter ‘A’. If you closely look at the URL, the code that’d be written for one letter (here in our case, ‘A’) could be easily replicated for other letters since ‘A’ is part of the URL which will be the only change for the link of other Alphabets.</p>
<p>Let us assign this URL to an R object <em>url</em> which could be passed on as the paramater to rvest’s <code>read_html() </code>function. </p>
<pre>
#url whose content to be scraped
url &lt;- 'http://www.atis.org/glossary/definitionsList.aspx?find=A&amp;kw=0'
#extracting the content of the given url
#url_content &lt;- read_html('http://www.atis.org/glossary/definitionsList.aspx?find=A&amp;kw=0')
url_content &lt;- read_html(url)
</pre>
<p><code>read_html()</code> parses the html page of the given url (as its parameter) and saves the result as an xml object. </p>
<p>To reiterate the objective, we are trying get the list of Telecom Keywords and as per the screenshot above, You could see that the Keywords are listed as Hyperlinks in the given url. </p>
<p>Hyperlinks in HTML is written in the following syntax:</p>
<pre>
<a rel="nofollow" href="https://www.google.com" target="_blank">Google</a>
</pre>
<p>Google is the Link Text Label that a browser would render and when clicked would take us to <a rel="nofollow" href="https://www.google.com/" target="_blank">www.google.com</a>. Hence it is evident that anchor tags in the page is what we are supposed to scrape/extract. But the issue with the current page is that, It’s not just the Keywords that are represented as Anchor Text (links) in the page but also there are a lot of other links (anchor tags) in the page. Hence to extract only the required information and filter out the irrelevant information, we need to find a pattern that helps us extract only the keywords links.</p>
<p>Have a look at this screenshot:<br>
<a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/12/telecom-dictionary-list.png?ssl=1" target="_blank"><img sizes="(max-width: 1095px) 100vw, 1095px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/12/telecom-dictionary-list.png?w=450&ssl=1" alt="" class="wp-image-9856" data-recalc-dims="1"></a></p>
<p>This screenshot shows that while the keywords are also represented as hyperlinks (Anchor Text), the differentiator is this ‘id’ element in the url. Only the links of the keywords have got this ‘id’ in the url and hence we can try extracting two information from the current page to get only the relevant information which in our case is – The Keywords: 1. Href value / Actual URL 2. Link Text. </p>
<pre>
#extracting all the links from the page 
links &lt;-  url_content %>% html_nodes('a') %>% html_attr('href')
#extracting all thhe link text from the page
text &lt;- url_content %>% html_nodes('a') %>% html_text()
</pre>
<p>With the example Hyperlink discussed earlier, the above code gives two informations. </p>
<p><code><a rel="nofollow" href="https://www.google.com/" target="_blank">www.google.com</a></code> is saved in <code>links</code> and <code>Google</code> is saved in <code>text</code>.</p>
<p>With <code>links</code> and <code>text</code> as columns, Let us build a rough dictionary (that’s not yet cleaned/filtered). </p>
<pre>
#creating a new dictonary of links and text extracted above
rough_dict &lt;- data.frame(links,text, stringsAsFactors = F)
head(rough_dict)
<em> links text
1 http://www.atis.org 2 definitionsList.aspx?find=A&amp;kw=0 A
3 definitionsList.aspx?find=B&amp;kw=0 B
4 definitionsList.aspx?find=C&amp;kw=0 C
5 definitionsList.aspx?find=D&amp;kw=0 D
6 definitionsList.aspx?find=E&amp;kw=0 E
</em>
</pre>
<p>As displayed above, <code>rough_dict</code> contains both signal (keywords) and noise (irrelevant links) and we have to filter the irrelevant out with the ‘<em>id</em>‘ pattern that we learnt earlier. </p>
<pre>
#filtering glossary terms leaving out irrelevant information
fair_dict &lt;- rough_dict %>% filter(str_detect(links, 'id')) %>% select(text)
tail(fair_dict)
<em> text
657 AN
658 axial propagation constant
659 analog component
660 axial ratio
661 analog computer
662 axial ray
</em></pre>
<p>And that’s how using <code>str_detect()</code> we can keep only links with ‘id’ in it and filter out the rest and building our <code>fair_dict</code>. As displayed in the above output, We have got 662 Keywords just for the letter ‘A’ and the same exercise could be repeated for the other letters available on the site. The only change that’s required is the <code>url</code> object. For example, Like this:</p>
<pre>
url &lt;- 'http://www.atis.org/glossary/definitionsList.aspx?find=B&amp;kw=0'
</pre>
<p>Note the ‘B’ in it and the same could be done for other available letters. This process could be further improved by making this scraping part a function and looping the function call over a character vector with all the available letters (which ideally is beyond the scope of this article’s objective and hence left out). The complete code used in this article is available on <a rel="nofollow" href="https://github.com/amrrs/datascienceplus_codes/blob/master/building_telecom_dictionary.R" target="_blank">my Github</a>.</p>
<ul></ul>
<p><strong>
<p>Related Post</p>
<p></p></strong>
<ol>
<li><a rel="nofollow" href="https://datascienceplus.com/scraping-javascript-rendered-web-content-using-r/" title="Permanent Link toScraping Javascript-rendered web content using R" target="_blank">Scraping Javascript-rendered web content using R</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/analysing-cryptocurrency-market-in-r/" title="Permanent Link toAnalysing Cryptocurrency Market in R" target="_blank">Analysing Cryptocurrency Market in R</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/time-series-analysis-in-r-part-3-getting-data-from-quandl/" title="Permanent Link toTime Series Analysis in R Part 3: Getting Data from Quandl" target="_blank">Time Series Analysis in R Part 3: Getting Data from Quandl</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/pulling-data-out-of-census-spreadsheets-using-r/" title="Permanent Link toPulling Data Out of Census Spreadsheets Using R" target="_blank">Pulling Data Out of Census Spreadsheets Using R</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/extracting-tables-from-pdfs-in-r-using-the-tabulizer-package/" title="Permanent Link toExtracting Tables from PDFs in R using the Tabulizer Package" target="_blank">Extracting Tables from PDFs in R using the Tabulizer Package</a></li>
</ol>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://datascienceplus.com/building-a-telecom-dictionary-scraping-web-using-rvest-in-r/"> R Programming – DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/jO_sWXQuMUY" height="1"></a></div>