<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/web-scraping-matrimonial-website-with-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.listendata.com/2018/02/web-scraping-website-with-r.html"> ListenData</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div dir="ltr">
<div dir="ltr">
<div>In this tutorial, we will cover how to extract information from a matrimonial website using R.  We will do web scraping which is a process of converting data available in unstructured format on the website to structured format which can be further used for analysis.</div>
<p>We will use a R package called <b>rvest</b> which was created by Hadley Wickham. This package simplifies the process of scraping web pages.</p>
<table cellspacing="0" align="center" cellpadding="0">
<tbody>
<tr>
<td><a rel="nofollow" href="https://i1.wp.com/2.bp.blogspot.com/-Y8lXmYAdLHA/WpPL3M78S7I/AAAAAAAAGv4/DqLn0jUTTBMyLVThJTvCeJh9VznWGajtACLcBGAs/s1600/web%2Bscraping.PNG?ssl=1" target="_blank"><img border="0" src="https://i1.wp.com/2.bp.blogspot.com/-Y8lXmYAdLHA/WpPL3M78S7I/AAAAAAAAGv4/DqLn0jUTTBMyLVThJTvCeJh9VznWGajtACLcBGAs/s1600/web%2Bscraping.PNG?resize=450%2C224&ssl=1" data-original-width="450" alt="Web Scraping in R" data-original-height="224" title="" data-recalc-dims="1"></a></td>
</tr>
<tr>
<td>Web Scraping in R</td>
</tr>
</tbody>
</table>
<p></p>
<h2>Install the required packages</h2>
<div>To download and install the rvest package, run the following command. We will also use dplyr which is useful for data manipulation tasks.</div>
<blockquote><p>install.packages(“rvest”)<br>install.packages(“dplyr”)</p></blockquote>
<h2>Load the required Libraries</h2>
<div>To make the libraries in use, you need to submit the program below.</div>
<blockquote><p>library(rvest)<br>library(dplyr)</p></blockquote>
<h2>Scrape Information from Matrimonial Website</h2>
<p>First we need to understand the structure of URL. See the URLs below.<br><a rel="nofollow" href="https://www.jeevansathi.com/punjabi-brides-girls" target="_blank">https://www.jeevansathi.com/punjabi-brides-girls</a><br><a rel="nofollow" href="https://www.jeevansathi.com/punjabi-grooms-boys" target="_blank">https://www.jeevansathi.com/punjabi-grooms-boys</a></p>
<div></div>
<div>The first URL takes you to the webpage wherein girls’ profiles of Punjabi community are shown whereas second URL provides details about boys’ profiles’ of Punjabi community.</div>
<div></div>
<div>We need to split the main URL into different elements so that we can access it. </div>
<blockquote><p>Main_URL = Static_URL + Mother_Tongue + Brides_Grooms</p></blockquote>
<p>Check out the following R code how to prepare the main URL. In the code, you need to provide the following details –</p>
<ol>
<li>Whether you are looking for girls’/boys’ profiles. Type <b>bride </b>to see girls’ profiles. Enter <b>groom </b>to check out boys’ profiles.</li>
<li>Select Mother Tongue. For example, punjabi, tamil etc.</li>
</ol>
<pre># Looking for bride/groom<br>Bride_Groom = &quot;bride&quot;<br># Possible Values : bride, groom<br><br># Select Mother Tongue<br>Mother_Tongue = &quot;punjabi&quot;<br># Possible Values<br># punjabi<br># tamil<br># bengali<br># telugu<br># kannada<br># marathi<br><br># URL<br>if (tolower(Bride_Groom) == &quot;bride&quot;) {<br>html = paste0('https://www.jeevansathi.com/',tolower(Mother_Tongue),'-brides-girls')<br>} else {<br>html = paste0('https://www.jeevansathi.com/',tolower(Mother_Tongue),'-grooms-boys')<br>}<br></pre>
<p><span><b>See the output : </b></span></p>
<pre>[1] &quot;https://www.jeevansathi.com/punjabi-brides-girls&quot;<br></pre>
<p></p>
<h3>Extract Profile IDs</h3>
<p>First you need to select parts of an html document using css selectors: <b>html_nodes()</b>. Use <b><a rel="nofollow" href="http://selectorgadget.com/" target="_blank">SelectorGadget</a> </b>which<b> </b>is a chrome extension available for free. It is the easiest and quickest way to find out which selector pulls the data that you are interested in.<br><b><br></b><b>How to use SelectorGadget : </b>Click on a page element that you would like your selector to match (it will turn green). It will then generate a minimal CSS selector for that element, and will highlight (yellow) everything that is matched by the selector.</p>
<blockquote><p>text = read_html(html) %>% html_nodes(“.profileContent .color11 a”) %>% html_text()<br>profileIDs = data.frame(ID = text)</p></blockquote>
<pre>         ID<br>1  ZARX0345<br>2  ZZWX5573<br>3  ZWVT2173<br>4  ZAYZ6100<br>5  ZYTS6885<br>6  ZXYV9849<br>7   TRZ8475<br>8   VSA7284<br>9  ZXTU1965<br>10 ZZSA6877<br>11 ZZSZ6545<br>12 ZYSW4809<br>13 ZARW2199<br>14 ZRSY0723<br>15 ZXAT2801<br>16 ZYXX8818<br>17 ZAWA8567<br>18  WXZ2147<br>19 ZVRT8875<br>20 ZWWR9533<br>21 ZYXW4043<br></pre>
<p>The basic functions in rvest are very user-friendly and robust. Explanation of these functions are listed below –</p>
<ol>
<li><b>read_html() :</b>  you can create a html document from a URL</li>
<li><b>html_nodes() :</b> extracts pieces out of HTML documents.</li>
<li><b>html_nodes(“.class”) :</b> calls node based on CSS class</li>
<li><b>html_nodes(“#class”) :</b> calls node based on <div>, <span>, <pre> id
<li><b>html_text() </b>: extracts only the text from HTML tag</li>
<li><b>html_attr()</b> : extracts contents of a single attribute</li>

<p></p>
<div><b><span>Difference between .class and #class</span></b>
<p><b>1.</b> <b>.class</b> targets the following element:</p>
<blockquote><p><span><div></div></span></p></blockquote>
<p><b>2.</b> <b>#class</b> targets the following element:</p>
<blockquote><p><span><div></div></span></p></blockquote>
</div>
<h3><span>Get Detailed Information of Profiles</span></h3>
<div>The following program performs the following tasks –
<ol>
<li>Loop through profile IDs</li>
<li>Pull information about Age, Height, Qualification etc.</li>
<li>Extract details about appearance</li>
<li>Fetch ‘About Me’ section of profiles</li>
</ol>
</div>
<pre># Get Detailed Information<br>finaldf = data.frame()<br>for (i in 1:length(profileIDs$ID)){<br>ID = profileIDs[i,1]<br>link = paste0(&quot;https://www.jeevansathi.com/profile/viewprofile.php?stype=4&amp;username=&quot;, ID)<br>FormattedInfo = data.frame(t(read_html(link) %>% html_nodes(&quot;.textTru li&quot;) %>%<br>                               html_text()))<br># Final Table<br>FormattedInfo = data.frame(ProfileID = ID,<br>                             Description = read_html(link) %>% <br>                             html_nodes(&quot;#myinfoView&quot;) %>%<br>                             html_text(), <br>                             Appearance = read_html(link) %>% <br>                             html_nodes(&quot;#section-lifestyle #appearanceView&quot;) %>%<br>                             html_text(),<br>                             FormattedInfo)<br><br>finaldf = bind_rows(finaldf, FormattedInfo)<br>}<br><br># Assign Variable Names<br>names(finaldf) = c(&quot;ProfileID&quot;, &quot;Description&quot;, &quot;Appearance&quot;, &quot;Age_Height&quot;, &quot;Qualification&quot;, &quot;Location&quot;, &quot;Profession&quot;, &quot;Mother Tongue&quot;, &quot;Salary&quot;, &quot;Religion&quot;, &quot;Status&quot;, &quot;Has_Children&quot;)<br></pre>
<div><a rel="nofollow" href="https://i2.wp.com/1.bp.blogspot.com/-ubL9VeMn4_U/WpP1ElPbXMI/AAAAAAAAGwY/7zE2fP2HhqARodTAFKbdl8K3CF4ZoCc4ACLcBGAs/s1600/details1.PNG?ssl=1" target="_blank"><img border="0" src="https://i0.wp.com/1.bp.blogspot.com/-ubL9VeMn4_U/WpP1ElPbXMI/AAAAAAAAGwY/7zE2fP2HhqARodTAFKbdl8K3CF4ZoCc4ACLcBGAs/s640/details1.PNG?resize=450%2C223&ssl=1" data-original-width="450" alt="Web Scraping Output" width="450" data-original-height="223" title="" data-recalc-dims="1"></a></div>
<div></div>
<div><a rel="nofollow" href="https://i2.wp.com/4.bp.blogspot.com/--J0Kmx-00nE/WpP1S72ctTI/AAAAAAAAGwc/cEXYzGfNaFYQ11Tx2j3KBG7PcG4TfWV9QCLcBGAs/s1600/details2.PNG?ssl=1" target="_blank"><img border="0" src="https://i2.wp.com/4.bp.blogspot.com/--J0Kmx-00nE/WpP1S72ctTI/AAAAAAAAGwc/cEXYzGfNaFYQ11Tx2j3KBG7PcG4TfWV9QCLcBGAs/s1600/details2.PNG?resize=450%2C174&ssl=1" data-original-width="450" alt="Web Scraping Output PartII" data-original-height="174" title="" data-recalc-dims="1"></a></div>
<h3>Download Display Pic</h3>
<p>To download display pic, you first need to fetch image URL of profile and then hit <b>download.file( )</b> function to download it. In the script below, you need to provide a profile ID.</p>
<blockquote><p># Download Profile Pic of a particular Profile<br>ID = “SWU6683”<br>text3 = read_html(html) %>% html_nodes(“.vtop”) %>% html_attr(‘src’)<br>pic = data.frame(cbind(profileIDs, URL = text3[!is.na(text3)]))<br>download.file(as.character(pic$URL[match(ID, pic$ID)]), “match.jpg”, mode = “wb”)<br># File saved as match.jpg</p></blockquote>
<p><b><span>Disclaimer</span></b></p>
<blockquote><p>We have accessed only publicly available data which does not require login or registration. The purpose is  not to cause any damage or copy the content from the website.</p></blockquote>
<p><b><span>Other Functions of rvest( )</span></b><br>You can extract, modify and submit forms with html_form(), set_values() and submit_form(). Refer the case study below –</p>
<p>You can collect google search result by submitting the google search form with search term. You need to supply search term. Here, I entered ‘<b>Datascience</b>‘ search term.</p>
<blockquote><p>library(rvest)<br>url       = “http://www.google.com”<br>pgsession = html_session(url)            <br>pgform    = html_form(pgsession)[[1]]</p>
<p># Set search term<br>filled_form = set_values(pgform, q=”Datascience”)<br>session = submit_form(pgsession,filled_form)</p>
<p># look for headings of first page<br>session %>% html_nodes(“.g .r a”) %>% html_text()</p></blockquote>

<pre> [1] &quot;Data science - Wikipedia&quot;                                          <br> [2] &quot;Data Science Courses | Coursera&quot;                                   <br> [3] &quot;Data Science | edX&quot;                                                <br> [4] &quot;Data science - Wikipedia&quot;                                          <br> [5] &quot;DataScience.com | Enterprise Data Science Platform Provider&quot;       <br> [6] &quot;Top Data Science Courses Online - Updated February 2018 - Udemy&quot;   <br> [7] &quot;Data Science vs. Big Data vs. Data Analytics - Simplilearn&quot;        <br> [8] &quot;What Is Data Science? What is a Data Scientist? What is Analytics?&quot;<br> [9] &quot;Online Data Science Courses | Microsoft Professional Program&quot;      <br>[10] &quot;News for Datascience&quot;                                              <br>[11] &quot;Data Science Course - Cognitive Class&quot;    <br></pre>


				
		
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.listendata.com/2018/02/web-scraping-website-with-r.html"> ListenData</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/iBIRvdFNNrc" height="1"></a></div></pre></span></div></ol></div></div>