<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/data-scientist-with-a-wine-hobby-part-i/"></div></div><div><a data-url="https://www.r-bloggers.com/data-scientist-with-a-wine-hobby-part-i/" data-counturl="https://www.r-bloggers.com/data-scientist-with-a-wine-hobby-part-i/" data-count="horizontal" data-text="Data Scientist with a wine hobby (Part I)" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://eighty20.github.io//package%20exploration/2016/09/11/wine-review/"> Pareto's Playground</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>After high school I made my way from Johannesburg, situated in the northern part of South Africa, to the famous wine country known as Stellenbosch in the south. Here for the first time I got a ton of exposure to wine and the countless varietals that make up this “drink of the gods”.</p>
<p>The one trick in wine tasting and exploring vini- and viticulture is the fact that the best way to learn about it all is going out to the farm and drinking the wine for yourself. Over the years I have become familiar with all the farms in the region and it can sometimes be a real prize when discovering a cellar which you have not visited.</p>
<p>As one can see from the map, there are quite a few farms that are ‘officially’ listed, but believe me there are a few hidden ones which one will never know of without some research.</p>
<p><img title="center" alt="center" src="https://i2.wp.com/eighty20.github.io/figures/Wine_review_fig/map_stellenbosch.jpg?w=400" data-recalc-dims="1"></p>
<p>To ease this process I have used <code>R</code> in order to compile a sort of wine farm repository for myself as a go to guide whenever I visit the region. I did this as an exercise to explore Hadley Wickham’s amazing <code>rvest</code> library which makes online data collection an ease.</p>
<p>The idea was to compile <em>wine review data</em> of the Stellenbosch wine region from <a target="_blank" href="http://www.winemag.com/">winemag.com</a>’s online database and use it to gain an analytical insight into this beautiful wine region in the Cape.</p>
<p>The first step in collecting the data was to deal with some housekeeping issues. First I assign the base URL to start the collecting from and collect the number of pages from the Stellenbosch search result as part of the <code>pagination</code> block in the html. Secondly I collect the information around the wine such as name, points and price. Lastly I use the information on this first page to create a function, <code>info_xtr()</code>, to extract information of each of the wines from their own specific page. Specifically I extract the written review of the wine, the date of review and also the alcohol content of the wine.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>rvest</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>site</span><span> </span><span>=</span><span> </span><span>read_html</span><span>(</span><span>&quot;http://www.winemag.com/?s=stellenbosch&amp;drink_type=wine&amp;page=1&quot;</span><span>)</span><span>
</span><span>nr_pages</span><span> </span><span>&lt;-</span><span> </span><span>html_nodes</span><span>(</span><span>site</span><span>,</span><span>&quot;.pagination li a&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>html_text</span><span>()</span><span> </span><span>%>%</span><span> </span><span>tail</span><span>(</span><span>.</span><span>,</span><span>1</span><span>)</span><span> </span><span>%>%</span><span> </span><span>as.numeric</span><span>
</span><span>wine_href</span><span> </span><span>&lt;-</span><span> </span><span>html_nodes</span><span>(</span><span>site</span><span>,</span><span>&quot;.review-listing&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>html_attr</span><span>(</span><span>&quot;href&quot;</span><span>)</span><span>
</span><span>info_xtr</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>x</span><span>){</span><span>
</span><span>page</span><span> </span><span>&lt;-</span><span> </span><span>read_html</span><span>(</span><span>x</span><span>)</span><span>
</span><span>review</span><span> </span><span>&lt;-</span><span> </span><span>page</span><span> </span><span>%>%</span><span> </span><span>html_nodes</span><span>(</span><span>&quot;p.description&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>html_text</span><span>
</span><span>review_info</span><span> </span><span>&lt;-</span><span> </span><span>page</span><span> </span><span>%>%</span><span> </span><span>html_nodes</span><span>(</span><span>&quot;.info.small-9.columns span span&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>html_text</span><span>()</span><span>
</span><span>cbind</span><span>(</span><span>review</span><span>,</span><span> </span><span>date</span><span> </span><span>=</span><span> </span><span>review_info</span><span>[</span><span>5</span><span>],</span><span> </span><span>Alc</span><span> </span><span>=</span><span> </span><span>review_info</span><span>[</span><span>1</span><span>])</span><span>
</span><span>}</span></code></pre>
</figure>
<p>The <code>info_xtr()</code> function returns a data.frame that contains the review, date of the specific review and the alcohol content:</p>
<figure>
<pre><code data-lang="r"><span>info_xtr</span><span>(</span><span>wine_href</span><span>[</span><span>1</span><span>])</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##      review                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
## [1,] &quot;A blend of 28% Cabernet Franc, 23% Cabernet Sauvignon, 21% Malbec, 18% Petit Verdot and 10% Merlot, this is a big, bold and powerful wine, with superb balance and a firm, concentrated structure that promises great long-term aging. Dense aromas and flavors of blackberry, boysenberry, cherry and fig are layered with notes of cocoa powder, roasted espresso bean and cigar tobacco. Ripe, rich and lush, but not overdone, hints of charred oak and black pepper linger on the finish. Drink 2018–2025.&quot;
##      date        Alc  
## [1,] &quot;12/1/2015&quot; &quot;14%&quot;</code></pre>
</figure>
<p>The wine selection data from the page is collated in a nice <code>data.frame()</code> for ease of reading. With all the information I need, I start my scraper to run through the website and collect all the information surrounding the wines</p>
<figure>
<pre><code data-lang="r"><span># Start the scraper ==============
</span><span>Wine_cellar</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>()</span><span>
</span><span>for</span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>nr_pages</span><span>)</span><span>
</span><span>{</span><span>
</span><span>cat</span><span>(</span><span>&quot;Wine review, page: &quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span>&quot;\n&quot;</span><span>)</span><span>
</span><span>site</span><span> </span><span>=</span><span> </span><span>read_html</span><span>(</span><span>paste0</span><span>(</span><span>&quot;http://www.winemag.com/?s=stellenbosch&amp;drink_type=wine&amp;page=&quot;</span><span>,</span><span>i</span><span>))</span><span>
</span><span>wine</span><span> </span><span>&lt;-</span><span> </span><span>html_nodes</span><span>(</span><span>site</span><span>,</span><span>&quot;.review-listing .title&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>html_text</span><span>()</span><span>
</span><span># extract specific information from title
</span><span> </span><span>wine_farms</span><span> </span><span>&lt;-</span><span> </span><span>gsub</span><span>(</span><span>&quot; \\d.*$&quot;</span><span>,</span><span>&quot;&quot;</span><span>,</span><span>wine</span><span>)</span><span>
</span><span>wine_year</span><span> </span><span>&lt;-</span><span> </span><span>gsub</span><span>(</span><span>&quot;[^0-9]&quot;</span><span>,</span><span>&quot;&quot;</span><span>,</span><span>wine</span><span>)</span><span>
</span><span>wine_name</span><span> </span><span>&lt;-</span><span> </span><span>gsub</span><span>(</span><span>&quot;^.*\\d &quot;</span><span>,</span><span>&quot;&quot;</span><span>,</span><span>wine</span><span>)</span><span> </span><span>%>%</span><span> </span><span>gsub</span><span>(</span><span>&quot; \\(Stellenbosch\\)&quot;</span><span>,</span><span>&quot;&quot;</span><span>,</span><span>.</span><span>)</span><span>
</span><span># extract review points
</span><span> </span><span>points</span><span> </span><span>&lt;-</span><span> </span><span>html_nodes</span><span>(</span><span>site</span><span>,</span><span>&quot;.info .rating&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>html_text</span><span>()</span><span> </span><span>%>%</span><span> </span><span>gsub</span><span>(</span><span>&quot; Points&quot;</span><span>,</span><span>&quot;&quot;</span><span>,</span><span>.</span><span>)</span><span>
</span><span># extract review price
</span><span> </span><span>price</span><span> </span><span>&lt;-</span><span> </span><span>html_nodes</span><span>(</span><span>site</span><span>,</span><span>&quot;span.price&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>html_text</span><span>()</span><span>
</span><span>wine_href</span><span> </span><span>&lt;-</span><span> </span><span>html_nodes</span><span>(</span><span>site</span><span>,</span><span>&quot;.review-listing&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>html_attr</span><span>(</span><span>&quot;href&quot;</span><span>)</span><span>
</span><span># here I go collect all the information from each of the wines seperately
</span><span> </span><span>reviews</span><span> </span><span>&lt;-</span><span> </span><span>sapply</span><span>(</span><span>wine_href</span><span>,</span><span> </span><span>function</span><span>(</span><span>x</span><span>)</span><span> </span><span>info_xtr</span><span>(</span><span>x</span><span>))</span><span>
</span><span>reviews</span><span> </span><span>&lt;-</span><span> </span><span>t</span><span>(</span><span>reviews</span><span>)</span><span> </span><span>row.names</span><span>(</span><span>reviews</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>NULL</span><span>
</span><span>colnames</span><span>(</span><span>reviews</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;review&quot;</span><span>,</span><span> </span><span>&quot;review_date&quot;</span><span>,</span><span> </span><span>&quot;Alc&quot;</span><span>)</span><span>
</span><span># bind all the information into a nicely formatted data.frame()
</span><span> </span><span>Wine_cellar</span><span>[[</span><span>i</span><span>]]</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>wine_farms</span><span>,</span><span> </span><span>wine_year</span><span>,</span><span> </span><span>wine_name</span><span>,</span><span> </span><span>points</span><span>,</span><span> </span><span>price</span><span>,</span><span> </span><span>wine_href</span><span>,</span><span> </span><span>reviews</span><span>,</span><span> </span><span>stringsAsFactors</span><span> </span><span>=</span><span> </span><span>F</span><span>)</span><span>
</span><span># as a rule I always save the already collected data, just in case the scraper stops unexpectedly
</span><span> </span><span>saveRDS</span><span>(</span><span>Wine_cellar</span><span>,</span><span>&quot;Wine_collection.RDS&quot;</span><span>)</span><span>
</span><span># I add in a sleep as not the flood the website that I am scraping with requests
</span><span> </span><span>Sys.sleep</span><span>(</span><span>runif</span><span>(</span><span>1</span><span>,</span><span>0</span><span>,</span><span>3</span><span>))</span><span>
</span><span>}</span></code></pre>
</figure>
<p>With my scraper having completed its task in retrieving all the necessary information from the website, I bind the information from the <code>Wine_cellar</code> list object. For brevity I truncate my sample by removing the <em>href</em> and <em>review</em> columns which takes up a lot of space</p>
<figure>
<pre><code data-lang="r"><span>Wine_all</span><span> </span><span>&lt;-</span><span> </span><span>bind_rows</span><span>(</span><span>Wine_cellar</span><span>)</span><span>
</span><span>Wine_all</span><span> </span><span>%>%</span><span> </span><span>select</span><span>(</span><span>-</span><span>c</span><span>(</span><span>wine_href</span><span>,</span><span> </span><span>review</span><span>))</span><span> </span><span>%>%</span><span> </span><span>str</span><span>()</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## 'data.frame':	1040 obs. of  7 variables:
##  $ wine_farms : chr  &quot;Mvemve Raats&quot; &quot;Mvemve Raats&quot; &quot;De Toren&quot; &quot;De Toren&quot; ...
##  $ wine_year  : chr  &quot;2012&quot; &quot;2011&quot; &quot;2012&quot; &quot;2012&quot; ...
##  $ wine_name  : chr  &quot;MR de Compostella Red&quot; &quot;MR de Compostella Red&quot; &quot;Fusion V Red&quot; &quot;Z Red&quot; ...
##  $ points     : chr  &quot;94&quot; &quot;94&quot; &quot;94&quot; &quot;92&quot; ...
##  $ price      : chr  &quot;$65&quot; &quot;$65&quot; &quot;$60&quot; &quot;$50&quot; ...
##  $ review_date: chr  &quot;12/1/2015&quot; &quot;12/1/2015&quot; &quot;12/1/2015&quot; &quot;12/1/2015&quot; ...
##  $ Alc        : chr  &quot;14%&quot; &quot;14%&quot; &quot;14%&quot; &quot;14%&quot; ...</code></pre>
</figure>
<p>As you can see from the information displayed, we now have a complete data-set of all reviews of the the wines in the Stellenbosch region. This information can now be used to analyse the traits of the wine region in a number of ways. I will explore this data in the next post and will see what interesting insights we can draw from it.</p>
<p>I am familiar with collecting online data using the <code>RCurl</code> package and have been wanting to see how I can integrate the <code>rvest</code> package into some of the work I do. I must admit, I love exploring online data with this package as the <code>html_node</code> function makes the collection a breeze. I would definitely recommend <code>rvest</code> as the go-to library for any online scraping needs.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/data-scientist-with-a-wine-hobby-part-i/"></div></div><div><a data-url="https://www.r-bloggers.com/data-scientist-with-a-wine-hobby-part-i/" data-counturl="https://www.r-bloggers.com/data-scientist-with-a-wine-hobby-part-i/" data-count="vertical" data-text="Data Scientist with a wine hobby (Part I)" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://eighty20.github.io//package%20exploration/2016/09/11/wine-review/"> Pareto's Playground</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/YCGW5rB1hsk"></a></div>