<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/goodreads-webscraping-and-text-analysis-with-r-part-1/"></div></div><div><a data-url="https://www.r-bloggers.com/goodreads-webscraping-and-text-analysis-with-r-part-1/" data-counturl="https://www.r-bloggers.com/goodreads-webscraping-and-text-analysis-with-r-part-1/" data-count="horizontal" data-text="GoodReads: Webscraping and Text Analysis with R (Part 1)" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://datascienceplus.com/goodreads-webscraping-and-text-analysis-with-r-part-1/"> DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Inspired by <a target="_blank" rel="nofollow" href="http://varianceexplained.org/r/yelp-sentiment/">this article about sentiment analysis</a> and <a target="_blank" rel="nofollow" href="http://zevross.com/blog/2015/05/19/scrape-website-data-with-the-new-r-package-rvest/">this guide to webscraping</a>, I have decided to get my hands dirty by scraping and analyzing a sample of reviews on the website <a target="_blank" rel="nofollow" href="http://goodreads.com/">Goodreads</a>.</p>
<p>The goal of this project is to demonstrate a complete example, going from data collection to machine learning analysis, and to illustrate a few of the dead ends and mistakes I encountered on my journey. We’ll be looking at the reviews for five popular romance books. I have voluntarily chosen books in the same genre in order to make comments text more homogeneous <em>a priori</em>; these five books are popular enough that I can easily pull a few thousands reviews for each, yielding a significant corpus with minimum effort. If you don’t like romance books, feel free to replicate the analysis with your genre of choice!</p>
<p>To make the article more digestible, I have divided it in three segments:</p>
<blockquote><ul>
<li>Part 1: Webscraping</li>
<li>Part 2: Exploratory data analysis and sentiment analysis</li>
<li>Part 3: Predictive analytics with machine learning</li>
</ul>
</blockquote>
<p>This post includes the part 1, and the two following parts will be posted on DataScience+ in 1-week intervals.</p>
<h2>Part 1: Webscraping</h2>
<p>Goodreads’s reviews are a trove of text content begging to be scraped, with an interesting non-text variable attached, the ratings left by the reviewers. But there is a problem: navigation between pages of comments is done through a javascript button, not an html link. Fear not: this problem has actually a pretty simple solution, through the use of the RSelenium package (which has a nice vignette <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/RSelenium/vignettes/RSelenium-basics.html">here</a>). </p>
<h3>Setup</h3>
<p>Let’s load the libraries we’ll require during the analyses, and define some variables we’ll use later.</p>
<pre>library(data.table)   # Required for rbindlist
library(dplyr)        # Required to use the pipes %>% and some table manipulation commands
library(magrittr)     # Required to use the pipes %>%
library(rvest)        # Required for read_html
library(RSelenium)    # Required for webscraping with javascript

url &lt;- &quot;https://www.goodreads.com/book/show/18619684-the-time-traveler-s-wife#other_reviews&quot;
book.title &lt;- &quot;The time traveler's wife&quot;
output.filename &lt;- &quot;GR_TimeTravelersWife.csv&quot;</pre>
<p>Note that I’m working on a book-by-book basis. This means we have to manually change the variables above and re-run the script for each book. This could be automated to work on a grander scale, but that’s good enough for what I want to do here. Also, I’d rather not overload Goodreads’s servers by pulling massive amounts of data from them. </p>
<p>Let’s then start the RSelenium server. I have had some trouble with Firefox, and I have had to reinstall a previous version of the browser (which Firefox frowns upon), so your mileage may vary here. </p>
<pre>
startServer()
remDr &lt;- remoteDriver(browserName = &quot;firefox&quot;, port = 4444) # instantiate remote driver to connect to Selenium Server
remDr$open() # open web browser
remDr$navigate(url) </pre>
<p>These instructions start an instance of Firefox and navigate to the url as if you were directly interacting with the browser. </p>
<p>We then initialize the data frame that we’ll be populating with the data later.</p>
<pre>
global.df &lt;- data.frame(book=character(),
                        reviewer = character(),
                        rating = character(),
                        review = character(), 
                        stringsAsFactors = F)
</pre>
<p>We are now ready to proceed with the webscraping process itself!</p>
<h3>The webscraping process</h3>
<p>To extract the content we want, we’ll be looping through the 100 pages or so of comments for each of the books. Here I remove the loop to show the code going through one page only and explain its workings line by line. </p>
<p>First, we need to identify “where” the reviews appear in the page code. This is done by using <a target="_blank" rel="nofollow" href="http://selectorgadget.com/">SelectorGadget</a>, a Chrome extension that allows you to identify CSS selectors. Once we have identified the proper CSS selector (here <em>“#bookReviews .stacked”</em>), we just pass its name to the <em>findElements</em> function of the RSelenium server. </p>
<pre>
reviews &lt;- remDr$findElements(&quot;css selector&quot;, &quot;#bookReviews .stacked&quot;)
</pre>
<p>We extract the html code for the reviews, then the text component. </p>
<pre>
reviews.html &lt;- lapply(reviews, function(x){x$getElementAttribute(&quot;outerHTML&quot;)[[1]]})
reviews.list % html_text()})
reviews.text &lt;- unlist(reviews.list)
</pre>
<p>We now have the text of the reviews in list format, but a rapid inspection of it shows that there is still a lot of work to do to get a clean text. This we will do by using regular expressions (regex).</p>
<h3>Cleaning the reviews with Regex</h3>
<p>In my experience with text analytics, regex are both a blessing and a curse. A blessing because how else can you remove all non-letters characters in a string in one short command? And a curse because it’s a fairly esoteric language that is hard to understand or remember when you re-read your code later. So if you are not familiar with regex, I would definitely advise very generous commenting at the brief moment in time when you actually understand what your code does. </p>
<pre>
# Removing all characters that are not letters or dash
reviews.text2 &lt;- gsub(&quot;[^A-Za-z\\-]|\\.+&quot;, &quot; &quot;, reviews.text)
# Removing the end of line characters and extra spaces
reviews.clean &lt;- gsub(&quot;\n|[ \t]+&quot;, &quot; &quot;, reviews.text2)  
</pre>
<p>In order to write these commands, I have found these resources useful:</p>
<ul>
<li><a target="_blank" rel="nofollow" href="http://www.regular-expressions.info/">http://www.regular-expressions.info/</a></li>
<li><a target="_blank" rel="nofollow" href="http://stat545.com/block022_regular-expression.html">http://stat545.com/block022_regular-expression.html</a></li>
<li><a target="_blank" rel="nofollow" href="https://stat.ethz.ch/R-manual/R-devel/library/base/html/regex.html">https://stat.ethz.ch/R-manual/R-devel/library/base/html/regex.html</a></li>
</ul>
<h3>Putting the reviews in table format</h3>
<p>We now have our reviews in a reasonably clean state. But due to the underlying structure of the html code, we have a problem: for each review, we have the name of the reviewer and his/her rating in one string, and the review in the following string. In addition to that, the system to preview reviews means that the beginning of the review appears twice in the string. We’ll have to clean all that, again using regex, to get our data in table format.</p>
<p>We start by counting the number of reviews we have, i.e. half the number of strings and creating a temporary data frame that we’ll use to store the data before transferring it to the main data frame. </p>
<pre>
n &lt;- floor(length(reviews)/2)
reviews.df &lt;- data.frame(book = character(n),
                         reviewer = character(n),
                         rating = character(n),
                         review = character(n), 
                         stringsAsFactors = F)
</pre>
<p>We loop through our strings and populate our data frame, by extracting the fields we want review by review, based on recurring stop words. A <em>for</em> loop will do for this non-production example, but for production code you’d probably want to vectorialize everything you can. </p>
<p>The following code might appear a bit cryptic so first I’ll explain what I’m going to do:</p>
<ul>
<li>In the first part, I identify several expressions that can appear between the reviewer’s name and the rating, and use them in a regex to determine the ending point of the name in the string, then extract the name.</li>
<li>In the second part, I identify several expressions that can appear at the end of the rating, and use them in a regex to determine the ending point of the rating; sometimes none of these expressions appear, so I have a conditional telling R to go to the end of the string if it finds none of the expressions (by convention, it returns the position as being -1). Then I extract the rating.</li>
<li>In the third part, I remove the beginning of each review, which is repeated in the html file, by looking for the position in the string where the first 50 characters of the string appear again. I have a conditional in place to deal with cases when the review is short enough that its beginning is not repeated. I deal with the end of the review in the same way I did with the end of the rating</li>
<li>Finally, note the structure of the loop: I’m not looping through the strings one by one, but through the reviews, each review taking 2 consecutive strings, hence the <code>2*j</code> and <code>2*j-1</code> indices. </li>
</ul>
<pre>
for(j in 1:n){
  reviews.df$book[j] &lt;- book.title
    
  #Isolating the name of the author of the review
  auth.rat.sep &lt;- regexpr(&quot; rated it | marked it | added it &quot;, 
                          reviews.clean[2*j-1]) 
  reviews.df$reviewer[j] &lt;- substr(reviews.clean[2*j-1], 5, auth.rat.sep-1)
    
  #Isolating the rating
  rat.end &lt;- regexpr(&quot;· | Shelves| Recommend| review of another edition&quot;,
                     reviews.clean[2*j-1])
  if (rat.end==-1){rat.end &lt;- nchar(reviews.clean[2*j-1])}
  reviews.df$rating[j] &lt;- substr(reviews.clean[2*j-1], auth.rat.sep+10, rat.end-1)
    
  #Removing the beginning of each review that was repeated on the html file
  short.str &lt;- substr(reviews.clean[2*j], 1, 50)
  rev.start &lt;- unlist(gregexpr(short.str, reviews.clean[2*j]))[2]
  if (is.na(rev.start)){rev.start &lt;- 1}
  rev.end &lt;- regexpr(&quot;\\.+more|Blog&quot;, reviews.clean[2*j])
  if (rev.end==-1){rev.end &lt;- nchar(reviews.clean[2*j])}
  reviews.df$review[j] &lt;- substr(reviews.clean[2*j], rev.start, rev.end-1)
  }
</pre>
<p>Now that our temporary data frame has been populated, we can transfer its content to our main data frame.</p>
<pre>
global.lst &lt;- list(global.df, reviews.df)
global.df &lt;- rbindlist(global.lst)
</pre>
<p>And finally, we tell RSelenium to “click” on the <em>next page</em> button, by passing the proper CSS selector that we identified with SelectorGadget. Final trick: I found that in the initial iterations, RSelenium was too slow to load the pages, and was not responding in time to the instructions at the beginning of the next loop, so we tell R to wait 3 seconds at the end of each loop. </p>
<pre>
NextPageButton &lt;- remDr$findElement(&quot;css selector&quot;, &quot;.next_page&quot;)
NextPageButton$clickElement()
Sys.sleep(3)
</pre>
<p>After closing the overall loop, we can save the final data frame in a file.</p>
<pre>
write.csv(global.df,output.filename)
</pre>
<p>The result data frame looks like this:</p>
<table>
<tr>
<th>book</th>
<th>reviewer</th>
<th>rating</th>
<th>review</th>
</tr>
<tr>
<td>The time traveler’s wife</td>
<td>Liz S</td>
<td>it was ok</td>
<td>I recently read…</td>
</tr>
<tr>
<td>Eleanor &amp; Park</td>
<td>Danielle</td>
<td>did not like it</td>
<td>Why can’t there be…</td>
</tr>
<tr>
<td>Me Before You</td>
<td>Swaps</td>
<td>it was amazing</td>
<td>This review has been…</td>
</tr>
</table>
<p>You can find the full code, including the loops I have omitted here, on <a target="_blank" rel="nofollow" href="https://github.com/BuissonFlorent/GoodReads_TextMining">my github</a>.</p>
<ul></ul>
<p><strong>
<p>Related Post</p>
<p>
<ol>
<li><a title="Permanent Link toEuro 2016 analytics: Who’s playing the toughest game?" target="_blank" rel="bookmark" href="http://datascienceplus.com/euro-2016-analytics-whos-playing-the-toughest-game/">Euro 2016 analytics: Who’s playing the toughest game?</a></li>
<li><a title="Permanent Link toIntegrating R with Apache Hadoop" target="_blank" rel="bookmark" href="http://datascienceplus.com/integrating-r-with-apache-hadoop/">Integrating R with Apache Hadoop</a></li>
<li><a title="Permanent Link toPlotting App for ggplot2" target="_blank" rel="bookmark" href="http://datascienceplus.com/plotting-app-for-ggplot2/">Plotting App for ggplot2</a></li>
<li><a title="Permanent Link toPerforming SQL selects on R data frames" target="_blank" rel="bookmark" href="http://datascienceplus.com/performing-sql-selects-on-r-data-frames/">Performing SQL selects on R data frames</a></li>
<li><a title="Permanent Link toDeveloping an R Tutorial shiny dashboard app" target="_blank" rel="bookmark" href="http://datascienceplus.com/developing-a-r-tutorial-shiny-dashboard-app/">Developing an R Tutorial shiny dashboard app</a></li>
</ol></strong>
<div>
<a target="_blank" href="http://feeds.feedburner.com/~ff/datascienceplus?a=9eNsc5VA6Tk:NFb7m4Rh810:D7DqB2pKExk"><img src="http://feeds.feedburner.com/~ff/datascienceplus?i=9eNsc5VA6Tk:NFb7m4Rh810:D7DqB2pKExk" border="0"></a> <a target="_blank" href="http://feeds.feedburner.com/~ff/datascienceplus?a=9eNsc5VA6Tk:NFb7m4Rh810:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/datascienceplus?d=qj6IDK7rITs" border="0"></a> <a target="_blank" href="http://feeds.feedburner.com/~ff/datascienceplus?a=9eNsc5VA6Tk:NFb7m4Rh810:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/datascienceplus?d=yIl2AUoC8zA" border="0"></a>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/goodreads-webscraping-and-text-analysis-with-r-part-1/"></div></div><div><a data-url="https://www.r-bloggers.com/goodreads-webscraping-and-text-analysis-with-r-part-1/" data-counturl="https://www.r-bloggers.com/goodreads-webscraping-and-text-analysis-with-r-part-1/" data-count="vertical" data-text="GoodReads: Webscraping and Text Analysis with R (Part 1)" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://datascienceplus.com/goodreads-webscraping-and-text-analysis-with-r-part-1/"> DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/tlUc8V638Xs"></a></div>