<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/retailers-heres-what-your-employees-are-saying-about-you-a-project-on-web-scraping-indeed-com/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.nycdatascience.com/student-works/retailers-heres-employees-saying-project-web-scraping-indeed-com/"> R – NYC Data Science Academy Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><span><b>To view the original source code, you can visit my Github page </b><a rel="nofollow" href="https://github.com/jackyip1/Indeed-Reviews" target="_blank"><b>here</b></a><b>. If you are interested in learning more about my background, you can visit my LinkedIn page </b><a rel="nofollow" href="https://www.linkedin.com/in/jackyip/" target="_blank"><b>here</b></a><b>.</b></span></p>
<p><span><strong>About This Project</strong></span></p>
<p><span>This is the second of four projects for the NYC Data Science Academy. In this project, students are required to collect their datasets using a method called webscraping. Essentially, webscraping is the process of collecting information (i.e. texts) from a series of websites into structured databases such as .csv files. Webscraping enables analyses to be done on data that is not already provided in a structured format.</span></p>
<p><span>In this analysis, we will scrape employee reviews for the 8 retail companies that made it to <a rel="nofollow" href="https://www.indeed.com/Best-Places-to-Work" target="_blank">Indeed’s 50 Best Corporate Employers in the US</a> list. See below for additional details regarding each of these 8 retail firms in discussion. The rank and overall score corresponds to the employer’s rank and overall score as they appear on Indeed’s website.</span></p>
<p><span><a rel="nofollow" href="https://i0.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/05/indeed1.png" target="_blank"><img sizes="(max-width: 975px) 100vw, 975px" src="https://i0.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/05/indeed1.png?w=450" alt="" class="wp-image-22109" data-recalc-dims="1"></a></span></p>
<p><span><strong>Objectives</strong></span></p>
<p><span>Given a two-week timeframe, the scope of the analysis was limited to these three primary objectives:</span></p>
<ul>
<li><span><span>To understand </span><span>when</span><span> employees are more likely to submit positive and negative reviews</span></span></li>
<li><span><span>To understand the </span><span>sentiments</span><span> and </span><span>emotions</span><span> associated with positive and negative reviews (i.e. Sentiment Analysis)</span></span></li>
<li><span><span>To understand the </span><span>topics</span><span> of positive and negative reviews (i.e. Topic Modeling)</span></span></li>
</ul>
<p><span><strong>Web Scraping with Scrapy</strong></span></p>
<p><span><a rel="nofollow" href="https://scrapy.org/" target="_blank">Scrapy</a> (a Python framework) was used for collecting employee reviews for each of the retailers on the 50 Best Corporate Employers List. Each review scraped contains the following elements:</span></p>
<ul>
<li><span>Company</span></li>
<li><span>Industry</span></li>
<li><span>Job Title</span></li>
<li><span>Date</span></li>
<li><span>Month</span></li>
<li><span>Location</span></li>
<li><span>Rating</span></li>
<li><span>Header</span></li>
<li><span>Comment (Main Review)</span></li>
<li><span>Pro</span></li>
<li><span>Con</span></li>
</ul>
<p><span>As seen in the illustration below, not every review contains a pro and a con, as those are optional fields for the reviewer. To see an example of a real review, you can visit Indeed’s Walt Disney Parks and Resorts review page <a rel="nofollow" href="https://www.indeed.com/cmp/Walt-Disney-Parks-and-Resorts/reviews" target="_blank">here</a>.</span></p>
<p><span><a rel="nofollow" href="https://i0.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/05/indeed2.png" target="_blank"><img sizes="(max-width: 975px) 100vw, 975px" src="https://i0.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/05/indeed2.png?w=450" alt="" class="wp-image-22111" data-recalc-dims="1"></a></span></p>
<p><span><strong>Employee Reviews At a Glance</strong></span></p>
<p><span>In total, we have scraped over 20,000 reviews. The number of reviews scraped for each firm varies significantly across the board. This is likely due to the difference in the number of employees hired by each firm across the nation.</span></p>
<p><span><a rel="nofollow" href="https://i2.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/05/indeed3.1.png" target="_blank"><img sizes="(max-width: 500px) 100vw, 500px" src="https://i0.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/05/indeed3.1-918x1024.png?w=450" alt="" class="wp-image-22114" data-recalc-dims="1"></a></span></p>
<p><span>Employee reviews for these eight retail firms tend to peak around the beginning of the year (January through April). This is expected as we know that retail is a cyclical industry in which the majority of sales happen during the holiday season (November through January). Because employees are oftentimes hired only for the duration of this period, it makes sense  that the majority of the reviews come in around the beginning of the year.</span></p>
<p><span><a rel="nofollow" href="https://i2.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/05/indeed3.2.png" target="_blank"><img sizes="(max-width: 500px) 100vw, 500px" src="https://i1.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/05/indeed3.2-908x1024.png?w=450" alt="" class="wp-image-22113" data-recalc-dims="1"></a></span></p>
<p><span>Among the eight retailers on the 50 Best Corporate Employers list, we see that the review ratings skew toward a score of 5, which is the highest score a review can get.</span></p>
<p><span><a rel="nofollow" href="https://i1.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/05/indeed3.3.png" target="_blank"><img sizes="(max-width: 500px) 100vw, 500px" src="https://i2.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/05/indeed3.3-925x1024.png?w=450" alt="" class="wp-image-22112" data-recalc-dims="1"></a></span></p>
<p><span>There is no significant differences in average rating across retailers.</span></p>
<p><span><a rel="nofollow" href="https://i0.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/05/indeed3.4.png" target="_blank"><img sizes="(max-width: 500px) 100vw, 500px" src="https://i0.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/05/indeed3.4-930x1024.png?w=450" alt="" class="wp-image-22116" data-recalc-dims="1"></a></span></p>
<p><span>There is no significant differences in average rating across months.</span></p>
<p><span><a rel="nofollow" href="https://i0.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/05/indeed3.5.png" target="_blank"><img sizes="(max-width: 500px) 100vw, 500px" src="https://i0.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/05/indeed3.5-913x1024.png?w=450" alt="" class="wp-image-22115" data-recalc-dims="1"></a></span></p>
<p><span><span><strong>When Are Employees At Retail Firms </strong></span><span><strong>More Likely to Submit Positive and Negative Reviews?</strong></span></span></p>
<p><span><strong><span>Average Employee Rating by Month</span></strong></span></p>
<p><span>When we consider reviews of all ratings (i.e. reviews with ratings 1 to 5), retailers receive the highest average ratings during the months of <strong>October, November, December, and January.</strong> Note that the ratings have been scaled to 0 to 1 to allow for comparison across firms and months.</span></p>
<p><span><a rel="nofollow" href="https://i0.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed4.1.png?ssl=1" target="_blank"><img sizes="(max-width: 975px) 100vw, 975px" src="https://i0.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed4.1.png?w=450&ssl=1" alt="" class="wp-image-22159" data-recalc-dims="1"></a></span></p>
<p><span><strong><span>Most Negative Reviews By Month (Reviews with Ratings <span>Below</span> 3)</span></strong></span></p>
<p><span>If we only consider the <span>count of negative reviews</span> (i.e. reviews with ratings 1 or 2), retailers on average receive the most negative reviews during the months of <strong>October, February, March, and April</strong> (i.e. these months scored higher across retailers)<strong>.</strong></span></p>
<p><span><a rel="nofollow" href="https://i2.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed4.2.png?ssl=1" target="_blank"><img sizes="(max-width: 975px) 100vw, 975px" src="https://i2.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed4.2.png?w=450&ssl=1" alt="" class="wp-image-22160" data-recalc-dims="1"></a></span></p>
<p><span><strong><span>Most Positive Reviews By Month (Reviews with Ratings <span>Above</span> 3)</span></strong></span></p>
<p><span>If we only consider the <span>count of positive reviews</span> (i.e. reviews with ratings 4 or 5), retailers on average receive the most positive reviews during the months of <strong>January, February, and April </strong>(i.e. these months scored higher across retailers).</span></p>
<p><span><a rel="nofollow" href="https://i1.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed4.3.png?ssl=1" target="_blank"><img sizes="(max-width: 975px) 100vw, 975px" src="https://i1.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed4.3.png?w=450&ssl=1" alt="" class="wp-image-22161" data-recalc-dims="1"></a></span></p>
<p><span><strong><span>Summary Observations</span></strong></span></p>
<p><span>In summary, while the highest average ratings concentrate toward the end of the year, the highest count of both positive and negative reviews are given during the beginning of the year. This aligns with our understanding that the retail industry is cyclical. In other words, employees are oftentimes hired specifically to aid sales around the holiday season (i.e. Nov to Jan). When their work concludes around the beginning of the year, that’s when we can expect employees to write the majority of the reviews (i.e. both positive and negative).</span></p>
<p><span><a rel="nofollow" href="https://i1.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed4.4.png?ssl=1" target="_blank"><img sizes="(max-width: 975px) 100vw, 975px" src="https://i1.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed4.4.png?w=450&ssl=1" alt="" class="wp-image-22162" data-recalc-dims="1"></a></span></p>
<p><span><strong>Understanding Employees Reviews Using Sentiment Analysis</strong></span></p>
<p><span>A sentiment analysis was done using <a rel="nofollow" href="https://github.com/mjockers/syuzhet" target="_blank">R’s Syuzhet</a> library to visualize the emotions across employee reviews. In the illustration below, each graph represents the intensity of the an emotion from January to December on a positive scale (i.e. 0 and above). For example, for Build-A-Bear Workshop’s ‘positive’ graph, we can see that there is an uptick in positive sentiment toward March and November, and each of those months scored approximately a 6. This is a high score relative to the scores recorded across all other emotions.</span></p>
<p><span>Generally, the sentiment observed align with our understanding that the majority of the employee reviews are positive, as we are narrowly focused on analyzing employee reviews among the best 8 corporate retail employers according to Indeed’s ranking. As an aside, it is interesting to see that Build-A-Bear Workshop and Trader Joe’s recorded more volatility in its scoring across the negative emotions (i.e. negative, sadness, fear, disgust, anger).</span></p>
<p><span><a rel="nofollow" href="https://i2.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed5.1.png?ssl=1" target="_blank"><img sizes="(max-width: 975px) 100vw, 975px" src="https://i2.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed5.1.png?w=450&ssl=1" alt="" class="wp-image-22165" data-recalc-dims="1"></a></span></p>
<p><span><strong>Using Topic Modeling to Identify Patterns Among Positive and Negative Employee Reviews</strong></span></p>
<p><span>Topic modeling is a form of unsupervised machine learning, and it can help us identify patterns by clustering similar items into groups. In this analysis, <a rel="nofollow" href="https://github.com/bmabey/pyLDAvis" target="_blank">pyLDAvis</a>, a Python library, was used to analyze the groups among positive and negative reviews (using pros and cons of the employee reviews). <span>Essentially, the library takes a list of documents as inputs (i.e. a list of reviews) and attempts to group the reviews based on common combinations of keywords appearing in each of these reviews.</span></span></p>
<p><span>The drawback of using topic modeling, as with other clustering techniques, is that the user has to specify the number of groups to cluster the documents into. This presents a catch-22 issue as the initial objective is to understand the number of topics or groups among your inputs. Given this hurdle, the user must use their business judgment as well as running multiple trials with different number of topics as inputs to determine results that are most interpretable. More sophisticated clustering libraries aid the users by automatically selecting the best number of topics, but this should only be used as a guide to begin the analysis.</span></p>
<p><span>In the illustration below, we can see that words like ‘great’ and ‘benefit,’ among many other words, are oftentimes seen together among pro reviews.</span></p>
<p><span><a rel="nofollow" href="https://i2.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed5.4.png?ssl=1" target="_blank"><img sizes="(max-width: 975px) 100vw, 975px" src="https://i2.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed5.4.png?w=450&ssl=1" alt="" class="wp-image-22166" data-recalc-dims="1"></a></span></p>
<p><span><strong><span>Observed Topics Among Pros and Cons in Employee Reviews</span></strong></span></p>
<p><span>Two topic modeling analyses were conducted using pro reviews and con reviews. Below are the most common combinations of words appearing in each of the groups for each analysis. It is not a surprise that the groups overlap in the common words identified among them, as we have chosen a relatively high number of topics. Nevertheless, the results provide a useful start for understanding what employees are thinking.</span></p>
<p><span><a rel="nofollow" href="https://i1.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed5.5.png?ssl=1" target="_blank"><img sizes="(max-width: 975px) 100vw, 975px" src="https://i1.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed5.5.png?w=450&ssl=1" alt="" class="wp-image-22167" data-recalc-dims="1"></a></span></p>
<p><span><strong>Visualizing Employee Reviews Using Word Clouds</strong></span></p>
<p><span><a rel="nofollow" href="https://cran.r-project.org/web/packages/wordcloud2/vignettes/wordcloud.html" target="_blank">R’s Wordcloud2</a> library was used to further aid the visualization of what employees are saying about these firms. Essentially, the library takes a list of words along with their frequencies, and produces a word cloud based on those inputs. The higher the frequency a word is associated with, the larger it appears in the word cloud.</span></p>
<p><span><strong><span>Word Cloud – Positive Reviews</span></strong></span></p>
<p><span>The positive reviews word cloud was created using only the pros from each of the reviews. (Note: Each review contains the main review, a pro, and a con.) Some of the common words we see among pros are ‘great,”work,’ ‘job,’’company,’ and ‘customers.’</span></p>
<p><span><a rel="nofollow" href="https://i1.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed5.2.png?ssl=1" target="_blank"><img sizes="(max-width: 975px) 100vw, 975px" src="https://i1.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed5.2.png?w=450&ssl=1" alt="" class="wp-image-22164" data-recalc-dims="1"></a></span></p>
<p><span><strong><span>Word Cloud – Negative Reviews</span></strong></span></p>
<p><span>Similarly, a word cloud was created using only the cons from each of the reviews. We can see that common words appearing include <em>work, management, hours, employees, </em>and<em> pay</em>.</span></p>
<p><span><a rel="nofollow" href="https://i2.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed5.3.png?ssl=1" target="_blank"><img sizes="(max-width: 975px) 100vw, 975px" src="https://i2.wp.com/blog.nycdatascience.com/wp-content/uploads/2017/06/indeed5.3.png?w=450&ssl=1" alt="" class="wp-image-22163" data-recalc-dims="1"></a></span></p>
<p><span><strong>Future Directions</strong></span></p>
<p><span>Beyond the scope of this project, below are other interesting areas that are worth looking into if additional time allows.</span></p>
<ul>
<li><span>Evaluate the credibility of Indeed’s Top 50 Ranking corporate employers by sampling comparable employers that did not make it to the list</span></li>
<li><span>Better understand the rationale why positive and negative reviews are concentrated in the months they were observed to be in, and evaluate whether this behavior is prevalent across other industries</span></li>
<li><span>Perform statistical methods to evaluate the relationships among variables</span></li>
</ul>
<p>The post <a rel="nofollow" href="http://blog.nycdatascience.com/student-works/retailers-heres-employees-saying-project-web-scraping-indeed-com/" target="_blank">Retailers: Here’s What Your Employees Are Saying About You – A Project on Web Scraping Indeed.com</a> appeared first on <a rel="nofollow" href="http://blog.nycdatascience.com/" target="_blank">NYC Data Science Academy Blog</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.nycdatascience.com/student-works/retailers-heres-employees-saying-project-web-scraping-indeed-com/"> R – NYC Data Science Academy Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/8VSMbeP-OVs" height="1"></a></div>