<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/automated-text-feature-engineering-using-textfeatures-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://datascienceplus.com/automated-text-feature-engineering-using-textfeatures-in-r/"> R Programming – DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>It could be the era of Deep Learning where it really doesn’t matter how big is your dataset or how many columns you’ve got. Still, a lot of Kaggle Competition Winners and Data Scientists emphasis on one thing that could put you on the top of the leaderboard in a Competition is “Feature Engineering”. Irrespective of how sophisticated your model is, good features will always help your Machine Learning Model building process better than others. </p>
<h3>What is Feature engineering?</h3>
<p>Features are nothing but columns/dimensions and Feature Engineering is the process of creating new Features or Predictors based on Domain Knowledge or Statistical Principles. Feature Engineering has always been around with Machine Learning but the latest in that is Automated Feature Engineering which has become a thing recently with Researchers started using Machine Learning itself to create new Features that can help in Model Accuracy. While most of the automated Featuring Engineering address numeric data, Text Data has always been left out in this race because of its inherent unstructured nature. No more, I could say. </p>
<h3>textfeatures – R package</h3>
<p>Michael Kearney, Assistant Professor in University of Missouri, well known in the R community for the modern twitter package <a rel="nofollow" href="https://github.com/mkearney/rtweet" target="_blank">rtweet</a>, has come up with a new R packaged called <code>textfeatures</code> that basically generates a bunch of features for any text data that you supply. Before you dream of Deep Learning based Package for Automated Text Feature Engineering, This isn’t that. This uses very simple Text Analysis principles and generates features like Number of Upper Case letters, Number of Punctuations – plain simple stuff and nothing fancy but pretty useful ones. </p>
<h3>Installation</h3>
<p><code>textfeatures</code> can be installed directly from <a rel="nofollow" href="https://cran.r-project.org/web/packages/textfeatures/index.html" target="_blank">CRAN</a> and the development version is available on <a rel="nofollow" href="https://github.com/mkearney/textfeatures" target="_blank">github</a>. </p>
<pre>
install.packages(&quot;textfeatures&quot;)
</pre>
<h2> Use Case</h2>
<p>In this post, we will use <code>textfeatures</code> package to generate features for Fifa official world cup ios app reviews from the UK. We will R package <code>itunesr</code> to extract reviews and <code>tidyverse</code> for data manipulation and plotting. </p>
<h3>Loading required Packages:<br>
</h3>
<p>Let us load all the required packages.</p>
<pre>
#install.packages(&quot;itunesr&quot;)
#install.packages(&quot;textfeatures&quot;)
#install.packages(&quot;tidyverse&quot;)

library(itunesr)
library(textfeatures)
library(tidyverse)

</pre>
<h3>Extracting recent Reviews:</h3>
<pre>
#Get UK Reviews of Fifa official world cup ios app
#https://itunes.apple.com/us/app/2018-fifa-world-cup-russia/id756904853?mt=8

reviews1 &lt;- getReviews(756904853,&quot;GB&quot;,1)
reviews2 &lt;- getReviews(756904853,&quot;GB&quot;,2)
reviews3 &lt;- getReviews(756904853,&quot;GB&quot;,3)
reviews4 &lt;- getReviews(756904853,&quot;GB&quot;,4)

#Combining all the reviews into one dataframe
reviews &lt;- rbind(reviews1,
                 reviews2,
                 reviews3,
                 reviews4)
</pre>
<h3>textfeatures Magic Begins:</h3>
<p>As we have got the reviews, Let us allow textfeatures to do its magic. We will use the function <code>textfeatures()</code> to do that. </p>
<pre>
#Combining all the reviews into one dataframe
reviews &lt;- rbind(reviews1,
                 reviews2,
                 reviews3,
                 reviews4)

# generate text features
feat &lt;- textfeatures(reviews$Review)

# check what all features generated 
glimpse(feat)
<em>Observations: 200
Variables: 17
$ n_chars 149, 13, 263, 189, 49, 338, 210, 186, 76, 14, 142, 114, 242, ...
$ n_commas 1, 0, 0, 0, 0, 1, 2, 1, 1, 0, 1, 1, 0, 3, 0, 0, 1, 0, 3, 1, 0...
$ n_digits 0, 0, 6, 3, 0, 4, 1, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 3, 0, 0, 0...
$ n_exclaims 0, 0, 2, 2, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 3, 0, 0...
$ n_extraspaces 1, 0, 0, 0, 0, 3, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 4, 0...
$ n_hashtags 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0...
$ n_lowers 140, 11, 225, 170, 46, 323, 195, 178, 70, 12, 129, 106, 233, ...
$ n_lowersp 0.9400000, 0.8571429, 0.8560606, 0.9000000, 0.9400000, 0.9557...
$ n_mentions 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0...
$ n_periods 2, 0, 5, 1, 0, 0, 3, 1, 2, 0, 2, 1, 1, 2, 0, 0, 4, 2, 0, 4, 0...
$ n_urls 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0...
$ n_words 37, 2, 55, 45, 12, 80, 42, 41, 14, 3, 37, 28, 50, 16, 15, 8, ...
$ n_caps 4, 1, 12, 8, 2, 7, 3, 4, 2, 2, 6, 4, 6, 2, 3, 1, 6, 4, 29, 9,...
$ n_nonasciis 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0...
$ n_puncts 2, 1, 7, 0, 1, 3, 4, 1, 1, 0, 4, 2, 2, 0, 1, 0, 1, 1, 0, 7, 2...
$ n_capsp 0.03333333, 0.14285714, 0.04924242, 0.04736842, 0.06000000, 0...
$ n_charsperword 3.947368, 4.666667, 4.714286, 4.130435, 3.846154, 4.185185, 4...
</em>
</pre>
<p>As you can see above, textfeatures have created 17 new features. Please note that these features will remain same for any text data. </p>
<h3>Visualizing the outcome:</h3>
<p>For this post, we wouldn’t build a Machine Learning Model but these features can very well be used for build a Classification model like Sentiment Classification or Category Classification. </p>
<p>But right now, we will just visualize the outcome with some features. </p>
<p>We can see if there’s any relations between a number of characters and number of characters per word, with respect to the Review rating. A hypothesis could be that people who give good rating wouldn’t write long or otherwise. We are not going to validate it here, but just visualizing using a scatter plot. </p>
<pre>
# merging features with original reviews
reviews_all % 
  ggplot(aes(n_charsperword, n_chars, colour = Rating)) + geom_point() 
</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2018/07/Features_plot1.png?ssl=1" target="_blank"><img sizes="(max-width: 1000px) 100vw, 1000px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2018/07/Features_plot1.png?w=450&ssl=1" alt="" class="wp-image-15255" data-recalc-dims="1"></a></p>
<p>Let’s bring in a different perspective to the same hypothesis with a different plot but while comparing against a number of words instead of a number of characters. </p>
<pre>
reviews_all %>% 
  ggplot(aes(n_charsperword, n_words)) + geom_point() +
  facet_wrap(~Rating) +
  stat_smooth()
</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2018/07/features_demo2.png?ssl=1" target="_blank"><img sizes="(max-width: 1000px) 100vw, 1000px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2018/07/features_demo2.png?w=450&ssl=1" alt="" class="wp-image-15256" data-recalc-dims="1"></a></p>
<p>Thus, you can use <code>textfeatures</code> to automatically generate new features and make a better understanding of your text data. Hope this post helps you get started with this beautiful package and if you’d like to know more on Text Analysis check out <a rel="nofollow" href="https://www.datacamp.com/courses/sentiment-analysis-in-r-the-tidy-way?tap_a=5644-dce66f&tap_s=10907-287229" target="_blank">this tutorial by Julia Silge</a>. The complete code used here is available on <a rel="nofollow" href="https://github.com/amrrs/blogpost_codes/blob/master/textfeatures_demo.R" target="_blank">my github</a>. </p>
<ul></ul>
<p><strong>
<p>Related Post</p>
<p></p></strong>
<ol>
<li><a rel="nofollow" href="https://datascienceplus.com/explaining-keras-image-classification-models-with-lime/" title="Permanent Link toExplaining Keras image classification models with LIME" target="_blank">Explaining Keras image classification models with LIME</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/image-classification-with-keras-in-roughly-100-lines-of-code/" title="Permanent Link toImage classification with keras in roughly 100 lines of code" target="_blank">Image classification with keras in roughly 100 lines of code</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/r-vs-python-image-classification-with-keras/" title="Permanent Link toR vs Python: Image Classification with Keras" target="_blank">R vs Python: Image Classification with Keras</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/update-can-we-predict-flu-outcome-with-machine-learning-in-r/" title="Permanent Link toUpdate: Can we predict flu outcome with Machine Learning in R?" target="_blank">Update: Can we predict flu outcome with Machine Learning in R?</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/evaluation-of-topic-modeling-topic-coherence/" title="Permanent Link toEvaluation of Topic Modeling: Topic Coherence" target="_blank">Evaluation of Topic Modeling: Topic Coherence</a></li>
</ol>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://datascienceplus.com/automated-text-feature-engineering-using-textfeatures-in-r/"> R Programming – DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/1ljgxqY_7xs" height="1"></a></div>