<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/an-overview-of-keyword-extraction-techniques/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.bnosac.be/index.php/blog/77-an-overview-of-keyword-extraction-techniques"> bnosac :: open analytical helpers</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In this blogpost, we will show 6 keyword extraction techniques which allow to<strong> find keywords in plain text</strong>. Keywords are frequently occuring words which occur somehow together in plain text. Common examples are New York, Monte Carlo, Mixed Models, Brussels Hoofdstedelijk Gewest, Public Transport, Central Station, p-values, …</p>
<p>If you master these techniques, it will allow you to easily step away from doing simple word frequency statistics to more business-relevant text summarisation. For this, we will use the <strong>udpipe R package</strong> (docs at <a rel="nofollow" href="https://cran.r-project.org/package=udpipe" target="_blank">https://CRAN.R-project.org/package=udpipe</a> or <a rel="nofollow" href="https://bnosac.github.io/udpipe/en" target="_blank">https://bnosac.github.io/udpipe/en</a>) which is the core R package you need for doing this type of t ext processing.We’ll basically show how to easily extract keywords as follows:</p>
<p>1. Find keywords by doing <strong>Parts of Speech tagging in order to identify nouns</strong><br>2. Find keywords based on<strong> Collocations and Co-occurrences</strong><br>3. Find keywords based on the<strong> Textrank algorithm</strong><br>4. Find keywords based on <strong>RAKE (rapid automatic keyword extraction)</strong><br>5. Find keywords by looking for <strong>Phrases</strong> (noun phrases / verb phrases)<br>6. Find keywords based on <strong>results of dependency parsing</strong> (getting the subject of the text)<strong><br></strong></p>
<p>These techniques will allow you to move away from showing silly word graphs to more relevant graphs containing keywords.</p>
<p><img alt="wordclouds" src="https://i2.wp.com/www.bnosac.be/images/bnosac/blog/wordclouds.png?w=456" data-recalc-dims="1"></p>
<h3><strong>Example</strong></h3>
<p>As an example we are going to use <strong>feedback in Spanish of customers going to an AirBnB appartment in Brussels</strong>. This data is part of the <a rel="nofollow" href="https://cran.r-project.org/package=udpipe" target="_blank">udpipe R</a> package. We extract the Spanish text and annotate it using the udpipe R package. Annotation performs <span><strong>tokenisation, parts of speech tagging, lemmatisation and dependency parsing</strong></span>.</p>
<pre>library(udpipe)<br>library(textrank)<br>## First step: Take the Spanish udpipe model and annotate the text. Note: this takes about 3 minutes<br>data(brussels_reviews)<br>comments &lt;- subset(brussels_reviews, language %in% &quot;es&quot;)<br>ud_model &lt;- udpipe_download_model(language = &quot;spanish&quot;)<br>ud_model &lt;- udpipe_load_model(ud_model$file_model)<br>x &lt;- udpipe_annotate(ud_model, x = comments$feedback)<br>x &lt;- as.data.frame(x)</pre>
<p><img alt="udpipe example airbnb" src="https://i0.wp.com/www.bnosac.be/images/bnosac/blog/udpipe-output-keywords1.png?w=456" data-recalc-dims="1"></p>
<p>Once we have the annotation, finding keywords is a breeze. Let’s show how this can be easily accomplished.</p>
<h3><strong>Option 1: Extracting only nouns</strong></h3>
<p>An easy way in order to find keywords is by looking at <strong>nouns</strong>. As each term has a Parts of Speech tag if you annotated text using the udpipe package, you can easily do this as follows.</p>
<pre>stats &lt;- subset(x, upos %in% &quot;NOUN&quot;)<br>stats &lt;- txt_freq(x = stats$lemma)<br>library(lattice)<br>stats$key &lt;- factor(stats$key, levels = rev(stats$key))<br>barchart(key ~ freq, data = head(stats, 30), col = &quot;cadetblue&quot;, main = &quot;Most occurring nouns&quot;, xlab = &quot;Freq&quot;)</pre>
<p><img alt="keywords plot1" src="https://i2.wp.com/www.bnosac.be/images/bnosac/blog/keywords-plot1.png?w=450" data-recalc-dims="1"></p>
<h3><strong>Option 2: Collocation &amp; co-occurrences</strong></h3>
<p>Although nouns are a great start, you are probably interested in multi-word expressions. You can get multi-word expression by looking either at <span>collocations</span> (words following one another), at word <span>co-occurrences within each sentence</span> or at word<span> co-occurrences of words which are close in the neighbourhood of one another</span>. These approaches can be executed as follows using the <a rel="nofollow" href="https://cran.r-project.org/package=udpipe" target="_blank">udpipe R</a> package. If we<strong> combine this with selecting only the nouns and adjectives</strong>, this becomes already nice.</p>
<pre>## Collocation (words following one another)<br>stats &lt;- keywords_collocation(x = x, <br>                             term = &quot;token&quot;, group = c(&quot;doc_id&quot;, &quot;paragraph_id&quot;, &quot;sentence_id&quot;),<br>                             ngram_max = 4)<br>## Co-occurrences: How frequent do words occur in the same sentence, in this case only nouns or adjectives<br>stats &lt;- cooccurrence(x = subset(x, upos %in% c(&quot;NOUN&quot;, &quot;ADJ&quot;)), <br>                     term = &quot;lemma&quot;, group = c(&quot;doc_id&quot;, &quot;paragraph_id&quot;, &quot;sentence_id&quot;))<br>## Co-occurrences: How frequent do words follow one another<br>stats &lt;- cooccurrence(x = x$lemma, <br>                     relevant = x$upos %in% c(&quot;NOUN&quot;, &quot;ADJ&quot;))<br>## Co-occurrences: How frequent do words follow one another even if we would skip 2 words in between<br>stats &lt;- cooccurrence(x = x$lemma, <br>                     relevant = x$upos %in% c(&quot;NOUN&quot;, &quot;ADJ&quot;), skipgram = 2)<br>head(stats)<br>      term1     term2 cooc<br>     barrio tranquilo   36<br>   estacion      tren   30<br> transporte   publico   23<br>     centro    ciudad   23<br>      pleno    centro   20<br>   estacion   central   19</pre>
<p>Visualisation of these co-occurrences can be done using a network plot as follows for the top 30 most frequent co-occurring nouns and adjectives.</p>
<pre>library(igraph)<br>library(ggraph)<br>library(ggplot2)<br>wordnetwork &lt;- head(stats, 30)<br>wordnetwork &lt;- graph_from_data_frame(wordnetwork)<br>ggraph(wordnetwork, layout = &quot;fr&quot;) +<br>  geom_edge_link(aes(width = cooc, edge_alpha = cooc), edge_colour = &quot;pink&quot;) +<br>  geom_node_text(aes(label = name), col = &quot;darkgreen&quot;, size = 4) +<br>  theme_graph(base_family = &quot;Arial Narrow&quot;) +<br>  theme(legend.position = &quot;none&quot;) +<br>  labs(title = &quot;Cooccurrences within 3 words distance&quot;, subtitle = &quot;Nouns &amp; Adjective&quot;)</pre>
<p><img alt="keywords plot2" src="https://i1.wp.com/www.bnosac.be/images/bnosac/blog/keywords-plot2.png?w=450" data-recalc-dims="1"></p>
<h3><strong>Option 3: Textrank (word network ordered by Google Pagerank)</strong></h3>
<p>Another approach for keyword detection is <strong>Textrank</strong>. Textrank is an algorithm implemented in the <a rel="nofollow" href="https://cran.r-project.org/package=textrank" target="_blank">textrank R package</a>. The algorithm allows to summarise text and as well allows to extract keywords. This is done by constructing a <strong>word network</strong> by looking if words are following one another. On top of that network the ‘Google Pagerank’ algorithm is applied to extract relevant words after which relevant words which are following one another are combined to get keywords. In the below example, we are interested in finding keywords using that algorithm of either nouns or adjectives following one another. You can see from the plot below that the keywords combines words together into multi-word expressions.</p>
<pre>stats &lt;- textrank_keywords(x$lemma, <br>                          relevant = x$upos %in% c(&quot;NOUN&quot;, &quot;ADJ&quot;), <br>                          ngram_max = 8, sep = &quot; &quot;)<br>stats &lt;- subset(stats$keywords, ngram > 1 &amp; freq >= 5)<br>library(wordcloud)<br>wordcloud(words = stats$keyword, freq = stats$freq)</pre>
<p><img src="https://i1.wp.com/www.bnosac.be/images/bnosac/blog/keywords-plot5.png?resize=424%2C340" alt="keywords plot5" width="424" data-recalc-dims="1" height="340"></p>
<h3><strong>Option 4: Rapid Automatic Keyword Extraction: RAKE</strong></h3>
<p>Next basic algorithm is called RAKE which is an acronym for <strong>Rapid Automatic Keyword Extraction</strong>. It looks for keywords by looking to a contiguous sequence of words which do not contain irrelevant words. Namely by</p>
<ol>
<li>calculating a score for each word which is part of any candidate keyword, this is done by
<ul>
<li>among the words of the candidate keywords, the algorithm looks how many times each word is occurring and how many times it co-occurs with other words</li>
<li><strong>each word gets a score which is the ratio of the word degree (how many times it co-occurs with other words) to the word frequency</strong></li>
</ul>
</li>
<li>a RAKE score for the full candidate keyword is calculated by summing up the scores of each of the words which define the candidate keyword</li>
</ol>
<pre>stats &lt;- keywords_rake(x = x, <br>                      term = &quot;token&quot;, group = c(&quot;doc_id&quot;, &quot;paragraph_id&quot;, &quot;sentence_id&quot;),<br>                      relevant = x$upos %in% c(&quot;NOUN&quot;, &quot;ADJ&quot;),<br>                      ngram_max = 4)<br>head(subset(stats, freq > 3))<br>               keyword ngram freq     rake<br> perfectas condiciones     2    4 2.000000<br>            unica pega     2    7 2.000000<br>           grand place     2    6 1.900000<br>   grandes anfitriones     2    4 1.809717<br>    transporte publico     2   21 1.685714<br>    buenos anfitriones     2    9 1.662281</pre>
<h3><strong>Option 5: Phrases</strong></h3>
<p>Next option is to extract phrases. These are defined as a <strong>sequence of Parts of Speech Tags</strong>. Common type of phrases are noun phrases or verb phrases. How does this work? Parts of Speech tags are recoded to one of the following one-letters: (A: adjective, C: coordinating conjuction, D: determiner, M: modifier of verb, N: noun or proper noun, P: preposition). Next you can define a regular expression to indicate a sequence of parts of speech tags which you want to extract from the text.</p>
<pre>## Simple noun phrases (a adjective+noun, pre/postposition, optional determiner and another adjective+noun)<br>x$phrase_tag &lt;- as_phrasemachine(x$upos, type = &quot;upos&quot;)<br>stats &lt;- keywords_phrases(x = x$phrase_tag, term = x$token, <br>                         pattern = &quot;(A|N)+N(P+D*(A|N)*N)*&quot;, <br>                         is_regex = TRUE, ngram_max = 4, detailed = FALSE)<br>head(subset(stats, ngram > 2))<br>                        keyword ngram freq<br>                   Gare du Midi     3   12<br>       pleno centro de Bruselas     4    6<br>               15 minutos a pie     4    4<br>               nos explico todo     3    4<br> primera experiencia con Airbnb     4    3<br>                   Gare du Nord     3    3</pre>
<h3><strong>Option 6: Use dependency parsing output to get the nominal subject and the adjective of it</strong></h3>
<p>In the last option, we will show how to use the results of the <strong>dependency parsing</strong>. When you executed the annotation using udpipe, the <em>dep_rel</em> field indicates how words are related to one another. A token is related to the parent using<em> token_id</em> and <em>head_token_id</em>. The<em> dep_rel</em> field indicates how words are linked to one another. The type of relations are defined at <a rel="nofollow" href="http://universaldependencies.org/u/dep/index.html" target="_blank">http://universaldependencies.org/u/dep/index.html</a>. For this exercise we are going to take the words which have as dependency relation <em>nsubj</em> indicating the <strong>nominal subject</strong> and we are adding to that the adjective which is changing the nominal subject.</p>
<p>In this way we can combine what are people talking about with the adjective they use when they talk about the subject.</p>
<pre>stats &lt;- merge(x, x, <br>           by.x = c(&quot;doc_id&quot;, &quot;paragraph_id&quot;, &quot;sentence_id&quot;, &quot;head_token_id&quot;),<br>           by.y = c(&quot;doc_id&quot;, &quot;paragraph_id&quot;, &quot;sentence_id&quot;, &quot;token_id&quot;),<br>           all.x = TRUE, all.y = FALSE, <br>           suffixes = c(&quot;&quot;, &quot;_parent&quot;), sort = FALSE)<br>stats &lt;- subset(stats, dep_rel %in% &quot;nsubj&quot; &amp; upos %in% c(&quot;NOUN&quot;) &amp; upos_parent %in% c(&quot;ADJ&quot;))<br>stats$term &lt;- paste(stats$lemma_parent, stats$lemma, sep = &quot; &quot;)<br>stats &lt;- txt_freq(stats$term)<br>library(wordcloud)<br>wordcloud(words = stats$key, freq = stats$freq, min.freq = 3, max.words = 100,<br>          random.order = FALSE, colors = brewer.pal(6, &quot;Dark2&quot;))</pre>
<p><img alt="keywords plot4" src="https://i2.wp.com/www.bnosac.be/images/bnosac/blog/keywords_plot4.png?w=456" data-recalc-dims="1"></p>
<p>Now up to you. Can you do the same on your own text?</p>
<p><strong>Credits:</strong> This analysis would not have been possible without the Spanish annotated treebanks (<a rel="nofollow" href="https://github.com/UniversalDependencies/UD_Spanish-GSD" target="_blank">https://github.com/UniversalDependencies/UD_Spanish-GSD</a> in particular as made available through <a rel="nofollow" href="http://universaldependencies.org/" target="_blank">http://universaldependencies.org</a>) and the UDPipe C++ library and models provided by Milan Straka (<a rel="nofollow" href="https://github.com/ufal/udpipe" target="_blank">https://github.com/ufal/udpipe</a>). All credits have to go there. </p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.bnosac.be/index.php/blog/77-an-overview-of-keyword-extraction-techniques"> bnosac :: open analytical helpers</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/pbAeV3IDcH8" height="1"></a></div>