<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/udpipe-version-0-7-for-natural-language-processing-nlp-alongside-tidytext-quanteda-tm/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.bnosac.be/index.php/blog/81-udpipe-version-0-7-for-natural-language-processing-nlp-alongside-tidytext-quanteda-tm"> bnosac :: open analytical helpers - bnosac :: open analytical helpers</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img src="https://i1.wp.com/www.bnosac.be/images/bnosac/blog/udpipe-rlogo.png?resize=319%2C178" alt="udpipe rlogo" width="319" data-recalc-dims="1" height="178">This blogpost announces the release of the udpipe R package version 0.7 on CRAN. <a rel="nofollow" href="https://bnosac.github.io/udpipe/en" target="_blank">udpipe</a> is an R package which does tokenization, parts of speech tagging, lemmatization, morphological feature tagging and dependency parsing. It’s main feature is that it is a lightweight R package which works on more than 50 languages and gives you rich NLP output out of the box. </p>
<p>The package was updated mainly in order to more easily work with the <a rel="nofollow" href="https://github.com/bnosac/crfsuite" target="_blank">crfsuite R package</a> which does entity/intent recogntion and chunking. The user-visible changes that were made are that udpipe now has a shorthand for working with text in the <a rel="nofollow" href="https://github.com/ropensci/tif" target="_blank">TIF format</a> and it now also allows to indicate the location of the token inside the original text. Next to this, version 0.7 also caches the udpipe models.</p>
<h3><strong>Example</strong></h3>
<p>Using udpipe (version >= 0.7) works as follows. First download the model of your language and next do the annotation.</p>
<pre>library(udpipe)<br>udmodel &lt;- udpipe_download_model(language = &quot;dutch&quot;)<br>x &lt;- udpipe(&quot;De federale regering besliste vandaag dat er een neutrale verpakking voor roltabak en sigaretten komt&quot;, <br>            object = udmodel)</pre>
<p>Since version 0.7, you can now also directly indicate the language. This will download the udpipe annotation model if it is not already downloaded. Please inspect the help of udpipe_download_model for more details on the languages available and the license of these.</p>
<pre>x &lt;- udpipe(&quot;Je veux qu’on me juge pour ce que je suis et non pour ce qu’était mon père&quot;, &quot;french&quot;)<br>x &lt;- udpipe(&quot;Europa lança taxas sobre navios para tirar lixo do fundo do mar.&quot;, &quot;portuguese&quot;)<br>x &lt;- udpipe(&quot;आपके इस स्नेह्पूर्ण और जोरदार स्वागत से मेरा हृदय आपार हर्ष से भर गया है। मैं आपको दुनिया के सबसे पौराणिक भिक्षुओं की तरफ से धन्यवाद् देता हूँ। मैं आपको सभी धर्मों की जननी कि तरफ से धन्यवाद् देता हूँ, और मैं आपको सभी जाति-संप्रदाय के लाखों-करोड़ों हिन्दुओं की तरफ से धन्यवाद् देता हूँ। मेरा धन्यवाद् उन वक्ताओं को भी जिन्होंने ने इस मंच से यह कहा कि दुनिया में शहनशीलता का विचार सुदूर पूरब के देशों से फैला है।&quot;, &quot;hindi&quot;)<br>x &lt;- udpipe(&quot;The economy is weak but the outlook is bright&quot;, &quot;english&quot;)<br>x &lt;- udpipe(&quot;Maxime y su mujer hicieron que nuestra estancia fuera lo mas comoda posible&quot;, &quot;spanish&quot;)<br>x &lt;- udpipe(&quot;A félmilliárdos MVM-támogatásból 433 milliót négy nyúlfarknyi imázsvideóra költött&quot;, &quot;hungarian&quot;)<br>x &lt;- udpipe(&quot;同樣,施力的大小不同,引起的加速度不同,最終的結果也不一樣,亦可以從向量的加成性來看&quot;, &quot;chinese&quot;)</pre>
<p>The result is a a data.frame with one row per doc_id and term_id containing all the <strong>tokens</strong> in the data, the <strong>lemma</strong>, the <strong>part of speech tags</strong>, the <strong>morphological features</strong>, the <strong>dependency relationship</strong> along the tokens and the <strong>location where the token is found in the original text</strong>.</p>
<p><img alt="screenshot udpipe chinese" src="https://i2.wp.com/www.bnosac.be/images/bnosac/blog/screenshot-udpipe-chinese.png?w=450" data-recalc-dims="1"></p>
<h3><strong>Use alongside other R packages</strong></h3>
<p>R has a rich NLP ecosystem. If you want to use <a rel="nofollow" href="https://cran.r-project.org/package=udpipe" target="_blank">udpipe</a> alongside other R packages, let’s enumerate some basic possibilities where we show how to easily extract lemma’s and text of parts of speech tags you are interested in:<br>Below we show how to use udpipe alongside the 3 popular R packages: <a rel="nofollow" href="https://cran.r-project.org/package=tidytext" target="_blank">tidytext</a>, <a rel="nofollow" href="https://cran.r-project.org/package=quanteda" target="_blank">quanteda</a> and <a rel="nofollow" href="https://cran.r-project.org/package=tm" target="_blank">tm</a> on the following data.frame in TIF format.</p>
<pre>rawdata &lt;- data.frame(doc_id = c(&quot;doc1&quot;, &quot;doc2&quot;), <br>                      text = c(&quot;The economy is weak but the outlook is bright.&quot;, <br>                               &quot;Natural Language Processing has never been more easy than this.&quot;), <br>                      stringsAsFactors = FALSE)</pre>
<h4>Using <strong>tidytext</strong></h4>
<p>In this code, we let tidytext do tokenisation and use udpipe to enrich the token list. Next we subset the data.frame of tokens by extracting only proper nouns, nouns and adjectives.</p>
<pre>library(tidytext)<br>library(udpipe)<br>library(dplyr)<br>x &lt;- unnest_tokens(rawdata, input = text, output = word)<br>x &lt;- udpipe(split(x$word, x$doc_id), &quot;english&quot;)<br>x &lt;- filter(x, upos %in% c(&quot;PROPN&quot;, &quot;NOUN&quot;, &quot;ADJ&quot;))</pre>
<h4>Using <strong>quanteda</strong></h4>
<p>In the code below, we let udpipe do tokenisation and provide the lemma’s back in quanteda’s tokens element.</p>
<pre>library(quanteda)<br>library(udpipe)<br>x &lt;- corpus(rawdata, textField = &quot;text&quot;)<br>tokens &lt;- udpipe(texts(x), &quot;english&quot;)<br>x$tokens &lt;-  as.tokenizedTexts(split(tokens$lemma, tokens$doc_id))</pre>
<h4>Using <strong>tm</strong></h4>
<p>Below, we get only the lemma’s of the nouns, proper nouns and adjectives and apply this using the tm_map functionality from tm.</p>
<pre>library(tm)<br>library(udpipe)<br>x &lt;- VCorpus(VectorSource(rawdata$text))<br>x &lt;- tm_map(x, FUN=function(txt){<br>  data &lt;- udpipe(content(txt), &quot;english&quot;)<br>  data &lt;- subset(data, upos %in% c(&quot;PROPN&quot;, &quot;NOUN&quot;, &quot;ADJ&quot;))<br>  paste(data$lemma, collapse = &quot; &quot;)<br>})</pre>
<p>UDPipe currently already uses deep learning techniques (e.g a GRU network) for doing the tokenisation but the dependency parsing was enhanced in 2018 by incorporating tensorflow. On the roadmap for a next release will be the integration of the UDPipe future enhancements (which got 3rd place at the <a rel="nofollow" href="http://universaldependencies.org/conll18/results.html" target="_blank">CoNLL shared task from 2018</a>) including these tensorflow components.</p>
<h3><strong>Training on Text Mining <br></strong></h3>
<p>Are you interested in how text mining techniques work, then you might be interested in the following data science courses that are held in the coming months.<img alt="rtraining" src="https://i1.wp.com/www.bnosac.be/images/bnosac/blog/rtraining.png?w=456" data-recalc-dims="1"></p>
<ul>
<li><strong>08-09/10/2018: Text mining with R. Brussels (Belgium). <a rel="nofollow" href="http://di-academy.com/bootcamp/" target="_blank">http://di-academy.com/bootcamp</a> + send mail to <a rel="nofollow" href="mailto:training@di-academy.com" target="_blank">training@di-academy.com</a></strong></li>
<li>15-16/10/2018: Statistical machine learning with R. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/statistical-machine-learning-with-r" target="_blank">here</a></li>
<li><strong>05-06/11/2018: Text mining with R. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/text-mining-with-r" target="_blank">here</a></strong></li>
<li>19-20/12/2018: Applied spatial modelling with R. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/applied-spatial-modelling-with-r" target="_blank">here</a></li>
<li>21-22/02/2018: Advanced R programming. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/AdvancedprogramminginR.html" target="_blank">here</a></li>
<li>13-14/03/2018: Computer Vision with R and Python. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/ComputervisionwithRandPython" target="_blank">here</a></li>
<li>     15/03/2019: Image Recognition with R and Python: Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/Imagerecognition" target="_blank">here</a></li>
<li><strong>01-02/04/2019: Text Mining with R. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/text-mining-with-r" target="_blank">here</a></strong></li>
</ul>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.bnosac.be/index.php/blog/81-udpipe-version-0-7-for-natural-language-processing-nlp-alongside-tidytext-quanteda-tm"> bnosac :: open analytical helpers - bnosac :: open analytical helpers</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/zEQvU51JgYY" height="1"></a></div>