<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/is-udpipe-your-new-nlp-processor-for-tokenization-parts-of-speech-tagging-lemmatization-and-dependency-parsing/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.bnosac.be/index.php/blog/68-is-udpipe-your-new-nlp-processor-for-tokenization-parts-of-speech-tagging-lemmatization-and-dependency-parsing"> bnosac :: open analytical helpers</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>If you work on natural language processing in a day-to-day setting which involves statistical engineering, at a certain timepoint you need to process your text with a number of text mining procedures of which the following ones are steps you must do before you can get usefull information about your text</p>
<ul>
<li>Tokenisation (splitting your full text in words/terms)</li>
<li>Parts of Speech (POS) tagging (assigning each word a syntactical tag like is the word a verb/noun/adverb/number/…)</li>
<li>Lemmatisation (a lemma means that the term we “are” is replaced by the verb to “be”, more information: <a rel="nofollow" href="https://en.wikipedia.org/wiki/Lemmatisation" target="_blank">https://en.wikipedia.org/wiki/Lemmatisation</a>)</li>
<li>Dependency Parsing (finding relationships between, namely between “head” words and words which modify those heads, allowing you to look to words which are maybe far away from each other in the raw text but influence each other)</li>
</ul>
<p><img alt="udpipe rlogo" src="https://i1.wp.com/www.bnosac.be/images/bnosac/blog/udpipe-rlogo.png?w=456" data-recalc-dims="1"></p>
<p>If you do this in <strong>R</strong>, there aren’t much available tools to do this. In fact there are none which</p>
<ol>
<li><strong>do this for multiple language</strong></li>
<li><strong>do not depend on external software dependencies (java/python)</strong></li>
<li><strong>which also allow you to train your own parsing &amp; tagging models.</strong></li>
</ol>
<p>Except <strong>R package udpipe</strong> (<a rel="nofollow" href="https://github.com/bnosac/udpipe" target="_blank">https://github.com/bnosac/udpipe</a>, <a rel="nofollow" href="https://cran.r-project.org/package=udpipe" target="_blank">https://CRAN.R-project.org/package=udpipe</a>) which satisfies these 3 criteria.</p>
<p>If you are interested in doing the annotation, pre-trained models are available for 50 languages (see ?udpipe_download_model) for details. Let’s show how this works on some Dutch text and what you get of of this.</p>
<p><code>library(udpipe)</code><br><code>dl &lt;- udpipe_download_model(language = &quot;dutch&quot;)</code><br><code>dl</code></p>
<p><code>language                                                                      file_model</code><br><code>   dutch C:/Users/Jan/Dropbox/Work/RForgeBNOSAC/BNOSAC/udpipe/dutch-ud-2.0-170801.udpipe</code></p>
<p><code>udmodel_dutch &lt;- udpipe_load_model(file = &quot;dutch-ud-2.0-170801.udpipe&quot;)</code><br><code>x &lt;- udpipe_annotate(udmodel_dutch,</code><br><code>                     x = &quot;Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.&quot;)</code><br><code>x &lt;- as.data.frame(x)</code><br><code>x</code></p>
<p>The result of this is a dataset where text has been splitted in paragraphs, sentences, words, words are replaced by their lemma (ging > ga, nam > neem), you get the universal parts of speech tags, detailed parts of speech tags, you get features of the word and with the head_token_id we see which words are influencing other words in the text as well as the dependency relationship between these words.</p>
<p><img alt="udpipe example" src="https://i0.wp.com/www.bnosac.be/images/bnosac/blog/udpipe-example.png?w=456" data-recalc-dims="1"></p>
<p>To go from that dataset to meaningfull visualisations like this one is than just a matter of a few lines of code. The following visualisation shows the co-occurrence of nouns with customer feedback on Airbnb appartment stays in Brussels (open data available at <a rel="nofollow" href="http://insideairbnb.com/get-the-data.html" target="_blank">http://insideairbnb.com/get-the-data.html</a>).</p>
<p><img alt="udpipe example coocurrence" src="https://i1.wp.com/www.bnosac.be/images/bnosac/blog/udpipe-example-coocurrence.png?w=456" data-recalc-dims="1"></p>
<p>In a next post, we’ll show how to train your own tagging models.</p>
<p>If you like this type of analysis or if you are interested in text mining with R, we have 3 upcoming courses planned on text mining. Feel free to register at the following links.</p>
<ul>
<ul>
<li>18-19/10/2017: Statistical machine learning with R. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/statistical-machine-learning-with-r" target="_blank">here</a></li>
<li><strong>08+10/11/2017: Text mining with R. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/text-mining-with-r" target="_blank">here</a></strong></li>
<li><strong>27-28/11/2017: Text mining with R. Brussels (Belgium). <a rel="nofollow" href="http://di-academy.com/bootcamp/" target="_blank">http://di-academy.com/bootcamp</a> + send mail to <span><a rel="nofollow" href="mailto:training@di-academy.com" target="_blank">training@di-academy.com</a></span></strong></li>
<li>19-20/12/2017: Applied spatial modelling with R. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/applied-spatial-modelling-with-r" target="_blank">here</a></li>
<li>20-21/02/2018: Advanced R programming. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/AdvancedprogramminginR.html" target="_blank">here</a></li>
<li>08-09/03/2018: Computer Vision with R and Python. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/ComputervisionwithRandPython" target="_blank">here</a></li>
<li><strong>22-23/03/2018: Text Mining with R. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/text-mining-with-r" target="_blank">here</a></strong></li>
</ul>
</ul>
<p>For business questions on text mining, feel free to contact BNOSAC by <a rel="nofollow" href="http://www.bnosac.be/index.php/contact/get-in-touch?id=0" target="_blank">sending us a mail here</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.bnosac.be/index.php/blog/68-is-udpipe-your-new-nlp-processor-for-tokenization-parts-of-speech-tagging-lemmatization-and-dependency-parsing"> bnosac :: open analytical helpers</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/eILXtxTyAys" height="1"></a></div>