<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/a-comparison-between-spacy-and-udpipe-for-natural-language-processing-for-r-users/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.bnosac.be/index.php/blog/75-a-comparison-between-spacy-and-udpipe-for-natural-language-processing-for-r-users"> bnosac :: open analytical helpers</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In the last few years, Natural Language Processing (NLP) has become more and more an open multi-lingual task instead of being held back by language, country and legal boundaries. With the advent of commonly used open data regarding natural language processing tasks as available at <a rel="nofollow" href="http://universaldependencies.org/" target="_blank">http://universaldependencies.org</a> one can now relatively easily compare different toolkits which perform natural language processing. In this post we compare the <a rel="nofollow" href="https://bnosac.github.io/udpipe/en/index.html" target="_blank">udpipe R package</a> to the <a rel="nofollow" href="https://cran.r-project.org/package=spacyr" target="_blank">spacyr R package.</a></p>
<h3>UDPipe – spaCy comparison</h3>
<p>A traditional natural language processing flow consists of a number of building blocks which can be used to structure your Natural Language Application on top of it. Namely</p>
<p>1. tokenisation<br>2. parts of speech tagging<br>3. lemmatisation<br>4. morphological feature tagging<br>5. syntactic dependency parsing<br>6. entity recognition<br>7. extracting word &amp; sentence meaning</p>
<p>Both of these R packages allow to do this where</p>
<ul>
<li>spacyr is an R package around the popular spaCy NLP toolkit: <a rel="nofollow" href="https://github.com/explosion/spaCy" target="_blank">https://github.com/explosion/spaCy</a></li>
<li>udpipe is an R package around the less known UDPipe NLP toolkit: <a rel="nofollow" href="https://github.com/ufal/udpipe" target="_blank">https://github.com/ufal/udpipe</a></li>
</ul>
<h3>Comparison</h3>
<p>In the comparison, we will provide general feedback on the following elements</p>
<ul>
<li>Languages which are covered by the tools</li>
<li>Ease of use</li>
<li>Annotation possibilities</li>
<li>Annotation accuracy of the models</li>
<li>Annotation speed   </li>
</ul>
<h3><a rel="nofollow" href="https://github.com/bnosac/GAlogger" target="_blank"><img alt="udpipe spacy" src="https://i1.wp.com/www.bnosac.be/images/bnosac/blog/udpipe-spacy.png?w=456" data-recalc-dims="1"></a>Annotation languages</h3>
<ul>
<li><strong><span>udpipe</span></strong> provides annotation models for <strong>more than 50 languages</strong> (afrikaans, ancient_greek-proiel, ancient_greek, arabic, basque, belarusian, bulgarian, catalan, chinese, coptic, croatian, czech-cac, czech-cltt, czech, danish, dutch-lassysmall, dutch, english-lines, english-partut, english, estonian, finnish-ftb, finnish, french-partut, french-sequoia, french, galician-treegal, galician, german, gothic, greek, hebrew, hindi, hungarian, indonesian, irish, italian, japanese, kazakh, korean, latin-ittb, latin-proiel, latin, latvian, lithuanian, norwegian-bokmaal, norwegian-nynorsk, old_church_slavonic, persian, polish, portuguese-br, portuguese, romanian, russian-syntagrus, russian, sanskrit, serbian, slovak, slovenian-sst, slovenian, spanish-ancora, spanish, swedish-lines, swedish, tamil, turkish, ukrainian, urdu, uyghur, vietnamese) of which 17 languages are released under a commercially more liberal license, the others are released under the CC-BY-SA-NC licence
<ul>
<li>With the udpipe R package, you can furthermore directly train models on top of the newest CONLLU files available at <a rel="nofollow" href="http://universaldependencies.org/" target="_blank">http://universaldependencies.org</a> (examples of this is provided at <a rel="nofollow" href="https://github.com/bnosac/udpipe.models.ud" target="_blank">https://github.com/bnosac/udpipe.models.ud</a>)</li>
</ul>
</li>
<li><span><strong>spaCy</strong> </span>provides currently models for <strong>8 languages</strong>: English/German/Spanish/Portugues/French/Italian/Dutch.
<ul>
<li>For English and German these were trained on data which is not available on <a rel="nofollow" href="http://universaldependencies.org/" target="_blank">http://universaldependencies.org</a>, for the other models they were trained on data from <a rel="nofollow" href="http://universaldependencies.org/" target="_blank">http://universaldependencies.org</a></li>
<li>In order to train your own models you need to do this directly in Python, the Python community is building these since the end of 2017.</li>
</ul>
</li>
</ul>
<h3>Ease of use</h3>
<ul>
<li>Both packages are on CRAN</li>
<li><strong>Models can be easily downloaded with both packages</strong>. For udpipe this is directly from R, for spacy this needs to be done in Python.</li>
<li><strong>udpipe has no external dependencies and can easily be installed</strong> with install.packages(‘udpipe’) and next you are ready to go</li>
<li><strong>installation of spacyr will probably give you some trouble</strong> namely it
<ul>
<li>requires installation of the Python package spacy which is more difficult to install on a local computer in your corporate office. You can easily get stuck in problems of Python versioning, 32 vs 64 bit architecture issues, admin rights or basic shell commands that you should be aware of</li>
<li>does not allow to switch seamlessly between 2 languages (you need to initialize and finalize) which is a burden if you live in a multi-language country like e.g. Belgium</li>
<li>spaCy models are constructed on different treebanks each following different guidelines which make cross-language downstream analysis more difficult to harmonise</li>
</ul>
</li>
</ul>
<h3><strong>Annotation accuracy of the models</strong></h3>
<p>As the spaCy and UDPipe models for Spanish, Portuguese, French, Italian and Dutch have been built on data from the same Universal Dependencies treebank (version 2.0) one can compare the accuracies of the different NLP processing steps (tokenisation, POS tagging, morphological feature tagging, lemmatisation, dependency parsing).<br>Evaluation is traditionally done by leaving out some sentences from the training part and seeing how good the model did on these hold-out sentences which were tagged by humans that’s why they are called ‘gold’.<br>Below you can find accuracy statistics for the different NLP tasks by using the conllu 2017 shared task evaluation script on the holdout test sets. These graphs basically show that</p>
<ul>
<li><strong>UDPipe provides better results</strong> for French, Italian &amp; Portuguese, equal results for Spanish and less good results for dependency parsing and treebank-specific tags for Dutch but better results for the universal parts of speech tags.</li>
<li>For English, only the Penn Treebank XPOS tag can be compared and spaCy shows less good results than what we expected when comparing to the UDPipe model</li>
</ul>
<h4><a rel="nofollow" href="https://github.com/bnosac/GAlogger" target="_blank"><img alt="results alignedaccuracy2" src="https://i1.wp.com/www.bnosac.be/images/bnosac/blog/results-alignedaccuracy1.png?w=456" data-recalc-dims="1"></a></h4>
<h4><a rel="nofollow" href="https://github.com/bnosac/GAlogger" target="_blank"><img alt="results alignedaccuracy2" src="https://i1.wp.com/www.bnosac.be/images/bnosac/blog/results-alignedaccuracy1.png?w=456" data-recalc-dims="1"></a></h4>
<p>Exact reproducible details on the evaluation can be found at <a rel="nofollow" href="https://github.com/jwijffels/udpipe-spacy-comparison." target="_blank">https://github.com/jwijffels/udpipe-spacy-comparison.</a> Feel free to provide comments there.</p>
<h3><strong>Annotation possibilities</strong></h3>
<ul>
<li>udpipe
<ul>
<li>allows to do tokenisation, parts of speech tagging, morphological feature tagging and dependency parsing</li>
<li>udpipe does not provide entity recognition and does not provide word vectors (you can use existing R packages for that (e.g <a rel="nofollow" href="http://text2vec.org/" target="_blank">text2vec</a>)</li>
</ul>
</li>
<li>spacyr
<ul>
<li>spaCy allows to do tokenisation, parts of speech tagging, morphological feature tagging and dependency parsing</li>
<li>On top of that it also does entity recognition</li>
<li>spacyr does not provide lemmatisation</li>
<li>spaCy also provides wordvectors (for English only) but they are not made available in spacyr</li>
</ul>
<p><strong>So if you need entity recognition, udpipe is not an option. If you need lemmatisation, spacyr is not an option.</strong></p>
</li>
</ul>
<h3><strong>Annotation speed</strong></h3>
<ul>
<li><strong>spacyr is in our experiments 5 times faster than udpipe</strong> for a comparable full annotation pipeline (Tokenisation, POS tagging, Lemmatisation, Feature tagging, Dependency Parsing) and comparable output (see code below)</li>
</ul>
<pre><samp>library(udpipe)</samp><br><samp>library(spacyr)</samp><br><samp>library(microbenchmark)</samp><br><samp>data(brussels_reviews, package = &quot;udpipe&quot;)<br>f_udpipe &lt;-function(x, model){</samp><br><samp>  x_anno &lt;-udpipe_annotate(model, x = x)</samp><br><samp>  x_anno &lt;-as.data.frame(x_anno)</samp><br><samp>  invisible()</samp><br><samp>}</samp><br><samp>f_spacy &lt;-function(x){</samp><br><samp>  x_anno &lt;-spacy_parse(x, pos = TRUE, tag = TRUE, lemma = TRUE, entity = FALSE, dependency = TRUE)</samp><br><samp>  invisible()</samp><br><samp>}<br>## Dutch</samp><br><samp>x &lt;-subset(brussels_reviews, language == &quot;nl&quot;)</samp><br><samp>x &lt;-x$feedback<br>ud_model &lt;-udpipe_download_model(language = &quot;dutch&quot;)</samp><br><samp>ud_model &lt;-udpipe_load_model(ud_model$file)</samp><br><samp>spacy_initialize(model = &quot;nl&quot;, python_executable = &quot;C:/Users/Jan/Anaconda3/python.exe&quot;)</samp><br><samp>microbenchmark(</samp><br><samp>  f_udpipe(x, model = ud_model),</samp><br><samp>  f_spacy(x),</samp><br><samp>  times = 2)</samp><br><samp>spacy_finalize()</samp></pre>
<h3>Enjoy</h3>
<p>Hope this provides you some guidance when you are thinking about extending your nlp workflow with more deeper natural language processing than merely sentiment analysis.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.bnosac.be/index.php/blog/75-a-comparison-between-spacy-and-udpipe-for-natural-language-processing-for-r-users"> bnosac :: open analytical helpers</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/iT9btuZ6T0w" height="1"></a></div>