<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/distributional-semantics-in-r-part-2-entity-recognition-w-opennlp/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.exactness.net/post/155292474180"> The Exactness of Mind</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>The R code for this tutorial on <b>Methods of Distributional Semantics in R</b> is found in the respective<a target="_blank" rel="nofollow" href="http://t.umblr.com/redirect?z=https%3A%2F%2Fgithub.com%2FGoranMilovanovic%2FDistributional-Semantics-in-R&t=MGU1N2Y0YzE5OGEwNWVmNzE5ZWI4MjZiOWZhNzBkYjQxYmUyYjY4Mix0R0xheVNjVQ%3D%3D&b=t%3AfxfLHZZpYe9t9Oek1n3iNg&p=http%3A%2F%2Fwww.exactness.net%2Fpost%2F154910180365%2Fdistributional-semantics-in-r-part-1-tm-classes&m=1"> GitHub repository</a>. You will find <b>.R</b>, <b>.Rmd</b>, and <b>.html</b> files corresponding to each part of this tutorial (e.g. <i>DistSemanticsBelgradeR-Part2.R</i>, <i>DistSemanticsBelgradeR-Part2.Rmd</i>, and <i>DistSemanticsBelgradeR-Part2.html</i>, for Part 2) there. All auxiliary files are also uploaded to the repository. </p>
<p>Following my<a target="_blank" rel="nofollow" href="http://www.exactness.net/post/153902588705/belgrader-meetup-methods-of-distributional"> Methods of Distributional Semantics in R BelgradeR Meetup</a> with<a target="_blank" rel="nofollow" href="http://t.umblr.com/redirect?z=http%3A%2F%2Fwww.datascience.rs%2F&t=NzlkYjliNTE5OGYyMzAxYWUyYzg1YTYyZDA1MjkxMmI2MGMwMTQ5ZSx0R0xheVNjVQ%3D%3D&b=t%3AfxfLHZZpYe9t9Oek1n3iNg&p=http%3A%2F%2Fwww.exactness.net%2Fpost%2F154910180365%2Fdistributional-semantics-in-r-part-1-tm-classes&m=1"> Data Science Serbia</a>, organized in <a target="_blank" rel="nofollow" href="http://t.umblr.com/redirect?z=http%3A%2F%2Fen.startit.rs%2F&t=YTMwMmQxZDlhMDQ4YWVhODRjMDA4NGJmZmQzNDU3MmU4NjQ1OGY3Nix0R0xheVNjVQ%3D%3D&b=t%3AfxfLHZZpYe9t9Oek1n3iNg&p=http%3A%2F%2Fwww.exactness.net%2Fpost%2F154910180365%2Fdistributional-semantics-in-r-part-1-tm-classes&m=1">Startit Center</a>, Belgrade,<br>
11/30/2016, several people asked me for the R code used for the analysis<br>
of William Shakespeare’s plays that was presented. I have decided to<br>
continue the development of the code that I’ve used during the Meetup in order to advance the examples<br>
that I have shown then into a more or less complete and comprehensible<br>
text-mining tutorial with<a target="_blank" rel="nofollow" href="http://t.umblr.com/redirect?z=https%3A%2F%2Fcran.r-project.org%2Fweb%2Fpackages%2Ftm%2Findex.html&t=NmI4MmIyYzQxNWRlNDRhMDJmNjkyN2Y5NjBiYjNkMWY5N2NmMzY2OSx0R0xheVNjVQ%3D%3D&b=t%3AfxfLHZZpYe9t9Oek1n3iNg&p=http%3A%2F%2Fwww.exactness.net%2Fpost%2F154910180365%2Fdistributional-semantics-in-r-part-1-tm-classes&m=1"> {tm}</a>,<a target="_blank" rel="nofollow" href="http://t.umblr.com/redirect?z=https%3A%2F%2Fcran.r-project.org%2Fweb%2Fpackages%2FopenNLP%2Findex.html&t=Y2VlMDhjZTQzM2Y2Mzk4ZDg3MzNlYWQ1NTlhNTg1YTdlNzIxMTBmNyx0R0xheVNjVQ%3D%3D&b=t%3AfxfLHZZpYe9t9Oek1n3iNg&p=http%3A%2F%2Fwww.exactness.net%2Fpost%2F154910180365%2Fdistributional-semantics-in-r-part-1-tm-classes&m=1"> {openNLP}</a>, and<a target="_blank" rel="nofollow" href="http://t.umblr.com/redirect?z=https%3A%2F%2Fcran.r-project.org%2Fweb%2Fpackages%2Ftopicmodels%2Findex.html&t=MGE3YWViYzhjNjY0ZWU3MjQzM2ExOTNkNjExNzdiNDA4ZjMxMzNiOSx0R0xheVNjVQ%3D%3D&b=t%3AfxfLHZZpYe9t9Oek1n3iNg&p=http%3A%2F%2Fwww.exactness.net%2Fpost%2F154910180365%2Fdistributional-semantics-in-r-part-1-tm-classes&m=1"> {topicmodels}</a> in R. All<br>
files in this <a target="_blank" rel="nofollow" href="http://t.umblr.com/redirect?z=https%3A%2F%2Fgithub.com%2FGoranMilovanovic%2FDistributional-Semantics-in-R&t=MGU1N2Y0YzE5OGEwNWVmNzE5ZWI4MjZiOWZhNzBkYjQxYmUyYjY4Mix0R0xheVNjVQ%3D%3D&b=t%3AfxfLHZZpYe9t9Oek1n3iNg&p=http%3A%2F%2Fwww.exactness.net%2Fpost%2F154910180365%2Fdistributional-semantics-in-r-part-1-tm-classes&m=1">GitHub</a> repository are a product of that work. </p>
<p>Part 2 will introduce named entity recognition with {openNLP}, and Apache project in Java interfaced by this nice R package that, in turn, relies on {NLP} classes. We will try to make machine learning (MaxEnt models offered in {openNLP} figure out the characters from Shakespeare’s plays, a quite difficult task given that the learning algorithms at our disposal were trained on contemporary English corpora.</p>
<figure data-orig-height="960" data-orig-width="1344"><img src="https://i2.wp.com/68.media.tumblr.com/7ef6b5869375197b8086e9ac8c29f177/tumblr_inline_oj5h0lDqOx1qa0hyw_540.png?resize=456%2C326" data-orig-height="960" data-recalc-dims="1" data-orig-width="1344"></figure>
<p><i>The accuracy of character recognition from Shakespeare’s comedies, tragedies, and histories; the black dashed line is the overall density. The results is not realistic (explanation given in the respective .Rmd and .hmtl files).</i></p>
<p>What I really want to show you here is how tricky and difficult it can be to do serious text-mining, and help you by exemplifying some steps that are necessary to ensure the consistency of results that you are expecting. The text-mining pipelines being developed here <i>are in no sense perfect or complete</i>; they are meant to demonstrate important problems and propose solutions rather than to provide a copy and paste ready chunks for future re-use. In essence, except in those cases where a standardized information extraction + text-mining pipeline is being developed (a situation where, by assumption, one periodically processes large text corpora, e.g. web-scraped news and other media reports, from various sources, in various formats, and where one simply needs to learn to live with approximations) every text-mining study will need a specific pipeline on its own. Chaining those <i>tm_map()</i> calls to various <i>content_transformers</i> from {tm} restlessly, while being ignorant of the necessary changes in parameters and different content-specific transformations – of which {tm} supports only a few – will simply not do.</p>
<figure data-orig-height="960" data-orig-width="1344"><img src="https://i1.wp.com/68.media.tumblr.com/bbfab6ea2861595c976e485c77196b21/tumblr_inline_oj5h5yb5yz1qa0hyw_540.png?resize=456%2C326" data-orig-height="960" data-recalc-dims="1" data-orig-width="1344"></figure>
<p>Don’t get hooked on the results presented in the {ggplot2} figure above; {openNLP} is not that successful in recognizing personal names from Shakespeare’s plays (in spite of the fact that it works great for contemporary English documents). I have helped it a bit, by doing something that is not applicable to real-world situations; go take a look at the code from <a target="_blank" rel="nofollow" href="https://github.com/GoranMilovanovic/Distributional-Semantics-in-R">this GitHub repository.</a></p>
<p>See you soon.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.exactness.net/post/155292474180"> The Exactness of Mind</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/cFUwA7sCh_Q"></a></div>