<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/cran-search-based-on-natural-language-processing/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.bnosac.be/index.php/blog/70-cran-search-based-on-natural-language-processing"> bnosac :: open analytical helpers</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>CRAN contains up to date (October 2017) more than 11500 R packages. If you want to scroll through all of these, you probably need to spend a few days, assuming you need 5 seconds per package and there are 8 hours in a day.</p>
<p>Since R version 3.4, we can also get a dataset will all packages, their dependencies, the package title, the description and even the installation errors which the packages have. Which makes the CRAN database with all packages an excellent dataset for doing text mining. If you want to get that dataset, just do as follows in R:</p>
<p><code>library(tools)<br>crandb &lt;- CRAN_package_db()</code></p>
<p>Based on that data the following <strong>CRAN NLP searcher</strong> app was built as shown below. I’ts available for inspection at <a rel="nofollow" href="http://datatailor.be:9999/app/cran_search" target="_blank"></a><a>http://datatailor.be:9999/app/cran_search </a>and is a tiny wrapper around the result of annotating the package title and package description using the udpipe R package: <a rel="nofollow" href="https://github.com/bnosac/udpipe" target="_blank">https://github.com/bnosac/udpipe</a></p>
<p><img alt="udpipe rlogo" src="https://i1.wp.com/www.bnosac.be/images/bnosac/blog/cran-search-corr.png?w=450" data-recalc-dims="1"></p>
<p>If you want to easily extract what is written in text without reading it, a common way is to do Parts of Speech tagging, extract the nouns and/or the verbs and then plot all co-occurrences / correlations and frequencies of the lemmata. The updipe package allows you exactly to do that. Annotating using Parts of Speech tagging, is pretty easy with udpipe_annotate function from the the udpipe R package (<a rel="nofollow" href="https://github.com/bnosac/udpipe" target="_blank">https://github.com/bnosac/udpipe</a>). Mark that this takes a while (as in +/- 30 minutes) and is probably something you want to run as a web-service or integrated stored procedure.</p>
<p><code>library(udpipe)<br>ud_model &lt;- udpipe_download_model(language = &quot;english&quot;)<br>ud_model &lt;- udpipe_load_model(ud_model$file_model)<br>crandb_annotated &lt;- udpipe_annotate(ud_model,<br>                                    x = paste(crandb$Title, crandb$Description, sep = &quot; \n &quot;),<br>                                    doc_id = crandb$Package)<br>crandb_annotated &lt;- as.data.frame(crandb_annotated)</code></p>
<p>Once we have that data annotated, making a web application which allows you to visualise, structure and display the CRAN packages content is pretty easy with tools like <a rel="nofollow" href="http://rmarkdown.rstudio.com/flexdashboard/" target="_blank">flexdashboard</a>. That’s exactly what this web application available at <a rel="nofollow" href="http://datatailor.be:9999/app/cran_search" target="_blank">http://datatailor.be:9999/app/cran_search</a> does. The application allows you</p>
<p><img alt="cran search cluster" src="https://i1.wp.com/www.bnosac.be/images/bnosac/blog/cran-search-cluster.png?resize=350%2C249" data-recalc-dims="1"></p>
<ul>
<li>List all packages which are part of a CRAN Task View</li>
<li>To search for CRAN packages based on what the author has written in the <span>package title and description</span></li>
<li>Based on the found CRAN packages which were searched for: <span>Visualise</span> the nouns and verbs in the package title and descriptions by using
<ul>
<li><strong>Word-coocurrence graphs</strong> indicating how many times each lemma occurs in the same package as another lemma</li>
<li><strong>Word-correlation graphs</strong> showing the positive correlations between the top n most occurring lemma’s in the packages</li>
<li><strong>Word clouds</strong> indicating the frequency of nouns/verbs or consecutive nouns/verbs (bigrams) in the package descriptions</li>
<li>Build a <strong>topic model</strong> (latent dirichlet allocation) to cluster packages and <strong>visualise</strong> them</li>
</ul>
</li>
</ul>
<p>The web application (flexdashboard) was launched on a small <a rel="nofollow" href="https://www.shinyproxy.io/" target="_blank">shinyproxy </a>server and is available here: <a rel="nofollow" href="http://datatailor.be:9999/app/cran_search" target="_blank">http://datatailor.be:9999/app/cran_search</a>. Can you find topics which are not yet covered by the CRAN task views yet? Can you find the content of the Rcpp-universe or the sp package universe?</p>
<p>If you are interested in these techniques, you can always subscribe for our text mining with R course at the following dates:</p>
<ul>
<li><strong>08+10/11/2017: Text mining with R. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/text-mining-with-r" target="_blank">here</a></strong></li>
<li><strong>27-28/11/2017: Text mining with R. Brussels (Belgium). <a rel="nofollow" href="http://di-academy.com/bootcamp/" target="_blank">http://di-academy.com/bootcamp</a> + send mail to <span><a>training@di-academy.com</a></span></strong></li>
<li><span><a rel="nofollow" href="mailto:training@di-academy.com" target="_blank"></a></span>19-20/12/2017: Applied spatial modelling with R. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/applied-spatial-modelling-with-r" target="_blank">here</a></li>
<li>20-21/02/2018: Advanced R programming. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/AdvancedprogramminginR.html" target="_blank">here</a></li>
<li>08-09/03/2018: Computer Vision with R and Python. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/ComputervisionwithRandPython" target="_blank">here</a></li>
<li><strong>22-23/03/2018: Text Mining with R. Leuven (Belgium). Subscribe <a rel="nofollow" href="https://lstat.kuleuven.be/training/coursedescriptions/text-mining-with-r" target="_blank">here</a></strong></li>
</ul>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.bnosac.be/index.php/blog/70-cran-search-based-on-natural-language-processing"> bnosac :: open analytical helpers</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/bNFiLeGwxO8" height="1"></a></div>