<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/rdocumentation-scoring-and-ranking/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.datacamp.com/community/blog/rdocumentation-ranking-scoring"> DataCamp Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>One of the core features of <a target="_blank" rel="nofollow" href="https://www.rdocumentation.org/">RDocumentation.org</a> is its search functionality. From the start, we wanted to have a super simple search bar that finds what you are looking for, without a complex form asking for a package name, function name, versions or anything else. Just a simple search bar.</p>
<p>In today’s <a target="_blank" rel="nofollow" href="http://tech.datacamp.com/">technical blog</a> post, we will highlight the technologies and techniques that we use to provide relevant and meaningful results to our users.</p>
<h2>Elasticsearch</h2>
<p>RDocumentation.org uses <a target="_blank" rel="nofollow" href="https://www.elastic.co/products/elasticsearch">Elasticsearch</a> to index and search through all R packages and topics.</p>
<blockquote>
<p>Elasticsearch is an open-source, scalable, distributed, enterprise-grade search engine.</p>
</blockquote>
<p>Elasticsearch is perfect for querying documentation because it doesn’t use conventional SQL data, but it stores documents in a JSON-like data structure instead. Each document is just a set of <em>key-value</em> pairs with simple data types (strings, numbers, lists, dates, …). Its distributed nature means Elasticsearch can be extremely fast.</p>
<p>An Elasticsearch cluster can have multiple indexes and each index can have multiple <em>document types</em>. A document type just describes what the structure of the document should look like. To learn more about Elasticsearch types, you can visit <a target="_blank" rel="nofollow" href="https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping.html">the guide on elastic.co</a>.</p>
<p>RDocumentation.org uses three different types: <code>package_version</code>, <code>topic</code>, and <code>package</code>. The first 2 are the main ones; let’s discuss <code>package</code> later.</p>
<p>Because RDocumentation.org is open-source, you can see the Elasticsearch’s mappings in our <a target="_blank" rel="nofollow" href="https://github.com/datacamp/RDocumentation-elasticsearch/blob/master/mappings_rdoc.json">github repo</a>. </p>
<h3>package_version type</h3>
<p>The <code>package_version</code> type is like a translation of the <code>DESCRIPTION</code> file of a package, it features the main field that one can find in there; <code>package_name</code>, <code>version</code>, <code>title</code>, <code>description</code>, <code>release_date</code>, <code>license</code>, <code>url</code>, <code>copyright</code>, <code>created_at</code>, <code>updated_at</code>, <code>latest_version</code>, <code>maintainer</code> and <code>collaborators</code>. The <code>maintainer</code> and <code>collaborators</code> are extracted from the <code>Authors</code> field in the <code>DESCRIPTION</code> file</p>
<h3>topic type</h3>
<p>The topic documents are parsed from the <code>Rd</code> files, the standard format of documentation in R. The <code>topic</code> type has the following keys: <code>name</code>, <code>title</code>, <code>description</code>, <code>usage</code>, <code>details</code>, <code>value</code>, <code>references</code>, <code>note</code>, <code>author</code>, <code>seealso</code>, <code>examples</code>, <code>created_at</code>, <code>updated_at</code>, <code>sections</code>, <code>aliases</code> and <code>keywords</code>.</p>
<h2>Scoring in Elasticsearch</h2>
<p>Before doing any scoring, Elasticseach first tries to reduce the set of candidates by checking if the document actually matches the query. Basically, a query is a word (or a set of words). Based on the query setting, Elasticsearch searches for a match in certain fields of certain types.</p>
<p>However, a match does not necessarily mean that the document is relevant; the same word can have different meanings in different contexts. Based on the query settings, we can filter by type and field, and include more contextual information. This contextual information will improve the relevancy and this is where scoring comes into place.</p>
<p>Elasticsearch uses <a target="_blank" rel="nofollow" href="http://lucene.apache.org/core/">Lucene</a> under the hood, so the scoring is based on Lucene’s <a target="_blank" rel="nofollow" href="https://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html#formula_coord">Practical Scoring Function</a> which brings together some models like the <a target="_blank" rel="nofollow" href="https://www.elastic.co/guide/en/elasticsearch/guide/current/scoring-theory.html#tfidf"><em>TF-IDF</em></a>, <a target="_blank" rel="nofollow" href="https://www.elastic.co/guide/en/elasticsearch/guide/current/scoring-theory.html#vector-space-model"><em>Vector Space Model</em></a> and <a target="_blank" rel="nofollow" href="https://www.elastic.co/guide/en/elasticsearch/guide/current/scoring-theory.html#boolean-model"><em>Boolean Model</em></a> to score the document.</p>
<p>If you want to lean more about how that function is used in Elasticsearch, you can check out <a target="_blank" rel="nofollow" href="https://www.elastic.co/guide/en/elasticsearch/guide/current/practical-scoring-function.html">this section of elastic.co guide</a>.</p>
<p>One way to improve relevancy is to apply a boost to some fields. For example, in the RDocumentation.org full search, we naturally boost fields like <code>package_name</code> and <code>title</code> for packages and <code>aliases</code> and <code>name</code> for topics.</p>
<h3>Boosting the popular documents</h3>
<p>Another effective way to improve relevancy is to boost documents based on their popularity. The idea behind that is that if a package is more popular, the user is more likely to search for this package. Showing the more populars packages first will increase the probability that we show what the user is actually looking for.</p>
<h4>Using downloads as a popularity measure</h4>
<p>There are multiple ways to measure popularity. We could use direct measures like votes or rankings that users give (like ratings on Amazon products), or indirect measures like the number of items sold or the number of views (for YouTube videos).</p>
<p>At RDocumentation.org, we chose the latter. More specifically, we use the number of downloads as a measure of popularity. Indirect measures are typically easier to collect because they don’t require active user input.</p>
<h4>Timeframing</h4>
<p>One problem that arises when using the number of downloads is that old packages will naturally have more total downloads than newer packages. That does not mean that they are more popular, however, they have just been around longer. What if a package was very popular years ago, but has now become obsolete and is no longer being actively used by the community?</p>
<p>To solve this problem, we only take into account the number of downloads in the last month. That way, older packages’ popularity score is not artificially boosted, and obsolete packages will quickly fade out.</p>
<h4>Direct vs Indirect downloads</h4>
<p>Another problem arises from reverse dependencies. R packages typically depend on a wide range of other packages. Packages with a lot of reverse dependencies will get downloaded way more than others. However, these packages are typically more low-level and are not used directly by the end-user. We have to watch out to not give their number of downloads too much weight.</p>
<p>As an example, take <a target="_blank" rel="nofollow" href="https://www.rdocumentation.org/packages/Rcpp/versions/0.12.9">Rcpp</a>. Over 70 percent of all packages on CRAN, the comprehensive R archive network, depend on this package, which obviously makes it the most downloaded R package. However, rather few R users will directly use this package and search for its documentation.</p>
<p>To solve this problem, we needed to separate <em>direct downloads</em> (downloads that happens because a user requested it) and <em>indirect downloads</em> (downloads that happen because a dependent packages was downloaded). To distinguish the direct and indirect downloads from the CRAN logs, we use the same heuristic described in the <a target="_blank" rel="nofollow" href="https://www.rdocumentation.org/packages/cran.stats/versions/0.1/topics/stats_logs">cran.stats package</a> by Arun Srinivasan.</p>
<p>We now have a meaningful popularity metric: <em>the number of direct downloads in the last month</em>. Elasticsearch provides an easy way to inject this additional information; for more details, check out <a target="_blank" rel="nofollow" href="https://www.elastic.co/guide/en/elasticsearch/guide/current/boosting-by-popularity.html">this article on elastic.co</a>.</p>
<p>The score is modified as follows:</p>
<div>
<pre>
<code>new_score = old_score * log(1 + number of direct downloads in the last month)
</code></pre>
</div>
<p>We use a <code>log()</code> function to smooth out the <code>number of downloads</code> value, because each subsequent downloads has less weight; the difference between 0 and 1000 downloads should have a bigger impact on a popularity score than the difference between 100.000 and 101.000 downloads.</p>
<p>This re-scoring improves the overall relevancy of the search results presented by RDocumentation.org and as a result, users can focus on reading documentation instead of searching for it.</p>
<p>If you want to find out more about how exactly the Elasticsearch query is implemented, you can take a look at the <a target="_blank" rel="nofollow" href="https://github.com/datacamp/RDocumentation-app">RDocumentation project on GitHub</a>. The query itself is located in the <a target="_blank" rel="nofollow" href="https://github.com/datacamp/RDocumentation-app/blob/master/api/controllers/SearchController.js"><code>SearchController</code></a>.</p>
<p>If you want to learn more about how RDocumentation.org is implemented, check out our repos:</p>
<ul>
<li>
<a target="_blank" rel="nofollow" href="https://github.com/datacamp/RDocumentation-app">RDocumentation-app</a>: The web application running rdocumentation.org.</li>
<li>
<a target="_blank" rel="nofollow" href="https://github.com/datacamp/RDocumentation-elasticsearch">RDocumentation-elasticsearch</a>: Configuration and feeders of the Elasticsearch server serving rdocumentation.org.</li>
<li>
<a target="_blank" rel="nofollow" href="https://github.com/datacamp/RDocumentation">RDocumentation</a>: R package to integrate rdocumentation.org into your R workflow</li>
<li>
<a target="_blank" rel="nofollow" href="https://github.com/datacamp/RDocumentation-lambda-worker">RDocumentation-lambda-worker</a>: AWS Lambda pipeline to parse package documentation for rdocumentation.org</li>
</ul>
<hr>
<p><strong>About RDocumentation  </strong></p>
<p><a target="_blank" rel="nofollow" href="http://www.rdocumentation.org/">RDocumentation</a> aggregates help documentation for R packages from CRAN, BioConductor, and GitHub – the three most common sources of current R documentation. RDocumentation.org goes beyond simply aggregating this information, however, by bringing all of this documentation to your fingertips via the RDocumentation package. The RDocumentation package overwrites the basic help functions from the utils package and gives you access to RDocumentation.org from the comfort of your RStudio IDE. Look up the newest and most popular R packages, search through documentation and post community examples. </p>
<p><a target="_blank" rel="nofollow" href="https://www.rdocumentation.org/"><img alt="" src="https://i0.wp.com/community.datacamp.com.s3.amazonaws.com/community/production/ckeditor_assets/pictures/140/content_b757fa44-7112-11e6-99e0-f20e8f3f93ff.png?w=456" data-recalc-dims="1"></a></p>
<p><center><a target="_blank" rel="nofollow" href="https://www.rdocumentation.org/register"><strong>Create an RDocumentation account today!</strong></a></center></p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.datacamp.com/community/blog/rdocumentation-ranking-scoring"> DataCamp Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/buu_vT6szaE"></a></div>