<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/elasticsearchr-a-lightweight-elasticsearch-client-for-r/"></div></div><div><a data-url="https://www.r-bloggers.com/elasticsearchr-a-lightweight-elasticsearch-client-for-r/" data-counturl="https://www.r-bloggers.com/elasticsearchr-a-lightweight-elasticsearch-client-for-r/" data-count="horizontal" data-text="elasticsearchr – a Lightweight Elasticsearch Client for R" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://alexioannides.com/2016/11/28/elasticsearchr-a-lightweight-elasticsearch-client-for-r/"> R – When Localhost Isn't Enough</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img title="Elasticsearchr" alt="" src="https://i1.wp.com/alexioannides.files.wordpress.com/2016/11/elasticsearchr2.png?w=456&ssl=1" data-recalc-dims="1"></p>
<h1>elasticsearchr: a Lightweight Elasticsearch Client for R</h1>
<p><a title="Elasticsearch" target="_blank" rel="nofollow" href="https://www.elastic.co/products/elasticsearch">Elasticsearch</a> is a distributed <a title="What is NoSQL?" target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/NoSQL">NoSQL</a> document store search-engine and <a title="Elasticsearch as a Column Store" target="_blank" rel="nofollow" href="https://www.elastic.co/blog/elasticsearch-as-a-column-store">column-oriented database</a>, whose <strong>fast</strong> (near real-time) reads and powerful aggregation engine make it an excellent choice as an ‘analytics database’ for R&amp;D, production-use or both. Installation is simple, it ships with sensible default settings that allow it to work effectively out-of-the-box, and all interaction is made via a set of intuitive and extremely <a title="Elasticsearch documentation" target="_blank" rel="nofollow" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html">well documented</a> <a title="RESTful?" target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Representational_state_transfer">RESTful</a> APIs. I’ve been using it for two years now and I am evangelical.</p>
<p>The <code>elasticsearchr</code> package implements a simple Domain-Specific Language (DSL) for indexing, deleting, querying, sorting and aggregating data in Elasticsearch, from within R. The main purpose of this package is to remove the labour involved with assembling HTTP requests to Elasticsearch’s REST APIs and processing the responses. Instead, users of this package need only send and receive data frames to Elasticsearch resources. Users needing richer functionality are encouraged to investigate the excellent <code>elastic</code> package from the good people at <a title="rOpenSci" target="_blank" rel="nofollow" href="https://github.com/ropensci/elastic">rOpenSci</a>.</p>
<p><span></span></p>
<p>This package is available on <a title="elasticsearchr on CRAN" target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/elasticsearchr/">CRAN</a> or from <a title="Alex's GitHub repository" target="_blank" rel="nofollow" href="https://github.com/AlexIoannides/elasticsearchr">this GitHub repository</a>. To install the latest development version from GitHub, make sure that you have the <code>devtools</code> package installed (this comes bundled with RStudio), and then execute the following on the R command line:</p>
<pre>
devtools::install_github(&quot;alexioannides/elasticsearchr&quot;)
</pre>
<h2>Installing Elasticsearch</h2>
<p>Elasticsearch can be downloaded <a title="Download" target="_blank" rel="nofollow" href="https://www.elastic.co/downloads/elasticsearch">here</a>, where the instructions for installing and starting it can also be found. OS X users (such as myself) can also make use of <a title="Homebrew for OS X" target="_blank" rel="nofollow" href="http://brew.sh/">Homebrew</a> to install it with the command,</p>
<pre>
$ brew install elasticsearch
</pre>
<p>And then start it by executing <code>$ elasticsearch</code> from within any Terminal window. Successful installation and start-up can be checked by navigating any web browser to <code><a target="_blank" rel="nofollow" href="http://localhost:9200/">http://localhost:9200</a></code>, where the following message should greet you (give or take the cluster name that changes with every restart),</p>
<pre>
{
  &quot;name&quot; : &quot;Kraven the Hunter&quot;,
  &quot;cluster_name&quot; : &quot;elasticsearch&quot;,
  &quot;version&quot; : {
    &quot;number&quot; : &quot;2.3.5&quot;,
    &quot;build_hash&quot; : &quot;90f439ff60a3c0f497f91663701e64ccd01edbb4&quot;,
    &quot;build_timestamp&quot; : &quot;2016-07-27T10:36:52Z&quot;,
    &quot;build_snapshot&quot; : false,
    &quot;lucene_version&quot; : &quot;5.5.0&quot;
  },
  &quot;tagline&quot; : &quot;You Know, for Search&quot;
}
</pre>
<h2>Elasticsearch 101</h2>
<p>If you followed the installation steps above, you have just installed a single Elasticsearch ‘node’. When <strong>not</strong> testing on your laptop, Elasticsearch usually comes in clusters of nodes (usually there are at least 3). The easiest easy way to get access to a managed Elasticsearch cluster is by using the <a title="Elastic Cloud" target="_blank" rel="nofollow" href="https://www.elastic.co/cloud/as-a-service">Elastic Cloud</a> managed service provided by <a title="Elastic corp." target="_blank" rel="nofollow" href="https://www.elastic.co/">Elastic</a> (note that Amazon Web Services offer something similar too). For the rest of this brief tutorial I will assuming you’re running a single node on your laptop (a great way of working with data that is too big for memory).</p>
<p>In Elasticsearch a ‘row’ of data is stored as a ‘document’. A document is a <a title="JSON" target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/JSON">JSON</a> object – for example, the first row of R’s <code>iris</code> dataset,</p>
<pre>
#   sepal_length sepal_width petal_length petal_width species
# 1          5.1         3.5          1.4         0.2  setosa
</pre>
<p>would be represented as follows using JSON,</p>
<pre>
{
  &quot;sepal_length&quot;: 5.1,
  &quot;sepal_width&quot;: 3.5,
  &quot;petal_length&quot;: 1.4,
  &quot;petal_width&quot;: 0.2,
  &quot;species&quot;: &quot;setosa&quot;
}
</pre>
<p>Documents are classified into ‘types’ and stored in an ‘index’. In a crude analogy with traditional SQL databases that is often used, we would associate an index with a database instance and the document types as tables within that database. In practice this example is not accurate – it is better to think of all documents as residing in a single – possibly sparse – table (defined by the index), where the document types represent non-unique sub-sets of columns in the table. This is especially so as fields that occur in multiple document types (within the same index), must have the same data-type – for example, if <code>&quot;name&quot;</code> exists in document type <code>customer</code> as well as in document type <code>address</code>, then <code>&quot;name&quot;</code> will need to be a <code>string</code> in both.</p>
<p>Each document is considered a ‘resource’ that has a Uniform Resource Locator (URL) associated with it. Elasticsearch URLs all have the following format: <code><a target="_blank" rel="nofollow" href="http://your_cluster:9200/your_index/your_doc_type/your_doc_id">http://your_cluster:9200/your_index/your_doc_type/your_doc_id</a></code>. For example, the above <code>iris</code> document could be living at <code><a target="_blank" rel="nofollow" href="http://localhost:9200/iris/data/1">http://localhost:9200/iris/data/1</a></code> – you could even point a web browser to this location and investigate the document’s contents.</p>
<p>Although Elasticsearch – like most NoSQL databases – is often referred to as being ‘schema free’, as we have already see this is not entirely correct. What is true, however, is that the schema – or ‘mapping’ as it’s called in Elasticsearch – does not <em>need</em> to be declared up-front (although you certainly can do this). Elasticsearch is more than capable of guessing the types of fields based on new data indexed for the first time.</p>
<p>For more information on any of these basic concepts take a look <a title="Basic Concepts" target="_blank" rel="nofollow" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/_basic_concepts.html">here</a></p>
<h2>elasticsearchr: a Quick Start</h2>
<p><code>elasticsearchr</code> is a <strong>lightweight</strong> client – by this I mean that it only aims to do ‘just enough’ work to make using Elasticsearch with R easy and intuitive. You will still need to read the <a title="Elasticsearch documentation" target="_blank" rel="nofollow" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html">Elasticsearch documentation</a> to understand how to compose queries and aggregations. What follows is a quick summary of what is possible.</p>
<h3>Elasticsearch Data Resources</h3>
<p>Elasticsearch resources, as defined by the URLs described above, are defined as <code>elastic</code> objects in <code>elasticsearchr</code>. For example,</p>
<pre>
es &lt;- elastic(&quot;http://localhost:9200&quot;, &quot;iris&quot;, &quot;data&quot;)
</pre>
<p>Refers to documents of type ‘data’ in the ‘iris’ index located on an Elasticsearch node on my laptop. Note that:<br>
– it is possible to leave the document type empty if you need to refer to all documents in an index; and,<br>
– <code>elastic</code> objects can be defined even if the underling resources have yet to be brought into existence.</p>
<h3>Indexing New Data</h3>
<p>To index (insert) data from a data frame, use the <code>%index%</code> operator as follows:</p>
<pre>
elastic(&quot;http://localhost:9200&quot;, &quot;iris&quot;, &quot;data&quot;) %index% iris
</pre>
<p>In this example, the <code>iris</code> dataset is indexed into the ‘iris’ index and given a document type called ‘data’. Note that I have not provided any document ids here. <strong>To explicitly specify document ids there must be a column in the data frame that is labelled <code>id</code></strong>, from which the document ids will be taken.</p>
<h3>Deleting Data</h3>
<p>Documents can be deleted in three different ways using the <code>%delete%</code> operator. Firstly, an entire index (including the mapping information) can be erased by referencing just the index in the resource – e.g.,</p>
<pre>
elastic(&quot;http://localhost:9200&quot;, &quot;iris&quot;) %delete% TRUE
</pre>
<p>Alternatively, documents can be deleted on a type-by-type basis leaving the index and it’s mappings untouched, by referencing both the index and the document type as the resource – e.g.,</p>
<pre>
elastic(&quot;http://localhost:9200&quot;, &quot;iris&quot;, &quot;data&quot;) %delete% TRUE
</pre>
<p>Finally, specific documents can be deleted by referencing their ids directly – e.g.,</p>
<pre>
elastic(&quot;http://localhost:9200&quot;, &quot;iris&quot;, &quot;data&quot;) %delete% c(&quot;1&quot;, &quot;2&quot;, &quot;3&quot;, &quot;4&quot;, &quot;5&quot;)
</pre>
<h3>Queries</h3>
<p>Any type of query that Elasticsearch makes available can be defined in a <code>query</code> object using the native Elasticsearch JSON syntax – e.g. to match every document we could use the <code>match_all</code> query,</p>
<pre>
for_everything &lt;- query('{
  &quot;match_all&quot;: {}
}')
</pre>
<p>To execute this query we use the <code>%search%</code> operator on the appropriate resource – e.g.,</p>
<pre>
elastic(&quot;http://localhost:9200&quot;, &quot;iris&quot;, &quot;data&quot;) %search% for_everything

#     sepal_length sepal_width petal_length petal_width    species
# 1            4.9         3.0          1.4         0.2     setosa
# 2            4.9         3.1          1.5         0.1     setosa
# 3            5.8         4.0          1.2         0.2     setosa
# 4            5.4         3.9          1.3         0.4     setosa
# 5            5.1         3.5          1.4         0.3     setosa
# 6            5.4         3.4          1.7         0.2     setosa
# ...
</pre>
<h3>Sorting Query Results</h3>
<p>Query results can be sorted on multiple fields by defining a <code>sort</code> object using the same Elasticsearch JSON syntax – e.g. to sort by <code>sepal_width</code> in ascending order the required <code>sort</code> object would be defined as,</p>
<pre>
by_sepal_width &lt;- sort('{&quot;sepal_width&quot;: {&quot;order&quot;: &quot;asc&quot;}}')
</pre>
<p>This is then added to a <code>query</code> object whose results we want sorted and executed using the <code>%search%</code> operator as before – e.g.,</p>
<pre>
elastic(&quot;http://localhost:9200&quot;, &quot;iris&quot;, &quot;data&quot;) %search% (for_everything + by_sepal_width)

#   sepal_length sepal_width petal_length petal_width    species
# 1          5.0         2.0          3.5         1.0 versicolor
# 2          6.0         2.2          5.0         1.5  virginica
# 3          6.0         2.2          4.0         1.0 versicolor
# 4          6.2         2.2          4.5         1.5 versicolor
# 5          4.5         2.3          1.3         0.3     setosa
# 6          6.3         2.3          4.4         1.3 versicolor
# ...
</pre>
<h3>Aggregations</h3>
<p>Similarly, any type of aggregation that Elasticsearch makes available can be defined in an <code>aggs</code> object – e.g. to compute the average <code>sepal_width</code> per-species of flower we would specify the following aggregation,</p>
<pre>
avg_sepal_width &lt;- aggs('{
  &quot;avg_sepal_width_per_species&quot;: {
    &quot;terms&quot;: {
      &quot;field&quot;: &quot;species&quot;,
      &quot;size&quot;: 3
    },
    &quot;aggs&quot;: {
      &quot;avg_sepal_width&quot;: {
        &quot;avg&quot;: {
          &quot;field&quot;: &quot;sepal_width&quot;
        }
      }
    }
  }
}')
</pre>
<p><em>(Elasticsearch 5.x users please note that when using the out-of-the-box mappings the above aggregation requires that <code>&quot;field&quot;: &quot;species&quot;</code> be changed to <code>&quot;field&quot;: &quot;species.keyword&quot;</code> – see <a title="Text fields in Elasticsearch 5.x" target="_blank" rel="nofollow" href="https://www.elastic.co/guide/en/elasticsearch/reference/5.0/breaking_50_mapping_changes.html">here</a> for more information as to why)</em></p>
<p>This aggregation is also executed via the <code>%search%</code> operator on the appropriate resource – e.g.,</p>
<pre>
elastic(&quot;http://localhost:9200&quot;, &quot;iris&quot;, &quot;data&quot;) %search% avg_sepal_width

#          key doc_count avg_sepal_width.value
# 1     setosa        50                 3.428
# 2 versicolor        50                 2.770
# 3  virginica        50                 2.974
</pre>
<p>Queries and aggregations can be combined such that the aggregations are computed on the results of the query. For example, to execute the combination of the above query and aggregation, we would execute,</p>
<pre>
elastic(&quot;http://localhost:9200&quot;, &quot;iris&quot;, &quot;data&quot;) %search% (for_everything + avg_sepal_width)

#          key doc_count avg_sepal_width.value
# 1     setosa        50                 3.428
# 2 versicolor        50                 2.770
# 3  virginica        50                 2.974
</pre>
<p>where the combination yields,</p>
<pre>
print(for_everything + avg_sepal_width)

# {
#     &quot;size&quot;: 0,
#     &quot;query&quot;: {
#         &quot;match_all&quot;: {
#
#         }
#     },
#     &quot;aggs&quot;: {
#         &quot;avg_sepal_width_per_species&quot;: {
#             &quot;terms&quot;: {
#                 &quot;field&quot;: &quot;species&quot;,
#                 &quot;size&quot;: 0
#             },
#             &quot;aggs&quot;: {
#                 &quot;avg_sepal_width&quot;: {
#                     &quot;avg&quot;: {
#                         &quot;field&quot;: &quot;sepal_width&quot;
#                     }
#                 }
#             }
#         }
#     }
# }
</pre>
<p>For comprehensive coverage of all query and aggregations types please refer to the rather excellent <a title="Elasticsearch documentation" target="_blank" rel="nofollow" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html">official documentation</a> (newcomers to Elasticsearch are advised to start with the ‘Query String’ query).</p>
<h3>Mappings</h3>
<p>Finally, I have included the ability to create an empty index with a custom mapping, using the <code>%create%</code> operator – e.g.,</p>
<pre>
elastic(&quot;http://localhost:9200&quot;, &quot;iris&quot;) %create% mapping_default_simple()
</pre>
<p>Where in this instance <code>mapping_default_simple()</code> is a default mapping that I have shipped with <code>elasticsearchr</code>. It switches-off the text analyser for all fields of type ‘string’ (i.e. switches off free text search), allows all text search to work with case-insensitive lower-case terms, and maps any field with the name ‘timestamp’ to type ‘date’, so long as it has the appropriate string or long format.</p>
<h2>Forthcoming Attractions</h2>
<p>I do not have a grand vision for <code>elasticsearchr</code> – I want to keep it a lightweight client that requires knowledge of Elasticsearch – but I would like to add the ability to compose major query and aggregation types, without having to type-out lots of JSON, and to be able to retrieve simple information like the names of all indices in a cluster, and all the document types within an index, etc. Future development will likely be focused in these areas, but I am open to your suggestions (open an issue <a title="elasticsearchr issues" target="_blank" rel="nofollow" href="https://github.com/AlexIoannides/elasticsearchr/issues">here</a>).</p>
<h2>Acknowledgements</h2>
<p>A big thank you to Hadley Wickham and Jeroen Ooms, the authors of the <code>httr</code> and <code>jsonlite</code> packages that <code>elasticsearchr</code> leans upon <em>heavily</em>.</p>
<p> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/alexioannides.wordpress.com/366/"><img alt="" src="http://feeds.wordpress.com/1.0/comments/alexioannides.wordpress.com/366/" border="0"></a> <img alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" border="0" data-recalc-dims="1"></p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/elasticsearchr-a-lightweight-elasticsearch-client-for-r/"></div></div><div><a data-url="https://www.r-bloggers.com/elasticsearchr-a-lightweight-elasticsearch-client-for-r/" data-counturl="https://www.r-bloggers.com/elasticsearchr-a-lightweight-elasticsearch-client-for-r/" data-count="vertical" data-text="elasticsearchr – a Lightweight Elasticsearch Client for R" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://alexioannides.com/2016/11/28/elasticsearchr-a-lightweight-elasticsearch-client-for-r/"> R – When Localhost Isn't Enough</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/vZA4hLzJ3sU"></a></div>