<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/image-description-with-microsofts-cognitive-services-and-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://flovv.github.io/Roxford_extension/"> Florian Teschner</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>A while back, I created the small package <a target="_blank" rel="nofollow" href="http://flovv.github.io/Image-Recognition/">Roxford</a> to access Microsoft’s Cognitive Services API in order to easily recognize objects in images.<br>
Back then Microsoft called the service “Project Oxford”, hence the name “Roxford”. Since then Microsoft extended their API to include image tagging, description and celebrity detection.<br>
In the following post, I will try to illustrate the functionality and how it is called through the package. To install the package, just follow the <a target="_blank" rel="nofollow" href="https://github.com/flovv/Roxford">guide</a>.</p>
<p>After installing the package, usage is (hopefully) straight forward. Set your API key and call the function, either by providing a path to a local image or to a remote image url. If you use remote images, use functions which end with “URL”.<br>
The following code exemplifies this for the “getDescription” for an image function.</p>
<figure>
<pre><code data-lang="r"><span>res</span><span> </span><span>&lt;-</span><span> </span><span>Roxford</span><span>::</span><span>getDescriptionResponseURL</span><span>(</span><span>url</span><span>,</span><span> </span><span>visionKey</span><span>,</span><span> </span><span>maxCandidates</span><span> </span><span>=</span><span> </span><span>4</span><span>)</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-3" src="https://i2.wp.com/flovv.github.io/figures/post16/unnamed-chunk-3-1.png?w=456" data-recalc-dims="1"></p>
<p>All the text in the chart is provided in the API response. First the title is the “caption text” and the confidence score attached to it. Additionally, the call returns a list of tags associated to the image. Both the tags and the caption look pretty good to me.</p>
<p>Compared to the previous version, the updated API offers a “tagging” service.<br>
While the tags in the image above are pretty extensive, this call returns just one categorization.</p>
<figure>
<pre><code data-lang="r"><span>resTag</span><span> </span><span>&lt;-</span><span> </span><span>Roxford</span><span>::</span><span>getTaggingResponseURL</span><span>(</span><span>url</span><span>,</span><span> </span><span>visionKey</span><span>)</span><span>
</span><span>resTag</span><span>[</span><span>&quot;tags&quot;</span><span>,</span><span> </span><span>]</span><span>$</span><span>name</span><span>
</span><span>##</span><span> </span><span>[</span><span>1</span><span>]</span><span> </span><span>&quot;person&quot;</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">resTag[&quot;tags&quot;, ]$confidence
## [1] &quot;0,999826610088348&quot;</code></pre>
</figure>
<p>Finally, the last extension is the possibility to call domain-specific models.<br>
In a a first step the API returns a list of what domain models are available (currently just ‘celebrities’).</p>
<figure>
<pre><code data-lang="text">##                  name categories
## models    celebrities    people_
## requestId        &lt;NA>       &lt;NA></code></pre>
</figure>
<p>In a second step, one can provide the image resource and the model specification (‘celebrities’) to get a classification of the provided image.</p>
<p><img alt="plot of chunk unnamed-chunk-6" src="https://i1.wp.com/flovv.github.io/figures/post16/unnamed-chunk-6-1.png?w=456" data-recalc-dims="1"></p>
<p>Again, all text in the chart is returned by the API. I wonder if Microsoft plans to open up their backend in order for others to plugin in this domain-specific modelling aspect. I could imagine that there are developers who would love to share their models through a centralized market place.<br>
Overall I find the API quite convenient to work with.<br>
Happy classification :).</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://flovv.github.io/Roxford_extension/"> Florian Teschner</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/OQ-1hhjW69o"></a></div>