<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/microsoft-cognitive-services-vision-api-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://www.stoltzmaniac.com/microsoft-cognitive-services-vision-api-in-r/"> R-Projects – Stoltzmaniac</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h1>Microsoft Cognitive Services Vision API in R</h1>
<hr>
<p>A little while ago I did a brief tutorial of the <a rel="nofollow" href="https://www.stoltzmaniac.com/google-vision-api-in-r-rooglevision/" target="_blank">Google Vision API using RoogleVision</a> created by Mark Edmonson. I couldn’t find anything similar to that in R for the <a rel="nofollow" href="https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/quickstarts/python#AnalyzeImage" target="_blank">Microsoft Cognitive Services API</a> so I thought I would give it a shot. I whipped this example together quickly to give it a proof-of-concept but I could certainly see myself building an R package to support this (unless someone can point to one – and please do if one exists)!</p>
<p>A quick example, sending this image retrieved the location of the human face and created a caption! Here’s my dog lined up next to his doppelganger:</p>
<p><img alt="plot of chunk unnamed-chunk-1" src="https://i2.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/09/unnamed-chunk-1-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>The API is extremely easy to access using RCurl and httr. There are <strong>a lot</strong> of options which can be accessed. In this example, I’ll just cover the basics of image detection and descriptions.</p>
<p><span></span></p>
<p>If you don’t want to spend time writing a bunch of code, you can simply use the “helper_functions.R” file I created and swap out your credentials and API endpoint to get it working.</p>
<h3>Getting Started With Microsoft Cognitive Services</h3>
<p>In order to get started, all you need is an <a rel="nofollow" href="https://azure.microsoft.com/en-us/free/" target="_blank">Azure Account</a> which is <strong>free</strong> if you can keep yourself under certain thresholds and limits. There is even a free trial period (at the time this was written, at least).</p>
<p>Once that is taken care of there are a few things you need to do:</p>
<ol>
<li>Login to <a rel="nofollow" href="https://portal.azure.com/" target="_blank">portal.azure.com</a></li>
<li>On the lefthand menu click “Add”<br>
<img alt="Figure 1" src="https://i1.wp.com/i.imgur.com/OPihH39.png?w=456&ssl=1" data-recalc-dims="1"></li>
<li>Click on “AI + Cognitive Services” and then the “Computer Vision API”<br>
<img alt="Figure 2" src="https://i0.wp.com/i.imgur.com/GMy2LFZ.png?w=456&ssl=1" data-recalc-dims="1"></li>
<li>Fill out the information required. You may have “Free Trial” under Subscription. Pay special attention to <strong>Location</strong> because this will be used in your API script<br>
<img alt="Figure 3" src="https://i0.wp.com/i.imgur.com/t7vg4vH.png?w=456&ssl=1" data-recalc-dims="1"></li>
<li>In the lefthand menu, click “Keys” underneath “Resource Management” and you will find what you need for credentials. Underneath your Endpoint URL, click on “Show access keys…” – <strong>copy your key</strong> and use it in your script (do not make this publicly accessible)<br>
<img alt="Figure 4" src="https://i0.wp.com/i.imgur.com/CKkC2nx.png?w=456&ssl=1" data-recalc-dims="1"></li>
<li>You’re ready to go!</li>
</ol>
<p>Now, you can write a script to utilize the power of the Microsoft Cognitive Vision API.</p>
<h4>What data can you get?</h4>
<p>There are <strong>a lot</strong> of features you can request. I’m only asking for: description, tags, categories, and faces. You can also return: image type, color, and adult. There are also details which can be returned such as: landmarks and celebrities.</p>
<p>Here is the setup and API call:</p>
<pre><code>image_url = 'https://imgur.com/rapIn0u.jpg'
visualFeatures = &quot;Description,Tags,Categories,Faces&quot;
# options = &quot;Categories, Tags, Description, Faces, ImageType, Color, Adult&quot;

details = &quot;Landmarks&quot;
# options = Landmarks, Celebrities

reqURL = paste(api_endpoint_url,
               &quot;?visualFeatures=&quot;,
               visualFeatures,
               &quot;&amp;details=&quot;,
               details,
               sep=&quot;&quot;)

APIresponse = POST(url = reqURL,
                   content_type('application/json'),
                   add_headers(.headers = c('Ocp-Apim-Subscription-Key' = api_key)),
                   body=list(url = image_url),
                   encode = &quot;json&quot;) 

df = content(APIresponse)
</code></pre>
<p>The dataframe returned looks messy, but isn’t too bad once you dive in. Take a look:</p>
<pre><code>str(df)
</code></pre>
<pre><code>## List of 5
##  $ tags       :List of 6
##   ..$ :List of 2
##   .. ..$ name      : chr &quot;dog&quot;
##   .. ..$ confidence: num 0.987
##   ..$ :List of 3
##   .. ..$ name      : chr &quot;mammal&quot;
##   .. ..$ confidence: num 0.837
##   .. ..$ hint      : chr &quot;animal&quot;
##   ..$ :List of 2
##   .. ..$ name      : chr &quot;looking&quot;
##   .. ..$ confidence: num 0.814
##   ..$ :List of 2
##   .. ..$ name      : chr &quot;animal&quot;
##   .. ..$ confidence: num 0.811
##   ..$ :List of 2
##   .. ..$ name      : chr &quot;posing&quot;
##   .. ..$ confidence: num 0.54
##   ..$ :List of 2
##   .. ..$ name      : chr &quot;staring&quot;
##   .. ..$ confidence: num 0.165
##  $ description:List of 2
##   ..$ tags    :List of 18
##   .. ..$ : chr &quot;dog&quot;
##   .. ..$ : chr &quot;mammal&quot;
##   .. ..$ : chr &quot;looking&quot;
##   .. ..$ : chr &quot;animal&quot;
##   .. ..$ : chr &quot;photo&quot;
##   .. ..$ : chr &quot;posing&quot;
##   .. ..$ : chr &quot;camera&quot;
##   .. ..$ : chr &quot;man&quot;
##   .. ..$ : chr &quot;standing&quot;
##   .. ..$ : chr &quot;smiling&quot;
##   .. ..$ : chr &quot;face&quot;
##   .. ..$ : chr &quot;white&quot;
##   .. ..$ : chr &quot;holding&quot;
##   .. ..$ : chr &quot;close&quot;
##   .. ..$ : chr &quot;wearing&quot;
##   .. ..$ : chr &quot;laying&quot;
##   .. ..$ : chr &quot;head&quot;
##   .. ..$ : chr &quot;teeth&quot;
##   ..$ captions:List of 1
##   .. ..$ :List of 2
##   .. .. ..$ text      : chr &quot;a close up of Albert Einstein and a dog posing for the camera&quot;
##   .. .. ..$ confidence: num 0.892
##  $ requestId  : chr &quot;2143e23a-14c8-47c4-9750-9bfc82381512&quot;
##  $ metadata   :List of 3
##   ..$ width : int 824
##   ..$ height: int 824
##   ..$ format: chr &quot;Jpeg&quot;
##  $ faces      :List of 1
##   ..$ :List of 3
##   .. ..$ age          : int 73
##   .. ..$ gender       : chr &quot;Male&quot;
##   .. ..$ faceRectangle:List of 4
##   .. .. ..$ left  : int 505
##   .. .. ..$ top   : int 241
##   .. .. ..$ width : int 309
##   .. .. ..$ height: int 309
</code></pre>
<h4>The top 5 descriptions returned are:</h4>
<pre><code>description_tags = df$description$tags
description_tags_tib = tibble(tag = character())
for(tag in description_tags){
  for(text in tag){
     if(class(tag) != &quot;list&quot;){  ## To remove the extra caption from being included
      tmp = tibble(tag = tag)
      description_tags_tib = description_tags_tib %>% bind_rows(tmp)
    } 
  }
}

knitr::kable(description_tags_tib[1:5,])
</code></pre>
<table>
<thead>
<tr>
<th align="left">tag</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">dog</td>
</tr>
<tr>
<td align="left">mammal</td>
</tr>
<tr>
<td align="left">looking</td>
</tr>
<tr>
<td align="left">animal</td>
</tr>
<tr>
<td align="left">photo</td>
</tr>
</tbody>
</table>
<p>The caption returned:</p>
<pre><code>captions = df$description$captions
captions_tib = tibble(text = character(), confidence = numeric())
for(caption in captions){
  tmp = tibble(text = caption$text, confidence = caption$confidence)
  captions_tib = captions_tib %>% bind_rows(tmp)
}
knitr::kable(captions_tib)
</code></pre>
<table>
<thead>
<tr>
<th align="left">text</th>
<th align="right">confidence</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">a close up of Albert Einstein and a dog posing for the camera</td>
<td align="right">0.891846</td>
</tr>
</tbody>
</table>
<h4>The metadata returned:</h4>
<pre><code>metadata = df$metadata
metadata_tib = tibble(width = metadata$width, height = metadata$height, format = metadata$format)
knitr::kable(metadata_tib)
</code></pre>
<table>
<thead>
<tr>
<th align="right">width</th>
<th align="right">height</th>
<th align="left">format</th>
</tr>
</thead>
<tbody>
<tr>
<td align="right">824</td>
<td align="right">824</td>
<td align="left">Jpeg</td>
</tr>
</tbody>
</table>
<h4>The locations of faces returned:</h4>
<pre><code>faces = df$faces
faces_tib = tibble(faceID = numeric(),
                   age = numeric(), 
                   gender = character(),
                   x1 = numeric(),
                   x2 = numeric(),
                   y1 = numeric(),
                   y2 = numeric())

n = 0
for(face in faces){
  n = n + 1
  tmp = tibble(faceID = n,
               age = face$age, 
               gender = face$gender,
               x1 = face$faceRectangle$left,
               y1 = face$faceRectangle$top,
               x2 = face$faceRectangle$left + face$faceRectangle$width,
               y2 = face$faceRectangle$top + face$faceRectangle$height)
  faces_tib = faces_tib %>% bind_rows(tmp)
}
#faces_tib
knitr::kable(faces_tib)
</code></pre>
<table>
<thead>
<tr>
<th align="right">faceID</th>
<th align="right">age</th>
<th align="left">gender</th>
<th align="right">x1</th>
<th align="right">x2</th>
<th align="right">y1</th>
<th align="right">y2</th>
</tr>
</thead>
<tbody>
<tr>
<td align="right">1</td>
<td align="right">73</td>
<td align="left">Male</td>
<td align="right">505</td>
<td align="right">814</td>
<td align="right">241</td>
<td align="right">550</td>
</tr>
</tbody>
</table>
<h4>A few more examples:</h4>
<p><img alt="plot of chunk unnamed-chunk-8" src="https://i0.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/09/unnamed-chunk-8-1-3.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p><img alt="plot of chunk unnamed-chunk-9" src="https://i0.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/09/unnamed-chunk-9-1-3.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p><img alt="plot of chunk unnamed-chunk-10" src="https://i2.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/09/unnamed-chunk-10-1-3.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p><img alt="plot of chunk unnamed-chunk-11" src="https://i0.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/09/unnamed-chunk-11-1-3.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p><img alt="plot of chunk unnamed-chunk-12" src="https://i1.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/09/unnamed-chunk-12-1-3.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>As always, you can find the code I used on my <a rel="nofollow" href="https://github.com/stoltzmaniac/ML-Image-Processing-R/tree/master/Microsoft%20Vision%20API" target="_blank">GitHub</a>.</p>
<p>Side note: I used a ton of for loops to access the data – not ideal… please let me know if you have better ways of dealing with nested data like this when you have unknown numbers of levels.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.stoltzmaniac.com/microsoft-cognitive-services-vision-api-in-r/"> R-Projects – Stoltzmaniac</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/lKH9KcO-XZs" height="1"></a></div>