<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/generate-image-captions-with-the-computer-vision-api/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2018/03/computer-vision-api.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>The Azure <a rel="nofollow" href="https://azure.microsoft.com/en-us/services/cognitive-services/computer-vision/?WT.mc_id=Revolutions-blog-davidsmi" target="_blank">Computer Vision API</a> can extract all sorts of interesting information from images — tags describing objects found in the images, locations of detected faces, and <a rel="nofollow" href="https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/home?WT.mc_id=Revolutions-blog-davidsmi" target="_blank">more</a> — but today I want to play around with just one: caption generation. I was inspired by <a rel="nofollow" href="https://twitter.com/picdescbot" target="_blank">@picdescbot</a> on Twitter, which selects <a rel="nofollow" href="https://commons.wikimedia.org/w/index.php?title=Special:Random" target="_blank">random images from Wikimedia Commons</a> and generates a caption using the API. The results are sometimes <a rel="nofollow" href="https://twitter.com/picdescbot/status/974540815409139712" target="_blank">impressive</a>, sometimes <a rel="nofollow" href="https://twitter.com/picdescbot/status/746583745076072448" target="_blank">funny</a>, and sometimes <a rel="nofollow" href="https://twitter.com/picdescbot/status/871969975145312256" target="_blank">bizarre</a>. On the bizarre side, recurring motifs include &quot;lush green field&quot;, and &quot;a clock tower on a building&quot;, but the bot doesn't discriminate when the confidence score returned by the API is low, so I wanted to take a look at that as well.</p>
<p>I'll use R to interface with the API with the code provided below. You can use the code too, but you'll need an Azure login first. If you don't have one, you can sign for a <a rel="nofollow" href="http://azure.com/free/?WT.mc_id=Revolutions-blog-davidsmi" target="_blank">free Azure account</a> (or <a rel="nofollow" href="https://azure.microsoft.com/free/students/?WT.mc_id=Revolutions-blog-davidsmi" target="_blank">Azure for Students</a>, which doesn't require a credit card), and get some free credits to boot. We won't be using any credits in this case, though, as the Computer Vision API has a free pricing tier: it's limited to 5,000 calls a month and 20 calls per minutes, but that's more than sufficient for our needs.</p>
<p>To generate a key for the Computer Vision API, visit the <a rel="nofollow" href="https://portal.azure.com/" target="_blank">Azure Portal</a> and click &quot;Create a Resource&quot;. Select &quot;AI + Cognitive Services&quot; and then &quot;Computer Vision API&quot;. Choose a name, your subscription (yours will be different), a data center (choose a region close to you), the pricing tier (F0 is the free tier), and create a new resource group to hold your keys:</p>
<p><a rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d2e60c8b970c-pi" target="_blank"><img border="0" alt="Vison api" title="Vison api" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d2e60c8b970c-800wi"></a></p>
<p>It'll take just a moment to set things up, but once that's done click select &quot;Overview&quot; to display the API endpoint, and &quot;Keys&quot; to display the API keys. (Two keys are generated, but you will only need to use Key 1.)</p>
<p>Now launch R, load a couple of packages we'll need, and save the endpoint and API key into R objects as shown below:</p>
<pre>library(tools)
library(httr)
vision_api_endpoint &lt;- &quot;https://westus.api.cognitive.microsoft.com/vision/v1.0&quot;
vision_api_key &lt;- &quot;7f1f01ac24064abd80970f41a90237e7&quot;
</pre>
<p>(Your endpoint may differ depending on the region you chose, and your API key will <em>definitely</em> be different: that one's invalid.) With those two pieces of information, we're all set to go.</p>
<p>Step 1 is to generate the URL of an image from Wikimedia commons. This is possible to do using the <a rel="nofollow" href="https://www.mediawiki.org/wiki/API:Main_page" target="_blank">Wikimedia API</a>, and easy once you know how. (It took me quite a while and a lot of trial and error to figure out <em>that</em> API, though.) The simple R function below will query the Wikimedia Commons API and return the URL of a random image file. It also checks that the image meets the requirements of the Computer Vision API, and throws an error if not.</p>
<p></p>
<p>You can now call the function to generate the URL of a random image. (If it throws an error, just try again.) It also returns as <a rel="nofollow" href="http://adv-r.had.co.nz/Data-structures.html#attributes" target="_blank">attrributes</a> the dimensions of the image and the description from the Wikimedia Commons page, which will be interesting to compare to the Computer Vision API generated caption.</p>
<pre>> random_image()
[1] &quot;https://upload.wikimedia.org/wikipedia/commons/b/b4/Villa_Malva.jpg&quot;
attr(,&quot;dims&quot;)
   w    h 
3072 2304 
attr(,&quot;desc&quot;)
[1] &quot;Villa Malva i Ramlösa brunnspark i Helsingborg.&quot;
</pre>
<p>Step 2 is to use the URL generated by that function as input to the Computer Vision API with the second R function below. It uses the global <code>vision_api_endpoint</code> and <code>vision_api_key</code> objects you defined earlier to call the Computer Vision API, requesting the Description (caption). It will also try and identify celebrities and famous landmarks, if it finds them in the image (for example, the generated caption for <a rel="nofollow" href="https://upload.wikimedia.org/wikipedia/commons/1/12/Jack_Reacher-_Never_Go_Back_Japan_Premiere_Red_Carpet-_Tom_Cruise_%2835375035831%29.jpg" target="_blank">this image</a> is &quot;TOM CRUISE wearing a suit and tie&quot;).</p>
<p></p>
<p>That function prints the URL, caption generated by the Computer Vision API and its confidence score (a value of 0 and 1), along with the Wikimedia Commons description for comparison. Let's give it a go. In each case I simply ran the two lines below. </p>
<pre>> u &lt;- random_image()
> image_caption(u)
</pre>
<p>Here are a few results:</p>
<p><a rel="nofollow" href="https://upload.wikimedia.org/wikipedia/commons/9/99/Falco-rusticolus.jpg" target="_blank"><img border="0" alt="image from upload.wikimedia.org" title="image from upload.wikimedia.org" src="http://a7.typepad.com/6a0105360ba1c6970c01b8d2e60fc7970c-800wi"></a><br><strong>Wikimedia description</strong>: Gerfalke (Falco rusticolus) in Westgrönland<br><strong>Computer Vision API caption</strong> (confidence: 92.9%): a bird that is standing in the grass</p>
<p><a rel="nofollow" href="https://upload.wikimedia.org/wikipedia/commons/7/70/DesdeAtrasSepia.jpg" target="_blank"><img border="0" alt="image from upload.wikimedia.org" title="image from upload.wikimedia.org" src="http://a2.typepad.com/6a0105360ba1c6970c01bb09ff03a2970d-800wi"></a><br><strong>Wikimedia description</strong>: Vista de la casa de la Hacienda Mozanga en 1995.<br><strong>Computer Vision API caption</strong> (confidence: 90.2%): a house with trees in the background</p>
<p><a rel="nofollow" href="https://upload.wikimedia.org/wikipedia/commons/3/35/Concierto_de_Borgore.jpg" target="_blank"><img border="0" alt="image from upload.wikimedia.org" title="image from upload.wikimedia.org" src="http://a1.typepad.com/6a0105360ba1c6970c01bb09ff03e1970d-800wi"></a><br><strong>Wikimedia description</strong>: Dj Israeli, en un concierto.<br><strong>Computer Vision API caption</strong> (confidence: 44.3%): a man riding on the back of a bicycle</p>
<p>That last example is a good lesson not to trust the captions when the confidence score is low! Let is know on any interesting, funny or strange captions you get in the comments below. </p>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2018/03/computer-vision-api.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/zTtSjUSOZ5w" height="1"></a></div>