<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/google-vision-api-in-r-rooglevision/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://www.stoltzmaniac.com/google-vision-api-in-r-rooglevision/"> R-Projects – Stoltzmaniac</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h2>Using the Google Vision API in R</h2>
<h3>Utilizing RoogleVision</h3>
<p>After doing <a rel="nofollow" href="https://www.stoltzmaniac.com/facial-recognition-in-r?utm_campaign=top_of_roogle_vision_post" target="_blank">my post last month</a> on OpenCV and face detection, I started looking into other algorithms used for pattern detection in images. As it turns out, Google has done a phenomenal job with their Vision API. It’s absolutely incredible the amount of information it can spit back to you by simply sending it a picture.</p>
<p>Also, it’s 100% free! I believe that includes 1000 images per month. Amazing!</p>
<p>In this post I’m going to walk you through the absolute basics of accessing the power of the Google Vision API using the <a rel="nofollow" href="https://github.com/cloudyr/RoogleVision" target="_blank">RoogleVision package in R</a>.</p>
<p>As always, we’ll start off loading some libraries. I wrote some extra notation around where you can install them within the code.</p>
<pre><code># Normal Libraries
library(tidyverse)

# devtools::install_github(&quot;flovv/RoogleVision&quot;)
library(RoogleVision)
library(jsonlite) # to import credentials

# For image processing
# source(&quot;http://bioconductor.org/biocLite.R&quot;)
# biocLite(&quot;EBImage&quot;)
library(EBImage)

# For Latitude Longitude Map
library(leaflet)
</code></pre>
<h4>Google Authentication</h4>
<p>In order to use the API, you have to authenticate. There is plenty of documentation out there about how to setup an account, create a project, download credentials, etc. Head over to <a rel="nofollow" href="https://console.cloud.google.com/" target="_blank">Google Cloud Console</a> if you don’t have an account already.</p>
<pre><code># Credentials file I downloaded from the cloud console
creds = fromJSON('credentials.json')

# Google Authentication - Use Your Credentials
# options(&quot;googleAuthR.client_id&quot; = &quot;xxx.apps.googleusercontent.com&quot;)
# options(&quot;googleAuthR.client_secret&quot; = &quot;&quot;)

options(&quot;googleAuthR.client_id&quot; = creds$installed$client_id)
options(&quot;googleAuthR.client_secret&quot; = creds$installed$client_secret)
options(&quot;googleAuthR.scopes.selected&quot; = c(&quot;https://www.googleapis.com/auth/cloud-platform&quot;))
googleAuthR::gar_auth()
</code></pre>
<h3>Now You’re Ready to Go</h3>
<p>The function getGoogleVisionResponse takes three arguments:</p>
<ol>
<li>imagePath</li>
<li>feature</li>
<li>numResults</li>
</ol>
<p>Numbers 1 and 3 are self-explanatory, “feature” has 5 options:</p>
<ul>
<li>LABEL_DETECTION</li>
<li>LANDMARK_DETECTION</li>
<li>FACE_DETECTION</li>
<li>LOGO_DETECTION</li>
<li>TEXT_DETECTION</li>
</ul>
<p>These are self-explanatory but it’s nice to see each one in action.</p>
<p>As a side note: there are also other features that the API has which aren’t included (yet) in the RoogleVision package such as “Safe Search” which identifies inappropriate content, “Properties” which identifies dominant colors and aspect ratios and a few others can be found at the <a rel="nofollow" href="https://cloud.google.com/vision/" target="_blank">Cloud Vision website</a></p>
<hr>
<h4>Label Detection</h4>
<p>This is used to help determine content within the photo. It can basically add a level of metadata around the image.</p>
<p>Here is a photo of our dog when we hiked up to Audubon Peak in Colorado:</p>
<p><img alt="plot of chunk unnamed-chunk-2" src="https://i1.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/07/unnamed-chunk-2-1.png?w=620&ssl=1" data-recalc-dims="1"></p>
<pre><code>dog_mountain_label = getGoogleVisionResponse('dog_mountain.jpg',
                                              feature = 'LABEL_DETECTION')
head(dog_mountain_label)
</code></pre>
<pre><code>##            mid           description     score
## 1     /m/09d_r              mountain 0.9188690
## 2 /g/11jxkqbpp mountainous landforms 0.9009549
## 3    /m/023bbt            wilderness 0.8733696
## 4     /m/0kpmf             dog breed 0.8398435
## 5    /m/0d4djn            dog hiking 0.8352048
</code></pre>
<p>All 5 responses were incredibly accurate! The “score” that is returned is how confident the Google Vision algorithms are, so there’s a 91.9% chance a mountain is prominent in this photo. I like “dog hiking” the best – considering that’s what we were doing at the time. Kind of a little bit too accurate…</p>
<hr>
<h4>Landmark Detection</h4>
<p>This is a feature designed to specifically pick out a recognizable landmark! It provides the position in the image along with the geolocation of the landmark (in longitude and latitude).</p>
<p>My wife and I took this selfie in at the Linderhof Castle in Bavaria, Germany.</p>
<pre><code>us_castle &lt;- readImage('us_castle_2.jpg')
plot(us_castle)
</code></pre>
<p><img alt="plot of chunk unnamed-chunk-4" src="https://i1.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/07/unnamed-chunk-4-1.png?w=620&ssl=1" data-recalc-dims="1"></p>
<p>The response from the Google Vision API was spot on. It returned “Linderhof Palace” as the description. It also provided a score (I reduced the resolution of the image which hurt the score), a boundingPoly field and locations.</p>
<ul>
<li>Bounding Poly – gives x,y coordinates for a polygon around the landmark in the image</li>
<li>Locations – provides longitude,latitude coordinates</li>
</ul>
<pre><code>us_landmark = getGoogleVisionResponse('us_castle_2.jpg',
                                      feature = 'LANDMARK_DETECTION')
head(us_landmark)
</code></pre>
<pre><code>##         mid      description     score
## 1 /m/066h19 Linderhof Palace 0.4665011
##                               vertices          locations
## 1 25, 382, 382, 25, 178, 178, 659, 659 47.57127, 10.96072
</code></pre>
<p>I plotted the polygon over the image using the coordinates returned. It does a great job (certainly not perfect) of getting the castle identified. It’s a bit tough to say what the actual “landmark” would be in this case due to the fact the fountains, stairs and grounds are certainly important and are a key part of the castle.</p>
<pre><code>us_castle &lt;- readImage('us_castle_2.jpg')
plot(us_castle)
xs = us_landmark$boundingPoly$vertices[[1]][1][[1]]
ys = us_landmark$boundingPoly$vertices[[1]][2][[1]]
polygon(x=xs,y=ys,border='red',lwd=4)
</code></pre>
<p><img alt="plot of chunk unnamed-chunk-6" src="https://i1.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/07/unnamed-chunk-6-1.png?w=620&ssl=1" data-recalc-dims="1"></p>
<p>Turning to the locations – I plotted this using the leaflet library. If you haven’t used leaflet, start doing so immediately. I’m a huge fan of it due to speed and simplicity. There are a lot of customization options available as well that you can check out.</p>
<p>The location = spot on! While it isn’t a shock to me that Google could provide the location of “Linderhof Castle” – it is amazing to me that I don’t have to write a web crawler search function to find it myself! That’s just one of many little luxuries they have built into this API.</p>
<pre><code>latt = us_landmark$locations[[1]][[1]][[1]]
lon = us_landmark$locations[[1]][[1]][[2]]
m = leaflet() %>%
  addProviderTiles(providers$CartoDB.Positron) %>%
  setView(lng = lon, lat = latt, zoom = 5) %>%
  addMarkers(lng = lon, lat = latt)
m</code><code>
</code></pre>
<p><img data-orig-file="https://i1.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/07/Screen-Shot-2017-07-31-at-11.39.24-AM.png?fit=1344%2C952&amp;ssl=1" data-orig-size="1344,952" data-medium-file="https://i1.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/07/Screen-Shot-2017-07-31-at-11.39.24-AM.png?fit=300%2C213&amp;ssl=1" data-attachment-id="499" data-image-meta="{" data-image-description="" src="https://i1.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/07/Screen-Shot-2017-07-31-at-11.39.24-AM.png?resize=458%2C325" data-permalink="https://www.stoltzmaniac.com/google-vision-api-in-r-rooglevision/screen-shot-2017-07-31-at-11-39-24-am/" data-large-file="https://i1.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/07/Screen-Shot-2017-07-31-at-11.39.24-AM.png?fit=620%2C439&amp;ssl=1" alt="" data-image-title="Screen Shot 2017-07-31 at 11.39.24 AM" sizes="(max-width: 458px) 100vw, 458px" data-comments-opened="0" class="wp-image-499" data-recalc-dims="1"></p>
<hr>
<h4>Face Detection</h4>
<p>My last blog post showed the OpenCV package utilizing the haar cascade algorithm in action. I didn’t dig into Google’s algorithms to figure out what is under the hood, but it provides similar results. However, rather than layering in each subsequent “find the eyes” and “find the mouth” and …etc… it returns more than you ever needed to know.</p>
<ul>
<li>Bounding Poly = highest level polygon</li>
<li>FD Bounding Poly = polygon surrounding each face</li>
<li>Landmarks = (funny name) includes each feature of the face (left eye, right eye, etc.)</li>
<li>Roll Angle, Pan Angle, Tilt Angle = all of the different angles you’d need per face</li>
<li>Confidence (detection and landmarking) = how certain the algorithm is that it’s accurate</li>
<li>Joy, sorrow, anger, surprise, under exposed, blurred, headwear likelihoods = how likely it is that each face contains that emotion or characteristic</li>
</ul>
<p>The likelihoods is another amazing piece of information returned! I have run about 20 images through this API and every single one has been accurate – very impressive!</p>
<p>I wanted to showcase the face detection and headwear first. Here’s a picture of my wife and I at “The Bean” in Chicago (side note: it’s awesome! I thought it was going to be really silly, but you can really have a lot of fun with all of the angles and reflections):</p>
<pre><code>us_hats_pic &lt;- readImage('us_hats.jpg')
plot(us_hats_pic)
</code></pre>
<p><img alt="plot of chunk unnamed-chunk-8" src="https://i2.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/07/unnamed-chunk-8-1.png?w=620&ssl=1" data-recalc-dims="1"></p>
<pre><code>us_hats = getGoogleVisionResponse('us_hats.jpg',
                                      feature = 'FACE_DETECTION')
head(us_hats)
</code></pre>
<pre><code>##                                 vertices
## 1 295, 410, 410, 295, 164, 164, 297, 297
## 2 353, 455, 455, 353, 261, 261, 381, 381
##                                 vertices
## 1 327, 402, 402, 327, 206, 206, 280, 280
## 2 368, 439, 439, 368, 298, 298, 370, 370
##
## landmarks...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           landmarks
##   rollAngle panAngle tiltAngle detectionConfidence landmarkingConfidence
## 1  7.103324 23.46835 -2.816312           0.9877176             0.7072066
## 2  2.510939 -1.17956 -7.393063           0.9997375             0.7268016
##   joyLikelihood sorrowLikelihood angerLikelihood surpriseLikelihood
## 1   VERY_LIKELY    VERY_UNLIKELY   VERY_UNLIKELY      VERY_UNLIKELY
## 2   VERY_LIKELY    VERY_UNLIKELY   VERY_UNLIKELY      VERY_UNLIKELY
##   underExposedLikelihood blurredLikelihood headwearLikelihood
## 1          VERY_UNLIKELY     VERY_UNLIKELY        VERY_LIKELY
## 2          VERY_UNLIKELY     VERY_UNLIKELY        VERY_LIKELY
</code></pre>
<pre><code>us_hats_pic &lt;- readImage('us_hats.jpg')
plot(us_hats_pic)

xs1 = us_hats$fdBoundingPoly$vertices[[1]][1][[1]]
ys1 = us_hats$fdBoundingPoly$vertices[[1]][2][[1]]

xs2 = us_hats$fdBoundingPoly$vertices[[2]][1][[1]]
ys2 = us_hats$fdBoundingPoly$vertices[[2]][2][[1]]

polygon(x=xs1,y=ys1,border='red',lwd=4)
polygon(x=xs2,y=ys2,border='green',lwd=4)
</code></pre>
<p><img alt="plot of chunk unnamed-chunk-10" src="https://i0.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/07/unnamed-chunk-10-1.png?w=620&ssl=1" data-recalc-dims="1"></p>
<p>Here’s a shot that should be familiar (copied directly from my last blog) – and I wanted to highlight the different features that can be detected. Look at how many points are perfectly placed:</p>
<pre><code>my_face_pic &lt;- readImage('my_face.jpg')
plot(my_face_pic)
</code></pre>
<p><img alt="plot of chunk unnamed-chunk-11" src="https://i2.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/07/unnamed-chunk-11-1.png?w=620&ssl=1" data-recalc-dims="1"></p>
<pre><code>my_face = getGoogleVisionResponse('my_face.jpg',
                                      feature = 'FACE_DETECTION')
head(my_face)
</code></pre>
<pre><code>##                               vertices
## 1 456, 877, 877, 456, NA, NA, 473, 473
##                               vertices
## 1 515, 813, 813, 515, 98, 98, 395, 395
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             landmarks
## landmarks ...
##    rollAngle  panAngle tiltAngle detectionConfidence landmarkingConfidence
## 1 -0.6375801 -2.120439  5.706552            0.996818             0.8222974
##   joyLikelihood sorrowLikelihood angerLikelihood surpriseLikelihood
## 1   VERY_LIKELY    VERY_UNLIKELY   VERY_UNLIKELY      VERY_UNLIKELY
##   underExposedLikelihood blurredLikelihood headwearLikelihood
## 1          VERY_UNLIKELY     VERY_UNLIKELY      VERY_UNLIKELY
</code></pre>
<pre><code>head(my_face$landmarks)
</code></pre>
<pre><code>## [[1]]
##                            type position.x position.y    position.z
## 1                      LEFT_EYE   598.7636   192.1949  -0.001859295
## 2                     RIGHT_EYE   723.1612   192.4955  -4.805475700
## 3          LEFT_OF_LEFT_EYEBROW   556.1954   165.2836  15.825399000
## 4         RIGHT_OF_LEFT_EYEBROW   628.8224   159.9029 -23.345352000
## 5         LEFT_OF_RIGHT_EYEBROW   693.0257   160.6680 -25.614508000
## 6        RIGHT_OF_RIGHT_EYEBROW   767.7514   164.2806   7.637372000
## 7         MIDPOINT_BETWEEN_EYES   661.2344   185.0575 -29.068363000
## 8                      NOSE_TIP   661.9072   260.9006 -74.153710000
...</code></pre>
<pre><code>my_face_pic &lt;- readImage('my_face.jpg')
plot(my_face_pic)

xs1 = my_face$fdBoundingPoly$vertices[[1]][1][[1]]
ys1 = my_face$fdBoundingPoly$vertices[[1]][2][[1]]

xs2 = my_face$landmarks[[1]][[2]][[1]]
ys2 = my_face$landmarks[[1]][[2]][[2]]

polygon(x=xs1,y=ys1,border='red',lwd=4)
points(x=xs2,y=ys2,lwd=2, col='lightblue')
</code></pre>
<p><img alt="plot of chunk unnamed-chunk-14" src="https://i1.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/07/unnamed-chunk-14-1.png?w=620&ssl=1" data-recalc-dims="1"></p>
<hr>
<h4>Logo Detection</h4>
<p>To continue along the Chicago trip, we drove by Wrigley field and I took a really bad photo of the sign from a moving car as it was under construction. It’s nice because it has a lot of different lines and writing the Toyota logo isn’t incredibly prominent or necessarily fit to brand colors.</p>
<p>This call returns:</p>
<ul>
<li>Description = Brand name of the logo detected</li>
<li>Score = Confidence of prediction accuracy</li>
<li>Bounding Poly = (Again) coordinates of the logo</li>
</ul>
<pre><code>wrigley_image &lt;- readImage('wrigley_text.jpg')
plot(wrigley_image)
</code></pre>
<p><img alt="plot of chunk unnamed-chunk-15" src="https://i1.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/07/unnamed-chunk-15-1.png?w=620&ssl=1" data-recalc-dims="1"></p>
<pre><code>wrigley_logo = getGoogleVisionResponse('wrigley_text.jpg',
                                   feature = 'LOGO_DETECTION')
head(wrigley_logo)
</code></pre>
<pre><code>##           mid description     score                               vertices
## 1 /g/1tk6469q      Toyota 0.3126611 435, 551, 551, 435, 449, 449, 476, 476
</code></pre>
<pre><code>wrigley_image &lt;- readImage('wrigley_text.jpg')
plot(wrigley_image)
xs = wrigley_logo$boundingPoly$vertices[[1]][[1]]
ys = wrigley_logo$boundingPoly$vertices[[1]][[2]]
polygon(x=xs,y=ys,border='green',lwd=4)
</code></pre>
<p><img alt="plot of chunk unnamed-chunk-17" src="https://i0.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/07/unnamed-chunk-17-1.png?w=620&ssl=1" data-recalc-dims="1"></p>
<hr>
<h4>Text Detection</h4>
<p>I’ll continue using the Wrigley Field picture. There is text all over the place and it’s fun to see what is captured and what isn’t. It appears as if the curved text at the top “field” isn’t easily interpreted as text. However, the rest is caught and the words are captured.</p>
<p>The response sent back is a bit more difficult to interpret than the rest of the API calls – it breaks things apart by word but also returns everything as one line. Here’s what comes back:</p>
<ul>
<li>Locale = language, returned as source</li>
<li>Description = the text (the first line is everything, and then the rest are indiviudal words)</li>
<li>Bounding Poly = I’m sure you can guess by now</li>
</ul>
<pre><code>wrigley_text = getGoogleVisionResponse('wrigley_text.jpg',
                                   feature = 'TEXT_DETECTION')
head(wrigley_text)
</code></pre>
<pre><code>##   locale
## 1     en

##                                                                                                        description
## 1 RIGLEY F\nICHICAGO CUBS\nORDER ONLINE AT GIORDANOS.COM\nTOYOTA\nMIDWEST\nFENCE\n773-722-6616\nCAUTION\nCAUTION\n
                                                                                             ORDER
##                                 vertices
## 1   55, 657, 657, 55, 210, 210, 852, 852
## 2 343, 482, 484, 345, 217, 211, 260, 266

</code></pre>
<pre><code>wrigley_image &lt;- readImage('wrigley_text.jpg')
plot(wrigley_image)

for(i in 1:length(wrigley_text$boundingPoly$vertices)){
  xs = wrigley_text$boundingPoly$vertices[[i]]$x
  ys = wrigley_text$boundingPoly$vertices[[i]]$y
  polygon(x=xs,y=ys,border='green',lwd=2)
}
</code></pre>
<p><img alt="plot of chunk unnamed-chunk-19" src="https://i0.wp.com/www.stoltzmaniac.com/wp-content/uploads/2017/07/unnamed-chunk-19-1.png?w=620&ssl=1" data-recalc-dims="1"></p>
<hr>
<p>That’s about it for the basics of using the Google Vision API with the RoogleVision library. I highly recommend tinkering around with it a bit, especially because it won’t cost you a dime.</p>
<p>While I do enjoy the math under the hood and the thinking required to understand alrgorithms, I do think these sorts of API’s will become the way of the future for data science. Outside of specific use cases or special industries, it seems hard to imagine wanting to try and create algorithms that would be better than ones created for mass consumption. As long as they’re fast, free and accurate, I’m all about making my life easier! From the hiring perspective, I much prefer someone who can get the job done over someone who can slightly improve performance (as always, there are many cases where this doesn’t apply).</p>
<p>Please comment if you are utilizing any of the Google API’s for business purposes, I would love to hear it!</p>
<p>As always you can find this on my <a rel="nofollow" href="https://github.com/stoltzmaniac/ML-Image-Processing-R" target="_blank">GitHub</a></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.stoltzmaniac.com/google-vision-api-in-r-rooglevision/"> R-Projects – Stoltzmaniac</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/h38ytEfxNj8" height="1"></a></div>