<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/using-microsofts-azure-face-api-to-analyze-videos-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://r-posts.com/using-microsofts-azure-face-api-to-analyze-videos-in-r/"> R-posts.com</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Microsoft had a cool API called “Emotion API”. With it you could submit a URL of a video, and the API would return a json file with the faces and emotions expressed in the video (per frame). However, that API never matured from preview mode, and in fact was deprecated on October 30th 2017 (it no longer works).</p>
<p> I stumbled upon the Emotion API when I read a post by <a rel="nofollow" href="https://blog.exploratory.io/analyzing-emotions-using-facial-expressions-in-video-with-microsoft-ai-and-r-8f7585dd0780" target="_blank">Kan Nishida from exploritory.io</a>, which analyzed the facial expressions of Trump and Clinton during a presedential debate last year.  However, that tutorial (like many others) no longer works, since it used the old Emotion API.</p>
<p> Lately I needed a tool for an analysis I did at work on the facial expressions in TV commercials. I had a list of videos showing faces, and I needed to code these faces into emotions.</p>
<p> I noticed that Microsft still offers a simpler “face API”. This API doesn’t work with videos, it only runs on still images (e.g. jpegs). I decided to use it and here are the results (bottom line – you can use it for videos after some prep work).</p>
<p> By the way, AWS and google have similar APIs (for images) called: Amazon Rekognition (not a typo) and Vision API respectively.</p>
<p> Here is guide on how to do a batch analysis of videos and turn them into a single data frame (or tibble) of emotions which are displayed in the videos per frame.</p>
<p> To use the API you need a key – if you don’t already have it, register to Microsoft’s Azure API and register a new service of type <a rel="nofollow" href="https://azure.microsoft.com/en-us/services/cognitive-services/face/" target="_blank">Face API</a>. You get an initial “gratis” credit of about $200 (1,000 images cost $1.5 so $200 is more than enough).</p>
<h3>Preparations</h3>
<p>First, we’ll load the packages we’re going to use <code>httr</code> to send our requests to the server, and <code>tidyverse</code> (mostly for <code>ggplot</code>, <code>dplyr</code>, <code>tidyr</code>, <code>tibble</code>). Also, lets define the API access point we will use, and the appropriate key. My face API service was hosted on west Europe (hence the URL starts with westeurope.api.</p>
<pre><code># ==== Load required libraries ====
library(tidyverse)
library(httr)
# ==== Microsoft's Azure Face API ====
end.point &lt;- &quot;https://westeurope.api.cognitive.microsoft.com/face/v1.0/detect&quot;
key1 &lt;- &quot;PUT YOUR SECRET KEY HERE&quot;</code></pre>
<p>To get things going, lets check that the API and key works. We’ll send a simple image (of me) to the API and see what comes out.</p>
<pre><code>sample.img.simple &lt;- POST(url = end.point,
                          add_headers(.headers = c(&quot;Ocp-Apim-Subscription-Key&quot; = key1)),
                          body = '{&quot;url&quot;:&quot;http://www.sarid-ins.co.il/files/TheTeam/Adi_Sarid.jpg&quot;}',
                          query = list(returnFaceAttributes = &quot;emotion&quot;),
                          accept_json())</code></pre>
<p>This is the simplest form of the API, which returns only emotions of the faces depicted in the image. You can ask for a lot of other features by setting them in the query parameter. For example, to get the emotions, age, gender, hair, makeup and accessories use <code>returnFaceAttributes = &quot;emotion,age,gender,hair,makeup,accessories&quot;</code>. </p>
<p> Here’s a <a rel="nofollow" href="https://westus.dev.cognitive.microsoft.com/docs/services/563879b61984550e40cbbe8d/operations/563879b61984550f30395236" target="_blank">full documentation of what you can get</a>.</p>
<p> Later on we’ll change the body parameter at the <code>POST</code> from a json which contains a URL to a local image file which will be uploaded to Microsoft’s servers.</p>
<p> For now, lets look at the response of this query (notice the reference is for the first identified face <code>[[1]]</code>, for an image with more faces, face <code>i</code> will appear in location <code>[[i]]</code>).</p>
<pre><code>as_tibble(content(sample.img.simple)[[1]]$faceAttributes$emotion) %>% t()
</code></pre>
<pre>##            [,1]
## anger     0.001
## contempt  0.062
## disgust   0.002
## fear      0.000
## happiness 0.160
## neutral   0.774
## sadness   0.001
## surprise  0.001</pre>
<p>You can see that the API is pretty sure I’m showing a neutral face (0.774), but I might also be showing a little bit of happiness (0.160). Anyway these are weights (probabilities to be exact), they will always sum up to 1. If you want a single classification you should probably choose the highest weight as the classified emotion. Other results such as gender, hair color, work similarly.</p>
<p> Now we’re ready to start working with videos. We’ll be building a number of functions to automate the process.</p>
<h3>Splitting a video to individual frames</h3>
<p>To split the video file into individual frames (images which we can send to the API), I’m going to (locally) use <a rel="nofollow" href="http://www.ffmpeg.org/" target="_blank">ffmpeg</a> by calling it from R (it is run externally by the system – I’m using windows for this). Assume that <code>file.url</code> contains the location of the video (can be online or local), and that <code>id.number</code> is a unique string identifier of the video.</p>
<pre><code>movie2frames &lt;- function(file.url, id.number){
  base.dir &lt;- &quot;d:/temp/facial_coding/&quot;
  dir.create(paste0(base.dir, id.number))
  system(
    paste0(
      &quot;ffmpeg -i &quot;, file.url, 
      &quot; -vf fps=2 &quot;, base.dir, 
      id.number, &quot;/image%07d.jpg&quot;)
        )
}</code></pre>
<p>The parameter <code>fps=2</code> in the command means that we are extracting two frames per second (for my needs that was a reasonable fps res, assuming that emotions don’t change that much during 1/2 sec).<br>
Be sure to change the directory location (<code>base.dir</code> from <code>d:/temp/facial_coding/</code>) to whatever you need. This function will create a subdirectory within the <code>base.dir</code>, with all the frames extracted by ffmpeg. Now were ready to send these frames to the API.</p>
<h3>A function for sending a (single) image and reading back emotions</h3>
<p>Now, I defined a function for sending an image to the API and getting back the results. You’ll notice that I’m only using a very small portion of what the API has to offer (only the faces). For the simplicity of the example, I’m reading only the first face (there might be more than one on a single image).</p>
<pre>send.face &lt;- function(filename) {
  face_res &lt;- POST(url = end.point,
                   add_headers(.headers = c(&quot;Ocp-Apim-Subscription-Key&quot; = key1)),
                   body = upload_file(filename, &quot;application/octet-stream&quot;),
                   query = list(returnFaceAttributes = &quot;emotion&quot;),
                   accept_json())
 
  if(length(content(face_res)) > 0){
    ret.expr &lt;- as_tibble(content(face_res)[[1]]$faceAttributes$emotion)
  } else {
    ret.expr &lt;- tibble(contempt = NA,
                       disgust = NA,
                       fear = NA,
                       happiness = NA,
                       neutral = NA,
                       sadness = NA,
                       surprise = NA)
   }
  return(ret.expr)
}</pre>
<h3>A function to process a batch of images</h3>
<p>As I mentioned, in my case I had videos, so I had to work with a batch of images (each image representing a frame in the original video). After splitting the video, we now have a directory full of jpgs and we want to send them all to analysis. Thus, another function is required to automate the use of <code>send.face()</code> (the function we had just defined).</p>
<pre>extract.from.frames &lt;- function(directory.location){
  base.dir &lt;- &quot;d:/temp/facial_coding/&quot;
  # enter directory location without ending &quot;/&quot;
  face.analysis &lt;- dir(directory.location) %>%
    as_tibble() %>%
    mutate(filename = paste0(directory.location,&quot;/&quot;, value)) %>%
    group_by(filename) %>%
    do(send.face(.$filename)) %>%
    ungroup() %>%
    mutate(frame.num = 1:NROW(filename)) %>%
    mutate(origin = directory.location)
  
  # Save temporary data frame for later use (so not to loose data if do() stops/fails)
  temp.filename &lt;- tail(stringr::str_split(directory.location, stringr::fixed(&quot;/&quot;))[[1]],1)
  write_excel_csv(x = face.analysis, path = paste0(base.dir, temp.filename, &quot;.csv&quot;))
  
  return(face.analysis)
}
</pre>
<p>The second part of the function (starting from “# Save temporary data frame for later use…”) is not mandatory. I wanted the results saved per frame batch into a file, since I used this function for a lot of movies (and you don’t want to loose everything if something temporarily doesn’t work). Again, if you do want the function to save its results to a file, be sure to change <code>base.dir</code> in <code>extract.from.frames</code> as well – to suit your own location.</p>
<p> By the way, note the use of <code>do()</code>. I could also probably use <code>walk()</code>, but <code>do()</code> has the benefit of showing you a nice progress bar while it processes the data. </p>
<p> Once you call this <code>results.many.frames &lt;- extract.from.frames(&quot;c:/some/directory&quot;)</code>, you will receive a nice tibble that looks like this one:</p>
<pre>## Observations: 796
## Variables: 11
## $ ..filename  d:/temp/facial_coding/119/image00001.jpg, d:/temp...
## $ anger        0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0...
## $ contempt     0.001, 0.001, 0.000, 0.001, 0.002, 0.001, 0.001, 0...
## $ disgust      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
## $ fear         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
## $ happiness    0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0...
## $ neutral      0.998, 0.998, 0.999, 0.993, 0.996, 0.997, 0.997, 0...
## $ sadness      0.000, 0.000, 0.000, 0.001, 0.001, 0.000, 0.000, 0...
## $ surprise     0.001, 0.001, 0.000, 0.005, 0.001, 0.002, 0.001, 0...
## $ frame.num    1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,...
## $ origin       d:/temp/facial_coding/119, d:/temp/facial_coding/...
</pre>
<h3>Visualization</h3>
<p>Here is the visualization of the emotion classification which were detected in this movie, as a function of frame number.</p>
<pre>res.for.gg &lt;- results.many.frames %>%
 select(anger:frame.num) %>%
 gather(key = emotion, value = intensity, -frame.num)
 glimpse(res.for.gg)</pre>
<pre> ## Observations: 6,368
 ## Variables: 3
 ## $ frame.num </pre>