<h1></h1><h1><div></div><div>Recently RStudio has released a <a href="https://rstudio.github.io/tensorflow/">package</a> that allows to use TensorFlow in R. Also recently several trained models for image classification have been <a href="https://github.com/tensorflow/models/blob/master/slim/README.md#Pretrained">released</a>. In this post I describe how to use the VGG16 model in R to produce an image classification like this:</div><div><a href="https://dl.dropboxusercontent.com/u/9154523/blog/Image_Classification_in%20_R/poodle_result.png"></a></div><div><br></div><div>(image taken from: <a href="https://www.flickr.com/photos/jonner/487461265">https://www.flickr.com/photos/jonner/487461265</a>)</div><div>The code is available on <a href="https://github.com/oduerr/tf_r/tree/master/load_vgg16">github</a>. In that directory there is also a python file <a href="https://github.com/oduerr/tf_r/tree/master/load_vgg16"><code>load_vgg16.py</code></a> for checking the validity of the R-code against the python implementation in which the models are published.</div><div>As a first step we download the VGG16 weights <code>vgg_16.tar.gz</code> from <a href="https://github.com/tensorflow/models/blob/master/slim/README.md#Pretrained">here</a> and extract it. You should get a file named <code>vgg_16.ckpt</code> which we will need later.</div></h1><h2>Building the model</h2><h1><div><div>We now define the model. Note that since the network is written in the default graph, we do a clean start by resetting the default graph.</div><pre><code><span>library</span><span>(</span><span>tensorflow</span><span>)</span><br><span>slim</span> <span>=</span> <span>tf</span><span>$</span><span>contrib</span><span>$</span><span>slim</span> <span>#Poor mans import tensorflow.contrib.slim as slim</span><br><span>tf</span><span>$</span><span>reset_default_graph</span><span>(</span><span>)</span> <span># Better to start from scratch</span></code></pre><div>We start with a placeholder <a href="https://rstudio.github.io/tensorflow/using_tensorflow_api.html#tensor_shapes">tensor</a> in which we later feed the images. The model works on a batch of images and thus needs a tensor of order 4 (an array having 4 indices). The first index of the tensor counts the image number and the second to 4th index is for the width, height, color. Since we want to allow for an arbitrary number of images of arbitrary size, we leave these dimensions open. We only specify that there should be 3 color channels (rgb). Then these images are rescaled with TensorFlow to the size (224, 224) as needed by the network.</div><pre><code><span># Resizing the images</span><br><span>images</span> <span>=</span> <span>tf</span><span>$</span><span>placeholder</span><span>(</span><span>tf</span><span>$</span><span>float32</span>, <span>shape</span><span>(</span><span>NULL</span>, <span>NULL</span>, <span>NULL</span>, <span>3</span><span>)</span><span>)</span><br><span>imgs_scaled</span> <span>=</span> <span>tf</span><span>$</span><span>image</span><span>$</span><span>resize_images</span><span>(</span><span>images</span>, <span>shape</span><span>(</span><span>224</span>,<span>224</span><span>)</span><span>)</span></code></pre><div>We are now defining the VGG16 model. Luckily there is a package <a href="https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/slim">TensorFlow-Slim</a> included in the TensorFlow installation, which allows to easily build networks.</div><pre><code><span># Definition of the network</span><br><span>library</span><span>(</span><span>magrittr</span><span>)</span> <br><span># The last layer is the fc8 Tensor holding the logits of the 1000 classes</span><br><span>fc8</span> <span>=</span> <span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>imgs_scaled</span>, <span>64</span>, <span>shape</span><span>(</span><span>3</span>,<span>3</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/conv1/conv1_1'</span><span>)</span> <span>%>%</span> <br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>64</span>, <span>shape</span><span>(</span><span>3</span>,<span>3</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/conv1/conv1_2'</span><span>)</span>  <span>%>%</span><br><span>slim</span><span>$</span><span>max_pool2d</span><span>(</span> <span>shape</span><span>(</span><span>2</span>, <span>2</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/pool1'</span><span>)</span>  <span>%>%</span><br><br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>128</span>, <span>shape</span><span>(</span><span>3</span>,<span>3</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/conv2/conv2_1'</span><span>)</span>  <span>%>%</span><br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>128</span>, <span>shape</span><span>(</span><span>3</span>,<span>3</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/conv2/conv2_2'</span><span>)</span>  <span>%>%</span><br><span>slim</span><span>$</span><span>max_pool2d</span><span>(</span> <span>shape</span><span>(</span><span>2</span>, <span>2</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/pool2'</span><span>)</span>  <span>%>%</span><br><br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>256</span>, <span>shape</span><span>(</span><span>3</span>,<span>3</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/conv3/conv3_1'</span><span>)</span>  <span>%>%</span><br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>256</span>, <span>shape</span><span>(</span><span>3</span>,<span>3</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/conv3/conv3_2'</span><span>)</span>  <span>%>%</span><br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>256</span>, <span>shape</span><span>(</span><span>3</span>,<span>3</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/conv3/conv3_3'</span><span>)</span>  <span>%>%</span><br><span>slim</span><span>$</span><span>max_pool2d</span><span>(</span><span>shape</span><span>(</span><span>2</span>, <span>2</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/pool3'</span><span>)</span>  <span>%>%</span><br><br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>512</span>, <span>shape</span><span>(</span><span>3</span>,<span>3</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/conv4/conv4_1'</span><span>)</span>  <span>%>%</span><br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>512</span>, <span>shape</span><span>(</span><span>3</span>,<span>3</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/conv4/conv4_2'</span><span>)</span>  <span>%>%</span><br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>512</span>, <span>shape</span><span>(</span><span>3</span>,<span>3</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/conv4/conv4_3'</span><span>)</span>  <span>%>%</span><br><span>slim</span><span>$</span><span>max_pool2d</span><span>(</span><span>shape</span><span>(</span><span>2</span>, <span>2</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/pool4'</span><span>)</span>  <span>%>%</span><br><br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>512</span>, <span>shape</span><span>(</span><span>3</span>,<span>3</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/conv5/conv5_1'</span><span>)</span>  <span>%>%</span><br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>512</span>, <span>shape</span><span>(</span><span>3</span>,<span>3</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/conv5/conv5_2'</span><span>)</span>  <span>%>%</span><br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>512</span>, <span>shape</span><span>(</span><span>3</span>,<span>3</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/conv5/conv5_3'</span><span>)</span>  <span>%>%</span><br><span>slim</span><span>$</span><span>max_pool2d</span><span>(</span><span>shape</span><span>(</span><span>2</span>, <span>2</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/pool5'</span><span>)</span>  <span>%>%</span><br><br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>4096</span>, <span>shape</span><span>(</span><span>7</span>, <span>7</span><span>)</span>, <span>padding</span><span>=</span><span>'VALID'</span>, <span>scope</span><span>=</span><span>'vgg_16/fc6'</span><span>)</span>  <span>%>%</span><br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>4096</span>, <span>shape</span><span>(</span><span>1</span>, <span>1</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/fc7'</span><span>)</span> <span>%>%</span> <br><br><span># Setting the activation_fn=NULL does not work, so we get a ReLU</span><br><span>slim</span><span>$</span><span>conv2d</span><span>(</span><span>1000</span>, <span>shape</span><span>(</span><span>1</span>, <span>1</span><span>)</span>, <span>scope</span><span>=</span><span>'vgg_16/fc8'</span><span>)</span>  <span>%>%</span><br><span>tf</span><span>$</span><span>squeeze</span><span>(</span><span>shape</span><span>(</span><span>1</span>, <span>2</span><span>)</span>, <span>name</span><span>=</span><span>'vgg_16/fc8/squeezed'</span><span>)</span></code></pre><div>We can visualize the model in tensorboard, by saving the default graph via:</div><pre><code><span>tf</span><span>$</span><span>train</span><span>$</span><span>SummaryWriter</span><span>(</span><span>'/tmp/dumm/vgg16'</span>, <span>tf</span><span>$</span><span>get_default_graph</span><span>(</span><span>)</span><span>)</span><span>$</span><span>close</span><span>(</span><span>)</span></code></pre><div>You can now open a shell and start tensorboard</div><pre><code>  tensorboard --logdir /tmp/dumm/</code></pre><div>You should get a result like: </div><div><a href="https://dl.dropboxusercontent.com/u/9154523/blog/Image_Classification_in%20_R/tensorboard.png"></a></div><div><br></div></div></h1><h2>Loading the weights</h2><h1><div><div>We start a Session and restore the model weights from the downloaded weight file.</div><pre><code>  <span>restorer</span> <span>=</span> <span>tf</span><span>$</span><span>train</span><span>$</span><span>Saver</span><span>(</span><span>)</span><br><span>sess</span> <span>=</span> <span>tf</span><span>$</span><span>Session</span><span>(</span><span>)</span><br><span>restorer</span><span>$</span><span>restore</span><span>(</span><span>sess</span>, <span>'/Users/oli/Dropbox/server_sync/tf_slim_models/vgg_16.ckpt'</span><span>)</span></code></pre></div></h1><h2>Loading the images</h2><h1><div><div>Now it’s time to load the image. The values have to be in the range of 0 to 255. Therefore I multiply the values by 255. Further, we need to feed the placeholder Tensor with an array of order 4.</div><pre><code><span>library</span><span>(</span><span>jpeg</span><span>)</span><br><span>img1</span> <span>&lt;-</span> <span>readJPEG</span><span>(</span><span>'apple.jpg'</span><span>)</span><br><span>d</span> <span>=</span> <span>dim</span><span>(</span><span>img1</span><span>)</span><br><span>imgs</span> <span>=</span> <span>array</span><span>(</span><span>255</span><span>*</span><span>img1</span>, <span>dim</span> <span>=</span> <span>c</span><span>(</span><span>1</span>, <span>d</span><span>[</span><span>1</span><span>]</span>, <span>d</span><span>[</span><span>2</span><span>]</span>, <span>d</span><span>[</span><span>3</span><span>]</span><span>)</span><span>)</span> <span>#We need array of order 4</span></code></pre></div></h1><h2>Feeding and fetching the graph</h2><h1><div><div>Now we have a graph in the session with the correct weights. We can do the predictions by feeding the placeholder tensor <code>images</code> with the value of the images stored in the array <code>imgs</code>. We fetch the <code>fc8</code> tensor from the graph and store it in <code>fc8_vals</code>.</div><pre><code><span>fc8_vals</span> <span>=</span> <span>sess</span><span>$</span><span>run</span><span>(</span><span>fc8</span>, <span>dict</span><span>(</span><span>images</span> <span>=</span> <span>imgs</span><span>)</span><span>)</span><br><span>fc8_vals</span><span>[</span><span>1</span><span>:</span><span>5</span><span>]</span> <span>#In python [-2.86833096 0.7060132 -1.32027602 -0.61107934 -1.67312801]</span></code></pre><pre><code>## [1] 0.0000000 0.7053483 0.0000000 0.0000000 0.0000000</code></pre><div>When comparing it with the python result, we see that negative values are clamped to zero. This is due to the fact that in this R implementation I could not deactivate the final ReLu operation. Nevertheless, we are only interested in the positive values which we transfer to probabilities for the certain classes via</div><pre><code><span>probs</span> <span>=</span> <span>exp</span><span>(</span><span>fc8_vals</span><span>)</span><span>/</span><span>sum</span><span>(</span><span>exp</span><span>(</span><span>fc8_vals</span><span>)</span><span>)</span></code></pre><div>We sort for the highest probabilities and also load the descriptions of the image net classes and produce the final plot.</div><pre><code><span>idx</span> <span>=</span> <span>sort.int</span><span>(</span><span>fc8_vals</span>, <span>index.return</span> <span>=</span> <span>TRUE</span>, <span>decreasing</span> <span>=</span> <span>TRUE</span><span>)</span><span>$</span><span>ix</span><span>[</span><span>1</span><span>:</span><span>5</span><span>]</span><br><br><span># Reading the class names</span><br><span>library</span><span>(</span><span>readr</span><span>)</span><br><span>names</span> <span>=</span> <span>read_delim</span><span>(</span><span>&quot;imagenet_classes.txt&quot;</span>, <span>&quot;\t&quot;</span>, <span>escape_double</span> <span>=</span> <span>FALSE</span>, <span>trim_ws</span> <span>=</span> <span>TRUE</span>,<span>col_names</span> <span>=</span> <span>FALSE</span><span>)</span></code></pre><pre><code><span>### Graph</span><br><span>library</span><span>(</span><span>grid</span><span>)</span><br><span>g</span> <span>=</span> <span>rasterGrob</span><span>(</span><span>img1</span>, <span>interpolate</span><span>=</span><span>TRUE</span><span>)</span> <br><span>text</span> <span>=</span> <span>&quot;&quot;</span><br><span>for</span> <span>(</span><span>id</span> <span>in</span> <span>idx</span><span>)</span> <span>{</span><br><span>text</span> <span>=</span> <span>paste0</span><span>(</span><span>text</span>, <span>names</span><span>[</span><span>id</span>,<span>]</span><span>[</span><span>[</span><span>1</span><span>]</span><span>]</span>, <span>&quot; &quot;</span>, <span>round</span><span>(</span><span>probs</span><span>[</span><span>id</span><span>]</span>,<span>5</span><span>)</span>, <span>&quot;\n&quot;</span><span>)</span> <br><span>}</span><br><br><span>library</span><span>(</span><span>ggplot2</span><span>)</span><br><span>ggplot</span><span>(</span><span>data.frame</span><span>(</span><span>d</span><span>=</span><span>1</span><span>:</span><span>3</span><span>)</span><span>)</span> <span>+</span> <span>annotation_custom</span><span>(</span><span>g</span><span>)</span> <span>+</span> <br><span>annotate</span><span>(</span><span>'text'</span>,<span>x</span><span>=</span><span>0.05</span>,<span>y</span><span>=</span><span>0.05</span>,<span>label</span><span>=</span><span>text</span>, <span>size</span><span>=</span><span>7</span>, <span>hjust</span> <span>=</span> <span>0</span>, <span>vjust</span><span>=</span><span>0</span>, <span>color</span><span>=</span><span>'blue'</span><span>)</span> <span>+</span> <span>xlim</span><span>(</span><span>0</span>,<span>1</span><span>)</span> <span>+</span> <span>ylim</span><span>(</span><span>0</span>,<span>1</span><span>)</span> </code></pre><div><a href="https://dl.dropboxusercontent.com/u/9154523/blog/Image_Classification_in%20_R/unnamed-chunk-9-1.png"></a></div><div>Now since we can load trained models, we can do many cool things like transfer learning etc. More maybe another time.</div><div><br></div></div></h1>