<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/transfer-learning-with-keras-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://flovv.github.io/Logo_detection_transfer_learning/"> Florian Teschner</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In my last posts ([here](http://flovv.github.io/Logo_detection_deep_learning/ and <a rel="nofollow" href="http://flovv.github.io/Logo_detection_deep_learning_part2" target="_blank">here</a>, I described how one can detect logos in images with R. The first results were promising and achieved a classification accuracy of ~50%. In this post i will detail how to do transfer learning (using a pre-trained network) to further improve the classification accuracy.</p>
<p>The dataset is a combination of the <a rel="nofollow" href="http://image.ntua.gr/iva/datasets/flickr_logos" target="_blank">Flickr27-dataset</a>, with 270 images of 27 classes and self-scraped images from google image search. In case you want to reproduce the analysis, you can download the set <a rel="nofollow" href="https://github.com/flovv/flovv.github.io/blob/master/data/LogoData.zip" target="_blank">here</a>.</p>
<p>In addition to the previous post, this time I wanted to use pre-trained image models, to see how they perform on the task of identifing brand logos in images.<br>
First, I tried to adapt the official example on the <a rel="nofollow" href="https://rstudio.github.io/keras/articles/applications.html" target="_blank">Keras-rstudio website</a>.</p>
<p>Here is a copy of the instructions:</p>
<figure>
<pre><code data-lang="r"><span># create the base pre-trained model
</span><span>base_model</span><span> </span><span>&lt;-</span><span> </span><span>application_inception_v3</span><span>(</span><span>weights</span><span> </span><span>=</span><span> </span><span>'imagenet'</span><span>,</span><span> </span><span>include_top</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span># add our custom layers
</span><span>predictions</span><span> </span><span>&lt;-</span><span> </span><span>base_model</span><span>$</span><span>output</span><span> </span><span>%>%</span><span> </span><span>layer_global_average_pooling_2d</span><span>()</span><span> </span><span>%>%</span><span> </span><span>layer_dense</span><span>(</span><span>units</span><span> </span><span>=</span><span> </span><span>1024</span><span>,</span><span> </span><span>activation</span><span> </span><span>=</span><span> </span><span>'relu'</span><span>)</span><span> </span><span>%>%</span><span> </span><span>layer_dense</span><span>(</span><span>units</span><span> </span><span>=</span><span> </span><span>200</span><span>,</span><span> </span><span>activation</span><span> </span><span>=</span><span> </span><span>'softmax'</span><span>)</span><span>
</span><span># this is the model we will train
</span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>keras_model</span><span>(</span><span>inputs</span><span> </span><span>=</span><span> </span><span>base_model</span><span>$</span><span>input</span><span>,</span><span> </span><span>outputs</span><span> </span><span>=</span><span> </span><span>predictions</span><span>)</span><span>
</span><span># first: train only the top layers (which were randomly initialized)
# i.e. freeze all convolutional InceptionV3 layers
</span><span>for</span><span> </span><span>(</span><span>layer</span><span> </span><span>in</span><span> </span><span>base_model</span><span>$</span><span>layers</span><span>)</span><span>
</span><span>layer</span><span>$</span><span>trainable</span><span> </span><span>&lt;-</span><span> </span><span>FALSE</span><span>
</span><span># compile the model (should be done *after* setting layers to non-trainable)
</span><span>model</span><span> </span><span>%>%</span><span> </span><span>compile</span><span>(</span><span>optimizer</span><span> </span><span>=</span><span> </span><span>'rmsprop'</span><span>,</span><span> </span><span>loss</span><span> </span><span>=</span><span> </span><span>'categorical_crossentropy'</span><span>)</span><span>
</span><span># train the model on the new data for a few epochs
</span><span>model</span><span> </span><span>%>%</span><span> </span><span>fit_generator</span><span>(</span><span>...</span><span>)</span><span>
</span><span># at this point, the top layers are well trained and we can start fine-tuning
# convolutional layers from inception V3. We will freeze the bottom N layers
# and train the remaining top layers.
</span><span> </span><span># let's visualize layer names and layer indices to see how many layers
# we should freeze:
</span><span>layers</span><span> </span><span>&lt;-</span><span> </span><span>base_model</span><span>$</span><span>layers</span><span>
</span><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>length</span><span>(</span><span>layers</span><span>))</span><span>
</span><span>cat</span><span>(</span><span>i</span><span>,</span><span> </span><span>layers</span><span>[[</span><span>i</span><span>]]</span><span>$</span><span>name</span><span>,</span><span> </span><span>&quot;\n&quot;</span><span>)</span><span>
</span><span># we chose to train the top 2 inception blocks, i.e. we will freeze
# the first 172 layers and unfreeze the rest:
</span><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>172</span><span>)</span><span>
</span><span>layers</span><span>[[</span><span>i</span><span>]]</span><span>$</span><span>trainable</span><span> </span><span>&lt;-</span><span> </span><span>FALSE</span><span>
</span><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>173</span><span>:</span><span>length</span><span>(</span><span>layers</span><span>))</span><span>
</span><span>layers</span><span>[[</span><span>i</span><span>]]</span><span>$</span><span>trainable</span><span> </span><span>&lt;-</span><span> </span><span>TRUE</span><span>
</span><span># we need to recompile the model for these modifications to take effect
# we use SGD with a low learning rate
</span><span>model</span><span> </span><span>%>%</span><span> </span><span>compile</span><span>(</span><span>
</span><span>optimizer</span><span> </span><span>=</span><span> </span><span>optimizer_sgd</span><span>(</span><span>lr</span><span> </span><span>=</span><span> </span><span>0.0001</span><span>,</span><span> </span><span>momentum</span><span> </span><span>=</span><span> </span><span>0.9</span><span>),</span><span> </span><span>loss</span><span> </span><span>=</span><span> </span><span>'categorical_crossentropy'</span><span>
</span><span>)</span><span>
</span><span># we train our model again (this time fine-tuning the top 2 inception blocks
# alongside the top Dense layers
</span><span>model</span><span> </span><span>%>%</span><span> </span><span>fit_generator</span><span>(</span><span>...</span><span>)</span></code></pre>
</figure>
<p>It is basically a three step process; 1) load an existing model and add some layers, 2) train the extended model on your own data, 3) set more layers trainable and fine-tune the model on your own data.</p>
<p>I have to admit that I struggled to make it work on my data. That’s why I will try to detail my approach. Hopefully it helps you to get better started with transfer learning.</p>
<p>Here is the full example that I got to work:</p>
<figure>
<pre><code data-lang="r"><span>require</span><span>(</span><span>keras</span><span>)</span><span>
</span><span>### Keras transfer learning example
</span><span> </span><span>################### Section 1 #########################
</span><span>img_width</span><span> </span><span>&lt;-</span><span> </span><span>32</span><span>
</span><span>img_height</span><span> </span><span>&lt;-</span><span> </span><span>32</span><span>
</span><span>batch_size</span><span> </span><span>&lt;-</span><span> </span><span>8</span><span>
</span><span>train_directory</span><span> </span><span>&lt;-</span><span> </span><span>&quot;flickrData/train&quot;</span><span>
</span><span>test_directory</span><span> </span><span>&lt;-</span><span> </span><span>&quot;flickrData/test&quot;</span><span>
</span><span>train_generator</span><span> </span><span>&lt;-</span><span> </span><span>flow_images_from_directory</span><span>(</span><span>train_directory</span><span>,</span><span> </span><span>generator</span><span> </span><span>=</span><span> </span><span>image_data_generator</span><span>(),</span><span>
</span><span>target_size</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>img_width</span><span>,</span><span> </span><span>img_height</span><span>),</span><span> </span><span>color_mode</span><span> </span><span>=</span><span> </span><span>&quot;rgb&quot;</span><span>,</span><span>
</span><span>class_mode</span><span> </span><span>=</span><span> </span><span>&quot;categorical&quot;</span><span>,</span><span> </span><span>batch_size</span><span> </span><span>=</span><span> </span><span>batch_size</span><span>,</span><span> </span><span>shuffle</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>seed</span><span> </span><span>=</span><span> </span><span>123</span><span>)</span><span>
</span><span>validation_generator</span><span> </span><span>&lt;-</span><span> </span><span>flow_images_from_directory</span><span>(</span><span>test_directory</span><span>,</span><span> </span><span>generator</span><span> </span><span>=</span><span> </span><span>image_data_generator</span><span>(),</span><span>
</span><span>target_size</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>img_width</span><span>,</span><span> </span><span>img_height</span><span>),</span><span> </span><span>color_mode</span><span> </span><span>=</span><span> </span><span>&quot;rgb&quot;</span><span>,</span><span> </span><span>classes</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>class_mode</span><span> </span><span>=</span><span> </span><span>&quot;categorical&quot;</span><span>,</span><span> </span><span>batch_size</span><span> </span><span>=</span><span> </span><span>batch_size</span><span>,</span><span> </span><span>shuffle</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>seed</span><span> </span><span>=</span><span> </span><span>123</span><span>)</span><span>
</span><span>train_samples</span><span> </span><span>=</span><span> </span><span>498</span><span>
</span><span>validation_samples</span><span> </span><span>=</span><span> </span><span>177</span><span>
</span><span>################### Section 2 #########################
#base_model &lt;-application_inception_v3(weights = 'imagenet', include_top = FALSE)
</span><span>base_model</span><span> </span><span>&lt;-</span><span> </span><span>application_vgg16</span><span>(</span><span>weights</span><span> </span><span>=</span><span> </span><span>'imagenet'</span><span>,</span><span> </span><span>include_top</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>### use vgg16 - as inception won't converge --- </span><span> </span><span>################### Section 3 #########################
## add your custom layers
</span><span>predictions</span><span> </span><span>&lt;-</span><span> </span><span>base_model</span><span>$</span><span>output</span><span> </span><span>%>%</span><span> </span><span>layer_global_average_pooling_2d</span><span>(</span><span>trainable</span><span> </span><span>=</span><span> </span><span>T</span><span>)</span><span> </span><span>%>%</span><span> </span><span>layer_dense</span><span>(</span><span>64</span><span>,</span><span> </span><span>trainable</span><span> </span><span>=</span><span> </span><span>T</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>layer_activation</span><span>(</span><span>&quot;relu&quot;</span><span>,</span><span> </span><span>trainable</span><span> </span><span>=</span><span> </span><span>T</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>layer_dropout</span><span>(</span><span>0.4</span><span>,</span><span> </span><span>trainable</span><span> </span><span>=</span><span> </span><span>T</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>layer_dense</span><span>(</span><span>27</span><span>,</span><span> </span><span>trainable</span><span>=</span><span>T</span><span>)</span><span> </span><span>%>%</span><span> </span><span>## important to adapt to fit the 27 classes in the dataset!
</span><span> </span><span>layer_activation</span><span>(</span><span>&quot;softmax&quot;</span><span>,</span><span> </span><span>trainable</span><span>=</span><span>T</span><span>)</span><span>
</span><span># this is the model we will train
</span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>keras_model</span><span>(</span><span>inputs</span><span> </span><span>=</span><span> </span><span>base_model</span><span>$</span><span>input</span><span>,</span><span> </span><span>outputs</span><span> </span><span>=</span><span> </span><span>predictions</span><span>)</span><span>
</span><span>################### Section 4 #########################
</span><span>for</span><span> </span><span>(</span><span>layer</span><span> </span><span>in</span><span> </span><span>base_model</span><span>$</span><span>layers</span><span>)</span><span>
</span><span>layer</span><span>$</span><span>trainable</span><span> </span><span>&lt;-</span><span> </span><span>FALSE</span><span>
</span><span>################### Section 5 #########################
</span><span>model</span><span> </span><span>%>%</span><span> </span><span>compile</span><span>(</span><span>
</span><span>loss</span><span> </span><span>=</span><span> </span><span>&quot;categorical_crossentropy&quot;</span><span>,</span><span>
</span><span>optimizer</span><span> </span><span>=</span><span> </span><span>optimizer_rmsprop</span><span>(</span><span>lr</span><span> </span><span>=</span><span> </span><span>0.003</span><span>,</span><span> </span><span>decay</span><span> </span><span>=</span><span> </span><span>1e-6</span><span>),</span><span> </span><span>## play with the learning rate
</span><span> </span><span>metrics</span><span> </span><span>=</span><span> </span><span>&quot;accuracy&quot;</span><span>
</span><span>)</span><span>
</span><span>hist</span><span> </span><span>&lt;-</span><span> </span><span>model</span><span> </span><span>%>%</span><span> </span><span>fit_generator</span><span>(</span><span>
</span><span>train_generator</span><span>,</span><span>
</span><span>steps_per_epoch</span><span> </span><span>=</span><span> </span><span>as.integer</span><span>(</span><span>train_samples</span><span>/</span><span>batch_size</span><span>),</span><span> </span><span>epochs</span><span> </span><span>=</span><span> </span><span>20</span><span>,</span><span> </span><span>validation_data</span><span> </span><span>=</span><span> </span><span>validation_generator</span><span>,</span><span>
</span><span>validation_steps</span><span> </span><span>=</span><span> </span><span>as.integer</span><span>(</span><span>validation_samples</span><span>/</span><span>batch_size</span><span>),</span><span>
</span><span>verbose</span><span>=</span><span>2</span><span>
</span><span>)</span><span>
</span><span>### saveable data frame obejct.
</span><span>histDF</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>acc</span><span> </span><span>=</span><span> </span><span>unlist</span><span>(</span><span>hist</span><span>$</span><span>history</span><span>$</span><span>acc</span><span>),</span><span> </span><span>val_acc</span><span>=</span><span>unlist</span><span>(</span><span>hist</span><span>$</span><span>history</span><span>$</span><span>val_acc</span><span>),</span><span> </span><span>val_loss</span><span> </span><span>=</span><span> </span><span>unlist</span><span>(</span><span>hist</span><span>$</span><span>history</span><span>$</span><span>val_loss</span><span>),</span><span>loss</span><span> </span><span>=</span><span> </span><span>unlist</span><span>(</span><span>hist</span><span>$</span><span>history</span><span>$</span><span>loss</span><span>))</span></code></pre>
</figure>
<p>The key change to the Rstudio sample code is to use a different pre-trained model. I use VGG16 over Inception_v3 as the later would work. (The Inception-model would not pick up any information and accuracy remains around the base rate.)<br>
But let’s see what the code does step by step.<br>
In section 1 the image data is prepared and loaded. One that setting that is important to set while loading the images is “class_mode = “categorical””, as we have 27 different image classes/labels to assign. In section 2 we load the pretrained image model.<br>
In section 3 we add custom layers. (In this case I copied the last layers from the previous post.) It is important to set the last layers to the number of labels (27) and the activation function to softmax.<br>
In section 4 we set the layers of the loaded image model to non-trainable.<br>
Section 5 sets the optimiser and finally trains the model. If the training process does not show improvements in terms of decreasing loss, try to increase the learning rate.<br>
The learning process is documented in the hist-object, which can be easily plotted.</p>
<p><img alt="plot of chunk unnamed-chunk-19" src="https://i1.wp.com/flovv.github.io/figures/post23/unnamed-chunk-19-1.png?w=456" data-recalc-dims="1"></p>
<p>After 50 Traing-epochs the accuracy is at 55% on the training 35% on the validation set.<br>
I assume that the accuracy can be further improved by training the full model or at least set more layers trainable and fine tune the full model as it is detailed in the R-Studio case. However, so far I tried various parameter settings without success. There is basically no improvement on the established 50%, yet.</p>
<p>I will update the post, as soon as I have figured it out.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://flovv.github.io/Logo_detection_transfer_learning/"> Florian Teschner</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/rUmnETTCUsA" height="1"></a></div>