<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/transfer-learning-with-augmented-data-for-logo-detection/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://flovv.github.io/Logo_detection_transfer_learning_part2/"> Florian Teschner</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>The last months, I have worked on brand logo detection in R with Keras. Starting with a <a rel="nofollow" href="http://flovv.github.io/Logo_detection_deep_learning/" target="_blank">model from scratch </a> <a rel="nofollow" href="http://flovv.github.io/Logo_detection_deep_learning_part2/" target="_blank">adding more data</a> and <a rel="nofollow" href="http://flovv.github.io/Logo_detection_transfer_learning/" target="_blank">using a pretrained model</a>. The goal is to build a (deep) neural net that is able to identify brand logos in images.</p>
<p>Just to recall, the dataset is a combination of the <a rel="nofollow" href="http://image.ntua.gr/iva/datasets/flickr_logos" target="_blank">Flickr27-dataset</a>, with 270 images of 27 classes and self-scraped images from google image search. In case you want to reproduce the analysis, you can download the set <a rel="nofollow" href="https://github.com/flovv/flovv.github.io/blob/master/data/LogoData.zip" target="_blank">here</a>.</p>
<p>In the last post, I used the VGG-16 pretrained model and showed that it can be trained to achieve an accuracy of 55% on the training 35% on the validation set.<br>
In this post, I will show how to further improve the model accuracy.</p>
<p>Keras (in R) provides a set of pretrained models:</p>
<ul>
<li>Xception</li>
<li>VGG16</li>
<li>VGG19</li>
<li>ResNet50</li>
<li>InceptionV3</li>
<li>MobileNet</li>
</ul>
<p>Naturally, it raises the question which model is best suited for the task at hand.<br>
The article <a rel="nofollow" href="https://www.analyticsvidhya.com/blog/2017/08/10-advanced-deep-learning-architectures-data-scientists/" target="_blank">10 advanced deep learning architectures</a> points out that Google Xception model performs better than VGG in transfer learning cases.</p>
<p>In addition to changing the pre-trained model, I wanted to see how data augmentation changes the results.<br>
The function “image_data_generator” takes the input data and randomly alters the original training images.</p>
<p>Here is the code:</p>
<figure>
<pre><code data-lang="r"><span>require</span><span>(</span><span>keras</span><span>)</span><span>
</span><span>### Xception Transfer-learning example
</span><span>img_width</span><span> </span><span>&lt;-</span><span> </span><span>75</span><span>
</span><span>img_height</span><span> </span><span>&lt;-</span><span> </span><span>75</span><span>
</span><span>batch_size</span><span> </span><span>&lt;-</span><span> </span><span>8</span><span>
</span><span>train_directory</span><span> </span><span>&lt;-</span><span> </span><span>&quot;flickrData/train&quot;</span><span>
</span><span>test_directory</span><span> </span><span>&lt;-</span><span> </span><span>&quot;flickrData/test&quot;</span><span>
</span><span>train_generator</span><span> </span><span>&lt;-</span><span> </span><span>flow_images_from_directory</span><span>(</span><span>train_directory</span><span>,</span><span> </span><span>generator</span><span> </span><span>=</span><span> </span><span>image_data_generator</span><span>(),</span><span>
</span><span>target_size</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>img_width</span><span>,</span><span> </span><span>img_height</span><span>),</span><span> </span><span>color_mode</span><span> </span><span>=</span><span> </span><span>&quot;rgb&quot;</span><span>,</span><span>
</span><span>class_mode</span><span> </span><span>=</span><span> </span><span>&quot;categorical&quot;</span><span>,</span><span> </span><span>batch_size</span><span> </span><span>=</span><span> </span><span>batch_size</span><span>,</span><span> </span><span>shuffle</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>seed</span><span> </span><span>=</span><span> </span><span>123</span><span>)</span><span>
</span><span>validation_generator</span><span> </span><span>&lt;-</span><span> </span><span>flow_images_from_directory</span><span>(</span><span>test_directory</span><span>,</span><span> </span><span>generator</span><span> </span><span>=</span><span> </span><span>image_data_generator</span><span>(),</span><span> </span><span>target_size</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>img_width</span><span>,</span><span> </span><span>img_height</span><span>),</span><span> </span><span>color_mode</span><span> </span><span>=</span><span> </span><span>&quot;rgb&quot;</span><span>,</span><span> </span><span>classes</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span> </span><span>class_mode</span><span> </span><span>=</span><span> </span><span>&quot;categorical&quot;</span><span>,</span><span> </span><span>batch_size</span><span> </span><span>=</span><span> </span><span>batch_size</span><span>,</span><span> </span><span>shuffle</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>seed</span><span> </span><span>=</span><span> </span><span>123</span><span>)</span><span>
</span><span>train_samples</span><span> </span><span>=</span><span> </span><span>498</span><span>
</span><span>validation_samples</span><span> </span><span>=</span><span> </span><span>177</span><span>
</span><span>########### generator to enhance the dataset, artificially alter the original data ################
</span><span>datagen</span><span> </span><span>&lt;-</span><span> </span><span>image_data_generator</span><span>(</span><span>
</span><span>rotation_range</span><span> </span><span>=</span><span> </span><span>20</span><span>,</span><span>
</span><span>width_shift_range</span><span> </span><span>=</span><span> </span><span>0.2</span><span>,</span><span>
</span><span>height_shift_range</span><span> </span><span>=</span><span> </span><span>0.2</span><span>,</span><span>
</span><span>horizontal_flip</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>
</span><span>)</span><span>
</span><span>train_augmented_generator</span><span> </span><span>&lt;-</span><span> </span><span>flow_images_from_directory</span><span>(</span><span>test_directory</span><span>,</span><span> </span><span>generator</span><span> </span><span>=</span><span> </span><span>datagen</span><span>,</span><span>
</span><span>target_size</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>img_width</span><span>,</span><span> </span><span>img_height</span><span>),</span><span> </span><span>color_mode</span><span> </span><span>=</span><span> </span><span>&quot;rgb&quot;</span><span>,</span><span> </span><span>classes</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span> </span><span>class_mode</span><span> </span><span>=</span><span> </span><span>&quot;categorical&quot;</span><span>,</span><span> </span><span>batch_size</span><span> </span><span>=</span><span> </span><span>batch_size</span><span>,</span><span> </span><span>shuffle</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>seed</span><span> </span><span>=</span><span> </span><span>123</span><span>)</span><span>
</span><span>## define the pretrained model, here: Xception </span><span>base_model</span><span> </span><span>&lt;-</span><span> </span><span>application_xception</span><span>(</span><span>weights</span><span> </span><span>=</span><span> </span><span>'imagenet'</span><span>,</span><span> </span><span>include_top</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>input_shape</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>img_width</span><span>,</span><span> </span><span>img_height</span><span>,</span><span> </span><span>3</span><span>))</span><span>
</span><span>predictions</span><span> </span><span>&lt;-</span><span> </span><span>base_model</span><span>$</span><span>output</span><span> </span><span>%>%</span><span> </span><span>layer_global_average_pooling_2d</span><span>(</span><span>trainable</span><span> </span><span>=</span><span> </span><span>T</span><span>)</span><span> </span><span>%>%</span><span> </span><span>layer_dense</span><span>(</span><span>64</span><span>,</span><span> </span><span>trainable</span><span> </span><span>=</span><span> </span><span>T</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>layer_activation</span><span>(</span><span>&quot;relu&quot;</span><span>,</span><span> </span><span>trainable</span><span> </span><span>=</span><span> </span><span>T</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>layer_dropout</span><span>(</span><span>0.4</span><span>,</span><span> </span><span>trainable</span><span> </span><span>=</span><span> </span><span>T</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>layer_dense</span><span>(</span><span>27</span><span>,</span><span> </span><span>trainable</span><span>=</span><span>T</span><span>)</span><span> </span><span>%>%</span><span> </span><span>## important to adapt to fit the 27 classes in the dataset!
</span><span> </span><span>layer_activation</span><span>(</span><span>&quot;softmax&quot;</span><span>,</span><span> </span><span>trainable</span><span>=</span><span>T</span><span>)</span><span>
</span><span># this is the model we will train
</span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>keras_model</span><span>(</span><span>inputs</span><span> </span><span>=</span><span> </span><span>base_model</span><span>$</span><span>input</span><span>,</span><span> </span><span>outputs</span><span> </span><span>=</span><span> </span><span>predictions</span><span>)</span><span>
</span><span>#################
</span><span>for</span><span> </span><span>(</span><span>layer</span><span> </span><span>in</span><span> </span><span>base_model</span><span>$</span><span>layers</span><span>)</span><span>
</span><span>layer</span><span>$</span><span>trainable</span><span> </span><span>&lt;-</span><span> </span><span>FALSE</span><span>
</span><span>summary</span><span>(</span><span>model</span><span>)</span><span>
</span><span>###################
</span><span>model</span><span> </span><span>%>%</span><span> </span><span>compile</span><span>(</span><span>
</span><span>loss</span><span> </span><span>=</span><span> </span><span>&quot;categorical_crossentropy&quot;</span><span>,</span><span>
</span><span>optimizer</span><span> </span><span>=</span><span> </span><span>optimizer_rmsprop</span><span>(</span><span>lr</span><span> </span><span>=</span><span> </span><span>0.003</span><span>,</span><span> </span><span>decay</span><span> </span><span>=</span><span> </span><span>1e-6</span><span>),</span><span>
</span><span>metrics</span><span> </span><span>=</span><span> </span><span>&quot;accuracy&quot;</span><span>
</span><span>)</span><span>
</span><span>hist</span><span> </span><span>&lt;-</span><span> </span><span>model</span><span> </span><span>%>%</span><span> </span><span>fit_generator</span><span>(</span><span>
</span><span>train_generator</span><span>,</span><span>
</span><span>steps_per_epoch</span><span> </span><span>=</span><span> </span><span>as.integer</span><span>(</span><span>train_samples</span><span>/</span><span>batch_size</span><span>),</span><span> </span><span>epochs</span><span> </span><span>=</span><span> </span><span>100</span><span>,</span><span> </span><span>validation_data</span><span> </span><span>=</span><span> </span><span>validation_generator</span><span>,</span><span>
</span><span>validation_steps</span><span> </span><span>=</span><span> </span><span>as.integer</span><span>(</span><span>validation_samples</span><span>/</span><span>batch_size</span><span>),</span><span>
</span><span>verbose</span><span>=</span><span>2</span><span>
</span><span>)</span><span>
</span><span>###################### Train on augmented: artificially altered data #######
</span><span>hist_aug</span><span> </span><span>&lt;-</span><span> </span><span>model</span><span> </span><span>%>%</span><span> </span><span>fit_generator</span><span>(</span><span>
</span><span>train_augmented_generator</span><span>,</span><span>
</span><span>steps_per_epoch</span><span> </span><span>=</span><span> </span><span>as.integer</span><span>(</span><span>train_samples</span><span>/</span><span>batch_size</span><span>),</span><span> </span><span>epochs</span><span> </span><span>=</span><span> </span><span>50</span><span>,</span><span> </span><span>validation_data</span><span> </span><span>=</span><span> </span><span>validation_generator</span><span>,</span><span>
</span><span>validation_steps</span><span> </span><span>=</span><span> </span><span>as.integer</span><span>(</span><span>validation_samples</span><span>/</span><span>batch_size</span><span>),</span><span>
</span><span>verbose</span><span>=</span><span>2</span><span>
</span><span>)</span></code></pre>
</figure>
<p>I trained the model first 100 epochs on the original training data and added 50 epochs on the augmented (altered) dataset.<br>
Plotting the training history shows that training on the original data results in a validation accuracy of ~57% after 50 epochs. After that the neither the training nor the validation accuracy increases any further. Further training the model on the augmented data (red colored lines) leads to another boost in the validation accuracy.</p>
<p><img alt="plot of chunk unnamed-chunk-3" src="https://i0.wp.com/flovv.github.io/figures/post24/unnamed-chunk-3-1.png?w=456" data-recalc-dims="1"></p>
<p>To sum up, just changing a couple of lines from the previous setup changes the network’s performance significantly. Using a different pre-trained network and adding data augmentation doubles the classification accuracy.</p>
<p>as a sidenote; it appears (to me) that the current DL landscape is very dynamic and fast evolving. It is a safe bet to say that the content of this post is probably outdated in 6 months. Just in the last month the Rstudio/Keras repository has significantly changed;</p>
<blockquote>
<p>Excluding merges, 3 authors have pushed 178 commits to master and 178 commits to all branches. On master, 349 files have changed and there have been 5,468 additions and 1,719 deletions.</p>
</blockquote>
<p>Kudos to the rstudio-team for the great work on the package.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://flovv.github.io/Logo_detection_transfer_learning_part2/"> Florian Teschner</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/2x13eNARiqQ" height="1"></a></div>