<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/image-clustering-with-keras-and-k-means/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://shirinsplayground.netlify.com/2018/10/keras_fruits_cluster/"> Shirin's playgRound</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>A while ago, I wrote two blogposts about <a rel="nofollow" href="https://shirinsplayground.netlify.com/2018/06/keras_fruits/" target="_blank">image classification with Keras</a> and about <a rel="nofollow" href="https://shirinsplayground.netlify.com/2018/06/keras_fruits_lime/" target="_blank">how to use your own models or pretrained models for predictions and using LIME to explain to predictions</a>.</p>
<p>Recently, I came across <a rel="nofollow" href="https://medium.com/@franky07724_57962/using-keras-pre-trained-models-for-feature-extraction-in-image-clustering-a142c6cdf5b1" target="_blank">this blogpost</a> on using Keras to extract learned features from models and use those to cluster images. It is written in Python, though – so I adapted the code to R. You find the results below.</p>
<p>One use-case for image clustering could be that it can make labelling images easier because – ideally – the clusters would pre-sort your images, so that you only need to go over them quickly and check that they make sense.</p>
<p></p>
<div>
<h2>Libraries</h2>
<p>Okay, let’s get started by loading the packages we need.</p>
<pre><code>library(keras)
library(magick)  # for preprocessing images
library(tidyverse)
library(imager)</code></pre>
<p></p>
</div>
<div>
<h2>Pretrained model</h2>
<p>And we load the VGG16 pretrained model but we exclude the laste layers.</p>
<pre><code>model &lt;- application_vgg16(weights = &quot;imagenet&quot;, 
                           include_top = FALSE)
model</code></pre>
<pre><code>## Model
## ___________________________________________________________________________
## Layer (type)                     Output Shape                  Param #     
## ===========================================================================
## input_1 (InputLayer)             (None, None, None, 3)         0           
## ___________________________________________________________________________
## block1_conv1 (Conv2D)            (None, None, None, 64)        1792        
## ___________________________________________________________________________
## block1_conv2 (Conv2D)            (None, None, None, 64)        36928       
## ___________________________________________________________________________
## block1_pool (MaxPooling2D)       (None, None, None, 64)        0           
## ___________________________________________________________________________
## block2_conv1 (Conv2D)            (None, None, None, 128)       73856       
## ___________________________________________________________________________
## block2_conv2 (Conv2D)            (None, None, None, 128)       147584      
## ___________________________________________________________________________
## block2_pool (MaxPooling2D)       (None, None, None, 128)       0           
## ___________________________________________________________________________
## block3_conv1 (Conv2D)            (None, None, None, 256)       295168      
## ___________________________________________________________________________
## block3_conv2 (Conv2D)            (None, None, None, 256)       590080      
## ___________________________________________________________________________
## block3_conv3 (Conv2D)            (None, None, None, 256)       590080      
## ___________________________________________________________________________
## block3_pool (MaxPooling2D)       (None, None, None, 256)       0           
## ___________________________________________________________________________
## block4_conv1 (Conv2D)            (None, None, None, 512)       1180160     
## ___________________________________________________________________________
## block4_conv2 (Conv2D)            (None, None, None, 512)       2359808     
## ___________________________________________________________________________
## block4_conv3 (Conv2D)            (None, None, None, 512)       2359808     
## ___________________________________________________________________________
## block4_pool (MaxPooling2D)       (None, None, None, 512)       0           
## ___________________________________________________________________________
## block5_conv1 (Conv2D)            (None, None, None, 512)       2359808     
## ___________________________________________________________________________
## block5_conv2 (Conv2D)            (None, None, None, 512)       2359808     
## ___________________________________________________________________________
## block5_conv3 (Conv2D)            (None, None, None, 512)       2359808     
## ___________________________________________________________________________
## block5_pool (MaxPooling2D)       (None, None, None, 512)       0           
## ===========================================================================
## Total params: 14,714,688
## Trainable params: 14,714,688
## Non-trainable params: 0
## ___________________________________________________________________________</code></pre>
<p></p>
</div>
<div>
<h2>Training images</h2>
<p>Next, I am writting a helper function for reading in images and preprocessing them.</p>
<pre><code>image_prep &lt;- function(x) {
  arrays &lt;- lapply(x, function(path) {
    img &lt;- image_load(path, target_size = c(224, 224))
    x &lt;- image_to_array(img)
    x &lt;- array_reshape(x, c(1, dim(x)))
    x &lt;- imagenet_preprocess_input(x)
  })
  do.call(abind::abind, c(arrays, list(along = 1)))
}</code></pre>
<p>My images are in the following folder:</p>
<pre><code>image_files_path &lt;- &quot;/Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/fruits-360/Training&quot;
sample_fruits &lt;- sample(list.dirs(image_files_path), 2)</code></pre>
<p>Because running the clustering on all images would take very long, I am randomly sampling 5 image classes.</p>
<pre><code>file_list &lt;- list.files(sample_fruits, full.names = TRUE, recursive = TRUE)
head(file_list)</code></pre>
<pre><code>## [1] &quot;/Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/fruits-360/Training/Apple Red 2/0_100.jpg&quot;  
## [2] &quot;/Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/fruits-360/Training/Apple Red 2/1_100.jpg&quot;  
## [3] &quot;/Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/fruits-360/Training/Apple Red 2/10_100.jpg&quot; 
## [4] &quot;/Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/fruits-360/Training/Apple Red 2/100_100.jpg&quot;
## [5] &quot;/Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/fruits-360/Training/Apple Red 2/101_100.jpg&quot;
## [6] &quot;/Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/fruits-360/Training/Apple Red 2/102_100.jpg&quot;</code></pre>
<p>For each of these images, I am running the <code>predict()</code> function of Keras with the VGG16 model. Because I excluded the last layers of the model, this function will not actually return any class predictions as it would normally do; instead we will get the output of the last layer: <code>block5_pool (MaxPooling2D)</code>.</p>
<p>These, we can use as learned features (or abstractions) of the images. Running this part of the code takes several minutes, so I save the output to a RData file (because I samples randomly, the classes you see below might not be the same as in the <code>sample_fruits</code> list above).</p>
<pre><code>vgg16_feature_list &lt;- data.frame()

for (image in file_list) {
  
  print(image)
  cat(&quot;Image&quot;, which(file_list == image), &quot;from&quot;, length(file_list))
  
  vgg16_feature &lt;- predict(model, image_prep(image))
  
  flatten &lt;- as.data.frame.table(vgg16_feature, responseName = &quot;value&quot;) %>%
    select(value)
  flatten &lt;- cbind(image, as.data.frame(t(flatten)))
  
  vgg16_feature_list &lt;- rbind(vgg16_feature_list, flatten)
}
save(vgg16_feature_list, file = &quot;/Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/vgg16_feature_list.RData&quot;)</code></pre>
<pre><code>load(&quot;/Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/vgg16_feature_list.RData&quot;)</code></pre>
<pre><code>dim(vgg16_feature_list)</code></pre>
<pre><code>## [1]   980 25089</code></pre>
</div>
<div>
<h2>Clustering</h2>
<p>Next, I’m comparing two clustering attempts:</p>
<ol>
<li>clustering original data and</li>
<li>clustering first 10 principal components of the data</li>
</ol>
<p>Here as well, I saved the output to RData because calculation takes some time.</p>
<pre><code>pca &lt;- prcomp(vgg16_feature_list[, -1],
              center = TRUE,
              scale = FALSE)
save(pca, file = &quot;/Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/pca.RData&quot;)</code></pre>
<pre><code>load(&quot;/Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/pca.RData&quot;)</code></pre>
<p>Plotting the first two principal components suggests that the images fall into 4 clusters.</p>
<pre><code>data.frame(PC1 = pca$x[, 1], 
           PC2 = pca$x[, 2]) %>%
  ggplot(aes(x = PC1, y = PC2)) +
    geom_point()</code></pre>
<p><img src="https://i0.wp.com/shirinsplayground.netlify.com/post/2018-10-06_keras_fruits_cluster_files/figure-html/unnamed-chunk-11-1.png?w=450&ssl=1" data-recalc-dims="1"></p>
<p>The <code>kMeans</code> function let’s us do <a rel="nofollow" href="https://en.wikipedia.org/wiki/K-means_clustering" target="_blank">k-Means clustering</a>.</p>
<pre><code>set.seed(50)
cluster_pca &lt;- kmeans(pca$x[, 1:10], 4)
cluster_feature &lt;- kmeans(vgg16_feature_list[, -1], 4)</code></pre>
<p>Let’s combine the resulting cluster information back with the image information and create a column class (abbreviated with the first three letters).</p>
<pre><code>cluster_list &lt;- data.frame(cluster_pca = cluster_pca$cluster, 
                           cluster_feature = cluster_feature$cluster,
                           vgg16_feature_list) %>%
  select(cluster_pca, cluster_feature, image) %>%
  mutate(class = gsub(&quot;/Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/fruits-360/Training/&quot;, &quot;&quot;, image),
         class = substr(class, start = 1, stop = 3))
head(cluster_list)</code></pre>
<pre><code>##   cluster_pca cluster_feature
## 1           3               2
## 2           3               2
## 3           3               2
## 4           3               2
## 5           3               2
## 6           3               2
##                                                                                            image
## 1 /Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/fruits-360/Training/Cocos/100_100.jpg
## 2 /Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/fruits-360/Training/Cocos/101_100.jpg
## 3 /Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/fruits-360/Training/Cocos/102_100.jpg
## 4 /Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/fruits-360/Training/Cocos/103_100.jpg
## 5 /Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/fruits-360/Training/Cocos/104_100.jpg
## 6 /Users/shiringlander/Documents/Github/DL_AI/Tutti_Frutti/fruits-360/Training/Cocos/105_100.jpg
##   class
## 1   Coc
## 2   Coc
## 3   Coc
## 4   Coc
## 5   Coc
## 6   Coc</code></pre>
<p>And let’s count the number of images in each cluster, as well their class.</p>
<ul>
<li>PCA:</li>
</ul>
<pre><code>cluster_list %>%
  count(cluster_pca, class)</code></pre>
<pre><code>## # A tibble: 4 x 3
##   cluster_pca class     n
##         </code></pre></div>