<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/cluster-analysis-naming-pattern-in-the-last-century/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://flovv.github.io/Cluster_analysis/"> Florian Teschner</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>A while back I got interested in (baby) surnames. It is interesting to follow how friends name their newly born.<br>
There are various naming “strategies”, a) pick a name that has been used in previous generations but now uncommon (e.g. in Germany that would be Oskar), b) pick a biblical name that never ages such as Johannes or David, or c) use a interesting sounding “foreign” name (e.g. Alina).</p>
<p>I thought it would be interesting to figure out if we can cluster names and uncover these patterns (if they do really exist.) in a real dataset.<br>
Next, I scraped the last 116 years of data from this <a rel="nofollow" href="https://www.beliebte-vornamen.de/" target="_blank">German source</a>. Due to Germany’s federal system there is no single consolidated database for birth statistics or naming data. Hence, the scraped data is sampled and only providing a ranking of the top 30 or so given birth names per year (at least seperated for boys and girls).</p>
<p>This is what the data looks like.</p>
<figure>
<pre><code data-lang="r"><span>head</span><span>(</span><span>total</span><span>,</span><span> </span><span>5</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##   year gender   name
## 1 1900   male  Erich
## 2 1900   male  Franz
## 3 1900   male  Georg
## 4 1900   male Gustav
## 5 1900   male Johann</code></pre>
</figure>
<p>Next, we caluclate the average year for each surname by averaging the years a name has made the top 30 list. As a second metric we calculate the popularity as the number of times a name appeared in the top 30 rankings.</p>
<figure>
<pre><code data-lang="r"><span>dc</span><span> </span><span>&lt;-</span><span> </span><span>data.table</span><span>::</span><span>dcast</span><span>(</span><span>total</span><span>,</span><span> </span><span>name</span><span>~</span><span>year</span><span>,</span><span> </span><span>fill</span><span>=</span><span>0</span><span>)</span><span>
</span><span>melted</span><span> </span><span>&lt;-</span><span> </span><span>data.table</span><span>::</span><span>melt</span><span>(</span><span>dc</span><span>,</span><span> </span><span>id.vars</span><span>=</span><span>&quot;name&quot;</span><span>)</span><span>
</span><span>melted</span><span>$</span><span>variable</span><span> </span><span>&lt;-</span><span> </span><span>as.numeric</span><span>(</span><span>as.character</span><span>(</span><span>melted</span><span>$</span><span>variable</span><span>))</span><span>
</span><span>dd</span><span> </span><span>&lt;-</span><span> </span><span>ddply</span><span>(</span><span>melted</span><span>,</span><span> </span><span>.</span><span>(</span><span>name</span><span>),</span><span> </span><span>summarise</span><span>,</span><span> </span><span>avgYear</span><span>=</span><span>weighted.mean</span><span>(</span><span>variable</span><span>,</span><span> </span><span>value</span><span>),</span><span> </span><span>popularity</span><span>=</span><span>sum</span><span>(</span><span>value</span><span>)</span><span>/</span><span>116</span><span>)</span><span>
</span><span>dd</span><span>[</span><span>dd</span><span>$</span><span>avgYear</span><span> </span><span>==</span><span> </span><span>min</span><span>(</span><span>dd</span><span>$</span><span>avgYear</span><span>),]</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##        name avgYear popularity
## 84 Dorothea    1900 0.00862069
## 90    Elise    1900 0.00862069</code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>dd</span><span>[</span><span>dd</span><span>$</span><span>avgYear</span><span> </span><span>==</span><span> </span><span>max</span><span>(</span><span>dd</span><span>$</span><span>avgYear</span><span>),]</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##       name avgYear popularity
## 255 Matteo    2016 0.00862069
## 267  Milan    2016 0.00862069</code></pre>
</figure>
<p>To determine the optimal cluster number, we use the <a rel="nofollow" href="https://bl.ocks.org/rpgove/0060ff3b656618e9136b" target="_blank">elbow technique</a>.</p>
<figure>
<pre><code data-lang="r"><span>######### How many cluster do we need?&quot; </span><span>
</span><span>clusterData</span><span> </span><span>&lt;-</span><span> </span><span>dc</span><span>[</span><span>2</span><span>:</span><span>117</span><span>]</span><span>
</span><span>ll</span><span> </span><span>&lt;-</span><span> </span><span>unlist</span><span>(</span><span>llply</span><span>(</span><span>1</span><span>:</span><span>10</span><span>,</span><span> </span><span>function</span><span>(</span><span>x</span><span>){</span><span>sum</span><span>(</span><span>kmeans</span><span>(</span><span>clusterData</span><span>,</span><span> </span><span>centers</span><span>=</span><span>x</span><span>)</span><span>$</span><span>withinss</span><span>)}))</span><span>
</span><span>ggplot</span><span>(</span><span>data.frame</span><span>(</span><span>x</span><span>=</span><span>1</span><span>:</span><span>10</span><span>,</span><span>y</span><span>=</span><span>ll</span><span>),</span><span> </span><span>aes</span><span>(</span><span>x</span><span>,</span><span>y</span><span>))</span><span>+</span><span>geom_point</span><span>()</span><span>+</span><span> </span><span>geom_line</span><span>()</span><span> </span><span>+</span><span> </span><span>xlab</span><span>(</span><span>&quot;Number of Clusters&quot;</span><span>)</span><span> </span><span>+</span><span> </span><span>ylab</span><span>(</span><span>&quot;Within groups sum of squares&quot;</span><span>)</span><span>+</span><span> </span><span>theme_bw</span><span>()</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-43" src="https://i1.wp.com/flovv.github.io/figures/post31/unnamed-chunk-43-1.png?w=456" data-recalc-dims="1"></p>
<p>We see that the chart exhibits a kink at 5. Hence, we subsequently do the analysis with 5 clusters.<br>
Next, we cluster (using Kmeans) the name data and calculate the average year for each cluster. Using a quick principle component analysis, we can visually inspect the name clusters.</p>
<figure>
<pre><code data-lang="r"><span>require</span><span>(</span><span>ggfortify</span><span>)</span><span>
</span><span>fit</span><span> </span><span>&lt;-</span><span> </span><span>kmeans</span><span>(</span><span>dc</span><span>[</span><span>2</span><span>:</span><span>117</span><span>],</span><span> </span><span>centers</span><span>=</span><span>5</span><span>)</span><span>
</span><span>dc</span><span>$</span><span>cluster</span><span> </span><span>&lt;-</span><span> </span><span>fit</span><span>$</span><span>cluster</span><span>
</span><span>mm</span><span> </span><span>&lt;-</span><span> </span><span>merge</span><span>(</span><span>dc</span><span>,</span><span> </span><span>dd</span><span>,</span><span> </span><span>by</span><span>=</span><span>&quot;name&quot;</span><span>)</span><span>
</span><span>clusterAge</span><span> </span><span>&lt;-</span><span> </span><span>ddply</span><span>(</span><span>mm</span><span>,</span><span> </span><span>.</span><span>(</span><span>cluster</span><span>),</span><span> </span><span>summarise</span><span>,</span><span> </span><span>m</span><span>=</span><span>mean</span><span>(</span><span>avgYear</span><span>))</span><span>
</span><span>subtitle</span><span> </span><span>&lt;-</span><span> </span><span>paste</span><span>(</span><span>&quot;Avg.Year per Cluster;&quot;</span><span>,</span><span> </span><span>paste</span><span>(</span><span>paste</span><span>(</span><span>&quot;C&quot;</span><span>,</span><span> </span><span>clusterAge</span><span>$</span><span>cluster</span><span>,</span><span> </span><span>&quot;:&quot;</span><span>,</span><span>round</span><span>(</span><span>clusterAge</span><span>$</span><span>m</span><span>)),</span><span> </span><span>collapse</span><span> </span><span>=</span><span>&quot; | &quot;</span><span>))</span><span>
</span><span>row.names</span><span>(</span><span>mm</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>mm</span><span>$</span><span>name</span><span>
</span><span>mm</span><span>$</span><span>cluster</span><span> </span><span>&lt;-</span><span> </span><span>as.factor</span><span>(</span><span>mm</span><span>$</span><span>cluster</span><span>)</span><span>
</span><span>autoplot</span><span>(</span><span>prcomp</span><span>(</span><span>dc</span><span>[,</span><span>2</span><span>:</span><span>118</span><span>]),</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>mm</span><span>,</span><span> </span><span>colour</span><span> </span><span>=</span><span> </span><span>'cluster'</span><span>,</span><span> </span><span>label</span><span>=</span><span>T</span><span>,</span><span> </span><span>frame</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>+</span><span> </span><span>ggtitle</span><span>(</span><span>&quot;Cluster Analysis&quot;</span><span>,</span><span> </span><span>subtitle</span><span> </span><span>=</span><span>subtitle</span><span>)</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-44" src="https://i1.wp.com/flovv.github.io/figures/post31/unnamed-chunk-44-1.png?w=456" data-recalc-dims="1"></p>
<p>Looking at the average years per cluster, we see that they are apparently well separated. The cluster supports this finding; Take cluster 5 as an example; Ingeborg, Walther, Charlotte and Hans are all rather old-fashioned names.</p>
<p>Sometimes mean values do not tell the whole picture. So, let’s see if we can identify patterns when we plot the distribution of average ages as well as a cluster’s popularity.</p>
<p><img alt="plot of chunk unnamed-chunk-45" src="https://i0.wp.com/flovv.github.io/figures/post31/unnamed-chunk-45-1.png?w=456" data-recalc-dims="1"></p>
<p>We see three similar distributions (cluster 5,4,2), just shifted by ~30 years. Two distributions are noteably different.<br>
Cluster 3 has a bi-modal shape: A first peak appears in the late 60s and bigger peak from 2000 onward. The most different is cluster 1; the distribution spreads out across the whole century. If we look at the popularity distributions we see that the same cluster 1 is the least popular. The names appear in less than 10% of all top 30 rankings.</p>
<p><img alt="plot of chunk unnamed-chunk-46" src="https://i2.wp.com/flovv.github.io/figures/post31/unnamed-chunk-46-1.png?w=456" data-recalc-dims="1"></p>
<p>Also, the older cluster appear to be slightly more popular in general.<br>
To sum up, with a cluster analysis we did not directly find the initially hypothized pattern. We did find though that names are clustered in time periods and grow out of fashion.</p>
<p>It is important to remember the input’s data structure. We created name cohorts by clustering across a binary coding over years. The rows indicate names and the columns the years. In a sense that is a wide sparse matrix. It is not the only way to structure the input data. Another way would be to aggregate the key features (e.g. max, min, average, sd of the appearing years) for each name first and cluster the names second.</p>
<p>Also, Kmeans is not a great pick for categorical/binary data. One could try using partitioning (clustering) around medoids (PAM), or kmodes. (I also tried to cluster the data using the klaR::kmodes function, but I could not see a substantial difference.)</p>
<p>In contrast to a supervised analysis, the unsupervised approach leaves more room for interpretation and degrees of freedom. Before writing this post, I looked for clustering examples with a focus on interpretation and applicability to business problems. The closest I found was this <a rel="nofollow" href="https://rpubs.com/gabrielmartos/ClusterAnalysis" target="_blank">analysis</a>. If you have a good hint, please send me an email.</p>
<p>If you want to reproduce the results, the dataset can be found <a rel="nofollow" href="https://github.com/flovv/flovv.github.io/tree/master/data" target="_blank">here</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://flovv.github.io/Cluster_analysis/"> Florian Teschner</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/7iYVgkUOkfg" height="1"></a></div>