<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/on-a-first-name-basis-with-statistics-sweden/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://staff.math.su.se/hoehle/blog/2017/03/25/scbnames.html"> Theory meets practice...</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h2>Abstract</h2>
<p>Jugding from recent R-Bloggers posts, it appears that many data scientists are concerned with scraping data from various media sources (Wikipedia, twitter, etc.). However, one should be aware that well structured and high quality datasets are available through state’s and country’s bureau of statistics. Increasingly these are offered to the public through direct database access, e.g., using a REST API. We illustrate the usefulness of such an approach by accessing data from Statistics Sweden.</p>
<p><center><br>
<img src="https://i1.wp.com/staff.math.su.se/hoehle/blog/figure/source/2017-03-25-scbnames/COLLISION26-1.png?w=456" data-recalc-dims="1"><br>
</center></p>
<p> <a target="_blank" rel="nofollow" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" src="https://i1.wp.com/i.creativecommons.org/l/by-sa/4.0/88x31.png?w=456&ssl=1" data-recalc-dims="1"></a> This work is licensed under a <a target="_blank" rel="nofollow" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>. The markdown+Rknitr source code of this blog is available under a <a target="_blank" rel="nofollow" href="https://www.gnu.org/licenses/gpl-3.0.html">GNU General Public License (GPL v3)</a> license from <a target="_blank" rel="nofollow" href="https://github.com/hoehleatsu/hoehleatsu.github.io"><img alt="github" src="https://i2.wp.com/staff.math.su.se/hoehle/blog/images/GitHub-Mark-32px.png?w=456" data-recalc-dims="1"></a>.</p>
<h2>Introduction</h2>
<p>Scandinavian countries are world-class when it comes to public registries. So when in need for reliable population data, this is the place to look. As an example, we access Statistics Sweden data by their API using the <code>pxweb</code> package developed by <a target="_blank" rel="nofollow" href="https://twitter.com/MansMeg">@MansMeg</a>, <a target="_blank" rel="nofollow" href="https://twitter.com/antagomir">@antagomir</a> and <a target="_blank" rel="nofollow" href="https://twitter.com/LCHansson">@LCHansson</a>. Love was the first speaker at a Stockholm R-Meetup some <a target="_blank" rel="nofollow" href="https://www.meetup.com/StockholmR/events/105738342/">years ago</a>, where I also gave a talk. Funny how such R-Meetups become useful many years after!</p>
<div>
<pre><code><span>library</span>(pxweb)</code></pre>
</div>
<p>By browsing the Statistics Sweden (in Swedish: Statistiska Centralbyrån (SCB)) data using their <a target="_blank" rel="nofollow" href="http://www.scb.se/en/finding-statistics/">web interface</a> one sees that they have two relevant first name datasets: one containing the <em>tilltalsnamn</em> of newborns for each year during 1998-2016 and one for the years 2004-2016. <strong>Note</strong>: A <a target="_blank" rel="nofollow" href="https://translate.google.com/translate?sl=auto&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=https%3A%2F%2Fsv.wikipedia.org%2Fwiki%2FTilltalsnamn&edit-text=&act=url"><em>tilltalsnamn</em></a> in Sweden is <em>the</em> first name (of several possible first names) by which a person is usually addressed. About 2/3 of the persons in the Swedish name registry indicate which of their first names is their tilltalsnamn. For the remaining persons it is automatically implied that their tilltalsnamn is the <a target="_blank" rel="nofollow" href="http://www.scb.se/hitta-statistik/statistik-efter-amne/befolkning/amnesovergripande-statistik/namnstatistik/produktrelaterat/Fordjupad-information/fragor-och-svar-om-namnstatistiken/#fem">first of the first names</a>. <strong>Also note:</strong> For reasons of data protection the 1998-2016 dataset contains only first names used 10 or more times in a given year, the 2004-2016 dataset contains only first names used 2 or more times in a given year.</p>
<p>Downloading such data through the SCB web-interface is cumbersome, because the downloads are limited to 50,000 data cells per query. Hence, one has to do several manual queries to get hold of the relevant data. This is where their <a target="_blank" rel="nofollow" href="http://www.scb.se/en_/About-us/Open-data-API/API-for-the-Statistical-Database-/">API</a> becomes a real time-saver. Instead of trying to fiddle with the API directly using <code>rjson</code> or <code>RJSONIO</code> we use the specially designed <code>pxweb</code> package to fetch the data. One can either use the web-interface to determine the name of the desired data matrix to query or navigate directly through the api using <code>pxweb</code>:</p>
<div>
<pre><code>d &lt;-<span> </span><span>interactive_pxweb</span>(<span>api =</span> <span>&quot;api.scb.se&quot;</span>, <span>version =</span> <span>&quot;v1&quot;</span>, <span>lang =</span> <span>&quot;en&quot;</span>)</code></pre>
</div>
<p>and select <em>Population</em> followed by <em>Name statistics</em> and then <code>BE0001T04Ar</code> or <code>BE0001T04BAr</code>, respectively, in order to obtain the relevant data and api download url. This leads to the following R code for download:</p>
<div>
<pre><code>names10 &lt;-<span> </span><span>get_pxweb_data</span>(
  <span>url =</span> <span>&quot;http://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0001/BE0001T04Ar&quot;</span>,
  <span>dims =</span> <span>list</span>(<span>Tilltalsnamn =</span> <span>c</span>(<span>'*'</span>),
              <span>ContentsCode =</span> <span>c</span>(<span>'BE0001AH'</span>),
              <span>Tid =</span> <span>c</span>(<span>'*'</span>)),
  <span>clean =</span> <span>TRUE</span>) %>%<span> </span>as.tbl</code></pre>
</div>
<p>For better usability we rename the columns a little and replace <code>NA</code> counts to be zero. For visualization we pick 10 random lines of the dataset.</p>
<div>
<pre><code>names10 &lt;-<span> </span>names10 %>%<span> </span><span>select</span>(-observations) %>%
<span> </span><span>rename</span>(<span>firstname=</span><span>`</span><span>first name normally used</span><span>`</span>,<span>counts=</span>values) %>%
<span> </span><span>mutate</span>(<span>counts =</span> <span>ifelse</span>(<span>is.na</span>(counts),<span>0</span>,counts))
##Look at 10 random lines
names10 %>%<span> </span><span>slice</span>(<span>sample</span>(<span>seq_len</span>(<span>nrow</span>(names10)),<span>size=</span><span>5</span>))</code></pre>
</div>
<pre><code>## # A tibble: 5 × 3
##   firstname   year counts
##      &lt;fctr> &lt;fctr>  &lt;dbl>
## 1   Leandro   2011     15
## 2    Marlon   2004      0
## 3    Andrej   2009      0
## 4     Ester   2002     63
## 5   Muhamed   1998      0</code></pre>
<p>Note: Each spelling variant of a name in the data is treated as a unique name. In similar fashion we download the <code>BE0001AL</code> dataset as <code>names2</code>.</p>
<p>We now join the two datasets into one large <code>data.frame</code> by</p>
<div>
<pre><code>names &lt;-<span> </span><span>rbind</span>(<span>data.frame</span>(names2,<span>type=</span><span>&quot;min02&quot;</span>), <span>data.frame</span>(names10,<span>type=</span><span>&quot;min10&quot;</span>))</code></pre>
</div>
<p>and thus got everything in place to compute the name collision probability over time using the <code>birthdayproblem</code> package (as shown in <a target="_blank" rel="nofollow" href="http://staff.math.su.se/hoehle/blog/2017/02/13/bday.html">previous posts</a>).</p>
<div>
<pre><code><span>library</span>(birthdayproblem)
collision &lt;-<span> </span>names %>%<span> </span><span>group_by</span>(year,type) %>%<span> </span><span>do</span>({
  <span>data.frame</span>(<span>p=</span><span>pbirthday_up</span>(<span>n=</span>26L, <span>prob=</span> .$counts /<span> </span><span>sum</span>(.$counts),<span>method=</span><span>&quot;mase1992&quot;</span>)$prob, <span>gini=</span> ineq::<span>Gini</span>(.$counts))
}) %>%<span> </span>ungroup %>%<span> </span><span>mutate</span>(<span>year=</span><span>as.numeric</span>(<span>as.character</span>(year)))</code></pre>
</div>
<p>And the resulting probabilities based on the two datasets <code>min02</code> (at least two instances of the name in a given year) and <code>min10</code> (at least ten instances of the name in a given year) can easily be visualized over time.</p>
<div>
<pre><code><span>ggplot</span>( collision, <span>aes</span>(<span>x=</span>year, <span>y=</span>p, <span>color=</span>type)) +<span> </span><span>geom_line</span>(<span>size=</span><span>1.5</span>) +
<span> </span><span>scale_y_continuous</span>(<span>label=</span>scales::percent,<span>limits=</span><span>c</span>(<span>0</span>,<span>1</span>)) +
<span> </span><span>xlab</span>(<span>&quot;Year&quot;</span>) +<span> </span><span>ylab</span>(<span>&quot;Probability&quot;</span>) +
<span> </span><span>ggtitle</span>(<span>&quot;Probability of a name collision in a class of 26 kids born in year YYYY&quot;</span>) +
<span> </span><span>scale_colour_discrete</span>(<span>name =</span> <span>&quot;Dataset&quot;</span>)</code></pre>
</div>
<p><img src="https://i1.wp.com/staff.math.su.se/hoehle/blog/figure/source/2017-03-25-scbnames/COLLISION26-1.png?w=456" data-recalc-dims="1"></p>
<p>As seen in similar plots for other countries, there is a decline in the collision probability over time. Note also that the two curves are upper limits to the true collision probabilities. The true probabilities, i.e. taking all tilltalsnamn into account, would be based on the hypothetical <code>min1</code> data set. These probabilities would be slightly, but not substantially, below the <code>min2</code> line. The same problem occurs, e.g., in the corresponding <a target="_blank" rel="nofollow" href="https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/livebirths/datasets/babynamesenglandandwalesbabynamesstatisticsboys">UK and Wales data</a>. Here, Table 6 is listing all first names with 3 or more uses, but not stating how many newborns have a name occurring once and twice, respectively. With all due respect for the need to anonymise the name statistics, it’s hard to understand why this summary figure is not automatically reported, so one would be able to at least compute correct totals or collision probabilities.</p>
<h2>Summary</h2>
<p>Altogether, I was still quite happy to get proper individual name data so the collision probabilities are – opposite to some of my <a target="_blank" rel="nofollow" href="http://staff.math.su.se/hoehle/blog/2017/02/13/bday.html">previous blog analyses</a> – exact!</p>
<div>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://staff.math.su.se/hoehle/blog/2017/03/25/scbnames.html"> Theory meets practice...</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/GYa8N96GRw4"></a></div>