Abstract:
We work out the math behind the so called income mountain plots used in the book &quot;Factfulness&quot; by Hans Rosling and use these insight to generate such plots using tidyverse code. The trip includes a mixture of log-normals, the density transformation theorem, histogram vs. density and then skipping all those details again to make nice moving mountain plots.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. The markdown+Rknitr source code of this blog is available under a GNU General Public License (GPL v3) license from github.
Introduction
Reading the book Factfulness by Hans Rosling seemed like a good thing to do during the summer months. The 'possibilistic' writing style is contagious and his TedEx presentations and media interviews are legendary teaching material on how to support your arguments with data. What a shame he passed away in 2017.
What is really enjoyable about the book is that the Gapminder web page allows you to study many of the graphs from the book interactively and contains the data for download. Being a fan of transparency and reproducibility, I got interested in the so called income mountain plots, which show how incomes are distributed within individuals of a population:
Screenshot of the 2010 income mountain plot. Free material from www.gapminder.org.
One notices that the &quot;mountains&quot; are plotted on a log-base-2 x-axis and without a y-axis annotation. Why? Furthermore, world income data usually involve mean income per country, so I got curious how/if these plots were made without access to finer granularity level data? Aim of this blog post is to answer these questions by using Gapminder data freely available from their webpage. The answer ended up as a nice tidyverse exercise and could serve as motivating application for basic probability course content.
Data Munging Gapminder
Data on income, population and Gini coefficient were needed to analyse the above formulated questions. I have done this previously in order to visualize the Olympic Medal Table Gapminder Style. We start by downloading the GDP data, which is the annual gross domestic product per capita by Purchasing Power Parities (PPP) measured in international dollars, fixed 2011 prices. Hence, the inflation over the years and differences in the cost of living between countries is accounted for and can thus be compared - see the Gapminder documentation for further details. We download the data from Gapminder where they are available in wide format as Excel-file. For tidyverse handling we reshape them into long format.
##Download gdp data from gapminder - available under a CC BY-4 license.
if (!file.exists(file.path(fullFigPath, &quot;gapminder-gdp.xlsx&quot;))) {
download.file(&quot;https://github.com/Gapminder-Indicators/gdppc_cppp/raw/master/gdppc_cppp-by-gapminder.xlsx&quot;, destfile=file.path(fullFigPath,&quot;gapminder-gdp.xlsx&quot;))
}
gdp_long %
rename(country=`geo.name`) %__%
select(-geo,-indicator,-indicator.name) %__%
gather(key=&quot;year&quot;, value=&quot;gdp&quot;, -country,) %__%
filter(!is.na(gdp))
Furthermore, we rescale GDP per year to daily income, because this is the unit used in the book.
gdp_long %% mutate(gdp = gdp / 365.25)
Similar code segments are written for (see the code on github for details)
the gini (gini_long) and population (pop_long) data
the regional group (=continent) each country belongs two (group)
The four data sources are then joined into one long tibble gm. For each year we also compute the fraction a country's population makes up of the world population that year (column w) as well as the fraction within the year and region the population makes up (column w_region) :
## # A tibble: 15,552 x 9
## country region code year gini gdp population w w_region
## ## 1 Afghanistan Asia AFG 1800 0.305 1.65 3280000 0.00347 0.00518 ## 2 Albania Europe ALB 1800 0.389 1.83 410445 0.000434 0.00192 ## 3 Algeria Africa DZA 1800 0.562 1.96 2503218 0.00264 0.0342 ## 4 Andorra Europe AND 1800 0.4 3.28 2654 0.00000280 0.0000124
## 5 Angola Africa AGO 1800 0.477 1.69 1567028 0.00166 0.0214 ## # ... with 1.555e+04 more rows
Income Mountain Plots
The construction of the income mountain plots is thoroughly described on the Gapminder webpage, but without mathematical detail. With respect to the math it says: &quot;Bas van Leeuwen shared his formulas with us and explained how to the math from ginis and mean income, to accumulated distribution shapes on a logarithmic scale.&quot; Unfortunately, the formulas are not shared with the reader. It's not black magic though: The income distribution of a country is assumed to be log-normal with a given mean \(\mu\) and standard deviation \(\sigma\) on the log-scale, i.e. \(X \sim \operatorname{LogN}(\mu,\sigma^2)\). From knowing the mean income \(\overline{x}\) of the distribution as well as the Gini index \(G\) of the distribution, one can show that it's possible to directly infer \((\mu, \sigma)\) of the log-normal distribution.
Because the Gini index of the log-normal distribution is given by \[
G = 2\Phi\left(\frac{\sigma}{\sqrt{2}}\right)-1,
\] where \(\Phi\) denotes the CDF of the standard normal distribution, and by knowing that the expectation of the log-normal is \(E(X) = \exp(\mu + \frac{1}{2}\sigma^2)\), it is possible to determine \((\mu,\sigma)\) as:
\[
\sigma = \frac{2}{\sqrt{2}} \Phi^{-1}\left(\frac{G+1}{2}\right)
\quad\text{and}\quad
\mu = \log(\overline{x}) - \frac{1}{2} \sigma^2.
\]
We can use this to determine the parameters of the log-normal for every country in each year.
Mixture distribution
The income distribution of a set of countries is now given as a Mixture distribution of log-normals, i.e. one component for each of the countries in the set with a weight proportional to the population of the country. As an example, the world income distribution would be a mixture of the 192 countries in the Gapminder dataset, i.e.
\[
f_{\text{mix}}(x) = \sum_{i=1}^{192} w_i \__\cdot
\__f_{\operatorname{LogN}}(x; \mu_i, \sigma_i^2), \quad\text{where}
\quad w_i = \frac{\text{population}_i}{\sum_{j=1}^{192} \text{population}_j},
\] and \(f_{\operatorname{LogN}}(x; \mu_i, \sigma_i^2)\) is the density of the log-normal distribution with country specific parameters. Note that we could have equally used the mixture approach to define the income of, e.g., a continent region. With the above definition we define standard R-functions for computing the PDF (dmix), CDF (pmix), quantile function (qmix) and a function for sampling from the distribution (rmix) - see the github code for details.
We use the mixture approach to compute the density of the world income distribution obtained by &quot;mixing&quot; all 192 log-normal distributions. This is shown below for the World income distribution of the year 2015. Note the \(\log_2\) x-axis. This presentation is Factfulness' preferred way of illustrating the skew income distribution.
##Restrict to year 2015
gm_recent % filter(year == 2015) %__% ungroup
##Make a data frame containing the densities of each region for
##the gm_recent dataset
df_pdf % mutate(x=2^log2x)
pdf_region % group_by(region) %__% do({
pdf %
summarise(region=&quot;Total&quot;,w=sum(w), pdf = sum(w_pdf)) ## Expectation of the distribution mean_mix %
summarise(mean=sum(w * exp(meanlog + 1/2*sdlog^2))) %$% mean
## Median of the distribution median_mix % filter(pdf_log2x == max(pdf_log2x)) %$% x
For illustration we compute a mixture distribution for each region using all countries within region. This is shown in the left pane. Note: because a log-base-2-transformation is used for the x-axis, we need to perform a change of variables, i.e. we compute the density for \(Y=\log_2(X)=g(X)\) where \(X\sim f_{\text{mix}}\), i.e. \[
f_Y(y) = \left| \frac{d}{dy}(g^{-1}(y)) \right| f_X(g^{-1}(y)) = \log(2) \cdot 2^y \cdot f_{\text{mix}}( 2^y) = \log(2) \cdot x \cdot f_{\text{mix}}(x), \text{ where } x=2^y.
\]
In the right pane we then show the region specific densities each weighted by their population fraction. These are then summed up to yield the world income shown as a thick blue line. The median of the resulting world income distribution is at 20.0 $/day, whereas the mean of the mixture is at an income of 39.9$/day and the mode (on the log-base-2 scale) is 17.1$/day. Note that the later is not transformation invariant, i.e. the value is not the mode of the income distribution, but of \(\log_2(X)\).
To get the income mountain plots as shown in Factfulness, we additionally need to obtain number of people on the \(y\)-axis and not density. We do this by partitioning the x-axis into non-overlapping intervals and then compute the number of individuals expected to fall into a given interval with limits \([l, u]\). Under our model this expectation is
\[n \cdot (F_{\text{mix}}(u)-F_{\text{mix}}(l)),\]
where \(F_{\text{mix}}\) is the CDF of the mixture distribution and \(n\) is the total world population. The mountain plot below shows this for a given partition with \(n=7,305,116,647\). Note that \(2.5\cdot 10^8\) corresponds to 250 mio people. Also note the \(\log_2\) x-axis, and hence (on the linear scale) unequally wide intervals of the partitioning. Contrary to Factfulness', I prefer to make this more explicit by indicating the intervals explicitly on the x-axis of the mountain plot, because it is about number of people in certain income brackets.
##Function to prepare the data.frame to be used in a mountain plot
make_mountain_df %
mutate(xm1 = lag(x), log2xm1=lag(log2x)) %__%
mutate(xm1=if_else(is.na(xm1),0,xm1),
log2xm1=if_else(is.na(log2xm1),0,log2xm1),
mid_log2 = (log2x+log2xm1)/2,
width = (x-xm1),
width_log2 = (log2x-log2xm1)) %__%
##Format the interval character representation
mutate(interval=if_else(xm1% do({
countries % rowwise %__%
mutate(
prob_mass = diff(pmix(c(xm1,x), meanlog=countries$meanlog, sdlog=countries$sdlog, w=countries$w_region)),
people = prob_mass * sum(countries$population)
)
temp %__% mutate(year = max(gm_df$year))
})
##Done
return(people)
}
##Create mountain plot data set for gm_recent with default spacing.
(people