<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/scratching-the-surface-of-gender-biases/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://shiring.github.io/shiny/2017/02/06/WGS_final"> Shirin's playgRound</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Today, I want to share my analysis of the World Gender Statistics dataset.</p>
<p>Last week I already introduced <a target="_blank" rel="nofollow" href="https://shiring.shinyapps.io/wgs_app/">my Shiny app</a>, where you can explore 160 measurements for 164 countries over 56 years. This week I’ve included a statistical analysis of these countries and measurements and put some finishing touches on the app.</p>
<p>Just in case anybody needed proof that gender bias is not only a problem of “other countries”, my analysis showed very nicely, that the Western world isn’t (yet, I hope) gender neutral in regards to many statistics!</p>
<p>You can also load the app via Github with the <strong>shiny</strong> package. <a target="_blank" rel="nofollow" href="https://github.com/ShirinG/WGS_app">There, you can also find the source code for the app</a>.</p>
<div>
<pre><code><span>library</span><span>(</span><span>shiny</span><span>)</span><span>
</span><span>runGitHub</span><span>(</span><span>&quot;ShirinG/WGS_app&quot;</span><span>)</span><span> </span></code></pre>
</div>
<p></p>
<h2>The data</h2>
<p>The data was downloaded from <a target="_blank" rel="nofollow" href="http://data.worldbank.org/">The World Bank’s Open Data project</a> via <a target="_blank" rel="nofollow" href="https://www.kaggle.com/theworldbank/world-gender-statistic">Kaggle</a>. The main data table shows</p>
<ul>
<li>Country.Name: the name of the country</li>
<li>Country.Code: the country’s code</li>
<li>Indicator.Name: the name of the variable that this row represents</li>
<li>Indicator.Code: a unique id for the variable</li>
<li>1960 – 2016: one column EACH for the value of the variable in each year it was available</li>
</ul>
<p>Unfortunately, the dataset doesn’t include a column that indicated which two female/ male statistics belong together and there is no consistent naming scheme either. Therefore, I chose to focus on those statistics, where the counterparts could be easily extracted: indicator codes that differed only in containing “.FE” or “.MA”. I then split the subsetted dataframe into female and male and produced a third dataset with the ratios between male and female values. To calculate the ratios, I added 0.001 to each data point, so that zero values would not produce NAs.</p>
<p>The finished datasets were saved as “R.Data” files, so that I could easily load them into the Shiny app.</p>
<div>
<pre><code><span>dataset</span><span> </span><span>&lt;-</span><span> </span><span>read.csv</span><span>(</span><span>&quot;Data.csv&quot;</span><span>)</span><span>
</span><span>dataset_subs</span><span> </span><span>&lt;-</span><span> </span><span>dataset</span><span>[</span><span>grep</span><span>(</span><span>&quot;.FE|.MA&quot;</span><span>,</span><span> </span><span>dataset</span><span>$</span><span>Indicator.Code</span><span>),</span><span> </span><span>]</span><span>
</span><span>head</span><span>(</span><span>dataset_subs</span><span>)</span><span>
</span><span>dataset_subs</span><span>$</span><span>Indicator.Name</span><span> </span><span>&lt;-</span><span> </span><span>as.character</span><span>(</span><span>dataset_subs</span><span>$</span><span>Indicator.Name</span><span>)</span><span>
</span><span>dataset_fem</span><span> </span><span>&lt;-</span><span> </span><span>dataset</span><span>[</span><span>grep</span><span>(</span><span>&quot;female&quot;</span><span>,</span><span> </span><span>dataset</span><span>$</span><span>Indicator.Name</span><span>),</span><span> </span><span>]</span><span>
</span><span>dataset_fem</span><span>$</span><span>Indicator.Name</span><span> </span><span>&lt;-</span><span> </span><span>gsub</span><span>(</span><span>&quot;female&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>dataset_fem</span><span>$</span><span>Indicator.Name</span><span>)</span><span>
</span><span>dataset_fem</span><span>$</span><span>Indicator.Name</span><span> </span><span>&lt;-</span><span> </span><span>gsub</span><span>(</span><span>&quot;,&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>dataset_fem</span><span>$</span><span>Indicator.Name</span><span>)</span><span>
</span><span>dataset_fem</span><span>$</span><span>Indicator.Code</span><span> </span><span>&lt;-</span><span> </span><span>gsub</span><span>(</span><span>&quot;.FE&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>dataset_fem</span><span>$</span><span>Indicator.Code</span><span>)</span><span>
</span><span>dataset_fem</span><span>$</span><span>gender</span><span> </span><span>&lt;-</span><span> </span><span>&quot;female&quot;</span><span>
</span><span>dataset_male</span><span> </span><span>&lt;-</span><span> </span><span>dataset</span><span>[</span><span>-</span><span>grep</span><span>(</span><span>&quot;female&quot;</span><span>,</span><span> </span><span>dataset</span><span>$</span><span>Indicator.Name</span><span>),</span><span> </span><span>]</span><span>
</span><span>dataset_male</span><span>$</span><span>Indicator.Name</span><span> </span><span>&lt;-</span><span> </span><span>gsub</span><span>(</span><span>&quot;male&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>dataset_male</span><span>$</span><span>Indicator.Name</span><span>)</span><span>
</span><span>dataset_male</span><span>$</span><span>Indicator.Name</span><span> </span><span>&lt;-</span><span> </span><span>gsub</span><span>(</span><span>&quot;,&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>dataset_male</span><span>$</span><span>Indicator.Name</span><span>)</span><span>
</span><span>dataset_male</span><span>$</span><span>Indicator.Code</span><span> </span><span>&lt;-</span><span> </span><span>gsub</span><span>(</span><span>&quot;.FE&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>dataset_male</span><span>$</span><span>Indicator.Code</span><span>)</span><span>
</span><span>dataset_male</span><span>$</span><span>gender</span><span> </span><span>&lt;-</span><span> </span><span>&quot;male&quot;</span><span>
</span><span>dataset_fem</span><span> </span><span>&lt;-</span><span> </span><span>dataset_fem</span><span>[</span><span>which</span><span>(</span><span>dataset_fem</span><span>$</span><span>Indicator.Name</span><span> </span><span>%in%</span><span> </span><span>dataset_male</span><span>$</span><span>Indicator.Name</span><span>),</span><span> </span><span>]</span><span>
</span><span>dataset_male</span><span> </span><span>&lt;-</span><span> </span><span>dataset_male</span><span>[</span><span>which</span><span>(</span><span>dataset_male</span><span>$</span><span>Indicator.Name</span><span> </span><span>%in%</span><span> </span><span>dataset_fem</span><span>$</span><span>Indicator.Name</span><span>),</span><span> </span><span>]</span><span>
</span><span>dataset_fem</span><span> </span><span>&lt;-</span><span> </span><span>dataset_fem</span><span>[</span><span>which</span><span>(</span><span>dataset_fem</span><span>$</span><span>Country.Code</span><span> </span><span>%in%</span><span> </span><span>dataset_male</span><span>$</span><span>Country.Code</span><span>),</span><span> </span><span>]</span><span>
</span><span>dataset_male</span><span> </span><span>&lt;-</span><span> </span><span>dataset_male</span><span>[</span><span>which</span><span>(</span><span>dataset_male</span><span>$</span><span>Country.Code</span><span> </span><span>%in%</span><span> </span><span>dataset_fem</span><span>$</span><span>Country.Code</span><span>),</span><span> </span><span>]</span><span>
</span><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>dataset_fem</span><span> </span><span>&lt;-</span><span> </span><span>arrange</span><span>(</span><span>dataset_fem</span><span>,</span><span> </span><span>Country.Code</span><span>)</span><span>
</span><span>dataset_male</span><span> </span><span>&lt;-</span><span> </span><span>arrange</span><span>(</span><span>dataset_male</span><span>,</span><span> </span><span>Country.Code</span><span>)</span><span>
</span><span>dataset_fem</span><span>$</span><span>Country.Code</span><span> </span><span>&lt;-</span><span> </span><span>as.character</span><span>(</span><span>dataset_fem</span><span>$</span><span>Country.Code</span><span>)</span><span>
</span><span>dataset_male</span><span>$</span><span>Country.Code</span><span> </span><span>&lt;-</span><span> </span><span>as.character</span><span>(</span><span>dataset_male</span><span>$</span><span>Country.Code</span><span>)</span><span>
</span><span>save</span><span>(</span><span>dataset_fem</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;dataset_fem.RData&quot;</span><span>)</span><span>
</span><span>save</span><span>(</span><span>dataset_male</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;dataset_male.RData&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>length</span><span>(</span><span>unique</span><span>(</span><span>dataset_fem</span><span>$</span><span>Indicator.Name</span><span>))</span><span> </span><span>==</span><span> </span><span>length</span><span>(</span><span>unique</span><span>(</span><span>dataset_male</span><span>$</span><span>Indicator.Name</span><span>))</span><span>
</span><span>for</span><span> </span><span>(</span><span>n</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>length</span><span>(</span><span>unique</span><span>(</span><span>dataset_fem</span><span>$</span><span>Indicator.Name</span><span>)))</span><span> </span><span>{</span><span>
</span><span>code</span><span> </span><span>&lt;-</span><span> </span><span>unique</span><span>(</span><span>dataset_fem</span><span>$</span><span>Indicator.Name</span><span>)[</span><span>n</span><span>]</span><span>
</span><span>print</span><span>(</span><span>code</span><span>)</span><span>
</span><span>fem</span><span> </span><span>&lt;-</span><span> </span><span>dataset_fem</span><span>[</span><span>which</span><span>(</span><span>dataset_fem</span><span>$</span><span>Indicator.Name</span><span> </span><span>==</span><span> </span><span>code</span><span>),</span><span> </span><span>]</span><span>
</span><span>male</span><span> </span><span>&lt;-</span><span> </span><span>dataset_male</span><span>[</span><span>which</span><span>(</span><span>dataset_male</span><span>$</span><span>Indicator.Name</span><span> </span><span>==</span><span> </span><span>code</span><span>),</span><span> </span><span>]</span><span>
</span><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>nrow</span><span>(</span><span>fem</span><span>))</span><span> </span><span>{</span><span>
</span><span>if</span><span> </span><span>(</span><span>i</span><span> </span><span>==</span><span> </span><span>1</span><span>)</span><span> </span><span>{</span><span>
</span><span>diff</span><span> </span><span>&lt;-</span><span> </span><span>(</span><span>male</span><span>[</span><span>i</span><span>,</span><span> </span><span>5</span><span>:</span><span>61</span><span>]</span><span> </span><span>+</span><span> </span><span>0.001</span><span>)</span><span> </span><span>/</span><span> </span><span>(</span><span>fem</span><span>[</span><span>i</span><span>,</span><span> </span><span>5</span><span>:</span><span>61</span><span>]</span><span> </span><span>+</span><span> </span><span>0.001</span><span>)</span><span>
</span><span>diff_table</span><span> </span><span>&lt;-</span><span> </span><span>cbind</span><span>(</span><span>male</span><span>[</span><span>i</span><span>,</span><span> </span><span>c</span><span>(</span><span>1</span><span>:</span><span>4</span><span>)],</span><span> </span><span>diff</span><span>)</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>diff</span><span> </span><span>&lt;-</span><span> </span><span>(</span><span>male</span><span>[</span><span>i</span><span>,</span><span> </span><span>5</span><span>:</span><span>61</span><span>]</span><span> </span><span>+</span><span> </span><span>0.001</span><span>)</span><span> </span><span>/</span><span> </span><span>(</span><span>fem</span><span>[</span><span>i</span><span>,</span><span> </span><span>5</span><span>:</span><span>61</span><span>]</span><span> </span><span>+</span><span> </span><span>0.001</span><span>)</span><span>
</span><span>diff_table</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>diff_table</span><span>,</span><span> </span><span>cbind</span><span>(</span><span>male</span><span>[</span><span>i</span><span>,</span><span> </span><span>c</span><span>(</span><span>1</span><span>:</span><span>4</span><span>)],</span><span> </span><span>diff</span><span>))</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span><span>if</span><span> </span><span>(</span><span>n</span><span> </span><span>==</span><span> </span><span>1</span><span>)</span><span> </span><span>{</span><span>
</span><span>diff_table_bind</span><span> </span><span>&lt;-</span><span> </span><span>diff_table</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>diff_table_bind</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>diff_table_bind</span><span>,</span><span> </span><span>diff_table</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span><span>diff_table_bind</span><span>$</span><span>Country.Code</span><span> </span><span>&lt;-</span><span> </span><span>as.character</span><span>(</span><span>diff_table_bind</span><span>$</span><span>Country.Code</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>measures</span><span> </span><span>&lt;-</span><span> </span><span>unique</span><span>(</span><span>diff_table_bind</span><span>$</span><span>Indicator.Name</span><span>)</span><span>
</span><span>save</span><span>(</span><span>measures</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;measures.RData&quot;</span><span>)</span><span>
</span><span>years</span><span> </span><span>&lt;-</span><span> </span><span>gsub</span><span>(</span><span>&quot;X&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>colnames</span><span>(</span><span>diff_table_bind</span><span>)[</span><span>-</span><span>c</span><span>(</span><span>1</span><span>:</span><span>4</span><span>)])</span><span>
</span><span>years</span><span> </span><span>&lt;-</span><span> </span><span>years</span><span>[</span><span>-</span><span>length</span><span>(</span><span>years</span><span>)]</span><span>
</span><span>save</span><span>(</span><span>years</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;years.RData&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<h2>The world map</h2>
<p>The map has been downloaded from <a target="_blank" rel="nofollow" href="http://www.naturalearthdata.com/downloads/110m-cultural-vectors/">the Natural Earth Data website</a>. The country borders were reduced by 200 meters with <a target="_blank" rel="nofollow" href="https://www.esri.de/produkte/arcgis-pro">ArcGIS Pro</a>, so that clicking within any country on the map would show the corresponding country’s border as the nearest point. <a target="_blank" rel="nofollow" href="https://www.esri.de/produkte/arcgis-pro">ArcGIS Pro</a> was also used to convert the map to <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Mercator_projection">Mercator projection</a>. The changed shapefiles can be downloaded from <a target="_blank" rel="nofollow" href="https://github.com/ShirinG/blog_posts_prep/tree/master/WGS/shapefiles/changed_borders">my Github repository</a>.</p>
<div>
<pre><code><span>library</span><span>(</span><span>rgdal</span><span>)</span><span>
</span><span>library</span><span>(</span><span>plyr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>scales</span><span>)</span><span>
</span><span>wmap_countries</span><span> </span><span>&lt;-</span><span> </span><span>readOGR</span><span>(</span><span>dsn</span><span> </span><span>=</span><span> </span><span>&quot;shapefiles/changed_borders&quot;</span><span>,</span><span> </span><span>layer</span><span> </span><span>=</span><span> </span><span>&quot;ne_110m_admin_0_countries_smaller_wm&quot;</span><span>)</span><span>
</span><span>wmap_countries_df</span><span> </span><span>&lt;-</span><span> </span><span>fortify</span><span>(</span><span>wmap_countries</span><span>)</span><span>
</span><span>wmap_countries</span><span>@</span><span>data</span><span>$</span><span>id</span><span> </span><span>&lt;-</span><span> </span><span>rownames</span><span>(</span><span>wmap_countries</span><span>@</span><span>data</span><span>)</span><span>
</span><span>wmap_countries_df_final</span><span> </span><span>&lt;-</span><span> </span><span>join</span><span>(</span><span>wmap_countries_df</span><span>,</span><span> </span><span>wmap_countries</span><span>@</span><span>data</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;id&quot;</span><span>)</span><span>
</span><span>wmap_countries_df_final</span><span>$</span><span>adm0_a3</span><span> </span><span>&lt;-</span><span> </span><span>as.character</span><span>(</span><span>wmap_countries_df_final</span><span>$</span><span>adm0_a3</span><span>)</span><span>
</span><span>save</span><span>(</span><span>wmap_countries_df_final</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;wmap_countries_smaller_df_final.RData&quot;</span><span>)</span><span>
</span><span>diff_table_bind</span><span> </span><span>&lt;-</span><span> </span><span>diff_table_bind</span><span>[</span><span>which</span><span>(</span><span>diff_table_bind</span><span>$</span><span>Country.Code</span><span> </span><span>%in%</span><span> </span><span>wmap_countries_df_final</span><span>$</span><span>adm0_a3</span><span>),</span><span> </span><span>]</span><span>
</span><span>save</span><span>(</span><span>diff_table_bind</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;diff_table_bind.RData&quot;</span><span>)</span><span>
</span><span>countries</span><span> </span><span>&lt;-</span><span> </span><span>as.character</span><span>(</span><span>unique</span><span>(</span><span>diff_table_bind</span><span>$</span><span>Country.Name</span><span>))</span><span>
</span><span>save</span><span>(</span><span>countries</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;countries.RData&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tidyr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>ggplot2</span><span>)</span><span>
</span><span>map_theme</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>theme</span><span>(</span><span>panel.grid.minor</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>panel.grid.major</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>panel.background</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>plot.background</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;white&quot;</span><span>),</span><span>
</span><span>panel.border</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>axis.line</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>axis.text.x</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>axis.text.y</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>axis.ticks</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>axis.title.x</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>axis.title.y</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>plot.title</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>18</span><span>)))</span><span>
</span><span>my_theme</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>base_size</span><span> </span><span>=</span><span> </span><span>12</span><span>,</span><span> </span><span>base_family</span><span> </span><span>=</span><span> </span><span>&quot;sans&quot;</span><span>){</span><span>
</span><span>theme_minimal</span><span>(</span><span>base_size</span><span> </span><span>=</span><span> </span><span>base_size</span><span>,</span><span> </span><span>base_family</span><span> </span><span>=</span><span> </span><span>base_family</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>
</span><span>axis.text</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>12</span><span>),</span><span>
</span><span>axis.title</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>14</span><span>),</span><span>
</span><span>panel.grid.major</span><span> </span><span>=</span><span> </span><span>element_line</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>),</span><span>
</span><span>panel.grid.minor</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>panel.background</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;white&quot;</span><span>),</span><span>
</span><span>strip.background</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;royalblue&quot;</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1</span><span>),</span><span>
</span><span>strip.text</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>face</span><span> </span><span>=</span><span> </span><span>&quot;bold&quot;</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>12</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;white&quot;</span><span>),</span><span>
</span><span>legend.position</span><span> </span><span>=</span><span> </span><span>&quot;right&quot;</span><span>,</span><span>
</span><span>panel.border</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>NA</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>0.5</span><span>)</span><span>
</span><span>)</span><span>
</span><span>}</span><span>
</span><span>colfunc</span><span> </span><span>&lt;-</span><span> </span><span>colorRampPalette</span><span>(</span><span>c</span><span>(</span><span>&quot;yellow&quot;</span><span>,</span><span> </span><span>&quot;red&quot;</span><span>))</span><span>
</span></code></pre>
</div>
<p></p>
<h3>Which countries are most biased?</h3>
<p>To explore differences between countries and statistics, I prepared a dataset that contains the first and last non-NA value in each time series of the male/ female ratio data (per statistic and country) to calculate whether the change over time was statistically significant. As for the world map plots, I am using the log2 of the ratios, so that ratios bigger and smaller than 1 can be more easily compared.</p>
<p>You can explore which countries have the strongest bias and the biggest change for each statistic in the app under the tab “Latest ratios”.</p>
<div>
<pre><code><span>year_table</span><span> </span><span>&lt;-</span><span> </span><span>diff_table_bind</span><span>[</span><span>which</span><span>(</span><span>diff_table_bind</span><span>$</span><span>Country.Code</span><span> </span><span>%in%</span><span> </span><span>wmap_countries_df_final</span><span>$</span><span>adm0_a3</span><span>),</span><span> </span><span>]</span><span>
</span><span># last non-NA value
</span><span>last_val</span><span> </span><span>&lt;-</span><span> </span><span>apply</span><span>(</span><span>year_table</span><span>[,</span><span> </span><span>grep</span><span>(</span><span>&quot;^X[0-9]+$&quot;</span><span>,</span><span> </span><span>colnames</span><span>(</span><span>year_table</span><span>))],</span><span> </span><span>1</span><span>,</span><span> </span><span>function</span><span>(</span><span>x</span><span>)</span><span> </span><span>na.omit</span><span>(</span><span>x</span><span>)[</span><span>length</span><span>(</span><span>na.omit</span><span>(</span><span>x</span><span>))]</span><span>
</span><span>)</span><span>
</span><span>last_val_df</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>unlist</span><span>(</span><span>last_val</span><span>))</span><span>
</span><span>last_val_df</span><span>$</span><span>year_of_val</span><span> </span><span>&lt;-</span><span> </span><span>gsub</span><span>(</span><span>&quot;([0-9]+)(.X)([0-9]+)&quot;</span><span>,</span><span> </span><span>&quot;\\3&quot;</span><span>,</span><span> </span><span>rownames</span><span>(</span><span>last_val_df</span><span>))</span><span>
</span><span>rownames</span><span>(</span><span>last_val_df</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>gsub</span><span>(</span><span>&quot;([0-9]+)(.X)([0-9]+)&quot;</span><span>,</span><span> </span><span>&quot;\\1&quot;</span><span>,</span><span> </span><span>rownames</span><span>(</span><span>last_val_df</span><span>))</span><span>
</span><span># first non-NA value
</span><span>first_val</span><span> </span><span>&lt;-</span><span> </span><span>apply</span><span>(</span><span>year_table</span><span>[,</span><span> </span><span>grep</span><span>(</span><span>&quot;^X[0-9]+$&quot;</span><span>,</span><span> </span><span>colnames</span><span>(</span><span>year_table</span><span>))],</span><span> </span><span>1</span><span>,</span><span> </span><span>function</span><span>(</span><span>x</span><span>)</span><span> </span><span>na.omit</span><span>(</span><span>x</span><span>)[</span><span>1</span><span>]</span><span>
</span><span>)</span><span>
</span><span>first_val_df</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>na.omit</span><span>(</span><span>first_val</span><span>))</span><span>
</span><span>year_table_last_val</span><span> </span><span>&lt;-</span><span> </span><span>cbind</span><span>(</span><span>year_table</span><span>[</span><span>rownames</span><span>(</span><span>last_val_df</span><span>),</span><span> </span><span>1</span><span>:</span><span>3</span><span>],</span><span> </span><span>last_val_df</span><span>,</span><span> </span><span>first_val_df</span><span>)</span><span>
</span><span>year_table_last_val</span><span>$</span><span>difference</span><span> </span><span>&lt;-</span><span> </span><span>(</span><span>year_table_last_val</span><span>$</span><span>unlist.last_val.</span><span> </span><span>+</span><span> </span><span>0.001</span><span>)</span><span> </span><span>/</span><span> </span><span>(</span><span>year_table_last_val</span><span>$</span><span>na.omit.first_val.</span><span> </span><span>+</span><span> </span><span>0.001</span><span>)</span><span>
</span><span>save</span><span>(</span><span>year_table_last_val</span><span>,</span><span> </span><span>file</span><span> </span><span>=</span><span> </span><span>&quot;year_table_last_val.RData&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p>To identify countries with strongest biases towards either men or women, I calculated the absolute log2 ratio of male/ female values and subsetted the top 10 countries with highest absolute bias. I then counted for how many statistics each country was in the top 10.</p>
<div>
<pre><code><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>length</span><span>(</span><span>measures</span><span>))</span><span> </span><span>{</span><span>
</span><span>m</span><span> </span><span>&lt;-</span><span> </span><span>measures</span><span>[</span><span>i</span><span>]</span><span>
</span><span>subs</span><span> </span><span>&lt;-</span><span> </span><span>subset</span><span>(</span><span>year_table_last_val</span><span>,</span><span> </span><span>Indicator.Name</span><span> </span><span>==</span><span> </span><span>m</span><span>)[,</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>4</span><span>)]</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>abs_ratio</span><span> </span><span>=</span><span> </span><span>abs</span><span>(</span><span>log2</span><span>(</span><span>unlist.last_val.</span><span>)))</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>desc</span><span>(</span><span>abs_ratio</span><span>))</span><span>
</span><span>subs</span><span>$</span><span>measure</span><span> </span><span>&lt;-</span><span> </span><span>m</span><span>
</span><span>if</span><span> </span><span>(</span><span>i</span><span> </span><span>==</span><span> </span><span>1</span><span>)</span><span> </span><span>{</span><span>
</span><span>subs_comb</span><span> </span><span>&lt;-</span><span> </span><span>subs</span><span>[</span><span>1</span><span>:</span><span>10</span><span>,</span><span> </span><span>]</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>subs_comb</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>subs_comb</span><span>,</span><span> </span><span>subs</span><span>[</span><span>1</span><span>:</span><span>10</span><span>,</span><span> </span><span>])</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>subs_comb_ordered</span><span> </span><span>&lt;-</span><span> </span><span>arrange</span><span>(</span><span>as.data.frame</span><span>(</span><span>table</span><span>(</span><span>subs_comb</span><span>$</span><span>Country.Name</span><span>)),</span><span> </span><span>desc</span><span>(</span><span>Freq</span><span>))</span><span>
</span><span>subs_comb</span><span>$</span><span>Country.Name</span><span> </span><span>&lt;-</span><span> </span><span>factor</span><span>(</span><span>subs_comb</span><span>$</span><span>Country.Name</span><span>,</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>subs_comb_ordered</span><span>$</span><span>Var1</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>subs_comb</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>Country.Name</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;navyblue&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.7</span><span>)</span><span> </span><span>+</span><span>
</span><span>coord_flip</span><span>()</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Most biased countries&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;For how many statistics were the countries in top 10 with highest absolute bias&quot;</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/shiny/2017/02/06/WGS_final_files/figure-markdown_github/unnamed-chunk-15-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>This plot shows that there are quite a few European/ Western countries with high biases in some statistics. For example:</p>
<div>
<pre><code><span>arrange</span><span>(</span><span>subs_comb</span><span>[</span><span>which</span><span>(</span><span>subs_comb</span><span>$</span><span>Country.Name</span><span> </span><span>%in%</span><span> </span><span>c</span><span>(</span><span>&quot;Germany&quot;</span><span>,</span><span> </span><span>&quot;Netherlands&quot;</span><span>,</span><span> </span><span>&quot;United States&quot;</span><span>,</span><span> </span><span>&quot;United Kingdom&quot;</span><span>)),</span><span> </span><span>-</span><span> </span><span>2</span><span>],</span><span> </span><span>Country.Name</span><span>,</span><span> </span><span>desc</span><span>(</span><span>abs_ratio</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##      Country.Name abs_ratio
## 1     Netherlands 1.4678833
## 2     Netherlands 1.4674207
## 3     Netherlands 1.3615747
## 4     Netherlands 1.1248210
## 5     Netherlands 0.9993447
## 6     Netherlands 0.6601651
## 7     Netherlands 0.5512176
## 8     Netherlands 0.1847042
## 9         Germany 8.2336197
## 10        Germany 2.0428320
## 11        Germany 1.1641113
## 12        Germany 1.0957438
## 13        Germany 0.9825083
## 14        Germany 0.7356144
## 15        Germany 0.6274894
## 16 United Kingdom 1.0932072
## 17 United Kingdom 0.8659095
## 18 United Kingdom 0.4347145
## 19  United States 1.3212071
##                                                                                    measure
## 1        Educational attainment completed Doctoral or equivalent population 25+ years  (%)
## 2  Educational attainment competed Doctoral or equivalent population 25+  (%) (cumulative)
## 3                              Borrowed from a store by buying on credit  (% age 15+) [w1]
## 4                           Prevalence of stunting height for age  (% of children under 5)
## 5                        Prevalence of underweight weight for age  (% of children under 5)
## 6                                                       Loan in the past year  (% age 15+)
## 7                      Coming up with emergency funds: somewhat possible  (% age 15+) [w2]
## 8                                                     Progression to secondary school  (%)
## 9                  Prevalence of severe wasting weight for height  (% of children under 5)
## 10                                          Part time employment  (% of total  employment)
## 11                                         Time-related underemployment  (% of employment)
## 12              Educational attainment completed lower secondary population 25+ years  (%)
## 13         Educational attainment completed short-cycle tertiary population 25+ years  (%)
## 14                                       Borrowed from family or friends  (% age 15+) [ts]
## 15                     Coming up with emergency funds: somewhat possible  (% age 15+) [w2]
## 16                               Borrowed for health or medical purposes  (% age 15+) [w2]
## 17       Average number of hours spent on unpaid domestic work (housework and child care) 
## 18                                   Borrowed any money in the past year  (% age 15+) [w2]
## 19                          Prevalence of stunting height for age  (% of children under 5)
</code></pre>
</div>
<p>So, let’s take a closer look at the countries: I am calculating the sums of absolute male/ female ratios (again log2) for each country and show it on the map:</p>
<div>
<pre><code><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>length</span><span>(</span><span>countries</span><span>))</span><span> </span><span>{</span><span>
</span><span>c</span><span> </span><span>&lt;-</span><span> </span><span>countries</span><span>[</span><span>i</span><span>]</span><span>
</span><span>subs</span><span> </span><span>&lt;-</span><span> </span><span>subset</span><span>(</span><span>year_table_last_val</span><span>,</span><span> </span><span>Country.Name</span><span> </span><span>==</span><span> </span><span>c</span><span>)[,</span><span> </span><span>c</span><span>(</span><span>2</span><span>,</span><span> </span><span>3</span><span>,</span><span> </span><span>4</span><span>)]</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>abs_ratio</span><span> </span><span>=</span><span> </span><span>abs</span><span>(</span><span>log2</span><span>(</span><span>unlist.last_val.</span><span>)))</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>desc</span><span>(</span><span>abs_ratio</span><span>))</span><span>
</span><span>if</span><span> </span><span>(</span><span>i</span><span> </span><span>==</span><span> </span><span>1</span><span>)</span><span> </span><span>{</span><span>
</span><span>subs_country</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>country</span><span> </span><span>=</span><span> </span><span>c</span><span>,</span><span> </span><span>Country.Code</span><span> </span><span>=</span><span> </span><span>subs</span><span>$</span><span>Country.Code</span><span>,</span><span> </span><span>sum_abs_ratio</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>subs</span><span>$</span><span>abs_ratio</span><span>))</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>subs_country</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>subs_country</span><span>,</span><span> </span><span>data.frame</span><span>(</span><span>country</span><span> </span><span>=</span><span> </span><span>c</span><span>,</span><span> </span><span>Country.Code</span><span> </span><span>=</span><span> </span><span>subs</span><span>$</span><span>Country.Code</span><span>,</span><span> </span><span>sum_abs_ratio</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>subs</span><span>$</span><span>abs_ratio</span><span>)))</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>left_join</span><span>(</span><span>subset</span><span>(</span><span>wmap_countries_df_final</span><span>,</span><span> </span><span>!</span><span>continent</span><span> </span><span>==</span><span> </span><span>&quot;Antarctica&quot;</span><span>),</span><span> </span><span>subs_country</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;adm0_a3&quot;</span><span> </span><span>=</span><span> </span><span>&quot;Country.Code&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>long</span><span>,</span><span> </span><span>lat</span><span>,</span><span> </span><span>group</span><span> </span><span>=</span><span> </span><span>group</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>sum_abs_ratio</span><span>))</span><span> </span><span>+</span><span>
</span><span>coord_equal</span><span>()</span><span> </span><span>+</span><span>
</span><span>map_theme</span><span> </span><span>+</span><span>
</span><span>geom_polygon</span><span>()</span><span> </span><span>+</span><span>
</span><span>geom_path</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;white&quot;</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>0.5</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;General gender bias per country&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Sum of absolute gender biases for all statistics&quot;</span><span>,</span><span>
</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;sum of absolute\nlog2 of male / female&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_gradient2</span><span>(</span><span>low</span><span> </span><span>=</span><span> </span><span>&quot;blue&quot;</span><span>,</span><span> </span><span>mid</span><span> </span><span>=</span><span> </span><span>&quot;blue&quot;</span><span>,</span><span> </span><span>high</span><span> </span><span>=</span><span> </span><span>&quot;red&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/shiny/2017/02/06/WGS_final_files/figure-markdown_github/unnamed-chunk-18-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h2>Statistical Analysis</h2>
<p>You can explore the results for the individual statistics in the app under “Analysis – Plots” and “Analysis – Tests”.</p>
<p></p>
<h3>The variables</h3>
<p>The world data contains information on each country. I used the following variables to test for statistical significance:</p>
<ol>
<li>economic status,</li>
<li>income group,</li>
<li>estimated population size,</li>
<li>estimated gross domestic product (GDP) and</li>
<li>continent.</li>
</ol>
<p>Because not all data was normally distributed I explored <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Nonparametric_statistics">non-parametric models</a>, as well as an <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Analysis_of_variance">ANOVA</a>. The plots for each statistic can be explored in my app, so here, I am only showing the results for which measures were statistically significant for the different variables.</p>
<p></p>
<h2>Wilcoxon Signed-Rank Test</h2>
<p>The <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test">Wilcoxon Signed-Rank test</a> can be used to test whether observations from repeated measures differ statistically from each other. Here, I am using it to test whether the most recent recorded value is significantly different from the first recorded value. Because I am testing the same data multiple times, I am correcting the p-values and only consider them significant with False Discovery Rate (FDR) below 10% (adjusted p-value &lt; 0.1).</p>
<div>
<pre><code><span>country.inf</span><span> </span><span>&lt;-</span><span> </span><span>subset</span><span>(</span><span>wmap_countries_df_final</span><span>[,</span><span> </span><span>c</span><span>(</span><span>17</span><span>,</span><span> </span><span>25</span><span>,</span><span> </span><span>26</span><span>,</span><span> </span><span>31</span><span>,</span><span> </span><span>32</span><span>,</span><span> </span><span>42</span><span>,</span><span> </span><span>43</span><span>,</span><span> </span><span>47</span><span>,</span><span> </span><span>48</span><span>,</span><span> </span><span>62</span><span>)],</span><span> </span><span>!</span><span>continent</span><span> </span><span>==</span><span> </span><span>&quot;Antarctica&quot;</span><span>)</span><span>
</span><span>country.inf</span><span> </span><span>&lt;-</span><span> </span><span>country.inf</span><span>[</span><span>!</span><span>duplicated</span><span>(</span><span>country.inf</span><span>),</span><span> </span><span>]</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>length</span><span>(</span><span>measures</span><span>))</span><span> </span><span>{</span><span>
</span><span>m</span><span> </span><span>&lt;-</span><span> </span><span>measures</span><span>[</span><span>i</span><span>]</span><span>
</span><span>stats_test</span><span> </span><span>&lt;-</span><span> </span><span>left_join</span><span>(</span><span>subset</span><span>(</span><span>year_table_last_val</span><span>,</span><span> </span><span>Indicator.Name</span><span> </span><span>==</span><span> </span><span>m</span><span>),</span><span> </span><span>country.inf</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Country.Code&quot;</span><span> </span><span>=</span><span> </span><span>&quot;adm0_a3&quot;</span><span>))</span><span>
</span><span>table</span><span> </span><span>&lt;-</span><span> </span><span>wilcox.test</span><span>(</span><span>log2</span><span>(</span><span>stats_test</span><span>$</span><span>na.omit.first_val.</span><span>),</span><span> </span><span>log2</span><span>(</span><span>stats_test</span><span>$</span><span>unlist.last_val.</span><span>),</span><span> </span><span>paired</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>table</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>Indicator.Name</span><span> </span><span>=</span><span> </span><span>paste</span><span>(</span><span>m</span><span>),</span><span>
</span><span>diff</span><span> </span><span>=</span><span> </span><span>(</span><span>median</span><span>(</span><span>stats_test</span><span>$</span><span>unlist.last_val.</span><span>)</span><span> </span><span>+</span><span> </span><span>0.0001</span><span>)</span><span> </span><span>/</span><span> </span><span>(</span><span>median</span><span>(</span><span>stats_test</span><span>$</span><span>na.omit.first_val.</span><span>)</span><span> </span><span>+</span><span> </span><span>0.0001</span><span>),</span><span>
</span><span>V</span><span> </span><span>=</span><span> </span><span>table</span><span>$</span><span>statistic</span><span>,</span><span>
</span><span>p.value</span><span> </span><span>=</span><span> </span><span>table</span><span>$</span><span>p.value</span><span>,</span><span>
</span><span>alternative</span><span> </span><span>=</span><span> </span><span>table</span><span>$</span><span>alternative</span><span>)</span><span>
</span><span>if</span><span> </span><span>(</span><span>i</span><span> </span><span>==</span><span> </span><span>1</span><span>)</span><span> </span><span>{</span><span>
</span><span>table_wilcox</span><span> </span><span>&lt;-</span><span> </span><span>table</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>table_wilcox</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>table_wilcox</span><span>,</span><span> </span><span>table</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span><span>table_wilcox</span><span>$</span><span>p.adj</span><span> </span><span>&lt;-</span><span> </span><span>p.adjust</span><span>(</span><span>table_wilcox</span><span>$</span><span>p.value</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;fdr&quot;</span><span>)</span><span>
</span><span>table_wilcox</span><span>$</span><span>significant</span><span> </span><span>&lt;-</span><span> </span><span>as.factor</span><span>(</span><span>ifelse</span><span>(</span><span>table_wilcox</span><span>$</span><span>p.adj</span><span> </span><span>&lt;</span><span> </span><span>0.1</span><span>,</span><span> </span><span>&quot;significant&quot;</span><span>,</span><span> </span><span>&quot;non-significant&quot;</span><span>))</span><span>
</span><span>sig</span><span> </span><span>&lt;-</span><span> </span><span>subset</span><span>(</span><span>table_wilcox</span><span>,</span><span> </span><span>significant</span><span> </span><span>==</span><span> </span><span>&quot;significant&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>left_join</span><span>(</span><span>subset</span><span>(</span><span>year_table_last_val</span><span>,</span><span> </span><span>Indicator.Name</span><span> </span><span>%in%</span><span> </span><span>sig</span><span>$</span><span>Indicator.Name</span><span>),</span><span> </span><span>country.inf</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Country.Code&quot;</span><span> </span><span>=</span><span> </span><span>&quot;adm0_a3&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>x</span><span>,</span><span> </span><span>y</span><span>,</span><span> </span><span>unlist.last_val.</span><span>,</span><span> </span><span>na.omit.first_val.</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>Indicator.Name</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>log2</span><span>(</span><span>y</span><span>),</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>x</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>x</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_boxplot</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.5</span><span>,</span><span> </span><span>outlier.shape</span><span> </span><span>=</span><span> </span><span>NA</span><span>)</span><span> </span><span>+</span><span>
</span><span>coord_flip</span><span>()</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_y_continuous</span><span>(</span><span>limits</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>-3</span><span>,</span><span> </span><span>3</span><span>))</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Measures with significant difference over time&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Wilcoxon Signed-Rank Test&quot;</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;log2(male/ female)&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/shiny/2017/02/06/WGS_final_files/figure-markdown_github/unnamed-chunk-37-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h2>Kruskal-Wallis Test</h2>
<p>The <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Kruskal%E2%80%93Wallis_one-way_analysis_of_variance">Kruskal-Wallis test</a> is a simple non-parametric statistical test. I used it here to test for each variable of interest whether it has a statistically significant effect on the most recent recorded value. Because I am testing the same data multiple times, I am correcting the p-values and only consider them significant with False Discovery Rate (FDR) below 10% (adjusted p-value &lt; 0.1).</p>
<div>
<pre><code><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>length</span><span>(</span><span>measures</span><span>))</span><span> </span><span>{</span><span>
</span><span>m</span><span> </span><span>&lt;-</span><span> </span><span>measures</span><span>[</span><span>i</span><span>]</span><span>
</span><span>stats_test</span><span> </span><span>&lt;-</span><span> </span><span>left_join</span><span>(</span><span>subset</span><span>(</span><span>year_table_last_val</span><span>,</span><span> </span><span>Indicator.Name</span><span> </span><span>==</span><span> </span><span>m</span><span>),</span><span> </span><span>country.inf</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Country.Code&quot;</span><span> </span><span>=</span><span> </span><span>&quot;adm0_a3&quot;</span><span>))</span><span>
</span><span>stats_test</span><span>$</span><span>economy_rank</span><span> </span><span>&lt;-</span><span> </span><span>as.numeric</span><span>(</span><span>as.character</span><span>(</span><span>gsub</span><span>(</span><span>&quot;(^[1-7])(.*)&quot;</span><span>,</span><span> </span><span>&quot;\\1&quot;</span><span>,</span><span> </span><span>stats_test</span><span>$</span><span>economy</span><span>)))</span><span>
</span><span>stats_test</span><span>$</span><span>income_grp_rank</span><span> </span><span>&lt;-</span><span> </span><span>as.numeric</span><span>(</span><span>as.character</span><span>(</span><span>gsub</span><span>(</span><span>&quot;(^[1-5])(.*)&quot;</span><span>,</span><span> </span><span>&quot;\\1&quot;</span><span>,</span><span> </span><span>stats_test</span><span>$</span><span>income_grp</span><span>)))</span><span>
</span><span>k_economy_rank</span><span> </span><span>&lt;-</span><span> </span><span>kruskal.test</span><span>(</span><span>log2</span><span>(</span><span>unlist.last_val.</span><span>)</span><span> </span><span>~</span><span> </span><span>economy_rank</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>stats_test</span><span>)</span><span>
</span><span>k_income_rank</span><span> </span><span>&lt;-</span><span> </span><span>kruskal.test</span><span>(</span><span>log2</span><span>(</span><span>unlist.last_val.</span><span>)</span><span> </span><span>~</span><span> </span><span>income_grp_rank</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>stats_test</span><span>)</span><span>
</span><span>k_pop</span><span> </span><span>&lt;-</span><span> </span><span>kruskal.test</span><span>(</span><span>log2</span><span>(</span><span>unlist.last_val.</span><span>)</span><span> </span><span>~</span><span> </span><span>pop_est</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>stats_test</span><span>)</span><span>
</span><span>k_gdp</span><span> </span><span>&lt;-</span><span> </span><span>kruskal.test</span><span>(</span><span>log2</span><span>(</span><span>unlist.last_val.</span><span>)</span><span> </span><span>~</span><span> </span><span>gdp_md_est</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>stats_test</span><span>)</span><span>
</span><span>k_continent</span><span> </span><span>&lt;-</span><span> </span><span>kruskal.test</span><span>(</span><span>log2</span><span>(</span><span>unlist.last_val.</span><span>)</span><span> </span><span>~</span><span> </span><span>continent</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>stats_test</span><span>)</span><span>
</span><span>kruskal_last_val</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>
</span><span>group</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;economy (rank)&quot;</span><span>,</span><span> </span><span>&quot;income_grp (rank)&quot;</span><span>,</span><span> </span><span>&quot;pop_est&quot;</span><span>,</span><span> </span><span>&quot;gdp_md_est&quot;</span><span>,</span><span> </span><span>&quot;continent&quot;</span><span>),</span><span>
</span><span>p.val</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>k_economy_rank</span><span>$</span><span>p.value</span><span>,</span><span> </span><span>k_income_rank</span><span>$</span><span>p.value</span><span>,</span><span> </span><span>k_pop</span><span>$</span><span>p.value</span><span>,</span><span> </span><span>k_gdp</span><span>$</span><span>p.value</span><span>,</span><span> </span><span>k_continent</span><span>$</span><span>p.value</span><span>))</span><span>
</span><span>kruskal_last_val</span><span>$</span><span>p.adj</span><span> </span><span>&lt;-</span><span> </span><span>p.adjust</span><span>(</span><span>kruskal_last_val</span><span>$</span><span>p.val</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;fdr&quot;</span><span>)</span><span>
</span><span>if</span><span> </span><span>(</span><span>i</span><span> </span><span>==</span><span> </span><span>1</span><span>)</span><span> </span><span>{</span><span>
</span><span>kruskal_last_val_df</span><span> </span><span>&lt;-</span><span> </span><span>kruskal_last_val</span><span>[,</span><span> </span><span>-2</span><span>]</span><span>
</span><span>colnames</span><span>(</span><span>kruskal_last_val_df</span><span>)[</span><span>2</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>paste</span><span>(</span><span>m</span><span>)</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>pre</span><span> </span><span>&lt;-</span><span> </span><span>kruskal_last_val</span><span>[,</span><span> </span><span>-2</span><span>]</span><span>
</span><span>colnames</span><span>(</span><span>pre</span><span>)[</span><span>2</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>paste</span><span>(</span><span>m</span><span>)</span><span>
</span><span>kruskal_last_val_df</span><span> </span><span>&lt;-</span><span> </span><span>left_join</span><span>(</span><span>kruskal_last_val_df</span><span>,</span><span> </span><span>pre</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;group&quot;</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>eco</span><span> </span><span>&lt;-</span><span> </span><span>t</span><span>(</span><span>kruskal_last_val_df</span><span>[</span><span>1</span><span>,</span><span> </span><span>-1</span><span>])</span><span>
</span><span>eco_sig</span><span> </span><span>&lt;-</span><span> </span><span>eco</span><span>[</span><span>which</span><span>(</span><span>eco</span><span> </span><span>&lt;</span><span> </span><span>0.1</span><span>),</span><span> </span><span>]</span><span>
</span><span>left_join</span><span>(</span><span>subset</span><span>(</span><span>year_table_last_val</span><span>,</span><span> </span><span>Indicator.Name</span><span> </span><span>%in%</span><span> </span><span>names</span><span>(</span><span>eco_sig</span><span>)),</span><span> </span><span>country.inf</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Country.Code&quot;</span><span> </span><span>=</span><span> </span><span>&quot;adm0_a3&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>y</span><span> </span><span>=</span><span> </span><span>log2</span><span>(</span><span>unlist.last_val.</span><span>),</span><span> </span><span>x</span><span> </span><span>=</span><span> </span><span>Indicator.Name</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>economy</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>economy</span><span>))</span><span> </span><span>+</span><span>
</span><span>coord_flip</span><span>()</span><span> </span><span>+</span><span>
</span><span>geom_boxplot</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.5</span><span>,</span><span> </span><span>outlier.shape</span><span> </span><span>=</span><span> </span><span>NA</span><span>)</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_y_continuous</span><span>(</span><span>limits</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>-3</span><span>,</span><span> </span><span>3</span><span>))</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Measures with significant difference in economy&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Kruskal-Wallis Test&quot;</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;log2(male/ female)&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/shiny/2017/02/06/WGS_final_files/figure-markdown_github/unnamed-chunk-41-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>inc</span><span> </span><span>&lt;-</span><span> </span><span>t</span><span>(</span><span>kruskal_last_val_df</span><span>[</span><span>2</span><span>,</span><span> </span><span>-1</span><span>])</span><span>
</span><span>inc_sig</span><span> </span><span>&lt;-</span><span> </span><span>inc</span><span>[</span><span>which</span><span>(</span><span>inc</span><span> </span><span>&lt;</span><span> </span><span>0.1</span><span>),</span><span> </span><span>]</span><span>
</span><span>left_join</span><span>(</span><span>subset</span><span>(</span><span>year_table_last_val</span><span>,</span><span> </span><span>Indicator.Name</span><span> </span><span>%in%</span><span> </span><span>names</span><span>(</span><span>inc_sig</span><span>)),</span><span> </span><span>country.inf</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Country.Code&quot;</span><span> </span><span>=</span><span> </span><span>&quot;adm0_a3&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>y</span><span> </span><span>=</span><span> </span><span>log2</span><span>(</span><span>unlist.last_val.</span><span>),</span><span> </span><span>x</span><span> </span><span>=</span><span> </span><span>Indicator.Name</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>income_grp</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>income_grp</span><span>))</span><span> </span><span>+</span><span>
</span><span>coord_flip</span><span>()</span><span> </span><span>+</span><span>
</span><span>geom_boxplot</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.5</span><span>,</span><span> </span><span>outlier.shape</span><span> </span><span>=</span><span> </span><span>NA</span><span>)</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_y_continuous</span><span>(</span><span>limits</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>-3</span><span>,</span><span> </span><span>3</span><span>))</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Measures with significant difference in income group&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Kruskal-Wallis Test&quot;</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;log2(male/ female)&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/shiny/2017/02/06/WGS_final_files/figure-markdown_github/unnamed-chunk-42-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>cont</span><span> </span><span>&lt;-</span><span> </span><span>t</span><span>(</span><span>kruskal_last_val_df</span><span>[</span><span>5</span><span>,</span><span> </span><span>-1</span><span>])</span><span>
</span><span>cont_sig</span><span> </span><span>&lt;-</span><span> </span><span>cont</span><span>[</span><span>which</span><span>(</span><span>cont</span><span> </span><span>&lt;</span><span> </span><span>0.1</span><span>),</span><span> </span><span>]</span><span>
</span><span>left_join</span><span>(</span><span>subset</span><span>(</span><span>year_table_last_val</span><span>,</span><span> </span><span>Indicator.Name</span><span> </span><span>%in%</span><span> </span><span>names</span><span>(</span><span>cont_sig</span><span>)),</span><span> </span><span>country.inf</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Country.Code&quot;</span><span> </span><span>=</span><span> </span><span>&quot;adm0_a3&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>y</span><span> </span><span>=</span><span> </span><span>log2</span><span>(</span><span>unlist.last_val.</span><span>),</span><span> </span><span>x</span><span> </span><span>=</span><span> </span><span>Indicator.Name</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>continent</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>continent</span><span>))</span><span> </span><span>+</span><span>
</span><span>coord_flip</span><span>()</span><span> </span><span>+</span><span>
</span><span>geom_boxplot</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.5</span><span>,</span><span> </span><span>outlier.shape</span><span> </span><span>=</span><span> </span><span>NA</span><span>)</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_y_continuous</span><span>(</span><span>limits</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>-3</span><span>,</span><span> </span><span>3</span><span>))</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Measures with significant difference in continent&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Kruskal-Wallis Test&quot;</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;log2(male/ female)&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/shiny/2017/02/06/WGS_final_files/figure-markdown_github/unnamed-chunk-43-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>For population and GDP, there were no statistically significant measures.</p>
<hr>
<p></p>
<div>
<pre><code><span>sessionInfo</span><span>()</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## R version 3.3.2 (2016-10-31)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 7 x64 (build 7601) Service Pack 1
## 
## locale:
## [1] LC_COLLATE=English_United States.1252 
## [2] LC_CTYPE=English_United States.1252   
## [3] LC_MONETARY=English_United States.1252
## [4] LC_NUMERIC=C                          
## [5] LC_TIME=English_United States.1252    
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] ggplot2_2.2.1 tidyr_0.6.1   dplyr_0.5.0  
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_0.12.9      digest_0.6.12    rprojroot_1.2    assertthat_0.1  
##  [5] plyr_1.8.4       grid_3.3.2       R6_2.2.0         gtable_0.2.0    
##  [9] DBI_0.5-1        backports_1.0.5  magrittr_1.5     scales_0.4.1    
## [13] evaluate_0.10    stringi_1.1.2    lazyeval_0.2.0   rmarkdown_1.3   
## [17] labeling_0.3     tools_3.3.2      stringr_1.1.0    munsell_0.4.3   
## [21] yaml_2.1.14      colorspace_1.3-2 htmltools_0.3.5  knitr_1.15.1    
## [25] tibble_1.2
</code></pre>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://shiring.github.io/shiny/2017/02/06/WGS_final"> Shirin's playgRound</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/-tu2H_nq_es"></a></div>