<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/tidyverse-practice-mapping-large-european-cities/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://sharpsightlabs.com/blog/map-european-cities-large-population/"> r-bloggers – SHARP SIGHT LABS</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a rel="nofollow" href="http://sharpsightlabs.com/join-newsletter-rapid-mastery/" target="_blank"><br>
<img sizes="(max-width: 600px) 100vw, 600px" src="https://i1.wp.com/sharpsightlabs.com/wp-content/uploads/2017/08/european-cities_large-pop_map-600x514.png?w=450" alt="" class="wp-image-3108" data-recalc-dims="1"><br>
</a></p>
<p>As noted in <a href="http://sharpsightlabs.com/blog/simple-practice-data-wrangling-iris-dataset/">several recent posts</a>, when you’re learning R and R’s Tidyverse packages, it’s important to <a href="http://sharpsightlabs.com/blog/simple-practice-maps-tidyverse/">break everything down into small units</a> that you can learn. </p>
<p>What that means is that you need to identify the most important tools and functions of the Tidyverse, and then practice them until you are fluent. </p>
<p>But once you have mastered the essential functions as isolated units, you need to put them together. By putting the individual piece together, you solidify your knowledge of how they work <em>individually</em> but also begin to learn how you can combine small tools together to create novel effects.</p>
<p>With that in mind, I want to show you another small project. Here, we’re going to use a fairly small set of functions to <a href="http://sharpsightlabs.com/blog/simple-practice-maps-tidyverse/">create a map</a> of the largest cities in Europe.</p>
<p>As we do this, pay attention:</p>
<ul>
<li>How many packages and functions do you really need?</li>
<li>Evaluate: how long would it really take to memorize each individual function? (Hint: it’s much, much less time than you think.)</li>
<li>Which functions have you seen before? Are some of the functions and techniques used more often than others (if you look across many different analyses)?</li>
</ul>
<p>Ok, with those questions in mind, let’s get after it.</p>
<p>First we’ll just load a few packages.</p>
<pre class="wp-code-highlight">
#==============
# LOAD PACKAGES
#==============

library(rvest)
library(tidyverse)
library(ggmap)
library(stringr)
</pre>
<p>Next, we’re going to use the  package to scrape data from Wikipedia. The data that we are gathering is data about the largest cities in Europe. You can read more about the data on Wikipedia.</p>
<pre class="wp-code-highlight">
#===========================
# SCRAPE DATA FROM WIKIPEDIA
#===========================
html.population &lt;- read_html('https://en.wikipedia.org/wiki/List_of_European_cities_by_population_within_city_limits')

df.euro_cities &lt;- html.population %>%
  html_nodes(&quot;table&quot;) %>%
  .[[2]] %>%
  html_table()


# inspect
df.euro_cities %>% head()
df.euro_cities %>% names()
</pre>
<p>Here at Sharp Sight, we haven’t worked with the  package in too many examples, so you might not be familiar with it.</p>
<p>Having said that, just take a close look. How many functions did we use from ? Could you memorize them? How long would it take?</p>
<p>Ok. Now we’ll do a little data cleaning.</p>
<p>First, we’re going to <a href="http://sharpsightlabs.com/blog/dplyr-intro-data-manipulation-with-r/">remove some of the variables using </a>. We are using the minus sign (‘-‘) in front of the names of the variables that we want to remove.</p>
<pre class="wp-code-highlight">
#============================
# REMOVE EXTRANEOUS VARIABLES
#============================
df.euro_cities &lt;- select(df.euro_cities, -Date, -Image, -Location, -`Ref.`, -`2011 Eurostat\npopulation[3]`)


# inspect
df.euro_cities %>% names()
</pre>
<p>After removing the variables that we don’t want, we only have four variables. These remaining raw variable names could be cleaned up a little.</p>
<p>Ideally, we want names that are lower case (because they are easier to type). We also want variable names that are brief and descriptive.</p>
<p>In this case, renaming these variables to be brief, descriptive, and lower-case is fairly straightforward. Here, we will use very simple variable names: like rank, city, country, and population.</p>
<p>To add these new variable names, we can simply assign them by using the  function.</p>
<pre class="wp-code-highlight">
#===============
# RENAME COLUMNS
#===============
colnames(df.euro_cities) &lt;- c(&quot;rank&quot;, &quot;city&quot;, &quot;country&quot;, &quot;population&quot;)


# inspect
df.euro_cities %>% names()
df.euro_cities %>% head()
</pre>
<p>Now that we have clean variable names, we will do a little modification of the data itself.</p>
<p>When we scraped the data from Wikipedia, some extraneous characters appeared in the  variable. Essentially, there were some leading digits and special characters that appear to be useless artifacts of the scraping process. We want to remove these extraneous characters and parse the population data into a proper numeric.</p>
<p>To do this, we will use a few functions from the  package. </p>
<p>First, we use  to extract the population data. When we do this, we are extracting everything from the ‘♠’ character to the end of the string (note: to do this, we are using a regular expression in ). </p>
<p>This is a quick way to get the numbers at the end of the string, but we actually don’t want to keep the ‘♠’ character. So, after we extract the population numbers (along with the ‘♠’), we then strip off the ‘♠’ character by using . </p>
<pre class="wp-code-highlight">
#========================================================================
# CLEAN UP VARIABLE: population
# - when the data are scraped, there are some extraneous characters
#   in the &quot;population&quot; variable.
#   ... you can see leading numbers and some other items
# - We will use stringr functions to extract the actual population data
#   (and remove the stuff we don't want)
# - We are executing this transformation inside dplyr::mutate() to 
#   modify the variable inside the dataframe
#========================================================================

df.euro_cities &lt;- df.euro_cities %>% mutate(population = str_extract(population, &quot;♠.*$&quot;) %>% str_replace(&quot;♠&quot;,&quot;&quot;) %>% parse_number())

df.euro_cities %>% head()
</pre>
<p>We will also do some quick data wrangling on the city names. Two of the city names on the Wikipedia page (Istanbul and Moscow) had footnotes. Because of this, those two city names had extra bracket characters when we read them in (e.g. “Istanbul[a]”). </p>
<p>We want to strip off those footnotes. To do this we will once again use  to strip away the information that we don’t want.</p>
<pre class="wp-code-highlight">
#==========================================================================
# REMOVE &quot;notes&quot; FROM CITY NAMES
# - two cities had extra characters for footnotes
#   ... we will remove these using stringr::str_replace and dplyr::mutate()
#==========================================================================

df.euro_cities &lt;- df.euro_cities %>% mutate(city = str_replace(city, &quot;\\[.\\]&quot;,&quot;&quot;))
  
df.euro_cities %>% head()
</pre>
<p>For the sake of making the data a little easier to explain, we’re going to filter the data to records where the population is over 1,000,000. </p>
<p>Keep in mind: this is a <a href="http://sharpsightlabs.com/blog/dplyr-intro-data-manipulation-with-r/">straightforward use of </a>; this is the sort of thing that you should be able to do with your eyes closed.</p>
<pre class="wp-code-highlight">
#=========================
# REMOVE CITIES UNDER 1 MM
#=========================

df.euro_cities &lt;- filter(df.euro_cities, population >= 1000000)

#=================
# COERCE TO TIBBLE
#=================

df.euro_cities &lt;- df.euro_cities %>% as_tibble()
</pre>
<p>Before we map the cities on a map, we need to get geospatial information. That is, we need to geocode these records.</p>
<p>To do this, we will use the  function to get the longitude and latitude.</p>
<p>After obtaining the geo data, we will join it back to the original data using .</p>
<pre class="wp-code-highlight">
#========================================================
# GEOCODE
# - here, we're just getting longitude and latitude data 
#   using ggmap::geocode()
#========================================================

data.geo &lt;- geocode(df.euro_cities$city)

df.euro_cities &lt;- cbind(df.euro_cities, data.geo)


#inspect
df.euro_cities
</pre>
<p>To map the data points, we also need a map that will sit in the background, underneath the points.</p>
<p>We will use the function <a href="http://sharpsightlabs.com/blog/simple-practice-maps-tidyverse/"> to get a world map</a>.</p>
<pre class="wp-code-highlight">
#==============
# GET WORLD MAP
#==============

map.europe &lt;- map_data(&quot;world&quot;)

</pre>
<p>Now that the data are clean, and we have a world map, we will plot the data.</p>
<pre class="wp-code-highlight">
#=================================
# PLOT BASIC MAP
# - this map is &quot;just the basics&quot;
#=================================

ggplot() +
  geom_polygon(data = map.europe, aes(x = long, y = lat, group = group)) +
  geom_point(data = df.euro_cities, aes(x = lon, y = lat, size = population), color = &quot;red&quot;, alpha = .3) +
  coord_cartesian(xlim = c(-9,45), ylim = c(32,70))
</pre>
<p>This first plot is a “first iteration.” In this version, we haven’t done any serious formatting. It’s just a “first pass” to make sure that the data are in the right format. If we had found anything “out of line,” we would go back to an earlier part of the analysis and modify our code to correct any problems in the data.</p>
<p><a rel="nofollow" href="http://sharpsightlabs.com/join-newsletter-rapid-mastery/" target="_blank"><br>
<img sizes="(max-width: 600px) 100vw, 600px" src="https://i2.wp.com/sharpsightlabs.com/wp-content/uploads/2017/08/european-cities_large-pop_map_noformat-600x517.png?w=450" alt="" class="wp-image-3110" data-recalc-dims="1"><br>
</a></p>
<p>Based on this plot, it looks like the data are essentially correct.</p>
<p>Now, we just want to “polish” the visualization by changing colors, fonts, sizes, etc.</p>
<pre class="wp-code-highlight">
#====================================================
# PLOT 'POLISHED' MAP
# - this version is formatted and cleaned up a little
#   just to make it look more aesthetically pleasing
#====================================================

#-------------
# CREATE THEME
#-------------
theme.maptheeme &lt;-
  theme(text = element_text(family = &quot;Gill Sans&quot;, color = &quot;#444444&quot;)) +
  theme(plot.title = element_text(size = 32)) +
  theme(plot.subtitle = element_text(size = 16)) +
  theme(panel.grid = element_blank()) +
  theme(axis.text = element_blank()) +
  theme(axis.ticks = element_blank()) +
  theme(axis.title = element_blank()) +
  theme(legend.background = element_blank()) +
  theme(legend.key = element_blank()) +
  theme(legend.title = element_text(size = 18)) +
  theme(legend.text = element_text(size = 10)) +
  theme(panel.background = element_rect(fill = &quot;#596673&quot;)) +
  theme(panel.grid = element_blank())


#------
# PLOT
#------
#fill = &quot;#AAAAAA&quot;,colour = &quot;#818181&quot;, size = .15)
ggplot() +
  geom_polygon(data = map.europe, aes(x = long, y = lat, group = group), fill = &quot;#DEDEDE&quot;,colour = &quot;#818181&quot;, size = .15) +
  geom_point(data = df.euro_cities, aes(x = lon, y = lat, size = population), color = &quot;red&quot;, alpha = .3) +
  geom_point(data = df.euro_cities, aes(x = lon, y = lat, size = population), color = &quot;red&quot;, shape = 1) +
  coord_cartesian(xlim = c(-9,45), ylim = c(32,70)) +
  labs(title = &quot;European Cities with Large Populations&quot;, subtitle = &quot;Cities with over 1MM population, within city limits&quot;) +
  scale_size_continuous(range = c(.7,15), breaks = c(1100000, 4000000, 8000000, 12000000), name = &quot;Population&quot;, labels = scales::comma_format()) +
  theme.maptheeme

</pre>
<p><a rel="nofollow" href="http://sharpsightlabs.com/join-newsletter-rapid-mastery/" target="_blank"><br>
<img sizes="(max-width: 600px) 100vw, 600px" src="https://i1.wp.com/sharpsightlabs.com/wp-content/uploads/2017/08/european-cities_large-pop_map-600x514.png?w=450" alt="" class="wp-image-3108" data-recalc-dims="1"><br>
</a></p>
<p>Not too bad.</p>
<p>Keep in mind that as a reader, you get to see the finished product: the finalized visualization and the finalized code.</p>
<p>But as always, the process for creating a visualization like this is <a href="https://www.r-bloggers.com/tidyverse-practice-mapping-large-european-cities/sharpsightlabs.com/blog/r-line-chart-and-datavis-process/">highly iterative</a>. If you work on a similar project, expect to change your code dozens of times. You’ll change your data-wrangling code as you work with the data and identify new items you need to change or fix. You’ll also change your  visualization code multiple times as you try different colors, fonts, and settings.</p>
<h1>If you master the basics, the hard things never seem hard</h1>
<p>Creating this visualization is actually not terribly hard to do, but if you’re somewhat new to R, it might seem rather challenging.</p>
<p>If you look at this, and it seems difficult then you need to understand: once you master the basics, the hard things <a href="https://www.amazon.com/C-Programming-Absolute-Beginners-Guide-ebook/dp/B00EANJ09U/ref=sr_1_5_twi_kin_2?ie=UTF8&qid=1504021618&sr=8-5&keywords=c+programming">never</a> seem hard.</p>
<p>What I mean by that, is that this visualization is nothing more than a careful application of a few dozen simple tools, arranged in a way to create something new.</p>
<p>Once you master individual tools from ggplot2, dplyr, and the rest of the Tidyverse, projects like this become very easy to execute.</p>
<h1>Sign up now, and discover how to rapidly master data science</h1>
<p>To rapidly master data science, you need to master the essential tools.</p>
<p>You need to know what tools are important, which tools are <em>not</em> important, and <em>how to practice</em>.</p>
<p>Sharp Sight is dedicated to teaching you how to master the tools of data science as quickly as possible.</p>
<p><a rel="nofollow" href="http://sharpsightlabs.com/join-newsletter-rapid-mastery/" target="_blank">Sign up now for our email list</a>, and you’ll receive regular tutorials and lessons. </p>
<p>You’ll learn:</p>
<ul>
<li>What data science tools you should learn (and what not to learn)</li>
<li>How to practice those tools</li>
<li>How to put those tools together to execute analyses and machine learning projects</li>
<li>… and more </li>
</ul>
<p>If you <a rel="nofollow" href="http://sharpsightlabs.com/join-newsletter-rapid-mastery/" target="_blank">sign up for our email list <em>right now</em></a>, you’ll also get access to our “Data Science Crash Course” <u>for free</u>.</p>
<p><a rel="nofollow" href="http://sharpsightlabs.com/join-newsletter-rapid-mastery/" target="_blank">
<p><strong>SIGN UP NOW</strong></p>
<p></p></a>
<p>The post <a rel="nofollow" href="http://sharpsightlabs.com/blog/map-european-cities-large-population/" target="_blank">Tidyverse practice: mapping large European cities</a> appeared first on <a rel="nofollow" href="http://sharpsightlabs.com/" target="_blank">SHARP SIGHT LABS</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://sharpsightlabs.com/blog/map-european-cities-large-population/"> r-bloggers – SHARP SIGHT LABS</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/BFDh73Ofo-I" height="1"></a></div>