<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/the-mathematics-genealogy-project-customizing-my-mathematical-family-tree/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.pieceofk.fr/?p=566"> R_EN – Piece of K</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Some time ago, Maëlle Salmon published a <a rel="nofollow" href="https://masalmon.eu/2018/06/18/mathtree/" target="_blank">very nice post</a> showing how she scraped her mathematical family tree from the <a rel="nofollow" href="https://www.genealogy.math.ndsu.nodak.edu/" target="_blank">Mathematics Genealogy Project</a>. Of course I immediately wanted to produce my own! I am not a mathematician myself, but one of my PhD supervisor has a PhD in mathematics. Which makes me the indirect descendant of a long lineage of famous mathematicians! As Maëlle kindly invited me to share my tree on Twitter, I decided to write this post to show how I customized the appearance of my tree to make it more sexy. [It is actually part of my job to plot sexy phylogenetic tree…]. So, this post is a remix of Maëlle’s post which was a remix of Nathalie Vialaneix’s <a rel="nofollow" href="http://tuxette.nathalievilla.org/?p=1292" target="_blank">post</a> <img alt="🙂" src="https://s.w.org/images/core/emoji/2.4/72x72/1f642.png" class="wp-smiley"></p>
<h1>1. Scraping more data</h1>
<p>First I modified a bit Maëlle’s scraping function to collect more data. I added Xpath selectors for the degree, the university, the country, the year and the title of the thesis. I am not going to use all these informations but now they are here if you need them.</p>
<pre>
library(magrittr)
library(DiagrammeR)
library(dplyr)
library(igraph)
library(purrr)
library(stringr)
library(rvest)
library(xml2)


.get_advisors &lt;- function(id_string = &quot;id.php?id=143630&quot;, sleep_time, terminal = FALSE){
  # small break to be nice
  Sys.sleep(sleep_time)
  
  # try to get the page
  page &lt;- glue::glue(&quot;https://www.genealogy.math.ndsu.nodak.edu/{id_string}&quot;) %>%
    httr::GET() 
  
  # try until it works but not more than 5 times
  try &lt;- 1
  while(httr::status_code(page) != 200 &amp; try &lt;= 5){
    Sys.sleep(sleep_time)
    page &lt;- glue::glue(&quot;https://www.genealogy.math.ndsu.nodak.edu/{id_string}&quot;) %>%
      httr::GET() 
    try = try + 1
  }
  
  # Now get student's data
  student_name &lt;- httr::content(page) %>%
    rvest::xml_nodes(xpath = '//h2[@style=&quot;text-align: center; margin-bottom: 0.5ex; margin-top: 1ex&quot;]') %>%
    rvest::html_text() %>%
    stringr::str_remove(&quot;\\\n&quot;)
  
  degree &lt;- httr::content(page) %>%
    rvest::xml_nodes(xpath = '//span[@style=&quot;margin-right: 0.5em&quot;]/span/preceding-sibling::text()') %>%
    rvest::html_text()
  
  university &lt;- httr::content(page) %>%
    rvest::xml_nodes(xpath = '//span[@style=&quot;margin-right: 0.5em&quot;]/span') %>%
    rvest::html_text()
  
  year &lt;- httr::content(page) %>%
    rvest::xml_nodes(xpath = '//span[@style=&quot;margin-right: 0.5em&quot;]/span/following-sibling::text()') %>%
    rvest::html_text() %>% 
    stringr::str_trim()
  
  country &lt;- httr::content(page) %>%
    rvest::xml_nodes(xpath = '//div[@style=&quot;line-height: 30px; text-align: center; margin-bottom: 1ex&quot;]/img') %>%
    rvest::html_attr(&quot;title&quot;)
  
  thesis_title &lt;- httr::content(page) %>%
    rvest::xml_nodes(xpath = '//span[@id=&quot;thesisTitle&quot;]') %>%
    rvest::html_text() %>%
    stringr::str_remove_all(&quot;\\\n&quot;)
  
  # Get all nodes corresponding to advisors
  # Thanks to their... formatting but it works
  all_advisors &lt;- httr::content(page) %>%
    rvest::xml_nodes(xpath = &quot;//p[@style='text-align: center; line-height: 2.75ex']&quot;) %>%
    rvest::html_nodes(&quot;a&quot;)
  
  if(terminal){
    name &lt;- NA
    id_string_advisors &lt;- NA
  } else {
    name &lt;- purrr::map_chr(all_advisors, rvest::html_text)
    id_string_advisors &lt;- purrr::map_chr(all_advisors, rvest::html_attr,
                                         &quot;href&quot;)
  }
  
  # Export results
  tibble::tibble(student_name = student_name,
                 degree = list(degree),
                 university = list(university),
                 year = list(year),
                 country = list(country),
                 thesis_title = list(thesis_title),
                 id_string_student = id_string,
                 name = name,
                 id_string = id_string_advisors)
}

get_advisors &lt;- memoise::memoise(.get_advisors)



#### Download and prepare data ####
# initial data.frame
df &lt;- get_advisors(me, 5)
new_df &lt;- df
keep_growing &lt;- TRUE

while(keep_growing){
  # get size to compare to size after a bit more scraping
  nrow1 &lt;- nrow(df)
  # get advisors for all new lines 
  # from the previous iterations
  new_df &lt;- purrr::map_df(new_df$id_string, get_advisors, sleep_time = 30)
  df &lt;- unique(rbind(df, new_df))
  # if the data.frame didn't grow, stop
  if(nrow(df) == nrow1){
    keep_growing &lt;- FALSE
  }
}


df &lt;- df %>% mutate(student_name = stringr::str_trim(student_name),
                    name = stringr::str_trim(name))


terminal_df &lt;- setdiff(df$id_string, df$id_string_student) %>% 
  map(get_advisors, terminal = TRUE, sleep_time = 30) %>% 
  bind_rows() %>%
  mutate(student_name = stringr::str_trim(student_name),
         name = stringr::str_trim(name))


</pre>
<h1>2. Labels</h1>
<p>I wanted to create labels with the name of the mathematician, the flag of the country and the year. Displaying a picture in a node with graphviz (using DiagrammeR) was not simple. Apparently it is possible to use some kind of basic HTML to format nodes but I failed to include images. Finally I decided to use emojis <img alt="🙂" src="https://s.w.org/images/core/emoji/2.4/72x72/1f642.png" class="wp-smiley"> Thanks to Hadley Wickham’s package <a rel="nofollow" href="https://github.com/hadley/emo" target="_blank">emo</a> it was fairly easy.</p>
<pre>
#### Construct the graph ####
# create nodes
labels &lt;-  unique(c(df$student_name, df$name))
nodes_df &lt;- create_node_df(n = length(labels))
nodes_df$label &lt;- labels

df_red &lt;- df %>%
  bind_rows(terminal_df) %>%
  filter(!duplicated(student_name)) %>% 
  right_join(nodes_df, by = c(&quot;student_name&quot; = &quot;label&quot;))


# create edges
edges_df &lt;- df[, c(&quot;name&quot;, &quot;student_name&quot;)]
edges_df &lt;- dplyr::left_join(edges_df, nodes_df,
                             by = c(&quot;name&quot; = &quot;label&quot;))
edges_df &lt;- dplyr::rename(edges_df, from = id)
edges_df &lt;- dplyr::left_join(edges_df, nodes_df,
                             by = c(&quot;student_name&quot; = &quot;label&quot;))
edges_df &lt;- dplyr::rename(edges_df, to = id)

edges_df &lt;- mutate(edges_df, rel = &quot;a&quot;, color = &quot;BurlyWood4&quot;)


# Create labels
years &lt;- map_chr(df_red$year, paste, collapse = &quot; &quot;) %>% 
  map_chr(str_trim) %>%
  str_replace_all(&quot;(?&lt;=[0-9]{4}) (?=[0-9]{4})&quot;, &quot;, &quot;) %>% 
  str_replace_all(&quot; /&quot;, &quot;, &quot;) %>% 
  paste0(&quot;(&quot;, ., &quot;)&quot;) %>% 
  str_replace_all(&quot;\\(\\)&quot;, &quot;&quot;)


df_red$country &lt;-
  map_if(df_red$country,
         (map_lgl(df_red$country,~ is.null(.x)) | map_lgl(df_red$country,~ length(.x) == 0)),
         function(x) &quot;&quot;) %>% 
  map(stringr::str_replace, &quot;UnitedKingdom&quot;, &quot;United Kingdom&quot;)

country_flag &lt;- map_if(df_red$country,
                       map_lgl(df_red$country,~ any(!.x == &quot;&quot;)),
                       function(x) map_chr(x, ~ as.character(emo::flag(paste0(&quot;^&quot;, .x, &quot;$&quot;))))) %>% 
  map_chr(paste, collapse = &quot; ⋅ &quot;)


label &lt;- paste0(nodes_df$label, &quot;\n&quot;, years, &quot;&quot;)
label &lt;- paste0(nodes_df$label, &quot;\n&quot;, country_flag, &quot;\n&quot;, years, &quot;&quot;)

nodes_df$label &lt;- label %>%
  str_replace_all(&quot;'&quot;, &quot; &quot;) %>%
  str_replace_all(&quot;[[:space:]]{2,}&quot;, &quot; &quot;)
</pre>
<h1>3. Adding my self</h1>
<pre>
nodes_df &lt;- bind_rows(nodes_df,
              tibble(id = nrow(nodes_df) + 1,
                type = NA, label = &quot;François Keck\n<img alt="🇫🇷" src="https://i0.wp.com/s.w.org/images/core/emoji/2.4/72x72/1f1eb-1f1f7.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1">\n(2016)&quot;))
          
edges_df &lt;- bind_rows(edges_df,
              tibble(name = &quot;AF&quot;, student_name = &quot;FK&quot;,
                 from = 1, type.x = NA,
                 to = nrow(nodes_df), type.y = NA,
                 rel = &quot;a&quot;, color = &quot;BurlyWood4&quot;))
</pre>
<h1>4. Customizing the style of the nodes</h1>
<p>I changed the color and shape of the nodes. For some obscure reasons the rectangle-based shapes of graphiz are not correctly rendered in Firefox on Ubuntu (labels are overlapping). It worked on Windows but then I couldn’t display my flags with colored emojis <img alt="🙁" src="https://s.w.org/images/core/emoji/2.4/72x72/1f641.png" class="wp-smiley"> The only solution I found was to fix the font size manually (see next point).</p>
<pre>
# Customizing the nodes
nodes_df &lt;- mutate(nodes_df,
                   shape = &quot;note&quot;,
                   color = &quot;Tan&quot;,
                   fillcolor = &quot;Moccasin&quot;,
                   style = &quot;filled&quot;)


# create a DiagrammeR dgr_graph object
dgr &lt;- create_graph(nodes_df = nodes_df,
                    edges_df = edges_df[, c(-1, -2)],
                    directed = TRUE)

# export the object to igraph format
# and then write it to a GraphViz DOT file
DiagrammeR::to_igraph(dgr) %>%
  igraph::write.graph(file = &quot;graph.dot&quot;,
                      format = &quot;dot&quot;) 
DiagrammeR::grViz(&quot;graph.dot&quot;, width = 4000, height = 5000)
</pre>
<p><a rel="nofollow" href="http://www.pieceofk.fr/?attachment_id=585" target="_blank"><img sizes="(max-width: 700px) 100vw, 700px" src="https://i0.wp.com/www.pieceofk.fr/wp-content/uploads/2018/07/Screenshot_2018-07-10-grViz.png?w=450" alt="" class="wp-image-585" data-recalc-dims="1"></a></p>
<h1>5. The final touch</h1>
<p>The viz.js library which stands behind DiagrammeR renders DOT objects in the browser via SVG. SVG files are XML-based and therefore can be directly processed with R. This gives us great power to manipulate the look of our tree.</p>
<p>First, we need to convert the widget to static HTML/SVG. We can do that in command line using Chromium in headerless mode to render the widget page.</p>
<pre>
DiagrammeR::grViz(&quot;graph.dot&quot;, width = 4000, height = 5000) %>% 
  htmlwidgets::saveWidget(&quot;index.html&quot;)
system(&quot;chromium-browser --headless --dump-dom index.html > genealogy.html&quot;)
</pre>
<p>Finally, I used R and <a rel="nofollow" href="https://cran.r-project.org/web/packages/xml2/index.html" target="_blank">xml2</a> to edit directly the SVG content and improve the look of the tree. In the code below I show how to fill the page and labels backgrounds with a texture image, how to fix the text size, and how to add a shadow effect on labels.</p>
<pre>

# Load and clean html
html &lt;- read_html(&quot;index.html&quot;) 

xml_find_all(html, '//script') %>% 
  xml_remove()

# Background
xml_find_all(html, '/html/body') %>% 
  xml_set_attr('style', 'background-image: url(&quot;ricepaper2.png&quot;); margin: 0px; padding: 40px;')

xml_find_all(html, '/html/body/div/div/svg/g/polygon') %>% 
  xml_set_attr('fill', 'transparent')


# Labels text size
xml_find_all(html, '//text') %>% 
  xml_text() %>% 
  str_detect(&quot;(^[A-Z])|(^\\()&quot;) %>%
  extract(xml_find_all(html, '//text'), .) %>% 
  xml_set_attr('font-size', '12')


# Labels background
xnodes &lt;- xml_find_all(html, '//polygon[@fill=&quot;#ffe4b5&quot;]')
xml_set_attr(xnodes, 'id', 'paper_tag')
xml_set_attr(xnodes, 'fill', 'url(#img1)')

xml_find_all(html, '/html/body/div/div/svg') %>%
  xml_add_child(read_xml('
  '), .where = 0)


# Labels shadow fx
xml_find_all(html, '//polygon[@id=&quot;paper_tag&quot;]') %>% 
  xml_set_attr('filter', 'url(#f3)')

xnodes &lt;- xml_find_all(html, '//*[@stroke=&quot;#d2b48c&quot;]')
  xml_set_attr(xnodes, 'stroke', '#000000')
  xml_set_attr(xnodes, 'stroke-width', '0.5')

xml_find_all(html, '/html/body/div/div/svg') %>%
  xml_add_child(read_xml('
  '), .where = 0)

write_html(html, &quot;genealogy.html&quot;)
</pre>
<p>I find the result pretty nice <img alt="🙂" src="https://s.w.org/images/core/emoji/2.4/72x72/1f642.png" class="wp-smiley"></p>
<p><a rel="nofollow" href="http://www.pieceofk.fr/?attachment_id=577" target="_blank"><img sizes="(max-width: 700px) 100vw, 700px" src="https://i0.wp.com/www.pieceofk.fr/wp-content/uploads/2018/07/Screenshot_2018-07-08-grViz.png?w=450" alt="" class="wp-image-577" data-recalc-dims="1"></a></p>
<p>And this is my mathematical family tree! I recognize some illustrious names here! Do you?</p>
<figure class="wp-caption"><a rel="nofollow" href="https://i1.wp.com/www.pieceofk.fr/wp-content/uploads/2018/07/Screenshot_2018-07-08-grViz.jpg" target="_blank"><img src="https://i0.wp.com/www.pieceofk.fr/wp-content/uploads/2018/07/Screenshot_2018-07-08-grViz-150x150.jpg?resize=150%2C150" alt="" width="150" class="wp-image-574" data-recalc-dims="1" height="150"></a><figcaption class="wp-caption-text">My tree</figcaption></figure>
<h3>Bonus:</h3>
<p>For a better parchment look, we can use the calligraphic font Tangerine for the labels. Note that some glyphs are unfortunately not supported by this font.</p>
<pre>  
# Labels font. 
xml_find_all(html, '//text') %>% 
  xml_text() %>% 
  str_detect(&quot;(^[A-Z])|(^\\()&quot;) %>%
  extract(xml_find_all(html, '//text'), .) %>% 
  xml_set_attr('font-size', '20')

xml_find_all(html, '//text') %>% 
  xml_text() %>% 
  str_detect(&quot;(^[A-Z])|(^\\()&quot;) %>%
  extract(xml_find_all(html, '//text'), .) %>% 
  xml_set_attr('font-family', 'Tangerine')

xml_find_all(html, '/html/head') %>%
  xml_add_child(read_xml(''))

</pre>
<p><a rel="nofollow" href="http://www.pieceofk.fr/?attachment_id=576" target="_blank"><img sizes="(max-width: 700px) 100vw, 700px" src="https://i2.wp.com/www.pieceofk.fr/wp-content/uploads/2018/07/Screenshot_2018-07-08-grViz1.png?w=450" alt="" class="wp-image-576" data-recalc-dims="1"></a></p>
<p><a rel="nofollow" data-provider="facebook" href="http://www.facebook.com/sharer.php?u=http%3A%2F%2Fwww.pieceofk.fr%2F%3Fp%3D566&t=The%20Mathematics%20Genealogy%20Project%3A%20Customizing%20my%20mathematical%20family%20tree&%23038;s=100&%23038;p&%23091;url&%23093;=http%3A%2F%2Fwww.pieceofk.fr%2F%3Fp%3D566&%23038;p&%23091;images&%23093;&%23091;0&%23093;=http%3A%2F%2Fwww.pieceofk.fr%2Fwp-content%2Fuploads%2F2018%2F07%2FScreenshot_2018-07-10-grViz.png&%23038;p&%23091;title&%23093;=The%20Mathematics%20Genealogy%20Project%3A%20Customizing%20my%20mathematical%20family%20tree" title="Share on Facebook" target="_blank"><img src="https://i0.wp.com/www.pieceofk.fr/wp-content/plugins/social-media-feather/synved-social/image/social/regular/32x32/facebook.png?resize=16%2C16" alt="Facebook" width="16" title="Share on Facebook" data-recalc-dims="1" height="16"></a><a rel="nofollow" data-provider="twitter" href="https://twitter.com/intent/tweet?url=http%3A%2F%2Fwww.pieceofk.fr%2F%3Fp%3D566&text=Hey%20check%20this%20out" title="Share on Twitter" target="_blank"><img src="https://i0.wp.com/www.pieceofk.fr/wp-content/plugins/social-media-feather/synved-social/image/social/regular/32x32/twitter.png?resize=16%2C16" alt="twitter" width="16" title="Share on Twitter" data-recalc-dims="1" height="16"></a></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.pieceofk.fr/?p=566"> R_EN – Piece of K</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/ibbqlWenWzg" height="1"></a></div>