If someone told me a decade ago (back before I'd ever heard the term &quot;roguelike&quot;) what I'd be doing today, I would have trouble believing this...Yet here we are. pic.twitter.com/N6Hh6A4tWl— Josh Ge (@GridSageGames) June 21, 2018
Abstract
In this post, I am going to show you how you can scrape tables from a website, and then create a package
with the tidied data to share with the world. The data I am going to scrape comes from a NetHack
public server (link). The data I discuss in this blog post is
available in the {nethack} package I created and I will walk you through the process of releasing
your package on CRAN. However, {nethack} is too large to be on CRAN (75 mb, while the maximum
allowed is 5mb), so you can install it to play around with the data from github:
devtools::install_github(&quot;b-rodrigues/nethack&quot;)
And to use it:
library(nethack)
data(&quot;nethack&quot;)
The data contains information on games played from 2001 to 2018; 322485 rows and 14 columns. I
will analyze the data in a future blog post. This post focuses on getting and then sharing the
data. By the way, all the content from the public server I scrape is under the CC BY 4.0 license.
I built the package by using the very useful {devtools} package.
Introduction
NetHack is a game released in 1987 that is still being played and developed today.
NetHack is a roguelike game, meaning that is has procedurally generated dungeons and permadeath.
If you die, you have to start over, and because the dungeons are procedurally generated, this means
that you cannot learn the layout of the dungeons you explore or know when ennemies are going
to attack or even what ennemies are going to attack. Ennemies are not the only thing that you have
to be careful about; you can die from a lot of different events, as you will see in this post.
Objects that you find, such as a silver ring, might be helpful in a run, but be cursed in the next run.
The latest version of the game, 3.6.1, was released on April 27th 2018, and this is how it looks like:
The graphics are… bare-bones to say the least. The game runs inside a terminal emulator and is
available for any platform. The goal of NetHack is to explore a dungeon and go down every
level until you find the Amulet of Yendor. Once you find this Amulet, you have to go all the way
back upstairs, enter and fight your way through the Elemental Planes, enter the final Astral Plane,
and then finally offer the Amulet of Yendor to your god to finish the game. Needless to say,
NetHack is very difficult and players can go years without ever finishing the game.
When you start an new game, you have to create a character, which can have several attributes.
You have to choose a race (human, elf, orc, etc), a role (tourist, samurai, mage, etc) and an
alignment (neutral, law, chaos) and these choices impact your base stats.
If you can’t get past the ASCII graphics, you might want to try
Pathos, which is NetHack with touch screen support
(on smartphones) and tile graphics:
You can install NetHack on your computer or you can play online on a pulbic server, such as
this one. There are several advantages when playing on a pubic server;
the player does not have to install anyhing, and we data enthusiasts have access to a mine of
information! For example, you can view the following table
which contains data on all the games played on October 25th 2018. These tables start in the year 2001,
and I am going to scrape the info from these tables, which will allow me to answer several questions.
For instance, what is the floor most players die on? What kills most players?
What role do players choose more often? I will explore this questions in a future blog post, but for
now I will focus on scraping the data and realeasing it as a package to CRAN.
Scraping the data
To scrape the data I wrote a big function that does several things:
library(&quot;tidyverse&quot;)
library(&quot;rvest&quot;)
scrape_one_day 60) %__%
mutate(dumplog = map(dumplog_links, ~possibly(read_lines_slow, otherwise = NA)(.))) %__%
mutate(time_in_seconds = ifelse(time_in_seconds == 61, NA, time_in_seconds))
saveRDS(clean_final, paste0(&quot;datasets/data_&quot;, date, &quot;.rds&quot;))
}
Let’s go through each part. The first part is a function that converts strings like “02:21:76” to
seconds:
convert_to_seconds %
as.numeric
time_in_seconds and then the links I am interested in:
knitr::include_graphics(&quot;/img/selectorgadget2.png&quot;)
Putting them together, I get the right “xpath”. But just as with the time of the run, dumplogs are
only available after a certain date. So in case the dumplogs column is empty, I relpace it with NA.
if(is_empty(dumplogs)){
print(&quot;dumplogs empty&quot;)
dumplogs %
mutate(dumplog_links = dumplogs)
print(paste0(&quot;cleaning data of date &quot;, date))
I scrape the table, and then join the dumplog links to the table inside a new column called “dumplog_links”.
Because what follows is a long process, I print a message to let me know the progress of the scraping.
Now the last part:
clean_final %
select(-x) %__%
rename(role = x_2,
race = x_3,
gender = x_4,
alignment = x_5) %__%
mutate(time_in_seconds = map(time, convert_to_seconds)) %__%
filter(!(death %in% c(&quot;quit&quot;, &quot;escaped&quot;)), time_in_seconds __ 60) %__%
mutate(dumplog = map(dumplog_links, ~possibly(read_lines_slow, otherwise = NA)(.))) %__%
mutate(time_in_seconds = ifelse(time_in_seconds == 61, NA, time_in_seconds))
I first remove and remane columns. Then I convert the “time” column into seconds and also remove
runs that lasted less than 60 seconds or that ended either in “quit” (the player left the game)
or “escaped” (the player left the dungeon and the game ended immediately). There are a lot of runs
like that and they’re not interesting. Finally, and this is what takes long, I create a new
list-column where each element is the contents of the dumplog for that run. I wrap read_lines_slow()
around purrr::possibly() because dumplogs are missing for certains runs and when I try to read
them I get an 404 error back. Getting such an error stops the whole process, so with purrr::possibly()
I can specify that in that case I want NA back. Basically, a function wrapped inside purrr::possibly()
never fails! Finally, if a game lasts for 61 seconds, I convert it back to NA (remember this was
used to avoid having problems with the filter() function).
Finally, I export what I scraped to disk:
saveRDS(clean_final, paste0(&quot;datasets/data_&quot;, date, &quot;.rds&quot;))
This is where I use the date; to name the data. This is really important because scraping takes
a very long time, so if I don’t write the progress to disk as it goes, I might lose hours of work
if my internet goes down, or if computer freezes or whatever.
In the lines below I build the links that I am going to scrape. They’re all of the form:
https://alt.org/nethack/gamesday.php?date=YYYYMMDD so it’s quite easy to create a list of
dates to scrape, for example, for the year 2017:
link Then name your package, create a git repository and then click on “Create Project”:
RStudio wil open the hello.R script which you can now modify. You got to learn from the best, so
I suggest that you modify hello.R by taking inspiration from the babynames package made by Hadley
Wickham which you can find here.
You do not need the first two lines, and can focus on lines 4 to 13. Then, rename the script to
data.R. This is how {nethack}'s looks like:
#' NetHack runs data.
#'
#' Data on NetHack runs scraped from https://alt.org/nethack/gamesday.php
#'
#' @format A data frame with 14 variables: \code{rank}, \code{score},
#' \code{name}, \code{time}, \code{turns}, \code{lev_max}, \code{hp_max}, \code{role}, \code{race},
#' \code{gender}, \code{alignment}, \code{death}, \code{date} and \code{dumplog}
#' \describe{
#' \item{rank}{The rank of the player on that day}
#' \item{score}{The score the player achieved on that run}
#' \item{name}{The name of the player}
#' \item{time}{The time the player took to finish the game}
#' \item{turns}{The number of turns the player played before finishing the game}
#' \item{lev_max}{The maximum character level the player achieved}
#' \item{hp_max}{The maximum character health points the player achieved}
#' \item{role}{The role the player chose to play as}
#' \item{race}{The race the player chose to play as}
#' \item{gender}{The gender the playr chose to play as}
#' \item{alignement}{The alignement the playr chose to play as}
#' \item{death}{The reason of death of the character}
#' \item{date}{The date the game took place}
#' \item{dumplog}{The log of the end game; this is a list column}
#' }
&quot;nethack&quot;
The comments are special, the “#” is followed by a '; these are special comments that will be
parsed by roxygen2::roxygenise() and converted to documentation files.
Next is the DESCRIPTION file. Here is how {nethack}’s looks like:
Package: nethack
Type: Package
Title: Data from the Video Game NetHack
Version: 0.1.0
Authors@R: person(&quot;Bruno André&quot;, &quot;Rodrigues Coelho&quot;, email = &quot;bruno@brodrigues.co&quot;,
role = c(&quot;aut&quot;, &quot;cre&quot;))
Description: Data from NetHack runs played between 2001 to 2018 on , a NetHack public server.
Depends: R (__= 2.10)
License: CC BY 4.0
Encoding: UTF-8
LazyData: true
RoxygenNote: 6.1.0
Adapt yours accordingly. I chose the license CC BY 4.0 because this was the licence under which
the original data was published. It is also a good idea to add a Vignette:
devtools::use_vignette(&quot;the_nethack_package&quot;)
Vignettes are very useful documentation with more details and examples.
It is also good practice to add the script that was used to scrape the data. Such scripts go into
data-raw/. Create this folder with:
devtools::use_data_raw()
This creates the data-raw/ folder where I save the script that scrapes the data. Now is time to
put the data in the package. Start by importing the data:
nethack