Link to webscraping the data
Link to Analysis, part 1
Introduction
This is the third blog post that deals with data from the game NetHack, and oh boy, did a lot of
things happen since the last blog post! Here’s a short timeline of the events:
I scraped data from alt.org/nethack and made a package with the data available on Github
(that package was too big for CRAN)
Then, I analyzed the data, focusing on what monsters kill the players the most, and also where
players die the most
@GridSageGames, developer of the roguelike Cogmind and moderator of the roguelike subreddit,
posted the blog post on reddit
I noticed that actually, by scraping the data like I did, I only got a sample of 100 daily games
This point was also discussed on Reddit, and bhhak, an UnNetHack developer (UnNetHack is a fork
of NetHack) suggested I used the xlogfiles instead
xlogfiles are log files generated by NetHack, and are also available on alt.org/nethack
I started scraping them, and getting a lot more data
I got contacted on twitter by @paxed, an admin of alt.org/nethack:
There was no need for scraping, we'll happily give out the whole database if you want.— Pasi Kallinen (@paxed) November 5, 2018
He gave me access to ALL THE DATA on alt.org/nethack!
The admins of alt.org/nethack will release all the data to the public!
So, I will now continue with the blog post I wanted to do in the first place; focusing now on what
roles players choose to play the most, and also which monsters they kill the most. BUT! Since all the data
will be released to the public, my {nethack} package that contains data that I scraped is not
that useful anymore. So I changed the nature of the package.
Now the package contains some functions: a function to parse and prepare the xlogfiles from NetHack that
you can download from alt.org/nethack (or from any other public server), a function
to download dumplogs such as this one. These dumplogs contain a lot of
info that I will extract in this blog post, using another function included in the {nethack} package.
The package also contains a sample of 6000 runs from NetHack version 3.6.1.
You can install the package with the following command line:
devtools::install_github(&quot;b-rodrigues/nethack&quot;)
The {nethack} package
In part 1 I showed what killed
players the most. Here, I will focus on what monsters players kill the most.
Let’s start by loading some packages:
library(tidyverse)
library(lubridate)
library(magrittr)
library(ggridges)
library(brotools)
library(rvest)
library(nethack)
Let’s first describe the data:
brotools::describe(nethack) %__% print(n = Inf)
## # A tibble: 23 x 17
## variable type nobs mean sd mode min max q05
## ## 1 deathdn… Nume… 6.00e3 8.45e-1 1.30e+0 2 0. 7.00e0 0. ## 2 deathlev Nume… 6.00e3 4.32e+0 3.69e+0 10 -5.00e0 4.50e1 1.00e0
## 3 deaths Nume… 6.00e3 8.88e-1 3.54e-1 1 0. 5.00e0 0. ## 4 endtime Nume… 6.00e3 1.53e+9 4.72e+6 1534… 1.52e9 1.54e9 1.53e9
## 5 hp Nume… 6.00e3 6.64e+0 4.96e+1 -1 -9.40e1 1.79e3 -8.00e0
## 6 maxhp Nume… 6.00e3 3.82e+1 5.29e+1 57 2.00e0 1.80e3 1.10e1
## 7 maxlvl Nume… 6.00e3 5.52e+0 6.36e+0 10 1.00e0 5.30e1 1.00e0
## 8 points Nume… 6.00e3 4.69e+4 4.18e+5 10523 0. 9.92e6 1.40e1
## 9 realtime Nume… 6.00e3 4.42e+3 1.60e+4 4575 0. 3.23e5 6.90e1
## 10 startti… Nume… 6.00e3 1.53e+9 4.72e+6 1534… 1.52e9 1.54e9 1.53e9
## 11 turns Nume… 6.00e3 3.60e+3 9.12e+3 6797 3.10e1 1.97e5 9.49e1
## 12 align Char… 6.00e3 NA NA Cha NA NA NA ## 13 align0 Char… 6.00e3 NA NA Cha NA NA NA ## 14 death Char… 6.00e3 NA NA kill… NA NA NA ## 15 gender Char… 6.00e3 NA NA Fem NA NA NA ## 16 gender0 Char… 6.00e3 NA NA Fem NA NA NA ## 17 killed_… Char… 6.00e3 NA NA fain… NA NA NA ## 18 name Char… 6.00e3 NA NA drud… NA NA NA ## 19 race Char… 6.00e3 NA NA Elf NA NA NA ## 20 role Char… 6.00e3 NA NA Wiz NA NA NA ## 21 dumplog List 1.33e6 NA NA NA NA NA ## 22 birthda… Date 6.00e3 NA NA NA NA NA ## 23 deathda… Date 6.00e3 NA NA NA NA NA ## # ... with 8 more variables: q25 , median , q75 ,
## # q95 , n_missing , n_unique , starting_date ,
## # ending_date All these columns are included in xlogfiles. The data was prepared using two functions, included
in {nethack}:
xlog %
ggplot(aes(x = turns, y = role, group = role, fill = role)) +
geom_density_ridges(scale = 6, size = 0.25, rel_min_height = 0.01) + theme_blog() + scale_fill_blog() + theme(axis.text.y = element_blank(),
axis.title.y = element_blank())
## Picking joint bandwidth of 486
I use the very cool {ggridges} package for that. The distribution seems to mostly be the same
(of course, one should do a statistical test to be sure), but the one for the role “Valkyrie”
seems to be quite different from the others. It is known that it is easier to win the game playing
as a Valkyrie, but a question remains: is it really easier as a Valkyrie, or do good players tend
to play as Valkyries more often?
Creatures vanquished, genocided or extinct
The dumplog lists which, and how many of which, creatures were vanquished during the run, as well
as creatures that were genocided and extinct. The player can genocide an entire species by reading
a scroll of genocide (or by sitting on a throne). A species gets extinct if the player manages to
kill every monster from that species (there’s other ways too, but for the sake of simplicity, let’s
just say that when the players kills every monster from a species, the species is extinct).
The following lines are an extract of a dumplog:
&quot;Vanquished creatures:&quot;
&quot; Baalzebub&quot;
&quot; Orcus&quot;
&quot; Juiblex&quot;
&quot;the Wizard of Yendor (4 times)&quot;
&quot; Pestilence (thrice)&quot;
&quot; Famine&quot;
&quot; Vlad the Impaler&quot;
&quot; 4 arch-liches&quot;
&quot; an arch-lich&quot;
&quot; a high priest&quot;
&quot;...&quot;
&quot;...&quot;
&quot;...&quot;
&quot;2873 creatures vanquished.&quot; If I want to analyze this, I have to first solve some problems:
Replace “a” and “an” by “1”
Put the digit in the string “(4 times)” in front of the name of the monster (going from “the Wizard of Yendor (4 times)” to “4 the Wizard of Yendor”)
Do something similar for “twice” and “thrice”
Put everything into singular (for example, arch-liches into arch-lich)
Trim whitespace
Extract the genocided or extinct status from the dumplog too
Finally, return a data frame with all the needed info
I wrote a function called extracted_defeated_monsters() and included it in the {nethack} package.
I discuss this function in appendix, but what it does is extracting information from dumplog files
about vanquished, genocided or extinct monsters and returns a tidy dataframe with that info. This
function has a lot of things going on inside it, so if you’re interested in learning more about
regular expression and other {tidyverse} tricks, I really encourage you to read its source code.
I can now easily add this info to my data:
nethack %%
mutate(monsters_destroyed = map(dumplog, ~possibly(extract_defeated_monsters, otherwise = NA)(.)))
Let’s take a look at one of them:
nethack$monsters_destroyed[[117]]
## # A tibble: 285 x 3
## value monster status
## ## 1 1 baalzebub ## 2 1 orcu ## 3 1 juiblex ## 4 4 the wizard of yendor ## 5 3 pestilence ## 6 1 famine ## 7 1 vlad the impaler ## 8 4 arch-lich ## 9 1 high priest ## 10 1 medusa ## # ... with 275 more rows
nethack$monsters_destroyed[[117]] %__% count(status)
## # A tibble: 3 x 2
## status n
## ## 1 extinct 2
## 2 genocided 7
## 3 276
The status variable tells us if that monster was genocided or extinct during that run. status
equal to “NA” means vanquished.
It is now possible to look at, say, the top 15 vanquished monsters (normalized):
nethack %__%
filter(!is.na(monsters_destroyed)) %__%
pull(monsters_destroyed) %__%
bind_rows %__%
group_by(monster) %__%
summarise(total = sum(value)) %__%
top_n(15) %__%
ungroup() %__%
mutate(norm_total = (total - min(total))/(max(total) - min(total))) %__%
mutate(monster = fct_reorder(monster, norm_total, .desc = FALSE)) %__%
ggplot() + geom_col(aes(y = norm_total, x = monster)) + coord_flip() + theme_blog() + scale_fill_blog() + ylab(&quot;Ranking&quot;) +
xlab(&quot;Monster&quot;)
## Selecting by total
In this type of graph, the most vanquished monster, “gnome” has a value of 1, and the least
vanquished one, 0. This normalization step is also used in the pre-processing step of machine learning
algorithms. This helps convergence of the gradient descent algorithm for instance.
Monsters can also get genocided or extinct. Let’s make a pie chart of the proportion of genocided
and extinct monsters (I lump monsters that are genocided or extinct less than 5% of the times
into a category called other). Because I want two pie charts, I nest the data after having grouped
it by the status variable. This is a trick I discussed in this blog post
and that I use very often:
nethack %__%
filter(!is.na(monsters_destroyed)) %__%
pull(monsters_destroyed) %__%
bind_rows %__%
filter(!is.na(status)) %__%
group_by(status) %__% count(monster) %__% mutate(monster = fct_lump(monster, prop = 0.05, w = n)) %__% group_by(status, monster) %__% summarise(total_count = sum(n)) %__%
mutate(freq = total_count/sum(total_count),
labels = scales::percent(freq)) %__%
arrange(desc(freq)) %__%
group_by(status) %__%
nest() %__%
mutate(pie_chart = map2(.x = status,
.y = data,
~ggplot(data = .y,
aes(x = &quot;&quot;, y = freq, fill = (monster))) + geom_col() + ggrepel::geom_label_repel(aes(label = labels), position = position_stack(vjust = 0.25), show.legend = FALSE) + coord_polar(&quot;y&quot;) + theme_blog() + scale_fill_blog() + ggtitle(.x) +
theme(legend.title = element_blank(),
panel.grid = element_blank(),
axis.text = element_blank(),
axis.title = element_blank())
)) %__%
pull(pie_chart)
## Warning in mutate_impl(.data, dots): Unequal factor levels: coercing to
## character
## Warning in mutate_impl(.data, dots): binding character and factor vector,
## coercing into character vector
## Warning in mutate_impl(.data, dots): binding character and factor vector,
## coercing into character vector
## [[1]]
## ## [[2]]
That was it for this one, the graphs are not that super sexy, but the amount of work that went
into making them was quite consequent. The main reason was that parsing xlogfiles was a bit tricky, but
the main challenge was extracting information from dumplog files. This proved to be a bit more
complicated than expected (just take a look at the source code of extract_defeated_monsters()
to get an idea…).
If you found this blog post useful, you might want to follow me on twitter for blog post updates.
Bonus plot
Correct number of daily games
The daily number of games are available here. Let’s
extract this info and remake the plot that shows the number of runs per day:
games %
html_nodes(xpath = '//table') %__%
html_table(fill = TRUE) This extracts all the tables and puts them into a list. Let’s take a look at one:
head(games[[1]])
## 2018 2018 2018 2018 2018 2018 2018 2018 2018 2018 2018 2018 2018 2018
## 1 NA 1 2 3 4 5 6 7 8 9 10 11 12
## 2 Jan 11639 275 370 394 363 392 276 288 324 297 411 413 430
## 3 Feb 10819 375 384 359 376 440 345 498 457 416 376 421 416
## 4 Mar 12148 411 403 421 392 447 391 451 298 350 309 309 369
## 5 Apr 13957 456 513 482 516 475 490 397 431 436 438 541 493
## 6 May 13361 595 509 576 620 420 443 407 539 440 446 404 282
## 2018 2018 2018 2018 2018 2018 2018 2018 2018 2018 2018 2018 2018 2018
## 1 13 14 15 16 17 18 19 20 21 22 23 24 25 26
## 2 331 341 318 483 408 424 464 412 371 430 348 315 359 375
## 3 385 367 443 324 283 341 385 398 361 379 399 276 455 460
## 4 390 358 362 345 388 360 411 382 371 400 410 417 328 431
## 5 593 537 396 578 403 435 526 448 339 377 476 492 528 393
## 6 265 358 419 564 483 429 423 299 424 404 450 408 355 409
## 2018 2018 2018 2018 2018
## 1 27 28 29 30 31
## 2 432 371 385 440 399
## 3 353 347 NA NA NA
## 4 386 484 493 486 395
## 5 407 421 463 477 NA
## 6 417 433 360 391 389
Let’s clean this up.
clean_table