{"id":6465,"date":"2017-09-25T06:17:04","date_gmt":"2017-09-25T11:17:04","guid":{"rendered":"https:\/\/rud.is\/b\/?p=6465"},"modified":"2018-03-07T17:01:43","modified_gmt":"2018-03-07T22:01:43","slug":"speeding-up-digital-arachinds","status":"publish","type":"post","link":"https:\/\/rud.is\/b\/2017\/09\/25\/speeding-up-digital-arachinds\/","title":{"rendered":"Speeding Up Digital Arachnids"},"content":{"rendered":"
\nspiderbar, spiderbar \nReads robots rules from afar.\nCrawls the web, any size; \nFetches with respect, never lies.\nLook Out! \nHere comes the spiderbar.\n\nIs it fast? \nListen bud, \nIt's got C++ under the hood.\nCan you scrape, from a site?\nTest with can_fetch(), TRUE == alright\nHey, there \nThere goes the spiderbar. \n<\/pre>\n
(Check the end of the post if you don’t recognize the lyrical riff.)<\/p>\n
Face front, true believer!<\/h3>\n
I’ve used and blogged about Peter Meissner’s most excellent robotstxt<\/code><\/a> package before<\/a>. It’s an essential tool for any ethical web scraper.<\/p>\n
But (there’s always a “but<\/em>“, right?), it was a definite bottleneck for an unintended package use case<\/a> earlier this year (yes, I still have not rounded out the corners on my “crawl delay” forthcoming post).<\/p>\n
I needed something faster for my bulk Crawl-Delay<\/code> analysis which led me to this small, spiffy C++ library<\/a> for parsing robots.txt<\/code> files. After a tiny bit of wrangling, that C++ library has turned into a small R package spiderbar<\/code><\/a> which is now hitting a CRAN mirror near you, soon. (CRAN — rightly so — did not like the unoriginal name rep<\/code>).<\/p>\n
How much faster?<\/h3>\n
I’m glad you asked!<\/p>\n
Let’s take a look at one benchmark: parsing robots.txt<\/code> and extracting Crawl-delay<\/code> entries. Just how much faster is spiderbar<\/code>?<\/p>\n
library(spiderbar)\r\nlibrary(robotstxt)\r\nlibrary(microbenchmark)\r\nlibrary(tidyverse)\r\nlibrary(hrbrthemes)\r\n\r\nrob <- get_robotstxt("imdb.com")\r\n\r\nmicrobenchmark(\r\n\r\n  robotstxt = {\r\n    x <- parse_robotstxt(rob)\r\n    x$crawl_delay\r\n  },\r\n\r\n  spiderbar = {\r\n    y <- robxp(rob)\r\n    crawl_delays(y)\r\n  }\r\n\r\n) -> mb1\r\n\r\nupdate_geom_defaults("violin", list(colour = "#4575b4", fill="#abd9e9"))\r\n\r\nautoplot(mb1) +\r\n  scale_y_comma(name="nanoseconds", trans="log10") +\r\n  labs(title="Microbenchmark results for parsing 'robots.txt' and extracting 'Crawl-delay' entries",\r\n       subtitle="Compares performance between robotstxt & spiderbar packages. Lower values are better.") +\r\n  theme_ipsum_rc(grid="Xx")<\/code><\/pre>\n
\"\"<\/a><\/p>\n
As you can see, it’s just a tad bit faster<\/em> ?.<\/p>\n
Now, you won’t notice that temporal gain in an interactive context but you absolutely will if you are cranking through a few million of them across a few thousand WARC files from the Common Crawl.<\/p>\n
But, I don’t care about Crawl-Delay<\/code>!<\/h3>\n
OK, fine. Do you care about fetchability? We can speed that up, too!<\/p>\n
rob_txt <- parse_robotstxt(rob)\r\nrob_spi <- robxp(rob)\r\n\r\nmicrobenchmark(\r\n\r\n  robotstxt = {\r\n    robotstxt:::path_allowed(rob_txt$permissions, "\/Vote")\r\n  },\r\n\r\n  spiderbar = {\r\n    can_fetch(rob_spi, "\/Vote")\r\n  }\r\n\r\n) -> mb2\r\n\r\nautoplot(mb2) +\r\n  scale_y_comma(name="nanoseconds", trans="log10") +\r\n  labs(title="Microbenchmark results for testing resource 'fetchability'",\r\n       subtitle="Compares performance between robotstxt & spiderbar packages. Lower values are better.") +\r\n  theme_ipsum_rc(grid="Xx")<\/code><\/pre>\n
\"\"<\/a><\/p>\n
Vectorized or it didn’t happen.<\/h3>\n
(Gosh, even Spider-Man got more respect!<\/em>)<\/p>\n
OK, this is a tough crowd, but we’ve got vectorization covered as well:<\/p>\n
microbenchmark(\r\n\r\n  robotstxt = {\r\n    paths_allowed(c("\/ShowAll\/this\/that", "\/Soundtracks\/this\/that", "\/Tsearch\/this\/that"), "imdb.com")\r\n  },\r\n\r\n  spiderbar = {\r\n    can_fetch(rob_spi, c("\/ShowAll\/this\/that", "\/Soundtracks\/this\/that", "\/Tsearch\/this\/that"))\r\n  }\r\n\r\n) -> mb3\r\n\r\nautoplot(mb3) +\r\n  scale_y_comma(name="nanoseconds", trans="log10") +\r\n  labs(title="Microbenchmark results for testing multiple resource 'fetchability'",\r\n       subtitle="Compares performance between robotstxt & spiderbar packages. Lower values are better.") +\r\n  theme_ipsum_rc(grid="Xx")<\/code><\/pre>\n
\"\"<\/a><\/p>\n
Excelsior!<\/h3>\n
Peter’s package does more than this one since it helps find the robots.txt<\/code> files and provides helpful data frames for more robots exclusion protocol content. And, we’ve got some plans<\/a> for package interoperability. So, stay tuned, true believer, for more spider-y goodness.<\/p>\n
You can check out the code and leave package questions or comments on GitHub<\/a>.<\/p>\n