Back? Good.<\/p>\n
If you explored that site a bit, you likely saw a link to their GitHub repo<\/a>. There you’ll find JavaScript source code and even some browser plugins along with the core “fingerprint database”, By now, you’ve likely guessed that this blog post is introducing an R port of “wappalyzer” and if you have made such a guess, you’re correct! The For the moment, one real function is exposed: If you played around on the Wappalyzer site you saw that it “spiders” many URLs to try to get a very complete picture of components that a given site may use. I just used the main jQuery site in the above example and we managed to get quite a bit of information from just that interaction.<\/p>\n Wappalyzer (and, hence Wappalyzer-proper runs in a browser-context, so it can pick up DOM elements and JavaScript environment variables which can provide some extra information not present in the static content.<\/p>\n As you can see from the returned data frame, each tech component has one or more associated categories as well as a version number (if it could be guessed) and a value indicating how confident the guess was. It will also show where the guess came from (headers, scripts, etc).<\/p>\n I started making a pure R port of Wappalyzer but there are some frustrating “gotchas” with the JavaScript regular expressions that would have meant spending time identifying all the edge cases. Since the main Wappalyzer source is in JavaScript it was trivial to down-port it to a On the other hand, keeping it in JavaScript has some advantages, especially with the advent of Note also that these tech stack analyses can be dreadfully slow. That’s not due to using V8. The ones that slow are slow in all the language ports. This is due to the way the regular expressions are checked against the site content and for just using regular expressions to begin with. I’ve got some ideas to speed things up and may also introduce some “guide rails” to prevent lengthy operations and avoiding some checks if page characteristics meet certain criteria. Drop an issue if you have ideas as well.<\/p>\n Well, you can interactively see what a site uses like we did above. But, we can also look for commonalities across a number of sites. We’ll do this in a small example now. You can skip the expository and just work with the RStudio project<\/a> if that’s how you’d rather roll.<\/p>\n I have a recent-ish copy of a list of Fortune 1000 companies and their industry sectors along with their website URLs. Let’s see how many of those sites we can still access and what we identify.<\/p>\n I’m going to pause for just a minute<\/em> to revisit some “rules” in the context of this operation.<\/p>\n Specifically, I’m not:<\/p>\n However, I’m also not:<\/p>\n I believe<\/em> I’ve threaded the ethical needle properly (and, I do this for a living), but if you’re uncomfortable with this you should use a different target URL list.<\/p>\n Back to the code. We’ll need some packages:<\/p>\n Now, I’ve included data in the RStudio project GH repo since it can be used later on to test out changes to Let’s take a look at our crawler code:<\/p>\n If you were expecting The next block does the tech stack identification:<\/p>\n We filter out non-“OK” (HTTP 200-ish response) Then we let We’re almost<\/em> down to data we can play with! Let’s join it back up with the original metadata:<\/p>\n Note that some orgs seem to have changed hands\/domains so we’re left with ~750 sites to play with (we could have tried to match more, but this is just an exercise). If you do the same Let’s see how many categories we picked up:<\/p>\n Plenty to play with!<\/p>\n Now, what do you think the most common tech component is across all these diverse sites?<\/p>\n I had suspected it was Let’s see how broadly each tech stack category is used across the sectors:<\/p>\napps.json<\/code>?<\/a>. If you poke around on their repo, you’ll find the repo’s wiki and eventually see that folks have made various unofficial ports to other languages<\/a>.<\/p>\nrappalyzer<\/code>?<\/a> package is a “work-in-progress” but it’s usable enough to get some feedback on the package API, solicit contributors and show some things you can do with it.<\/p>\nJust rappalyze something already!<\/h3>\n
rappalyze()<\/code>. Feed it a URL string or an httr<\/code> response<\/code> object and — if the analysis was fruitful — you’ll get back a data frame with all the identified tech stack components. Let’s see what jquery.com<\/code>?<\/a> is made of:<\/p>\ndevtools::install_github("hrbrmstr\/rappalyzer")\r\n\r\nlibrary(rappalyzer)\r\n\r\nrappalyze("https:\/\/jquery.com\/")\r\n## # A tibble: 8 x 5\r\n## tech category match_type version confidence\r\n## <chr> <chr> <chr> <chr> <dbl>\r\n## 1 CloudFlare CDN headers <NA> 100\r\n## 2 Debian Operating Systems headers <NA> 100\r\n## 3 Modernizr JavaScript Frameworks script <NA> 100\r\n## 4 Nginx Web Servers headers <NA> 100\r\n## 5 PHP Programming Languages headers 5.4.45 100\r\n## 6 WordPress CMS meta 4.5.2 100\r\n## 7 WordPress Blogs meta 4.5.2 100\r\n## 8 jQuery JavaScript Frameworks script 1.11.3 100<\/code><\/pre>\nrappalyzer<\/code>) works by comparing sets of regular expressions against different parts of a site’s content. For now, rappalyzer<\/code> checks:<\/p>\n\n
A peek under the hood<\/h3>\n
V8<\/code>?<\/a>-compatible version and expose it via an R function. A longer-term plan is to deal with the regular expression conversion, but I kinda needed the functionality sooner than later (well, wanted<\/em> it in R, anyway).<\/p>\nchimera<\/code>?<\/a> (a phantomjs alternative that can put a full headless browser into a V8 engine context). Using that would mean getting the V8<\/code> package ported to more modern version of the V8 source code which isn’t trivial, but doable. Using chimera<\/code> would make it possible to identify even more tech stack components.<\/p>\nWhy is this useful?<\/h3>\n
Crawling 1,000 orgs<\/h3>\n
\n
robots.txt<\/code> file<\/li>\n\n
library(hrbrthemes)\r\nlibrary(tidyverse)\r\nlibrary(curl)\r\nlibrary(httr)\r\nlibrary(rvest)\r\nlibrary(stringi)\r\nlibrary(urltools)\r\nlibrary(rappalyzer) # devtools::install_github("hrbrmstr\/rappalyzer")\r\nlibrary(rprojroot)\r\n# I'm also using a user agent shortcut from the splashr package which is on CRAN<\/code><\/pre>\nrappalyzer<\/code>. That’s the reason for the if<\/code>\/file.exists<\/code> tests. As I’ve said before, be kind to sites you scrape and cache content whenever possible to avoid consuming resources that you don’t own.<\/p>\nrt <- find_rstudio_root_file()\r\n\r\nif (!file.exists(file.path(rt, "data", "f1k_gets.rds"))) {\r\n\r\n f1k <- read_csv(file.path(rt, "data", "f1k.csv"))\r\n\r\n targets <- pull(f1k, website)\r\n\r\n results <- list()\r\n errors <- list()\r\n\r\n OK <- function(res) {\r\n cat(".", sep="")\r\n results <<- c(results, list(res))\r\n }\r\n\r\n BAD <- function(err_msg) {\r\n cat("X", sep="")\r\n errors <<- c(errors, list(err_msg))\r\n }\r\n\r\n pool <- multi_set(total_con = 20)\r\n\r\n walk(targets, ~{\r\n multi_add(\r\n new_handle(url = .x, useragent = splashr::ua_macos_chrome, followlocation = TRUE, timeout = 60),\r\n OK, BAD\r\n )\r\n })\r\n\r\n multi_run(pool = pool)\r\n\r\n write_rds(results, file.path(rt, "data", "f1k_gets.rds"), compress = "xz")\r\n\r\n} else {\r\n results <- read_rds(file.path(rt, "data", "f1k_gets.rds"))\r\n}<\/code><\/pre>\nread_html()<\/code> and\/or GET()<\/code> calls you’re likely surprised at what you see. We’re trying to pull content from a thousand web sites, some of which may not be there anymore (for good or just temporarily). Sequential calls would need many minutes with error handling. We’re using the asynchronous\/parallel capabilities in the curl<\/code>?<\/a> package to setup all 1,000 requests with the ability to capture the responses. There’s a hack-ish “progress” bar that uses .<\/code> for “good” requests and X<\/code> for ones that didn’t work. It still takes a little time (and you may need to tweak the total_con<\/code> value on older systems) but way less than sequential GET<\/code>s would have, and any errors are implicitly handled (ignored).<\/p>\nif (!file.exists(file.path(rt, "data", "rapp_results.rds"))) {\r\n\r\n results <- keep(results, ~.x$status_code < 300)\r\n map(results, ~{\r\n list(\r\n url = .x$url,\r\n status_code = .x$status_code,\r\n content = .x$content,\r\n headers = httr:::parse_headers(.x$headers)\r\n ) -> res\r\n class(res) <- "response"\r\n res\r\n }) -> results\r\n\r\n # this takes a *while*\r\n pb <- progress_estimated(length(results))\r\n map_df(results, ~{\r\n pb$tick()$print()\r\n rap_df <- rappalyze(.x)\r\n if (nrow(rap_df) > 0) rap_df <- mutate(rap_df, url = .x$url)\r\n }) -> rapp_results\r\n\r\n write_rds(rapp_results, file.path(rt, "data", "rapp_results.rds"))\r\n\r\n} else {\r\n rapp_results <- read_rds(file.path(rt, "data", "rapp_results.rds"))\r\n}<\/code><\/pre>\ncurl<\/code> results and turn them into just enough<\/em> of an httr<\/code> request<\/code> object to be able to work with them.<\/p>\nrappalyzer<\/code> work. I had time to kill so I let it run sequentially. In a production or time-critical context, I’d use the future<\/code>?<\/a> package.<\/p>\nleft_join(\r\n mutate(rapp_results, host = domain(rapp_results$url)) %>%\r\n bind_cols(suffix_extract(.$host))\r\n ,\r\n mutate(f1k, host = domain(website)) %>%\r\n bind_cols(suffix_extract(.$host)),\r\n by = c("domain", "suffix")\r\n) %>%\r\n filter(!is.na(name)) -> rapp_results\r\n\r\nlength(unique(rapp_results$name))\r\n## [1] 754<\/code><\/pre>\ncurl<\/code>-fetching exercise on your own, you may get fewer or more total sites. Internet crawling is fraught with peril and it’s not really possible to conveniently convey 100% production code in a blog post.<\/p>\nComparing “Fortune 1000” org web stacks<\/h3>\n
xdf <- distinct(rapp_results, name, sector, category, tech)\r\n\r\nsort(unique(xdf$category))\r\n## [1] "Advertising Networks" "Analytics" "Blogs"\r\n## [4] "Cache Tools" "Captchas" "CMS"\r\n## [7] "Ecommerce" "Editors" "Font Scripts"\r\n## [10] "JavaScript Frameworks" "JavaScript Graphics" "Maps"\r\n## [13] "Marketing Automation" "Miscellaneous" "Mobile Frameworks"\r\n## [16] "Programming Languages" "Tag Managers" "Video Players"\r\n## [19] "Web Frameworks" "Widgets"<\/code><\/pre>\ncount(xdf, tech, sort=TRUE)\r\n## # A tibble: 115 x 2\r\n## tech n\r\n## <chr> <int>\r\n## 1 jQuery 572\r\n## 2 Google Tag Manager 220\r\n## 3 Modernizr 197\r\n## 4 Microsoft ASP.NET 193\r\n## 5 Google Font API 175\r\n## 6 Twitter Bootstrap 162\r\n## 7 WordPress 150\r\n## 8 jQuery UI 143\r\n## 9 Font Awesome 118\r\n## 10 Adobe Experience Manager 69\r\n## # ... with 105 more rows<\/code><\/pre>\njQuery<\/code> (and hinted at that when I used it as the initial rappalyzer<\/code> example). In a future blog post we’ll look at just how vulnerable orgs are based on which CDNs they use (many use similar jQuery and other resource CDNs, and use them insecurely).<\/p>\ngroup_by(xdf, sector) %>%\r\n count(category) %>%\r\n ungroup() %>%\r\n arrange(category) %>%\r\n mutate(category = factor(category, levels=rev(unique(category)))) %>%\r\n ggplot(aes(category, n)) +\r\n geom_boxplot() +\r\n scale_y_comma() +\r\n coord_flip() +\r\n labs(x=NULL, y="Tech\/Services Detected across Sectors",\r\n title="Usage of Tech Stack Categories by Sector") +\r\n theme_ipsum_rc(grid="X")<\/code><\/pre>\n
![\"\"](\"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2017\/09\/plot_zoom_png.png?resize=510%2C327&ssl=1\")
<\/a><\/p>\n