{"id":11859,"date":"2019-02-03T06:48:55","date_gmt":"2019-02-03T11:48:55","guid":{"rendered":"https:\/\/rud.is\/b\/?p=11859"},"modified":"2019-02-04T17:27:28","modified_gmt":"2019-02-04T22:27:28","slug":"r-package-update-urlscan","status":"publish","type":"post","link":"https:\/\/rud.is\/b\/2019\/02\/03\/r-package-update-urlscan\/","title":{"rendered":"R Package Update: urlscan"},"content":{"rendered":"

The urlscan<\/code>?<\/a> package (an interface to the urlscan.io<\/a> API) is now at version 0.2.0 and supports urlscan.io’s authentication requirement when submitting a link for analysis. The service is handy if you want to learn about the details — all the gory technical details — for a website.<\/p>\n

For instance, say you wanted to check on r-project.org<\/code>. You could manually go to the site, enter that into the request bar and wait for the result<\/a>:<\/p>\n

\"\"<\/a><\/p>\n

Or, you can use R!. First pick your preferred social coding site, read through the source (this is going to be new advice for every post starting with this one. Don’t blindly trust code from any social coding site) an then install the urlscan<\/code> package:<\/p>\n

devtools::install_git(\"https:\/\/git.sr.ht\/~hrbrmstr\/urlscan\")\n# or\ndevtools::install_gitlab(\"hrbrmstr\/urlscan\")\n# or\ndevtools::install_github(\"hrbrmstr\/urlscan\")\n<\/code><\/pre>\n
Next, head on over back to urlscan.io and grab an API key<\/a> (it’s free). Stick that in your ~\/.Renviron<\/code> under URLSCAN_API_KEY<\/code> and then readRenviron(\"~\/.Renviron\")<\/code> in your R console.<\/p>\n
Now, let’s check out r-project.org<\/code>.<\/p>\n
library(urlscan)\nlibrary(tidyverse)\n\nrproj <- urlscan_search(\"r-project.org\")\n\nrproj\n##   URL Submitted: https:\/\/r-project.org\/\n##   Submission ID: eb2a5da1-dc0d-43e9-8236-dbc340b53772\n## Submission Type: public\n## Submission Note: Submission successful\n<\/code><\/pre>\n
There is more data in that rproj<\/code> object but we have enough to get more detailed results back. Note that site will return an error when you use urlscan_result()<\/code> if it hasn’t finished the analysis yet.<\/p>\n
rproj_res <- urlscan_result(\"eb2a5da1-dc0d-43e9-8236-dbc340b53772\", include_shot = TRUE)\n\nrproj_res\n##             URL: https:\/\/www.r-project.org\/\n##         Scan ID: eb2a5da1-dc0d-43e9-8236-dbc340b53772\n##       Malicious: FALSE\n##      Ad Blocked: FALSE\n##     Total Links: 8\n## Secure Requests: 19\n##    Secure Req %: 100%\n<\/code><\/pre>\n
That rproj_res<\/code> holds quite a bit of data and makes no assumptions about how you want to use it so you will need to do some wrangling with it to find out. The rproj_res$scan_result<\/code> entry contains entries with the following information:<\/p>\n