{"id":10848,"date":"2018-05-30T12:19:23","date_gmt":"2018-05-30T17:19:23","guid":{"rendered":"https:\/\/rud.is\/b\/?p=10848"},"modified":"2018-05-30T12:19:23","modified_gmt":"2018-05-30T17:19:23","slug":"os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package","status":"publish","type":"post","link":"https:\/\/rud.is\/b\/2018\/05\/30\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\/","title":{"rendered":"OS Secrets Exposed: Extracting Extended File Attributes and Exploring Hidden Download URLs With The xattrs Package"},"content":{"rendered":"<p>Most modern operating systems keep secrets from you in many ways. One of these ways is by associating <a href=\"https:\/\/en.wikipedia.org\/wiki\/Extended_file_attributes\">extended file attributes<\/a> with files. These attributes can serve useful purposes. For instance, macOS uses them to identify when files have passed through the <a href=\"https:\/\/support.apple.com\/en-us\/HT202491\">Gatekeeper<\/a> or to store the URLs of files that were downloaded via Safari (though most other browsers add the <code>com.apple.metadata:kMDItemWhereFroms<\/code> attribute now, too).<\/p>\n<p>Attributes are nothing more than a series of key\/value pairs. They key must be a character value &amp; unique, and it&#8217;s fairly standard practice to keep the value component under 4K. Apart from that, you can put anything in the value: text, binary content, etc.<\/p>\n<p>When you&#8217;re in a terminal session you can tell that a file has extended attributes by looking for an <code>@<\/code> sign near the permissions column:<\/p>\n<pre>\n$ cd ~\/Downloads\n$ ls -l\ntotal 264856\n-rw-r--r--@ 1 user  staff     169062 Nov 27  2017 1109.1968.pdf\n-rw-r--r--@ 1 user  staff     171059 Nov 27  2017 1109.1968v1.pdf\n-rw-r--r--@ 1 user  staff     291373 Apr 27 21:25 1804.09970.pdf\n-rw-r--r--@ 1 user  staff    1150562 Apr 27 21:26 1804.09988.pdf\n-rw-r--r--@ 1 user  staff     482953 May 11 12:00 1805.01554.pdf\n-rw-r--r--@ 1 user  staff  125822222 May 14 16:34 RStudio-1.2.627.dmg\n-rw-r--r--@ 1 user  staff    2727305 Dec 21 17:50 athena-ug.pdf\n-rw-r--r--@ 1 user  staff      90181 Jan 11 15:55 bgptools-0.2.tar.gz\n-rw-r--r--@ 1 user  staff    4683220 May 25 14:52 osquery-3.2.4.pkg\n<\/pre>\n<p>You can work with extended attributes from the terminal with the <code>xattr<\/code> command, but do you really want to go to the terminal every time you want to examine these secret settings (now that you know your OS is keeping secrets from you)?<\/p>\n<p>I didn&#8217;t think so. Thus begat the <a href=\"https:\/\/github.com\/hrbrmstr\/xattrs\"><code>xattrs<\/code>? package<\/a>.<\/p>\n<h3>Exploring Past Downloads<\/h3>\n<p>Data scientists are (generally) inquisitive folk and tend to accumulate things. We grab papers, data, programs (etc.) and some of those actions are performed in browsers. Let&#8217;s use the <code>xattrs<\/code> package to rebuild a list of download URLs from the extended attributes on the files located in <code>~\/Downloads<\/code> (if you&#8217;ve chosen a different default for your browsers, use that directory).<\/p>\n<p>We&#8217;re not going to work with the entire package in this post (it&#8217;s really straightforward to use and has a README on the GitHub site along with extensive examples) but I&#8217;ll use one of the example files from the directory listing above to demonstrate a couple functions before we get to the main example.<\/p>\n<p>First, let&#8217;s see what is hidden with the RStudio disk image:<\/p>\n<pre><code class=\"language-r\">\nlibrary(xattrs)\nlibrary(reticulate) # not 100% necessary but you'll see why later\nlibrary(tidyverse) # we'll need this later\n\nlist_xattrs(\"~\/Downloads\/RStudio-1.2.627.dmg\")\n## [1] \"com.apple.diskimages.fsck\"            \"com.apple.diskimages.recentcksum\"    \n## [3] \"com.apple.metadata:kMDItemWhereFroms\" \"com.apple.quarantine\"   \n<\/code><\/pre>\n<p>There are four keys we can poke at, but the one that will help transition us to a larger example is <code>com.apple.metadata:kMDItemWhereFroms<\/code>. This is the key Apple has standardized on to store the source URL of a downloaded item. Let&#8217;s take a look:<\/p>\n<pre><code class=\"language-r\">\nget_xattr_raw(\"~\/Downloads\/RStudio-1.2.627.dmg\", \"com.apple.metadata:kMDItemWhereFroms\")\n##   [1] 62 70 6c 69 73 74 30 30 a2 01 02 5f 10 4c 68 74 74 70 73 3a 2f 2f 73 33 2e 61 6d 61\n##  [29] 7a 6f 6e 61 77 73 2e 63 6f 6d 2f 72 73 74 75 64 69 6f 2d 69 64 65 2d 62 75 69 6c 64\n##  [57] 2f 64 65 73 6b 74 6f 70 2f 6d 61 63 6f 73 2f 52 53 74 75 64 69 6f 2d 31 2e 32 2e 36\n##  [85] 32 37 2e 64 6d 67 5f 10 2c 68 74 74 70 73 3a 2f 2f 64 61 69 6c 69 65 73 2e 72 73 74\n## [113] 75 64 69 6f 2e 63 6f 6d 2f 72 73 74 75 64 69 6f 2f 6f 73 73 2f 6d 61 63 2f 08 0b 5a\n## [141] 00 00 00 00 00 00 01 01 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00\n## [169] 00 00 00 89\n<\/code><\/pre>\n<p>Why &#8220;raw&#8221;? Well, as noted above, the value component of these attributes can store anything and this one definitely has embedded nul[l]s (<code>0x00<\/code>) in it. We can try to read it as a string, though:<\/p>\n<pre><code class=\"language-r\">\nget_xattr(\"~\/Downloads\/RStudio-1.2.627.dmg\", \"com.apple.metadata:kMDItemWhereFroms\")\n## [1] \"bplist00\\xa2\\001\\002_\\020Lhttps:\/\/s3.amazonaws.com\/rstudio-ide-build\/desktop\/macos\/RStudio-1.2.627.dmg_\\020,https:\/\/dailies.rstudio.com\/rstudio\/oss\/mac\/\\b\\vZ\"\n<\/code><\/pre>\n<p>So, we can <em>kinda<\/em> figure out the URL but it&#8217;s definitely not pretty. The general practice of Safari (and other browsers) is to use a <a href=\"https:\/\/developer.apple.com\/library\/content\/documentation\/Cocoa\/Conceptual\/PropertyLists\/AboutPropertyLists\/AboutPropertyLists.html\">binary property list<\/a> to store metadata in the value component of an extended attribute (at least for these URL references).<\/p>\n<p>There will eventually be a native <a href=\"https:\/\/github.com\/ebarnard\/rust-plist\">Rust-backed<\/a> property list reading package for R, but we can work with that binary plist data in two ways: first, via the <code>read_bplist()<\/code> function that comes with the <code>xattrs<\/code> package and wraps Linux\/BSD or macOS system utilities (which are super expensive since it also means writing out data to a file each time) <em>or<\/em> turn to Python which already has this capability. We&#8217;re going to use the latter.<\/p>\n<p>I like to prime the Python setup with <code>invisible(py_config())<\/code> but that is not really necessary (I do it mostly b\/c I have a wild number of Python &#8212; don&#8217;t judge &#8212; installs and use the <code>RETICULATE_PYTHON<\/code> env var for the one I use with R). You&#8217;ll need to install the <code>biplist<\/code> module via <code>pip3 install bipist<\/code> or <code>pip install bipist<\/code> depending on your setup. I highly recommended using Python 3.x vs 2.x, though.<\/p>\n<pre><code class=\"language-r\">\nbiplist <- import(\"biplist\", as=\"biplist\")\n\nbiplist$readPlistFromString(\n  get_xattr_raw(\n    \"~\/Downloads\/RStudio-1.2.627.dmg\", \"com.apple.metadata:kMDItemWhereFroms\"\n  )\n)\n## [1] \"https:\/\/s3.amazonaws.com\/rstudio-ide-build\/desktop\/macos\/RStudio-1.2.627.dmg\"\n## [2] \"https:\/\/dailies.rstudio.com\/rstudio\/oss\/mac\/\" \n<\/code><\/pre>\n<p>That's <em>much<\/em> better.<\/p>\n<p>Let's work with metadata for the whole directory:<\/p>\n<pre><code class=\"language-r\">\nlist.files(\"~\/Downloads\", full.names = TRUE) %>% \n  keep(has_xattrs) %>% \n  set_names(basename(.)) %>% \n  map_df(read_xattrs, .id=\"file\") -> xdf\n\nxdf\n## # A tibble: 24 x 4\n##    file            name                                  size contents   \n##    <chr>           <chr>                                <dbl> <list>     \n##  1 1109.1968.pdf   com.apple.lastuseddate#PS               16 <raw [16]> \n##  2 1109.1968.pdf   com.apple.metadata:kMDItemWhereFroms   110 <raw [110]>\n##  3 1109.1968.pdf   com.apple.quarantine                    74 <raw [74]> \n##  4 1109.1968v1.pdf com.apple.lastuseddate#PS               16 <raw [16]> \n##  5 1109.1968v1.pdf com.apple.metadata:kMDItemWhereFroms   116 <raw [116]>\n##  6 1109.1968v1.pdf com.apple.quarantine                    74 <raw [74]> \n##  7 1804.09970.pdf  com.apple.metadata:kMDItemWhereFroms    86 <raw [86]> \n##  8 1804.09970.pdf  com.apple.quarantine                    82 <raw [82]> \n##  9 1804.09988.pdf  com.apple.lastuseddate#PS               16 <raw [16]> \n## 10 1804.09988.pdf  com.apple.metadata:kMDItemWhereFroms   104 <raw [104]>\n## # ... with 14 more rows\n\n## count(xdf, name, sort=TRUE)\n## # A tibble: 5 x 2\n##   name                                     n\n##   <chr>                                <int>\n## 1 com.apple.metadata:kMDItemWhereFroms     9\n## 2 com.apple.quarantine                     9\n## 3 com.apple.lastuseddate#PS                4\n## 4 com.apple.diskimages.fsck                1\n## 5 com.apple.diskimages.recentcksum         1\n<\/code><\/pre>\n<p>Now we can focus on the task at hand: recovering the URLs:<\/p>\n<pre><code class=\"language-r\">\nlist.files(\"~\/Downloads\", full.names = TRUE) %>% \n  keep(has_xattrs) %>% \n  set_names(basename(.)) %>% \n  map_df(read_xattrs, .id=\"file\") %>% \n  filter(name == \"com.apple.metadata:kMDItemWhereFroms\") %>% \n  mutate(where_from = map(contents, biplist$readPlistFromString)) %>% \n  select(file, where_from) %>% \n  unnest() %>% \n  filter(!where_from == \"\")\n## # A tibble: 15 x 2\n##    file                where_from                                                       \n##    <chr>               <chr>                                                            \n##  1 1109.1968.pdf       https:\/\/arxiv.org\/pdf\/1109.1968.pdf                              \n##  2 1109.1968.pdf       https:\/\/www.google.com\/                                          \n##  3 1109.1968v1.pdf     https:\/\/128.84.21.199\/pdf\/1109.1968v1.pdf                        \n##  4 1109.1968v1.pdf     https:\/\/www.google.com\/                                          \n##  5 1804.09970.pdf      https:\/\/arxiv.org\/pdf\/1804.09970.pdf                             \n##  6 1804.09988.pdf      https:\/\/arxiv.org\/ftp\/arxiv\/papers\/1804\/1804.09988.pdf           \n##  7 1805.01554.pdf      https:\/\/arxiv.org\/pdf\/1805.01554.pdf                             \n##  8 athena-ug.pdf       http:\/\/docs.aws.amazon.com\/athena\/latest\/ug\/athena-ug.pdf        \n##  9 athena-ug.pdf       https:\/\/www.google.com\/                                          \n## 10 bgptools-0.2.tar.gz http:\/\/nms.lcs.mit.edu\/software\/bgp\/bgptools\/bgptools-0.2.tar.gz \n## 11 bgptools-0.2.tar.gz http:\/\/nms.lcs.mit.edu\/software\/bgp\/bgptools\/                    \n## 12 osquery-3.2.4.pkg   https:\/\/osquery-packages.s3.amazonaws.com\/darwin\/osquery-3.2.4.p\u2026\n## 13 osquery-3.2.4.pkg   https:\/\/osquery.io\/downloads\/official\/3.2.4                      \n## 14 RStudio-1.2.627.dmg https:\/\/s3.amazonaws.com\/rstudio-ide-build\/desktop\/macos\/RStudio\u2026\n## 15 RStudio-1.2.627.dmg https:\/\/dailies.rstudio.com\/rstudio\/oss\/mac\/             \n<\/code><\/pre>\n<p>(There are multiple URL entries due to the fact that some browsers preserve the path you traversed to get to the final download.)<\/p>\n<p>Note: if Python is not an option for you, you <em>can<\/em> use the hack-y <code>read_bplist()<\/code> function in the package, but it will be much, much slower and you'll need to deal with an ugly <code>list<\/code> object vs some quaint text vectors.<\/p>\n<h3>FIN<\/h3>\n<p>Have some fun exploring what other secrets your OS may be hiding from you and if you're on Windows, give this a go. I have no idea if it will compile or work there, but if it does, definitely report back!<\/p>\n<p>Remember that the package lets you <em>set<\/em> and <em>remove<\/em> extended attributes as well, so you can use them to store metadata with your data files (they don't always survive file or OS transfers but if you keep things local they can be an interesting way to tag your files) or clean up items you do not want stored.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Most modern operating systems keep secrets from you in many ways. One of these ways is by associating extended file attributes with files. These attributes can serve useful purposes. For instance, macOS uses them to identify when files have passed through the Gatekeeper or to store the URLs of files that were downloaded via Safari [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":3,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":""},"categories":[91],"tags":[],"class_list":["post-10848","post","type-post","status-publish","format-standard","hentry","category-r"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.3 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>OS Secrets Exposed: Extracting Extended File Attributes and Exploring Hidden Download URLs With The xattrs Package - rud.is<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/rud.is\/b\/2018\/05\/30\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"OS Secrets Exposed: Extracting Extended File Attributes and Exploring Hidden Download URLs With The xattrs Package - rud.is\" \/>\n<meta property=\"og:description\" content=\"Most modern operating systems keep secrets from you in many ways. One of these ways is by associating extended file attributes with files. These attributes can serve useful purposes. For instance, macOS uses them to identify when files have passed through the Gatekeeper or to store the URLs of files that were downloaded via Safari [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/rud.is\/b\/2018\/05\/30\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\/\" \/>\n<meta property=\"og:site_name\" content=\"rud.is\" \/>\n<meta property=\"article:published_time\" content=\"2018-05-30T17:19:23+00:00\" \/>\n<meta name=\"author\" content=\"hrbrmstr\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"hrbrmstr\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2018\\\/05\\\/30\\\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2018\\\/05\\\/30\\\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\\\/\"},\"author\":{\"name\":\"hrbrmstr\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\"},\"headline\":\"OS Secrets Exposed: Extracting Extended File Attributes and Exploring Hidden Download URLs With The xattrs Package\",\"datePublished\":\"2018-05-30T17:19:23+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2018\\\/05\\\/30\\\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\\\/\"},\"wordCount\":791,\"commentCount\":1,\"publisher\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\"},\"articleSection\":[\"R\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/rud.is\\\/b\\\/2018\\\/05\\\/30\\\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2018\\\/05\\\/30\\\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\\\/\",\"url\":\"https:\\\/\\\/rud.is\\\/b\\\/2018\\\/05\\\/30\\\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\\\/\",\"name\":\"OS Secrets Exposed: Extracting Extended File Attributes and Exploring Hidden Download URLs With The xattrs Package - rud.is\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#website\"},\"datePublished\":\"2018-05-30T17:19:23+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2018\\\/05\\\/30\\\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/rud.is\\\/b\\\/2018\\\/05\\\/30\\\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/2018\\\/05\\\/30\\\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/rud.is\\\/b\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"OS Secrets Exposed: Extracting Extended File Attributes and Exploring Hidden Download URLs With The xattrs Package\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#website\",\"url\":\"https:\\\/\\\/rud.is\\\/b\\\/\",\"name\":\"rud.is\",\"description\":\"&quot;In God we trust. All others must bring data&quot;\",\"publisher\":{\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/rud.is\\\/b\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\\\/\\\/rud.is\\\/b\\\/#\\\/schema\\\/person\\\/d7cb7487ab0527447f7fda5c423ff886\",\"name\":\"hrbrmstr\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\",\"width\":460,\"height\":460,\"caption\":\"hrbrmstr\"},\"logo\":{\"@id\":\"https:\\\/\\\/i0.wp.com\\\/rud.is\\\/b\\\/wp-content\\\/uploads\\\/2023\\\/10\\\/ukr-shield.png?fit=460%2C460&ssl=1\"},\"description\":\"Don't look at me\u2026I do what he does \u2014 just slower. #rstats avuncular \u2022 ?Resistance Fighter \u2022 Cook \u2022 Christian \u2022 [Master] Chef des Donn\u00e9es de S\u00e9curit\u00e9 @ @rapid7\",\"sameAs\":[\"http:\\\/\\\/rud.is\"],\"url\":\"https:\\\/\\\/rud.is\\\/b\\\/author\\\/hrbrmstr\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"OS Secrets Exposed: Extracting Extended File Attributes and Exploring Hidden Download URLs With The xattrs Package - rud.is","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/rud.is\/b\/2018\/05\/30\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\/","og_locale":"en_US","og_type":"article","og_title":"OS Secrets Exposed: Extracting Extended File Attributes and Exploring Hidden Download URLs With The xattrs Package - rud.is","og_description":"Most modern operating systems keep secrets from you in many ways. One of these ways is by associating extended file attributes with files. These attributes can serve useful purposes. For instance, macOS uses them to identify when files have passed through the Gatekeeper or to store the URLs of files that were downloaded via Safari [&hellip;]","og_url":"https:\/\/rud.is\/b\/2018\/05\/30\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\/","og_site_name":"rud.is","article_published_time":"2018-05-30T17:19:23+00:00","author":"hrbrmstr","twitter_card":"summary_large_image","twitter_misc":{"Written by":"hrbrmstr","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/rud.is\/b\/2018\/05\/30\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\/#article","isPartOf":{"@id":"https:\/\/rud.is\/b\/2018\/05\/30\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\/"},"author":{"name":"hrbrmstr","@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"headline":"OS Secrets Exposed: Extracting Extended File Attributes and Exploring Hidden Download URLs With The xattrs Package","datePublished":"2018-05-30T17:19:23+00:00","mainEntityOfPage":{"@id":"https:\/\/rud.is\/b\/2018\/05\/30\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\/"},"wordCount":791,"commentCount":1,"publisher":{"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"articleSection":["R"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/rud.is\/b\/2018\/05\/30\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/rud.is\/b\/2018\/05\/30\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\/","url":"https:\/\/rud.is\/b\/2018\/05\/30\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\/","name":"OS Secrets Exposed: Extracting Extended File Attributes and Exploring Hidden Download URLs With The xattrs Package - rud.is","isPartOf":{"@id":"https:\/\/rud.is\/b\/#website"},"datePublished":"2018-05-30T17:19:23+00:00","breadcrumb":{"@id":"https:\/\/rud.is\/b\/2018\/05\/30\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/rud.is\/b\/2018\/05\/30\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/rud.is\/b\/2018\/05\/30\/os-secrets-exposed-extracting-extended-file-attributes-and-exploring-hidden-download-urls-with-the-xattrs-package\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/rud.is\/b\/"},{"@type":"ListItem","position":2,"name":"OS Secrets Exposed: Extracting Extended File Attributes and Exploring Hidden Download URLs With The xattrs Package"}]},{"@type":"WebSite","@id":"https:\/\/rud.is\/b\/#website","url":"https:\/\/rud.is\/b\/","name":"rud.is","description":"&quot;In God we trust. All others must bring data&quot;","publisher":{"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/rud.is\/b\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/rud.is\/b\/#\/schema\/person\/d7cb7487ab0527447f7fda5c423ff886","name":"hrbrmstr","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","url":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","contentUrl":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1","width":460,"height":460,"caption":"hrbrmstr"},"logo":{"@id":"https:\/\/i0.wp.com\/rud.is\/b\/wp-content\/uploads\/2023\/10\/ukr-shield.png?fit=460%2C460&ssl=1"},"description":"Don't look at me\u2026I do what he does \u2014 just slower. #rstats avuncular \u2022 ?Resistance Fighter \u2022 Cook \u2022 Christian \u2022 [Master] Chef des Donn\u00e9es de S\u00e9curit\u00e9 @ @rapid7","sameAs":["http:\/\/rud.is"],"url":"https:\/\/rud.is\/b\/author\/hrbrmstr\/"}]}},"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p23idr-2OY","jetpack_likes_enabled":true,"jetpack-related-posts":[{"id":11082,"url":"https:\/\/rud.is\/b\/2018\/07\/22\/new-apache-drill-udf-for-processing-twitter-tweet-text\/","url_meta":{"origin":10848,"position":0},"title":"New Apache Drill UDF for Processing Twitter Tweet Text","author":"hrbrmstr","date":"2018-07-22","format":false,"excerpt":"There are many ways to gather Twitter data for analysis and many R and Python (et al) libraries make full use of the Twitter API when building a corpus to extract useful metadata for each tweet along with the text of each tweet. However, many corpus archives are minimal and\u2026","rel":"","context":"In &quot;Apache Drill&quot;","block_context":{"text":"Apache Drill","link":"https:\/\/rud.is\/b\/category\/apache-drill\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":11215,"url":"https:\/\/rud.is\/b\/2018\/08\/04\/digging-into-mbox-details-a-tale-of-tm-reticulate\/","url_meta":{"origin":10848,"position":1},"title":"Digging into mbox details: A tale of tm &#038; reticulate","author":"hrbrmstr","date":"2018-08-04","format":false,"excerpt":"\u2728 I had to processes a bunch of emails for a $DAYJOB task this week and my \"default setting\" is to use R for pretty much everything (this should come as no surprise). Treating mail as data is not an uncommon task and many R packages exist that can reach\u2026","rel":"","context":"In &quot;Python&quot;","block_context":{"text":"Python","link":"https:\/\/rud.is\/b\/category\/python-2\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":10287,"url":"https:\/\/rud.is\/b\/2018\/05\/18\/lmx-ot-nosj-interchanging-classic-data-formats-with-single-blackmagic-incantations\/","url_meta":{"origin":10848,"position":2},"title":"&#8220;LMX ot NOSJ!&#8221; Interchanging Classic Data Formats With Single blackmagic Incantations","author":"hrbrmstr","date":"2018-05-18","format":false,"excerpt":"The D.C. Universe magic hero Zatanna used spells (i.e. incantations) to battle foes and said spells were just sentences said backwards, hence the mixed up jumble in the title. But, now I'm regretting not naming the package zatanna and reversing the function names to help ensure they're only used deliberately\u2026","rel":"","context":"In &quot;R&quot;","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":13026,"url":"https:\/\/rud.is\/b\/2021\/04\/14\/avoiding-the-mdls-command-line-round-trip-with-swiftrswift_function\/","url_meta":{"origin":10848,"position":3},"title":"Avoiding The mdls Command Line Round Trip With swiftr::swift_function()","author":"hrbrmstr","date":"2021-04-14","format":false,"excerpt":"The last post showed how to work with the macOS mdls command line XML output, but with {swiftr} we can avoid the command line round trip by bridging the low-level Spotlight API (which mdls uses) directly in R via Swift. If you've already played with {swiftr} before but were somewhat\u2026","rel":"","context":"In &quot;macOS&quot;","block_context":{"text":"macOS","link":"https:\/\/rud.is\/b\/category\/macos\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":18584,"url":"https:\/\/rud.is\/b\/2024\/03\/23\/vulnchecks-free-community-kev-cve-apis-code-golang-cli-utility\/","url_meta":{"origin":10848,"position":4},"title":"VulnCheck&#8217;s Free Community KEV &#038; CVE APIs  (Code &#038; Golang CLI Utility)","author":"hrbrmstr","date":"2024-03-23","format":false,"excerpt":"VulnCheck has some new, free API endpoints for the cybersecurity community. Two extremely useful ones are for their extended version of CISA\u2019s KEV, and an in-situ replacement for NVD\u2019s sad excuse for an API and soon-to-be-removed JSON feeds. There are two ways to work with these APIs. One is retrieve\u2026","rel":"","context":"In &quot;APIs&quot;","block_context":{"text":"APIs","link":"https:\/\/rud.is\/b\/category\/apis\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4560,"url":"https:\/\/rud.is\/b\/2016\/07\/26\/use-quick-formula-functions-in-purrrmap-base-vs-tidtyverse-idiom-comparisonsexamples\/","url_meta":{"origin":10848,"position":5},"title":"Use quick formula functions in purrr::map (+ base vs tidtyverse idiom comparisons\/examples)","author":"hrbrmstr","date":"2016-07-26","format":false,"excerpt":"I've converted the vast majority of my *apply usage over to purrr functions. In an attempt to make this a quick post, I'll refrain from going into all the benefits of the purrr package. Instead, I'll show just one thing that's super helpful: formula functions. After seeing this Quartz article\u2026","rel":"","context":"In &quot;R&quot;","block_context":{"text":"R","link":"https:\/\/rud.is\/b\/category\/r\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts\/10848","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/comments?post=10848"}],"version-history":[{"count":0,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/posts\/10848\/revisions"}],"wp:attachment":[{"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/media?parent=10848"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/categories?post=10848"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rud.is\/b\/wp-json\/wp\/v2\/tags?post=10848"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}