<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/digging-into-mbox-details-a-tale-of-tm-reticulate/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2018/08/04/digging-into-mbox-details-a-tale-of-tm-reticulate/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>

<div><a rel="nofollow" href="https://rud.is/b/2018/08/04/digging-into-mbox-details-a-tale-of-tm-reticulate/#f0" target="_blank"><img alt="✨" src="https://i1.wp.com/s.w.org/images/core/emoji/11/72x72/2728.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"></a></div>
<p>I had to processes a bunch of emails for a <code>$DAYJOB</code> task this week and my “default setting” is to use R for pretty much everything (this should come as no surprise). Treating mail as data is not an uncommon task and many R packages exist that can reach out and grab mail from servers or work directly with local mail archives.</p>
<h3>Mbox’in off the rails on a crazy tm<sup><a rel="nofollow" href="https://rud.is/b/2018/08/04/digging-into-mbox-details-a-tale-of-tm-reticulate/#f1" target="_blank">1</a></sup></h3>
<p>This particular mail corpus is in <a rel="nofollow" href="https://www.loc.gov/preservation/digital/formats/fdd/fdd000383.shtml" target="_blank"><code>mbox</code><img alt="🔗" src="https://i0.wp.com/s.w.org/images/core/emoji/11/72x72/1f517.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"></a> format since it was saved via Apple Mail. It’s one big text file with each message appearing one after the other. The format has been around for decades, and R’s <code>tm</code> package — via the <code>tm.plugin.mail</code> plugin package — can process these <code>mbox</code> files.</p>
<p>To demonstrate, we’ll use an Apple Mail archive excerpt from a set of R mailing list messages as they are not private/sensitive:</p>
<pre><code>library(tm)
library(tm.plugin.mail)

# point the tm corpus machinery to the mbox file and let it know the timestamp format since it varies
VCorpus(
  MBoxSource(&quot;~/Data/test.mbox/mbox&quot;),
  readerControl = list(
    reader = readMail(DateFormat = &quot;%a, %e %b %Y %H:%M:%S %z&quot;)
  )
) -> mbox

str(unclass(mbox), 1)
## List of 3
##  $ content:List of 198
##  $ meta   : list()
##   ..- attr(*, &quot;class&quot;)= chr &quot;CorpusMeta&quot;
##  $ dmeta  :'data.frame': 198 obs. of  0 variables

str(unclass(mbox[[1]]), 1)
## List of 2
##  $ content: chr [1:476] &quot;Try this:&quot; &quot;&quot; &quot;> library(lubridate)&quot; &quot;> library(tidyverse)&quot; ...
##  $ meta   :List of 9
##   ..- attr(*, &quot;class&quot;)= chr &quot;TextDocumentMeta&quot;

str(unclass(mbox[[1]]$meta), 1)
## List of 9
##  $ author       : chr &quot;jim holtman &lt;jholtman@gmail.com>&quot;
##  $ datetimestamp: POSIXlt[1:1], format: &quot;2018-08-01 15:01:17&quot;
##  $ description  : chr(0) 
##  $ heading      : chr &quot;Re: [R] read txt file - date - no space&quot;
##  $ id           : chr &quot;&lt;CAAxdm-5rNfu-+zE2PNkpudM33-coJBaiCrChWuDePtZfFerAjw@mail.gmail.com>&quot;
##  $ language     : chr &quot;en&quot;
##  $ origin       : chr(0) 
##  $ header       : chr [1:145] &quot;Delivered-To: bob@rud.is&quot; &quot;Received: by 2002:ac0:e681:0:0:0:0:0 with SMTP id b1-v6csp950182imq;&quot; &quot;        Wed, 1 Aug 2018 08:02:23 -0700 (PDT)&quot; &quot;X-Google-Smtp-Source: AAOMgpcdgBD4sDApBiF2DpKRfFZ9zi/4Ao32Igz9n8vT7EgE6InRoa7VZelMIik7OVmrFCRPDBde&quot; ...
##  $              : NULL</code></pre>
<p>We’re using <code>unclass()</code> since the <code>str()</code> output gets a bit crowded with all of the <code>tm</code> class attributes stuck in the output display.</p>
<p>The <code>tm</code> suite is designed for text mining. My task had nothing to do with text mining and I really just needed some header fields and body content in a data frame. If you’ve been working with R for a while, some things in the <code>str()</code> output will no doubt cause a bit of angst. For instance:</p>
<ul>
<li><code>datetimestamp: POSIXlt[1:1],</code> : <code>POSIXlt</code> <img alt="😒" src="https://i0.wp.com/s.w.org/images/core/emoji/11/72x72/1f612.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"> and data frames really don’t mix well</li>
<li><code>description  : chr(0)</code> / <code>origin       : chr(0)</code>: zero-length character vectors <img alt="☹" src="https://i2.wp.com/s.w.org/images/core/emoji/11/72x72/2639.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"></li>
<li><code>$              : NULL</code> : Blank element name with a <code>NULL</code> value…<em>I Don’t Even</em> <img alt="🤦🏽‍♀" src="https://i1.wp.com/s.w.org/images/core/emoji/11/72x72/1f926-1f3fd-200d-2640-fe0f.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"><sup><a rel="nofollow" href="https://rud.is/b/2018/08/04/digging-into-mbox-details-a-tale-of-tm-reticulate/#f2" target="_blank">2</a></sup></li>
</ul>
<p>The <code>tm</code> suite is also <em>super opinionated</em> and “helpfully” left out a ton of headers (though it did keep the source for the complete headers around). Still, we can roll up our sleeves and turn that into a data frame:</p>
<pre><code># helper function for cleaner/shorter code
`%|0|%` &lt;- function(x, y) { if (length(x) == 0) y else x }

# might as well stay old-school since we're using tm
do.call(
  rbind.data.frame,
  lapply(mbox, function(.x) {

    # we have a few choices, but this one is pretty explicit abt what it does
    # so we'll likely be able to decipher it quickly in 2 years when/if we come
    # back to it

    data.frame(
      author = .x$meta$author %|0|% NA_character_,
      datetimestamp = as.POSIXct(.x$meta$datetimestamp %|0|% NA),
      description = .x$meta$description %|0|% NA_character_,
      heading = .x$meta$heading %|0|% NA_character_,
      id = .x$meta$id %|0|% NA_character_,
      language = .x$meta$language %|0|% NA_character_,
      origin = .x$meta$origin %|0|% NA_character_,
      header = I(list(.x$meta$header %|0|% NA_character_)),
      body = I(list(.x$content %|0|% NA_character_)),
      stringsAsFactors = FALSE
    )

  })
) %>%
  glimpse()
## Observations: 198
## Variables: 9
## $ author        </code></pre>