<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/mining-sent-email-for-self-knowledge/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://dgarmat.github.io/Mining-Sent-Email-for-Self-Knowledge/"> Dan Garmat's Blog -- R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>How can we use data analytics to increase our self-knowledge? Along with biofeedback from digital devices like <a rel="nofollow" href="https://dgarmat.github.io/Calories-vs-Sleep/" target="_blank">FitBit</a>, less structured sources such as sent emails can provide insights.</p>
<p>E.g. here it seems my communication took a sudden more positive turn in 2013. Let’s see what else shakes out of my sent email corpus.</p>
<p><img alt="monthly_sentiment" title="monthly_sentiment" src="https://i2.wp.com/dgarmat.github.io/images/sent_mail00.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p><span></span></p>
<p>In <a rel="nofollow" href="https://www.mango-solutions.com/blog/snakes-in-a-package-combining-python-and-r-with-reticulate" target="_blank">Snakes in a Package: combining Python and R with reticulate</a> Adnan Fiaz uses a download of personal gmail from <a rel="nofollow" href="https://takeout.google.com/" target="_blank">Google Takeout</a> to extract <a rel="nofollow" href="http://r-bloggers.com/" target="_blank">R-bloggers</a> post counts from subject lines. To handle gmail’s choice of mbox file format, rather than write a new R package to parse mbox files, he uses <a rel="nofollow" href="https://rstudio.github.io/reticulate/articles/introduction.html" target="_blank">reticulate</a> to import a Python package, <a rel="nofollow" href="https://docs.python.org/2/library/mailbox.html" target="_blank">mailbox</a>. His approach seems a great use case for reticulate – when you want to take advantage of a highly developed Python package in R.</p>
<h3>Loading Email Corpus into R</h3>
<p>I wanted to mine my own emails for sentiment and see if I can learn anything about myself. Has my sent mail showed signs of mood trends over time? I started by following his example:</p>
<div>
<div>
<pre><code><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>library</span><span>(</span><span>stringr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tidytext</span><span>)</span><span>
</span><span>library</span><span>(</span><span>lubridate</span><span>)</span><span>
</span><span>library</span><span>(</span><span>reticulate</span><span>)</span><span>
</span><span>mailbox</span><span> </span><span>&lt;-</span><span> </span><span>import</span><span>(</span><span>&quot;mailbox&quot;</span><span>)</span><span>
</span><span>sent</span><span> </span><span>&lt;-</span><span> </span><span>mailbox</span><span>$</span><span>mbox</span><span>(</span><span>&quot;Sent-001.mbox&quot;</span><span>)</span><span>
</span><span>message</span><span> </span><span>&lt;-</span><span> </span><span>sent</span><span>$</span><span>get_message</span><span>(</span><span>11L</span><span>)</span><span>
</span><span>message</span><span>$</span><span>get</span><span>(</span><span>&quot;Date&quot;</span><span>)</span><span>
</span><span># [1] &quot;Mon, 23 Jul 2018 20:01:33 -0700&quot;</span><span>
</span><span>message</span><span>$</span><span>get</span><span>(</span><span>&quot;Subject&quot;</span><span>)</span><span>
</span><span># [1] &quot;Re: Ptfc schedules&quot;</span><span>
</span></code></pre>
</div>
</div>
<p>Loading in email #11, can see it’s about Portland Football Club’s schedule. I wanted to see the body of the email, but found the normal built-in documentation doesn’t exist for Python modules</p>
<div>
<div>
<pre><code><span>?</span><span>get_message</span><span>
</span><span># No documentation for ‘get_message’ in specified packages and libraries:</span><span>
</span><span># you could try ‘??get_message’</span><span>
</span><span>?</span><span>mailbox</span><span>
</span><span># No documentation for ‘mailbox’ in specified packages and libraries:</span><span>
</span><span># you could try ‘??mailbox’</span><span>
</span></code></pre>
</div>
</div>
<p>Returning <code>message</code> prints the whole thing, but with much additional unneeded formatting. So worked around it with nested <code>sub()</code> and <code>gsub()</code> commands on specific example emails to get down to the text I wrote and sent, only.</p>
<p>It starts with this already difficult to understand call</p>
<div>
<div>
<pre><code><span>sub</span><span>(</span><span>&quot;.*Content-Transfer-Encoding: quoted-printable&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>gsub</span><span>(</span><span>&quot;=E2=80=99&quot;</span><span>,</span><span> </span><span>&quot;'&quot;</span><span>,</span><span> </span><span>gsub</span><span>(</span><span>&quot;>&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>sub</span><span>(</span><span>&quot;On [A-Z][a-z]{2}.*&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>gsub</span><span>(</span><span>&quot;\n|\t&quot;</span><span>,</span><span> </span><span>&quot; &quot;</span><span>,</span><span> </span><span>message</span><span>)))))</span><span>
</span></code></pre>
</div>
</div>
<p>And, after much guess-try-see-what’s-left-and-add-another-<code>sub()</code>, ended up with this ugly function that does semi-reasonably for my goal of sentiment analysis:</p>
<div>
<div>
<pre><code><span>parse_sent_message</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>email</span><span>){</span><span>
</span><span>substr</span><span>(</span><span>
</span><span>gsub</span><span>(</span><span>&quot;-top:|-bottom:|break-word&quot;</span><span>,</span><span>&quot;&quot;</span><span>,</span><span>
</span><span>sub</span><span>(</span><span>&quot;Content-Type: application/pdf|Mime-Version: 1.0.*&quot;</span><span>,</span><span>&quot;&quot;</span><span>,</span><span>
</span><span>sub</span><span>(</span><span>&quot;.*charset ISO|charset UTF-8|charset us-ascii&quot;</span><span>,</span><span>&quot;&quot;</span><span>,</span><span>
</span><span>sub</span><span>(</span><span>&quot;.*Content-Transfer-Encoding: 7bit&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>sub</span><span>(</span><span>&quot;orwarded message.*&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>gsub</span><span>(</span><span>&quot;=|\&quot;&quot;</span><span>,</span><span> </span><span>&quot; &quot;</span><span>,</span><span> </span><span>gsub</span><span>(</span><span>&quot; &quot;</span><span>,</span><span> </span><span>&quot; &quot;</span><span>,</span><span> </span><span>gsub</span><span>(</span><span>&quot;= &quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>sub</span><span>(</span><span>&quot;.*Content-Transfer-Encoding: quoted-printable&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>sub</span><span>(</span><span>&quot;.*charset=UTF-8&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>gsub</span><span>(</span><span>&quot;=E2=80=99|'&quot;</span><span>,</span><span> </span><span>&quot;'&quot;</span><span>,</span><span> </span><span>gsub</span><span>(</span><span>&quot;>|&lt;&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>sub</span><span>(</span><span>&quot;On [A-Z][a-z]{2}.*&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>gsub</span><span>(</span><span>&quot;\n|\t|&lt;div||<br>&quot;</span><span>,</span><span> </span><span>&quot; &quot;</span><span>,</span><span> </span><span>email</span><span>))))))))))))))),</span><span> </span><span>1</span><span>,</span><span> </span><span>10000</span><span>)</span><span>
</span><span>}</span><span>
</span><span>parse_sent_message</span><span>(</span><span>message</span><span>)</span><span>
</span><span># [1] &quot; Hey aren't you planning to go to Seattle the 16th? Trying to figure out my days off schedule &quot;</span><span>
</span></code></pre>
</div>
</div>
<p>Good to go. I tried using the R <a rel="nofollow" href="https://github.com/MangoTheCat/mailman" target="_blank">mailman</a> wrapper, but ran into issues, so went back to the imported mailbox module. Importing and parsing took a few minutes:</p>
<div>
<div>
<pre><code><span>message</span><span>$</span><span>get</span><span>(</span><span>&quot;From&quot;</span><span>)</span><span> </span><span># check this email index 11 if from my email address</span><span>
</span><span>myemail</span><span> </span><span>&lt;-</span><span> </span><span>message</span><span>$</span><span>get</span><span>(</span><span>&quot;From&quot;</span><span>)</span><span> </span><span># since it is, save as myemail to check the rest</span><span>
</span><span>keys</span><span> </span><span>&lt;-</span><span> </span><span>sent</span><span>$</span><span>keys</span><span>()</span><span>
</span><span># keys &lt;-keys[1:3000] # uncomment if want to run the below on a subset to see if it works</span><span>
</span><span>number_of_messages</span><span> </span><span>&lt;-</span><span> </span><span>length</span><span>(</span><span>keys</span><span>)</span><span>
</span><span>pb</span><span> </span><span>&lt;-</span><span> </span><span>utils</span><span>::</span><span>txtProgressBar</span><span>(</span><span>max</span><span>=</span><span>number_of_messages</span><span>)</span><span>
</span><span>sent_messages</span><span> </span><span>&lt;-</span><span> </span><span>data_frame</span><span>(</span><span>sent_date</span><span> </span><span>=</span><span> </span><span>as.character</span><span>(</span><span>NA</span><span>),</span><span> </span><span>text</span><span> </span><span>=</span><span> </span><span>rep</span><span>(</span><span>as.character</span><span>(</span><span>NA</span><span>),</span><span> </span><span>number_of_messages</span><span>))</span><span>
</span><span>for</span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>seq_along</span><span>(</span><span>keys</span><span>)){</span><span>
</span><span>message</span><span> </span><span>&lt;-</span><span> </span><span>sent</span><span>$</span><span>get_message</span><span>(</span><span>keys</span><span>[</span><span>i</span><span>])</span><span>
</span><span>if</span><span>(</span><span>is.character</span><span>(</span><span>message</span><span>$</span><span>get</span><span>(</span><span>&quot;From&quot;</span><span>))){</span><span>
</span><span>if</span><span> </span><span>(</span><span>message</span><span>$</span><span>get</span><span>(</span><span>&quot;From&quot;</span><span>)</span><span> </span><span>%in%</span><span> </span><span>myemail</span><span>){</span><span>
</span><span>sent_messages</span><span>[</span><span>i</span><span>,</span><span> </span><span>1</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>message</span><span>$</span><span>get</span><span>(</span><span>&quot;Date&quot;</span><span>)</span><span>
</span><span>sent_messages</span><span>[</span><span>i</span><span>,</span><span> </span><span>2</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>parse_sent_message</span><span>(</span><span>message</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span><span>utils</span><span>::</span><span>setTxtProgressBar</span><span>(</span><span>pb</span><span>,</span><span> </span><span>i</span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
</div>
<p>If the message is not from me, it is saved as <code>NA</code>. What percent of mail flagged “sent” was not from <code>myemail</code>?</p>
<div>
<div>
<pre><code><span>sum</span><span>(</span><span>is.na</span><span>(</span><span>sent_messages</span><span>$</span><span>text</span><span>))</span><span> </span><span>/</span><span> </span><span>number_of_messages</span><span>
</span><span># [1] 0.6664132</span><span>
</span></code></pre>
</div>
</div>
<p>67%.<br>
Removing them and doing some additional processing, can see these 11,093 remaining sent emails range from November of 2014 to September of 2018 with a median date of October of 2013.</p>
<div>
<div>
<pre><code><span>sent_messages</span><span> </span><span>&lt;-</span><span> </span><span>sent_messages</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>!</span><span>is.na</span><span>(</span><span>text</span><span>))</span><span>
</span><span>sent_messages</span><span> </span><span>&lt;-</span><span> </span><span>sent_messages</span><span> </span><span>%>%</span><span> </span><span>mutate</span><span>(</span><span>sent_date</span><span> </span><span>=</span><span> </span><span>dmy_hms</span><span>(</span><span>sent_date</span><span>))</span><span>
</span><span># remove duplicates per month</span><span>
</span><span>sent_messages</span><span> </span><span>&lt;-</span><span> </span><span>sent_messages</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>year_sent</span><span> </span><span>=</span><span> </span><span>year</span><span>(</span><span>sent_date</span><span>),</span><span>
</span><span>month_sent</span><span> </span><span>=</span><span> </span><span>month</span><span>(</span><span>sent_date</span><span>))</span><span> </span><span>%>%</span><span> </span><span>group_by</span><span>(</span><span>year_sent</span><span>,</span><span> </span><span>month_sent</span><span>,</span><span> </span><span>text</span><span>)</span><span> </span><span>%>%</span><span> </span><span>top_n</span><span>(</span><span>1</span><span>,</span><span> </span><span>wt</span><span> </span><span>=</span><span> </span><span>sent_date</span><span>)</span><span> </span><span>%>%</span><span> </span><span>ungroup</span><span>()</span><span>
</span><span>sent_messages</span><span> </span><span>%>%</span><span> </span><span>summary</span><span>(</span><span>sent_date</span><span>)</span><span>
</span><span># sent_date text year_sent month_sent </span><span>
</span><span># Min. :2004-11-10 01:42:04 Length:11093 Min. :2004 Min. : 1.000 </span><span>
</span><span># 1st Qu.:2010-07-17 20:39:10 Class :character 1st Qu.:2010 1st Qu.: 3.000 </span><span>
</span><span># Median :2013-10-01 22:12:08 Mode :character Median :2013 Median : 6.000 </span><span>
</span><span># Mean :2013-03-24 10:55:30 Mean :2013 Mean : 6.416 </span><span>
</span><span># 3rd Qu.:2015-09-18 19:45:21 3rd Qu.:2015 3rd Qu.: 9.000 </span><span>
</span><span># Max. :2018-09-30 01:35:02 Max. :2018 Max. :12.000 </span><span>
</span></code></pre>
</div>
</div>
<p>While median date comes a bit later than the chronological midpoint seemingly implies slightly more emails later, from the chart above, it’s probably more due to missing years of data.</p>
<h2>Sentiment Analysis</h2>
<p>Julia Silge and David Robinson have put together an excellent online reference on text mining at <a rel="nofollow" href="https://www.tidytextmining.com/" target="_blank">Text Mining with R</a> so with some slight work can follow their analyses with email data. Using their <code>tidytext</code> package, quickly see a lot of html formatting tags still made it past my <code>gsub()</code> gauntlet.</p>
<div>
<div>
<pre><code><span>tidy_emails</span><span> </span><span>&lt;-</span><span> </span><span>sent_messages</span><span> </span><span>%>%</span><span>
</span><span>unnest_tokens</span><span>(</span><span>word</span><span>,</span><span> </span><span>text</span><span>)</span><span>
</span><span>tidy_emails</span><span>
</span><span># # A tibble: 886,870 x 4</span><span>
</span><span># sent_date year_sent month_sent word </span><span>
</span><span># </span></code></pre></div></div>