<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/wrangling-wikileaks-dms/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://colinfay.me/wikileaks/"> Colin Fay</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Using R to turn raw data into browsable and reusable content.</p>
<p><span></span></p>
<h2>About</h2>
<p>On the 29th of July 2018, Emma Best published on her website the copy of<br>
11k+ wikileaks Twitter DM :<br>
<a rel="nofollow" href="https://emma.best/2018/07/29/11000-messages-from-private-wikileaks-chat-released/" target="_blank">https://emma.best/2018/07/29/11000-messages-from-private-wikileaks-chat-released/</a></p>
<p>To be honnest, I’m not really interested in the content of this dataset.<br>
What really interested me is that it’s <strong>raw data</strong> (copied and pasted<br>
text) waiting to be parsed, and that I could <strong>use R to turn these<br>
elements into a reusable and browsable content</strong>.</p>
<h2>The results</h2>
<p>Here are the links to the pages I’ve created with R from this dataset:</p>
<ul>
<li><a rel="nofollow" href="https://colinfay.me/wikileaksdm/index.html" target="_blank">Home</a> has the full<br>
dataset, to search and download.</li>
<li><a rel="nofollow" href="https://colinfay.me/wikileaksdm/timeline.html" target="_blank">Timeline</a> has a<br>
series of time-related content: notably DMs by years, and daily<br>
count of DMs.</li>
<li><a rel="nofollow" href="https://colinfay.me/wikileaksdm/users.html" target="_blank">Users</a> holds the<br>
dataset for each users.</li>
<li><a rel="nofollow" href="https://colinfay.me/wikileaksdm/mention_urls.html" target="_blank">mentions_urls</a><br>
holds the extracted mentions and urls</li>
<li><a rel="nofollow" href="https://colinfay.me/wikileaksdm/methodo.html" target="_blank">methodo</a> contains the<br>
methodology used for the data wrangling</li>
</ul>
<h2>Methodology</h2>
<h3>Extracting the content</h3>
<p>As I wanted to use the data offline (and not re-download it each time I<br>
compile the outputs), I’ve first extracted and saved the dataset as a<br>
.txt. You can now see it at <a rel="nofollow" href="https://colinfay.me/wikileaksdm/raw.txt" target="_blank">https://colinfay.me/wikileaksdm/raw.txt</a>.</p>
<p>Here is the code<br>
used:</p>
<div>
<div>
<pre><code><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span></code></pre>
</div>
</div>
<div>
<div>
<pre><code>## ── Attaching packages ────────────────────────────────────────── tidyverse 1.2.1 ──

## ✔ ggplot2 3.0.0     ✔ purrr   0.2.5
## ✔ tibble  1.4.2     ✔ dplyr   0.7.5
## ✔ tidyr   0.8.1     ✔ stringr 1.3.1
## ✔ readr   1.1.1     ✔ forcats 0.3.0

## ── Conflicts ───────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
</code></pre>
</div>
</div>
<div>
<div>
<pre><code><span>library</span><span>(</span><span>rvest</span><span>)</span><span>
</span></code></pre>
</div>
</div>
<div>
<div>
<pre><code>## Loading required package: xml2

## 
## Attaching package: 'rvest'

## The following object is masked from 'package:purrr':
## 
##     pluck

## The following object is masked from 'package:readr':
## 
##     guess_encoding
</code></pre>
</div>
</div>
<div>
<div>
<pre><code><span># Reading the page</span><span>
</span><span>doc</span><span> </span><span>&lt;-</span><span> </span><span>read_html</span><span>(</span><span>&quot;https://emma.best/2018/07/29/11000-messages-from-private-wikileaks-chat-released/&quot;</span><span>)</span><span>
</span><span># Extracting the paragraphs</span><span>
</span><span>doc</span><span> </span><span>&lt;-</span><span> </span><span>doc</span><span> </span><span>%>%</span><span> </span><span># Getting the p</span><span>
</span><span>html_nodes</span><span>(</span><span>&quot;p&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span># Getting the text</span><span>
</span><span>html_text</span><span>()</span><span>
</span><span># Removing the empty lines</span><span>
</span><span>doc</span><span> </span><span>&lt;-</span><span> </span><span>doc</span><span>[</span><span>!</span><span> </span><span>nchar</span><span>(</span><span>doc</span><span>)</span><span> </span><span>==</span><span> </span><span>0</span><span>]</span><span>
</span><span># Lines 1 to 9 are the content of the blogpost, not the content of the conversation. </span><span>
</span><span>doc</span><span>[</span><span>1</span><span>:</span><span>9</span><span>]</span><span>
</span></code></pre>
</div>
</div>
<div>
<div>
<pre><code>## [1] &quot;“Objectivity is short-hand for not having a significant pre-conceived agenda, eliding facts the audience would be interested in, or engaging in obvious falsehoods.” ~ WikiLeaks&quot;                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
## [2] &quot;Presented below are over 11,000 messages from the WikiLeaks + 10 chat, from which only excerpts have previously been published.&quot;                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
## [3] &quot;The chat is presented nearly in its entirety, with only a handful redactions made to protect the privacy and personal information of innocent, third parties as well as the already public name of an individual who has sent hate mail, made legal threats and who the source for the DMs considers a threat. It is at the request of the source that Mark’s full name is redacted, leaving only his first name and his initials (which he specified is alright). Though MGT’s full name is already public and easily discoverable, the source’s wishes are being respected. Beyond this individual, the redactions don’t include any information that’s relevant to understanding WikiLeaks or their activities.&quot;
## [4] &quot;The chat log shows WikiLeaks’ private attitudes, their use of FOIA laws, as well as discussions about WikiLeaks’ lobbying and attempts to “humiliate” politicians, PR and propaganda efforts (such as establishing a “medium term truth” for “phase 2”), troll operations, attempts to engineer situations where WikiLeaks would be able to sue their critics, and in some instances where WikiLeaks helped direct lawsuits filed by third parties or encouraged criminal investigations against their opponents. In some instances, the chats are revealing. In others, they show a mundane consistency with WikiLeaks’ public stances. A few are provocative and confounding.&quot;                                   
## [5] &quot;The extract below was created using DMArchiver, and is presented as pure text to make it easier to search and to provide as much metadata as possible (i.e. times as well as dates). The formatting is presented as-is, and shows users’ display names rather than their twitter handles. (Note: Emmy B is @GreekEmmy, not the author.)&quot;                                                                                                                                                                                                                                                                                                                                                                           
## [6] &quot;CW: At various points in the chat, there are examples of homophobia, transphobia, ableism, sexism, racism, antisemitism and other objectionable content and language. Some of these are couched as jokes, but are still likely to (and should) offend, as a racist or sexist jokes doesn’t cease to be racist or sexist because of an expected or desired laugh. Attempts to dismiss of these comments as “ironic” or “just trolling” merely invites comparisons to 4chan and ironic nazis. These comments, though offensive, are included in order to present as full and complete a record as possible and to let readers judge the context, purpose and merit of these comments for themselves.&quot;                
## [7] &quot; &quot;                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
## [8] &quot;If any current or former staffers, volunteers or hackers wants to add to my growing collection of leaks from within #WikiLeaks, please reach out. DMs are open and I’m EmmaBest on Wire.&quot;                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
## [9] &quot;— Emma Best (\u1d1c//ғ\u1d0f\u1d1c\u1d0f) \U0001f3f3‍\U0001f308 (@NatSecGeek) June 28, 2018&quot;
</code></pre>
</div>
</div>
<div>
<div>
<pre><code><span>doc</span><span>[</span><span>10</span><span>]</span><span>
</span></code></pre>
</div>
</div>
<div>
<div>
<pre><code>## [1] &quot;[2015-05-01 13:52:11] </code></pre></div></div>