<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/who-wrote-that-anonymous-nyt-op-ed-text-similarity-analyses-with-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2018/09/anonymous-nyt-op-ed.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>In US politics news, the <em>New York Times</em> took the unusual step this week of publishing an <a rel="nofollow" href="https://www.nytimes.com/2018/09/05/opinion/trump-white-house-anonymous-resistance.html" target="_blank">anonymous op-ed</a> from a current member of the White House (assumed to be a cabinet member or senior staffer). Speculation about the identity of the author is, of course, rife. Much of the attention has focused on the use of specific words in the article, but can data science provide additional clues? In the last 48 hours, several R users have employed text similarity analysis to try and identify likely culprits.</p>
<p><a rel="nofollow" href="http://varianceexplained.org/r/op-ed-text-analysis/" target="_blank">David Robinson</a> followed a similar process to the one he used to identify which of <a rel="nofollow" href="https://github.com/hadley/rvest" target="_blank">President Trump's tweets</a> were written by him personally (as opposed to a staffer). He downloaded the tweets written by 69 Cabinet members and WH staff who maintain accounts on on Twitter (about 1.5 million words total). Then, he used the TD-IDF (term-frequency inverse-document-frequency) distance measure to find the rate at which certain words are more frequently (or less frequently) by the each subject. Comparing those rates to the rate of use in the op-ed. After ruling out the VP (and the President himself), this analysis suggests Secretary of State Pompeo as a suspect, in part due to his preference for the word &quot;malign&quot;.</p>
<div><a rel="nofollow" href="http://a1.typepad.com/6a0105360ba1c6970c022ad38f9e59200d-pi" target="_blank"><img border="0" alt="David Robinson's analysis" title="David Robinson's analysis" src="http://a1.typepad.com/6a0105360ba1c6970c022ad38f9e59200d-800wi"></a>
<div>David Robinson's analysis</div>
</div>
<p><a rel="nofollow" href="https://www.rjionline.org/stories/we-put-data-science-to-the-test-to-try-to-uncover-the-mystery-author-of-the" target="_blank">Michael Kearney</a> also used tweets as the comparison text, but limited the suspect pool to cabinet members only. He compared the text of those tweets to the NYT op-ed on using 107 numeric features, such as use of capitalizaion, punctuation, whitespace, and sentence length and structure. On that basis, again ruling out the President and VP, the closest match is Secretary Pompeo.</p>
<div><a rel="nofollow" href="http://a5.typepad.com/6a0105360ba1c6970c022ad38f9e5d200d-pi" target="_blank"><img border="0" alt="Michael Kearney's analysis" title="Michael Kearney's analysis" src="http://a5.typepad.com/6a0105360ba1c6970c022ad38f9e5d200d-800wi"></a>
<div>Michael Kearney's analysis</div>
</div>
<p><a rel="nofollow" href="https://github.com/kanishkamisra/inside-trumps-white-house" target="_blank">Kanishka Misra</a> also used tweets from cabinet members, but used the Burows' Delta metric to compare their text to that of the op-ed; those tweets with similar absolute word frequencies to the op-ed would be &quot;closest&quot; by this measure. Secretary of Transportation Elaine Chao is the closest match using this method.</p>
<div><a rel="nofollow" href="http://a5.typepad.com/6a0105360ba1c6970c022ad38f9e9d200d-pi" target="_blank"><img border="0" alt="Kanishka Misra's analysis" title="Kanishka Misra's analysis" src="http://a5.typepad.com/6a0105360ba1c6970c022ad38f9e9d200d-800wi"></a>
<div>Kanishka Misra's analysis</div>
</div>
<p>The authors take great pains to point out the limitations of these analyses: not all possible authors are active on Twitter; some of those likely don't write all their own tweets; Twitter is a medium that uses very different language than a newspaper editorial; and it's even possible the author was deliberately concealing their authorship through their language choices. But even so, the use of data has highlighted some unusual turns of phrase in the original op-ed that might have otherwise gone unobserved.</p>
<p>In each case the analysis was performed using the <a rel="nofollow" href="https://www.r-project.org/" target="_blank">R language</a>, and the authors make use of a common set of R packages to assist with the analysis:</p>
<ul>
<li><a rel="nofollow" href="https://github.com/hadley/rvest" target="_blank">rvest</a> to scrape data from the NYT website,</li>
<li><a rel="nofollow" href="https://rtweet.info/" target="_blank">rtweet</a>, to download the text of tweets from likely suspects,</li>
<li><a rel="nofollow" href="https://github.com/juliasilge/tidytext" target="_blank">tidytext</a>, to tokenize and otherwise prepare the texts for analysis, and</li>
<li><a rel="nofollow" href="https://github.com/kanishkamisra/inside-trumps-white-house" target="_blank">widyr</a> to reshape the data for distance calculations</li>
</ul>
<p>The full R code behind the analysis is provided by each author, and if nothing else provides interesting examples of different methods for text similarity analysis. These articles are a great demonstration of the power of rapid, in-depth data science with R: the op-ed was only published two days ago!</p>
<p> </p>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2018/09/anonymous-nyt-op-ed.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/6i1pcLGWiwg" height="1"></a></div>