<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/using-r-to-gain-insights-into-the-emotional-journeys-in-war-and-peace/"></div></div><div><a data-url="https://www.r-bloggers.com/using-r-to-gain-insights-into-the-emotional-journeys-in-war-and-peace/" data-counturl="https://www.r-bloggers.com/using-r-to-gain-insights-into-the-emotional-journeys-in-war-and-peace/" data-count="horizontal" data-text="Using R to Gain Insights into the Emotional Journeys in War and Peace" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2016/12/war-and-peace.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p><em>by Wee Hyong Tok, Senior Data Scientist Manager at Microsoft</em></p>
<p>How do you read a novel in record time, and gain insights into the emotional journey of main characters, as they go through various trials and tribulations, as an exciting story unfolds from chapter to chapter?</p>
<p>I remembered my experiences when I start reading a novel, and I get intrigued by the story, and simply cannot wait to get to the last chapter. I also recall many conversations with friends on some of the interesting novels that I have read awhile back, and somehow have only vague recollection of what happened in a specific chapter. In this post, I’ll work through how we can use R to analyze the English translation of War and Peace.</p>
<p><a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/War_and_Peace">War and Peace</a> is a novel by Leo Tolstoy, and captures the salient points about Russian history from the period 1805 to 1812. The novel consists of the stories of five families, and captures the trials and tribulations of various characters (e.g. Natasha and Andre). The novel consists of about 1400 pages, and is one of the longest novels that have been written.</p>
<p>We hypothesize that if we can build a dashboard (shown below), this will allow us to gain insights into the emotional journey undertaken by the characters in War and Peace.</p>
<p><a target="_blank" rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d23f7462970c-pi"><img title="Warpeace1" alt="Warpeace1" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d23f7462970c-800wi" border="0"></a></p>
<p>For example, using the dashboard below, and correlating it with the War and Peace story, we can see how Andrey falling in love with Natsha in Book 6 have created a green (positive) highlight in his life. We can also see how Natsha’s family bankruptcy in Book 6 overshadows her budding romance with Andrey. In addition, we can also observe how Pierre emotional journey stays red (negative), especially in Book 11-14, as he almost killed himself, and finally ending up in prison.</p>
<h2>Building the War and Peace Visualization</h2>
<p>In this post, I’ll show you how we use the <a target="_blank" rel="nofollow" href="https://github.com/rstudio/d3heatmap">d3heatmap</a> package to build the War and Peace visualization. Let’s start with a dataset, where we used Azure Data Lake Analytics, and U-SQL extensions to analyze the English translation of War and Peace.</p>
<h3>Let’s Get Started</h3>
<p>The <a target="_blank" rel="nofollow" href="https://machinelearningdatasets.blob.core.windows.net/public/warandpeace/warandpeaceSentimentKeyPhrase.csv">dataset</a> (23Mb CSV) consists of 6 columns. V1, V2, V3 captures the year, book, and chapter respectively. V4 is a line of text from the novel. V5 denotes the key phrase extracted from each sentence, and V6 denotes the sentiment associated with the key phrase.</p>
<p>To get started, we will first specify the R libraries that we will use to create the War and Peace heatmap visualization.</p>
<div>
<pre><span># Load libs</span>
library (dplyr)
library(d3heatmap)
library(tidyr)
library(stringr)
library(reshape)
</pre>
</div>
<p>Next, we load the data that is stored in Azure Blob storage.</p>
<div>
<pre><span># Read the war and peace dataset, with key phrase and sentiment extracted</span>
warpeace <span>&lt;-</span> read.csv(file<span>=</span><span>&quot;https://machinelearningdatasets.blob.core.windows.net/public/warandpeace/warandpeaceSentimentKeyPhrase.csv&quot;</span>,
  header<span>=</span><span>FALSE</span>, sep<span>=</span><span>&quot;,&quot;</span>)
</pre>
</div>
<p>We focus on some of the main characters that we want to analyze by specifying them, as follows:</p>
<div>
<pre><span># Specify the characters in War and Peace</span>
bookCharacters <span>&lt;-</span> c(<span>&quot;Cyril&quot;</span>,  <span>&quot;Pierre&quot;</span>,  <span>&quot;Nicholas&quot;</span>,
        <span>&quot;Rostóva&quot;</span>,<span>&quot;Natalie&quot;</span>,<span>&quot;Natasha&quot;</span>,<span>&quot;Catiche&quot;</span>, <span>&quot;Ilyá&quot;</span>,
        <span>&quot;Nikolenka&quot;</span>,  <span>&quot;Pétya&quot;</span>,<span>&quot;Véra&quot;</span>,<span>&quot;Sónya&quot;</span>, <span>&quot;Nicholas&quot;</span>,
        <span>&quot;Alpatych&quot;</span>, <span>&quot;Vasíli&quot;</span>, <span>&quot;Anatole&quot;</span>, <span>&quot;Leyla&quot;</span>, <span>&quot;Borís&quot;</span>,<span>&quot;Mitenka&quot;</span>,
        <span>&quot;Berg&quot;</span>, <span>&quot;Bourienne&quot;</span>, <span>&quot;Lorrain&quot;</span>, <span>&quot;Michael Ivánovich&quot;</span>,
        <span>&quot;Timókhin&quot;</span>,<span>&quot;Kozlovski&quot;</span>, <span>&quot;Nesvítski&quot;</span>, <span>&quot;Kirsten&quot;</span>,  <span>&quot;Bilibin&quot;</span>,  <span>&quot;Bagration&quot;</span>,
        <span>&quot;Murat&quot;</span>,  <span>&quot;Tushin&quot;</span>,<span>&quot;Alpatych&quot;</span>)
</pre>
</div>
<p>Next, we find the rows in the dataset where each of the characters appeared.</p>
<div>
<pre><span># Create an initial data frame</span>
wpBase <span>&lt;-</span> warpeace[grep(<span>&quot;Mack&quot;</span>,warpeace<span>$</span>V5),]
wpBase<span>$</span>character <span>&lt;-</span> <span>&quot;Mack&quot;</span>

<span># For each character in the book, find the rows with the specific book character</span>
<span>for</span> (person <span>in</span> bookCharacters) {
  wpNew <span>&lt;-</span> warpeace[grep(person,warpeace<span>$</span>V5),]
  <span>if</span> ( nrow(wpNew) <span>></span> <span>0</span> ) {
    wpNew<span>$</span>character <span>&lt;-</span> person
    wpBase <span>&lt;-</span> rbind(wpBase,wpNew)
  }
}
</pre>
</div>
<p>In the book, a character might appear with different names. For example, Natalie and Natsha refer to the same person (depending on whether it is referred to in Russian or French). We attempt to resolve these same-name character references as follows:</p>
<div>
<pre><span># Resolve to same-name references</span>
wpBase<span>$</span>character[wpBase<span>$</span>character<span>==</span><span>&quot;NatÃ¡sha&quot;</span>] <span>&lt;-</span> <span>&quot;Natasha&quot;</span>
wpBase<span>$</span>character[wpBase<span>$</span>character<span>==</span><span>&quot;Natalie&quot;</span>] <span>&lt;-</span> <span>&quot;Natasha&quot;</span>

wpBase<span>$</span>character[wpBase<span>$</span>character<span>==</span><span>&quot;Monsieur Pierre&quot;</span> ] <span>&lt;-</span> <span>&quot;Pierre&quot;</span>
wpBase<span>$</span>character[wpBase<span>$</span>character<span>==</span><span>&quot;Count BezÃºkhov&quot;</span>] <span>&lt;-</span> <span>&quot;Pierre&quot;</span>

wpBase<span>$</span>character[wpBase<span>$</span>character<span>==</span><span>&quot;Prince Andrew&quot;</span>] <span>&lt;-</span> <span>&quot;Andrey&quot;</span>
wpBase<span>$</span>character[wpBase<span>$</span>character<span>==</span><span>&quot;Andrew&quot;</span>] <span>&lt;-</span> <span>&quot;Andrey&quot;</span>
wpBase<span>$</span>character[wpBase<span>$</span>character<span>==</span><span>&quot;Andrey&quot;</span>] <span>&lt;-</span> <span>&quot;Andrey&quot;</span>
</pre>
</div>
<p>Finally, we aggregate the sentiments of the characters by the book they appeared in, and then format it so that it can be used as inputs to d3heatmap. We used various useful functions, including melt and cast from the <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/reshape/">reshape package</a>.</p>
<div>
<pre><span># Aggregate by character, and then order by book</span>
warpeaceG <span>&lt;-</span> wpBase <span>%>%</span> group_by(V2,character) <span>%>%</span> summarize(sentimentOverall<span>=</span>sum(V6))
warpeaceOrderByBook <span>&lt;-</span> warpeaceG[order(warpeaceG<span>$</span>V2),]

md <span>&lt;-</span> melt(warpeaceOrderByBook, id <span>=</span> (c(<span>&quot;V2&quot;</span>, <span>&quot;character&quot;</span>)))
castd <span>&lt;-</span> cast(md, character <span>~</span> V2)

<span># Replace all NA with 0</span>
castd[is.na(castd)] <span>&lt;-</span> <span>0</span>
</pre>
</div>
<p>This reshapes the data from</p>
<p><a target="_blank" rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb0958b8fa970d-pi"><img title="Warpeace2" alt="Warpeace2" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb0958b8fa970d-800wi" border="0"></a></p>
<p>to produce the following:</p>
<p><a target="_blank" rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d23f777b970c-pi"><img title="Warpeace3" alt="Warpeace3" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d23f777b970c-800wi" border="0"></a></p>
<p>This is then used as inputs to d3heatmap to produce the War and Peace character sentiment visualization.</p>
<div>
<pre>d3heatmap(castd,scale <span>=</span> <span>&quot;column&quot;</span>, colors <span>=</span> <span>&quot;Spectral&quot;</span>,
          dendrogram <span>=</span> <span>&quot;none&quot;</span>, Rowv <span>=</span> <span>FALSE</span>, Colv <span>=</span> <span>FALSE</span>)
</pre>
</div>
<p>This produces the following heatmap.</p>
<p><a target="_blank" rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c8b5aa31970b-pi"><img title="Warpeace4" alt="Warpeace4" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c8b5aa31970b-800wi" border="0"></a></p>
<p>I can now use the same approach to gain insights into any book, and understand which chapters of a novel I want to read!</p>
<p>Resources</p>
<ul>
<li><a target="_blank" rel="nofollow" href="https://blogs.msdn.microsoft.com/azuredatalake/2016/11/22/u-sql-advanced-analytics-introducing-cognitive-scenarios-for-text-and-imaging/">Find out</a> how to use U-SQL in Azure Data Lake Analytics to detect key phrases.</li>
<li><a target="_blank" rel="nofollow" href="https://machinelearningdatasets.blob.core.windows.net/public/warandpeace/warandpeaceSentimentKeyPhrase.csv">Download</a> the dataset here.</li>
<li>You can see this in action in <a target="_blank" rel="nofollow" href="https://channel9.msdn.com/Events/Machine-Learning-and-Data-Sciences-Conference/Data-Science-Summit-2016/K001">Joseph Sirosh's keynote presentation at the Data Science Summit</a>, where Matt Winkler demonstrated using how you can use Azure Data Lake Analytics to perform key phrase extractions and sentiment analysis at scale, over the entire collection of War and Peace books.</li>
</ul>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/using-r-to-gain-insights-into-the-emotional-journeys-in-war-and-peace/"></div></div><div><a data-url="https://www.r-bloggers.com/using-r-to-gain-insights-into-the-emotional-journeys-in-war-and-peace/" data-counturl="https://www.r-bloggers.com/using-r-to-gain-insights-into-the-emotional-journeys-in-war-and-peace/" data-count="vertical" data-text="Using R to Gain Insights into the Emotional Journeys in War and Peace" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2016/12/war-and-peace.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/Oqs79fFGIJo"></a></div>