<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/movies-and-series-subtitles-in-r-with-subtools/"></div></div><div><a data-url="https://www.r-bloggers.com/movies-and-series-subtitles-in-r-with-subtools/" data-counturl="https://www.r-bloggers.com/movies-and-series-subtitles-in-r-with-subtools/" data-count="horizontal" data-text="Movies and Series subtitles in R with subtools" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.pieceofk.fr/?p=437"> R_EN – Piece of K</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Every time I download subtitles for a movie or a series episode, I cannot help thinking about all this text that could be analyzed with text mining methods. As my PhD came to its end in April and I started a postdoc in September, I could use some time of my <em>looong</em> summer break to work on a new R package, <code>subtools</code> which aims to provide a toolbox to read, manipulate and write subtitles files in R.</p>
<p>In this post I will present briefly the functions of <code>subtools</code>. For more details you can check the documentation.</p>
<h2>Installation</h2>
<p>The package is available on <a target="_blank" href="https://github.com/fkeck/subtools">GitHub</a>.<br>
You can install it using the devtools package:</p>
<pre>devtools::install_github(&quot;fkeck/subtools&quot;)</pre>
<h2>Read subtitles</h2>
<p>A subtitles file can be read directly from R with the function <code>read.subtitles</code>. Currently, four formats of subtitles are supported: SubRip, (Advanced) Substation Alpha, MicroDVD and SubViewer. The parsers are probably not optimal but they seem to do the job as expected, at least with valid files. The package also provides some wrappers to import whole directories of series subtitles (see <code>read.subtitles.season</code>, <code>read.subtitles.serie</code> and <code>read.subtitles.multiseries</code>)</p>
<h2>Manipulate subtitles</h2>
<p>The <code>subtools</code> package stores imported subtitles as simple S3 objects of class <code>Subtitles</code>. They are lists with two main components : subtitles (IDs, timecodes and texts) and optional meta-data. Multiple Subtitles objects can be stored as a list of class <code>MultiSubtitles</code>. The subtools package provides functions to easily manipulate <code>Subtitles</code> and <code>MultiSubtitles</code> objects.</p>
<p>Basically, you can :</p>
<ul>
<li>combine subtitles objects with <code>combineSubs</code></li>
<li>extract parts of subtitles with <code>[.Subtitles</code></li>
<li>clean subtitles content with <code>cleanTags</code>, <code>cleanCaptions</code> and <code>cleanPatterns</code></li>
<li>reorganize subtitles as sentences with <code>sentencify</code></li>
</ul>
<h2>Extract/convert/write subtitles</h2>
<p>Although you can conduct statistical analyses on subtitles objects, the package subtools is not designed for text mining. The following functions allow you to convert subtitles to other classes/format to analyze them. You can:</p>
<ul>
<li>extract text content as a simple character string with <code>rawText</code></li>
<li>convert subtitles and meta-data to a virtual corpus with <code>tmCorpus</code> if you want to work with the standard text mining framework <a target="_blank" href="https://rstudio-pubs-static.s3.amazonaws.com/31867_8236987cf0a8444e962ccd2aec46d9c3.html">tm</a>.</li>
<li>convert subtitles and meta-data to a data.frame with <code>subDataFrame</code>. If you want to use tidy data principles with <a target="_blank" href="https://cran.r-project.org/web/packages/tidytext/vignettes/tidytext.html">tidytext</a> and <a target="_blank" href="https://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html">dplyr</a>, you should probably start here.</li>
<li>finally, it’s also possible to write subtitles objects to a file with <code>write.subtitles</code>. Though, it is unclear to me if there is any sense in doing that <img alt="😀" class="wp-smiley" src="https://i2.wp.com/s.w.org/images/core/emoji/2/72x72/1f600.png?w=456&ssl=1" data-recalc-dims="1"></li>
</ul>
<h2>Application: Game of Thrones subtitles wordcloud</h2>
<p>To illustrate how subtools can be used to get started with a subtitles analysis project, I propose to create a wordcloud showing the most frequent words in the popular TV series Game of Thrones. We will use subtools to import the subtitles, the <code>tm</code> and <code>SnowballC</code> packages to pre-process the data and finally <code>wordcloud</code> to generate the cloud. I will not provide the data I use here, because subtitles files are in a grey zone concerning licensing. But no worries, it’s pretty easy to find subtitles on Internet. <img alt="😉" class="wp-smiley" src="https://i2.wp.com/s.w.org/images/core/emoji/2/72x72/1f609.png?w=456&ssl=1" data-recalc-dims="1"></p>
<pre>library(subtools)
library(tm)
library(SnowballC)
library(wordcloud)</pre>
<p>Because the subtitles are correctly organized in directories, we import them in one command line using the function <code>read.subtitles.serie</code>. The nice thing is that this function will try to automatically extract basic meta-data (like series title, season number and episode number) from directories/files name.</p>
<pre>a &lt;- read.subtitles.serie(dir = &quot;/subs/Game of Thrones/&quot;)</pre>
<p><code>a</code> is a <code>MultiSubtitles</code> object with 60 Subtitles elements (episodes). We can convert it directly to a tm corpus using tmCorpus. Note that meta-data are preserved.</p>
<pre>c &lt;- tmCorpus(a)</pre>
<p>And then, we can prepare the data:</p>
<pre>
c &lt;- tm_map(c, content_transformer(tolower))
c &lt;- tm_map(c, removePunctuation)
c &lt;- tm_map(c, removeNumbers)
c &lt;- tm_map(c, removeWords, stopwords(&quot;english&quot;))
c &lt;- tm_map(c, stripWhitespace)</pre>
<p>Compute a term-document matrix and aggregate counts by season:</p>
<pre>
TDM &lt;- TermDocumentMatrix(c)
TDM &lt;- as.matrix(TDM)
vec.season &lt;- rep(1:6, each = 10)
TDM.season &lt;- t(apply(TDM, 1, function(x) tapply(x, vec.season, sum)))
colnames(TDM.season) &lt;- paste(&quot;S&quot;, 1:6)</pre>
<p>And finally plot the cloud!</p>
<pre>set.seed(100)
comparison.cloud(TDM.season, title.size = 1, max.words = 100, random.order = T)</pre>
<figure class="wp-caption"><a target="_blank" rel="attachment wp-att-467" href="http://www.pieceofk.fr/?attachment_id=467"><img sizes="(max-width: 594px) 100vw, 594px" alt="Subtitles wordcloud of the six seasons of Game of Thrones." class="wp-image-467" src="https://i1.wp.com/www.pieceofk.fr/wp-content/uploads/2016/10/got_wordcloud.png?resize=456%2C468" data-recalc-dims="1"></a><br>
<figcaption class="wp-caption-text">Subtitles wordcloud of the six seasons of Game of Thrones.</figcaption>
</figure>
<p>Few words about this plot. Like every wordcloud, I think it’s a very simple and limited descriptive way to represent the information. However, I like it. The people who have watched the TV show will look at it and say « Oh of course! ». In one hundred word, the cloud is not revealing the scenario of GoT, but for each season I can see one or two critical events popping out (wedding, kill/joffrey, queen/shame, hodor/hold/door).<br>
What I find funny here (perhaps interesting?), is that these very important and emotional moments are supported in the dialogs by the repetition of one or two keywords. I don’t know if this is exclusive to GoT, or if it’s a trick of my mind, or something else. And I will not make any hypothesis, I’m not a linguist. But this is the first idea which came to my mind and I wanted to write it down.</p>
<p>Now, there are plenty of text-mining ideas and hypotheses and methods that can be tested with movies and series subtitles. So have fun <img alt="🙂" class="wp-smiley" src="https://i0.wp.com/s.w.org/images/core/emoji/2/72x72/1f642.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p><a title="Share on Facebook" data-provider="facebook" target="_blank" rel="nofollow" href="http://www.facebook.com/sharer.php?u=http%3A%2F%2Fwww.pieceofk.fr%2F%3Fp%3D437&t=Movies%20and%20Series%20subtitles%20in%20R%20with%20subtools&%23038;s=100&%23038;p&%23091;url&%23093;=http%3A%2F%2Fwww.pieceofk.fr%2F%3Fp%3D437&%23038;p&%23091;images&%23093;&%23091;0&%23093;=http%3A%2F%2Fwww.pieceofk.fr%2Fwp-content%2Fuploads%2F2016%2F10%2Fgot_wordcloud.png&%23038;p&%23091;title&%23093;=Movies%20and%20Series%20subtitles%20in%20R%20with%20subtools"><img title="Share on Facebook" alt="Facebook" src="https://i0.wp.com/www.pieceofk.fr/wp-content/plugins/social-media-feather/synved-social/image/social/regular/32x32/facebook.png?resize=16%2C16" data-recalc-dims="1"></a><a title="Share on Twitter" data-provider="twitter" target="_blank" rel="nofollow" href="https://twitter.com/intent/tweet?url=http%3A%2F%2Fwww.pieceofk.fr%2F%3Fp%3D437&text=Hey%20check%20this%20out"><img title="Share on Twitter" alt="twitter" src="https://i0.wp.com/www.pieceofk.fr/wp-content/plugins/social-media-feather/synved-social/image/social/regular/32x32/twitter.png?resize=16%2C16" data-recalc-dims="1"></a></p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/movies-and-series-subtitles-in-r-with-subtools/"></div></div><div><a data-url="https://www.r-bloggers.com/movies-and-series-subtitles-in-r-with-subtools/" data-counturl="https://www.r-bloggers.com/movies-and-series-subtitles-in-r-with-subtools/" data-count="vertical" data-text="Movies and Series subtitles in R with subtools" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.pieceofk.fr/?p=437"> R_EN – Piece of K</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/pfA8gOhfTmo"></a></div>