<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/r%e2%81%b6-using-pandoc-from-r-a-neat-package-for-reading-subtitles/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/04/30/r%E2%81%B6-using-pandoc-from-r-a-neat-package-for-reading-subtitles/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Once I realized that my planned, larger post would not come to fruition today I took the R⁶ post (i.e. “minimal expository, keen focus) route, prompted by a Twitter discussion with some R mates who needed to convert “lightly formatted” Microsoft Word (<code>docx</code>) documents to markdown. Something like this:</p>
<p><img alt="" class="wp-image-5855" src="https://i1.wp.com/rud.is/b/wp-content/uploads/2017/04/simple.png?resize=468%2C371&ssl=1" data-recalc-dims="1"></p>
<p>to:</p>
<pre><code>Does pandoc work?
=================

Simple document with **bold** and *italics*.</code></pre>
<p>This is definitely a job that <a target="_blank" rel="nofollow" href="http://pandoc.org/"><code>pandoc</code></a> can handle.</p>
<p><code>pandoc</code> is a <a target="_blank" rel="nofollow" href="https://www.haskell.org/">Haskell</a> (yes, Haskell) program created by <a target="_blank" rel="nofollow" href="http://johnmacfarlane.net/">John MacFarlane</a> and is an amazing tool for transcoding documents. And, if you’re a “modern” R/RStudio user, you likely use it every day because it’s ultimately what powers <code>rmarkdown</code> / <code>knitr</code>. </p>
<p>Yes, you read that correctly. You’re beautiful PDF, Word and HTML R reports are powered by — and, would not be possible without — Haskell.</p>
<p>Doing the aforementioned conversion from <code>docx</code> to markdown is super-simple from R:</p>
<pre><code>rmarkdown::pandoc_convert(&quot;simple.docx&quot;, &quot;markdown&quot;, output=&quot;simple.md&quot;)</code></pre>
<p>Give the help on <code>rmarkdown::pandoc_convert()</code> a read as well as the very thorough and helpful documentation over at [<code>pandoc.org</code>])(http://pandoc.org) to see the power available at your command.</p>
<h3>Just One More Thing</h3>
<p>This section — technically — violates the R⁶ principle so you can stop reading if you’re a purist <img alt="🙂" class="wp-smiley" src="https://s.w.org/images/core/emoji/2/72x72/1f642.png"></p>
<p>There’s a neat, non-on-CRAN package by François Keck called <code>subtools</code> — <a target="_blank" rel="nofollow" href="https://github.com/fkeck/subtools">https://github.com/fkeck/subtools</a> which can slice, dice and reformat digital content subtitles. There are <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Comparison_of_video_player_software#Subtitle_ability">multiple formats</a> for these subtitle files and it seems to be able to handle them all. </p>
<p>There was a post (earlier in April) about <a target="_blank" rel="nofollow" href="http://datameetsmedia.com/ranking-the-negativity-of-black-mirror-episodes-with-sentiment-analysis/">Ranking the Negativity of Black Mirror Episodes</a>. That post is python and I’ve never had time to fully replicate it in R. </p>
<p>Here’s a snippet (sans expository) that can get you started pulling in subtitles into R and <code>tidytext</code>. I would have written scraper code but the various subtitle aggregation sites make that a task suited for something like my <a target="_blank" rel="nofollow" href="https://github.com/hrbrmstr/splashr"><code>splashr</code></a> package and I just had no cycles to write the code. So, I grabbed the first season of “The Flash” and use the Bing sentiment lexicon from <code>tidytext</code> to see how the season looked.</p>
<p>The overall scoring for a given episode is naive and can definitely be improved upon.</p>
<p>Definitely drop a link to anything you create in the comments!</p>
<pre><code># devtools::install_github(&quot;fkeck/subtools&quot;)

library(subtools)
library(tidytext)
library(hrbrthemes)
library(tidyverse)

data(stop_words)

bing &lt;- get_sentiments(&quot;bing&quot;)
afinn &lt;- get_sentiments(&quot;afinn&quot;)

fils &lt;- list.files(&quot;flash/01&quot;, pattern = &quot;srt$&quot;, full.names = TRUE)

pb &lt;- progress_estimated(length(fils))

map_df(1:length(fils), ~{

  pb$tick()$print()

  read.subtitles(fils[.x]) %>%
    sentencify() %>%
    .$subtitles %>%
    unnest_tokens(word, Text) %>%
    anti_join(stop_words, by=&quot;word&quot;) %>%
    inner_join(bing, by=&quot;word&quot;) %>%
    inner_join(afinn, by=&quot;word&quot;) %>%
    mutate(season = 1, ep = .x)

}) %>% as_tibble() -> season_sentiments


count(season_sentiments, ep, sentiment) %>%
  mutate(pct = n/sum(n),
         pct = ifelse(sentiment == &quot;negative&quot;, -pct, pct)) -> bing_sent

ggplot() +
  geom_ribbon(data = filter(bing_sent, sentiment==&quot;positive&quot;),
              aes(ep, ymin=0, ymax=pct, fill=sentiment), alpha=3/4) +
  geom_ribbon(data = filter(bing_sent, sentiment==&quot;negative&quot;),
              aes(ep, ymin=0, ymax=pct, fill=sentiment), alpha=3/4) +
  scale_x_continuous(expand=c(0,0.5), breaks=seq(1, 23, 2)) +
  scale_y_continuous(expand=c(0,0), limits=c(-1,1),
                     labels=c(&quot;100%\nnegative&quot;, &quot;50%&quot;, &quot;0&quot;, &quot;50%&quot;, &quot;positive\n100%&quot;)) +
  labs(x=&quot;Season 1 Episode&quot;, y=NULL, title=&quot;The Flash — Season 1&quot;,
       subtitle=&quot;Sentiment balance per episode&quot;) +
  scale_fill_ipsum(name=&quot;Sentiment&quot;) +
  guides(fill = guide_legend(reverse=TRUE)) +
  theme_ipsum_rc(grid=&quot;Y&quot;) +
  theme(axis.text.y=element_text(vjust=c(0, 0.5, 0.5, 0.5, 1)))</code></pre>
<p><a target="_blank" rel="nofollow" href="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/04/flash.png?ssl=1"><img alt="" class="wp-image-5866" src="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/04/flash.png?resize=780%2C418&ssl=1" data-recalc-dims="1">
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2017/04/30/r%E2%81%B6-using-pandoc-from-r-a-neat-package-for-reading-subtitles/"> R – rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/YIL6oBMu9R4"></a></div></a>