<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/nlp-on-nprs-commencement-addresses/"></div></div><div><a data-url="http://www.r-bloggers.com/nlp-on-nprs-commencement-addresses/" data-counturl="http://www.r-bloggers.com/nlp-on-nprs-commencement-addresses/" data-count="horizontal" data-text="NLP on NPR’s Commencement Addresses" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://data-steve.github.io/nlp-on-commencement-addresses/"> data_steve</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img alt="" src="http://i1.wp.com/data-steve.github.io/images/1*z2iYrZI9xUMoBuIKVO_wQg.png?w=456" data-recalc-dims="1"></p>
<p>Vocativ did an interesting <a target="_blank" href="http://www.vocativ.com/interactive/usa/us-politics/presidential-readability/">analysis of the President’s State of the Union (SOTU)<br>
speeches</a>. They showed that across the past couple hundred years and<br>
many Presidents, SOTU speeches have been targeted at audiences with<br>
lower and lower education levels.</p>
<p>Vocativ’s in-print interpretation of the downward sloping trend was that a speeches have gotten<br>
less sophisticated. Their recommended share-tweet for the article was more blunt:</p>
<p><img alt="" src="http://i1.wp.com/data-steve.github.io/images/share-tweet.png?w=456" data-recalc-dims="1"></p>
<p>We think there could be another interpretation, a growing populist bent in US culture that<br>
influenced even how POTUS did the SOTU.</p>
<h3>Idea</h3>
<p><a target="_blank" href="http://data-steve.github.io/nlp-on-commencement-addresses/www.github.com/trinker">Tyler</a> and I wanted to test for that broad cultural influence in other formal<br>
addresses. But what other kinds of speeches happen as regularly and<br>
consistently as SOTU? And then it struck us: commencement speeches! And<br>
what better test?</p>
<p>If commencement speeches are always targeted at an on-average relatively more<br>
educated audience of college graduates (and their families), then if we<br>
still see the same trend in language, this would be a fairly decent test<br>
that a cultural shift was influencing public address in general.</p>
<p><strong>In the following series of posts</strong>, I’ll share the methods, data, and<br>
insights we found. We not only replicate Vocativ’s analysis, we extend<br>
it in several interesting ways, using different types of text analysis<br>
and a view inspired views of the data.</p>
<h5>Methods</h5>
<p>Commencing with Commencements.</p>
<p>Since the Vocativ article uses a <a target="_blank" href="https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests">common grade-level readability formula</a><br>
to measure the education level of the audience, all we really needed was<br>
to scrape some commencement addresses and apply the same readability<br>
algorithm. Also, we thought we’d extend the methodology by using several<br>
readability measures averaged together for robustness.</p>
<h5>Data</h5>
<p>Luckily for us, NPR had built a great web app to collect as many<br>
<a target="_blank" href="http://apps.npr.org/commencement/">commencement addresses</a> as they<br>
could readily find: 350+ speeches, in fact. So, with a bit of<br>
BeautifulSoup code (more on that to come!), we were able to scrape the<br>
website for many of the cool insights that data and portal afford.</p>
<h5>First Results</h5>
<p>But not to leave you hanging until our next post: when we compare our<br>
results (first graph above) to the interactive visualization that Vocativ produced, we do in<br>
fact find the same relative downward trend among the 280 usable<br>
speeches, from 12th grade level in the 1950s to 8th grade level in 2014.<br>
In this time series plot, we aggregate speeches by year, and scatter<br>
their occurrence with a trend line and bivariate correlation measure.</p>
<p>The negative time trend is not proof-positive of the populist<br>
interpretation, as opposed to a more curmudgeonly view on American<br>
education; but it is at least a stronger test than the original on new<br>
data and independent contexts. Finding the same results generalize<br>
across these instances, pretty much exactly, does say something for the<br>
populist interpretation.</p>
<h3>The Code</h3>
<p>We provide the code and data to reproduce the longitudinal readability<br>
graph below. The data provided is the output from the <em>readability</em><br>
function in the <a target="_blank" href="https://github.com/trinker/readability">GitHub <strong>readability</strong> package</a>, not the raw text (we<br>
intend to release the full text based data set at the end of the blog<br>
series).</p>
<h4>Load Packages &amp; Data</h4>
<div>
<pre><code>if (!require('pacman')) install.packages(&quot;pacman&quot;)
pacman::p_load_gh(&quot;trinker/syllable&quot;, &quot;trinker/readability&quot;)
pacman::p_load(dplyr, qdap, ggplot2, RCurl, readr, foreign)


## read in readability data
readability_scores_by_id_year &lt;-
&quot;https://raw.githubusercontent.com/trinker/data_store/master/readability_scores_by_id_year.csv&quot; %>%
    getURL() %>%
    textConnection() %>% 
    read.csv() %>% 
    tbl_df()
</code></pre>
</div>
<p><img alt="" src="http://i2.wp.com/data-steve.github.io/images/1*qzqFK7mzzavPMOF9V-nOeg.png?w=456" data-recalc-dims="1"></p>
<p>The output shows the <em>speech id</em>, <em>year</em>, <em>5 readability scores</em><br>
(<a target="_blank" href="https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests">Flesch Kincaid</a>,<br>
<a target="_blank" href="https://en.wikipedia.org/wiki/Gunning_fog_index">Gunning Fog Index</a>,<br>
<a target="_blank" href="https://en.wikipedia.org/wiki/Coleman%E2%80%93Liau_index">Coleman Liau</a>,<br>
<a target="_blank" href="https://en.wikipedia.org/wiki/SMOG">SMOG</a>, &amp; <a target="_blank" href="https://en.wikipedia.org/wiki/Automated_readability_index">Automated Readability Index</a>), and<br>
an <em>aggregate readability score</em>. (Vocativ uses the first of the above<br>
five scores.) Readability scores are generally, computed from a<br>
combination of simple counts and ratios of a of characters, syllables,<br>
words, and sentences. The gist is that complex words and sentences make<br>
the comprehension of these texts more difficult.</p>
<h5>Munging and Visualizing</h5>
<p>To interpret the data we created the <em>changes in speech readability over<br>
time visualization</em>, using an aggregate of average grade level<br>
readability scores. Below, the <strong>dplyr</strong> &amp; <strong>ggplot</strong> packages make<br>
light work of the data munging and plotting task.</p>
<div>
<pre><code>readability_scores_by_id_year %>%
    group_by(Year) %>%
    summarize(
        Readability = mean(Average_Grade_Level),
        n = n()
    ) %>%
    ggplot() +
        geom_smooth(aes(x=Year, y=Readability), fill=NA, size=1,
            method='lm', color='blue') +
        geom_point(aes(x=Year, y=Readability, size=n), alpha=.3) + 
        geom_point(aes(x=Year, y=Readability), size=1) +
        scale_size_continuous(range = c(3,10), name='N Speeches') +
        theme_bw() +
        theme(
            panel.grid.major.x=element_blank(),
            panel.grid.minor.x=element_blank(), 
            panel.border = element_blank(), 
            axis.line = element_line(color=&quot;grey50&quot;) 
        ) +
        scale_x_continuous(
            expand = c(.01, 0), 
            limits = c(1955, 2015), 
            breaks = seq(1955, 2015, by=5)
        ) +
        ggtitle('Average Yearly Readabilitynof Speeches Over Time') +
        annotate('text', x = 1962, y = 10.7, 
            label = 'italic(r[xy] == -.299)', 
            parse = TRUE, color='blue'
        )+
        ylab('Grade Level')
</code></pre>
</div>
<h3>To Come in Future Posts</h3>
<p>Next we hope to post about choices we made for scraping websites:<br>
specifically, how Python’s BeautifulSoup faired and what it was like to<br>
run the project on Jupyter/iPython Notebook vs an IDE.</p>
<p>In the meanwhile, go check out Tyler’s readability package and github star it for doing such a great job. Follow the button below.</p>
<p><a target="_blank" class="twitter-share-button" data-via="data_steve" data-size="large" data-hashtags="rstats, textanalytics,NLP" data-dnt="true" href="https://twitter.com/share">Tweet</a><a data-style="mega" target="_blank" href="https://github.com/trinker/readability" data-icon="octicon-star">readability</a></p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/nlp-on-nprs-commencement-addresses/"></div></div><div><a data-url="http://www.r-bloggers.com/nlp-on-nprs-commencement-addresses/" data-counturl="http://www.r-bloggers.com/nlp-on-nprs-commencement-addresses/" data-count="vertical" data-text="NLP on NPR’s Commencement Addresses" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://data-steve.github.io/nlp-on-commencement-addresses/"> data_steve</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/Qtki1PmA5YQ"></a></div>