<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/fivethirtyeights-polling-data-for-the-us-presidential-election/"></div></div><div><a data-url="https://www.r-bloggers.com/fivethirtyeights-polling-data-for-the-us-presidential-election/" data-counturl="https://www.r-bloggers.com/fivethirtyeights-polling-data-for-the-us-presidential-election/" data-count="horizontal" data-text="FiveThirtyEight’s polling data for the US Presidential election" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://ellisp.github.io/blog/2016/10/29/538-pollsters"> Peter's stats stuff - R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h2>3,000+ voting intention surveys</h2>
<p>Like many others around the world, I have been watching with interest the democratic process in the United States of America. One of the most influential and watched websites is <a target="_blank" rel="nofollow" href="http://projects.fivethirtyeight.com/2016-election-forecast/?ex_cid=rrpromo#plus">FiveThirtyEight.com</a>, headed by Nate Silver, author of the excellent popularisation of the craft of statistical time series forecasting <a target="_blank" rel="nofollow" href="https://www.amazon.com/Signal-Noise-Many-Predictions-Fail-but/dp/0143125087">The Signal and the Noise</a>. I have no intention of adding substantive commentary to the crowded space of analysis of US voting behaviour (some of which, and not just from FiveThirtyEight, is extremely high quality BTW), but I wanted at least poke around the data. In particular I’m interested in who is doing all these polls, and what does FiveThirtyEight do to the numbers before putting them into their forecasting models.</p>
<p>In pale grey text at the bottom of the election prediction page, FiveThirytyEight have a link to “Download CSV of polls”.</p>
<p>There are more than 9,000 (at the time of writing) rows in this spreadsheet, but a quick look shows that there are only 3,067 unique values of the <code>poll_id</code> field. Each poll is listed three times in this dataset: once each for “polls-only”, “polls-plus” and “now-cast”. Here’s R code for downloading the data and a first quick look at it:</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tidyr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>ggplot2</span><span>)</span><span>
</span><span>library</span><span>(</span><span>viridis</span><span>)</span><span>
</span><span>library</span><span>(</span><span>ggthemes</span><span>)</span><span>
</span><span># this next code will need to be adapted if you don't have a ../data/ folder...
# alternatively, if that link stops working, there's a static copy at http://ellisp.github.io/data/polls.csv
</span><span>www</span><span> </span><span>&lt;-</span><span> </span><span>&quot;http://projects.fivethirtyeight.com/general-model/president_general_polls_2016.csv&quot;</span><span>
</span><span>download.file</span><span>(</span><span>www</span><span>,</span><span> </span><span>destfile</span><span> </span><span>=</span><span> </span><span>&quot;../data/polls.csv&quot;</span><span>)</span><span>
</span><span># download data
</span><span>polls_orig</span><span> </span><span>&lt;-</span><span> </span><span>read.csv</span><span>(</span><span>&quot;../data/polls.csv&quot;</span><span>,</span><span> </span><span>stringsAsFactors</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>table</span><span>(</span><span>table</span><span>(</span><span>polls_orig</span><span>$</span><span>poll_id</span><span>))</span><span>
</span><span>## 3 ## 3067 </span><span>
</span><span>table</span><span>(</span><span>polls_orig</span><span>$</span><span>type</span><span>)</span><span>
</span><span>## now-cast polls-only polls-plus </span><span>##</span><span> </span><span>3067</span><span> </span><span>3067</span><span> </span><span>3067</span><span> </span></code></pre>
</figure>
<h2>Poll weight?</h2>
<p>The polling data has a a <code>poll_wt</code> field. At first I thought this would estimate the value of the poll based on the grade of its provider’s quality (they are all rated from D to A+) and sample size, but it turns out to be more complicated than that, and to be based largely on how long before the election the data was collected. The values for polls-only and polls-plus (both of which forecast the election on its actual date) are identical; the now-cast weights are slightly higher. Sample size and pollster grade also contribute to the determination of <code>poll_wt</code>. Many of the older surveys have a <code>poll_wt</code> of pretty much zero, meaning they are not being used at all in estimating current predictions. All this can be seen in the graphic below.</p>
<p><img alt="weights" src="http://ellisp.github.io/img/0062-weights.svg"></p>
<p>That graphic was created with this code:</p>
<figure>
<pre><code data-lang="r"><span>grades</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;A+&quot;</span><span>,</span><span> </span><span>&quot;A&quot;</span><span>,</span><span> </span><span>&quot;A-&quot;</span><span>,</span><span> </span><span>&quot;B+&quot;</span><span>,</span><span> </span><span>&quot;B&quot;</span><span>,</span><span> </span><span>&quot;B-&quot;</span><span>,</span><span> </span><span>&quot;C+&quot;</span><span>,</span><span> </span><span>&quot;C&quot;</span><span>,</span><span> </span><span>&quot;C-&quot;</span><span>,</span><span> </span><span>&quot;D&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>)</span><span>
</span><span># parse the dates as proper dates, and add structure to the grade
</span><span>polls</span><span> </span><span>&lt;-</span><span> </span><span>polls_orig</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>
</span><span>startdate</span><span> </span><span>=</span><span> </span><span>as.Date</span><span>(</span><span>startdate</span><span>,</span><span> </span><span>&quot;%m/%d/%Y&quot;</span><span>),</span><span>
</span><span>enddate</span><span> </span><span>=</span><span> </span><span>as.Date</span><span>(</span><span>enddate</span><span>,</span><span> </span><span>&quot;%m/%d/%Y&quot;</span><span>),</span><span>
</span><span>grade</span><span> </span><span>=</span><span> </span><span>ordered</span><span>(</span><span>grade</span><span>,</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>grades</span><span>)</span><span>
</span><span>)</span><span>
</span><span>polls</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>enddate</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>poll_wt</span><span>,</span><span> </span><span>colour</span><span> </span><span>=</span><span> </span><span>type</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>samplesize</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_y_sqrt</span><span>()</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>~</span><span>grade</span><span>)</span></code></pre>
</figure>
<p>For the rest of this post I’ll be focusing on the “polls-only” data.</p>
<h2>Raw versus adjusted?</h2>
<p>Each poll has two sets of percentages for each of four presidential candidates (Clinton, Trump, McMullin and Johnson – but McMullin is often missing). The two sets are labelled “raw” and “adj”. My understanding of the FiveThirtyEight approach is that the “adj” value reflects calibration for historical statistical bias of the individual polls (ie if they consistently have overestimated party X in the past, their reported proportion for party X is adjusted downwards). This is perfectly sound and standard, and it’s good that both the adjusted and unadjusted data are made available. Neither the raw nor adjusted series add up to 100, presumably because of undecided or otherwise non-informative respondents.</p>
<p>Here is a comparison of the raw and adjusted intended vote for Clinton:</p>
<p><img alt="rawadj" src="http://ellisp.github.io/img/0062-raw-adj.svg"></p>
<p>The diagonal line shows parity – points on the line represent surveys where the adjusted value is exactly the same as raw. We see that much of the adjustment for Clinton is upwards. Before any conclusions are drawn from that, let’s check in on the same graphic for Trump:</p>
<p><img alt="rawadj" src="http://ellisp.github.io/img/0062-raw-adj-trump.svg"></p>
<p>We see that Trump’s vote also gets a share of upwards adjustment. Before leaving this, let’s look at the distribution of adjustment factors for them both:</p>
<p><img alt="rawadj" src="http://ellisp.github.io/img/0062-raw-adj-densities.svg"></p>
<p>For both candidates, the most common adjustment is actually very slightly downwards (ie just below 1). Clinton’s estimated intended votes do get adjusted up a little more than do Trump’s. With adjustments based on historical relationships between polls and results, this is the sort of thing to be uncertain about in this far-from-usual election year; but FiveThirtyEight only have history to go on. Certainly in the methodological pieces on their website like <a target="_blank" rel="nofollow" href="http://fivethirtyeight.com/features/election-update-why-our-model-is-more-bullish-than-others-on-trump/">this one</a>, they pay all due respect to this aspect of uncertainty.</p>
<figure>
<pre><code data-lang="r"><span>pollsonly</span><span> </span><span>&lt;-</span><span> </span><span>filter</span><span>(</span><span>polls</span><span>,</span><span> </span><span>type</span><span> </span><span>==</span><span> </span><span>&quot;polls-only&quot;</span><span>)</span><span>
</span><span># compare the raw values with the adjusted values for Clinton
</span><span>pollsonly</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>rawpoll_clinton</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>adjpoll_clinton</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_abline</span><span>(</span><span>intercept</span><span> </span><span>=</span><span> </span><span>0</span><span>,</span><span> </span><span>slope</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>colour</span><span> </span><span>=</span><span> </span><span>&quot;grey75&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.4</span><span>,</span><span> </span><span>aes</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>samplesize</span><span>,</span><span> </span><span>colour</span><span> </span><span>=</span><span> </span><span>grade</span><span>))</span><span> </span><span>+</span><span>
</span><span>scale_color_viridis</span><span>(</span><span>discrete</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>option</span><span> </span><span>=</span><span> </span><span>&quot;magma&quot;</span><span>)</span><span> </span><span>+</span><span> </span><span>guides</span><span>(</span><span>colour</span><span> </span><span>=</span><span> </span><span>guide_legend</span><span>(</span><span>override.aes</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>2</span><span>)))</span><span> </span><span>+</span><span>
</span><span>coord_equal</span><span>()</span><span>
</span><span># For Trump:
</span><span>pollsonly</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>rawpoll_trump</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>adjpoll_trump</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_abline</span><span>(</span><span>intercept</span><span> </span><span>=</span><span> </span><span>0</span><span>,</span><span> </span><span>slope</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>colour</span><span> </span><span>=</span><span> </span><span>&quot;grey75&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.4</span><span>,</span><span> </span><span>aes</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>samplesize</span><span>,</span><span> </span><span>colour</span><span> </span><span>=</span><span> </span><span>grade</span><span>))</span><span> </span><span>+</span><span>
</span><span>scale_color_viridis</span><span>(</span><span>discrete</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>option</span><span> </span><span>=</span><span> </span><span>&quot;magma&quot;</span><span>)</span><span> </span><span>+</span><span> </span><span>guides</span><span>(</span><span>colour</span><span> </span><span>=</span><span> </span><span>guide_legend</span><span>(</span><span>override.aes</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>2</span><span>)))</span><span> </span><span>+</span><span>
</span><span>coord_equal</span><span>()</span><span>
</span><span># Distribution of adjustment factors:
</span><span>pollsonly</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Trump</span><span> </span><span>=</span><span> </span><span>adjpoll_trump</span><span> </span><span>/</span><span> </span><span>rawpoll_trump</span><span>,</span><span>
</span><span>Clinton</span><span> </span><span>=</span><span> </span><span>adjpoll_clinton</span><span> </span><span>/</span><span> </span><span>rawpoll_clinton</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>Trump</span><span>,</span><span> </span><span>Clinton</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>candidate</span><span>,</span><span> </span><span>adj_ratio</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>adj_ratio</span><span>,</span><span> </span><span>colour</span><span> </span><span>=</span><span> </span><span>candidate</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>candidate</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_density</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.3</span><span>)</span><span> </span><span>+</span><span>
</span><span>ggtitle</span><span>(</span><span>&quot;Density of adjustment ratios for two main candidates&quot;</span><span>)</span></code></pre>
</figure>
<h2>Trends over time</h2>
<p>Note that in the graphics above I’m ignoring the regional element of the data – some of the surveys’ target population is the whole USA, some are for specific locations, from 14 for the District of Colombia to 109 for the key state of Florida. This is the main reason for the large variance in estimated intended vote for Senator Clinton when looking at all the polls together.</p>
<p>More interesting is seeing trends in voting intention over time. In the media, these are usually shown as lovely clean time series charts with a single, simple line for each candidate. The reality of the data is much messier of course. Here’s the estimated intended vote for both Clinton and Trump, from just the surveys aiming to estimate voting behaviour of the whole U.S.A., illustrating the raw versus adjusted data:</p>
<p><img alt="clintonprogress" src="http://ellisp.github.io/img/0062-clinton.svg"></p>
<p>I like this graphic because it illustrates several things:</p>
<ul>
<li>the <em>big</em> range of individual survey results. All these surveys are trying to measure the same thing – intended vote in the same election, from the same population. Nothing better illustrates the folly of paying attention to only individual polls than looking at this spread. Pooling them – which is implicitly done by the smoothing blue line – is the way to go.</li>
<li>one result of the adjustment process is to reduce the variance in survey results. For both Clinton and Trump, the adjusted poll results are tighter than the raw data.</li>
<li>the basic story – Clinton is doing better than Trump – isn’t changed by the adjustment process</li>
<li>a cluster of recent large sample size, “B” grade surveys have particularly low raw values for both Clinton and Trump and these are both adjusted upwards. Checking the data, these turn out to be “Google Consumer Surveys”, which are apparently judged by FiveThirtyEight to materially understate the probable vote for major party candidates and overstate the chances of Johnson (Johnson’s percentages in these surveys are typically adjusted down about three percentage points by FiveThirtyEight)</li>
</ul>
<p>It’s useful to think of each pollster’s data as a group of longitudinal results. This suggests a variant of our last graphic, the standard longitudinal spaghetti plot:<br>
<img alt="pollster-lines" src="http://ellisp.github.io/img/0062-pollster-lines.svg"></p>
<figure>
<pre><code data-lang="r"><span># show probability of voting Clinton and Trump over time, just the national polls:
</span><span>pollsonly</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>state</span><span> </span><span>==</span><span> </span><span>&quot;U.S.&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>enddate</span><span>,</span><span> </span><span>rawpoll_clinton</span><span>,</span><span> </span><span>adjpoll_clinton</span><span>,</span><span> </span><span>rawpoll_trump</span><span>,</span><span> </span><span>adjpoll_trump</span><span>,</span><span> </span><span>samplesize</span><span>,</span><span> </span><span>grade</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>variable</span><span>,</span><span> </span><span>value</span><span>,</span><span> </span><span>-</span><span>enddate</span><span>,</span><span> </span><span>-</span><span>samplesize</span><span>,</span><span> </span><span>-</span><span>grade</span><span>)</span><span> </span><span>%>%</span><span> </span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>enddate</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>value</span><span> </span><span>/</span><span> </span><span>100</span><span>))</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>~</span><span>variable</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.4</span><span>,</span><span> </span><span>aes</span><span>(</span><span>colour</span><span> </span><span>=</span><span> </span><span>grade</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>samplesize</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_smooth</span><span>(</span><span>aes</span><span>(</span><span>weight</span><span> </span><span>=</span><span> </span><span>samplesize</span><span>),</span><span> </span><span>span</span><span> </span><span>=</span><span> </span><span>0.5</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_color_viridis</span><span>(</span><span>&quot;Pollster grade&quot;</span><span>,</span><span> </span><span>discrete</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>option</span><span> </span><span>=</span><span> </span><span>&quot;magma&quot;</span><span>)</span><span> </span><span>+</span><span> </span><span>guides</span><span>(</span><span>colour</span><span> </span><span>=</span><span> </span><span>guide_legend</span><span>(</span><span>override.aes</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>2</span><span>)))</span><span> </span><span>+</span><span>
</span><span>scale_y_continuous</span><span>(</span><span>&quot;Percentage of intended vote\n&quot;</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>percent</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_size_area</span><span>(</span><span>&quot;Sample size&quot;</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>comma</span><span>)</span><span> </span><span>+</span><span>
</span><span>ggtitle</span><span>(</span><span>&quot;Intended vote in the US Presidential election&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;National surveys only&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;End date of survey&quot;</span><span>,</span><span>
</span><span>caption</span><span> </span><span>=</span><span> </span><span>&quot;Data compiled by FiveThirtyEight, analysis by http://ellisp.github.io&quot;</span><span>)</span><span>
</span><span># longitudinal version, with lines connecting pollsters:
</span><span>pollsonly</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>state</span><span> </span><span>==</span><span> </span><span>&quot;U.S.&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>enddate</span><span>,</span><span> </span><span>rawpoll_clinton</span><span>,</span><span> </span><span>adjpoll_clinton</span><span>,</span><span> </span><span>rawpoll_trump</span><span>,</span><span> </span><span>adjpoll_trump</span><span>,</span><span> </span><span>samplesize</span><span>,</span><span> </span><span>grade</span><span>,</span><span> </span><span>pollster</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>variable</span><span>,</span><span> </span><span>value</span><span>,</span><span> </span><span>-</span><span>enddate</span><span>,</span><span> </span><span>-</span><span>samplesize</span><span>,</span><span> </span><span>-</span><span>grade</span><span>,</span><span> </span><span>-</span><span>pollster</span><span>)</span><span> </span><span>%>%</span><span> </span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>enddate</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>value</span><span> </span><span>/</span><span> </span><span>100</span><span>,</span><span> </span><span>colour</span><span> </span><span>=</span><span> </span><span>grade</span><span>))</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>~</span><span>variable</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>aes</span><span>(</span><span>group</span><span> </span><span>=</span><span> </span><span>pollster</span><span>))</span><span> </span><span>+</span><span>
</span><span>scale_color_viridis</span><span>(</span><span>&quot;Pollster grade&quot;</span><span>,</span><span> </span><span>discrete</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>option</span><span> </span><span>=</span><span> </span><span>&quot;magma&quot;</span><span>)</span><span> </span><span>+</span><span> </span><span>guides</span><span>(</span><span>colour</span><span> </span><span>=</span><span> </span><span>guide_legend</span><span>(</span><span>override.aes</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>2</span><span>)))</span><span> </span><span>+</span><span>
</span><span>scale_y_continuous</span><span>(</span><span>&quot;Percentage of intended vote\n&quot;</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>percent</span><span>)</span><span> </span><span>+</span><span>
</span><span>ggtitle</span><span>(</span><span>&quot;Intended vote in the US Presidential election&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;National surveys only, grouped by pollster&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;End date of survey&quot;</span><span>,</span><span>
</span><span>caption</span><span> </span><span>=</span><span> </span><span>&quot;Data compiled by FiveThirtyEight, analysis by http://ellisp.github.io&quot;</span><span>)</span></code></pre>
</figure>
<h2>Who are all these pollsters?</h2>
<p>Finally, I wanted a graphic that showed the relative influence of the various pollsters on FiveThirtyEight’s polling forecasts. This takes me back to the <code>poll_wt</code> variable.</p>
<p>Unfortunately visualising all 180 pollsters is a non-trivial problem. After trialling various ways of showing pollster grade, sample size, and total weight simultaneously I fell back on a straightforward wordcloud:</p>
<p><img alt="mainimage" src="http://ellisp.github.io/img/0062-pollsters-cloud.svg"></p>
<p>It could definitely be improved on with some imagination.</p>
<p>Survey Monkey polls are the most influential in the data set despite their relatively low grade (C-) because of the sheer number of them – at least three surveys in every state so far plus 30 national surveys. Ipsos, Google and Survey Monkey between them have surveyed 1.75 <em>million</em> people so far in this campaign.</p>
<p>Final code snippet – creating the wordcloud:</p>
<figure>
<pre><code data-lang="r"><span># create summary pollster info. Most pollsters always have the same grade, a few
# sometimes have different grades and sometimes have their grade missing
</span><span>palette</span><span> </span><span>&lt;-</span><span> </span><span>magma</span><span>(</span><span>length</span><span>(</span><span>grades</span><span>))[</span><span>length</span><span>(</span><span>grades</span><span>)</span><span>:</span><span>1</span><span>]</span><span>
</span><span>pollsters</span><span> </span><span>&lt;-</span><span> </span><span>pollsonly</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>pollster</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>graden</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>grade</span><span> </span><span>==</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>NA</span><span>,</span><span> </span><span>as.numeric</span><span>(</span><span>grade</span><span>)))</span><span> </span><span>%>%</span><span>
</span><span>summarise</span><span>(</span><span>grade</span><span> </span><span>=</span><span> </span><span>round</span><span>(</span><span>mean</span><span>(</span><span>graden</span><span>,</span><span> </span><span>na.rm</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)),</span><span>
</span><span>totalweight</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>poll_wt</span><span>),</span><span>
</span><span>totalsample</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>samplesize</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>grade</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>is.na</span><span>(</span><span>grade</span><span>),</span><span> </span><span>11</span><span>,</span><span> </span><span>grade</span><span>),</span><span>
</span><span>grade</span><span> </span><span>=</span><span> </span><span>factor</span><span>(</span><span>grade</span><span>,</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>11</span><span>,</span><span> </span><span>labels</span><span> </span><span>=</span><span> </span><span>grades</span><span>),</span><span>
</span><span>grade_colours</span><span> </span><span>=</span><span> </span><span>palette</span><span>[</span><span>as.numeric</span><span>(</span><span>grade</span><span>)])</span><span> </span><span>%>%</span><span>
</span><span>ungroup</span><span>()</span><span> </span><span>par</span><span>(</span><span>bg</span><span> </span><span>=</span><span> </span><span>&quot;grey40&quot;</span><span>)</span><span>
</span><span>set.seed</span><span>(</span><span>223</span><span>)</span><span>
</span><span>wordcloud</span><span>(</span><span>words</span><span> </span><span>=</span><span> </span><span>pollsters</span><span>$</span><span>pollster</span><span>,</span><span>
</span><span>freq</span><span> </span><span>=</span><span> </span><span>pollsters</span><span>$</span><span>totalweight</span><span> </span><span>*</span><span> </span><span>10000</span><span>,</span><span>
</span><span>colors</span><span> </span><span>=</span><span> </span><span>pollsters</span><span>$</span><span>grade_colours</span><span>,</span><span>
</span><span>random.order</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span>
</span><span>ordered.colors</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>family</span><span> </span><span>=</span><span> </span><span>&quot;myfont&quot;</span><span>)</span><span>
</span><span>grid.text</span><span>(</span><span>0.55</span><span>,</span><span> </span><span>0.03</span><span>,</span><span> </span><span>hjust</span><span> </span><span>=</span><span> </span><span>0</span><span>,</span><span>
</span><span>label</span><span> </span><span>=</span><span> </span><span>&quot;Size is mapped to total combined weight as at 29/10/2016;\nColour is mapped to pollster grade (lighter is better).&quot;</span><span>,</span><span> </span><span>gp</span><span> </span><span>=</span><span> </span><span>gpar</span><span>(</span><span>col</span><span> </span><span>=</span><span> </span><span>&quot;grey90&quot;</span><span>,</span><span> </span><span>fontfamily</span><span> </span><span>=</span><span> </span><span>&quot;myfont&quot;</span><span>,</span><span> </span><span>cex</span><span> </span><span>=</span><span> </span><span>0.8</span><span>))</span><span>
</span><span>grid.text</span><span>(</span><span>0.03</span><span>,</span><span> </span><span>0.97</span><span>,</span><span> </span><span>hjust</span><span> </span><span>=</span><span> </span><span>0</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>&quot;The pollster data used by FiveThirtyEight&quot;</span><span>,</span><span> </span><span>gp</span><span> </span><span>=</span><span> </span><span>gpar</span><span>(</span><span>col</span><span> </span><span>=</span><span> </span><span>&quot;grey90&quot;</span><span>,</span><span> </span><span>fontfamily</span><span> </span><span>=</span><span> </span><span>&quot;myfont&quot;</span><span>,</span><span> </span><span>cex</span><span> </span><span>=</span><span> </span><span>1.5</span><span>))</span><span>
</span><span>legend</span><span>(</span><span>&quot;bottomleft&quot;</span><span>,</span><span> </span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>bg</span><span> </span><span>=</span><span> </span><span>&quot;grey45&quot;</span><span>,</span><span> </span><span>box.lty</span><span> </span><span>=</span><span> </span><span>0</span><span>,</span><span>
</span><span>legend</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>levels</span><span>(</span><span>pollsters</span><span>$</span><span>grade</span><span>)[</span><span>1</span><span>:</span><span>10</span><span>],</span><span> </span><span>&quot;None&quot;</span><span>),</span><span> </span><span>text.col</span><span> </span><span>=</span><span> </span><span>palette</span><span>)</span></code></pre>
</figure>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/fivethirtyeights-polling-data-for-the-us-presidential-election/"></div></div><div><a data-url="https://www.r-bloggers.com/fivethirtyeights-polling-data-for-the-us-presidential-election/" data-counturl="https://www.r-bloggers.com/fivethirtyeights-polling-data-for-the-us-presidential-election/" data-count="vertical" data-text="FiveThirtyEight’s polling data for the US Presidential election" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://ellisp.github.io/blog/2016/10/29/538-pollsters"> Peter's stats stuff - R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/pA6CKTm31CY"></a></div>