<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/the-long-view-on-new-zealand-political-polls-by-ellis2013nz/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://ellisp.github.io/blog/2017/09/09/nzpolls-again"> Peter's stats stuff - R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h2>The long view</h2>
<p>There’s been noise in the public debate on the forthcoming New Zealand election about the value of voting intention polls. There’s often not enough attention to the long view on these things, so here is a graphic of 15 years of polling data:</p>
<p><img width="100%" src="http://ellisp.github.io/img/0110-polls.svg"></p>
<p>Taken all together, the polls are probably doing better than many people think.</p>
<p>There’s been a bit of a systematic underestimate of New Zealand First support perhaps, and maybe last election an overestimate of the Greens vote, but it’s not too bad.</p>
<p>The other thing we see in that chart is that voting intention obviously changes over time. This is something that’s often forgotten when all the attention is on just the latest number. But look at the 2004 surge in National’s support after <a rel="nofollow" href="https://en.wikipedia.org/wiki/Orewa_Speech" target="_blank">Don Brash’s controversial “Orewa” speech</a>, followed by a partial collapse to not much more than it had been six months earlier. Or the interesting high levels in National support in the year or so before the 2008 and 2011 elections followed by swift movement down in the lead up to the election. Or Labour’s journey between 2011 and 2014 – growth in support up to solidly in the 30% range before a collapse in the last year to the election. In any of these cases, taking an average of an individual pollsters’ results over time and treating it as a prediction of the election day result is just unfair. People do change their minds.</p>
<p>I always think in terms of there being three sources of uncertainty in translating a poll result into a prediction of the election:</p>
<ul>
<li>uncertainty from polls’ relatively small sample size, which is published as the margin of error of “plus or minus two or three percent” and is the only indication of uncertainty that is usually given prominence</li>
<li>the rest of what is known to statisticians as “total survey error”, which comes from things like systematic small under- or over-estimates from subtleties in the polling companies’ methods, or ways that sampling people in reality doesn’t resemble just picking golf balls from a bucket (for example, unlike the golf balls, certain types of people may systematically refuse to talk to surveyers)</li>
<li>even if we knew for sure how people would vote today, people change their minds right up to the last minute.</li>
</ul>
<p>My <a rel="nofollow" href="http://ellisp.github.io/elections/combined.html" target="_blank"><em>forecasts</em></a> using this data try to take all of that into account. Only on the evening of 23 September will we know how that went.</p>
<p>The source of this data is ultimately the news organisations commissioning the polls and the volunteers contributing to <a rel="nofollow" href="https://en.m.wikipedia.org/wiki/Opinion_polling_for_the_New_Zealand_general_election,_2017" target="_blank">Wikipedia</a>. The data is collated in my <a rel="nofollow" href="https://github.com/ellisp/nzelect" target="_blank">nzelect R package</a>.</p>
<p>Here’s the R code that drew the graphic:</p>
<figure>
<pre><code data-lang="r"><span># install the latest version of nzelect, with recent polls
</span><span>devtools</span><span>::</span><span>install_github</span><span>(</span><span>&quot;ellisp/nzelect/pkg1&quot;</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>library</span><span>(</span><span>scales</span><span>)</span><span>
</span><span>library</span><span>(</span><span>nzelect</span><span>)</span><span>
</span><span>library</span><span>(</span><span>forcats</span><span>)</span><span>
</span><span>pollsters</span><span> </span><span>&lt;-</span><span> </span><span>unique</span><span>(</span><span>polls</span><span>$</span><span>Pollster</span><span>)</span><span>
</span><span>pollsters</span><span> </span><span>&lt;-</span><span> </span><span>pollsters</span><span>[</span><span>!</span><span>grepl</span><span>(</span><span>&quot;Election result&quot;</span><span>,</span><span> </span><span>pollsters</span><span>)]</span><span>
</span><span>palette</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;black&quot;</span><span>,</span><span> </span><span>hue_pal</span><span>()(</span><span>15</span><span>))</span><span>
</span><span>names</span><span>(</span><span>palette</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;Election result&quot;</span><span>,</span><span> </span><span>pollsters</span><span>)</span><span>
</span><span>polls</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>Party</span><span> </span><span>%in%</span><span> </span><span>c</span><span>(</span><span>&quot;Labour&quot;</span><span>,</span><span> </span><span>&quot;National&quot;</span><span>,</span><span> </span><span>&quot;NZ First&quot;</span><span>,</span><span> </span><span>&quot;Green&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Party</span><span> </span><span>=</span><span> </span><span>fct_reorder</span><span>(</span><span>Party</span><span>,</span><span> </span><span>-</span><span>VotingIntention</span><span>),</span><span>
</span><span>Pollster</span><span> </span><span>=</span><span> </span><span>fct_reorder</span><span>(</span><span>Pollster</span><span>,</span><span> </span><span>VotingIntention</span><span>,</span><span> </span><span>fun</span><span> </span><span>=</span><span> </span><span>length</span><span>,</span><span> </span><span>.desc</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>MidDate</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>VotingIntention</span><span>,</span><span> </span><span>colour</span><span> </span><span>=</span><span> </span><span>Pollster</span><span>))</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>~</span><span>Party</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free_y&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>()</span><span> </span><span>+</span><span>
</span><span>geom_text</span><span>(</span><span>aes</span><span>(</span><span>label</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>Pollster</span><span> </span><span>==</span><span> </span><span>&quot;Election result&quot;</span><span>,</span><span> </span><span>&quot;O&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>)),</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>8</span><span>,</span><span> </span><span>colour</span><span> </span><span>=</span><span> </span><span>&quot;black&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_y_continuous</span><span>(</span><span>label</span><span> </span><span>=</span><span> </span><span>percent</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Polling date&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Voting intention&quot;</span><span>,</span><span>
</span><span>caption</span><span> </span><span>=</span><span> </span><span>&quot;Peter's Stats Stuff, http://ellisp.github.io&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_colour_manual</span><span>(</span><span>values</span><span> </span><span>=</span><span> </span><span>palette</span><span>)</span><span> </span><span>+</span><span>
</span><span>ggtitle</span><span>(</span><span>&quot;Fifteen years of opinion polls in New Zealand&quot;</span><span>,</span><span>
</span><span>&quot;Intended party vote for the next election (election results shown in black)\nPolling results taken from Wikipedia, compiled in the nzelect R package&quot;</span><span>)</span></code></pre>
</figure>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://ellisp.github.io/blog/2017/09/09/nzpolls-again"> Peter's stats stuff - R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/7n2ZILg4910" height="1"></a></div>