<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/presidential-election-predictions-2016-an-asa-competition/"></div></div><div><a data-url="https://www.r-bloggers.com/presidential-election-predictions-2016-an-asa-competition/" data-counturl="https://www.r-bloggers.com/presidential-election-predictions-2016-an-asa-competition/" data-count="horizontal" data-text="Presidential Election Predictions 2016 (an ASA competition)" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://www.r-statistics.com/2016/08/presidential-election-predictions-2016/"> R – R-statistics blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><em><strong><span>Guest post by <a target="_blank" href="http://research.pomona.edu/johardin/">Jo</a></span> <span><a target="_blank" href="http://research.pomona.edu/johardin/">Hardin</a>, </span></strong><span>professor of mathematics, Pomona College.</span></em></p>
<p><a name="asas-prediction-competition"></a><span>ASA’s Prediction Competition</span></p>
<p><span>In this election year, the American Statistical Association (ASA) has put together a competition for students to predict the exact percentages for the winner of the 2016 presidential election. They are offering cash prizes for the entry that gets closest to the national vote percentage and that best predicts the winners for each state and the District of Columbia. For more details see:</span></p>
<p><a target="_blank" href="http://thisisstatistics.org/electionprediction2016/"><span>http://thisisstatistics.org/electionprediction2016/</span></a></p>
<p><span>To get you started, I’ve written an analysis of data scraped from fivethirtyeight.com. The analysis uses weighted means and a formula for the standard error (SE) of a weighted mean. For your analysis, you might consider a similar analysis on the state data (what assumptions would you make for a new weight function?). Or you might try some kind of model – either a generalized linear model or a Bayesian analysis with an informed prior. The world is your oyster!</span></p>
<p><a name="getting-the-data"></a><span>Getting the Data</span></p>
<p> </p>
<p><span>Thanks to the Internet, there is a lot of polling data which is publicly accessible. For the competition, you are welcome to get your data from anywhere. However, I’m going to take mine from 538.</span> <a target="_blank" href="http://projects.fivethirtyeight.com/2016-election-forecast/national-polls/"><span>http://projects.fivethirtyeight.com/2016-election-forecast/national-polls/</span></a> <span>(Other good sources of data are</span> <a target="_blank" href="http://www.realclearpolitics.com/epolls/latest_polls/"><span>http://www.realclearpolitics.com/epolls/latest_polls/</span></a> <span>and</span> <a target="_blank" href="http://elections.huffingtonpost.com/pollster/2016-general-election-trump-vs-clinton"><span>http://elections.huffingtonpost.com/pollster/2016-general-election-trump-vs-clinton</span></a> <span>and</span> <a target="_blank" href="http://www.gallup.com/products/170987/gallup-analytics.aspx"><span>http://www.gallup.com/products/170987/gallup-analytics.aspx</span></a><span>)</span></p>
<p><span>Note the date indicated above as to when this R Markdown file was written. That’s the day the data were scraped from 538. If you run the Markdown file on a different day, you are likely to get different results as the polls are constantly being updated.</span></p>
<p><span>Because the original data were scraped as a JSON file, it gets pulled into R as a list of lists. The data wrangling used to convert it into a tidy format is available from the source code at</span> <a target="_blank" href="https://github.com/hardin47/prediction2016/blob/master/predblog.Rmd"><span>https://github.com/hardin47/prediction2016/blob/master/predblog.Rmd</span></a><span>.</span></p>
<p><span>url =</span><span> “http://projects.fivethirtyeight.com/2016-election-forecast/national-polls/”</span><br>
<span>doc &lt;-</span> <span>htmlParse</span><span>(url, </span><span>useInternalNodes =</span> <span>TRUE</span><span>)</span></p>
<p><span>sc =</span> <span>xpathSApply</span><span>(doc, </span><span>“//script[contains(., ‘race.model’)]”</span><span>, </span><br>
<span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span>function(x) </span><span>c</span><span>(</span><span>xmlValue</span><span>(x), </span><span>xmlAttrs</span><span>(x)[[</span><span>“href”</span><span>]]))</span></p>
<p><span>jsobj =</span> <span>gsub</span><span>(</span><span>“.*race.stateData = (.*);race.pathPrefix.*”</span><span>, </span><span>“</span><span>\\</span><span>1″</span><span>, sc)</span></p>
<p><span>data =</span> <span>fromJSON</span><span>(jsobj)</span><br>
<span>allpolls &lt;-</span> <span>data$polls</span></p>
<p><span>#unlisting the whole thing</span><br>
<span>indx &lt;-</span> <span>sapply</span><span>(allpolls, length)</span><br>
<span>pollsdf &lt;-</span> <span>as.data.frame</span><span>(</span><span>do.call</span><span>(rbind, </span><span>lapply</span><span>(allpolls, </span><span>`</span><span>length&lt;-</span><span>`</span><span>, </span><span>max</span><span>(indx))))</span></p>
<p>
<p><a name="a-quick-visualization"></a><span>A Quick Visualization</span></p>
<p><span>Before coming up with a prediction for the vote percentages for the 2016 US Presidential Race, it is worth trying to look at the data. The data are in a tidy form, so ggplot2 will be the right tool for visualizing the data.</span></p>
<p><span>ggplot</span><span>(</span><span>subset</span><span>(allpolldata, ((polltypeA ==</span><span> “now”</span><span>) &amp;</span> <span>(endDate ></span> <span>ymd</span><span>(</span><span>“2016-08-01”</span><span>)))), </span><br>
<span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span>aes</span><span>(</span><span>y=</span><span>adj_pct, </span><span>x=</span><span>endDate, </span><span>color=</span><span>choice)) +</span><br>
<span>  </span><span>geom_line</span><span>() +</span> <span>geom_point</span><span>(</span><span>aes</span><span>(</span><span>size=</span><span>wtnow)) +</span><br>
<span>  </span><span>labs</span><span>(</span><span>title =</span> <span>“Vote percentage by date and poll weight</span><span>\n</span><span>“</span><span>, </span><br>
<span> </span> <span> </span> <span>y =</span> <span>“Percent Vote if Election Today”</span><span>, </span><span>x =</span> <span>“Poll Date”</span><span>, </span><br>
<span> </span> <span> </span> <span>color =</span> <span>“Candidate”</span><span>, </span><span>size=</span><span>“538 Poll</span><span>\n</span><span>Weight”</span><span>)</span></p>
<p><img alt="" src="https://i2.wp.com/www.r-statistics.com/wp-content/uploads/2016/08/57bf3a9e062c7_img.png?resize=484%2C388" data-recalc-dims="1"></p>
<p><a name="a-quick-analysis"></a><span>A Quick Analysis</span></p>
<p><span>Let’s try to think about the percentage of votes that each candidate will get based on the</span> <span>now cast</span> <span>polling percentages. We’d like to weight the votes based on what 538 thinks (hey, they’ve been doing this longer than I have!), the sample size, and the number of days since the poll closed.</span></p>
<p><img alt="" src="https://i1.wp.com/www.r-statistics.com/wp-content/uploads/2016/08/57bf3a9e063b4_img.png?resize=204%2C50" data-recalc-dims="1"></p>
<p><span>Using my weight, I’ll calculate a weighted average and a weighted SE for the predicted percent of votes. (The SE of the weighted variance is taken from Cochran (1977) and cited in Gatz and Smith (1995).) The weights can be used to calculate the average or the running average for the</span> <span>now cast</span> <span>polling percentages.</span></p>
<p><span>allpolldata2 &lt;-</span> <span>allpolldata %>%</span><br>
<span>  </span><span>filter</span><span>(wtnow ></span> <span>0</span><span>) %>%</span><br>
<span>  </span><span>filter</span><span>(polltypeA ==</span><span> “now”</span><span>) %>%</span><br>
<span>  </span><span>mutate</span><span>(</span><span>dayssince =</span> <span>as.numeric</span><span>(</span><span>today</span><span>() –</span> <span>endDate)) %>%</span><br>
<span>  </span><span>mutate</span><span>(</span><span>wt =</span> <span>wtnow *</span> <span>sqrt</span><span>(sampleSize) /</span> <span>dayssince) %>%</span><br>
<span>  </span><span>mutate</span><span>(</span><span>votewt =</span> <span>wt*pct) %>%</span><br>
<span>  </span><span>group_by</span><span>(choice) %>%</span><br>
<span>  </span><span>arrange</span><span>(choice, -dayssince) %>%</span><br>
<span>  </span><span>mutate</span><span>(</span><span>cum.mean.wt =</span> <span>cumsum</span><span>(votewt) /</span> <span>cumsum</span><span>(wt)) %>%</span><br>
<span>  </span><span>mutate</span><span>(</span><span>cum.mean =</span> <span>cummean</span><span>(pct))</span></p>
<p><a name="plotting-the-cumulative-mean-weighted-me"></a><span>Plotting the Cumulative Mean / Weighted Mean</span></p>
<p><span>In tidy format, the data are ready to plot. Note that the cumulative mean is much smoother than the cumulative weighted mean because the weights are much heavier toward the later polls.</span></p>
<p><span>ggplot</span><span>(</span><span>subset</span><span>(allpolldata2, ( endDate ></span> <span>ymd</span><span>(</span><span>“2016-01-01”</span><span>))), </span><br>
<span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span>aes</span><span>(</span><span>y=</span><span>cum.mean, </span><span>x=</span><span>endDate, </span><span>color=</span><span>choice)) +</span><br>
<span>  </span><span>geom_line</span><span>() +</span> <span>geom_point</span><span>(</span><span>aes</span><span>(</span><span>size=</span><span>wt)) +</span><br>
<span>    </span><span>labs</span><span>(</span><span>title =</span> <span>“Cumulative Mean Vote Percentage</span><span>\n</span><span>“</span><span>, </span><br>
<span> </span> <span> </span> <span>y =</span> <span>“Cumulative Percent Vote if Election Today”</span><span>, </span><span>x =</span> <span>“Poll Date”</span><span>, </span><br>
<span> </span> <span> </span> <span>color =</span> <span>“Candidate”</span><span>, </span><span>size=</span><span>“Calculated Weight”</span><span>)</span></p>
<p><img alt="" src="https://i0.wp.com/www.r-statistics.com/wp-content/uploads/2016/08/57bf3a9e06514_img.png?resize=484%2C388" data-recalc-dims="1"></p>
<p><span>ggplot</span><span>(</span><span>subset</span><span>(allpolldata2, (endDate ></span> <span>ymd</span><span>(</span><span>“2016-01-01”</span><span>))), </span><br>
<span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span>aes</span><span>(</span><span>y=</span><span>cum.mean.wt, </span><span>x=</span><span>endDate, </span><span>color=</span><span>choice)) +</span><br>
<span>  </span><span>geom_line</span><span>() +</span> <span>geom_point</span><span>(</span><span>aes</span><span>(</span><span>size=</span><span>wt)) +</span><br>
<span>  </span><span>labs</span><span>(</span><span>title =</span> <span>“Cumulative Weighted Mean Vote Percentage</span><span>\n</span><span>“</span><span>, </span><br>
<span> </span> <span> </span> <span>y =</span> <span>“Cumulative Weighted Percent Vote if Election Today”</span><span>, </span><span>x =</span> <span>“Poll Date”</span><span>, </span><br>
<span> </span> <span> </span> <span>color =</span> <span>“Candidate”</span><span>, </span><span>size=</span><span>“Calculated Weight”</span><span>)</span></p>
<p><img alt="" src="https://i2.wp.com/www.r-statistics.com/wp-content/uploads/2016/08/57bf3a9e066c2_img.png?resize=484%2C388" data-recalc-dims="1"></p>
<p><span>Additionally, the weighted average and the SE of the average (given by Cochran (1977)) can be computed for each candidate. Using the formula, we have our prediction of the final percentage of the popular vote for each major candidate!</span></p>
<p><span>pollsummary &lt;-</span> <span>allpolldata2 %>%</span><br>
<span>  </span><span>select</span><span>(choice, pct, wt, votewt, sampleSize, dayssince) %>%</span><br>
<span>  </span><span>group_by</span><span>(choice) %>%</span><br>
<span>  </span><span>summarise</span><span>(</span><span>mean.vote =</span> <span>weighted.mean</span><span>(pct, wt, </span><span>na.rm=</span><span>TRUE</span><span>),</span><br>
<span> </span> <span> </span> <span> </span> <span> </span> <span> </span> <span> </span><span>std.vote =</span> <span>sqrt</span><span>(</span><span>weighted.var.se</span><span>(pct, wt, </span><span>na.rm=</span><span>TRUE</span><span>)))</span></p>
<p><span>pollsummary</span></p>
<p><span>## # A tibble: 2 x 3</span><br>
<span>##    choice mean.vote  std.vote</span><br>
<span>##     &lt;chr>     &lt;dbl>     &lt;dbl></span><br>
<span>## 1 Clinton  43.64687 0.5073492</span><br>
<span>## 2   Trump  39.32071 1.1792667</span></p>
<p><a name="other-peoples-advice"></a><span>Other people’s advice</span></p>
<p><span>Prediction is very difficult, especially about the future. – Niels Bohr</span></p>
<p><span>Along with good data sources, you should also be able to find information about prediction and modeling. I’ve provided a few resources to get you started.</span></p>
<p><span>•</span><span>             </span><span>Andrew Gelman:</span> <a target="_blank" href="http://andrewgelman.com/2016/08/17/29654/"><span>http://andrewgelman.com/2016/08/17/29654/</span></a></p>
<p><span>•</span><span>             </span><span>Sam Wang:</span> <a target="_blank" href="http://election.princeton.edu/2016/08/21/sharpening-the-forecast/"><span>http://election.princeton.edu/2016/08/21/sharpening-the-forecast/</span></a></p>
<p><span>•</span><span>             </span><span>Fivethirtyeight:</span> <a target="_blank" href="http://fivethirtyeight.com/features/a-users-guide-to-fivethirtyeights-2016-general-election-forecast/"><span>http://fivethirtyeight.com/features/a-users-guide-to-fivethirtyeights-2016-general-election-forecast/</span></a></p>
<p><span>•</span><span>             </span><span>Christensen and Florence:</span> <a target="_blank" href="http://www.amstat.org/misc/tasarticle.pdf"><span>http://www.amstat.org/misc/tasarticle.pdf</span></a> <span>and</span> <a target="_blank" href="https://tofu.byu.edu/electionpollproject/"><span>https://tofu.byu.edu/electionpollproject/</span></a></p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/presidential-election-predictions-2016-an-asa-competition/"></div></div><div><a data-url="https://www.r-bloggers.com/presidential-election-predictions-2016-an-asa-competition/" data-counturl="https://www.r-bloggers.com/presidential-election-predictions-2016-an-asa-competition/" data-count="vertical" data-text="Presidential Election Predictions 2016 (an ASA competition)" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.r-statistics.com/2016/08/presidential-election-predictions-2016/"> R – R-statistics blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/8xRCQW_EYC4"></a></div>