<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/weighted-survey-data-with-power-bi-compared-to-dplyr-sql-or-survey-by-ellis2013nz/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://freerangestats.info/blog/2018/04/11/weighted-survey-data-with-power-bi"> free range statistics - R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h2>A conundrum for Microsoft Power BI</h2>
<p>I’ve been familiarising myself with Microsoft <em>Power BI</em>, which features prominently in any current discussion on data analysis and dissemination tools for organisations. It’s a good tool with some nice features. I won’t try to do a full review here, but just ruminate on one aspect – setting it up for non-specialists to explore weighted survey data. For this, I want to be able to do appropriately weighted cross-tabs, but I’m not expecting anything that is either more sophisticated or more upstream in the data processing chain. Actually <em>creating</em> the weights, and estimating sampling uncertainty based on them, is something for another tool like R.</p>
<p>Judging from discussion threads like <a rel="nofollow" href="http://community.powerbi.com/t5/Desktop/case-weighting-in-data/td-p/51599" target="_blank">this one</a> I’m not the only one who wishes you could just say “apply case weights” in the way that you would with SPSS or a market research cross tab tool. In fact, there are some tutorials out there on elaborate and painful ways of getting around this problem that seem totally surreal to me, being used to the ease with which R or Stata deal with such problems.</p>
<p><em>Caveat on what follows – my total experience with Power BI can be measured in hours rather than days, so please take the below with a grain of salt. I may have missed something important.</em></p>
<p>I was worried that inability to deal with weighted data could be a deal breaker for the purpose I was thinking of, and when I found out that a recent release <a rel="nofollow" href="https://powerbi.microsoft.com/en-us/blog/tag/weighted-average/" target="_blank">proudly touted the ability to do a weighted average</a> (in a way that didn’t even help me much) I nearly gave up on it in disgust. Power BI lets the developer write R code and at one point I was considering the successful workflow was to pass everything through to R and send it back, before realising that this made no sense at all – might as well do the whole thing in R if that’s what it takes.</p>
<p>However, a few hours of experimentation and trying to get my head around a different way of thinking, and it turns out the solution wasn’t too difficult. It all comes down to understanding the way Power BI differentiates between static <em>columns</em> of data as opposed to <em>measures</em> which are calculated on the fly.</p>
<p>Once I’d cracked the problem I made a couple of Power BI reports with weighted microdata from complex surveys to be sure it worked generally. Here’s one that’s been made with public data, the New Zealand International Visitor Survey. It took about 20 minutes to make this, after I’d familiarised myself with the toolkit on another (non-public) dataset. It’s live and interactive, in fact interactive in too many ways to try to describe, so just have a play with it:</p>
<p><iframe width="450" allowfullscreen="true" src="https://app.powerbi.com/view?r=eyJrIjoiNjNhZDc3MGEtNDkzOS00ZDAwLWE0NTUtNzBmOTViZWU0NDIwIiwidCI6IjBlNzE2NzI4LTA1MDItNDkwZS04YTU2LTk0Y2Q5N2I1YWE2OCJ9"></iframe></p>
<p><em>Disclaimer – I’ve been responsible for that survey in the past, but not for more than a year now. What follows is very much written as a private citizen.</em></p>
<h2>Introducing some data</h2>
<p>This example survey is one I’ve <a rel="nofollow" href="https://freerangestats.info/blog/2018/02/03/ivs" target="_blank">blogged about before</a>. It’s an on-going survey of 5,000 to 10,000 tourists per year on their departure from New Zealand. Sample size, questionnaire and mode have varied over time, but the Ministry of Business, Innovation and Employment <a rel="nofollow" href="http://www.mbie.govt.nz/info-services/sectors-industries/tourism/tourism-research-data/ivs/data-download" target="_blank">publish a backcast set of the microdata</a> that is as comparable across time as is possible. It’s about 24MB to download. For today’s demo, I’m only going to use the simplest part of the data – the <code>vw_IVSSurveyMainHeader</code> table which has one row for each of the 125,000 respondents since 1997. Here’s code to download it, including a couple of convenience R functions that MBIE use to help classify countries into groupings (dated I’m afraid – I can criticise them because I wrote them myself in 2011). I also reduce the dataset to just 8 columns so when I get into Power BI I won’t have to deal with the complexity of the full data:</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>readr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>survey</span><span>)</span><span>
</span><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>source</span><span>(</span><span>&quot;https://github.com/nz-mbie/mbie-r-package-public/raw/master/pkg/R/CountryManip.R&quot;</span><span>)</span><span> </span><span># for CountryGroup
</span><span>source</span><span>(</span><span>&quot;https://github.com/nz-mbie/mbie-r-package-public/raw/master/pkg/R/NZTourism.R&quot;</span><span>)</span><span> </span><span># for rename.levels
</span><span>
</span><span># download survey data from the MBIE (Ministry of Business, Innovation and Employment) website
</span><span>download.file</span><span>(</span><span>&quot;http://www.mbie.govt.nz/info-services/sectors-industries/tourism/tourism-research-data/ivs/documents-image-library/vw_IVS.zip&quot;</span><span>,</span><span>
</span><span>mode</span><span> </span><span>=</span><span> </span><span>&quot;wb&quot;</span><span>,</span><span> </span><span>destfile</span><span> </span><span>=</span><span> </span><span>&quot;vw_IVS.zip&quot;</span><span>)</span><span>
</span><span>unzip</span><span>(</span><span>&quot;vw_IVS.zip&quot;</span><span>)</span><span>
</span><span>ivs</span><span> </span><span>&lt;-</span><span> </span><span>read_csv</span><span>(</span><span>&quot;IVS/vw_IVSSurveyMainHeader.csv&quot;</span><span>)</span><span>
</span><span>ivs_sub</span><span> </span><span>&lt;-</span><span> </span><span>ivs</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Country</span><span> </span><span>=</span><span> </span><span>CountryGroup</span><span>(</span><span>CORNextYr</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>Country</span><span>,</span><span> </span><span>Year</span><span>,</span><span> </span><span>PopulationWeight</span><span>,</span><span> </span><span>WeightedSpend</span><span>,</span><span> </span><span>POV</span><span>,</span><span> </span><span>Gender</span><span>,</span><span> </span><span>AgeRange</span><span>,</span><span> </span><span>SurveyResponseID</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>rename</span><span>(</span><span>Spend</span><span> </span><span>=</span><span> </span><span>WeightedSpend</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>as_tibble</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>Year</span><span>)</span><span>
</span><span># save a copy to use as the data source for Power BI </span><span>write_delim</span><span>(</span><span>ivs_sub</span><span>,</span><span> </span><span>path</span><span> </span><span>=</span><span> </span><span>&quot;ivs-1997-to-2017.txt&quot;</span><span>,</span><span> </span><span>delim</span><span> </span><span>=</span><span> </span><span>&quot;|&quot;</span><span>)</span></code></pre>
</figure>
<p>Of these variables:</p>
<ul>
<li><code>POV</code> stands for “purpose of visit”, a key concept in tourism data analysis that will be familiar to travellers from many countries’ arrival or departure cards.</li>
<li><code>WeightedSpend</code> actually means “outlier-treated spend”</li>
<li><code>PopulationWeight</code> is the survey weight, after all sorts of complex post-stratification including for age, gender, airport, country of residence and purpose of visit.</li>
</ul>
<p>Now, I’m interested in weighted counts of people for various combinations of dimensions (like year and purpose), and also in weighted averages and totals of continuous variables like “spend” and “nights in New Zealand”. If I were using <code>dplyr</code>, to get those yearly summary estimates for the years since 2011 I’d do something like:</p>
<figure>
<pre><code data-lang="r"><span>ivs_sub</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>Year</span><span> </span><span>></span><span> </span><span>2010</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>Year</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>summarise</span><span>(</span><span>total_spend</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>Spend</span><span> </span><span>*</span><span> </span><span>PopulationWeight</span><span>),</span><span>
</span><span>people</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>PopulationWeight</span><span>),</span><span>
</span><span>mean_spend</span><span> </span><span>=</span><span> </span><span>total_spend</span><span> </span><span>/</span><span> </span><span>people</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>desc</span><span>(</span><span>Year</span><span>))</span></code></pre>
</figure>
<p>Or the exact equivalent operation in SQL:</p>
<figure>
<pre><code data-lang="sql"><span>SELECT</span>
  <span>SUM</span><span>(</span><span>Spend</span> <span>*</span> <span>PopulationWeight</span><span>)</span>                         <span>AS</span> <span>total_spend</span><span>,</span>
  <span>SUM</span><span>(</span><span>PopulationWeight</span><span>)</span>                                 <span>AS</span> <span>people</span><span>,</span>
  <span>SUM</span><span>(</span><span>Spend</span> <span>*</span> <span>PopulationWeight</span><span>)</span> <span>/</span> <span>SUM</span><span>(</span><span>PopulationWeight</span><span>)</span> <span>AS</span> <span>mean_spend</span><span>,</span>
  <span>Year</span>
<span>FROM</span> <span>ivs_sub</span>
<span>WHERE</span> <span>Year</span> <span>></span> <span>2010</span>
<span>GROUP</span> <span>BY</span> <span>Year</span>
<span>ORDER</span> <span>BY</span> <span>Year</span> <span>DESC</span></code></pre>
</figure>
<p>I love <code>dplyr</code> because the code is ordered in the way I think of the operation: take a dataset, filter it, group it by some variable, summarise it in a particular way and then sort the results. Whereas in SQL you have to look down near the bottom for the <code>FROM</code> and <code>WHERE</code> statements to see what data you’re talking about. But it really doesn’t matter in this sort of case, they both work fine and fast and are pretty readable.</p>
<p>Those are both (to my mind) database-y ways of telling a computer to do something. A more statistically oriented way is to create a new object that somehow encompasses the survey design and its weights, and abstract the weighting of estimates away from the user. That’s the approach taken (with greatly varying degrees of statistical rigour) by commercial cross-tab tools used by market researchers, statistical packages like SPSS and Stata, and Thomas Lumley’s <code>survey</code> package in R. Here’s how you’d get mean spend per year this way (I’m ignoring the complexity in the survey design as I’m only interested in the point estimates for today)</p>
<figure>
<pre><code data-lang="r"><span>ivs_svy</span><span> </span><span>&lt;-</span><span> </span><span>svydesign</span><span>(</span><span>~</span><span>1</span><span>,</span><span> </span><span>weights</span><span> </span><span>=</span><span> </span><span>~</span><span>PopulationWeight</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>ivs_sub</span><span>)</span><span>
</span><span>mean_spend</span><span> </span><span>&lt;-</span><span> </span><span>svyby</span><span>(</span><span>~</span><span>Spend</span><span>,</span><span> </span><span>~</span><span>Year</span><span>,</span><span> </span><span>design</span><span> </span><span>=</span><span> </span><span>ivs_svy</span><span>,</span><span> </span><span>FUN</span><span> </span><span>=</span><span> </span><span>svymean</span><span>)</span><span>
</span><span>tail</span><span>(</span><span>round</span><span>(</span><span>mean_spend</span><span>))</span></code></pre>
</figure>
<p>which has these results:</p>
<figure>
<pre><code data-lang="r"><span>></span><span> </span><span>tail</span><span>(</span><span>round</span><span>(</span><span>mean_spend</span><span>))</span><span>
</span><span>Year</span><span> </span><span>Spend</span><span> </span><span>se</span><span>
</span><span>2012</span><span> </span><span>2012</span><span> </span><span>2772</span><span> </span><span>49</span><span>
</span><span>2013</span><span> </span><span>2013</span><span> </span><span>2717</span><span> </span><span>48</span><span>
</span><span>2014</span><span> </span><span>2014</span><span> </span><span>2879</span><span> </span><span>50</span><span>
</span><span>2015</span><span> </span><span>2015</span><span> </span><span>3440</span><span> </span><span>69</span><span>
</span><span>2016</span><span> </span><span>2016</span><span> </span><span>3225</span><span> </span><span>63</span><span>
</span><span>2017</span><span> </span><span>2017</span><span> </span><span>3187</span><span> </span><span>75</span></code></pre>
</figure>
<p>It’s noticeably slower than <code>dplyr</code> or SQL, but that’s because it’s doing a lot more calculation and giving you the appropriate sampling error as well as the point estimates. And once you’ve invested in creating the survey design object, it’s a lot simpler to forget about the weights and just use <code>svymean</code>, <code>svytotal</code>, <code>svyquantile</code> and so on.</p>
<h2>Weights in Power BI</h2>
<p>Power BI is an eco-system rather than a single tool, with three main parts: a desktop application, a web service, and a mobile app. A typical (but by no means the only) workflow is to do some analysis in the desktop application and create an interactive report or dashboard; and “publish” it to the web service where it can be shared either with other Power BI users, or simply as a web page like my example earlier.</p>
<p>Power BI is an amazing tool with things like natural language queries, but unfortunately there’s no simple way to just say “weight the data please, for all subsequent analyses”. So we have to do it old-school, something closer to those original <code>dplyr</code> or SQL queries.</p>
<p>For simple counts this is actually easy – we just need to tell it to report the sum of weights for each combination of variables. This fits in very nicely with how Power BI sees the world, which is basically as a giant pivot table. So no problem there.</p>
<p>For totals of spend (or another numeric variable), it’s also fairly straightforward. You need to create a new <em>column</em> of weight multiplied by the original value, added to the original data rectangle. This column is just going to be a bunch of static numbers. It’s defined this way in Power BI:</p>
<div>
<pre><code>spend_by_weight = 'ivs-1997-to-2017'[PopulationWeight] * 'ivs-1997-to-2017'[Spend]
</code></pre>
</div>
<p>Now this column can be used as the value cell in reporting tables and charts and we’re all fine. It’s annoying to have to create a persistent column for each weighted numeric variable rather than do it on the fly (as we did in SQL and <code>dplyr</code>) during the grouping and aggregation, but the gain comes with all the automated filtering interactivity of working in Power BI.</p>
<p>The weighted average is more complex. Imagine we now have a table with a row for each year and a column for total spend and for total visitors. We just want to divide spend by visitors, right? That’s what happens in <code>dplyr</code>, where we took advantage of the fact that variables created first in the <code>summarise()</code> statement can then be referred to further down in the same query.</p>
<p>I wasted a fair bit of time fiddling with how my reporting table was defined before I understood that the problem comes from all the flexibility for the end user such tables have in Power BI, which puts constraints on the developer. In particular, if the user selects another graph in the same page (or a report-wide filter), the data behind all linked tables gets automatically filtered (go back to my report at the top of the post and try it). This is like adding new <code>filter()</code> functions to our <code>dplyr</code> statement (or <code>WHERE</code> clauses in SQL). Power BI won’t let you treat the columns in a reporting table as first class objects in their own right; and there’s no way to add a column to the original static data that can be just neatly aggregated into a weighted average for any combination of filter, slice and dice that is required once it gets into that reporting table.</p>
<p>It turns out that the way around this is to define a “measure”, which is a more powerful concept in Power BI than a simple static column, even though it appears in a data source’s column list and looks similar. We define the measure we want this way, referring to the <code>spend_by_weight</code> column we’d already made for use in aggregating totals:</p>
<div>
<pre><code>mean_spend = sum('ivs-1997-to-2017'[spend_by_weight]) / sum('ivs-1997-to-2017'[PopulationWeight]) 
</code></pre>
</div>
<p>Once defined this way, the new <code>mean_spend</code> measure will be calculated dynamically and correctly for whatever combination of variables it is combined with in the visualization. It’s like defining part of <code>dplyr</code>’s <code>summarise()</code> clause in advance, then whatever the user defines as <code>group_by</code> variables (by pointing and clicking) kicks the measure into action.</p>
<p>So, in summary, to work with weighted data in Power BI you need</p>
<ul>
<li>a column of weights (obviously)</li>
<li>for each numeric variable you want to the weighted total of by any particular slice and dice, a new static <em>column</em> of the original value multiplied by the weight</li>
<li>for each numeric variable you want the weighted mean of by any particular slice and dice, a new dynamic <em>measure</em> defined in advance of the sum of the column defined in the step above (ie total of that variable), divided by the sum of the aggregated weights (ie population)</li>
</ul>
<p>Note that this means two extra variables (one column and one measure) for each existing numeric variable. This has some implications for the most effective data model to use – more normalization, with long and skinny relational tables covering several “variables” probably better than a wide table with a column for each. Which makes sense for all sorts of other reasons too.</p>
<p>I have other thoughts about Power BI (or close competitors like Tableau) and its place in the world vis a vis either hand-crafted JavaScript or the Shiny platform, but they can wait for a later post.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://freerangestats.info/blog/2018/04/11/weighted-survey-data-with-power-bi"> free range statistics - R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/jawTOxSitWQ" height="1"></a></div>