<div> It’s been a long time since I’ve posted a Statistics Sunday post! Now that I’m moved out of my apartment and into my house, I have a bit more time on my hands, but work has been quite busy. Today, I’m preparing for 2 upcoming <a rel="nofollow" href="http://www.deeplytrivial.com/2018/09/statistics-sunday-what-is-standard.html" target="_blank">standard-setting studies</a> by drawing a sample of items from 2 of our exams. So I thought I’d share what I’m up to in order to pass on some of these new psychometric tricks I’ve learned to help me with this project.
<p>Because I can’t share data from our item banks, I’ll generate a fake dataset to use in my demonstration. For the exams I’m using for my upcoming standard setting, I want to draw a large sample of items, stratified by both item difficulty (so that I have a range of items across the <a rel="nofollow" href="http://www.deeplytrivial.com/2017/09/great-minds-in-statistics-georg-rasch.html" target="_blank">Rasch</a> difficulties) and item domain (the topic from the exam outline that is assessed by that item). Let’s pretend I have an exam with 3 domains, and a bank of 600 items. I can generate that data like this:</p>
<div>
<div>
<div>
<pre><span>domain1</span> <span>&lt;-</span> <span>data.frame</span><span>(</span><span>domain</span> <span>=</span> <span>1</span><span>,</span> <span>b</span> <span>=</span> <span>sort</span><span>(</span><span>rnorm</span><span>(</span><span>200</span><span>)))</span><br><span>domain2</span> <span>&lt;-</span> <span>data.frame</span><span>(</span><span>domain</span> <span>=</span> <span>2</span><span>,</span> <span>b</span> <span>=</span> <span>sort</span><span>(</span><span>rnorm</span><span>(</span><span>200</span><span>)))</span><br><span>domain3</span> <span>&lt;-</span> <span>data.frame</span><span>(</span><span>domain</span> <span>=</span> <span>3</span><span>,</span> <span>b</span> <span>=</span> <span>sort</span><span>(</span><span>rnorm</span><span>(</span><span>200</span><span>)))</span><br></pre>
</div>
</div>
</div>
<p>The variable domain is the domain label, and b is the item difficulty. I decided to sort that varible within each dataset so I can easily see that it goes across a range of difficulties, both positive and negative.</p>
<div>
<div>
<div>
<pre><span>head</span><span>(domain1)</span><br></pre>
</div>
<div>
<pre>##   domain         b<br>## 1      1 -2.599194<br>## 2      1 -2.130286<br>## 3      1 -2.041127<br>## 4      1 -1.990036<br>## 5      1 -1.811251<br>## 6      1 -1.745899<br></pre>
</div>
<div>
<pre><span>tail</span><span>(domain1)</span><br></pre>
</div>
<div>
<pre>##     domain        b<br>## 195      1 1.934733<br>## 196      1 1.953235<br>## 197      1 2.108284<br>## 198      1 2.357364<br>## 199      1 2.384353<br>## 200      1 2.699168<br></pre>
</div>
</div>
</div>
<p>If I desire, I can easily combine these 3 datasets into 1:</p>
<div>
<div>
<div>
<pre><span>item_difficulties</span> <span>&lt;-</span> <span>rbind</span><span>(domain1, domain2, domain3)</span><br></pre>
</div>
</div>
</div>
<p>I can also easily visualize my item difficulties, by domain, as a group of histograms using ggplot2:</p>
<div>
<div>
<div>
<pre><span>library</span><span>(tidyverse)</span><br></pre>
</div>
<div>
<pre><span>item_difficulties</span> <span>%>%</span><br>  <span>ggplot</span><span>(</span><span>aes</span><span>(b))</span> <span>+</span><br>  <span>geom_histogram</span><span>(</span><span>show.legend</span> <span>=</span> <span>FALSE</span><span>)</span> <span>+</span><br>  <span>labs</span><span>(</span><span>x</span> <span>=</span> <span>&quot;Item Difficulty&quot;</span><span>,</span> <span>y</span> <span>=</span> <span>&quot;Number of Items&quot;</span><span>)</span> <span>+</span><br>  <span>facet_wrap</span><span>(</span><span>~</span><span>domain,</span> <span>ncol</span> <span>=</span> <span>1</span><span>,</span> <span>scales</span> <span>=</span> <span>&quot;free&quot;</span><span>)</span> <span>+</span><br>  <span>theme_classic</span><span>()</span><br></pre>
</div>
<div>
<pre>## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.<br></pre>
</div>
</div>
<div><a rel="nofollow" href="https://i2.wp.com/4.bp.blogspot.com/-hRfW802ko8Y/W8OpSzHrnDI/AAAAAAAAOCc/DpiGUSp0TjsgAQdqzbfn0njsaIxdkpeUgCLcBGAs/s1600/unnamed-chunk-4-1.png?ssl=1" target="_blank"><img border="0" src="https://i0.wp.com/4.bp.blogspot.com/-hRfW802ko8Y/W8OpSzHrnDI/AAAAAAAAOCc/DpiGUSp0TjsgAQdqzbfn0njsaIxdkpeUgCLcBGAs/s320/unnamed-chunk-4-1.png?resize=320%2C320&ssl=1" data-original-width="450" width="320" data-original-height="504" data-recalc-dims="1" height="320"></a></div>
</div>
<p>Now, let’s say I want to draw 100 items from my item bank, and I want them to be stratified by difficulty and by domain. I’d like my sample to range across the potential item difficulties fairly equally, but I want my sample of items to be weighted by the percentages from the exam outline. That is, let’s say I have an outline that says for each exam: 24% of items should come from domain 1, 48% from domain 2, and 28% from domain 3. So I want to draw 24 from domain1, 48 from domain2, and 28 from domain3. Drawing such a random sample is pretty easy, but I also want to make sure I get items that are very easy, very hard, and all the levels in between.</p>
<p>I’ll be honest: I had trouble figuring out the best way to do this with a continuous variable. Instead, I decided to classify items by quartile, then drew an equal number of items from each quartile.</p>
<p>To categorize by quartile, I used the following code:</p>
<div>
<div>
<div>
<pre><span>domain1</span> <span>&lt;-</span> <span>within</span><span>(domain1, quartile</span> <span>&lt;-</span> <span>as.integer</span><span>(</span><span>cut</span><span>(b,</span> <span>quantile</span><span>(b,</span> <span>probs</span> <span>=</span> <span>0</span><span>:</span><span>4</span><span>/</span><span>4</span><span>),</span> <span>include.lowest</span> <span>=</span> <span>TRUE</span><span>)))</span><br></pre>
</div>
</div>
</div>
<p>The code uses the quantile command, which you may remember from my post on <a rel="nofollow" href="http://www.deeplytrivial.com/2018/02/statistics-sunday-quantile-regression.html" target="_blank">quantile regression</a>. The nice thing about using quantiles is that I can define that however I wish. So I didn’t have to divide my items into quartiles (groups of 4); I could have divided them up into more or fewer groups as I saw fit. To aid in drawing samples across domains of varying percentages, I’d probably want to pick a quantile that is a common multiple of the domain percentages. In this case, I purposefully designed the outline so that 4 was a common multiple.</p>
<p>To draw my sample, I’ll use the sampling library (which you’ll want to install with install.packages(“sampling”) if you’ve never done so before), and the strata function.</p>
<div>
<div>
<div>
<pre><span>library</span><span>(sampling)</span><br><span>domain1_samp</span> <span>&lt;-</span> <span>strata</span><span>(domain1,</span> <span>&quot;quartile&quot;</span><span>,</span> <span>size</span> <span>=</span> <span>rep</span><span>(</span><span>6</span><span>,</span> <span>4</span><span>),</span> <span>method</span> <span>=</span> <span>&quot;srswor&quot;</span><span>)</span><br></pre>
</div>
</div>
</div>
<p>The resulting data frame has 4 variables – the quartile value (since that was used for stratification), the ID_unit (row number from the original dataset), probability of being selected (in this case equal, since I requested equally-sized strata), and stratum number. So I would want to merge my item difficulties into this dataset, as well as any identifiers I have so that I can pull the correct items. (For the time being, we’ll just pretend row number is the identifier, though this is likely not the case for large item banks.)</p>
<div>
<div>
<div>
<pre><span>domain1</span><span>$</span><span>ID_unit</span> <span>&lt;-</span> <span>as.numeric</span><span>(</span><span>row.names</span><span>(domain1))</span><br><span>domain1_samp</span> <span>&lt;-</span> <span>domain1_samp</span> <span>%>%</span><br>  <span>left_join</span><span>(domain1,</span> <span>by</span> <span>=</span> <span>&quot;ID_unit&quot;</span><span>)</span><br><span>qplot</span><span>(domain1_samp</span><span>$</span><span>b)</span><br></pre>
</div>
<div>
<pre>## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.<br></pre>
</div>
</div>
<div><a rel="nofollow" href="https://i0.wp.com/1.bp.blogspot.com/-1LT4akxAXBM/W8OpYRS5qDI/AAAAAAAAOCg/_pVYqUvt3IkH045VxHUIaifvWahmqd8CgCLcBGAs/s1600/unnamed-chunk-7-1.png?ssl=1" target="_blank"><img border="0" src="https://i1.wp.com/1.bp.blogspot.com/-1LT4akxAXBM/W8OpYRS5qDI/AAAAAAAAOCg/_pVYqUvt3IkH045VxHUIaifvWahmqd8CgCLcBGAs/s320/unnamed-chunk-7-1.png?resize=320%2C320&ssl=1" data-original-width="450" width="320" data-original-height="504" data-recalc-dims="1" height="320"></a></div>
</div>
<p>For my upcoming study, my sampling technique is a bit more nuanced, but this gives a nice starting point and introduction to what I’m doing. </p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.deeplytrivial.com/2018/10/statistics-sunday-some-psychometric.html"> Deeply Trivial</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/8SgZ5xWRWgc" height="1"></a></div></div>