<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/examining-inter-rater-reliability-in-a-reality-baking-show/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://mattansb.blogspot.com/2018/10/examining-inter-rater-reliability-in.html"> Mattan S. Ben-Shachar</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a rel="nofollow" href="https://en.wikipedia.org/wiki/Game_of_Chefs" target="_blank">Game of Chefs</a> is an Israeli reality cooking competition show, where chefs compete for the title of “Israel’s most talented chef”. The show has four stages: blind auditions, training camp, kitchen battles, and finals. Here I will examine the 4th season’s (<i><a rel="nofollow" href="https://reshet.tv/yummies/games-of-chef-konditor/" target="_blank">Game of Chefs: Confectioner</a></i>) inter-rater reliability in the blind auditions, using some simple R code.</p>
<h2>The Format</h2>
<div>
<div>In the blind auditions, candidates have 90 minutes to prepare a dessert, confection or other baked good, which is then sent to the show’s four judges for a blind taste test. The judges don’t see the candidate, and know nothing about him or her, until after they deliver their decision – A “pass” decision is signified by the judge granting the candidate a kitchen knife; a “fail” decision is signified by the judge not granting a knife. A candidate who receives a knife from at least three of the judges continues on to the next stage, the training camp.
<table cellspacing="0" align="center" cellpadding="0">
<tbody>
<tr>
<td><a rel="nofollow" href="https://i2.wp.com/img.playbuzz.com/image/upload/q_auto:good,f_auto,fl_lossy,w_640,c_limit/v1539072677/hiot174quriwj8lltgx1.png?ssl=1" target="_blank"><img border="0" src="https://i2.wp.com/img.playbuzz.com/image/upload/q_auto:good,f_auto,fl_lossy,w_640,c_limit/v1539072677/hiot174quriwj8lltgx1.png?resize=450%2C307&ssl=1" data-original-width="450" width="450" data-original-height="307" data-recalc-dims="1"></a></td>
</tr>
<tr>
<td>The 4 judges and host, from left to right: Yossi Shitrit, Erez Komarovsky, Miri Bohadana (host), Assaf Granit, and Moshik Roth</td>
</tr>
</tbody>
</table>
</div>
<h2>Inter-Rater Reliability</h2>
<div>I’ve watched all 4 episodes of the blind auditions (for purely academic reasons!), for a total of 31 candidates. For each dish, I recorded each of the four judges’ verdict (fail / pass).
<p>Let’s load the data.</p></div>
</div>
<pre><code>df &lt;- read.csv('https://github.com/mattansb/blogCode/raw/master/2018_10_18%20Inter-Rater%20Reliability/IJR.csv')<br>head(df)<br><br>##   Moshik Assaf Erez Yossi<br>## 1      1     1    1     1<br>## 2      1     0    0     0<br>## 3      1     1    1     0<br>## 4      1     0    0     0<br>## 5      0     1    1     1<br>## 6      1     1    1     1<br></code></pre>
<p>We will need the following packages:</p>
<pre><code>library(dplyr) # for manipulating the data<br>library(psych) # for computing Choen's Kappa<br>library(corrr) # for manipulating matrices<br><br></code></pre>
<p>We can now use the <code>psych</code> package to compute <a rel="nofollow" href="https://en.wikipedia.org/wiki/Cohen%27s_kappa" target="_blank">Cohen’s Kappa coefficient</a> for inter-rater agreement for categorical items, such as the fail/pass categories we have here. \$\kappa\$ ranges from -1 (full disagreement) through 0 (no pattern of agreement) to +1 (full agreement). Normally, \$\kappa\$ is computed between two raters, and for the case of more than two raters, the mean \$\kappa\$ across all pair-wise raters is used.</p>
<pre><code>cohen.kappa(df)<br><br>## Cohen Kappa (below the diagonal) and Weighted Kappa (above the diagonal) <br>## For confidence intervals and detail print with all=TRUE<br>##        Moshik Assaf  Erez  Yossi<br>## Moshik   1.00  0.34  0.17   0.29<br>## Assaf    0.34  1.00  0.17   0.68<br>## Erez     0.17  0.17  1.00  -0.16<br>## Yossi    0.29  0.68 -0.16   1.00<br>## <br>## Average Cohen kappa for all raters  0.25<br>## Average weighted kappa for all raters  0.25<br></code></pre>
<p>We can see that overall \$\kappa=0.25\$ – surprisingly low for what might be expected from a group of pristine, accomplished, professional chefs and bakers in a <i>blind</i> taste test.</p>
<p>When examining the pair-wise coefficients, we can also see that Erez seems to be in lowest agreement with each of the other judges (and even in a slight disagreement with Yossi!). This might be because Erez is new on the show (this is his first season as judge), but it might also be because of the four judges, he is the only one who is actually a confectioner (the other 3 are restaurant chefs).</p>
<div>For curiosity’s sake, let’s also look at the \$\kappa\$ coefficient between each judge’s rating and the total fail/pass decision, based on whether a dish got a “pass” from at least 3 judges.</div>
<pre><code>df &lt;- mutate(df,<br>             PASS = as.numeric(rowSums(df) >= 3))<br>head(df)<br><br>##   Moshik Assaf Erez Yossi PASS<br>## 1      1     1    1     1    1<br>## 2      1     0    0     0    0<br>## 3      1     1    1     0    1<br>## 4      1     0    0     0    0<br>## 5      0     1    1     1    1<br>## 6      1     1    1     1    1<br></code></pre>
<p>We can now use the wonderful new <code><a rel="nofollow" href="https://cran.r-project.org/package=corrr" target="_blank">corrr</a></code> package, which is intended for exploring correlations, but can also generally be used to manipulate any symmetric matrix in a tidy-fashion.</p>
<pre><code>cohen.kappa(df)$cohen.kappa %>%<br>  as_cordf() %>%<br>  focus(PASS)<br><br>## # A tibble: 4 x 2<br>##   rowname  PASS<br>##   </code></pre>