<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/interpolating-statistical-tables/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://davegiles.blogspot.com/2018/01/interpolating-statistical-tables.html"> Econometrics Beat: Dave Giles' Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div dir="ltr">
<div><span>We’ve all experienced it. You go to use a statistical table – Standard Normal, Student-t, F, Chi Square – and the line that you need simply isn’t there in the table. That’s to say the table simply isn’t detailed enough for our purposes.</span></div>
<div></div>
<p><span><br></span><span>One question that always comes up when students are first being introduced to such tables is:</span></p>
<div>
<blockquote><p><span>“Do I just interpolate linearly between the nearest entries on either side of the desired value?”</span></p></blockquote>
</div>
<div><span>Not that these exact words are used, typically. For instance, a student might ask if they should take the average of the two closest values. How should you respond?</span><br><span><br></span><br><a name="more"></a><span>The correct answer to the question is: “No – not in general”. To which we should add that the correct way to deal with this situation depends on:</span></div>
<ul>
<li><span>Which distribution we’re dealing with.</span></li>
<li><span>Whether we are trying to retrieve a “missing” <i>quantile</i> for the distribution, or trying to determine a tail area (<i>probability</i>) associated with a given quantile.</span></li>
</ul>
<div><span>For example, suppose that we wish to find the 70th. percentile for a Chi-square distribution with 67 degrees of freedom; or the p-value associated with a F-value of 0.65, when the numerator and denominator degrees of freedom are 17 and 33.</span></div>
<div><span><span><br></span><span>With regard to the first of these calculations, suppose that we have access to <a rel="nofollow" href="http://web.uvic.ca/~dgiles/blog/Chi-square-table-ed.pdf" target="_blank">this table</a> of percentiles for the Chi-square distribution. Using <i>this table alone</i>, the best that we could do by way of reporting the 70th.percentile when the degrees of freedom are 67, would be to say that it lies somewhere between 46.459 and 85.527. Let’s face it – that’s not particularly helpful!</span></span><br><span><span><br></span><span>A similar situation arises if we want to compute the p-value for the F-distribution mentioned above, and we have access only to the tables that we usually find in text books.</span></span><br><span><span><br></span><span>Of course, if we have access to our favourite econometric/statistical package, it’s a simple matter to compute the desired value. <span>To get the exact answers with EViews, we’d use the command:</span></span></span></div>
<p><span><b><span><br></span></b><b>             scalar q70 = @qchisq(0.70, 67)</b>  </span><br><span><b><br></b>and the answer would be 72.554.</span><br><span><span><br></span><span>Similarly, the p-value referred to above could be computed exactly by using the command:</span></span><br><span><b><br></b><b>             scalar pval = 1 – @cfdist(0.65,17,33)</b></span><br><span><span><br></span><span>and the answer would be 0.826.</span></span><br><span><span><br></span><span>In R, we’d use the commands:</span></span><br><span><b><br></b></span></p>
<div><span><span>             </span><b>qchisq(0.70, df=67)   </b><span>and   </span><b> pf(0.65, 17, 33, lower.tail=FALSE)</b></span></div>
<div><span><span><br></span><span>to produce exactly the same results.</span></span><br><span><span><br></span><span>But what about the answer to your student’s question about interpolation? Suppose you’re stuck with the statistical table, and you don’t have access to your econometrics or statistical package. What do you do?</span></span></div>
<div><span><br></span></div>
<div><span>Some of you will remember when this was a daily occurrence! We didn’t have a laptop, a tablet, or even a pocket calculator.</span><br><span><span><br></span>We could look at various  common distributions here. However, by way of illustration, l<span>et’s take just the </span><b>Student-t Distribution</b><span>.</span><b> </b><span>W</span><span>e’ll consider appropriate ways to compute (interpolate) a </span><i>quantile </i><span>(</span><i>percentile</i><span>), first for a non-tabulated tail area probability; and secondly, for a non-tabulated degrees of freedom.</span></span></div>
<div><span><span><br></span><span><span>1.  <i>Interpolating Between Tail Areas</i></span></span></span><br><span><span><br></span><span>Suppose that we have v = 20 degrees of freedom and we want to find the quantile (percentile) that will give us an area of 0.075 under the <i>right</i> tail of the Student-t density function.</span></span><br><span><span><br></span><span>Note that 0.075 is (mid-way) between 0.05 and 0.10, so any standard table (<i>e.g</i>., <a rel="nofollow" href="http://www.itl.nist.gov/div898/handbook/eda/section3/eda3672.htm" target="_blank">here</a>) will tell us that the quantile that we want is between 1.325 and 1.725. The (<i>arithmetic</i>) average of these two percentiles is (1.325 + 1.725) /2 = 1.525.</span></span><br><span><span><br></span><span>What we’re actually doing is linearly (straight-line) interpolating between the two original values:</span></span><br><span><br></span><span><span>             1.525 = 1.325 + (1.725 – 1.325)[(0.075 – 0.1) / (0.05 – 0.1)]</span></span><br><span><span><br></span><span>However, using either the</span><span> EViews command,</span><b>  scalar qval = @qtdist(0.925,20) </b><span>or the</span><b> </b><span>R command, </span><b>qt(0.925, df=20)</b><span>, </span><span>we get the answer,</span><b> </b><span>1.497036. This <i>isn’t</i> the result that we got by linearly interpolating between the percentiles on either side!</span></span><br><span><span><br></span><span>(Recall that we wanted a <i>right-tail</i> area of 0.075, which implies a <i>left-tail</i> area of 0.925.)</span></span><br><span><span><br></span><span>Why isn’t the linear interpolation working? </span></span><span>Well, intuitively, it’s a consequence of the curvature of the density function (or the distribution function).</span><br><span><span><br></span></span><span>So, how can we deal with this?</span><br><span><span><br></span><span></span></span><br><span>Hoaglin <i>et al.</i> (1991) show that we can get a very good approximation to the correct value for the quantile (1.497036) by interpolating using the (base 10) logarithms of the tail areas to construct the interpolating weights.</span><br><span><span><br></span><span><span>Noting that </span><span>log<sub>10</sub>(0.075) = -1.1249387366083 ; </span><span>log<sub>10</sub>(0.1) =  -1.0 ; and </span><span>log<sub>10</sub>(0.05) = -1.301029995663981, the appropriate calculation for the desired quantile becomes:</span></span></span><br><span><span><br></span><span>q </span><span>= 1.325 + (1.725 – 1.325)[(-1.1249387366083 – (-1)) / (-1.301029995663981 – (-1))] </span></span><br><span>   = 1.49104</span>
<div><span><span><br></span><span>That’s more like it, even though it isn’t perfect!</span></span><br><span><br></span></div>
<p><span>2.  <i>Interpolating Between Degrees of Freedom</i></span><br><span><span><i><br></i></span><span>Now let’s consider a different situation – one where the table that we’re using for the t-distribution includes the (right-hand) tail area that we want – say, 5%. However, our particular degrees of freedom (v) are nowhere to be found in the table.</span></span><br><span><span><br></span><span>Actually, this isn’t likely to be as much of a problem as the situation we’ve just considered above. Most t-tables cover every degree of freedom, for small-to-moderate values of v; and when v is large the standard-Normal approximation will usually suffice.</span></span><br><span><span><br></span><span>However, let’s suppose that we want an accurate answer, and by way of an example, consider a 5% (right) tail area, and 53 degrees of freedom.</span></span><br><span><span><br></span><span>In this case we can use an <i>harmonic interpolation</i>, rather than a linear one. This is like switching from an arithmetic mean to an harmonic mean, so we use the the values of (1 / v) to construct the interpolation weights.</span></span><br><span><span><br></span><span>(Isn’t it nice to come across a situation where that harmonic mean that you learned about actually gets used! Another situation arises with <a rel="nofollow" href="http://davegiles.blogspot.ca/2012/01/extracting-correct-meaning-from-data.html" target="_blank">certain index numbers</a>.)</span></span><br><span><span><br></span><span>Let’s go back to our example. From any Student-t table you can find that the quantiles that determine 5% in the <i>right tail</i> of the density are 1.671 and 1.684 for v = 60 and v = 40, respectively. Here’s what happens if you just use a naive linear interpolation to get the quantile for v = 53:</span></span><br><span><span><br></span><span>              q = </span><span>1.671 + (1.684 – 1.671)[(53 – 60) / (40 – 60)] = 1.6755 </span></span><br><span><span><br></span><span>Using the EViews command <b>scalar q = @qtdist(0.95 , 53),</b> or the R code <b>qt(0.95, df = 53</b>), you can verify that the exact answer is 1.674116.</span></span><br><span><span><br></span><span>Applying the harmonic interpolation, we get:</span></span><br><span><span><br></span><span><span>            q = </span><span>1.671 + (1.684 – 1.671)[((1/53) – (1/60)) / ((1/40) – (1/60))] = 1.67443.</span></span></span><br><span><span><br></span><span>This is certainly <i>a little</i> more accurate than the result obtained above by linear interpolation. However, the gain in accuracy will vary, depending on the significance level and degrees of freedom that we’re considering.</span></span><br><span><span><br></span></span><span><span>When we’re working with the Chi-Square, F, or other distributions, similar methods are available. However, the details differ a little from those used for the Student-t distribution. The references given below will give you some guidance.</span></span><br><span><span><br></span></span><span><span>In a sense, the take-away message here is very simple – don’t use “straight-line” interpolation when the function is curved!</span></span><br><span><span><br></span></span><strong><span>References</span></strong></div>
<div><span><span><br></span><span><a rel="nofollow" href="http://www.amazon.com/Fundamentals-Exploratory-Analysis-Probability-Statistics/dp/0471527351/ref=sr_1_3?s=books&ie=UTF8&qid=1327861689&sr=1-3" target="_blank">Hoaglin, D. C.</a>, F. Mosteller, and J. W. Tukey, 1991. <em>Fundamentals of Exploratory Analysis of Variance</em>. Wiley, New York.</span></span><br><span><span><span><br></span><span><a rel="nofollow" href="http://www.ams.org/journals/mcom/1959-13-067/S0025-5718-1959-0105789-1/S0025-5718-1959-0105789-1.pdf" target="_blank">Salton, G<b>.</b></a>, 1959. The use of the central limit theorem for interpolating in tables of probability </span></span><span>distribution functions. <i>Mathematical Tables and Other Aids to Computation</i>, 13, 213-216.</span></span><br><span><span><br></span><span><a rel="nofollow" href="https://www.jstor.org/stable/2985223?seq=1#page_scan_tab_contents" target="_blank">Zinger, A.</a>, 1964. On interpolation in tables of the F-distribution. <i>Journal of the Royal Society</i>, <i>Series C</i> (<i>Applied Statistics</i>), 13, 51-53.</span></span><br><span><br></span><span><br></span></div>
<div><span>© 2018, David E. Giles</span></div>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://davegiles.blogspot.com/2018/01/interpolating-statistical-tables.html"> Econometrics Beat: Dave Giles' Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/5gEL4GQZgHQ" height="1"></a></div>