<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/prediction-intervals-for-glms-part-ii/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.fromthebottomoftheheap.net/2017/05/01/glm-prediction-intervals-ii/"> From the Bottom of the Heap - R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>
One of my more popular <a target="_blank" rel="nofollow" href="http://stackoverflow.com/a/14424417/429846">answers</a> on StackOverflow concerns the issue of prediction intervals for a generalized linear model (GLM). Comments, even on StackOverflow, aren’t a good place for a discussion so I thought I’d post something hereon my blog that went into a bit more detail as to why, for some common types of GLMs, prediction intervals aren’t that useful and require a lot more thinking about what they mean and how they should be calculated. I’ve broken it into two and in this, the second part, I look at Possion models.
</p>
<p>
The second example — purely because I happen to have it handy from teaching this semester — is from <span data-cites="Korner-Nievergelt2015-tk">Korner-Nievergelt et al. (2015)</span>, and concerns the number of breeding pairs of the common whitethroat (<em>Silvia communis</em>). This species likes to inhabit field margins and fallow lands and has been adversely affected by intensive agricultural activities reducing these types of habitat on the landscape. As a mitigiation effort, wildflower fields are sown and left largely unmanaged for several years. The data come from a study looking at how the number of breeding pairs of common whitethroat change as the composition and structure of the plant community changes over time. The data are in the <strong>blmeco</strong> package available on CRAN.
</p>
<figure>
<pre><code data-lang="r"><span>## install.packages(&quot;blmeco&quot;) # first, if not already installed
</span><span>library</span><span>(</span><span>&quot;blmeco&quot;</span><span>)</span>
<span>data</span><span>(</span><span>wildflowerfields</span><span>)</span>
<span>library</span><span>(</span><span>&quot;ggplot2&quot;</span><span>)</span>
<span>theme_set</span><span>(</span><span>theme_bw</span><span>())</span></code></pre>
</figure>
<p>
The example in <span data-cites="Korner-Nievergelt2015-tk">Korner-Nievergelt et al. (2015)</span> uses a Poisson GLM with a quadratic effect of the variable <code>age</code>. Instead I’ll use a Poisson GAM, but in all other respects the analysis follows that from the text book (only the year 2007 data are used, field size transformed to hectares).
</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>&quot;mgcv&quot;</span><span>)</span>
<span>wf</span> <span>&lt;-</span> <span>transform</span><span>(</span><span>subset</span><span>(</span><span>wildflowerfields</span><span>,</span> <span>year</span> <span>==</span> <span>2007</span><span>),</span> <span>size</span> <span>=</span> <span>size</span> <span>/</span> <span>100</span><span>)</span>
<span>wf</span> <span>&lt;-</span> <span>transform</span><span>(</span><span>wf</span><span>,</span> <span>size.z</span> <span>=</span> <span>(</span><span>size</span> <span>-</span> <span>mean</span><span>(</span><span>size</span><span>))</span> <span>/</span> <span>sd</span><span>(</span><span>size</span><span>))</span>
<span>mod</span> <span>&lt;-</span> <span>gam</span><span>(</span><span>bp</span> <span>~</span> <span>s</span><span>(</span><span>age</span><span>,</span> <span>k</span> <span>=</span> <span>6</span><span>)</span> <span>+</span> <span>size.z</span> <span>+</span> <span>offset</span><span>(</span><span>log</span><span>(</span><span>size</span><span>)),</span> <span>data</span> <span>=</span> <span>wf</span><span>,</span> <span>family</span> <span>=</span> <span>poisson</span><span>,</span> <span>method</span> <span>=</span> <span>&quot;REML&quot;</span><span>)</span>
<span>summary</span><span>(</span><span>mod</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">Family: poisson 
Link function: log 

Formula:
bp ~ s(age, k = 6) + size.z + offset(log(size))

Parametric coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept)  -1.1791     0.3333  -3.538 0.000404 ***
size.z       -0.5283     0.2893  -1.826 0.067861 .  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Approximate significance of smooth terms:
         edf Ref.df Chi.sq p-value  
s(age) 2.608  3.223  7.323   0.074 .
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

R-sq.(adj) =  0.207   Deviance explained = 46.3%
-REML = 33.589  Scale est. = 1         n = 41</code></pre>
</figure>
<p>
The primary variable of interest shows a moderate amount of non-linearity, similar to that of the quadratic effect of <code>age</code> in the version from the text book, though the effect of field age is weak at best. The fitted model is illustrated graphically below, holding <code>size</code> constant at the mean field size
</p>
<figure>
<pre><code data-lang="r"><span>ilink</span> <span>&lt;-</span> <span>family</span><span>(</span><span>mod</span><span>)</span><span>$</span><span>linkinv</span>
<span>newd</span> <span>&lt;-</span> <span>with</span><span>(</span><span>wf</span><span>,</span> <span>expand.grid</span><span>(</span><span>age</span> <span>=</span> <span>seq</span><span>(</span><span>min</span><span>(</span><span>age</span><span>),</span> <span>max</span><span>(</span><span>age</span><span>),</span> <span>length</span> <span>=</span> <span>300</span><span>),</span>
                             <span>size</span> <span>=</span> <span>1</span><span>,</span> <span>size.z</span> <span>=</span> <span>0</span><span>))</span>
<span>newd</span> <span>&lt;-</span> <span>cbind</span><span>(</span><span>newd</span><span>,</span> <span>as.data.frame</span><span>(</span><span>predict</span><span>(</span><span>mod</span><span>,</span> <span>newd</span><span>,</span> <span>type</span> <span>=</span> <span>&quot;link&quot;</span><span>,</span> <span>se.fit</span> <span>=</span> <span>TRUE</span><span>)))</span>
<span>newd</span> <span>&lt;-</span> <span>transform</span><span>(</span><span>newd</span><span>,</span> <span>fitted</span> <span>=</span> <span>ilink</span><span>(</span><span>fit</span><span>),</span> <span>upper</span> <span>=</span> <span>ilink</span><span>(</span><span>fit</span> <span>+</span> <span>(</span><span>2</span> <span>*</span> <span>se.fit</span><span>)),</span>
                  <span>lower</span> <span>=</span> <span>ilink</span><span>(</span><span>fit</span> <span>-</span> <span>(</span><span>2</span> <span>*</span> <span>se.fit</span><span>)))</span>
<span>ggplot</span><span>(</span><span>wf</span><span>,</span> <span>aes</span><span>(</span><span>x</span> <span>=</span> <span>age</span><span>,</span> <span>y</span> <span>=</span> <span>bp</span><span>/</span><span>size</span><span>))</span> <span>+</span>
    <span>geom_ribbon</span><span>(</span><span>data</span> <span>=</span> <span>newd</span><span>,</span> <span>mapping</span> <span>=</span> <span>aes</span><span>(</span><span>ymin</span> <span>=</span> <span>lower</span><span>,</span> <span>ymax</span> <span>=</span> <span>upper</span><span>,</span> <span>x</span> <span>=</span> <span>age</span><span>),</span>
                <span>alpha</span> <span>=</span> <span>0.1</span><span>,</span> <span>inherit.aes</span> <span>=</span> <span>FALSE</span><span>)</span> <span>+</span>
    <span>geom_line</span><span>(</span><span>data</span> <span>=</span> <span>newd</span><span>,</span> <span>mapping</span> <span>=</span> <span>aes</span><span>(</span><span>y</span> <span>=</span> <span>fitted</span><span>))</span> <span>+</span>
    <span>geom_jitter</span><span>(</span><span>width</span> <span>=</span> <span>0.1</span><span>,</span> <span>height</span> <span>=</span> <span>0</span><span>)</span> <span>+</span>
    <span>labs</span><span>(</span><span>x</span> <span>=</span> <span>&quot;Age [years]&quot;</span><span>,</span> <span>y</span> <span>=</span> <span>expression</span><span>(</span><span>&quot;Number of Breeding Pairs [&quot;</span> <span>*</span> <span>pairs</span> <span>~</span> <span>ha</span><span>^</span><span>{</span><span>-1</span><span>}</span> <span>*</span> <span>&quot;]&quot;</span><span>),</span>
         <span>title</span> <span>=</span> <span>&quot;Common Whitethroat densities in Wildflower Fields&quot;</span><span>,</span>
         <span>subtitle</span> <span>=</span> <span>&quot;Estimated densities for average field of ~1.8 ha&quot;</span><span>)</span></code></pre>
</figure>
<figure>
<img alt="The fitted GAM for the common whitethroat data, showing the estimated number of breeding pairs per hectare with a 95% pointwise confidence interval. The points are the observed densities of breeding pairs." src="https://i2.wp.com/www.fromthebottomoftheheap.net/assets/img/posts/glm-prediction-intervals-ii-plot-wildflowers-1.png?w=456" data-recalc-dims="1">
<figcaption>
The fitted GAM for the common whitethroat data, showing the estimated number of breeding pairs per hectare with a 95% pointwise confidence interval. The points are the observed densities of breeding pairs.<br>
</figcaption>
</figure>
<p>
So far so good, but how do we interpret this model? For simplicity, lets assume that fields only come in integer ages. What the model implies is that for each integer age the observations are best fitted by (or described by; or generated from) a Poisson model with parameter <span>()</span> equal to the value of the solid line at each particular age. This value of <span>()</span> is just an estimate of the true value and so we might envisage the observations for each year as having come from Poisson distributions with values of <span>()</span> given by the values of the upper and lower confidence band also shown in the figure above. For fields of two and five years of age these distributions look like this
</p>
<figure>
<img src="https://i0.wp.com/www.fromthebottomoftheheap.net/assets/img/posts/glm-prediction-intervals-ii-implied-poisson-distributions-1.png?w=456" data-recalc-dims="1"><br>
</figure>
<p>
The fitted Poisson distributions for the two field ages are shown by the green points and lines in the figure above. The effect of field age is to shift the estimated Poisson distribution to the right, towards on average higher numbers of breeding pairs. The uncertainty in the estimated model is shown by the orange and blue points and lines; these are based on the lower and upper 95% pointwise confidence interval on the estimate mean number of breeding pairs for fields of two and five years of age. The orange points illustrate the Poisson distribution from which the points might have been derived if the true value of <span>()</span> were at the lower end of the confidence interval. The blue points show the Poisson distribution if the true values of <span>()</span> was at the upper end of the confidence interval. Each of these distributions implies, potentially at least, different predicted numbers of breeding pairs.
</p>
<p>
We have estimated the expected number of breeding pairs given the age of the field and it’s size. We also have a (pointwise) 95% confidence interval on that expectation. As before, this isn’t a prediction interval, so what would one of those look like in this case? Somewhat similar to those we created for the binomial GLM earlier, except now we have posterior densities (the probability density implied by the Poisson distribution with <span>()</span> given as a function of field age) for all the integers 0–∞, although once we get above 10 breeding pairs the density is going to be effectively 0 even if not technically so.
</p>
<p>
Note that I said integers above; we can’t have 2.5 breeding pairs <em>as a prediction</em>. Hence any prediction interval is really talking about points of probability for each integer <span>({0, 1, 2, })</span> (even if we might consider a much smaller upper limit than that) not a continuous interval. Having said that, perhaps I’m being to pedantic? In some instances, the upper and lower 2.5<sup>th</sup> and 97.5<sup>th</sup> probability quantiles of the implied Poisson distribution do begin to look more like a prediction interval.
</p>
<p>
To illustrate I’ll work my way through some code to illustrate some ways of thinking about what the fitted model says in terms of predicting the numbers of breeding pairs of common whitethroats. First a little bit of prep; I’ll illustrate various intervals for two hypothetical fields or average size, one created two years ago and a second five years ago.
</p>
<figure>
<pre><code data-lang="r"><span>p</span> <span>&lt;-</span> <span>data.frame</span><span>(</span><span>age</span> <span>=</span> <span>c</span><span>(</span><span>2</span><span>,</span> <span>5</span><span>),</span> <span>size</span> <span>=</span> <span>1</span><span>,</span> <span>size.z</span> <span>=</span> <span>0</span><span>)</span>
<span>pred</span> <span>&lt;-</span> <span>setNames</span><span>(</span><span>as.data.frame</span><span>(</span><span>predict</span><span>(</span><span>mod</span><span>,</span> <span>p</span><span>,</span> <span>se.fit</span> <span>=</span> <span>TRUE</span><span>)),</span> <span>c</span><span>(</span><span>&quot;fit&quot;</span><span>,</span> <span>&quot;se&quot;</span><span>))</span>
<span>p</span> <span>&lt;-</span> <span>cbind</span><span>(</span><span>p</span><span>,</span> <span>pred</span><span>)</span>
<span>p</span> <span>&lt;-</span> <span>transform</span><span>(</span><span>p</span><span>,</span> <span>lower</span> <span>=</span> <span>ilink</span><span>(</span><span>fit</span> <span>-</span> <span>(</span><span>2</span> <span>*</span> <span>se</span><span>)),</span>
               <span>lambda</span> <span>=</span> <span>ilink</span><span>(</span><span>fit</span><span>),</span> <span>upper</span> <span>=</span> <span>ilink</span><span>(</span><span>fit</span> <span>+</span> <span>(</span><span>2</span> <span>*</span> <span>se</span><span>)))</span>
<span>p</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">  age size size.z        fit        se      lower    lambda     upper
1   2    1      0 -1.7184424 0.5805902 0.05615594 0.1793453 0.5727752
2   5    1      0 -0.1295676 0.3260103 0.45767855 0.8784752 1.6861589</code></pre>
</figure>
<p>
<code>p</code> contains the estimated value of <span>()</span> (the expected number of breeding pairs), and the upper and lower 95% pointwise interval about this expected count, for the two fields. First, the 95% interval for the model estimated <span>()</span> for the younger of the two fields, based on <code>qpois()</code>, the quantile function of the conditional distribution of the number of breeding pairs given field age
</p>
<figure>
<pre><code data-lang="r"><span>qpois</span><span>(</span><span>c</span><span>(</span><span>0.025</span><span>,</span> <span>0.975</span><span>),</span> <span>lambda</span> <span>=</span> <span>p</span><span>[</span><span>1</span><span>,</span> <span>&quot;lambda&quot;</span><span>])</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">[1] 0 1</code></pre>
</figure>
<p>
Hence we might expect either 0, 1 breeding pairs. But, we haven’t accounted for the uncertainty in the estimated <span>()</span>. At the lower end of of the 95% interval on the estimated <span>()</span> the prediction interval would be
</p>
<figure>
<pre><code data-lang="r"><span>qpois</span><span>(</span><span>c</span><span>(</span><span>0.025</span><span>,</span> <span>0.975</span><span>),</span> <span>lambda</span> <span>=</span> <span>p</span><span>[</span><span>1</span><span>,</span> <span>&quot;lower&quot;</span><span>])</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">[1] 0 1</code></pre>
</figure>
<p>
and
</p>
<figure>
<pre><code data-lang="r"><span>qpois</span><span>(</span><span>c</span><span>(</span><span>0.025</span><span>,</span> <span>0.975</span><span>),</span> <span>lambda</span> <span>=</span> <span>p</span><span>[</span><span>1</span><span>,</span> <span>&quot;upper&quot;</span><span>])</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">[1] 0 2</code></pre>
</figure>
<p>
for the upper end, leading to a prediction interval of 0–2 breeding pairs. The same prediction interval for the five year old field would be
</p>
<figure>
<pre><code data-lang="r"><span>qpois</span><span>(</span><span>c</span><span>(</span><span>0.025</span><span>,</span> <span>0.975</span><span>),</span> <span>lambda</span> <span>=</span> <span>c</span><span>(</span><span>p</span><span>[</span><span>2</span><span>,</span> <span>&quot;lower&quot;</span><span>],</span> <span>p</span><span>[</span><span>2</span><span>,</span> <span>&quot;upper&quot;</span><span>]))</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">[1] 0 5</code></pre>
</figure>
<p>
We can also look at the probability densities of the poisson distribution for the estimated value of <span>()</span> and its 95% confidence interval. The table below shows these probability densities for a two-year old field
</p>
<table>
<caption>
Posterior densities for selected numbers of breeding pairs for a two year old field. Columns show the densities for a poisson distribution with <span>()</span> equal to the esimated value and the lower and upper limits on the estimated values for this field.<br>
</caption>
<thead>
<tr>
<th>
# of pairs
</th>
<th>
lower
</th>
<th>
estimate
</th>
<th>
upper
</th>
</tr>
</thead>
<tbody>
<tr>
<td>
0
</td>
<td>
0.9454
</td>
<td>
0.8358
</td>
<td>
0.5640
</td>
</tr>
<tr>
<td>
1
</td>
<td>
0.0531
</td>
<td>
0.1499
</td>
<td>
0.3230
</td>
</tr>
<tr>
<td>
2
</td>
<td>
0.0015
</td>
<td>
0.0134
</td>
<td>
0.0925
</td>
</tr>
<tr>
<td>
3
</td>
<td>
0.0000
</td>
<td>
0.0008
</td>
<td>
0.0177
</td>
</tr>
<tr>
<td>
4
</td>
<td>
0.0000
</td>
<td>
0.0000
</td>
<td>
0.0025
</td>
</tr>
<tr>
<td>
5
</td>
<td>
0.0000
</td>
<td>
0.0000
</td>
<td>
0.0003
</td>
</tr>
<tr>
<td>
6
</td>
<td>
0.0000
</td>
<td>
0.0000
</td>
<td>
0.0000
</td>
</tr>
<tr>
<td>
7
</td>
<td>
0.0000
</td>
<td>
0.0000
</td>
<td>
0.0000
</td>
</tr>
</tbody>
</table>
<p>
For example, we’d expect to observe no breeding pairs in 95–56% of average sized, two-year old fields. The same values are shown for a five-year old tree in the table below
</p>
<table>
<caption>
Posterior densities for selected numbers of breeding pairs for a five year old field. Columns show the densities for a poisson distribution with <span>()</span> equal to the esimated value and the lower and upper limits on the estimated values for this field.<br>
</caption>
<thead>
<tr>
<th>
# of pairs
</th>
<th>
lower
</th>
<th>
estimate
</th>
<th>
upper
</th>
</tr>
</thead>
<tbody>
<tr>
<td>
0
</td>
<td>
0.6328
</td>
<td>
0.4154
</td>
<td>
0.1852
</td>
</tr>
<tr>
<td>
1
</td>
<td>
0.2896
</td>
<td>
0.3649
</td>
<td>
0.3123
</td>
</tr>
<tr>
<td>
2
</td>
<td>
0.0663
</td>
<td>
0.1603
</td>
<td>
0.2633
</td>
</tr>
<tr>
<td>
3
</td>
<td>
0.0101
</td>
<td>
0.0469
</td>
<td>
0.1480
</td>
</tr>
<tr>
<td>
4
</td>
<td>
0.0012
</td>
<td>
0.0103
</td>
<td>
0.0624
</td>
</tr>
<tr>
<td>
5
</td>
<td>
0.0001
</td>
<td>
0.0018
</td>
<td>
0.0210
</td>
</tr>
<tr>
<td>
6
</td>
<td>
0.0000
</td>
<td>
0.0003
</td>
<td>
0.0059
</td>
</tr>
<tr>
<td>
7
</td>
<td>
0.0000
</td>
<td>
0.0000
</td>
<td>
0.0014
</td>
</tr>
</tbody>
</table>
<p></p>
<p>
I could repeat the process of simulating breeding pairs from the poisson distributions with estimated values of <span>()</span> but the code to illustrate this gets tedious and this post is long enough already.
</p>
<p>
The prediction intervals for the Poisson model are starting to look more like intervals than the ones for the binomial model we looked at earlier. They’re still not something we can easily convey on a plot like we can with linear models and <code>predict.lm</code>, however.
</p>
<p>
For continous conditional distributions, prediction “intervals” act like their linear model counterparts, as long as we take the extra step of computing the prediction interval using the probability quantile function (the <code>qfoo()</code> functions in R where <code>foo</code> is the abbreviation for the distribution) and potentially include the uncertainty in the estimated expectations (fitted values on the response scale) as we did in both examples above.
</p>
<p>
Ok, I think that’s enough modelling pedantry for one [Ed: er-um, <em>two</em>] post.
</p>
<div>
<h3>
References<br>
</h3>
<div>
<p>
Korner-Nievergelt, F., Felten, S. von, Roth, T., Gulat, J., Almasi, B., and Korner-Nievergelt, P. (2015). <em>Bayesian data analysis in ecology using linear models with R, BUGS, and stan</em>. Elsevier Science &amp; Technology Books.
</p>
</div>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.fromthebottomoftheheap.net/2017/05/01/glm-prediction-intervals-ii/"> From the Bottom of the Heap - R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/cb9IuXYvMH4"></a></div>