<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/sub-gaussian-property-for-the-beta-distribution-part-3-final/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://statisfaction.wordpress.com/2017/12/26/sub-gaussian-property-for-the-beta-distribution-part-3-final/"> R – Statisfaction</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div data-shortcode="caption" class="wp-caption"><img data-orig-file="https://statisfaction.files.wordpress.com/2017/12/beta_to_bernoulli.gif?w=456" data-orig-size="400,400" data-medium-file="https://statisfaction.files.wordpress.com/2017/12/beta_to_bernoulli.gif?w=456?w=400" data-attachment-id="6984" data-image-meta="{" data-image-description="" src="https://statisfaction.files.wordpress.com/2017/12/beta_to_bernoulli.gif?w=456" data-permalink="https://statisfaction.wordpress.com/2017/12/26/sub-gaussian-property-for-the-beta-distribution-part-3-final/beta_to_bernoulli/" data-large-file="https://statisfaction.files.wordpress.com/2017/12/beta_to_bernoulli.gif?w=456?w=400" alt="beta_to_bernoulli" data-image-title="beta_to_bernoulli" data-comments-opened="1" class="wp-image-6984" data-recalc-dims="1">
<p class="wp-caption-text">When a <strong><span><span>Beta</span> random <span>variable</span></span></strong> wants to act like a <strong><span>Bernoulli</span></strong>: convergence of optimal proxy variance.</p>
</div>
<p>In this third and last post about the Sub-Gaussian property for the Beta distribution [1] (<a rel="nofollow" href="https://statisfaction.wordpress.com/2017/05/02/sub-gaussian-property-for-the-beta-distribution-part-1/" target="_blank">post 1</a> and <a rel="nofollow" href="https://statisfaction.wordpress.com/2017/12/20/sub-gaussian-property-for-the-beta-distribution-part-2/" target="_blank">post 2</a>), I would like to show the interplay with the Bernoulli distribution as well as some connexions with optimal transport (OT is a hot topic in general, and also on this blog with Pierre’s posts on <a rel="nofollow" href="https://statisfaction.wordpress.com/?s=wasserstein" target="_blank">Wasserstein ABC</a>).<span></span></p>
<p>Let us see how sub-Gaussian proxy variances can be derived from transport inequalities. To this end, we need first to introduce the <strong>Wasserstein distance</strong> (of order 1) between two probability measures <em>P</em> and <em> Q</em> on a space <img alt="mathcal{X}" title="mathcal{X}" src="https://s0.wp.com/latex.php?latex=%5Cmathcal%7BX%7D&bg=fff&%23038;fg=222&%23038;s=0" class="latex">. It is defined wrt a distance <em>d</em> on <img alt="mathcal{X}" title="mathcal{X}" src="https://s0.wp.com/latex.php?latex=%5Cmathcal%7BX%7D&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> by</p>
<p><img alt="W(P,Q)=inf_{piinPi(P,Q)}int_{mathcal{X}timesmathcal{X}}d(x,y)pi(text{d}x,text{d}y)," title="W(P,Q)=inf_{piinPi(P,Q)}int_{mathcal{X}timesmathcal{X}}d(x,y)pi(text{d}x,text{d}y)," src="https://s0.wp.com/latex.php?latex=W%28P%2CQ%29%3D%5Cinf_%7B%5Cpi%5Cin%5CPi%28P%2CQ%29%7D%5Cint_%7B%5Cmathcal%7BX%7D%5Ctimes%5Cmathcal%7BX%7D%7Dd%28x%2Cy%29%5Cpi%28%5Ctext%7Bd%7Dx%2C%5Ctext%7Bd%7Dy%29%2C&bg=fff&%23038;fg=222&%23038;s=0" class="latex"></p>
<p>where <img alt="Pi(P,Q)" title="Pi(P,Q)" src="https://s0.wp.com/latex.php?latex=%5CPi%28P%2CQ%29&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> is the set of probability measures on <img alt="mathcal{X}times mathcal{X}" title="mathcal{X}times mathcal{X}" src="https://s0.wp.com/latex.php?latex=%5Cmathcal%7BX%7D%5Ctimes+%5Cmathcal%7BX%7D&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> with fixed marginal distributions respectively <img alt="P" title="P" src="https://s0.wp.com/latex.php?latex=P&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> and <img alt="Q." title="Q." src="https://s0.wp.com/latex.php?latex=Q.&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> Then, a probability measure <img alt="P" title="P" src="https://s0.wp.com/latex.php?latex=P&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> is said to satisfy a <strong>transport inequality</strong> with positive constant <img alt="sigma" title="sigma" src="https://s0.wp.com/latex.php?latex=%5Csigma&bg=fff&%23038;fg=222&%23038;s=0" class="latex">, if for any probability measure <img alt="Q" title="Q" src="https://s0.wp.com/latex.php?latex=Q&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> dominated by <img alt="P" title="P" src="https://s0.wp.com/latex.php?latex=P&bg=fff&%23038;fg=222&%23038;s=0" class="latex">,</p>
<p><img alt="W(P,Q) leqsigmasqrt{2 D(Q||P)}," title="W(P,Q) leqsigmasqrt{2 D(Q||P)}," src="https://s0.wp.com/latex.php?latex=W%28P%2CQ%29+%5Cleq%5Csigma%5Csqrt%7B2+D%28Q%7C%7CP%29%7D%2C&bg=fff&%23038;fg=222&%23038;s=0" class="latex"></p>
<p>where <img alt="D(Q||P)" title="D(Q||P)" src="https://s0.wp.com/latex.php?latex=D%28Q%7C%7CP%29&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> is the entropy, or Kullback–Leibler divergence, between <img alt="P" title="P" src="https://s0.wp.com/latex.php?latex=P&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> and <img alt="Q" title="Q" src="https://s0.wp.com/latex.php?latex=Q&bg=fff&%23038;fg=222&%23038;s=0" class="latex">. The nice result proven by Bobkov and Götze (1999) [2] is that the constant <img alt="sigma^2" title="sigma^2" src="https://s0.wp.com/latex.php?latex=%5Csigma%5E2&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> is a sub-Gaussian proxy variance for <em>P</em>.</p>
<p>For a discrete space <img alt="mathcal{X}" title="mathcal{X}" src="https://s0.wp.com/latex.php?latex=%5Cmathcal%7BX%7D&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> equipped with the Hamming metric, <img alt="d(x,y) = mathbf{1}_{{xneq y}}" title="d(x,y) = mathbf{1}_{{xneq y}}" src="https://s0.wp.com/latex.php?latex=d%28x%2Cy%29+%3D+%5Cmathbf%7B1%7D_%7B%5C%7Bx%5Cneq+y%5C%7D%7D&bg=fff&%23038;fg=222&%23038;s=0" class="latex">, the induced Wasserstein distance reduces to the total variation distance, <img alt="W(P,Q) = Vert P-QVert_{text{TV}}" title="W(P,Q) = Vert P-QVert_{text{TV}}" src="https://s0.wp.com/latex.php?latex=W%28P%2CQ%29+%3D+%5CVert+P-Q%5CVert_%7B%5Ctext%7BTV%7D%7D&bg=fff&%23038;fg=222&%23038;s=0" class="latex">. In that setting, Ordentlich and Weinberger (2005) [3] proved the distribution-sensitive transport inequality:</p>
<p><img alt="Vert P-QVert_{text{TV}} leq sqrt{frac{1}{g(mu_P)}D(Q||P)}," title="Vert P-QVert_{text{TV}} leq sqrt{frac{1}{g(mu_P)}D(Q||P)}," src="https://s0.wp.com/latex.php?latex=%5CVert+P-Q%5CVert_%7B%5Ctext%7BTV%7D%7D+%5Cleq+%5Csqrt%7B%5Cfrac%7B1%7D%7Bg%28%5Cmu_P%29%7DD%28Q%7C%7CP%29%7D%2C&bg=fff&%23038;fg=222&%23038;s=0" class="latex"></p>
<p>where the function <img alt="g" title="g" src="https://s0.wp.com/latex.php?latex=g&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> is defined by <img alt="g(mu)=frac{1}{1-2mu}lnfrac{1-mu}{mu}" title="g(mu)=frac{1}{1-2mu}lnfrac{1-mu}{mu}" src="https://s0.wp.com/latex.php?latex=g%28%5Cmu%29%3D%5Cfrac%7B1%7D%7B1-2%5Cmu%7D%5Cln%5Cfrac%7B1-%5Cmu%7D%7B%5Cmu%7D&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> and the coefficient <img alt="mu_P" title="mu_P" src="https://s0.wp.com/latex.php?latex=%5Cmu_P&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> is called the balance coefficient of <img alt="P" title="P" src="https://s0.wp.com/latex.php?latex=P&bg=fff&%23038;fg=222&%23038;s=0" class="latex">, and is defined by <img alt="mu_P=underset{Asubset mathcal{X}}maxmin{P(A),1-P(A)}" title="mu_P=underset{Asubset mathcal{X}}maxmin{P(A),1-P(A)}" src="https://s0.wp.com/latex.php?latex=%5Cmu_P%3D%5Cunderset%7BA%5Csubset+%5Cmathcal%7BX%7D%7D%5Cmax%5Cmin%5C%7BP%28A%29%2C1-P%28A%29%5C%7D&bg=fff&%23038;fg=222&%23038;s=0" class="latex">. In particular, the Bernoulli balance coefficient is easily shown to coincide with its mean. Hence, applying the result of Bobkov and Götze (1999) [2] to the above transport inequality yields a distribution-sensitive proxy variance of <img alt="frac{1}{2g(mu)}=frac{1-2mu}{2ln((1-mu)/mu)}" title="frac{1}{2g(mu)}=frac{1-2mu}{2ln((1-mu)/mu)}" src="https://s0.wp.com/latex.php?latex=%5Cfrac%7B1%7D%7B2g%28%5Cmu%29%7D%3D%5Cfrac%7B1-2%5Cmu%7D%7B2%5Cln%28%281-%5Cmu%29%2F%5Cmu%29%7D&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> for the <span>Bernoulli</span> with mean <img alt="mu" title="mu" src="https://s0.wp.com/latex.php?latex=%5Cmu&bg=fff&%23038;fg=222&%23038;s=0" class="latex">, as <span>plotted in blue</span> above.</p>
<p>In the Beta distribution case, we have not been able to extend this transport inequality methodology since the support is not discrete. However, a nice limiting argument holds. Consider a sequence of Beta<img alt="(alpha,beta)" title="(alpha,beta)" src="https://s0.wp.com/latex.php?latex=%28%5Calpha%2C%5Cbeta%29&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> random variables with fixed mean <img alt="mu=frac{alpha}{alpha+beta}" title="mu=frac{alpha}{alpha+beta}" src="https://s0.wp.com/latex.php?latex=%5Cmu%3D%5Cfrac%7B%5Calpha%7D%7B%5Calpha%2B%5Cbeta%7D&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> and with a sum <img alt="alpha+beta" title="alpha+beta" src="https://s0.wp.com/latex.php?latex=%5Calpha%2B%5Cbeta&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> going to zero. This converges to a Bernoulli random variable with mean <img alt="mu" title="mu" src="https://s0.wp.com/latex.php?latex=%5Cmu&bg=fff&%23038;fg=222&%23038;s=0" class="latex">, and we have shown that the limiting <span>optimal proxy variance of such a sequence of Beta with decreasing sum</span> <img alt="alpha+beta" title="alpha+beta" src="https://s0.wp.com/latex.php?latex=%5Calpha%2B%5Cbeta&bg=fff&%23038;fg=222&%23038;s=0" class="latex"> is the one of the <span>Bernoulli</span>.</p>
<h4>References</h4>
<p>[1] Marchal, O. and Arbel, J. (2017), <a rel="nofollow" href="https://projecteuclid.org/euclid.ecp/1507860211" target="_blank">On the sub-Gaussianity of the Beta and Dirichlet distributions</a>. Electronic Communications in Probability, 22:1–14, 2017. Code on <a rel="nofollow" href="https://github.com/jarbel/Beta-sub-Gaussian" target="_blank">GitHub</a>.<br>
[2] Bobkov, S. G. and Götze, F. (1999). Exponential integrability and transportation cost related to logarithmic Sobolev inequalities. Journal of Functional Analysis, 163(1):1–28.<br>
[3] Ordentlich, E. and Weinberger, M. J. (2005). A distribution dependent refinement of Pinsker’s inequality. IEEE Transactions on Information Theory, 51(5):1836–1840.</p>
<p> <a rel="nofollow" href="https://feeds.wordpress.com/1.0/gocomments/statisfaction.wordpress.com/6983/" target="_blank"><img border="0" alt="" src="https://feeds.wordpress.com/1.0/comments/statisfaction.wordpress.com/6983/"></a> <img border="0" alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" data-recalc-dims="1"></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://statisfaction.wordpress.com/2017/12/26/sub-gaussian-property-for-the-beta-distribution-part-3-final/"> R – Statisfaction</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/3EHI9orSYnY" height="1"></a></div>