<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/sliced-wasserstein-estimation-of-mixtures/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://xianblog.wordpress.com/2017/11/28/sliced-wasserstein-estimation-of-mixtures/"> R – Xi'an's Og</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><strong><a rel="nofollow" href="https://arxiv.org/abs/1711.05376" target="_blank"><img data-orig-file="https://xianblog.files.wordpress.com/2017/11/puzler.png?w=450&amp;h=340&amp;fit=450%2C340&amp;resize=450%2C340" data-orig-size="710,537" data-medium-file="https://xianblog.files.wordpress.com/2017/11/puzler.png?w=450&amp;h=340&amp;fit=450%2C340&amp;resize=450%2C340?w=300" data-attachment-id="34868" data-image-meta="{" data-image-description="" src="https://xianblog.files.wordpress.com/2017/11/puzler.png?w=450&h=340&fit=450%2C340&resize=450%2C340" data-permalink="https://xianblog.wordpress.com/2017/11/28/sliced-wasserstein-estimation-of-mixtures/puzler/" data-large-file="https://xianblog.files.wordpress.com/2017/11/puzler.png?w=450&amp;h=340&amp;fit=450%2C340&amp;resize=450%2C340?w=450" alt="" data-image-title="puzler" sizes="(max-width: 450px) 100vw, 450px" data-comments-opened="1" class="wp-image-34868" data-recalc-dims="1"></a>A</strong> <a rel="nofollow" href="https://arxiv.org/abs/1711.05376" target="_blank">paper</a> by Soheil Kolouri and co-authors was <a rel="nofollow" href="https://arxiv.org/abs/1711.05376" target="_blank">arXived</a> last week about using Wasserstein distance for inference on multivariate Gaussian mixtures. The basic concept is that the parameter is estimated by minimising the p-Wasserstein distance to the empirical distribution, smoothed by a Normal kernel. As the general Wasserstein distance is quite costly to compute, the approach relies on a sliced version, which means computing the Wasserstein distance between one-dimensional projections of the distributions. Optimising over the directions is an additional computational constraint.</p>
<blockquote>
<p><em><span>“To fit a finite GMM to the observed data, one is required to answer the following questions: 1) how to estimate the number of mixture components needed to represent the data, and 2) how to estimate the parameters of the mixture components.”</span></em></p>
</blockquote>
<p>The <a rel="nofollow" href="https://arxiv.org/abs/1711.05376" target="_blank">paper</a> contains a most puzzling comment opposing maximum likelihood estimation to minimum Wasserstein distance estimation on the basis that the later would not suffer from multimodality. This sounds incorrect as the multimodality of a mixture model (likelihood) stems from the lack of identifiability of the parameters. If all permutations of these parameters induce exactly the same distribution, they all stand at the same distance from the data distribution, whatever the distance is. Furthermore, the above tartan-like picture clashes with the representation of the log-likelihood of a Normal mixture, as exemplified by the picture below based on a 150 sample with means 0 and 2, same unit variance, and weights 0.3 and 0.7, which shows a smooth if bimodal structure:<a rel="nofollow" href="https://xianblog.files.wordpress.com/2017/11/mixlik.jpg" target="_blank"><img data-orig-file="https://xianblog.files.wordpress.com/2017/11/mixlik.jpg?w=450&amp;h=450&amp;fit=450%2C450&amp;resize=450%2C450" data-orig-size="480,480" data-medium-file="https://xianblog.files.wordpress.com/2017/11/mixlik.jpg?w=450&amp;h=450&amp;fit=450%2C450&amp;resize=450%2C450?w=300" data-attachment-id="34869" data-image-meta="{" data-image-description="" src="https://xianblog.files.wordpress.com/2017/11/mixlik.jpg?w=450&h=450&fit=450%2C450&resize=450%2C450" data-permalink="https://xianblog.wordpress.com/2017/11/28/sliced-wasserstein-estimation-of-mixtures/mixlik/" data-large-file="https://xianblog.files.wordpress.com/2017/11/mixlik.jpg?w=450&amp;h=450&amp;fit=450%2C450&amp;resize=450%2C450?w=450" alt="" data-image-title="mixlik" sizes="(max-width: 450px) 100vw, 450px" data-comments-opened="1" class="wp-image-34869" data-recalc-dims="1"></a>And for the same dataset, my attempt at producing a Wasserstein “energy landscape” does return a multimodal structure (this is the surface of minus the logarithm of the 2-Wasserstein distance):<a rel="nofollow" href="https://xianblog.files.wordpress.com/2017/11/mixtein.jpg" target="_blank"><img data-orig-file="https://xianblog.files.wordpress.com/2017/11/mixtein.jpg?w=450&amp;h=450&amp;fit=450%2C450&amp;resize=450%2C450" data-orig-size="480,480" data-medium-file="https://xianblog.files.wordpress.com/2017/11/mixtein.jpg?w=450&amp;h=450&amp;fit=450%2C450&amp;resize=450%2C450?w=300" data-attachment-id="34870" data-image-meta="{" data-image-description="" src="https://xianblog.files.wordpress.com/2017/11/mixtein.jpg?w=450&h=450&fit=450%2C450&resize=450%2C450" data-permalink="https://xianblog.wordpress.com/2017/11/28/sliced-wasserstein-estimation-of-mixtures/mixtein/" data-large-file="https://xianblog.files.wordpress.com/2017/11/mixtein.jpg?w=450&amp;h=450&amp;fit=450%2C450&amp;resize=450%2C450?w=450" alt="" data-image-title="mixtein" sizes="(max-width: 450px) 100vw, 450px" data-comments-opened="1" class="wp-image-34870" data-recalc-dims="1"></a><em><span>“Jin et al. proved that with random initialization, the EM algorithm will converge to a bad critical point with high probability.”</span></em></p>
<p>This statement is most curious in that the “probability” in the assessment must depend on the choice of the random initialisation, hence on a sort of prior distribution that is not explicited in the paper. Which remains blissfully unaware of Bayesian approaches.</p>
<p>Another [minor mode] puzzling statement is that the p-Wasserstein distance is defined on the space of probability measures with finite p-th moment, which does not make much sense when what matters is rather the finiteness of the expectation of the distance d(X,Y) raised to the power p. A lot of the maths details either do not make sense or seem superfluous.</p>
<p>Filed under: <a rel="nofollow" href="https://xianblog.wordpress.com/category/books/" target="_blank">Books</a>, <a rel="nofollow" href="https://xianblog.wordpress.com/category/pictures/" target="_blank">pictures</a>, <a rel="nofollow" href="https://xianblog.wordpress.com/category/statistics/r-statistics/" target="_blank">R</a>, <a rel="nofollow" href="https://xianblog.wordpress.com/category/statistics/" target="_blank">Statistics</a> Tagged: <a rel="nofollow" href="https://xianblog.wordpress.com/tag/arxiv/" target="_blank">arXiv</a>, <a rel="nofollow" href="https://xianblog.wordpress.com/tag/em-algorithm/" target="_blank">EM algorithm</a>, <a rel="nofollow" href="https://xianblog.wordpress.com/tag/finite-mixtures/" target="_blank">finite mixtures</a>, <a rel="nofollow" href="https://xianblog.wordpress.com/tag/label-switching/" target="_blank">label switching</a>, <a rel="nofollow" href="https://xianblog.wordpress.com/tag/log-likelihood/" target="_blank">log-likelihood</a>, <a rel="nofollow" href="https://xianblog.wordpress.com/tag/multimodality/" target="_blank">multimodality</a>, <a rel="nofollow" href="https://xianblog.wordpress.com/tag/wasserstein-distance/" target="_blank">Wasserstein distance</a> <a rel="nofollow" href="https://feeds.wordpress.com/1.0/gocomments/xianblog.wordpress.com/34854/" target="_blank"><img border="0" alt="" src="https://feeds.wordpress.com/1.0/comments/xianblog.wordpress.com/34854/"></a> <img border="0" alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" data-recalc-dims="1"></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://xianblog.wordpress.com/2017/11/28/sliced-wasserstein-estimation-of-mixtures/"> R – Xi'an's Og</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/EdP7jR6cgjs" height="1"></a></div>