<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/contours-of-statistical-penalty-functions-as-gif-images/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.alexejgossmann.com/contours_of_statistical_penalty_functions_as_GIFs/"> Alexej's blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Many statistical modeling problems reduce to a minimization problem of the general form:</p>
<p></p>
<p>or</p>
<p></p>
<p>where $f$ is some type of <em>loss function</em>, $\mathbf{X}$ denotes the data, and $g$ is a <em>penalty</em>, also referred to by other names, such as “regularization term” (problems (1) and (2-3) are often equivalent by the way). Of course both, $f$ and $g$, may depend on further parameters.</p>
<p>There are multiple reasons why it can be helpful to check out the contours of such penalty functions $g$:</p>
<ol>
<li>When $\boldsymbol{\beta}$ is two-dimensional, the solution of problem (2-3) can be found by simply taking a look at the contours of $f$ and $g$.</li>
<li>That builds intuition for what happens in more than two dimensions, and in other more general cases.</li>
<li>From a Bayesian point of view, problem (1) can often be interpreted as an <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Maximum_a_posteriori_estimation">MAP</a> estimator, in which case the contours of $g$ are also contours of the prior distribution of $\boldsymbol{\beta}$.</li>
</ol>
<p>Therefore, it is meaningful to visualize the set of points that $g$ maps onto the unit ball in $\mathbb{R}^2$, i.e., the set</p>
<p></p>
<p>Below you see GIF images of such sets $B\subscript{g}$ for various penalty functions $g$ in 2D, capturing the effect of varying certain parameters in $g$. The covered penalty functions include the family of $p$-norms, the elastic net penalty, the fused penalty, and the sorted $\ell_1$ norm.</p>
<p>:white_check_mark: R code to reproduce the GIFs is provided.</p>
<h2>p-norms in 2D</h2>
<p>First we consider the $p$-norm,</p>
<p></p>
<p>with a varying parameter $p \in (0, \infty]$ (which actually isn’t a proper <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Norm_(mathematics)">norm</a> for $p &lt; 1$). Many statistical methods, such as <em>LASSO</em> and <em>Ridge Regression</em>, employ $p$-norm penalties. To find all $\boldsymbol{\beta}$ on the boundary of the 2D unit $p$-norm ball, given $\beta_1$ (the first entry of $\boldsymbol{\beta}$), $\beta_2$ is easily obtained as</p>
<p></p>
<p><img title="p-norm balls" alt="Loading..." src="https://i2.wp.com/www.alexejgossmann.com/images/penalty_function_contours/p-norm_balls.gif?w=456" data-recalc-dims="1"></p>
<p></p>
<h2>Elastic net penalty in 2D</h2>
<p>The elastic net penalty can be written in the form</p>
<p></p>
<p>for $\alpha\in(0,1)$. It is quite popular with a variety of regression-based methods (such as the <em>Elastic Net</em>, of course). We obtain the corresponding 2D unit “ball”, by calculating $\beta\subscript{2}$ from a given $\beta\subscript{1}\in[-1,1]$ as</p>
<p></p>
<p><img title="elastic net balls" alt="Loading..." src="https://i1.wp.com/www.alexejgossmann.com/images/penalty_function_contours/elastic_net_balls.gif?w=456" data-recalc-dims="1"></p>
<h2>Fused penalty in 2D</h2>
<p>The <em>fused</em> penalty can be written in the form</p>
<p></p>
<p>It encourages neighboring coefficients $\beta\subscript{i}$ to have similar values, and is utilized by the <em>fused LASSO</em> and similar methods.</p>
<p><img title="fused penalty" alt="Loading..." src="https://i0.wp.com/www.alexejgossmann.com/images/penalty_function_contours/fused_penalty_balls.gif?w=456" data-recalc-dims="1"></p>
<p>(Here I have simply evaluated the fused penalty function on a grid of points in $[-2,2]^2$, because figuring out equations in parametric form for the above polygons was too painful for my taste… :stuck_out_tongue:)</p>
<h2>Sorted L1 penalty in 2D</h2>
<p>The Sorted $\ell\subscript{1}$ penalty is used in a number of regression-based methods, such as <em>SLOPE</em> and <em>OSCAR</em>. It has the form</p>
<p></p>
<p>where $\lvert \beta \rvert\subscript{(1)} \geq \lvert \beta \rvert\subscript{(2)} \geq \ldots \geq \lvert \beta \rvert\subscript{(m)}$ are the absolute values of the entries of $\boldsymbol{\beta}$ arranged in a decreasing order. In 2D this reduces to</p>
<p></p>
<p><img title="sorted L1 norm balls" alt="Loading..." src="https://i2.wp.com/www.alexejgossmann.com/images/penalty_function_contours/sorted_L1_balls.gif?w=456" data-recalc-dims="1"></p>
<h2>Difference of p-norms</h2>
<p>It holds that</p>
<p></p>
<p>or more generally, for all $p$-norms it holds that</p>
<p></p>
<p>Thus, it is meaningful to define a penalty function of the form</p>
<p></p>
<p>for $\alpha\in[0,1]$, which results in the following.</p>
<p><img title="l1 norm minus l2 norm balls" alt="Loading..." src="https://i0.wp.com/www.alexejgossmann.com/images/penalty_function_contours/l1-l2_balls.gif?w=456" data-recalc-dims="1"></p>
<p>We visualize the same for varying $p \geq 1$ fixing $\alpha = 0.6$, i.e., we define</p>
<p></p>
<p>and we obtain the following GIF.</p>
<p><img title="l1 norm minus lp norm balls" alt="Loading..." src="https://i0.wp.com/www.alexejgossmann.com/images/penalty_function_contours/l1-lp_balls.gif?w=456" data-recalc-dims="1"></p>
<h1>Code</h1>
<p>The R code uses the libraries <code>dplyr</code> for data manipulation, <code>ggplot2</code> for generation of figures, and <code>magick</code> to combine the individual images into a GIF.</p>
<p>Here are the R scripts that can be used to reproduce the above GIFs:</p>
<ol>
<li><a target="_blank" rel="nofollow" href="https://github.com/agisga/2D_norm_balls/blob/master/R/p-norm.R">p-norms in 2D</a></li>
<li><a target="_blank" rel="nofollow" href="https://github.com/agisga/2D_norm_balls/blob/master/R/elastic_net.R">Elastic net penalty in 2D</a></li>
<li><a target="_blank" rel="nofollow" href="https://github.com/agisga/2D_norm_balls/blob/master/R/fused.R">Fused penalty in 2D</a></li>
<li><a target="_blank" rel="nofollow" href="https://github.com/agisga/2D_norm_balls/blob/master/R/sorted_L1.R">Sorted L1 penalty in 2D</a></li>
<li><a target="_blank" rel="nofollow" href="https://github.com/agisga/2D_norm_balls/blob/master/R/l1-l2.R">Difference of $p$-norms: $\ell\subscript{1} – \ell\subscript{2}$ in 2D</a></li>
<li><a target="_blank" rel="nofollow" href="https://github.com/agisga/2D_norm_balls/blob/master/R/l1-lp.R">Difference of $p$-norms: $\ell\subscript{1} – \ell\subscript{p}$ in 2D</a></li>
</ol>
<p>Should I come across other interesting penalty functions that make sense in 2D, then I will add corresponding further visualizations to the same Github repository.</p>
<p><font size="3"><br>
<a target="_blank" rel="nofollow" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" src="https://i2.wp.com/i.creativecommons.org/l/by/4.0/80x15.png?w=456&ssl=1" data-recalc-dims="1"></a><br>This work is licensed under a <a target="_blank" rel="nofollow" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.<br>
</font></p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.alexejgossmann.com/contours_of_statistical_penalty_functions_as_GIFs/"> Alexej's blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/YbfozhWJ7gk"></a></div>