<p>Many statistical modeling problems reduce to a minimization problem of the general form:</p>
<p>or</p>
<p>where $f$ is some type of loss function, $\mathbf{X}$ denotes the data, and $g$ is a penalty, also referred to by other names, such as “regularization term” (problems (1) and (2-3) are often equivalent by the way). Of course both, $f$ and $g$, may depend on further parameters.</p>
<p>There are multiple reasons why it can be helpful to check out the contours of such penalty functions $g$:</p>
<ol><li>When $\boldsymbol{\beta}$ is two-dimensional, the solution of problem (2-3) can be found by simply taking a look at the contours of $f$ and $g$.</li>
<li>That builds intuition for what happens in more than two dimensions, and in other more general cases.</li>
<li>From a Bayesian point of view, problem (1) can often be interpreted as an <a href="https://en.wikipedia.org/wiki/Maximum_a_posteriori_estimation">MAP</a> estimator, in which case the contours of $g$ are also contours of the prior distribution of $\boldsymbol{\beta}$.</li>
</ol><p>Therefore, it is meaningful to visualize the set of points that $g$ maps onto the unit ball in $\mathbb{R}^2$, i.e., the set</p>
<p>Below you see GIF images of such sets $B\subscript{g}$ for various penalty functions $g$ in 2D, capturing the effect of varying certain parameters in $g$. The covered penalty functions include the family of $p$-norms, the elastic net penalty, the fused penalty, and the sorted $\ell_1$ norm.</p>
<p>:white_check_mark: R code to reproduce the GIFs is provided.</p>
<h2>p-norms in 2D</h2>
<p>First we consider the $p$-norm,</p>
<p>with a varying parameter $p \in (0, \infty]$ (which actually isn’t a proper <a href="https://en.wikipedia.org/wiki/Norm_(mathematics)">norm</a> for $p &lt; 1$). Many statistical methods, such as LASSO and Ridge Regression, employ $p$-norm penalties. To find all $\boldsymbol{\beta}$ on the boundary of the 2D unit $p$-norm ball, given $\beta_1$ (the first entry of $\boldsymbol{\beta}$), $\beta_2$ is easily obtained as</p>
<p></p>
<h2>Elastic net penalty in 2D</h2>
<p>The elastic net penalty can be written in the form</p>
<p>for $\alpha\in(0,1)$. It is quite popular with a variety of regression-based methods (such as the Elastic Net, of course). We obtain the corresponding 2D unit “ball”, by calculating $\beta\subscript{2}$ from a given $\beta\subscript{1}\in[-1,1]$ as</p>
<p></p>
<h2>Fused penalty in 2D</h2>
<p>The fused penalty can be written in the form</p>
<p>It encourages neighboring coefficients $\beta\subscript{i}$ to have similar values, and is utilized by the fused LASSO and similar methods.</p>
<p></p>
<p>(Here I have simply evaluated the fused penalty function on a grid of points in $[-2,2]^2$, because figuring out equations in parametric form for the above polygons was too painful for my taste… :stuck_out_tongue:)</p>
<h2>Sorted L1 penalty in 2D</h2>
<p>The Sorted $\ell\subscript{1}$ penalty is used in a number of regression-based methods, such as SLOPE and OSCAR. It has the form</p>
<p>where $\lvert \beta \rvert\subscript{(1)} \geq \lvert \beta \rvert\subscript{(2)} \geq \ldots \geq \lvert \beta \rvert\subscript{(m)}$ are the absolute values of the entries of $\boldsymbol{\beta}$ arranged in a decreasing order. In 2D this reduces to</p>
<p></p>
<h2>Difference of p-norms</h2>
<p>It holds that</p>
<p>or more generally, for all $p$-norms it holds that</p>
<p>Thus, it is meaningful to define a penalty function of the form</p>
<p>for $\alpha\in[0,1]$, which results in the following.</p>
<p></p>
<p>We visualize the same for varying $p \geq 1$ fixing $\alpha = 0.6$, i.e., we define</p>
<p>and we obtain the following GIF.</p>
<p></p>
<h1>Code</h1>
<p>The R code uses the libraries <code>dplyr</code> for data manipulation, <code>ggplot2</code> for generation of figures, and <code>magick</code> to combine the individual images into a GIF.</p>
<p>Here are the R scripts that can be used to reproduce the above GIFs:</p>
<ol><li><a href="https://github.com/agisga/2D_norm_balls/blob/master/R/p-norm.R">p-norms in 2D</a></li>
<li><a href="https://github.com/agisga/2D_norm_balls/blob/master/R/elastic_net.R">Elastic net penalty in 2D</a></li>
<li><a href="https://github.com/agisga/2D_norm_balls/blob/master/R/fused.R">Fused penalty in 2D</a></li>
<li><a href="https://github.com/agisga/2D_norm_balls/blob/master/R/sorted_L1.R">Sorted L1 penalty in 2D</a></li>
<li><a href="https://github.com/agisga/2D_norm_balls/blob/master/R/l1-l2.R">Difference of $p$-norms: $\ell\subscript{1} - \ell\subscript{2}$ in 2D</a></li>
<li><a href="https://github.com/agisga/2D_norm_balls/blob/master/R/l1-lp.R">Difference of $p$-norms: $\ell\subscript{1} - \ell\subscript{p}$ in 2D</a></li>
</ol><p>Should I come across other interesting penalty functions that make sense in 2D, then I will add corresponding further visualizations to the same Github repository.</p>
<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"></a><br>This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.