Integral probability metric

In probability theory, integral probability metrics are types of distance functions between probability distributions, defined by how well a class of functions can distinguish the two distributions. Many important statistical distances are integral probability metrics, including the Wasserstein-1 distance and the total variation distance. In addition to theoretical importance, integral probability metrics are widely used in areas of statistics and machine learning.

The name "integral probability metric" was given by German statistician Alfred Müller;^[1] the distances had also previously been called "metrics with a ζ-structure."^[2]

Definition

Integral probability metrics (IPMs) are distances on the space of distributions over a set ${\displaystyle {\mathcal {X}}}$, defined by a class ${\displaystyle {\mathcal {F}}}$ of real-valued functions on ${\displaystyle {\mathcal {X}}}$ as $${\displaystyle D_{\mathcal {F}}(P,Q)=\sup _{f\in {\mathcal {F}}}{\big |}\mathbb {E} _{X\sim P}f(X)-\mathbb {E} _{Y\sim Q}f(Y){\big |}=\sup _{f\in {\mathcal {F}}}{\big |}Pf-Qf{\big |};}$$ here the notation P f refers to the expectation of f under the distribution P. The absolute value in the definition is unnecessary, and often omitted, for the usual case where for every ${\displaystyle f\in {\mathcal {F}}}$ its negation ${\displaystyle -f}$ is also in ${\displaystyle {\mathcal {F}}}$.

The functions f being optimized over are sometimes called "critic" functions;^[3] if a particular ${\displaystyle f^{*}\in {\mathcal {F}}}$ achieves the supremum, it is often termed a "witness function"^[4] (it "witnesses" the difference in the distributions). These functions try to have large values for samples from P and small (likely negative) values for samples from Q; this can be thought of as a weaker version of classifers, and indeed IPMs can be interpreted as the optimal risk of a particular classifier.^{[5]: sec. 4}

The choice of ${\displaystyle {\mathcal {F}}}$ determines the particular distance; more than one ${\displaystyle {\mathcal {F}}}$ can generate the same distance.^[1]

For any choice of ${\displaystyle {\mathcal {F}}}$, ${\displaystyle D_{\mathcal {F}}}$ satisfies all the definitions of a metric except that we may have we may have ${\displaystyle D_{\mathcal {F}}(P,Q)=0}$ for some P ≠ Q; this is variously termed a "pseudometric" or a "semimetric" depending on the community. For instance, using the class ${\displaystyle {\mathcal {F}}=\{x\mapsto 0\}}$ which only contains the zero function, ${\displaystyle D_{\mathcal {F}}(P,Q)}$ is identically zero. ${\displaystyle D_{\mathcal {F}}}$ is a metric if and only if ${\displaystyle {\mathcal {F}}}$ separates points on the space of probability distributions, i.e. for any P ≠ Q there is some ${\displaystyle f\in {\mathcal {F}}}$ such that ${\displaystyle Pf\neq Qf}$;^[1] most, but not all, common particular cases satisfy this property.

Examples

All of these examples are metrics except when noted otherwise.

• The Wasserstein-1 distance (also called earth mover's distance), via its dual representation, has ${\displaystyle {\mathcal {F}}}$ the set of 1-Lipschitz functions.
• The related Dudley metric is generated by the set of bounded 1-Lipschitz functions.
• The total variation distance can be generated by ${\displaystyle {\mathcal {F}}=\{f:{\mathcal {X}}\to \{0,1\}\}}$, so that ${\displaystyle {\mathcal {F}}}$ is a set of indicator functions for any event, or by the larger class ${\displaystyle {\mathcal {F}}=\{f:{\mathcal {X}}\to [0,1]\}}$.
• The closely related Radon metric is generated by continuous functions bounded in [-1, 1].
• The Kolmogorov metric used in the Kolmogorov-Smirnov test has a function class of indicator functions, ${\displaystyle {\mathcal {F}}=\{1_{(-\infty ,t]}:t\in \mathbb {R} \}}$.
• The kernel maximum mean discrepancy (MMD) has ${\displaystyle {\mathcal {F}}}$ the unit ball in a reproducing kernel Hilbert space. This distance is particularly easy to estimate from samples, requiring no optimization; it is a proper metric exactly when the underlying kernel is characteristic.^[6]
• The energy distance, as a special case of the maximum mean discrepancy,^[7] is generated by the unit ball in a particular reproducing kernel Hilbert space.
• Defining ${\displaystyle {\mathcal {F}}}$ by functions with a bounded Sobolev norm gives a useful distance for generative modeling, among other applications.^[8]
• Functions with bounded Besov norm generalize many other forms of IPM and are amenable to theoretical analysis.^[9][10]
• Many variants of generative adversarial networks and classifer-based two-sample tests^[11][12] use a "neural net distance"^[13][14] where ${\displaystyle {\mathcal {F}}}$ is a class of neural networks; these are not metrics for typical fixed-size networks, but could be for other classifiers. For Wasserstein GANs in particular, it has been argued that analysis in terms of this distance and not the Wasserstein they approximate is very important to the behavior of these models.^[13][15][16]

Relationship to f-divergences

The f-divergences are probably the best-known way to measure dissimilarity of probability distributions. It has been shown^{[5]: sec. 2} that the only functions which are both IPMs and f-divergences are of the form ${\displaystyle c\,\operatorname {TV} (P,Q)}$, where ${\displaystyle c\in [0,\infty ]}$ and ${\displaystyle \operatorname {TV} }$ is the total variation distance between distributions.

One major difference between f-divergences and most IPMs is that when P and Q have disjoint support, all f-divergences take on a constant value;^[17] by contrast, IPMs where functions in ${\displaystyle {\mathcal {F}}}$ are "smooth" can give "partial credit." For instance, consider the sequence ${\displaystyle \delta _{1/n}}$ of Dirac measures at 1/n; this sequence converges in distribution to ${\displaystyle \delta _{0}}$, and many IPMs satisfy ${\displaystyle D_{\mathcal {F}}(\delta _{1/n},\delta _{0})\to 0}$, but no nonzero f-divergence can satisfy this. That is, many IPMs are continuous in weaker topologies than f-divergences. This property is sometimes of substantial importance,^[18] although other options also exist, such as considering f-divergences between distributions convolved with continuous noise.^[18][19]

Estimation from samples

Because IPM values between discrete distributions are often sensible, it is often reasonable to estimate ${\displaystyle D_{\mathcal {F}}(P,Q)}$ using a simple "plug-in" estimator: ${\displaystyle D_{\mathcal {F}}({\hat {P}},{\hat {Q}})}$ where ${\displaystyle {\hat {P}}}$ and ${\displaystyle {\hat {Q}}}$ are empirical measures of sample sets. These empirical distances can be computed exactly for some classes ${\displaystyle {\mathcal {F}}}$;^[5] estimation quality varies depending on the distance, but can be minimax-optimal in certain settings.^[14][20][21]

When exact maximization is not available or too expensive, another commonly used scheme is to divide the samples into "training" sets (with empirical measures ${\displaystyle {\hat {P}}_{\mathit {train}}}$ and ${\displaystyle {\hat {Q}}_{\mathit {train}}}$) and "test" sets (${\displaystyle {\hat {P}}_{\mathit {test}}}$ and ${\displaystyle {\hat {Q}}_{\mathit {test}}}$), find ${\displaystyle {\hat {f}}}$ approximately maximizing ${\displaystyle {\big |}{\hat {P}}_{\mathit {train}}f-{\hat {Q}}_{\mathit {train}}f{\big |}}$, then use ${\displaystyle {\big |}{\hat {P}}_{\mathit {test}}{\hat {f}}-{\hat {Q}}_{\mathit {test}}{\hat {f}}{\big |}}$ as an estimate.^{[22][12][23][24]} This estimator can possibly be consistent, but has a negative bias^{[22]: thm. 2}. In fact, no unbiased estimator can exist for any IPM^{[22]: thm. 3}, although there is for instance an unbiased estimator of the squared maximum mean discrepancy.^[4]

References

1. Müller, Alfred (June 1997). "Integral Probability Metrics and Their Generating Classes of Functions". Advances in Applied Probability. 29 (2): 429–443. doi:10.2307/1428011. JSTOR 1428011. S2CID 124648603.
2. Zolotarev, V. M. (January 1984). "Probability Metrics". Theory of Probability & Its Applications. 28 (2): 278–302. doi:10.1137/1128025.
3. Arjovsky, Martin; Chintala, Soumith; Bottou, Léon (2017-07-17). "Wasserstein Generative Adversarial Networks". International Conference on Machine Learning. PMLR: 214–223.
4. Gretton, Arthur; Borgwardt, Karsten M.; Rasche, Malte J.; Schölkopf, Bernhard; Smola, Alexander (2012). "A Kernel Two-Sample Test" (PDF). Journal of Machine Learning Research. 13: 723–773.
5. Sriperumbudur, Bharath K.; Fukumizu, Kenji; Gretton, Arthur; Schölkopf, Bernhard; Lanckriet, Gert R. G. (2009). "On integral probability metrics, φ-divergences and binary classification". arXiv:0901.2698 [cs.IT].
6. Fukumizu, Kenji; Gretton, Arthur; Sun, Xiaohui; Schölkopf, Bernhard (2007). "Kernel Measures of Conditional Dependence". Advances in Neural Information Processing Systems.
7. Sejdinovic, Dino; Sriperumbudur, Bharath; Gretton, Arthur; Fukumizu, Kenji (2013). "Equivalence of distance-based and RKHS-based statistics in hypothesis testing". The Annals of Statistics. 41 (5): 2263–2291. arXiv:1207.6076. doi:10.1214/13-aos1140. S2CID 8308769.
8. Mroueh, Youssef; Li, Chun-Liang; Sercu, Tom; Raj, Anant; Cheng, Yu (2018). "Sobolev GAN". International Conference on Learning Representations. arXiv:1711.04894.
9. Uppal, Ananya; Singh, Shashank; Póczos, Barnabás (2019). "Nonparametric Density Estimation & Convergence Rates for GANs under Besov IPM Losses". Advances in Neural Information Processing Systems. arXiv:1902.03511.
10. Uppal, Ananya; Singh, Shashank; Póczos, Barnabás (2020). "Robust Density Estimation under Besov IPM Losses". Advances in Neural Information Processing Systems. arXiv:2004.08597.
11. Kim, Ilmun; Ramdas, Aaditya; Singh, Aarti; Wasserman, Larry (February 2021). "Classification accuracy as a proxy for two-sample testing". The Annals of Statistics. 49 (1). arXiv:1703.00573. doi:10.1214/20-AOS1962. S2CID 17668083.
12. Lopez-Paz, David; Oquab, Maxime (2017). "Revisiting Classifier Two-Sample Tests". International Conference on Learning Representations. arXiv:1610.06545.
13. Arora, Sanjeev; Ge, Rong; Liang, Yingyu; Ma, Tengyu; Zhang, Yi (2017). "Generalization and Equilibrium in Generative Adversarial Nets (GANs)". International Conference on Machine Learning. arXiv:1703.00573.
14. Ji, Kaiyi; Liang, Yingbin (2018). "Minimax Estimation of Neural Net Distance". Advances in Neural Information Processing Systems. arXiv:1811.01054.
15. Stanczuk, Jan; Etmann, Christian; Lisa Maria Kreusser; Schönlieb, Carola-Bibiane (2021). "Wasserstein GANs Work Because They Fail (To Approximate the Wasserstein Distance)". arXiv:2103.01678 [stat.ML].
16. Mallasto, Anton; Montúfar, Guido; Gerolin, Augusto (2019). "How Well do WGANs Estimate the Wasserstein Metric?". arXiv:1910.03875 [cs.LG].
17. Sutherland, Danica J. "Computing Jensen-Shannon Divergence between discrete and continuous distribution". Stack Exchange Network. Retrieved 18 July 2023.
18. Arjovsky, Martin; Bettou, Léon (2017). "Towards Principled Methods for Training Generative Adversarial Networks". International Conference on Learning Representations. arXiv:1701.04862.
19. Sønderby, Casper Kaae; Caballero, Jose; Theis, Lucas; Shi, Wenzhe; Huszár, Ferenc (2017). "Amortised MAP Inference for Image Super-resolution". International Conference on Learning Representations. Appendix C. arXiv:1610.04490.
20. Tolstikhin, Ilya O.; Sriperumbudur, Bharath K.; Schölkopf, Bernhard (2016). "Minimax Estimation of Maximum Mean Discrepancy with Radial Kernels". Advances in Neural Information Processing Systems.
21. Singh, Shashank; Póczos, Barnabás (2018). "Minimax Distribution Estimation in Wasserstein Distance". arXiv:1802.08855 [math.ST].
22. Bińkowski, Mikołaj; Sutherland, Danica J.; Arbel, Michael; Gretton, Arthur (2018). "Demystifying MMD GANs". International Conference on Learning Representations. arXiv:1801.01401.
23. Liu, Feng; Xu, Wenkai; Lu, Jie; Zhang, Guangquan; Gretton, Arthur; Sutherland, Danica J. (2020). "Learning Deep Kernels for Non-Parametric Two-Sample Tests". International Conference on Machine Learning. arXiv:2002.09116.
24. Kübler, Jonas M.; Jitkrittum, Wittawat; Schölkopf, Bernhard; Muandent, Krikamol (2021). "A Witness Two-Sample Test". International Conference on Artificial Intelligence and Statistics. arXiv:2102.05573.