Slide deck on 3 concepts for statistical divergences:

- *comparative convexity* and a generalization of Bregman divergences
- computing divergences with *maximal invariant*
- embedding Fisher-Rao metrics onto submanifolds and *projective divergences*
👉franknielsen.github.io/SlidesVideo/

Distances omnipresent in information sciences:
- Probability: convergence thms
- Statistics: estimators/scoring rules
- Information theory: mutual information
- Signal proc.: factorization
- ML: loss functions
- Information geometry: canonical structures

franknielsen.github.io/Divergence/

Precursors of statistical manifolds in information geometry:
- Space of statistical parameters by Hotelling (1930)
- Statistical field by Mahalanobis (1936)
- Population space by Rao (1945)

2 meanings for statistical manifolds:
-mfd of statistical models
-mfd with dual structure

I agree with this quote by replacing 'is not about' by 'is not only about' 🙂

'Mathematics is not about numbers, equations, computations, or algorithms: it is about understanding'
-- William Paul Thurston

3 definitions of the Jensen-Shannon divergence yielding 3 different generalizations!

① JSD = Symmetrization of KLD

② JSD = concave gap induced by Shannon entropy

③ JSD = variational KLD divergence

mdpi.com/1099-4300/23/4…

Sad news. RIP.

Talk next week on recent works on distances with applications in statistics & ML at meeting edmsa.sciencesconf.org

Introduce concepts:
- comparative convexity & Bregman divergences
- maximal invariant & f-divergences
- Fisher Rao & projective distances

x.com/frnknlsn/statu…

Fisher-Rao geometry of statistical model of univariate normal distributions amounts to Poincaré hyperbolic upper plane U.

Isometric embedding of a *band* of U onto the pseudosphere of negative curvature.

Hilbert proved that no full isometric embedding of U in R3 is possible.

Overview of the key principles of main statistical distances used in machine learning:
- f-divergences
- Bregman divergences
- Optimal transport

👉franknielsen.github.io/Divergence/ind…

(Figure from mdpi.com/1099-4300/22/1… )

Prof. Mahalanobis (famous for Mahalanobis Δ^2 distance) mentions analogy with relativity: model space of distributions by a 'statistical field' !

Introduced work of Einstein and Minkowski, english translation published by the university of Calcutta.

👉 franknielsen.github.io/IG/

Nice to see that at least 6 papers at #ICML24 have coresets in their titles !
Hope for green data ML/AI !

x.com/frnknlsn/statu…

Softplus is a strictly convex activation function smoothing ReLU.
It is related to LogSumExp (LSE) by fixing one argument to zero.
LSE is only convex but fixing one argument to zero makes it strictly convex and yields the softplus function.
tinyurl.com/MonteCarloIG

Enjoy browsing the list of ICML'24 paper titles!

👉 icml.cc/virtual/2024/p…

4.V

The Theorem of the Day #TOTD theoremoftheday is Bregman’s Theorem 🏇🏻🏇🏻🏇🏻🏇🏻

theoremoftheday.org/CombinatorialT…

Variational divergences = minimizers of averages of divergences.

Sibson's information radius yields Jensen-Shannon variational divergence with centroid = arithmetic mean.

Symmetrize any divergence as a variational divergence using an abstract mean:

mdpi.com/1099-4300/23/4…

My short reviews:

① Excellent panorama of information geometry (IG) by its Founder. Most accessible text!
⬇️
② Great description of dual structures of IG with exercises but need diff geo background
⬇️
③ Most rigorous text. Foundations, statistical invariance

Giant Kite Festival in Japan