Wei-Yin Ko just updated the arxiv preprint for this work:

arxiv.org/abs/2303.00586

Extensive additional experiments across additional datasets and settings.

Congrats to Wei-Yin Ko Daniel D'souza  for leading with Randall Balestriero 🎉

Interestingly the ReLU and Swish relation is well understood from a spline viewpoint akin to the relation between k-NN and isotropic GMM:
deterministic vs probabilistic region assignment!

The same goes for absolute value vs Mish, and many more!

More at openreview.net/forum?id=Syxt2…

VICReging with Randall at NeurIPS.

arxiv.org/abs/2303.00633

Such issue is hard/costly to mitigate without a clear and applicable mathematical model of Generative AI. One step in that direction was presented in
arxiv.org/abs/2110.08009
arxiv.org/abs/2203.01993
without such provable solution we will always be at the mercy of a discrepancy...