we do a little devops

Are rating system predictive accuracies stable over time? Pretty much yea Using hyperparams swept on data up to 2023-03-31, I evaluated on 20 esports datasets in 3 month chunks. Relative ordering is 100% stable, Glicko>TrueSkill>Elo

holy shit Claude is generating ascii art to explain C to me

Claude agrees with me that it's weird that Glicko only uses opponent variance to compute win prob for a player (not their own). And thus P(A beats B) != 1 - P(B beats A) (I compute both, use them in their respective updates then average)

finally bought a bookshelf, love staring at my sci-fi and ml sections

My gf agreed we can name our dog Jokic if I wake up with 20k followers. She set an impossible condition because she really doesn't wanna name him/her Jokic. I don't ask for much, but I ask for this miracle.

cmon APM is a vanity metric that's been Goodharted to hell

It's really a bad practice to use Elo as an evaluation metric for LLMs. I can't believe I'm still seeing papers in good venues doing it Elo assumes the strengths are time varying, you need to use a Bradley-Terry or similar model to aggregate paired comparisons in this setting