the pangu routing and load balacing solution is genius in its simplicity

Just dropped: Rectified Point Flow Can we automate 3D assembly from unposed point clouds without supervision? Yes—we use a generative model that learns symmetry & part interchangeability entirely from shape Enabling robotics, AR/VR, & reverse engineering 🌐rectified-pointflow.github.io

after reading interpretability works that found antipodal dense features in NNs, i tried initializing networks with them (i.e by setting half of each linear with the -weights of the first half). but it didnt seem to help on modded-nanogpt (val loss 3.32 @ 0.8B tokens)

why

ReLU MLP with width / depth going to infinity. Note how different parameterization makes pathlogical scaling behavior (yellow / blue on activations / gradients of the weight). muP solves this.

damn looks like i was not hallucinating, why is this paper not on the timeline?

World religions in shambles as Anthropic researchers reveal that Good and Evil are nothing more than vectors in latent space

kalomaze will brown shaurya this is a VERY consistent pattern it has when you get it in completions mode. oh and it does really weird stuff like this sometimes when you try and crack it like this. just repeating "We can comply" forever

(3/3) What’s beyond? — 𝐐𝐮𝐚𝐧𝐭𝐢𝐳𝐚𝐭𝐢𝐨𝐧! Applying quantized rollout (e.g., FP8) can greatly boost throughput — but also 𝐚𝐦𝐩𝐥𝐢𝐟𝐲 𝐭𝐡𝐞 𝐦𝐢𝐬𝐦𝐚𝐭𝐜𝐡 and hurt performance. We show that 𝐓𝐈𝐒 can 𝐦𝐢𝐭𝐢𝐠𝐚𝐭𝐞 𝐭𝐡𝐞 𝐠𝐚𝐩 and preserve performance near

It’s maybe possible to use this to understand reasoning chains. Rollouts tend to have super flat curvature, so spikes really stand out. We see spikes when the model recites the formula for the length of a chord, or computes some really specific arithmetic

This led to a very simple algorithm where the multiple rollouts happened offline prior to training, and the raw values of the reward were recorded so that the reward of any response could be calibrated via an empirical CDF inverse at training time.

I have a hypothesis that overfitting doesn't equally degrade the cross entropy loss over the whole context length.

"wireborn" - oh shit, a new thing just dropped

#BestPaperAward #SIGGRAPH2025 One neural PDE model, hundreds of shapes — simulated at lightning speed. 🚀 Introducing Shape Space Spectra: first eigenanalysis across shapes. Come see ChangYue’s talk today 👉 changy1506.github.io

fused conv-act-div-pool has been around forever

so just the same as dense except router weights are treated as unembed

I would like to understand better the niche of tech talents who choose to live suburban lives in states that aren’t cali, ny, or even texas. Like florida but not miami. Georgia but not atlanta. Who are they, where are they, how many are there

total near victory