You could just do MatFormer training + Matryoshka quantization to get a here of inference checkpoints (at multiple sizes and quantization) from a single full precision checkpoint

Very happy to release those models to the community. Lots of care went into the architecture design. 1. It has low cache requirements (the gated convolutions only require a cache size of batch_size x 3 x d_model) 2. It has fewer FLOPs than a standard transformer, even at short

What's the frontier for OSS multimodal models ? Qwen-omni?

You could have a functional RL + inference engine in ~2.5k LOC. 1. RL2 (github.com/ChenmienTan/RL2) 2. nano-vllm (github.com/GeeeekExplorer…)

VLM learn to "see" by aligning the representation from the image encoder to the text representation from the LLM, using image-text pairs. Has anyone tried to make a VLM learn to "see" purely through exploration/RL? A setup could be to 1. feed the encoded image (with or

Thank god for uv replacing conda slop

Mixture-of-experts has this nice property of being optimal w.r.t scaling: 1. compute 2. suffering to write the infra to do training, quantization & inference

fixing up the QAT pipeline at 2AM one day before a model delivery

Training: 282+1=284 in bf16, lfg some regularization for my training. Inference: Leaving torch.backends.cudnn.allow_tf32 = True, too bad bruv, that's a 20 point drop on AIME

Watching Zettlemoyer's lecture at the Simons Institute, this man is goated

Really keen on adaptive sparse architectures for multi-modal models. It makes a lot of sense to reduce the difficulties of training large MM models (training instabilities induced by modality competition e.g. logit growth.) It also just scales better. MoMA is a good example of

never settling for slow evals in your research is the hill I'll die on.

A structured version of this e.g Matryoskha/ nested dropout would allow to train end-to-end a knob to adjust between recall and memory usage. We could even offload the unused KV cache to RAM, and bring it back later if better recall is needed.