I read someone disdainfully talking about how optimizing CUDA kernels was a miserable job. Perhaps for many, but some people take joy in the clearly defined world of make-go-faster. Leaders don’t need the actual skills, but they should have a sense of what is possible with them.

🤯 Full body tracking now possible using only WiFi signals A deep neural network maps the phase and amplitude of WiFi signals to UV coordinates within 24 human regions The model can estimate the dense pose of multiple subjects by utilizing WiFi signals as the only input 🧵

Sparsity people trying to publish in ML conferences

Join Roberto at #SC23 on Nov 16 at 11:30AM as he unveils VENOM - a new sparse matrix format enabling arbitrary N:M patterns on Sparse Tensor Cores, natively restricted to 2:4. Plus, explore Spatha 🗡️, our sparse library for VENOM achieving up to 37x over SOTA dense methods.

Join us at #SC23 on Nov 16 at 11:30AM as we unveil VENOM - a new sparse matrix format enabling arbitrary N:M patterns on Sparse Tensor Cores, natively restricted to 2:4. Plus, explore Spatha 🗡️, our sparse library for VENOM achieving up to 37x over SOTA dense methods.

Don't miss Roberto's talk at #SC23 tomorrow at 11:30AM as he unveils VENOM - a new sparse matrix format enabling arbitrary N:M patterns on Sparse Tensor Cores, natively restricted to 2:4. 👉 sc23.supercomputing.org/attend/digital…

Is NVIDIA's 2:4 sparsity not enough for your ambitions with #AI models? Then check out Roberto's structured sparse extension format to go to nearly arbitrary sparsities. Getting >10x inference speedups using sparse tensor cores! youtube.com/watch?v=hif1Eq… arxiv.org/abs/2310.02065

📰Xa dispoñible no #RUC e #Zenodo o traballo do grupo #GAC da Facultade de Informática da UDC: "STuning-DL: Model-Driven Autotuning of Sparse GPU Kernels for Deep Learning" (doi.org/10.1109/ACCESS…). hdl.handle.net/2183/36810 & zenodo.org/uploads/114893…. #cienciaaberta

Happy to release the write-up on the MARLIN kernel for fast LLM inference, now supporting 2:4 sparsity! Led by Elias Frantar & Roberto López Castro Paper: arxiv.org/abs/2408.11743 Code: github.com/IST-DASLab/Spa… MARLIN is integrated with vLLM thanks to @neuralmagic!

Sparse-Marlin is here and integrated into vLLM! This GPU-optimized kernel accelerates matrix multiplication with 4-bit quantized weights and 2:4 sparsity, achieving 5.3x speedups on NVIDIA GPUs (Ampere/Ada). Maintains efficiency with batch sizes up to 32. Links below.

Code: github.com/IST-DASLab/Spa… Paper: arxiv.org/abs/2408.11743 Made possible by: Dan Alistarh, Elias Frantar, Roberto López Castro. Shout out to Neural Magic engineers who swiftly integrated Sparse-Marlin into #vLLM for immediate use.

2:4 Sparsity + AI at Meta Llama-3.1: At @neuralmagic, we've developed a recipe to produce very competitive sparse LLMs, and we are starting by open-sourcing the first one: Sparse-Llama-3.1-8B-2of4. We also show how to leverage it for blazingly fast inference in vLLM.

4) The 2:4 models are compatible with quantization. We apply GPTQ to quantize weights to 4-bit integers, but modify it such that it preserves the 2:4 sparsity pattern. The resulting model has a minimal drop in acc, but runs super fast with Sparse-Marlin kernel in vLLM

Today, Red Hat completed the acquisition of Red Hat AI (formerly Neural Magic), a pioneer in software and algorithms that accelerate #GenAI inference workloads. Read how we are accelerating our vision for #AI’s future: red.ht/408kJ8K.

Happy to release #HALO, a Hadamard-Assisted Lower-Precision scheme that enables INT8/FP6 full #finetuning (FFT) of LLMs. with @mmnnn76 Rush Tabesh Roberto López Castro Torsten Hoefler 🇨🇭 Dan Alistarh Paper: arxiv.org/pdf/2501.02625 Code: github.com/IST-DASLab/HALO 1/4

Yesterday, Jiale and Roberto presented the paper MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models at PPoPP 2025 in Las Vegas! Want to know more? Check out the paper here👇dl.acm.org/doi/10.1145/37… #HPC Torsten Hoefler 🇨🇭 ETH CS Department

Our QuEST paper was selected for Oral Presentation at ICLR Sparsity in LLMs Workshop at ICLR 2025 workshop! QuEST is the first algorithm with Pareto-optimal LLM training for 4bit weights/activations, and can even train accurate 1-bit LLMs. Paper: arxiv.org/abs/2502.05003 Code: github.com/IST-DASLab/QuE…

Introducing MoE-Quant, a fast version of GPTQ for MoEs, with: * Optimized Triton kernels and expert&data parallelism * Quantizes the 671B DeepSeekV3/R1 models in 2 hours on 8xH100 * ~99% accuracy recovery for 4bit R1 on *reasoning* tasks, and 100% recovery on leaderboards [1/3]

We are introducing Quartet, a fully FP4-native training method for Large Language Models, achieving optimal accuracy-efficiency trade-offs on NVIDIA Blackwell GPUs! Quartet can be used to train billion-scale models in FP4 faster than FP8 or FP16, at matching accuracy. [1/4]