The code for RoSA: Accurate Parameter-Efficient Fine-Tuning via Sparse + Low-Rank Adapters is now available: github.com/IST-DASLab/RoSA along with a PEFT integration github.com/IST-DASLab/pef… As a bonus, we also QRoSA, which implements the same idea, but with quantized base weights.

Happy to introduce #HALO lower-precision fine-tuning for LLMs. With proper Hadamard transforms, #HALO enables accurate INT8/FP6 fine-tuning—lossless speedups up to 1.41×. 📄 Paper: arxiv.org/pdf/2501.02625 💻 Code: github.com/IST-DASLab/HALO #LLM #Quantization

(1/n) In our #ICLR2025 paper, we explore a fundamental issue that enables prompt injections: 𝐋𝐋𝐌𝐬’ 𝐢𝐧𝐚𝐛𝐢𝐥𝐢𝐭𝐲 𝐭𝐨 𝐬𝐞𝐩𝐚𝐫𝐚𝐭𝐞 𝐢𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧𝐬 𝐟𝐫𝐨𝐦 𝐝𝐚𝐭𝐚 𝐢𝐧 𝐭𝐡𝐞𝐢𝐫 𝐢𝐧𝐩𝐮𝐭 ✅ Definition of separation 👉 SEP Benchmark 🔍 LLM evals on SEP

Our QuEST paper was selected for Oral Presentation at ICLR Sparsity in LLMs Workshop at ICLR 2025 workshop! QuEST is the first algorithm with Pareto-optimal LLM training for 4bit weights/activations, and can even train accurate 1-bit LLMs. Paper: arxiv.org/abs/2502.05003 Code: github.com/IST-DASLab/QuE…