We wrote up some of the best practices we feel are useful for ML projects github.com/labmlai/labml/… Here's a summary 🧵👇

Deep Learning job interview questions, fully solved & covering a wide range of key AI topics: bit.ly/4bv6FL9 credit: Daily AI Papers

Encoding floating point arrays with Base64 gives a 4x compression over JSON 🚀. Quite useful when you have to transfer larger arrays. Encoded float arrays can be included as a string in the JSON objects keeping existing structures intact. Doesn’t require any external libraries.

.labml.ai deep learning experiment monitoring app got significantly more responsive after hehehehe implemented base64 encoding for long float arrays instead of plain JSON. It uses 4X less data transfer.

We’ve open-sourced our LLM attention visualization library. It generates interactive visualizations of attention matrices with just a few lines of Python code in notebooks. hehehehe cleaned up and polished the existing code to make it open source.

✨ Annotated DL Paper Implementation repository reached 50K stars. It has implementations of a wide range of deep learning concepts including Transformers and variations, StyleGAN, Stable Diffusion, Normalization layers, RL, Optimizers... 🧶👇

The machine generated Chinese translation of annotated paper implementations repo is being improved with manual translations by @pengchzn 🙏 @pengchzn has already finished the basic transformer including multi-head attention. 👇

🎉 Excited share that we add a distribution visualization to our library, Inspectus. It plots the full distribution of data across training steps. This helps better understand how the training is going and instantly see the impact of outliers. 👇

I first found plotting the distribution useful when I was trying RL algorithms on Atari around 2018/19. I used Tensorboard back then. It was quite useful to look at the score distribution of the rollouts. It showed how the policy was behaving clearer than looking at the mean

We should be able to release an update of labml experiment monitoring library very soon 😂 It has a bunch of cool new features

We’ve been training NVIDIA Mistral-NeMo-Minitron-8B-Base for math reasoning on the GSM8K-Aug dataset, and we have a version with a 70.2% gsm8k score, up from a 58.5% cot score (reported in the paper LLM Pruning and distillation). 👇

Annotated PyTorch implementation of of LoRA (Low Lank Adaptation of LLMs) 📝 Code + Notes: nn.labml.ai/lora/index.html 📎 Paper: arxiv.org/abs/2106.09685 LoRA freezes the pre-trained model and trains smaller injected weights, enabling faster and memory efficient fine-tuning. 👇

Our open source deep learning experiment monitoring library now has 2000 stars! Thank you

We added token visualization to Inspectus. It lets you visualize metrics associated with tokens such as loss, entropy, KL div, etc. It works on notebooks and pretty easy to use. 👇

📢 We are excited to announce Notbad v1.0 Mistral 24B, a new reasoning model trained in math and Python coding. This model is built upon the Mistral AI Small 24B 2501 and has been further trained with reinforcement learning on math and coding.

You can now download the Notbad v1.0 Mistral 24B model from Hugging Face huggingface.co/notbadai/notba… Try it on chat.labml.ai

We're open-sourcing a math reasoning dataset with 270k samples, generated by our RL-based self-improved Mistral 24B 2501 model and used to train Notbad v1.0 Mistral 24B. Available on Hugging Face: huggingface.co/datasets/notba…

Uploaded the dataset of 270k math reasoning samples that we used to finetune Notbad v1.0 Mistral 24B (MATH-500=77.52% GSM8k Platinum=97.55%) to Hugging Face (link in reply) Follow NOTBAD AI for updates

We are releasing an updated reasoning model with improvements on IFEval scores (77.9%) than our previous model (only 51.4%). 👇 Links to try the model and to download weights below

The new training also improved GPQA from 64.2% to 67.3% and MMLU Pro from 64.2% to 67.3%. This model was also trained with the same reasoning datasets we used to train the v1.0 model. We mixed more general instruction data with answers sampled from the