Hey folks, since we've been asked so many times, here is a quick demo of what we're building. This is a small LLaMA2 sharded on 1 NVIDIA RTX 4090 (in Paris), 1 AMD 6800XT (in Corendos's flat) and 1 Google Cloud TPU v2 over Tailscale. Exact same code, all built on my Mac

Dear folks, what are your average tok/s on Llama-3.1-8B BF16 on A100 and H100 for batch_size = 1 ?

Very proud to release what we've been working on ZML in the past few months !

So we're working on a custom async HTTP/1.1 server for the ZML LLM serving stack. This is on a single thread, M3 Max. Uses stackful coroutines and Mitchell Hashimoto's libxev. Not gonna lie, pretty happy about the performance.

H200 go brrrr with ZML's llmd

We can finally say it publicly: poolside got their models running on Amazon Web Services’s Trainium/Inferentia thanks to ZML. No code change. Promise kept.

Here is a quick demo of ZML running with MLX (cc Awni Hannun). It's a very early demo and needs lots more work, but it's alive.

> Tweet random de réponse "Lisa a aimé votre réponse" > Autre tweet random de réponse "Brenda a aimé votre réponse" L'aubaine, X Entreprise est à -30% !

zml.ai

Hey guys, website is up 👀

If you're at FOSDEM, come see renerocksai⚡ and Guillaume Wenzek who will be presenting ZML

In today’s Vatnik Soup, I’ll introduce how the Elon’s Election Interference Machine™ (EIM) works. Since Musk & his broligarch allies won the presidency for Trump, they’ve now harnessed the machine to interfere in European elections, including the ones in Germany & Romania. 1/23

So the show with Jonathan Ross absolutely crushed last week but left me with many more questions. Today is one of our most nerdy but fantastic episodes. - Why Google wins the AI arms race - Why OpenAI is not best positioned - NVIDIA vs. AMD - Inference vs training - The

We made this 👀

Somehow it keeps on getting faster 👀

We've been cooking. zml/llmd will soon get another platform.

And after 1 week of work, here is zml/llmd running transparently on TPU with full prefill/decode paged attention. No code change, single flag, as it should be.

Come see the future !

Sex is good but have you tried loading tensors from a SSD to a 4090 at 15GB/s ?

We, ZML, push the boundaries of what's possible 😎