Almost a Kessler syndrome for LLMs arxiv.org/abs/2305.17493 arxiv.org/abs/2307.01850

The AI world is in a GPU crunch and meanwhile NERSC is offering 50% off its A100 GPUs (rest.nersc.gov/REST/announcem…). They could make a killing by backfilling idle capacity with commercial workloads 💰 💰 💰

$0.25/mile

arxiv.org/abs/2308.13418

Finally found the introverts' dining area at SJC. 🙂

Regulation starts at roughly two orders of magnitude larger than a ~70B Transformer trained on 2T tokens -- which is ~5e24. Note: increasing the size of the dataset OR the size of the transformer increases training flops. The (rumored) size of GPT-4 is regulated.

Inception used 1.5X less compute than AlexNet and 12X less than VGG, outperforming both. The trend continued with mobile net... etc. IMO, today's LLMs are insanely inefficient/compute. Regulations that impose limits on the amount of compute spent on AI training will just

So tempting to just group a collection of "specialized"s and call it "general" arxiv.org/abs/2311.02462

Today's AI (credit: Balazs Szegedy using DALL-E-3)

Me: Can you draw a very normal image? ChatGPT: Here is a very normal image depicting a tranquil suburban street scene during the daytime. Me: Not bad, but can you go more normal than that? (cont.)

How do we capture local features across multiple resolutions? While standard convolutional layers work only on a fixed input-resolution, we design local neural operators that learn integral and differential kernels, and are principled ways to extend standard convolutions to

And today is T - 2 weeks for my other @NVIDIA #GTC session - a fireside chat with Christian Szegedy, cofounder of @xAI. Christian is one of the seminal research figures in the Deep Learning community, but the main focus of our chat will be on something that he has been working very

Why do neural nets generalize so well?🤔 There's a ton of work on SGD, flat minima, ... but the root cause is that their inductive biases somehow match properties of real-world data.🌎 We've examined these inductive biases in *untrained* networks.🎲 arxiv.org/abs/2403.02241 ⬇️

Mechanism for feature learning in neural networks and backpropagation-free machine learning models | Science science.org/doi/10.1126/sc…

Congress could use more computer scientists like Rep. Ted Lieu

Wordle 1,006 5/6 🟩⬜⬜⬜⬜ 🟩🟩⬜⬜⬜ 🟩🟩🟩⬜⬜ 🟩🟩🟩🟩⬜ 🟩🟩🟩🟩🟩

Large multimodal models often lack precise low-level perception needed for high-level reasoning, even with simple vector graphics. We bridge this gap by proposing an intermediate symbolic representation that leverages LLMs for text-based reasoning. mikewangwzhl.github.io/VDLM 🧵1/4

I hope this PDP-10 makes it to the excellent Computer History Museum geekwire.com/2024/seattles-…

Knuth - Claude's cycles problem cs.stanford.edu/~knuth/papers/… solved using Claude Opus 4.6