we run 18 million EC2 instances per month. At our scale, we see very rare bugs very frequently. Last week, we received *half* an HTTP request. Not a HTTP 206, literally half a request. Content-Length was 2350 bytes. Body was actually 1200 bytes, and was truncated mid json doc.

We are hiring! If you're interested in writing Rust, running high performance infrastructure at massive scale: DM me your GitHub :)

We made our test suite run 30x faster... by manipulating time. We have hundreds of video processing tests that are slow because they need to run in "real time". But by intercepting syscalls, and returning faked results, we could run the system clock 30x faster! 🧵👇️

thanks pg&e for reducing my screen time

for those outside SF, the context is that the power has been out since 2:30 the original ETA for resolution was 3:45. It's been pushed back incrementally, and now it's 10:00

what other books belong in the cracked engineer starter pack?

AI coding agents have been less useful for us than we expected. They're quite good for problems that are "in the training set". But our workload is very atypical and very large scale, so the majority of the hard problems we encounter are unique and never-seen-before.

today we run 18 million EC2 instances per month three years ago, we outgrew Kubernetes as an eng team of 3 To replace it, we built our own high-performance cluster scaler that can manage hundreds of thousands of nodes, and can scale capacity +500% in less than 10 sec

We are hiring! If you're interested in writing Rust, running high performance infrastructure at massive scale: DM me your GitHub :)

we run postgres with 20k concurrent writers. at this scale, it's mandatory to deeply understand the internals of every piece of infra we ran into a performance bottleneck, where our query performance was limited by the rate that Linux could fork child processes! 🧵 👇️

fun javascript fact, if you pass a string numeric to the Date constructor, you get a totally crazy result new Date("0") results in Jan 1, 2000 new Date("1") results in Jan 1, 2001 (+1 year, ok....) new Date("2") results in Feb 1, 2001 (+1 month, ??????) 🧵👇

Recall.ai is the API for meeting recording. We are infra for 1000’s of companies including HubSpot, ClickUp, and Rippling. In 2025 we grew 3x, raised our Series B and learned what it actually means to find strong PMF and execute. We only made it because of the help

We launch over 18 million EC2 instances per month. One thing we've learned is that cold-booting an EC2 instance is very slow, but this happens for a surprising reason. When you boot an EC2 instance from an AMI, creating a new root EBS volume is near-instantaneous. However,

We are hiring! If you're interested in writing Rust, running high performance infrastructure at massive scale: DM me your GitHub :)

10 things you didn’t know about the EC2 API, part 1. Fact: RunInstances returning a 201 with N instances does not mean you have actually been allocated N instances. Any number of those instances can fail to boot with Server.InsufficientInstanceCapacity.

We launched 18 million EC2 instances last month. 10 things you didn’t know about the EC2 API, part 2. Fact: AWS's code path for booting new EC2 instances is buggy. When you boot a lot (e.g. 10,000s) of EC2 instances you often see failures with Server.InternalError

We are hiring! If you're interested in launching millions of instances, operating large-scale video processing clusters, and writing high-performance Rust: DM me your GitHub :)

Did you know AWS has a quota for quota increase requests???

it's also not possible to request an increase to your quota increase request quota 🤔

getting to the scale where running a load test costs real money