Latency, Throughput, and Napkin Math

Part one of the series, and we start with physics because the rest of the book is negotiable and this chapter isn’t. Every design sits on top of a few hard numbers — how long it takes to read from memory versus disk versus across a network — and engineers who’ve internalized them make better calls in seconds than others make in a week of meetings. You don’t need them memorized to the nanosecond. You need them by feel, as orders of magnitude.

The numbers, rounded to what you’ll remember

Jeff Dean’s famous “latency numbers every programmer should know,” rounded hard enough to actually stick:

The right-hand column is the trick that makes them stick: multiply everything by a billion so a nanosecond becomes a second. Now the gaps are visceral. RAM is minutes; SSD is days; a cross-continent round trip is five years. That single intuition — memory is absurdly close, the network is absurdly far — explains why caching exists, why the N+1 query is a disaster, and why we put CDNs near users.

Why “average latency” lies

Report a single latency number and you’ll report the mean, and the mean will lie to you. Latency distributions have long tails: most requests are fast, a few are catastrophically slow, and the average hides both. What you actually want are percentiles:

• p50 (median) — half of requests are faster than this. The typical experience.
• p99 — 99% are faster; 1% are slower. The bad-day experience.
• p99.9 — the truly unlucky.

Why the tail matters more than the average: on a page that makes 20 backend calls, the slowest of the 20 sets the page’s speed. With a p99 of 1 second, a page fanning out to 20 calls has roughly a 1 - 0.99^20 ≈ 18% chance of hitting at least one of those slow calls. Your “1% slow” backend just made nearly one in five page loads slow. Tail latency is the real latency for anything that fans out — which is almost everything.

Average says:  "requests take 80ms"   ← comforting, useless
p50 / p99 say: "usually 40ms, but 1 in 100 takes 1.2s, and
                pages that make 20 calls feel that 1.2s often"

This is also why chasing the average is the wrong optimization. Shaving the median from 40ms to 30ms feels good and changes nothing; taming a 1.2s p99 changes how the whole app feels.

Napkin math: the five-minute estimate that prevents months of waste

Before designing anything, estimate its scale. Not precisely — within an order of magnitude is plenty, and that’s usually enough to make the decision. Say I’m sizing the appointment system for a clinic product:

Clinics:              1,000
Appointments/clinic:  50 per day
Writes:               1,000 × 50           = 50,000 appointments/day
                      50,000 / 86,400 s    ≈ 0.6 writes/second
Reads (view-heavy):   ~100× writes         ≈ 60 reads/second
Data/year:            50,000 × 365 × ~1 KB ≈ 18 GB/year

Now look at what that tells you. Under one write per second. Sixty reads per second. Eighteen gigabytes a year. That is nothing. A single modest MySQL box handles thousands of writes per second and stores that dataset for a decade without blinking. Every instinct to shard, to add a queue, to reach for a distributed database — deleted by five minutes of arithmetic. The system needs good indexes and maybe a read cache, full stop.

That’s the real power of napkin math: it’s not for proving you need scale, it’s usually for proving you don’t. The estimate that says “one server, for years” is the one that saves the most money.

Caveats and best practices

• Estimate before you architect, every time. The number that says “you don’t need it” is the most valuable output. Cargo-culting a big-tech design onto a small-tech problem is the most common expensive mistake in the field.
• Measure percentiles, not averages. Your error tracking and metrics should graph p50/p95/p99. If a dashboard shows only the mean, it’s hiding your worst user experiences.
• Know your read:write ratio early. It’s the single number that most shapes the design — read-heavy wants caches and replicas, write-heavy wants a different data model entirely.
• Sequential beats random, memory beats disk, local beats network — always. When a design feels slow, find where it crossed one of those boundaries more than it needed to. That’s usually the N+1 query or the chatty API.
• Round aggressively. “About 100 nanoseconds,” “about a millisecond,” “tens of GB.” Precision here is false confidence; the order of magnitude is what drives the decision.

Conclusion

Feel the gaps → RAM = minutes, SSD = days, cross-continent = 5 years
Tail > average → p99 is the real latency once a page fans out
Napkin math → data size + RPS + read:write, to one order of magnitude
Usual result → "one server, for years" — the estimate that saves the most

The numbers are the ground truth every later chapter negotiates against. Feel the distance from memory to disk to network, trust percentiles over averages, and run the five-minute estimate before the five-week build. Next: scaling and statelessness — what to do when one server genuinely isn’t enough, and the one property your app needs before it can scale out at all.