How One Server Holds a Million Connections

Someone asked me this on a call last week, half as a trick question: how many connections can one server hold at once? The honest answer is “more than you’d guess, and the number that kills you isn’t the one you’re thinking of.” A single, ordinary Linux box can hold a million open TCP connections. Not a cluster — one machine. WhatsApp did two million on one server back in 2011, on hardware that a mid-range homelab now embarrasses. The reason it feels impossible is that most of us carry around a mental model that was never built for it.

The model that breaks

The intuitive picture is one connection, one thread. A client connects, you hand it a thread, that thread blocks on read() until bytes show up, does the work, writes a reply, loops. Clean. It’s how the first generation of servers worked, and it’s still how a lot of tutorial code works.

It falls over fast. A thread needs a stack, and on Linux the default stack reservation is 8 MB of address space (a smaller amount actually committed, but not nothing). Stack aside, each thread is a scheduling entity the kernel has to track and switch between. Do the arithmetic on a million:

You run out of RAM long before a million, and even if you didn’t, the scheduler would spend all its time switching contexts instead of doing work. Thread-per-connection dies somewhere in the low tens of thousands. That wall has a name.

C10K: the pivot

In 1999 Dan Kegel wrote up the C10K problem — how do you serve ten thousand clients on one server at once? At the time that was the frontier. The answer that won wasn’t “faster threads.” It was to stop pairing a thread with a connection at all.

Here’s the insight: at any given moment, almost all of your connections are doing nothing. A million people with a chat app open are, mostly, just sitting there with a connection held open. You don’t need a million threads waiting on a million sockets. You need one (or a handful of) threads that ask the kernel a single question — which of these million sockets have something for me right now? — and then service only those.

That question used to be expensive. The old select() and poll() calls make you hand the kernel the entire list of sockets every single time, and the kernel walks all of them: O(n) per call, which is death when n is a million. The fix was readiness notification that scales:

• epoll on Linux (kernel 2.6, 2002)
• kqueue on the BSDs and macOS

You register your sockets once. The kernel then hands you back only the ones that are ready. The cost is proportional to the number of active connections, not the number of open ones. That single change — from “scan everything” to “tell me only what changed” — is the engine under nginx, Node.js, Redis, HAProxy, and every other server that holds a lot of sockets with very few threads. It’s the whole ballgame.

A connection at rest is almost free

Once you drop the thread, you have to ask what a connection actually costs, because that’s the real budget. An established, idle TCP connection is essentially:

• one file descriptor in your process,
• one struct sock and friends in kernel memory (a few KB of bookkeeping),
• a send buffer and a receive buffer.

The buffers are where the money goes. Linux sizes them dynamically between the floor and ceiling in net.ipv4.tcp_rmem and tcp_wmem, and the defaults are generous — tens to hundreds of KB per socket — because they’re tuned for throughput on a few connections, not a million idle ones. That default is the difference between fitting in RAM and not:

1,000,000 connections × 200 KB of buffers  = ~200 GB   ← won't fit
1,000,000 connections × 4 KB of buffers     = ~4 GB     ← fits easily

For a workload of mostly-idle connections that exchange small messages — chat, presence, push, live dashboards — you deliberately tune the buffers down. You’re trading peak per-connection throughput (which idle connections don’t use) for the ability to hold a staggering number of them. That’s the core move behind the big numbers.

The four walls you actually hit

Getting to a million is less about raw horsepower and more about knowing which limits to raise. There are four, and the defaults are set for a laptop, not a fleet.

1. File descriptors. Every connection is a file descriptor, and the default soft limit is a laughable 1024 (ulimit -n). You raise the per-process limit, and the system-wide ceilings fs.file-max and fs.nr_open behind it. This is the wall everyone hits first, and it shows up as the maddening Too many open files.

2. Memory — the socket buffers above. The one that’s easy to forget until the OOM killer reminds you. This is the real ceiling; everything else is a config flag.

3. The port myth. Here’s the one people get wrong constantly. “You only have 65,535 ports, so a server tops out at 64k connections.” False, and it matters. A TCP connection is identified by the four-tuple: source IP, source port, destination IP, destination port. Your server listens on one port — say 443 — and every one of a million clients connects to that same port. They’re distinguished by their IP and port, not yours. So one listening port holds millions of connections, no problem. The 65k limit is real, but it applies to outbound connections from one source IP to one destination — it’s a client-side and proxy-side concern, not a “how many can my server accept” concern.

4. The stateful firewall / conntrack. The sneaky one. If your box runs a stateful firewall (netfilter’s connection tracking), every connection also takes a slot in the nf_conntrack table, and nf_conntrack_max has its own default that’s nowhere near a million. Blow past it and new connections get dropped with a cryptic log line while your application looks innocent. Either raise it or bypass conntrack for the traffic that doesn’t need it.

None of these four is exotic. They’re sysctl lines and a LimitNOFILE in your service file. But if you don’t know they exist, you’ll conclude the machine “can’t do it” when really you just hit a 1994-era default.

The napkin math

The rounded numbers I keep in my head (same spirit as the latency numbers — orders of magnitude, not precision):

Add it up and a million idle connections is roughly 8–16 GB of RAM plus some file descriptors — comfortably inside a single modern server, and within reach of a beefy homelab node. The machine isn’t the bottleneck. The defaults are, and the application memory per connection is, which is why the servers that win at this are written to hold almost nothing per connection.

It’s been done, more than once

This isn’t theory. The proof points are worth knowing:

• WhatsApp, 2011 — famously got 2 million connections on a single FreeBSD server, and later pushed past 3M. Their whole engineering culture was “hold a lot of connections cheaply,” built on Erlang, whose lightweight processes are practically designed for it.
• Phoenix / Elixir, 2015 — the team demoed 2 million concurrent WebSocket connections on one box. Same lineage: the BEAM VM gives you millions of cheap processes, so one-process-per-connection actually works where one-OS-thread never could.
• The C10M problem, 2013 — Robert Graham argued at Shmoocon that to go ten times further you have to stop asking the kernel for help at all, and move the TCP stack into user space (the DPDK / kernel-bypass world). That’s the frontier past a million, and it’s a different kind of engineering — but it exists, and it’s why “ten million on one box” is a real target and not a joke.

Different languages, different decades, same shape: don’t give a connection a thread, and don’t let it own much memory.

The part everyone misses

Here’s the twist that reframes the whole thing. Idle connections are a memory-and-file-descriptor problem. Active connections are a CPU problem, and they are completely different.

A million connections that sit there holding a keepalive is a storage question — fit the bookkeeping in RAM and you’re done. A million connections all sending a message right now is a firehose: a million wakeups, a million buffer allocations, a million trips through your serialization and your business logic, possibly a million garbage-collector headaches. The connection count was never the hard part. The work per active connection is.

That’s why the real engineering at scale is about keeping the per-message path cheap — no allocation you can avoid, no lock you can skip, no O(n) broadcast where an O(1) one would do. Holding the connections open is the easy 90%. Doing something with them is the other 90%.

Why a working dev should care

You are probably never going to hand-tune a box for a million sockets. I haven’t. But the mental model changes decisions you make every week:

• WebSockets and live features aren’t as scary as they look. Holding thousands of open connections for presence, notifications, or a live dashboard is cheap if you don’t spend memory per connection. The instinct to reach for polling “because open connections are expensive” is usually backwards.
• Connection pools exist for the opposite reason. Your app pools database connections precisely because those connections are expensive — a Postgres backend is a whole process with real memory, nothing like an idle socket. Knowing which connections are cheap and which are costly is the whole game.
• When something “can’t scale,” suspect a default, not the machine. Too many open files, a conntrack table overflowing, a buffer setting eating your RAM — these masquerade as hard limits and are almost always one config line away from gone.

A million connections on one server isn’t a magic trick. It’s what you get when you stop pairing each client with a thread, keep almost nothing per connection, and raise four limits that someone set to 1024 a long time ago. The box was always capable. We just have to ask it correctly.