Skip to content

← Writing

engineering

Replication and Quorums

· Jerwin Arnado · 6 min read ·

Part five of the series. Replication is keeping copies of your data on more than one machine, and it’s the concrete machinery behind the last two chapters: it’s why partitions matter and where replication lag comes from. You replicate for two reasons — read scaling (more copies to read from) and redundancy (a copy survives when a server dies) — and you pay for both in consistency. This is how the machinery actually works.

Leader/follower: the common case

The default topology, and the one you’ll use 95% of the time: one leader (primary) takes all writes; one or more followers (replicas) copy the leader’s changes and serve reads.

         writes          ┌──────────┐
  client ───────────────▶│  LEADER  │
                         └────┬─────┘
              replication ────┼──── replication
                         ▼          ▼
                   ┌──────────┐ ┌──────────┐
       reads ─────▶│ FOLLOWER │ │ FOLLOWER │◀───── reads
                   └──────────┘ └──────────┘

This maps perfectly onto a read-heavy app — the appointment system from chapter one was ~100:1 reads to writes, so sending all those reads to followers takes enormous load off the leader. In Laravel it’s config, not code:

// config/database.php — writes go to the leader, reads fan out to followers
'mysql' => [
    'read'  => ['host' => ['10.0.0.2', '10.0.0.3']],  // followers
    'write' => ['host' => ['10.0.0.1']],              // leader
    'sticky' => true,                                  // read-your-writes, per last chapter
],

Note sticky again — replication is exactly where the read-your-writes bug lives, so the fix travels with the topology.

Synchronous vs asynchronous: the lag knob

When does the leader consider a write “done”? Two answers, and this is the PACELC latency-vs-consistency trade made physical:

  • Synchronous: the leader waits for follower(s) to confirm before telling the client “saved.” No lag — a follower read right after the write is current. Cost: every write is as slow as the slowest follower, and if a follower is down, writes stall.
  • Asynchronous: the leader confirms immediately and ships changes to followers in the background. Fast writes, always available — but followers lag, so reads can be stale. This is the default, and the source of nearly all replication-lag bugs.

Most systems run async for speed and manage the staleness with the middle-menu consistency guarantees. Some run semi-synchronous — wait for one follower, not all — as a middle ground: durability of the write survives the leader dying, without waiting on the whole fleet.

Failover, and the split-brain nightmare

The leader dies. Something must promote a follower to be the new leader — that’s failover, and it’s where replication gets genuinely dangerous. The hard problem is split-brain: if the old leader isn’t actually dead but merely partitioned off, and you promote a follower, now you have two leaders both accepting writes. They diverge, and when the network heals you have two conflicting versions of reality and no clean way to merge them.

Network hiccup, not a real death:
  OLD LEADER (isolated, still taking writes) ── ✂ ── FOLLOWER promoted to new leader
                                                       (also taking writes)
  Network heals → two histories, conflicting writes, data loss on merge. Split-brain.

This is why “just automate failover” is harder than it sounds, and why the mechanism that prevents split-brain is the same idea that tunes consistency: quorums.

Quorums: the dial between fast and safe

A quorum is a simple, powerful rule: require a majority to agree before an action counts. With N copies, require W nodes to acknowledge a write and R nodes to serve a read. The magic condition:

W + R > N   ⇒   any read set overlaps any write set ⇒ reads see the latest write

With N = 3: choose W = 2, R = 2. 2 + 2 > 3, so every read of 2 nodes is guaranteed to include at least one node that saw the latest write. You get strong-ish consistency without waiting for all nodes — tolerating one being down. And the same “majority” idea prevents split-brain: a partitioned minority can’t form a majority, so it can’t elect a leader or accept writes. It knows it’s the losing side and steps down.

The quorum is a dial, not a switch — it’s how you tune the consistency/latency trade numerically:

Setting (N=3) Behavior
W=3, R=1 Fast reads, slow/fragile writes (all must ack) — read-heavy, rarely-written data
W=1, R=3 Fast writes, slow reads — write-heavy, rarely-read data
W=2, R=2 Balanced, survives one node down — the sensible default

This is the same tradeoff from CAP and PACELC, now with actual knobs you set to numbers.

Caveats and best practices

  • Async replication is the default, and its lag is a feature you must manage, not a bug to ignore. Use read-your-writes where it matters and accept staleness where it doesn’t. Graph the lag — it’s an early warning that a follower or the network is struggling.
  • Read replicas scale reads, not writes. All writes still funnel through one leader. When the write volume is the ceiling, replication doesn’t help — that’s when you look at partitioning/sharding (a stretch topic beyond this core series).
  • Automated failover needs fencing. Without a majority-quorum or an explicit fencing token to shut the old leader out, automated failover can cause the split-brain it was meant to survive. Managed databases (RDS, Cloud SQL) handle this for you — a strong reason to use them over hand-rolled replication.
  • Test failover before you need it. A replica you’ve never promoted and a backup you’ve never restored are the same untested safety net. Kill the leader in staging and watch what happens.
  • Quorum math is your consistency dial. W + R > N is the line to remember: cross it for strong reads, stay under it for speed. Write it down when you configure any quorum store.

Conclusion

Leader/follower → one writer, many readers; scales a read-heavy app cheaply
Sync vs async → sync = no lag but slow/fragile; async = fast but stale (the default)
Failover → promoting a follower risks split-brain if the old leader isn't truly dead
Quorum → W + R > N guarantees reads see latest; majority also prevents split-brain

Replication is the engine under the whole Consistency & Data section — it buys read scale and redundancy, and it manufactures every lag and split-brain risk you then have to manage with consistency models and quorums. Master leader/follower and W + R > N and the rest is tuning. That closes this section; next we move between services: sync vs async communication.