Skip to content

← Writing

engineering

Designing for Failure

· Jerwin Arnado · 7 min read ·

Part nine of the series. Every previous chapter assumed things mostly work. This one assumes they don’t — because at scale, something is always failing. A disk, a network link, a third-party API, a deploy gone wrong. The Availability & Recovery stack post covered backups and getting back up after a fall; this chapter is about the patterns that keep one component’s failure from becoming your outage in the first place. The mindset shift: stop designing for the happy path and hoping; design for failure and be pleasantly surprised when things work.

The cascade: how one failure becomes an outage

The scary thing about failure in a connected system isn’t the first failure — it’s the cascade. Service A calls slow service B; A’s threads pile up waiting on B; A runs out of threads; now A is down too, and everything calling A starts to fall. One slow dependency takes down the whole chain. The entire toolkit below exists to break that chain — to contain a failure to the component that had it.

Payment gateway gets slow (not even down — just slow)
  → your checkout threads all block waiting on it
  → your web server runs out of threads
  → your ENTIRE site stops responding, including pages that never touch payments
One slow dependency, total outage. This is what we design against.

Timeouts: the foundation

Everything starts here. A call with no timeout is a call that can hang forever, and a hung call holds a thread hostage — the first link in the cascade. Every network call — database, cache, HTTP to a third party — needs an aggressive, explicit timeout.

// No timeout = hang forever when the dependency is slow. Always set one.
$response = Http::timeout(3)->connectTimeout(1)->post($gatewayUrl, $payload);

The default in most HTTP clients is no timeout or an absurdly long one. That default is a latent outage. Set timeouts short enough that a slow dependency fails fast instead of dragging you down with it — a failed request you can handle; a hung one you can’t.

Retries — with backoff and jitter

Many failures are transient: a blip, a brief network hiccup, a momentary overload. Retrying fixes those. But naive retries make things worse, and understanding why is important:

  • Retrying instantly hammers a struggling service — it’s already overloaded, and you just sent the same request again immediately.
  • Retrying in lockstep synchronizes clients — everyone fails at once, everyone retries at the same instant, creating a wave that re-overloads the recovering service. The stampede, again.

The fix is exponential backoff with jitter: wait longer between each attempt (1s, 2s, 4s…), and add randomness so clients don’t retry in sync.

// Exponential backoff (2s, 4s, 8s...) with jitter so retries don't synchronize
public function backoff(): array
{
    return [2, 4, 8];
}
public $tries = 3;
// Laravel adds jitter; or randomize yourself: sleep(2 ** $attempt + rand(0, 1000)/1000)

And the non-negotiable prerequisite: retries are only safe if the operation is idempotent. Retrying a non-idempotent charge is how you double-bill. Backoff + jitter + idempotency is the complete, safe retry.

Circuit breakers: stop knocking on a dead door

Retries handle transient failures. When a dependency is genuinely down, retrying is pointless — you’re wasting time and resources hammering a corpse, and slow-failing every request that depends on it. A circuit breaker watches the failure rate and, once it crosses a threshold, “trips”: for a cooldown period it fails instantly without even trying the dependency.

CLOSED  → calls pass through, failures counted
        → too many failures? trip →
OPEN    → calls fail INSTANTLY (don't even try) for a cooldown, giving the dep time to recover
        → cooldown elapsed? →
HALF-OPEN → let one test call through. Success → CLOSED. Failure → back to OPEN.

The breaker does two things: it fails fast (a tripped call returns in microseconds instead of timing out over 3 seconds, so your threads stay free), and it gives the dying dependency room to recover instead of piling on. This is the primary tool against the cascade.

Bulkheads and graceful degradation

Two more patterns complete the kit:

  • Bulkheads: named after a ship’s watertight compartments — one floods, the ship stays afloat. Isolate resources so one struggling dependency can’t consume all your capacity. Give the flaky third-party integration its own limited connection pool or its own queue; when it backs up, it exhausts its pool, not the threads your core app needs.
  • Graceful degradation: when a non-critical dependency fails, lose the feature, not the page. Recommendations service down? Render the product page without the “you might also like” section. Search down? Show a message and keep the rest of the site working. The art is deciding, per dependency, what to do when it’s gone — and defaulting to “degrade, don’t crash.”
// Graceful degradation: the page survives the recommendation service dying
try {
    $recommendations = $this->recommender->for($user);   // behind a circuit breaker + timeout
} catch (ServiceUnavailable) {
    $recommendations = collect();   // lose the widget, keep the page
}

Caveats and best practices

  • Set a timeout on every network call. No exceptions. The un-timed call is the single most common cause of a “why is the whole site down?” incident. Audit for defaults — they’re usually “wait forever.”
  • Retry only idempotent operations, always with backoff + jitter. Instant lockstep retries turn a blip into an outage; retrying non-idempotent writes turns one action into several. Both halves matter.
  • Add circuit breakers to your flakiest dependencies. Anything external and critical — payment gateways, third-party APIs. Failing fast keeps a dead dependency from consuming your capacity while it’s down.
  • Decide degradation per dependency, in advance. For each external thing, answer now: “if this is down, do we lose a feature or the whole page?” The answer should almost always be the feature. Write it down before the incident, not during.
  • Test failure deliberately. Turn off the cache, block the payment gateway, add latency in staging. The failure paths you never exercise are the ones that surprise you at 3am — the same lesson as an untested backup.

Conclusion

Cascade → one slow dependency blocks threads → the whole chain falls
Timeout → every network call, aggressive, explicit — or it hangs forever
Retry → exponential backoff + jitter, and ONLY if idempotent
Circuit breaker → trip after N failures, fail fast, let the dependency recover
Bulkhead → isolate resources so one flood doesn't sink the ship
Degrade → lose the feature, not the page

Failure isn’t an edge case at scale — it’s the steady state, and the difference between a resilient system and a fragile one is entirely in how it contains failure. Aggressive timeouts, idempotent retries with jitter, circuit breakers on the flaky stuff, bulkheads to compartmentalize, and graceful degradation as the default response to “it’s gone.” Assume everything breaks; make sure the break stays small. Next section — The User’s Clock — flips to perception with optimistic UI and prefetching.