The xz Backdoor: Open Source's Near-Miss

Archive note: this is a backdated post, written years later while rebuilding this site. It’s dated to the moment it covers, but the hindsight is real.

On March 29, Andres Freund — a Postgres developer at Microsoft — published the most consequential “hey, something’s weird” email in the history of open source. He’d noticed SSH logins consuming ~500ms more CPU than they should, plus some valgrind noise, and pulled the thread. At the end of the thread: a deliberate backdoor in xz/liblzma 5.6.0–5.6.1, the compression library that — through a chain involving systemd — links into sshd on major Linux distributions.

The payload targeted the front door of, approximately, every server on Earth. It shipped in bleeding-edge distro releases (Fedora 40 beta, Debian sid) and was days to weeks from reaching stable distributions — at which point the attacker would have held authentication-bypass capability against a meaningful fraction of the internet. It was caught by one engineer’s refusal to ignore half a second.

The part that should keep you up: it wasn’t code

Log4Shell was an accident. node-ipc was a maintainer’s protest. This was a multi-year intelligence operation against a person.

The account “Jia Tan” appeared in 2021 and spent years becoming a trusted xz contributor: useful patches, patient presence. In parallel, sockpuppet accounts pressured the original maintainer, Lasse Collin — a volunteer maintaining a foundational library alone, unpaid — about slow releases and his mental-health-related absences, until co-maintainership passed to Jia Tan. The backdoor itself was tradecraft: hidden in binary test files, activated only during package builds, visible in neither the git source nor casual review. Then social-engineering pushes to get the poisoned versions into distros faster.

Every layer of the attack exploited the same vulnerability, and it isn’t in any CVE database: the open-source commons runs on exhausted volunteers whose trust is grantable to anyone patient enough to earn it. The xkcd tower-of-blocks panel stopped being a joke in 2021; this week it became a confirmed nation-state-grade target list. (Attribution is unproven, but the patience, skill, and objective speak a clear dialect.)

What actually worked, and what to do

The honest, uncomfortable scorecard: defense-in-depth mostly didn’t catch this — luck and one human did. But the response showcased open source’s actual superpower: within hours of disclosure, the entire world could audit the same code, distros shipped reverts, and the full forensic timeline was public. Try that with a proprietary vendor’s build pipeline.

The practical residue, for those of us running servers and homelabs:

1. Slow lanes are a security feature. The backdoor reached only bleeding-edge distros. Debian stable’s legendary conservatism — the thing we joke about — is precisely what bought the time for discovery. My new-release quarantine rule gets a vindication sticker.
2. SSH hygiene is non-negotiable anyway: keys not passwords, no exposed port 22 to the world, VPN-first access. The next backdoor may not get caught.
3. Investigate your anomalies. The entire defense of the internet came down to someone benchmarking 500 milliseconds. The unexplained slow thing in your system is sometimes just entropy. Sometimes it is not.
4. Fund the load-bearing volunteers. Every analysis lands here, so let it be repeated until budgets move: critical infrastructure maintained by one burned-out person is not a quirky open-source tradition. It is an attack surface with a name and a sleep schedule.

We get exactly one free lesson like this. The next Jia Tan read the same post-mortems we did — and there is no reason to believe there’s only one.