# Write-Ahead Logging: The Only Trick Durability Has

> Every database, filesystem, and message broker converges on the same move: append the intention to a sequential log, fsync, then update the real structures whenever. WAL is one idea wearing a dozen names — and its failure modes repeat too.

URL: https://harshith.in/blog/write-ahead-logging/
Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/
Tags: databases, systems, filesystems

A transaction touches five pages scattered across a B-tree. Crash
mid-write and the tree is neither old nor new — it's shrapnel. You
cannot make five random writes atomic; disks don't offer that. But you
can make *one sequential append* durable, and that asymmetry is the
whole foundation: *write what you intend to do into a log, make the log
durable, then do it.* Postgres WAL, InnoDB redo log, ext4's journal,
Kafka, Raft's log, RocksDB — one idea, a dozen uniforms. Learn it once
with the failure modes attached and half of storage engineering unlocks.

## The Protocol

```text
  1. append to log:  "txn 42: page 7 bytes 96..128 := <new>" (+ more records)
  2. append:         "txn 42 COMMIT"
  3. fsync the log   ◀── the transaction is durable HERE
  4. report success
  5. update the real pages in memory; write them back... eventually
```

The rule that names the technique: **no data page reaches disk before
the log records describing its changes** (write-*ahead*). Recovery is
then mechanical: replay the log forward from the last checkpoint (redo),
and for transactions with no COMMIT record, either ignore their records
(redo-only designs) or roll them back (undo). Crash at any step and you
get either all of txn 42 or none of it.

Notice what got bought. **Performance**, not just safety: the durable
write path became *sequential appends to one file* — the access pattern
both disks and SSDs love — while the random-write chaos happens lazily
in memory and trickles back in the background. And **group commit**
compounds it: ten transactions committing within the same millisecond
share a single fsync (one flush, ten durability guarantees) — the
mechanism behind every "how does this database do 50k commits/sec on
one disk" number, and behind knobs like Postgres's `commit_delay` and
Kafka producers' batching. Latency-vs-throughput at the fsync boundary
is *the* tunable of durable systems: relaxations like
`synchronous_commit = off` (ack before flush, bounded loss window on
crash) are honest engineering trades when stated, and quiet data-loss
bugs when unstated — MongoDB's early unacknowledged-write defaults
taught the industry that lesson publicly.

## Where WAL Implementations Get Hurt

The protocol is simple; the storage stack underneath is treacherous,
and the classic wounds are all documented incidents:

- **fsync must mean it.** Consumer drives ack writes from volatile
  cache; a power cut eats "durable" bytes. Barriers/FUA and honest
  disks (or capacitor-backed enterprise SSDs) are prerequisites, not
  options. Worse: Linux fsync *error* semantics — a failed fsync could
  mark pages clean, so retrying it "succeeds" over lost data. That's
  fsyncgate (2018): Postgres now PANICs on fsync failure, on the theory
  that crash-and-recover-from-WAL beats believing a liar.
- **Torn pages.** An 8 KB page write isn't atomic against 4 KB-sector
  power loss — half old, half new, checksum garbage. Postgres logs
  **full-page images** on first touch after each checkpoint (why WAL
  volume spikes right after checkpoints); InnoDB routes through its
  **doublewrite buffer**. Same threat, two armors.
- **Checkpoints exist to bound recovery**, and they're the other
  latency villain: flushing all dirty pages at once creates I/O storms
  (hence spread/fuzzy checkpointing, `checkpoint_completion_target`),
  while checkpointing rarely means longer crash-recovery replays. The
  triangle — recovery time, steady-state I/O, WAL retention — has no
  fourth option.

One more gift for free: the log is a **replication feed**. Ship WAL to
a replica and replay it — physical replication, PITR archives, and
(decoded) logical CDC pipelines like Debezium are all "read someone
else's WAL." The log turns out to be the database's most reusable
interface — which is the observation Kafka generalized into a company,
and Raft formalized into consensus: replicate the log, and state
machines replaying it agree by construction.

## Takeaways

- Random multi-page atomicity is impossible; one durable sequential
  append is easy — WAL converts the first problem into the second.
- Durability point = log fsync; group commit amortizes it and is where
  commit throughput comes from.
- The stack lies: verify write caches/barriers, treat fsync failure as
  fatal (fsyncgate), and armor against torn pages (FPI/doublewrite).
- Checkpoints trade steady-state I/O against recovery time; tune the
  spread, accept the triangle.
- The WAL is also the replication/CDC/consensus interface — the log is
  the database, everything else is a cache of it.