# Raft: Consensus You Can Actually Trace

> Raft won because you can hold the whole protocol in your head: terms, a leader, and one log-matching rule. The mechanics of election and replication — and the operational corners (fsync, membership changes, pre-vote) where deployments actually bleed.

URL: https://harshith.in/blog/raft-in-practice/
Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/
Tags: distributed-systems, systems

I've written before about why quorums exist; this is the protocol layer
above them. Raft (Ongaro & Ousterhout, 2014) was explicitly designed
around *understandability* — the paper's actual thesis — and it worked:
etcd, Consul, CockroachDB, TiKV, and most Kafka deployments (KRaft) now
bet their correctness on it. The payoff of learning it properly is that
distributed-systems logs stop being mystical: `term 847`,
`leader election lost`, `proposal dropped` are all lines you can trace
back to specific rules.

## The Frame: Terms and One Leader

Time divides into numbered **terms**; each term has at most one leader,
elected by majority vote. All writes flow through the leader — Raft
buys simplicity by refusing multi-master entirely. Every message
carries the sender's term; seeing a higher term instantly demotes you
to follower. That one rule quietly handles most of the "stale leader"
zoo: a partitioned ex-leader rejoins, hears term 848 > its 847, and
steps down before doing damage.

**Election**: followers expect heartbeats; silence past a randomized
timeout (150–300 ms class) triggers candidacy — increment term, vote
for self, request votes. Majority → leader. The randomization is the
whole liveness trick (simultaneous candidacies split votes; random
timeouts make repeats improbable), and the *vote-granting rule* is the
first safety pillar: a node refuses its vote to any candidate whose
log is less up-to-date than its own. Elected leaders therefore already
hold every committed entry — recovery requires no log transfer to the
new leader, ever.

## Replication: The Log Matching Property

The leader appends client commands to its log and ships them via
AppendEntries, each carrying the *previous* entry's (index, term) as a
consistency probe:

```text
  leader:   ... [5:t2] [6:t3] [7:t3]        AppendEntries(prev=6:t3, [7:t3])
  follower: ... [5:t2] [6:t3]           ✓ matches → append 7
  follower: ... [5:t2] [6:t2]           ✗ reject → leader backs up, overwrites
```

Induction does the rest: if a follower matches at (index, term), its
entire prefix matches — divergent suffixes (from crashed leaders'
unreplicated tail entries) get found and overwritten mechanically. An
entry is **committed** once replicated on a majority *and* — the subtle
clause that fixed a real bug class (the paper's Figure 8) — the leader
only counts replication of entries *from its own term*; older-term
entries commit implicitly behind them. Committed entries feed the state
machine, and the guarantee stack — elected leaders have all committed
entries + log matching + commit rule — yields the property everything
else stands on: **committed means durable across any minority of
failures, in order, exactly once in the log** (delivery to *your*
application is another story — see below).

## Where Deployments Actually Bleed

The protocol is the easy half; a decade of etcd/Consul incident
reports names the hard one:

- **fsync is part of the protocol.** Votes and log entries must hit
  stable storage before being answered — a node that votes, crashes,
  and forgets its vote can elect two leaders in one term. Slow disks
  therefore *are* slow consensus: etcd's
  `wal_fsync_duration_seconds` alerting exists because commit latency
  is disk latency; running consensus on shared/burstable storage is a
  well-documented way to spend a weekend.
- **Elections during GC/CPU stalls.** A leader that pauses past the
  timeout gets deposed; the cluster churns through terms
  ("leadership transferred" storms). Tuning timeouts against your
  worst-case pause — and using **pre-vote** (a candidate first checks
  it *could* win, without incrementing terms) plus leader
  stickiness — is what keeps a flaky node from term-bombing a healthy
  cluster. If your logs show terms climbing by hundreds, it's this.
- **Membership changes are a protocol, not a config edit.** Jumping
  straight from {A,B,C} to {A,B,C,D,E} lets two disjoint majorities
  exist mid-change. Raft's answers (joint consensus, or
  one-node-at-a-time — what etcd enforces, now with learner/non-voting
  states to catch new nodes up safely) are why "just add two replicas
  at once" is refused by good implementations.
- **Reads need care too.** Serving reads from any leader replica risks
  stale data from a deposed leader; linearizable reads use ReadIndex
  (confirm leadership with a quorum round) or leases — the reason
  etcd distinguishes `--consistency=l` from serializable reads, and a
  knob every Raft-backed store exposes somewhere.

Scope honesty completes the picture: Raft totally orders a *log* —
it says nothing about exactly-once effects in your application
(dedup/idempotency still yours), it costs a quorum round-trip per
write (don't put it on hot paths that don't need it), and a majority
loss halts writes by design — availability was traded for the
guarantee, which is the correct trade for the configuration/metadata/
coordination tier it dominates.

## Takeaways

- Terms + majority elections + higher-term-wins handle stale leaders;
  randomized timeouts handle liveness.
- Log matching lets one (index, term) probe verify entire prefixes;
  commit = majority replication of a current-term entry.
- Consensus latency = fsync latency; monitor WAL sync times before
  blaming the network.
- Pre-vote and single-node membership changes are what production
  Raft looks like; their absence is what incidents look like.
- Raft orders the log — idempotency, read consistency modes, and the
  cost of quorum rounds remain your problems.