Raft: What the Paper Means in Production

The paper in brief

"In Search of an Understandable Consensus Algorithm" by Diego Ongaro and John Ousterhout appeared at USENIX ATC in 2014. Raft solves the same problem as Paxos, getting a group of servers to agree on a sequence of commands so they can act as a replicated state machine. Its distinguishing goal is understandability: the authors decomposed the problem and ran a user study comparing how well students learned Raft and Paxos.

Key ideas

Strong leader. At any time one server is the leader. Clients send commands to it, it appends them to its log, and it replicates them to followers. Data flows in one direction, from leader to followers, which simplifies reasoning.

Terms and leader election. Time is divided into numbered terms. A follower that hears nothing from a leader for a randomized timeout becomes a candidate, increments the term and asks for votes. A candidate wins with a majority. Randomized timeouts make split votes rare and quickly resolved.

Log replication and commitment. An entry is committed once the leader has stored it on a majority of servers. The log matching property ensures that if two logs contain an entry with the same index and term, the logs are identical up to that point.

Safety rules. A voter refuses a candidate whose log is less up to date than its own, so any elected leader holds all committed entries. A leader only counts replicas to commit entries from its own term, a subtle rule that the paper explains with a worked scenario.

Membership changes. The paper describes changing cluster membership using a joint consensus approach, and a simpler single-server-change approach was later described by Ongaro in his dissertation.

What breaks in practice

Understandable does not mean trivial to implement. Implementers repeatedly find bugs around the subtle parts: persisting term and vote before replying, handling stale leaders, snapshotting and log compaction, and membership changes. The persistence requirement matters most. A server that forgets its vote after a restart can vote twice in a term and break safety, so correct fsync behavior is part of correctness, not only of performance.

Timeouts need to fit the environment. If the election timeout is shorter than network or disk latency spikes, a healthy leader is deposed, and the cluster churns. If it is too long, failover is slow. Multi-region clusters make this tuning harder, since the quorum latency now includes the slowest necessary link.

Throughput is limited by the single leader. All writes pass through one machine, and each write waits for a majority round trip. Reads need care too. Serving a read from a leader that has been deposed without knowing it returns stale data, so implementations use read-index or lease mechanisms, and leases depend on assumptions about clocks.

Cluster size is a trade-off. Three or five voting members is typical. More members tolerate more failures but make each commit wait on more replicas and add little benefit in most deployments.

Raft also assumes crash failures, not malicious ones. It does not tolerate Byzantine behavior.

Modern descendants

Raft is used widely. etcd, the store that holds Kubernetes cluster state, uses it. HashiCorp Consul and Nomad use it, and CockroachDB and TiKV run many independent Raft groups, one per data range. Apache Kafka introduced KRaft, a Raft-based metadata quorum, to remove its dependency on ZooKeeper. Because the paper is a clear specification, many independent libraries exist, which also means quality varies.

Key takeaways

• Raft provides replicated state machines with a strong leader, majority commit and randomized election timeouts.
• Durable storage of term and vote before responding is a safety requirement.
• Election timeouts must be tuned against real network and disk latency.
• A single leader bounds write throughput, and safe reads need extra mechanisms.
• Understandability helped adoption, but production-grade implementations still demand deep testing.