Borg to Kubernetes: What the Lineage Means in Production

The paper in brief

"Large-scale cluster management at Google with Borg" by Abhishek Verma and colleagues appeared at EuroSys in 2015. It describes the system that runs Google's workloads across clusters it calls cells. A companion article by Brendan Burns, Brian Grant, David Oppenheimer, Eric Brewer and John Wilkes, "Borg, Omega, and Kubernetes", appeared in ACM Queue in 2016 and discusses lessons from the three systems. Kubernetes was open sourced by Google in 2014 and reflects experience with Borg.

Key ideas

Declarative jobs and tasks. Users describe what they want, such as a job with a number of tasks and their resource needs, and the system works to make reality match. The Borg paper describes a central Borgmaster, replicated for availability, and an agent on each machine called the Borglet.

Mixing workloads for utilization. Borg runs long-running services and batch jobs on the same machines, with priorities and preemption so high-priority work can displace lower-priority work. Sharing machines raises utilization, which the paper presents as a major cost saving.

Resource declarations and limits. Tasks state resource requests, and the system enforces limits. The paper discusses reclaiming resources that tasks reserved but do not use.

Names and discovery. Tasks get stable names and the system supports service discovery and load balancing, so applications do not hardcode machine addresses.

Operational lessons. The authors list what they would do differently, including that the job as the only grouping mechanism is limiting, and that labels would be a more flexible way to organize things. Kubernetes took this up with labels and selectors, and with pods as the unit of scheduling.

What breaks in practice

Kubernetes is Borg-inspired, not Borg itself. The control plane differs: Kubernetes keeps state in etcd behind an API server, and independent controllers watch that state and reconcile it. This level-triggered, eventually consistent style is resilient, and it surprises people who expect a command to take effect immediately. A change is a statement of intent, and convergence can take time or never happen if something is wrong.

Resource requests and limits are the most common source of trouble. Requests drive scheduling, limits drive throttling or termination. Setting them wrong leads to wasted capacity, noisy neighbors, CPU throttling or out-of-memory kills. The utilization benefits reported in Borg depend on disciplined resource accounting, which many clusters never reach.

Scale and complexity are a cost. A cluster scheduler is a distributed system with its own failure modes, and the control plane itself becomes a critical dependency. Small teams often adopt it for workloads that did not need it.

State is hard. Borg and Kubernetes were designed first around replaceable tasks. Running databases needs persistent volumes, stable identities and careful operators, which came later and remain harder than stateless services.

Multi-tenancy is weaker than the isolation the large-scale paper assumes. Containers share a kernel, and strong isolation between untrusted tenants needs extra measures.

Modern descendants

Kubernetes is now the dominant container orchestrator, offered as a managed service by major cloud providers, and it is maintained under the Cloud Native Computing Foundation. Its ideas, such as declarative configuration, reconciliation loops, labels and custom resources, spread well beyond scheduling. Nomad and Apache Mesos explored related designs. Google has published that Borg remains in use internally.

Key takeaways

• Borg's central idea is declarative desired state with a control plane that reconciles toward it.
• Sharing machines between services and batch work raises utilization, but only with accurate resource accounting.
• Kubernetes inherits lessons from Borg and differs in architecture, using labels, pods and etcd-backed controllers.
• Requests and limits are the most consequential settings for stability and cost.
• Stateful workloads and strong tenant isolation need more than the basic scheduler provides.