GKE node-pool upgrades: how maxSurge and maxUnavailable affect capacity

Google Cloud quietly made one of those doc clarifications that will cost on-call teams sleep: in GKE Standard node-pool upgrades, maxSurge is the number of extra nodes the control plane will create to keep your capacity up, while maxUnavailable is the count of nodes it will drain at the same time. That distinction is obvious once you read it, but many ops teams have been operating on implicit assumptions — and those assumptions break during autoscaler-driven, multi-region, and GPU-heavy workloads.

Here’s the operational reality: set maxSurge too low and upgrades will drain your nodes faster than the cluster can add replacements. Combine that with aggressive PodDisruptionBudgets, non-preemptible GPUs, or tight CPU/memory headroom, and the upgrade becomes a classic "out of resources" event. Set maxUnavailable too high and you intentionally permit concurrent drains that reduce capacity. They’re different knobs; they need to be tuned together.

Surge upgrades and capacity math

GKE performs surge-style rolling upgrades by creating up to maxSurge additional nodes, cordoning and draining up to maxUnavailable nodes, then deleting old nodes when replacements are ready. Think of maxSurge as the temporary buffer and maxUnavailable as the drain concurrency limit. If your cluster autoscaler or node auto-provisioning is configured to scale slowly, or if pod scheduling is blocked by admission controllers, taints, or lack of GPU capacity, those extra nodes might not appear fast enough.

Two concrete failure modes I've seen in customer environments recently:

A node pool running GPU inference with maxSurge: 0 and maxUnavailable: 1. During upgrades, the control plane drained nodes while the autoscaler tried to create GPU nodes, but quota and placement delays meant pods couldn't schedule, causing service errors.
A mixed-instance node pool with low headroom and maxSurge: 1, maxUnavailable: 2. The cluster briefly had fewer schedulable CPUs than expected because two drains removed capacity before the single surged node became ready.

Those are preventable with a short checklist: raise maxSurge to match the worst-case replacement latency for your node image and zone; set maxUnavailable conservatively for stateful or latency-sensitive workloads; and validate that autoscaler and node auto-provisioning quotas cover the surge.

Why Google’s clarification matters now

Recent GCP updates include Cloud Run worker pools, additional Vertex AI Gemini variants, and finer-grained billing attributes for AI and container workloads. Teams are mixing more GPU/AI workloads and non-HTTP workers, so cluster upgrade behavior is a higher-stakes problem. If you run worker pools across regions, your Kubernetes node pools are more likely to pump and drain capacity unpredictably during upgrades.

The correct engineering stance is not "leave defaults". Google did the right thing by spelling the semantics out, but the onus is on platform teams to run upgrade rehearsals against realistic workloads: GPUs, long-start containers, sidecars that delay readiness, admission webhooks, and heavy autoscaler interactions. Treat maxSurge as deliberate capacity, not a placebo.

A short actionable nudge (opinionated): defaulting both knobs to small values with no test coverage is malpractice. If you rely on bursty AI inference or long-running background workers, set maxSurge to at least the number of nodes you need to absorb a worst-case drain latency, and make maxUnavailable reflect how many simultaneous disruptions your SLA tolerates.

Final thought: this is one of those infra clarifications that separates teams who understand their upgrade surface from teams who will be surprised at 03:00. The product updates around Cloud Run workers, Gemini variants, and billing granularity are all useful, but the real operational risk right now is subtle orchestration: node surges, API quotas, and scheduling edge-cases. Audit your node-pool upgrade knobs this week and run a realistic upgrade drill — you’ll either sleep through the next maintenance window, or you’ll know exactly why you won't.

GKE node-pool upgrades: how maxSurge and maxUnavailable affect capacity

Sources

Cloud Run worker pools GA: pull-based non-HTTP workers and multi-region HA pattern

Gemini Pro preview on Vertex AI and the Gemini API — Cloud Run worker pools GA

Cloud Run worker pools GA — pull-based non-HTTP workers and multi-region failover pattern