Google just shipped an inference stack that treats model latency and safety as platform responsibilities, not ad-hoc application hacks. llm-d 1.0 is a Kubernetes-native, open-source distributed inference stack that runs on GKE and spans vLLM, TGI (Text Generation Inference), Google TPUs and NVIDIA GPUs, with verified Helm charts and a new GKE Inference Gateway that does L7 routing, prefix-aware endpoint selection, cross-cluster/regional failover and integrated safety checks (Model Armor). In production runs cited by Google, prefix-cache hit rates doubled (35% → 70%), time-to-first-token (TTFT) for coding agents dropped >35%, and P95 TTFT during bursts fell by >50%.
The important detail is that Google shipped platform primitives—routing, tiered caching, and safety—so the stack can exploit repeated prefixes and fail over across regions without application-level glue. The result is real user-facing latency improvements for agentic and coding workloads where the beginning of the prompt dominates cost and user experience.
llm-d 1.0 details matter. The stack supports vLLM and TGI backends, verified Helm charts for GKE, and hardware targets that include TPUs and NVIDIA GPUs. The Inference Gateway can run llm-d in a gateway role and make prefix-aware endpoint selection decisions at L7, factoring prefix-cache state and safety checks into routing. Cross-cluster and regional failover are treated as first-class capabilities.
Why prefix-aware routing matters
For interactive and code-generation workloads, the first tokens are everything. Prefix caches (warm caches keyed by prefixes or prompt fragments) are extremely effective, but only if your router sends requests to nodes holding warmed caches. A naive load balancer will smash caches and amplify cold-start tail latencies. Making the gateway prefix-aware moves that complexity into platform code with consistent semantics across clusters: route to warmed instances, fall back across regions intelligently, and surface cache-hit metrics to operators. That’s how Google reported a 70% prefix-cache hit rate and much lower P95s.
Model Armor in the gateway is the other operationally significant piece. Pushing safety and policy checks into the L7 path forces decisions at the traffic boundary where enforcement and telemetry are most efficient. Teams used to bolt on safety post-hoc, scattering policy enforcement and observability. Centralizing checks at the gateway simplifies enforcement, but it also becomes a chokepoint and a new attack surface you must monitor and audit.
Open model choice and the platform play
Google emphasized broader model choice by enabling the platform to route to different model providers via provider adapters and integrations. In practice that means operators can configure the gateway to proxy or route to hosted Google models, on-prem models, or externally hosted model endpoints. That’s good for avoiding vendor lock-in; it’s harder for ops because you now manage heterogeneous runtimes and consistency semantics across providers.
Other pieces worth noting: Google said larger Gemini-class models are available for regulated and air-gapped deployments via Google Distributed Cloud, and GKE node pools with high-end NVIDIA GPUs (including Blackwell-class hardware where offered) are being positioned for large-scale simulation and training. If you’re building simulation-driven agent training, that hardware plus GKE networking and the llm-d stack is intended to be a full-path solution.
This is the right call from a platform POV. Teams kept re-implementing routing, cache warmers, and safety checks in app code—painful, brittle, and inconsistent. Packaging those concerns into a verified, deployable stack with Helm charts reduces repeated engineering. That said, don’t fool yourself: you’re not handing off complexity; you’re moving it into infrastructure. Expect new operational burdens—cache coherence, gateway performance, regional failover choreography, audit trails for gateway-enforced policies, and observability for prefix-cache behavior.
If you care about interactive LLM latency, treat llm-d 1.0 as a platform milestone, not a drop-in reduction of work. Start planning SLOs around TTFT and P95 during bursts, instrument prefix-cache hit rates, and bake safety-policy observability into your dashboards. For a concise extra read on deployment scaffolds for agent runtimes on GKE, see Google's Agent Development Kit coverage: Google Agent Development Kit (ADK): Cloud Run, Gemini runtimes, and GKE deployment scaffolds.
Prediction: within 12 months, every cloud provider will ship inference gateways that do routing + cache steering + policy enforcement. Teams that still treat LLMs like stateless HTTP backends will be surprised by their first bursty traffic incident. Operating models are shifting—the battle for latency and safety is now fought at the gateway.