Google Cloud just turned an academic idea1distributed, prefix-aware inference1into a production-grade fabric. llm-d 1.0 arrives with stable v1 APIs and first-class integration into GKE Inference Gateway, and Google reports prefix-cache hit rates moving from 35% to 70% and P95 time-to-first-token dropping by more than 50% under bursty chat load. If you operate agentic chat systems, thats the single most material performance lever you havent been using.
llm-d isnt a new model; its an open-source inference stack that coordinates TPUs, GPUs, and software runtimes to present low-latency, cache-friendly inference as a networked service. With v1 APIs and GKE Inference Gateway wiring, teams get three operational features out of the box: prefix-aware routing (so identical conversation prefixes hit cached activations), cross-region failover, and Model Armor checks (policy gates and sanity filters applied before traffic reaches a model). Those are orthogonal to raw model size 1 they attack tail latency and safety at the routing layer.
Why prefix-aware routing matters
Chat workloads are bursty in both users and tokens. A lot of the expensive work is repeated: identical system prompts, shared boilerplate, or partially overlapping contexts across parallel sessions. llm-ds prefix cache records activations for those prefixes and reuses them; when combined with an inference gateway that understands routing by prefix, you get cache locality and fewer token-generation cycles on the slow path. The numbers Google published are not trivial: moving from 35% to ~70% cache hit means far fewer GPUs/TPUs running hot for the same QPS, and a halved P95 time-to-first-token changes how you architect SLAs and autoscalers.
Operationally useful features
- Prefix-aware routing: the gateway can route a request to a node or node pool that already holds the cached prefix activations, reducing generation work.
- Cross-region/global failover: llm-d plus the gateway can redirect traffic on partial regional failure while trying to preserve cache affinity where possible.
- Model Armor checks: pluggable policy and safety checks applied at the routing layer to drop or reroute risky queries before they consume model cycles.
These are not vapor features. Integration with GKE Inference Gateway means you can run the fabric inside your cluster topology, retain observability with Prometheus and OpenTelemetry, and stitch policies into existing ingress and service-mesh controls. Live inference is not just a CPU/GPU problem 1 its a networking and operational-consistency problem.
Context: agents and runtimes
This release lands alongside Googles broader push on agent runtimes (Agent Runtime, Cloud Run, GKE, agent sandboxes, and a new Agent Compute Engine preview). If youre building agent stacks, expect to combine runtime choices with inference routing and secure tool connectivity (for example, private service connections). I covered Agent Compute Engine earlier and its worth reading alongside this: Agent Compute Engine (preview): hardware-isolated runtime for LLM agents.
What platform teams should do right now
Treat inference routing as part of the control plane. If you only worry about model selection and raw throughput, youll keep paying for GPUs and chasing tail latency with ad-hoc scale rules. llm-d + GKE Inference Gateway moves much of that work into deterministic routing and cache control1meaning platform engineers can own latency guarantees again. Add tests for prefix-cache eviction, observe token-level latencies at the gateway, and codify Model Armor policies into CI for safety compliance.
A final take: this is overdue and decisive. Distributed inference has been stuck between model engineering and infra for too long; llm-d 1.0 forces the separation of concerns platform teams wanted1routing, caching, and safety belong in the networked inference fabric, not as hacks inside model-serving containers. If you run agentic or heavily templated chat apps, shift the conversation on your roadmap from "more GPUs" to "smarter routing". The teams that do will get predictable tail latency and much lower operating cost per chat.
One question to watch: will vendors standardize a small set of routing semantics (prefix identity, activation reuse, cache sharding) or will each cloud build its own incompatible fabric? The former gives us a usable ecosystem; the latter just buys more vendor lock-in under the name of performance.