Google Cloud just pushed a production playbook for running inference at cluster scale: llm-d 1.0. The surprise isn't that it exists — it's the level of operational detail baked into a Kubernetes-native stack: prefix-aware routing (cache-aware request routing), multi-tenant flow control, a tiered KV cache offload, cross-cluster/global failover and a managed L7 Inference Gateway that glues it together.
Put bluntly: this moves a lot of the hard work of latency and cost engineering out of ad-hoc application code and into the cluster platform. The numbers Google published are concrete: llm-d's prefix-cache hit rate on the Gemini serving path rose from ~35% to ~70%, time-to-first-token for coding agents dropped by >35%, and p95 TTFT fell by >50% during bursty chat traffic. Those are the kinds of improvements that matter for real user-facing agent workloads.
llm-d supports off-the-shelf inference runtimes (vLLM and other popular runtimes) and common accelerators (TPUs, GPUs) and exposes traffic-control primitives so different tenants or agents don't stomp each other during spikes. The GKE Inference Gateway provides managed L7 routing for llm-d, with cross-cluster and cross-region scaling and global failover — and it layers in built-in safety checks before traffic hits model endpoints. That combination is telling: Google expects teams to run model-serving topologies across clusters and regions and to treat safety and observability as first-class platform responsibilities.
This is the right call. For years platform teams have duct-taped sharded caches, homegrown request routers, and noisy co-tenancy mitigations around single-node inference services. Centralizing routing and cache semantics at the cluster level reduces duplicated effort and makes predictable SLAs possible. But it also expands the attack surface and operational responsibilities in ways teams underestimating will regret.
Operational implications you should care about now
- Cache semantics: prefix-aware routing and a tiered KV offload means cache-consistency and eviction policies now matter for correctness, not just performance. If you rely on cached prompt prefixes for context reconstruction, design your invalidation strategy now.
- Multi-tenancy & flow control: llm-d's traffic shaping prevents noisy neighbor effects, but it requires platform-owned admission controls and quotas. That's now part of your SRE charter.
- Safety & auditing: the built-in safety filters are useful, but upstream filtering isn't a substitute for end-to-end audit trails. Expect to run additional detection or logging layers for high-risk workflows.
Cloud Run + MCP: a new deployment pattern
Google also documented a Cloud Run pattern for hosting a private Model Connector endpoint (MCP) that Gemini can talk to over a /mcp JSON-RPC-style endpoint. The recommended deployment is a private, authenticated service with a warmed worker and these flags to keep the instance available:
gcloud run deploy my-mcp \
--image=gcr.io/my-project/mcp-server:latest \
--region=us-central1 \
--no-cpu-throttling \
--min-instances=1 \
--ingress=internal \
--no-allow-unauthenticatedThat --no-cpu-throttling + --min-instances=1 combo is important: connector services often poll and maintain persistent state; you need a non-ephemeral runtime to avoid cold-starts and missed messages. If you still treat Cloud Run as only for strictly stateless request/response workloads, this should change your view.
Gemini for government services
Separately, Google announced a preview of Gemini for government services with government-specific skills, agency-system integrations and deployment options that target high-assurance boundaries. If you run public-sector workloads, this makes Gemini more viable — but operational controls, identity integration, and the data boundary will require platform-level plumbing rather than ad-hoc pulls.
What’s missing (and what matters)
Two practical gaps: I couldn't find authoritative recent updates on GCP pricing for these llm-d paths or Vertex AI release-note entries explicitly tied to llm-d. That suggests teams will need to pilot to surface real cost and operational behaviors before adopting at scale.
Final take: llm-d 1.0 is the infrastructure move platform engineering teams should have made two years ago. It standardizes request routing, caching and failover for model-serving while exposing the knobs teams actually need to hit production SLAs. But standardization also means responsibility: if you adopt llm-d, you must own cache correctness, flow-control policies, and safety audits — and if you don't, your reduction in latency will be the least of your problems. For teams who treat models as a remote API, this is the nudge that will make inference into a first-class infra concern.
Related reading: see the Google Agent Development Kit scaffolds for Cloud Run and GKE and the practical writeup on the Gemini persistent runtime for deployment recipes.