Google Cloud's biggest operational pivot this week isn't another faster GPU SKU—it's a rethink of where and how agents run. Agent Compute Engine (preview) introduces a hardware‑isolated sandbox for LLM agents and ships alongside Agent Substrate (an open zero‑trust execution runtime for Kubernetes) and a sweep of governance controls that reach into Cloud Run, GKE, hybrid stacks and air‑gapped deployments. In short: agent execution is now a first‑class trust boundary.
A new trust boundary: hardware‑isolated agents
Agent Compute Engine promises hardware isolation and a runtime purpose‑built for agent workflows. Agent Substrate, its Kubernetes counterpart, adds an interesting operational primitive: the ability to snapshot idle agent memory and resume later. Snapshot/resume is a great optimization, but it also creates a long‑lived artifact you must protect—memory snapshots are just another credential store if they contain tool tokens, API keys, or conversational context with sensitive PII.
Google's governance narrative isn't limited to a single runtime. Cloud Run and other serverless platforms can use Workload Identity and short‑lived tokens to authenticate agent tool calls, and governance controls are being extended across Agent Substrate, Cloud Run, GKE, Agent Compute Engine, cross‑cloud and on‑prem clients. That authenticated agent→tool connectivity is extremely useful—it's also a new attack surface that most orgs' IAM and audit models aren't designed to cover. If you think of agents as ephemeral bots, this announcement should make you stop and redesign your identity flow.
Distributed inference: llm‑d finally speaks Kubernetes
llm‑d is shaping up as an open, Kubernetes‑native distributed inference stack that integrates with projects like vLLM, supports prefix‑aware routing and KV caching, and enables disaggregation across accelerator pools. GKE Inference Gateway, a managed component, adds L7 routing, global failover and model safety checks as first‑class controls. The practical effect is that inference topologies—mixed accelerator pools, tiered caching, and prefix‑aware routing—now have an operator story that fits Kubernetes rather than living as bespoke orchestration scripts.
This is overdue. Until now, building a multi‑tier inference surface with cache locality, prefix routing and global failover meant stitching together custom tooling. llm‑d and the GKE Inference Gateway give operators primitives that map to real SLOs: cold‑start avoidance, region failover, and model safety checks.
GKE‑level primitives that matter
Several cluster features landed that matter for AI orchestration: managed RDMA support exposes RDMA‑capable NICs as schedulable Kubernetes resources so RDMA‑aware apps can be placed correctly, and Kueue's topology‑aware scheduling improvements make it easier to co‑schedule GPU/TPU pods across racks. Google also expanded its GPU instance lineup with newer H100‑backed and RTX‑class options aimed at large‑scale training and simulation workloads.
Sovereignty: Gemini models on Google Distributed Cloud
Gemini models are available on Google Distributed Cloud in air‑gapped and sovereign configurations, letting teams run multimodal, long‑context models within jurisdictional boundaries—useful for regulated industries. Google didn't announce major public API or pricing changes tied to this availability, so procurement and integration work will be the gating factor for many teams.
What platform teams should actually do (opinionated)
Treat agent runtimes as distinct trust planes. Enforce least privilege for token injection (via Workload Identity and short‑lived credentials), audit snapshot storage and lifecycle, and bake model safety checks and prefix‑aware routing into your inference ingress. If you don't rework IAM, RBAC and network policies with these primitives in mind, you'll end up with agent tokens floating in Cloud Storage or other object stores and a compliance nightmare.
Also: stop treating inference as a single‑node problem. llm‑d and GKE Inference Gateway make a multi‑tier, multi‑accelerator topology operationally tractable; build for cache locality, scheduling topologies and global failover now, not later.
Final thought
This wave of announcements isn't incremental. Google is drawing execution boundaries around agents and then providing orchestration, routing, safety checks and sovereign deployments for those boundaries. Platform teams who ignore the distinction between an agent control plane and a model inference plane will pay for it in risk and operational complexity. Within a year, your infra diagrams should show an "agent compute" plane with its own policies—or you're going to be running expensive firefights when an agent leaks credentials or snapshots.
Sources
- Introducing Agent Compute Engine and expanding agent governance
- Announcing llm-d 1.0: Open distributed inference for agents on Google Cloud TPUs and GPUs
- Google Cloud infrastructure at NVIDIA GTC Berlin 2026
- Scale robotics with Google Cloud physical AI and NVIDIA
- New innovations in Google Distributed Cloud