Google's reference architecture does something subtle and important: it moves model selection out of the backend and up into the request path, using a Cloud Run payload inspector to extract an OpenAI-style model identifier and inject an X-Gateway-Model-Name header for URL-map based routing. That one design choice—model-aware routing at the load‑balancer layer—changes how you think about identity, observability, and policy for private inference.
Two concrete pieces land at the same time. First, Agent Compute Engine (preview) gives teams a managed runtime for agent execution. Second, Agent Substrate is an open-source, zero-trust execution runtime you can run on GKE if you need portability, CRD-level customization, or full control over sidecars and networking. Those are complementary, not competing, and Google explicitly baked governance surfaces across the Agent runtime, Cloud Run, GKE, and Agent Compute Engine in the preview.
The operational pattern Google documented for MCP (multi‑customer proxy / model control plane) apps makes the integration explicit: sign-in and workforce authorization happen against the configured identity and workforce pools for Gemini-capable access controls, while the MCP server itself runs as a private Cloud Run service. That private MCP plugs into Discovery Engine, the agent platform surfaces, and Organization Policy; critically, it relies on Cloud Run service accounts and Workload Identity (or federated identity flows) so agent-to-tool requests can carry auditable identity tokens. In short: user identity and agent identity can be injected and audited at the platform edge, not faked by individual containers.
Where this gets interesting for platform engineers is the mixed-backend inference architecture. The documented flow uses a small Cloud Run extension to parse OpenAI-compatible request bodies, pull out the model field, and add X-Gateway-Model-Name. A Google Cloud Load Balancer URL-map routes to backends (a GKE-hosted inference gateway, Cloud Run services, the agent platform, regional internal HTTP(S) load balancers or Private Service Connect endpoints, hybrid clusters, or external clouds) based on that header. On the security side the path wires in Apigee for identity and quota enforcement and a prompt/response screening layer for content policy checks before the request ever touches a model host.
That design is smart for three reasons:
- It centralizes routing logic so your backends can be heterogeneous (specialized GPU pools on GKE, light CPUs on Cloud Run, vendor-hosted inference) without changing client code.
- It surfaces model-level metadata at the load‑balancer where you already implement quotas, observability hooks, and policy enforcement.
- It keeps private inference private: Private Service Connect, internal HTTP(S) load balancers, and GKE-hosted inference gateways are first-class options for backends.
But it also raises operational complexity. Moving model selection into the edge requires a model registry and clear ownership: who updates URL-maps when a model is deprecated or hot-fixed? How do you audit which code paths can flip the X-Gateway-Model-Name? The Cloud Run inspection extension is a small, convenient piece of code, but it becomes a critical trust boundary. Treating it as a sidecar hack will get you bitten.
This is the right call from Google: provide a managed runtime (Agent Compute Engine) for teams that want a boxed, governed experience and an open substrate for teams that must own networking, CRDs, or custom schedulers. But platform teams must treat the routing plane and identity injection as first-class components: automated URL-map deployments, model-aware SLOs, and model-level telemetry are no longer optional.
If you’re building private MCP deployments with Gemini, deploy the MCP server to Cloud Run as shown and configure Cloud Run service accounts and Workload Identity so the MCP can obtain and present identity tokens for downstream calls. But don’t stop at connectivity—add model ownership, a CI gate for URL-map changes, and prompt/response rules in your screening layer before any model becomes routable. That’s where governance actually happens.
This announcement isn’t just another managed service feature. It signals that cloud vendors expect platform teams to operationalize models the same way they do services: identity at the edge, policy before execution, and routing that understands the model as an addressable resource. If you still treat models like glorified HTTP endpoints with ad‑hoc proxies, expect costly surprises. Start treating your model registry, URL-maps, and policy attachments as the primary surface area for security and reliability.
For more on the agent runtime and governance implications, see my previous coverage of the preview: Google Cloud Agent Compute Engine (preview): managed agent runtime and expanded governance.
My prediction: within a year you'll see toolchains that manage URL-maps, model-canarying, and screening rules as part of the model CI pipeline. If your platform isn’t ready to automate those changes safely, you’ll end up with brittle, manual edges—and that’s exactly the sort of incident that will force a re-architecture mid-deployment.