The conversation in platform engineering has shifted from assembling toolchains to shipping product experiences. The primary failure mode for an internal developer platform (IDP) is not a missing CI tool but the lack of product discipline: small, focused Minimum Viable Platforms (MVPs), opinionated golden paths developers actually use, and outcome-based metrics. This article is a pragmatic playbook for senior engineers and platform product leads: design opinionated defaults, enforce deterministic guardrails, instrument adoption and outcomes with Four Keys–style telemetry, and keep a controlled path for edge cases.
Treat golden paths as product features
Golden paths are shipped surfaces, not just docs. Treat each golden path like an external product feature: versioned APIs, discoverability in the developer catalog, an SLA for maintenance, and telemetry that reports successful end-to-end completion.
Concrete elements of a golden path:
- A scaffolder template (Backstage Scaffolder task or repo template) that creates the repo, CI pipeline, default manifests, and a feature-flag entry.
- An opinionated CI workflow with a small, audited set of runnable steps (checkout, language/runtime setup, lint/test, security scans) that also emits a platform telemetry event when runs succeed or fail.
- A standard CD target: an Argo CD Application manifest, Flux Kustomize/Helm pattern, or your chosen GitOps App of record.
Example: a minimal GitHub Actions workflow that installs dependencies, runs tests, and emits a telemetry event to your platform ingestion endpoint. Note: send telemetry to your platform's event API (collector or ingestion service) rather than attempting to speak raw OTLP wire format from a shell script.
# .github/workflows/ci.yml
name: CI — Golden Path
on:
workflow_dispatch: {}
push:
branches: ["main"]
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Use Node 18
uses: actions/setup-node@v4
with:
node-version: 18
- name: Install and Test
run: |
npm ci
npm run lint --if-present
npm test
- name: Publish telemetry event
env:
TELEMETRY_ENDPOINT: ${{ secrets.TELEMETRY_HTTP_ENDPOINT }}
run: |
cat <<EOF > /tmp/event.json
{
"event": "ci_success",
"repo": "${{ github.repository }}",
"commit": "${{ github.sha }}",
"workflow": "${{ github.workflow }}",
"timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
}
EOF
curl -s -X POST "$TELEMETRY_ENDPOINT/events" -H 'Content-Type: application/json' --data-binary @/tmp/event.jsonThis pattern gives you a uniform place to collect pipeline-level telemetry and a reproducible onboarding flow.
Standardization with controlled flexibility
Standardization reduces cognitive load and enables safe automation. The pattern that scales is opinionated defaults plus policy-based exceptions.
- Opinionated defaults: a documented platform contract such as default deployment strategy (RollingUpdate with maxSurge=25%), standard probe configuration, built-in OpenTelemetry SDK and collector export path, and a canonical Prometheus metrics endpoint (/metrics). Make these defaults visible in the scaffolder output and the catalog entry.
- Policy engine for exceptions: use a Kubernetes admission controller (OPA Gatekeeper, OPA with admission integration, or Kyverno) as the enforcement point. Enforce deterministic guardrails (no privileged containers, required liveness/readiness probes, image provenance) and provide an explicit exception path via a ticketed workflow or a Backstage approval flow.
Example Rego policy (deny images not hosted on a corporate registry unless an approved provenance annotation is present). This is illustrative—adjust keys and admission input shapes to match your admission controller integration.
package platform.policies.images
default allow = false
allow {
# Accept images hosted on the approved registry prefix
images := [c.image | c := input.request.object.spec.template.spec.containers[_]]
all_images_valid(images)
}
all_images_valid(images) {
not some i
startswith(images[i], "ghcr.io/")
}
deny[msg] {
not allow
images := [c.image | c := input.request.object.spec.template.spec.containers[_]]
msg = sprintf("images %v must be on ghcr.io or include approved provenance annotations", [images])
}The platform's role is to make the compliant path easy and exceptions explicit and auditable.
Instrumentation: Four Keys as the outcome lens
If you treat the platform as a product, instrument both adoption signals (catalog views, template usages) and engineering outcomes (Four Keys: deployment frequency, lead time for changes, mean time to restore, and change failure rate).
Practical telemetry sources:
- CI/CD events emitted from GitHub Actions, Jenkins, or Tekton into a central event pipeline (an ingestion API or collector). Prefer structured events (JSON) that include repo, commit, workflow, environment, and status.
- Incident and on-call timelines from PagerDuty, Opsgenie, or your incident tracker to compute MTTR.
- Catalog metadata (Backstage catalog-info.yaml) showing ownership, component kind, and environment targets to join events to teams.
A Four Keys-style BigQuery example to compute lead time for changes (time from commit to first successful production deploy). Adjust table/field names to match your event schema.
-- Lead time for changes: commit -> first successful production deploy
WITH commits AS (
SELECT commit_sha, repo, MIN(commit_time) AS commit_time
FROM `project.repo_commits`
WHERE commit_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY commit_sha, repo
),
prod_deploys AS (
SELECT repo, commit_sha, MIN(deploy_time) AS deploy_time
FROM `project.deploy_events`
WHERE environment = 'production' AND status = 'success'
GROUP BY repo, commit_sha
)
SELECT
DATE(commits.commit_time) AS day,
APPROX_QUANTILES(TIMESTAMP_DIFF(prod_deploys.deploy_time, commits.commit_time, SECOND), 100)[OFFSET(50)]/3600.0 AS median_lead_hours,
COUNT(DISTINCT commits.commit_sha) AS commits_count
FROM commits
JOIN prod_deploys USING(repo, commit_sha)
GROUP BY day
ORDER BY day DESC
LIMIT 30;Surface these metrics in a platform dashboard (Backstage plugin, Grafana, or a BI dashboard) and track adoption metrics (scaffolder template usages, catalog searches) alongside outcomes. The key signal is the delta in lead time and deployment frequency between teams that adopt the golden path and those that do not.
AI accelerates workflows but not control planes
AI accelerates developer work (scaffolding, triage, runbook suggestions) but increases the need for strong interfaces and deterministic guardrails:
- Automation at scale raises blast-radius. Ensure LLM-generated scaffolds produce the same golden-path outputs as your canonical templates.
- Use model-in-the-loop systems that propose changes but route outputs through the platform's approval and CI gates: generate PRs via the scaffolder API, run the canonical CI workflow, and emit the same telemetry.
- Detect drift with continuous policy checks (OPA/Kyverno) and telemetry anomaly detection so AI-suggested changes don't evade patterns.
An integration pattern: an LLM agent that writes repository content, calls the Scaffolder REST API to create a task or run a template, and lets the platform's CI/CD and policy systems enforce delivery and telemetry capture.
Implementation specifics to wire today
- Scaffolding: Backstage Scaffolder REST API (POST /scaffolder/v1/tasks or the template action endpoints) or GitHub repo templates with the Actions dispatch API. Record template usage in catalog entries (kind: Component, schemaVersion: 1.0.0).
- Policy: OPA with admission hooks, Gatekeeper, or Kyverno for Kubernetes admission policies. Keep policies in a platform repo and gate exceptions via a Backstage approval workflow tied to audit logs.
- Observability: require OpenTelemetry SDK usage and a central collector endpoint. Standardize exporter environment variables (OTEL_EXPORTER_OTLP_ENDPOINT or your platform's TELEMETRY_HTTP_ENDPOINT) and document runtime instrumentation expectations.
- Feature flags: centralize in Unleash, LaunchDarkly, or an equivalent. Enforce a naming convention (org.team.service.feature) and include staged rollout configuration in scaffolder output.
- Secrets and tokens: use a managed secrets store (HashiCorp Vault, AWS Secrets Manager) and enforce short-lived service tokens for platform agents.
Recommended next steps (practical)
- Start small: ship a single MVP golden path for a common service type (for example: Node 18 HTTP service) and instrument Four Keys signals for teams that adopt it.
- Make defaults explicit and enforceable: document platform contracts (rolling update strategy, probes, metrics endpoint, feature-flag convention) and enforce them with admission policies and the scaffolder.
- Assign product ownership: a platform product lead or team should own the golden path SLAs, roadmap, and outcome measurements.
- Integrate AI thoughtfully: allow LLMs to generate scaffolds but route their outputs through the same Scaffolder/CICD pipeline and policy checks.
- Prove value with telemetry: instrument CI/CD events, deploy events, incidents, and catalog interactions before expanding to more features.
If you already run a platform, reframe: stop proliferating integrations and harden a small set of product-grade golden paths, add deterministic policy enforcement, and instrument adoption and outcomes. The shift is from tool integration to defining, shipping, and iterating on product experiences teams actually use and measure.