When Git Main Is Not What Production Serves

You merged to main. The build passed. You closed the ticket.

Meanwhile www still runs last Tuesday.

That gap — git truth vs routing truth — is one of the most expensive illusions in small-team SaaS. Staged publish and build-time tests stop bad code from going live; they do not guarantee good code ever becomes LIVE when webhooks stall, Git integration breaks, or a deploy sits READY but never gets the domain.

Two different questions

Cowpin already treats the first as non-negotiable (docs/deploy-safety.md). The second needed its own monitor (COW-170).

What “drift” looks like in practice

Common shapes:

1. Pushes to GitHub stop creating Vercel builds (token, integration, or quota).
2. A build finishes READY but auto-publish is off and the promote step never ran.
3. LIVE lags main because an older build was promoted manually while a newer one waits.
4. You think you deployed because git push succeeded — but push ≠ live after staged release.

Operators feel this as “the fix is on main but customers still see the bug.” Engineers burn cycles re-debugging code that was already fixed.

What we monitor

The AWS synthetic monitor (same Lambda family as staged publish checks) now runs git deploy drift on schedule:

1. Read commits/main from the GitHub API.
2. Read Vercel LIVE production deployment metadata (githubCommitSha).
3. If they match → OK.
4. If main has a READY build that is not LIVE yet → allow a grace window (default 20 minutes) for normal staged promote latency.
5. If drift exceeds grace → email + file monitor alert (same channel as other deploy failures).

Implementation: monitoring/synthetic-monitor/git-deploy-drift.mjs, wired from synthetic-monitor/index.mjs. Local ad-hoc check: node scripts/verify-vercel-git-deploy.mjs.

This complements — does not replace — deployment.error webhooks and the “unpublished build > 30 min” email from the release step.

Why 20 minutes?

Shorter windows cry wolf during legitimate staged checks (landing npm honesty, DB paths, warm-up after promote). Longer windows let a stuck pipeline lie to support all afternoon. Twenty minutes is a pragmatic default for a site that usually publishes within a few minutes of green checks; tune if your promote path is slower.

What we did not build

• Automatic promote on drift — dangerous if the waiting build failed checks you did not see.
• Rollback — we still prefer “bad builds never publish” over rewinding code against migrated schema.

Drift detection is tell a human; staged gates remain block bad traffic.

Pattern you can steal

1. Pick a single source of truth for “intended” (GitHub main or release tag).
2. Pick a single source of truth for “serving” (Vercel LIVE, Fly release, etc.).
3. Compare SHAs on a schedule independent of your CI vendor.
4. Alert with both SHAs and a link to your promote/runbook doc.

If you only run tests in CI but deploy elsewhere, this check is often the fastest way to catch “we forgot to wire the last mile.”

Cowpin deploy docs

Full pipeline: docs/deploy-safety.md — Vercel build gate, staged publish, webhooks, and the drift row in the honesty table.

Building a bookmark product? The same discipline applies: ship importers and API paths only when you can prove they are what strangers hit on www — not just what main contains.

Start a free trial · Automate saves with the API.