Skip to content
Tech

Why does my schema migration run before checking if deployed services survive it?

83

Opportunity

When you push a schema change to a production database, the migration executes before any tool verifies that the service versions currently running can read and write the new shape. A dropped column or renamed field reaches the database before old pods finish draining, and rollback restores the schema but not data written by the new version in the meantime. ORM tools generate SQL from diffs but model nothing about deployment topology, concurrent service versions, or live read-write patterns. Teams compensate with expand-contract rituals documented in wikis and tribal knowledge, not enforced by tooling. A single misordered step triggered multi-service cascading failures and a documented $4.5 million incident in the last year alone.

Why it matters

A pre-flight compatibility check that models your live service topology would turn the riskiest step in a deployment into something you can reason about before hitting apply.

How I score the opportunity

The Opportunity Score is my own read, not a measurement: how much it hurts, how often it bites, and how little exists to solve it today. Higher means I think it is more worth building.

Severity9/10

How much pain it causes when it shows up.

Frequency8/10

How often people actually run into it.

Whitespace7/10

How little good tooling exists for it today.

More problems worth solving