Skip to content
Tech

Why do I only find out I tagged a metric wrong when the invoice arrives?

82

Opportunity

Tagging a metric with a high-cardinality attribute like a user ID, session ID, or request path multiplies a single time-series into hundreds of millions, and the bill follows a week later rather than a CI failure follows a minute later. By 2026, observability spend has reached 15 to 25 percent of total cloud spend at affected SaaS companies, with documented single-service incidents generating $90,000 in charges over a weekend. Every major backend, including Datadog, Grafana Cloud, and Prometheus, detects cardinality explosions only after ingestion, meaning the only signal is a budget alert or an invoice line item, not a pull request annotation. The OpenTelemetry SDK added optional cardinality caps in early 2026, but these are runtime circuit-breakers that silently drop data rather than upstream gates that catch the problem before it ships. There is no widely adopted CI primitive that ana

Why it matters

Every engineer who has tagged a metric with a user ID has had a bad Tuesday; catching it at the pull request instead of on the invoice permanently changes the economics of instrumentation.

How I score the opportunity

The Opportunity Score is my own read, not a measurement: how much it hurts, how often it bites, and how little exists to solve it today. Higher means I think it is more worth building.

Severity8/10

How much pain it causes when it shows up.

Frequency8/10

How often people actually run into it.

Whitespace8/10

How little good tooling exists for it today.

More problems worth solving