A city telemetry service adds a Prometheus label for every client MAC and also stores raw Meraki response bodies inside distributed traces. After a week the metrics store is huge and traces contain PII. What observability design mistake does this illustrate?

This city telemetry design illustrates unbounded cardinality and payload-heavy instrumentation instead of bounded, useful telemetry. A Prometheus label for every client MAC creates a new time series for each unique MAC; in a municipal wireless estate that set grows without a practical upper bound, ballooning storage, slowing queries, and stressing scrape targets. Embedding raw Meraki response bodies in distributed traces multiplies size again and frequently captures personally identifiable information that should never ride along in routine observability backends. Good observability design deliberately chooses what not to emit: aggregate or hash identifiers, sample traces, and store redacted summaries rather than every raw body.

Failing to sample host CPU more aggressively is not the mistake shown here; CPU scrape frequency is unrelated to MAC-label explosion or PII in traces. Choosing HTTP instead of gRPC for the Meraki client library is an API-transport preference, not an observability cardinality or privacy failure. Using Webex API notifications instead of email for on-call pages is a paging channel choice and does not cause metrics cardinality or trace PII problems.

Exam caveat: high-cardinality labels sometimes look useful for drill-down until the metrics backend collapses—prefer bounded dimensions such as site, SSID class, or AP model and reserve per-client detail for sampled logs or short-lived investigative queries. Operational check: after deploying label changes, inspect series count growth day-over-day and run a trace sample review for unexpected PII fields before promoting the scraper to citywide production.