Your observability bill is now bigger than your compute bill. What do you cut first, and what must you never lose?
Attribute observability cost by signal, service and pricing dimension. Then cut by value per byte: debug and duplicate logs at source, logs that should be counters, hot retention nobody queries. Protect audit and security logs, error events and SLO signals from sampling, and never cut by a flat percentage.
Answer
Short answer
Attribute the observability cost before cutting anything. Split it by signal, by emitting service and by what the vendor bills on, because the spend is usually concentrated in a few sources. Cut log volume by value per byte: debug and duplicated logs at the source, log lines that are really counters, and hot retention nobody queries. Never lose audit and security logs, error events, or the metrics behind your SLOs, and never apply one percentage cut across the board.
Was this answer helpful?
Likely follow-ups
- Legal says audit logs must be kept for years, and they are a large share of the bill. How do you store them without paying hot-index prices?
- A team argues their debug logs are the only way they diagnose a flaky integration. What do you offer them in place of always-on debug logging?
- How would you stop the bill drifting back up six months after the clean-up?
- Your vendor bills by ingested bytes, and a new pricing tier bills by indexed bytes. How does that change which logs you keep?
Related questions
- A maintenance script updates your permissions table and every user in one tenant can suddenly edit everything. What should have stopped that, and how would you have found out?hardAlso on audit-logging6 min
- Latency on a checkout endpoint has tripled and nobody knows why. What should your metrics, logs and traces already have told you before anyone opens an editor?hardAlso on sampling6 min
- Your cloud bill is too high. Where does the money go, and how do you bring it down without degrading the service?hardAlso on observability-cost6 min
- A customer's API token turns up in a public repository. What do you do in the next hour, and what in your token design decides how bad this is?hardAlso on audit-logging7 min