Structured Logging That Pays for Itself: Schema, Context, and Cost Discipline

Logging is the telemetry everyone has and almost nobody designs. Metrics get dashboards reviews and traces get instrumentation sprints, while logs just… accumulate — printf archaeology wrapped in an ingestion bill that grows 40% annually until finance asks why the observability vendor costs more than the compute. The waste and the weakness have the same root: logs treated as text a developer emits rather than what production logging actually is — a stream of structured events, designed for machines to filter and humans to trust, with an explicit cost model. Here's the design.

#Events, not sentences

The foundational shift: a log line is a JSON event with a schema, not a sentence with variables interpolated into it. "Payment failed for order 12345 after 3 retries" is a sentence — grep-able if you remember the phrasing, aggregatable never. The event version carries event: payment_failed, order_id, retry_count, failure_reason, duration_ms — and now "failure rate by reason over time" is a query instead of a regex safari.

The schema disciplines that make event streams queryable at estate scale: a small set of standard fields on every event (timestamp, severity, service, version, environment, plus the correlation set below) — adopt an existing convention (OpenTelemetry's semantic conventions or the ECS-class schemas) rather than inventing one, because the payoff of shared field names compounds across every service and every dashboard; stable event names treated like API surface (renaming payment_failed breaks every alert and saved query downstream — version and deprecate, don't mutate); consistent types per field (an order_id that's a string here and a number there breaks indexing in ways discovered at query time); and one event per logical occurrence, not a narrative of five lines that must be mentally reassembled — the request-summary event (one rich line per request with outcome, duration, and key attributes — the classic "canonical log line" pattern) is worth more than fifty step-by-step murmurs, and costs less.

#Context is what makes logs debuggable

A log line's value is mostly its joinability. The non-negotiable correlation set: trace and request IDs (propagated via OpenTelemetry context — the integration that transforms logs from standalone artifacts into the detail layer of distributed traces: find the slow trace, jump to its logs, done), principal context (tenant/user identifiers — pseudonymous where privacy demands), and causal context where it exists (job ID, message ID, saga/workflow ID — the threads that let you follow one business transaction across services and hours). Inject these once at the boundary (middleware, contextvars/MDC — the async-safe variants covered in the respective stack discussions) so every log line in the request's lifetime carries them without per-callsite effort. A codebase where engineers manually remember to include the request ID is a codebase where a third of the lines don't have it — and that third is always where the incident is.

The counterpart discipline is what must never be in logs: secrets, tokens, passwords (redaction at the logging-library layer, plus scanners in CI and on the stream — because one debug line logging full request headers is a credential leak with retention), and unminimized personal data (logs are the compliance surface nobody maps — data-protection rules apply to them fully, deletion requests included, and "it's just logs" is not a defense anyone enjoys presenting).

#Levels, sampling, and the cost model

Level discipline, stripped of folklore: ERROR means something requiring action failed (and should be alarm-worthy in aggregate — an ERROR that's routine is mislabeled); WARN means degraded-but-handled (retries that succeeded, fallbacks taken — the early-warning band); INFO carries the business-and-request events that tell the system's story (the canonical lines); DEBUG exists for humans actively investigating and is off in production by default — with the crucial modern amendment that it should be dynamically enable-able (per service, per tenant, per request-percentage, at runtime, without deploys) because "redeploy with debug logging" is a 45-minute incident tax that runtime toggles reduce to seconds.

Then the economics, because logging is priced per byte and emitted per request: not every event deserves 100% capture. The sane defaults — keep all ERROR/WARN, keep 100% of request summaries for errored/slow requests, sample the happy-path INFO stream (1–10% preserves statistical visibility at a tenth of the bill), always-keep anything tied to a sampled trace (consistency with your tracing decisions, or the join breaks), and let high-volume DEBUG-adjacent events earn their retention explicitly. Add tiered retention — hot searchable days, warm queryable weeks, cold object-storage archive for the compliance horizon — and the bill drops from "growth-limiting" to "proportionate," which is the actual goal: the organizations that log everything forever at full index eventually respond by logging nothing useful, and the pendulum's both ends lose.

The operating loop that keeps it healthy: a monthly look at ingestion by service and event (the top ten events are reliably 80% of volume, and half of them serve no query anyone has run — delete or sample them), alerts on log-volume anomalies (a 10× spike is either an incident or an accidental debug-loop, both worth knowing), and the standing question in code review — who queries this line, and what decision does it inform? — which is the logging version of the test-value question, and prunes with the same efficiency.

#The summary schema

Structured events against a shared convention; stable names treated as API; one rich canonical line per request; correlation IDs injected at boundaries and present everywhere; secrets and PII engineered out at the library layer; levels that mean things, with DEBUG dynamically switchable; sampling and tiered retention as designed policy rather than vendor-bill archaeology; and a review loop that deletes what nobody queries. None of it is glamorous, all of it compounds — because logs are where every incident investigation actually ends up at 3 AM, and the difference between fifteen minutes and three hours of that investigation was decided months earlier, by whether someone designed the events or just let the printfs accumulate.