The Structured Logging Fields That Shorten an Incident

The minimum correlators

The first and most important structured logging fields are those that narrow down the incident to a specific failed request, in a specific service, at a specific time. These fields ensure that every logged event is independently identifiable by when, where, and what it was. Every log entry should include at minimum timestamp, service, event or message, a unique trace_id that correlates all events across all services in a single transaction, and a request_id that distinguishes each individual request within that trace.,,,

• timestamp (time) defines the exact moment the log entry was generated, making it possible to order and correlate events, and to compare behavior against other times of the day or day-to-day.
• service (string) is the name by which a specific deployment or process is known. A log entry from an isolated failing service can be quickly identified and compared to other logs from the same environment and version. A source that explicitly recommends logging the deployed code version in the same field list as the operational context; Uptrace mentions version, but not in the outage-specific framing requested.
• event (string) or message (string) records what happened or the outcome. Since engineers search log entries by message, this string should exactly match the error message that comes out of the language or framework the service is written in, not an internal string or code path.
• trace_id (string) is a globally unique identifier that correlates all components involved in a single transaction. A single user request can invoke multiple services, each of which may generate multiple events. The trace ID lets the obscure error in one service be tied to the full set of logs for the entire transaction.,,,
• request_id (string) is a per-request identifier that distinguishes between a service's handling of multiple incoming requests. If the same service handles high volumes of concurrent requests, each with its own events, this qualifier makes it possible to reconstruct even similar requests separately.

The context that narrows the blast radius

Once an incident has been narrowed to a single failing service handling a single transaction, the next task is to narrow it further down to a particular user or class of users, the particular deployment in which this is happening, and the specific piece of the business flow that failed. To enable this, logs must include an extensive but scoped set of context fields. Sources for these claims

• environment (string) distinguishes between staging and production, or from one production environment to the next. Separating logs by environment ensures that only production logs are flared up, and avoids too many alerts being sent.
• version (string) records the currently deployed version of the service codebase. In a continuous delivery workflow, with multiple environments and options for canary or gradually rolled out versions, this distinction makes it possible to know which version a particular incident fell into and to narrow any search to the relevant set of pods or containers.,
• host (string) or instance_id (string) distinguishes one pod or container from another in a service with multiple running instances. Typically this is built in to the platform and is often unavailable to the service code itself, but it can be obtained via environment variables and should be logged out in order to know exactly which one out of many an event came from.
• Many incidents are request-based, and a complete set of HTTP details makes it possible to reconstruct even long-completed transactions to understand who they were for, which piece of logic was executing, and what went wrong. At minimum, a log should include, where relevant, the http_method (string) e.g. GET or POST, the http_path (string) e.g. /shop, the http_status (int) e.g. 404 or 500, and the duration_ms (int) time in milliseconds taken to execute.,,
• user_id (string) serves in concert with HTTP specifics to understand which particular user was affected by a request-based issue. If a feature like the shopping cart is implicated, order_id (string) fills the same role. Any business logic-specific identifiers, like these, add searchability of a functionally specific kind that lets engineers use the log to reconstruct precise end-user paths.,,

Domain fields for the thing being debugged

While the core set of fields above keeps log entries consistent and independently identifiable, additional fields are required to locate the specific business logic or the specific sub-component that a failing event represents. This part of the log structure depends on the domain and the architecture of the application, but certain principles should guide the selection of additional fields. Domain-specific identifiers map an individual log event to a particular trace or request that a developer is seeking. While it's important to add these identifiers for something like user_id, order_id, or product_id, log entries should be kept distinct from message strings. Similarly, while outcome or reason may map to a specific business pain point, these fields are better logged separately from the core event or message than combined into a single string.,, {todo confirm the instances where outcome and message push the following warning to the copy}

A primary source that explicitly recommends a retry attempt field for production logs.

It is tempting, especially in an event-based or microservices architecture, to add fields for every service that participated in a particular event. However, structured logs should, in practice, assume that those events are recovered through the trace_id. The fewer fields are added, the less verbosity is introduced and the faster it is to spot the relevant ones, though critical information should be included if it narrows down the incident.

How structured logging is represented in standards and platforms

Once logs have been structured as a JSON object, with fields carrying the identifying and contextual information needed to pin an incident down, it can be interpreted by both humans and machines. But it is also portable across platforms and query-able using standard semantics.,,. The OpenTelemetry project documents a set of standard log fields as part of its open source schematic conventions. These fields, spanning general correlations like service or trace_id, span errors like exception.type and exception.message, and domain-specific fields like http.url, provide both an important head start in how to name log fields and a point of understanding for how to query them., Google Cloud similarly invites structured logs into its documentation as a set of JSON-encoded mappings that documents their eventual form as a Google Cloud LogEntry. In this case, fields like httpRequest.traceId, httpRequest.requestUrl, or labels.user_id document not only how log entries are recorded, but how they are naturally indexed and accessed in circumventing platforms.

The anti-patterns to exclude

While structured logging helps preserve the smallest set of fields needed to reconstruct failing requests, there are certain mistakes to avoid with the potential to counteract its benefits. A primary source that explicitly warns against secrets reaching the log in the exact form requested. Compiling log messages from variables, concatenating user-provided information, or performing string interpolation in the log entry forces the engineer to parse and understand variable content in order to discern relevant information. This makes the log harder to write, read and search. It should be avoided in favor of following clear, standardized logging patterns and naming conventions. A primary source that explicitly warns against unbounded cardinality in log fields in the exact form requested.

Similarly, any unbounded-field cardinality can make structured logs unseachable. Unbounded cardinality occurs when a log field can accept an infinite number of unique values, making it effectively a free-text field. This precludes the field from being indexed or searched efficiently. The trace_id and request_id are exceptions to this, as they identify unique and separate transactions, not just unique log entries. A primary source that explicitly warns against assembled message strings from variables in the exact form requested.

Finally, a service should never output secrets in logs, including API keys, encryption keys, database connections, or credentials. The log records should be treated as searchable, accessible records that can circulate to many humans and platforms. Any sensitive or PII information should be scrubbed with masking or encryption before logging.

What the operator needs at 3 a.m.

A properly structured log that includes only the universally useful and domain-specific incident handlers provides a clear workflow for the engineer called at 3 a.m. to understand and reproduce a failed request.

1. Understand the problem by tying the failed event to a failing request. timestamp, service, event, trace_id, and request_id alone are usually enough to understand the general scale of the problem. Sources for these claims
2. Pin points the failing service to one environment. environment field combined with the exigent fields mean the incidents can be understood in terms of a specific environment, not across all pods.
3. Localize it to the exact version of the service. version determines whether the incident is tied to a newly deployed feature or a vulnerable part of the codebase.,
4. Measure the exact duration, time of day, and behavior in this pod. Combined host or instance_id and duration_ms can indicate whether the problem is with a specific pod, which pods are involved, and whether this is an isolated incident or a cascading failure. Sources for these claims
5. Find the precise traffic or transactions affected. http_method, http_path, http_status, and user_id can pinpoint a breaking change or a problematic user, or group of users.,,

By preserving these fields and omitting information that expands too much on its semantic meaning or clouds its filtering and querying, the structured log recording remains as simple as possible while staying useful in the face of an incident.