Observability from day one (not day ninety)

Every team agrees observability matters. Almost every team wires it in month three, during the first incident, with the dashboard assembled while production burns. The argument of this post is small and stubborn: the cheapest moment to add telemetry is in the same PR that ships the route — and after that moment, the price only compounds.

Day ninety is the expensive day

Retrofitting observability means re-deriving intent: what *should* this endpoint’s latency be? Which downstream call is allowed to fail? The engineer who knew has shipped four features since, or left. On day one those answers are free — they are sitting in the PR description. Telemetry written alongside the code is documentation that happens to be queryable.

Logs tell you what the code did. Traces tell you what you meant.

The starting kit

OpenTelemetry’s auto-instrumentation gets you eighty percent — HTTP, fetch, the database driver — in about forty lines. The remaining twenty percent is one custom span per business operation, named after the domain, not the function.

import { NodeSDK } from '@opentelemetry/sdk-node'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'

export const sdk = new NodeSDK({
  serviceName: process.env.SERVICE_NAME,
  traceExporter: new OTLPTraceExporter(),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': { enabled: false },
    }),
  ],
})

sdk.start()

• One span per business operation: order.confirm, not handlePost.
• Attributes carry the domain: order id, tenant, plan — the things you will actually filter by at 2 a.m.
• The PR template asks one question: “how will we know this works in prod?” — answered with a metric name or a trace link.

What it costs and what it caught

The kit adds roughly fifteen minutes to a feature PR. In exchange, our last three incidents were diagnosed from the trace view before the responder finished reading the alert — including one N+1 a code review had blessed twice. Day-one telemetry did not make us faster at fixing bugs. It made the bugs arrive already explained.