Observability

Audience: Sysadmin Status: ✅ Ready

How telemetry leaves a TelosMUD fleet and where it lands: metrics, logs, and traces all flow over OTLP to a single OpenTelemetry Collector, which fans them out to a Grafana LGTM stack — Loki (logs), Grafana, Tempo (traces), and Prometheus (metrics). Everything here is opt-in and off by default; an untouched deployment emits nothing and pays nothing. The three signals cross-link: a histogram spike carries an exemplar to the trace that produced it, and a traced log line carries the trace_id back to Loki.

Related: Running at Scale (the metric catalogue and what to alert on), Deployment (ports and the dev overlay), Sysadmin Reference.

The pipeline

flowchart LR
    S["services<br/>(world · gate · account)"] -->|OTLP/gRPC :4317| C["OTel Collector<br/>(single ingest)"]
    L["container stdout"] -.->|filelog, k8s only| C
    C -->|:8889 scrape| P["Prometheus<br/>(metrics, 7d)"]
    C -->|OTLP| K["Loki<br/>(logs, 7d)"]
    C -->|OTLP| T["Tempo<br/>(traces, dev overlay)"]
    P --> G["Grafana"]
    K --> G
    T --> G

The collector is the single ingest and scrape surface — the Go services expose no /metrics endpoint of their own, and nothing scrapes them directly. That indirection is what lets the same service binary run identically in dev compose and in Kubernetes while the backend differs.

Metrics

Every service can export OTLP/gRPC metrics, but export turns on only when OTEL_EXPORTER_OTLP_ENDPOINT (or the metrics-specific var) is set; otherwise each record is a negligible no-op. The collector re-exposes everything as a Prometheus scrape on :8889. The export interval is the SDK default of 60 s, which is right for a long-running box.

The endpoint needs a URL scheme. The OTel Go SDK parses OTEL_EXPORTER_OTLP_ENDPOINT as a URL, so a bare otel-collector:4317 is read as scheme otel-collector with an empty address — the gRPC dial fails and metrics silently never leave the process. Write http://otel-collector:4317. This bit both the compose stack and the first Kubernetes wiring; if a dashboard is empty, check the scheme first.

The instrument catalogue — telos.zone.tick_lag_ms and the rest — lives on Running at Scale → Metrics, alongside what each one means and what to alert on.

Logs

The primary sink is always stdout — structured JSON, one logger tagged with the service name (Running at Scale → Logging). A deployment gets logs into Loki one of two ways, and they are deliberately mutually exclusive so nothing double-ships:

• The OTLP bridge (compose / dev). Set TELOS_OTEL_LOGS=1 and an OTLP endpoint, and slog is fanned to both stdout (unchanged) and OTLP, through the same connection the metrics use, into the collector’s logs pipeline → Loki. It is independent of DEBUG and off by default. If TELOS_OTEL_LOGS is set but no endpoint is configured, the service warns rather than silently skipping — that silence is exactly what once hid an un-wired service.
• filelog (Kubernetes). The collector runs a filelog receiver over /var/log/pods and ships container stdout to Loki directly, so the OTLP bridge stays off on k8s. (Docker Desktop’s in-VM logs aren’t filelog-readable, which is why compose uses the bridge instead.)

Defense-in-depth against sensitive logs. The engine already refuses to log raw player input (#454) and caps builder-controlled values (Lua and content). On top of that, the Kubernetes collector runs a sensitive-drop processor that discards any log record matching a raw-input message or key before it reaches Loki — so a future upstream regression still can’t persist a tell, chat line, or link code.

Traces

Distributed tracing is wired, on the same OTLP path and the same env gate as metrics — a TracerProvider with a W3C composite propagator (TraceContext + Baggage) set globally and unconditionally, so a hop that doesn’t itself export still understands and forwards an incoming traceparent. With no endpoint configured the global no-op provider stays in place, so a started span is non-recording — zero cost when unconfigured, exactly like metrics.

Sampling is head-based — ParentBased(TraceIDRatioBased), default ratio 1.0, overridable via the standard OTEL_TRACES_SAMPLER_ARG — wrapped by a carve-out that records a handful of spans at 100 % regardless of the ratio. The carve-out is for the rarest, highest-value traces, which must never be sampled away.

What is traced (and what deliberately isn’t)

• Cross-shard handoff + AdoptZone lease handover — the flagship, 100 %-sampled trace. This is the neither-owns/both-own window where three separate rounds each found a real bug, so it is the one you most want a recording of. These are root spans, not children: the triggering north command is not traced (see below), and parenting a multi-second cross-shard operation to a player’s move would misattribute its lifetime anyway. Per-hop child spans (prepare / directory_claim / abort; rpc / lease_flip) localize which hop was slow, and the failure path distinguishes “destination unreachable” from “destination rejected” from “ownership conflict.”
• Session attach — the login-latency path a player directly experiences, crossing Redis, Postgres, and the session lock. A bounded span covers the handshake only (first Recv → session lock), with child spans per hop (epoch-resume, snapshot load, ownership claim, session-lock). The Play stream is a context anchor, not a span: it is one bidi call for the whole session, so the span deliberately ends at attach and the multi-hour reader/writer loop is out of scope. Ordinary head ratio, not 100 %.
• The comms bus and the zone mailbox — trace context crosses a NATS publish through the message envelope (not NATS headers, so one mechanism works across the in-proc MemBus too), joined producer→consumer by a span link, not parent-child. That is not stylistic: JetStream is at-least-once, so a delivery may be the Nth redelivery arriving long after the producer span ended — a link (“caused by”) is truthful where parent-child (“contained in”) would lie, and a delivery_attempt attribute makes a redelivery visibly one. The zone-mailbox span starts at dequeue (so a message dropped under load before it is dequeued never orphans a started-never-ended span) and records queue_wait_ms — the inbox-saturation signal at the one-core hot-zone ceiling.
• Deliberately not traced (a recorded decision, not an omission): there is no per-command span and no gRPC stream interceptor on the Play stream. So the reader loop and per-command log sites carry no span — tracing covers the cross-cutting latency paths above, not every verb.

The metric → trace → log pivot

The three signals are wired to click through to each other:

• Metric → trace (exemplars). The flagship is busLag (telos.bus.deliver_lag_ms): it is recorded on the producer’s span context (extracted from the bus trace envelope), so otel-go’s trace-based exemplar filter attaches a trace exemplar for a sampled producer trace — a spike in the histogram becomes a click through to a trace of one request that produced it. Note that the headline scale metric, tick_lag_ms, deliberately carries no exemplar: it lives on a zone-lifetime context, never a request span, and spanning a heartbeat is an explicit anti-goal.
• Trace → log. A traceHandler on the stdout logger stamps trace_id/span_id onto any line logged with a context carrying a live span, so clicking a slow span in Tempo lands on that request’s Loki lines. It fires only on the *Context slog forms (the ctx-less forms have no span) and is zero-cost when tracing is off. The span-carrying log sites on the traced paths were converted to the *Context forms; the rest stay as-is, since a path with no span gains nothing.

Cardinality is a security boundary, not just hygiene

Every zone-labelled metric and every span attribute is routed through the same discipline: the template, never the player-mintable zone instance id. Returning the instance id would let any player minting dungeon instances spray unbounded label values — a player-triggerable cardinality bomb on a single-node backend, i.e. a self-service monitoring outage. The same rule bars per-player subjects (only a bounded subject kind — tell/chan/roster/… — is recorded, never telos.comms.tell.<playerId>), account UUIDs and character names from span attributes (those are structured log fields), and bounds a gate metric to the host rather than host:port (the ephemeral source port would mint a dead series per reconnect). Producer links extract only the span-context id from the envelope — attacker-controllable baggage never crosses.

Where you can see traces today

Dev compose renders the full pivot. make up brings up otel-lgtm, which bundles Tempo; the overlay collector fans traces to it, its Prometheus keeps OTLP exemplars (--enable-feature=exemplar-storage), and Grafana provisions the Prometheus→Tempo exemplar link. So a local spike-to-trace-to-log click-through works end to end.

The reference k8s deployment still defers Tempo. The staging manifests emit spans but do not yet run a Tempo backend — an empty datasource reads to an operator as “tracing is broken,” so it waits until the backend is deployed. Metrics and logs are live there; traces are the remaining staging step.

Running it locally: the compose overlay

The Grafana stack lives in a separate compose overlay file so the base stack stays byte-for-byte identical for CI (whose smoke/e2e jobs reference the base deploy/docker-compose.yml directly and must never pay for four observability containers they don’t read). The human default make up merges the overlay in; make up-base is the lean path without it:

make up          # base stack + the LGTM overlay — the default (Grafana on http://localhost:3000)
make up-base     # base stack only, no observability (the lean path CI/smoke use)
make down        # stop everything; dashboards + TSDB persist under deploy/data/lgtm

The overlay adds one grafana/otel-lgtm all-in-one container (arm64-native; bundles Prometheus), publishes only Grafana, on 127.0.0.1:3000 (loopback, per the port-hardening pass), points the collector at an overlay config that also fans to the bundled backends, and sets TELOS_OTEL_LOGS=1 on the long-running services. Reach for make up-base when you want the leaner stack with no exporter-retry noise.

The reference deployment (Kubernetes)

The public telosMUD-infra repo is a working AWS EKS deployment of the whole fleet — Terraform + Kustomize, with a one-click up/down lifecycle — and is the concrete reference for the how-tos here. (It ran on single-node Oracle k3s until mid-2026; if you are reading an older write-up, see the migration note.) Its observability layer deploys the LGTM backends as discrete manifests (not the dev all-in-one) with the following load-bearing decisions:

• Retention is configured before first ingest — and note why has changed with the platform. Loki deletes nothing by default, so without an explicit retention policy logs live forever. On the old k3s topology that was a node-level hazard: local-path PVCs were directories on the node root FS and didn’t enforce size, so an unbounded backend could fill / and take Postgres, NATS, and the game down with it. On EKS the PVCs are gp3 EBS volumes and are genuinely size-bounded, so the blast radius is now contained to the backend itself — a full volume stops ingestion and you lose observability, not the game. That is a much better failure mode but still one to avoid, so the policy is unchanged: Loki keeps 7 days, Prometheus keeps 7 days and 2 GB (the size cap is what saves you during a cardinality blow-up, when time-based retention alone is too slow), on a ~7 GiB PVC budget. A NodeDiskFillingUp alert still fires at 80 % root-FS from collector hostmetrics — now guarding against runaway emptyDir usage and image churn rather than the backends.
• The collector is one DaemonSet: OTLP :4317 ingest (re-exposed :8889 for Prometheus), filelog → Loki over OTLP, and hostmetrics feeding that disk alert. It scales cleanly to N nodes.
• Grafana is public but guarded. It gets its own hostname (not a path on the player-facing host, which would risk shadowing the OAuth callback) behind an independent ingress-nginx basic-auth gate in front of Grafana’s own login — the basis on which public exposure is safe, given Grafana’s pre-auth CVE history. The htpasswd lives under the auth key of a grafana-basic-auth Secret, and a missing Secret makes nginx return 503 — fail-closed, not fail-open. Grafana itself is hardened: admin password from a Secret, anonymous access / sign-up / unsigned plugins all off, pinned to a patched release.
• A default-deny NetworkPolicy closes lateral movement. The cluster shipped allow-all; four internet-adjacent observability workloads plus a tokenless account gRPC in staging made a compromised obs pod a real path to Postgres/account/Loki. Ingress is default-deny with explicit allows for every real path (datastores from the game services only, :8889 from Prometheus, Loki from collector + Grafana, Grafana from ingress-nginx, and the ACME HTTP-01 solver — without which cert renewal silently breaks). Egress stays open so DNS/health work. On EKS, enforcement is the VPC CNI’s NetworkPolicy support, and the ingress source is the ingress-nginx namespace (it was kube-system under k3s/Traefik — a detail that silently breaks the policy if you port it across unchanged).

Certificates split by reachability. The web and Grafana hostnames use cert-manager HTTP-01 through ingress-nginx, but the gate’s telnet-TLS certificate uses DNS-01: the gate is exposed on its own AWS NLB for raw TCP, which an HTTP-01 challenge can’t reach. cert-manager talks to Route53 via IRSA — no DNS API tokens anywhere.

The lifecycle is one click, and DNS is automated. up runs Terraform then deploys; down drains the NLBs and EBS volumes before terraform destroy. external-dns (Route53 via IRSA) writes the web and gate CNAMEs from the live NLB hostnames, and the deploy workflow creates the app secrets and applies the ClusterIssuers itself — so there are no manual kubectl/DNS steps. Each environment gets its own delegated Route53 subzone, with external-dns and cert-manager IRSA scoped to only that subzone, so a compromised staging pod cannot touch production names. The only create-once manual surface left is the OIDC role, the Terraform state bucket, the GitHub OAuth app, the root Route53 zone, and the CI secrets.

CI/CD guards worth mirroring: a PR check renders every Kustomize overlay with the pinned kustomize and schema-validates it with kubeconform -strict (the deploy workflow otherwise has no checks between merge and a live cluster); and production apply — Kubernetes, Terraform, and the up/down lifecycle alike — is gated behind a manual-approval Environment, so a push to main auto-applies staging only and production is one-click-approved via workflow_dispatch. That gate matters more now that down exists: it is the difference between an approval prompt and a one-click production teardown.