Scoped Event Bus

Audience: Engine Developer Status: ✅ Ready

The scoped event bus is how systems and content couple loosely across zone boundaries. A zone signals up to a director; a director broadcasts down to zones. Content publishes with signal_region / signal_world and reacts with on_region / on_world. Two transport lanes back it: a transient lane for down-broadcasts and a durable JetStream lane for the state-changing signal-up reports.

Addressing: one subject per scope

The bus layers scope addressing over the comms transport: one subject per scope — telos.scope.world, telos.scope.region.<id>, telos.scope.zone.<id> — with the event name and payload in the message body. So one subscription per scope receives all of that scope’s events and dispatches by name. Subject building validates region/zone ids against a strict charset (alphanumeric, -, _, :, ≤128) to prevent subject/wildcard injection.

Two lanes

Lane	Transport	Used for	Guarantee
Transient (`Signal`/`Subscribe`)	NATS core	down-broadcasts (state deltas + remote effects)	at-most-once; a blip drops it
Durable (`SignalDurable`/`SubscribeDurable`)	JetStream, stream `WORLD_EVENTS`	signal-up (state-changing zone→director reports)	file storage, 30-day retention, publish-side dedup, at-least-once

The durable stream’s subject root is defined as the same constant the publisher uses, so publisher subject and stream binding can never drift.

Signal-up and broadcast-down share one subject per scope — so reserved event names are enforced on the content side too. A durable signal-up publish is an ordinary NATS publish to telos.scope.<kind>.<id>, and every shard core-subscribes that same subject for down-broadcasts. A zone script calling signal_world("scope.state.set", {key = "pvp", value = true}) therefore had its frame delivered to every read-replica as though a director had written it — bypassing the director, its lease and its CAS entirely. The zone-side publish took the event name straight off the Lua stack with no check; the reserved-name guard had only ever covered the director-script surface, which is the narrower and more trusted of the two.

The version fence made the consequence strictly worse rather than better: before it, a forged delta set a wrong value until the next legitimate write; after it, a forged version far ahead of the real one makes every subsequent legitimate director write drop as stale — and since nothing reads through, a single frame permanently freezes that key fleet-wide. A probe reproduced exactly that: a forged pvp=true survived six later director writes.

The reserved list therefore lives in the bus package, where both ends can share it — the world package cannot import the director, and restating the list is how the two would drift. The director keeps its own scheduler names on top and deliberately not in the shared list: boss.died is a legitimate signal-up that the spawn scheduler consumes, so reserving it globally would break a shipped feature. A guard too broad is as bad as one too narrow. A replica also warns when a delta’s version jumps implausibly far ahead — logged, not rejected, because a genuine large gap (a long transient outage) must still apply and silently freezing a key would be worse than the gap.

Residual: this closes the content path, which is the trust boundary the guard exists for. It does not stop a process that already holds a trusted NATS identity from publishing the same frame directly — the broker’s per-identity authorization matrix now denies the untrusted gate publish outright and constrains every role to its own subjects, so this is narrowed to a world/director credential (which legitimately down-broadcasts on this subject). Fully closing it needs the up and down subjects split apart, so even a trusted publisher physically cannot reach the down channel — a wire change with a migration, tracked separately.

Publish (write-up) and react

signal_region / signal_world are plain Lua globals — deliberately kept off the read-only world/region tables so reads and writes stay visually distinct. They never run on a zone goroutine: the call buffers a job to a channel (depth 256, drop-with-log on full), and the shard’s single drain loop always publishes on the durable tier so a state-changing report survives a broker blip and a director restart.

on_region / on_world register into the entity’s shared handler table under a namespaced key (world:<event>), so they share the per-instance script lifecycle (drop on despawn, rebuild on hot reload — see Lua Sandbox Internals). A director broadcast arrives at the shard’s dispatcher: a reserved EventStateSet becomes a replica-update delta; any other event is posted to the target zones — world-scope events fan to every hosted zone, region-scope events only to that region’s members. On the zone goroutine the fire path first primes every scripted entity (so a freshly-spawned mob’s handler is registered before the scan) then fires matching handlers.

Ordering, idempotency, failover

Signal-up consume is leader-bound: the durable consumer is subscribed only while a director is leader and torn down on losing leadership; a standby never consumes. The consumer id is stable per scope, so a restart resumes from the last ack rather than replaying everything. Consumer config: deliver-all (backlog in stream order, then live), explicit ack, max-deliver 5, 30 s ack-wait. Each consumed event (other than the scheduler’s reserved boss.died) is handed to the pack’s content on_signal handler (the world-director script) — so orchestration reactions are now content, not just engine code.
Apply-once over at-least-once: each delivery is applied on the director’s actor goroutine and deduped by a per-source high-water watermark — seq ≤ applied[source] is idempotently suppressed. The key is <source>:<seq> where source is the emitter’s run-unique id (fresh per restart, so keys never collide across runs).
Ack/NAK back-pressure: the bus goroutine posts to the inbox and waits on an ack channel bounded by a 5 s timeout; a wedged actor or a shutdown NAKs → redelivered to the next leader.
Paced redelivery. An explicit NAK redelivers immediately — the ack-wait only governs a hung or lost ack, and a consumer’s BackOff isn’t applied to an explicit NAK. Since a NAK is issued for transient reasons (a target mid-handoff, an emit blip, an overloaded zone), an un-paced NAK burns the whole redelivery budget in milliseconds and the message is parked — permanent loss, because a durable consumer is never deleted and so never replays. Durable NAKs are therefore issued with a delay on a ramp-then-hold schedule (200 ms → 1 s → 3 s → 10 s → 30 s, MaxDeliver 10, covering ~164 s of transient), and the same schedule is installed as the consumer’s BackOff for the ack-wait-expiry path. MaxAckPending = 1 is load-bearing rather than tuning: a delayed NAK would otherwise let a later message be delivered ahead of the one awaiting redelivery — advancing the per-source watermark past it, so the predecessor is suppressed as a duplicate on redelivery and silently lost. It is now a per-consumer option (default 1, double-clamped so no caller can inherit NATS’s 0 == unlimited posture by accident) rather than a hardcoded transport constant; both ordered consumers (COMMS_TELL, WORLD_EVENTS) pass no option and keep the serializing posture. Raising it above 1 is the lever a future reorder-tolerant consumer — one that dedups with a seen-set instead of a high-water — would pull, and it must not be set on the two ordered consumers, which would inherit the reordering hazard with no throughput gain. That serialization is also why a stalled message is an orchestration outage rather than a delayed message: it blocks every later message on its consumer for the whole retry window, and WORLD_EVENTS has one consumer per scope for the entire fleet. telos.commbus.durable_stalled_total exists to alert on that while it is still recoverable — see Sysadmin Reference.
The ack decision is a tri-state, not a bool: only RetryTransient spends the redelivery budget, while an undeliverable poison message is dropped immediately so it can never consume a transient’s budget.

Documented constraint (not a bug): the watermark is sound only for a gap-free, in-order consumer — which the director actor spine is, and whose payloads are order-insensitive (audit = logging, pull = idempotent-by-SHA). An order-sensitive director reaction would need a seen-set instead.

Cascade budgets — an important nuance

The scoped bus has no cross-scope cascade/loop budget of its own. The depth/width bounds (maxEventDepth=8, maxEventHandlers=256, maxEventCascadeDepth=32) live in the in-zone event bus and govern the local reaction a broadcast triggers — each on_world/on_region handler fires with a fresh root context (depth 0). Cross-scope width is bounded only by the 256-deep signal queue (drop-on-full) and the low-rate design assumption.

Consequence to design around: a handler that itself calls signal_world/signal_region starts a new cascade with a new budget. Nothing structurally bounds a content-authored cross-zone signal loop — it rests on content discipline, not an engine guard.

Consistency properties along a cross-zone path

Writes up, reads down, single writer. A zone never mutates another scope; it signals up. The director is the sole writer of scope state, applying on its one goroutine; zones read local replicas.
Durability where it matters. Signal-up rides the durable lane and survives broker blips + director restart; state down-broadcasts go through Set (persisted + broadcast), while pure remote effects (Broadcast) are transient/best-effort.
Exactly-once effect across restart. A capstone test asserts precisely three boss kills counted (not lost, not doubled) across a mid-sequence director restart — via the stable durable consumer resuming from the last ack, the per-source watermark suppressing redelivery, and CAS-persisted state.
Failover handoff. A message consumed-but-not-applied when leadership flips is NAK’d and redelivered to the promoted leader; a coordinated pull consumed by a demoting leader is requeued, not ack-dropped; the persisted-state CAS rejects a stale leader’s late write.
Broadcast ordering caveat. Transient down-broadcasts have no cross-message ordering beyond per-subscription serial delivery; the durable/ordered/per-source guarantee applies to the signal-up lane and the director’s single-writer application, not the down-leg.