Orchestration and Directors

Audience: Engine Developer Status: ✅ Ready

Some consequences span more than one zone: a region-wide invasion counter, a world-boss respawn timer, a coordinated content pull across the fleet. Those live in the director tier — telos-director — hosted out-of-band from the simulation shards so orchestration never competes with zone ticks for CPU. A director is itself an actor, one level up: the single owning writer of one scope’s state, elected per scope so exactly one instance is live while others stand warm.

Related: Scoped Event Bus (how consequences travel), Distributed Systems Model, Content-Pack Operations (coordinated pull), Running at Scale.

The director as an actor

Director.Run is a select over four sources — ctx.Done(), the inbox, a heartbeat ticker, and an election sub-ticker — exactly the zone actor model one level up. Every scope-state read/write happens on this one goroutine, so its state maps need no lock. Ingress is only via post → inbox.

Scope state is CAS-backed: Get/Set are synchronous round-trips onto the actor goroutine, reading an in-memory cache that lazily loads from Postgres on a miss. Set writes through an optimistic-concurrency CAS on the key’s version; on a CAS loss it reloads the winning value and returns an error rather than clobbering.

A signal is NAK’d if the write its handler asked for never landed. handleSignal used to run the handler, advance the per-source applied-seq high-water, and ack — regardless of whether the write succeeded. Because every director binds the same durable consumer name, that ack consumes the event fleet-wide: the live leader never re-sees it and the consequence is simply gone. The signal is now NAK’d for redelivery on a lost write, and the high-water is deliberately left alone so the redelivery isn’t then suppressed as a duplicate.

Two widenings matter. The trigger is any unsuccessful store return, not just a CAS loss — a plain Postgres blip reaches the identical end state by a far more common route — and it is recorded through a deferred check at the single point every scope-state write funnels through, so a future early-return inherits the guarantee. That single point is on the Director, not threaded out of the handler signature, because a handler return is defeated three ways: a script’s pcall swallows the Lua error, the schedule composition discards returns, and the schedule-state save calls the setter directly without ever holding a script API.

A prerequisite defect had to be fixed first, and it inverts the intuition about CAS losses: the setter CASed on its cached version, which is 0 for any key this process never read, and the store rejects 0 against an existing row. So a restarted director’s first blind write to each pre-existing key lost the CAS with no concurrent writer anywhere — precisely the derived-write pattern the scripting idempotency contract recommends. A CAS loss therefore meant “cold cache” far more often than “concurrent writer”, and NAKing on it would have requeued a signal on every ordinary restart. The setter now seeds the version on a cache miss.

The setter also refuses a write from a non-leader, since leadership can lapse during a handler, after the dispatch gate has already passed. This does not fully close that window: a version CAS detects a lost update but cannot say who was entitled to win. Closing it needs an ownership fence in the CAS predicate — a lease epoch on the row, exactly as owner_epoch does for characters.

Leader election

Exactly one live director owns a given scope (a region or the world); others are warm standbys. Leadership is a Redis time-fenced lease CAS — the same primitive as zone leases — so the CAS is the final split-brain arbiter. The lease id is director:world or director:region:<ref>; TTL is 9 s, renewed every TTL/3, so a crash lets a standby claim within ~TTL. campaign fails safe: if it can’t confirm the lease (Redis unreachable), the director steps down rather than risk double-leading. A standby still serves Get/Set (loading from the store on demand) but gates all active orchestration on IsLeader().

What the director drives

Per tick (leader-gated):

• Scheduled spawns — the boss-respawn scheduler (Loot, Spawns & Crafting).
• Mail dead-letter reap — leader-gated and cooldown-checked, the DELETE offloaded to a single-flighted worker. Notably it is not lease-fenced: it rests on the reap being an idempotent DELETE ... WHERE sent_at < cutoff, so a brief failover overlap deletes the same already-eligible rows, never live data.
• Channel-roster aggregation — a per-channel listener roster (for GMCP Comm.Channel.Players) spans shards, so no single shard can author it; the leader director periodically reads the whole cross-shard presence roster (each entry carries the player’s hear-set), inverts it to each channel’s listener set, and publishes the changed channels’ rosters to a per-channel transient subject the gate forwards. The blocking read + publishes are single-flighted off the actor goroutine. Because a roster is convergent state, not an event, a diff dropped on the fire-and-forget subject would never be retried — so every tenth poll does a full, un-diffed republish (bounding staleness and seeding a subscriber who joined a quiet channel), and the first poll after a leader failover is always a full resync. Concealed players are omitted. See Scoped Event Bus.

Driven by signals or other paths:

• Down-broadcasts — state deltas and remote effects, transient over the scoped bus.
• Durable signal-up consume — bound to leadership (a standby never consumes).
• Placement coordination — a leader-only loop that reads per-zone occupancy weights, plans a load-aware spread, and issues rebalance-drain directives the owning shards execute via the handoff (with a per-zone cooldown, locality-aware colocation, and reservation-based drain-target selection) (Running at Scale).
• Coordinated content pull — leader-gated, single-flighted, heavy git+Postgres work offloaded to a worker whose context derives from Run’s, with NAK-on-not-leader requeuing to the promoted leader (Content-Pack Operations).

Shutdown calls resign() (releasing the lease for immediate standby takeover) then waits on in-flight workers.

The scope hierarchy

The levels are entity → room → zone → region → world. The two upper levels are the routable scopes a director owns; a zone is a director’s remote-effect target, not an owned scope.

• Zone → region mapping is pure content: a region_defs entry {ref, name, zones[]}. A zone belongs to at most one region, and ref doubles as the scoped-event subject token telos.scope.region.<ref>. (The demo ships one region, heartlands, spanning midgaard + darkwood.)
• Persistence: world_state(key) and region_state(region_id, key), each a JSONB value plus a version for the optimistic CAS. Only the owning director writes these, but the CAS still defends against a just-demoted leader.
• Reads-down (CQRS): each zone keeps a read-only replica of world/region state, updated only on the zone goroutine by a posted delta message. Lua reads it synchronously and lock-free via world.flag/world.get/region:get.
• A down-broadcast carries the store’s version, and the replica fences on it. The payload used to be {key, value} and nothing more, so a replica had no way to tell a current push from a duplicated or reordered one; it now carries the version the scope-state CAS assigned, and the replica keeps the highest version applied per key and drops anything at or below it. The reason this matters is not the obvious one: rapid set churn does not reproduce it (single-publisher transient delivery is strictly ordered end-to-end with no redelivery). The reachable case is two directors publishing inside the lease-observation-lag window — and the damage is lasting, because there is no re-read anywhere. world.flag and region:get read the local replica only, so a replica left holding a superseded value stays wrong until the next write of that key or a full reseed. A sticky “war active” is exactly that shape.
  • The version must be the store’s, never a per-director counter. The store’s counter lives in the row and is incremented by the CAS, so it is monotonic per key across a failover: a promoted leader loads the row and continues the sequence. A per-director counter would reset to 0 on every failover, and the fence would then drop every good push until it climbed back past what each replica had seen — permanently, for a rarely-written key. A fence that drops good pushes is worse than no fence.
  • Three easily-inverted rules: version 0 means unversioned (apply it, record nothing) — reading 0 as “oldest” would make an upgraded zone reject every push from an un-upgraded director mid-rolling-deploy; an unknown key applies unconditionally, since rejecting it would freeze a seeded replica forever; and a delete records its version too, or a reordered older set resurrects it.
  • Seeds are versioned as well, and that was not optional: applying a seed must reset the fence (a version from the replica’s previous life describes a value the snapshot just overwrote), which would otherwise leave the fence empty at exactly the boot and drain-adoption moments it exists to cover. A snapshot that cannot supply a version degrades to unfenced — the safe direction, since unfenced means “apply the next delta”, never “freeze this key”.
• A zone’s replica is seeded before the zone is exposed. Boot zones are seeded from a snapshot before the shard subscribes to the transient scope bus. A zone hosted after boot — a drain adoption — used to bypass that entirely and start with an empty replica, learning each world/region key only when it was next broadcast: a sticky world flag like “war active” read false on the adopted zone, possibly forever, at exactly the failover this subsystem exists to survive. The seed must also run before the zone is published, not after: once it is visible, a world delta can already be sitting in its inbox, and applying a seed is a full-map replace that would clobber the newer state. So the zone is seeded while still invisible to both the world fan-out and the region routing, then adopted.

The world-director script

Orchestration logic is content, not engine code. A pack can carry a world_script — sandboxed Lua the world director runs — that defines on_signal(event, payload), invoked once per signal-up event the director consumes. The script reacts to its zones’ events by writing world scope state and broadcasting the consequence down, which each zone then applies locally through its on_world handlers (the golden rule: the director never reaches into a zone). It runs in the shared sandbox core (Lua Sandbox Internals) — the same allowlist, caps, chokepoint, and circuit breaker as a zone VM — with a director host table instead of the world’s mud/self surface:

director.*	Effect
get(key)	read a world-scope-state value
set(key, value)	write world-scope state (through the director’s single-writer CAS)
broadcast(event, payload)	fan a transient down-event to the member zones
log(msg)	structured log line

Four properties keep a content script from corrupting engine state — the same “content is untrusted at scale” posture as zone Lua, applied to the control plane:

• on_signal sees every non-boss.died signal, including engine-internal ones (content.reload.audit, content.pull.result) — only the scheduler’s boss.died is intercepted upstream. So a script must guard on its own event names and never write a catch-all branch. (WithWorldScript composes before WithSchedules, so the scheduler’s reserved handling stays outermost and the Lua handler only ever sees the rest.)
• Handlers must be idempotent. Durable delivery is at-least-once, so a redelivery re-invokes on_signal; write derived values (director.set("last_boss", p.boss)), never a blind increment, or an event double-counts.
• director.set refuses engine-reserved keys (the schedule:* prefix), so a script can’t clobber the scheduler’s own state.
• Instanced zones are excluded from reserved schedule events. A single spawn.boss fanned out to every hosted zone would otherwise spawn the boss — with its full loot table — in the template and in every live private copy, and each kill would reschedule the one shared world timer, last-writer-wins. Instances are withheld from that fan-out, and mud.zone() exists so content filters on the live zone id (the demo pack’s ev.zone ~= "darkwood" idiom was a worked example of getting this wrong). Symmetrically, signal_region / signal_world from an instance are refused loudly — the signal envelope drops source, so a director could not tell one party’s private progress from the shared world’s — while region reads still resolve through the template, so content gating on region state isn’t silently inert in every copy.
• director.broadcast refuses engine-reserved event names — scope.state.set (which a zone read-replica would apply as a state delta, bypassing the director’s single-writer CAS) and the content.* namespace — so a script can’t forge a state-set or a fake pull/reload status.

Authoring lives on Pack Lua Hooks → world-director script and the world_script manifest field on Pack MUD Settings. Only the world director runs a script today; region-director scripts (and VM teardown) are a tracked follow-up, as are NAK-on-CAS-loss and persisting the applied-signal high-water.

Worked cross-zone path (the shipped demo boss loop)

sequenceDiagram
    participant D as world director (leader)
    participant Z1 as zone (darkwood)
    participant Z2 as other hosted zones
    D->>Z1: spawn.boss DOWN (world scope, transient)
    D->>Z2: spawn.boss DOWN (fan-out to all hosted zones)
    Z1->>Z1: herald on_world("spawn.boss") filters zone, mud.spawn(boss)
    Note over Z1: players fight and kill the boss
    Z1->>D: signal_world("boss.died", {ref}) UP (durable)
    D->>D: dedup by watermark → onBossDied → AfterDeath (next = now + interval) via CAS

Scope of the shipped fleet: the world director runs the pack’s world_script, and telos-director now also builds one director per region_defs entry, each running that region’s own optional script — see Per-region director scripts. The demo pack’s world script reacts to a boss_slain signal by recording the fallen boss in world state and broadcasting a realm-wide announcement down — the worked cross-zone loop that used to live only in a test is now shippable content.

Per-region director scripts

A region_defs entry may carry a script: the region-scoped sibling of world_script, run in a sandboxed VM on that region’s own director actor.

This closed a scope that was write-dead in a way the zone side did not reflect. Content’s signal_region already published durable signal-up events, and zones already subscribed their region and applied its state deltas — but nothing consumed the region subject and nothing ever wrote region state, so those events sat in WORLD_EVENTS until they aged out, and region:get could only ever read rows no code path produced. Region directors are the missing consumer.

Three hazards were found building it, all worth knowing:

• An empty region ref would have minted a phantom world director. director.New("") is the world director, so a region_defs entry with ref omitted — which the ref-charset lint skips by design — produced a second director sharing the world’s lease id, instance id and durable consumer name. Both claimed leadership (the lease CAS treats a same-owner claim as a renewal), the world’s signal stream was split between them, and the region’s script ran against world state. Refused at the construction site and structurally in the option, so the next caller inherits the refusal.
• The consumer id was not injective. . and : are both legal in a region ref and both were substituted to -, so heart:lands and heart-lands — distinct subjects, distinct leases — collapsed onto one durable consumer. Because Consume calls CreateOrUpdateConsumer, the second director would repoint the shared consumer’s filter at itself and each leadership flip would flip it back, intermittently blackholing a region’s stream. It is now a hash of the ref.
• The director tier hardcoded the demo pack. Harmless while it only read schedules and the world script; not harmless once regions are built from it — on a deployment serving reference the feature would be silently inert, and with a demo row co-existing the directors would own the demo’s regions while shards derive membership from the operator’s packs. It now resolves packs the same way telos-world does.

A leadership change does not destroy a director. It was believed that a director losing leadership is torn down and leaks its gopher-lua VM. Measured, it is not: the same long-lived object campaigns, loses, and campaigns again — losing the lease flips an atomic bool and tears down the durable consumer, and nothing else — and an un-Closed runtime holds no goroutine, OS handle or timer, so a dropped VM is fully reclaimable. Adding the “fix” would have been worse than the non-problem: Runtime.Close nils the LState, CallGlobal dereferences it unguarded, and OnSignal has no recover, so a late signal would crash the director process.