Distributed Systems Model

Audience: Engine Developer Status: ✅ Ready

TelosMUD scales a persistent, stateful text world horizontally by sharding it into zones and giving each zone exactly one owner. The invariants that make that safe — single-writer ownership, time-fenced leases, monotonic placement, apply-once event delivery — are collected here. This is the “why it’s correct” companion to the mechanisms documented on Zone Runtime & Actor Model, Cross-Shard Handoff, Scoped Event Bus, and Persistence & Durability. For the operator’s view see Running at Scale.

The four services and how they scale

Service	Scaling model	Coordination
`telos-gate`	horizontal, stateless edge	routes a session to its shard via the Redis directory; any gate can serve any player
`telos-world`	horizontal by zone, single-writer per zone	claims zones from a pool via Redis CAS leases
`telos-director`	singleton per scope, leader-elected	Redis lease; warm standbys (Orchestration & Directors)
`telos-account`	horizontal, stateless behind Postgres	off the hot path — the world trusts a signed assertion

The gate holds no authoritative game state, so you add instances freely. Session→shard routing is Redis-directory-driven: the player’s placement record names their zone, and ShardForZone resolves whichever shard currently owns it. A cross-shard move carries the destination address in the world’s Redirect frame, so there is no gate affinity.

The single-writer spine

Each zone is one goroutine that owns all its entities; every mutation funnels through its inbox. No in-zone locks, deterministic ordering, and a clean unit of placement. Two-writer prevention is enforced at the directory, not in-process:

Time-fenced lease CAS: claimZone grants ownership only if the zone is unowned, expired, or already yours, using the single Redis TIME clock so a skewed shard can’t steal a live lease. Default lease 15 s, renewed at TTL/3.
Duplicate shard-id refused: registering a shard id that a different live endpoint holds is rejected — the guard that stops two same-id owners both reading as renewals.
Renewal restarts on re-adoption. A zone the rebalancer moves away and later moves back (A → B → A) must restart lease renewal. If the “handed off” marker were set-only, the shard would host and serve the zone while renewing nothing; once the 15 s lease lapsed, ShardForZone would resolve nobody and ClaimZone — fenced only against a live lease — would grant the zone to any shard that asked, while this one was still writing to it. That is a guaranteed second writer on a routine operation, so the marker is cleared and renewal restarted on re-adoption, and the adopting state is bounded.
Monotonic placement epoch: epoch is a per-player monotonic fence. Only the handoff coordinator bumps it, and the handoff CAS demands a strictly greater epoch — so a stale or duplicated handoff can’t roll a player back to a shard they’ve left.
Monotonic ownership epoch: owner_epoch is a second, separate per-character fence living on the durable row, minted atomically by every ownership claim (login and handoff alike) and applied as its own conjunct on every save. Do not confuse the two. The placement epoch lives in the evictable directory and fences routing; owner_epoch lives in Postgres and fences writes at the sink. That split is deliberate: a durability fence whose high-water mark came from evictable Redis would refuse a legitimate player’s every save for the life of their session, so the mint floor is max(directory, row) and only ever raises. See Persistence & Durability.

Cross-boundary interaction is message-passing only (the gRPC Play stream and the Handoff RPC); no shared mutation ever crosses a shard.

The placement record

The directory keeps one record per player. It is the reconnect-routing spine, and it carries more weight than its name suggests — three subsystems read it:

Field	Read by	Meaning
`zone`	the gate, on reconnect	the routing key. Resolve it through `ShardForZone` to get the zone’s current owner
`epoch`	the handoff CAS	the monotonic fence; also the existence key for the tell/mail oracle
`nonce`	the logout tombstone	a per-session token that fences a same-shard relog (below)
`shard`	nothing, for routing	vestigial for routing; dropped by the logout tombstone

Why zone, not shard. A shard id says where the player was. The moment that zone is rebalanced onto another shard — or that shard exits — the id is stale, and a returning player was dropped into the home zone’s start room, losing their durable location. Routing by zone makes an offline rebalance transparent, because ShardForZone always names the current owner.

Every residency writes a placement. Originally the record had exactly one writer, the cross-shard handoff CAS — so a player who simply logged in and stayed put had no placement at all, which made them unroutable on reconnect and invisible to the tell/mail existence oracle (“there is no player by that name”). The world now registers a placement whenever a player becomes resident in a zone: fresh login, link-dead resume, cross-shard arrival, and the intra-shard zone walk — which changes zone without changing shard or epoch, and so never triggers the handoff CAS.

That required a second writer with different semantics. The handoff CAS demands a strictly greater epoch; a login re-registers at the epoch it just resumed from, so reusing the CAS would make every login a silent no-op. registerPlacement therefore accepts an equal epoch, refuses a strictly newer one, and keeps the stored epoch at the maximum. This is safe precisely because an epoch maps to exactly one shard: only the handoff coordinator bumps it, so an equal-epoch write can only ever rewrite the zone within the shard that already owns the player. The hand-off to the background writer coalesces per player rather than dropping a full FIFO.

Logout writes a fenced tombstone, not a delete. Deleting the record would ship three regressions at once: the tell/mail existence oracle would refuse tells to an offline character; a delayed or retried handoff write would find no current value and apply, resurrecting a stale placement; and the returning player would lose their zone. So logout drops only the shard field, and only when the record still names this shard at this epoch — a compare-and-delete, so a clear racing a fast relog or a handoff is a no-op. Existence is therefore keyed on epoch (which every writer sets), and the tombstone also writes the quitting zone, since a logout offered while a zone-change registration is still pending would otherwise leave a stale zone behind. An outright ClearPlayer still exists — as character deletion. The placement is an existence-and-routing oracle, not a liveness one: because it persists across logout (a tombstone, and every login re-registers it), a present record answers “does this name exist, and where does it route” — not “is the player connected right now”. A live-connection decision — for instance, whether to ping a mail recipient the moment mail arrives versus letting them see it on their next mail — reads the presence roster (the live-connection oracle) instead, and only to gate the action: the tell/mail publish subject is still derived from the sanitized recipient name, never from a roster entry, so “presence never routes” holds.

The shard+epoch compare-and-delete fences a relog on a different shard and a cross-shard handoff, but not a same-shard relog: a fresh login resumes the same epoch (registration accepts an equal epoch by design), so a late-draining clear matched the live record on both axes and could blank a connected player’s shard field — safe only via world-side single-writer ordering the directory can’t see. So each session mints a nonce, stamped into the record on every write; the logout clear carries the quitting session’s nonce and only deletes if it still matches. A relog rewrites the nonce, so a stale clear is rejected and the live record survives. (The fence is present-only, so a legacy or handoff-CAS-only record with no nonce is still clearable on shard + epoch alone.)

Placement: claim-from-pool, not declare

World servers claim zones from a shared pool rather than statically declaring them. On boot a shard registers its id→endpoint, then walks the configured pool and wins each free zone via the directory CAS; a zone already owned by a live peer is skipped. Liveness is decentralized and works even with no director running — claim-from-pool plus lease expiry is the failover mechanism (a crashed shard’s zones become unclaimed when its 15 s leases lapse). A server that wins nothing runs as a warm standby. The director’s placement role is an optimizer, not a dependency.

The bootstrap core zone is hosted locally + unleased on every shard, so a fresh/empty fleet still serves a lobby.

The durability ladder as a distributed invariant

State is checkpointed to Redis (~10 s) and Postgres (~60 s), under two distinct guards. state_version is optimistic-concurrency contention control — the CAS WHERE state_version = $expected reports only that somebody wrote since you read, and every caller answers a loss by re-reading and rebasing. It is therefore not an ownership fence: a stale writer rebases exactly as a legitimate one does, which is how a zombie owner’s 60-second-old snapshot could roll the live copy back. The ownership fence is a second column, owner_epoch — a monotonic generation minted atomically by every claim (login and handoff), applied as its own conjunct WHERE owner_epoch <= $k that a rebase cannot reach, so an epoch names exactly one live copy. See Persistence & Durability. On load the freshest of {row, checkpoint} wins by (owner_epoch, state_version), with a state_version tie broken toward the checkpoint — since a checkpoint is dumped at the pre-CAS version it ties the row while carrying newer content, so the tie-break is what makes the crash-loss window genuinely the ~10 s checkpoint cadence rather than the ~60 s flush. See Persistence & Durability.

Event delivery guarantees

The scoped event bus splits transient down-broadcasts (at-most-once, NATS core) from durable signal-up (at-least-once, JetStream). Correctness comes from apply-once over at-least-once: the director consumes the durable stream only while leader (stable consumer id → resume from last ack) and dedups by a per-source monotonic watermark. A capstone test proves exactly-once effect — precisely three boss kills counted across a mid-sequence director restart.

Failure modes & degradation

The world degrades rather than crashing:

Redis down → single-shard mode; cross-shard exits sealed.
Postgres down → ephemeral characters; hot-reload disabled.
NATS down → comms/tells/hot-reload disabled.
A dead gate → gRPC server keepalive pings reclaim leaked world streams.

Security-sensitive gaps fail closed on boot rather than running open: a gate with no account target, a shard with no handoff verify key, an account service with no caller token, or a pack-set divergence each refuse to start unless TELOS_ALLOW_INSECURE is explicitly set (Deployment).

The scaling ceiling and honest gaps

The fundamental limit is one zone = one core: a zone is a single goroutine with a 250 ms heartbeat budget, and all combat rounds and affect ticks run inline on it. telos.zone.tick_lag_ms (how far past 250 ms a heartbeat fired) is the headline scale signal. A single hot zone cannot exceed one core — it does not shard within itself. Instanced zones are now implemented, but they are an isolation mechanism, not transparent load-sharding: a runtime-minted copy is private, and occupants of different copies cannot see or interact with each other. So instancing solves the dungeon/party case, and does not relieve a single crowded public zone — that ceiling stands. There is no hard players-per-box constant in code; capacity is measured with the bot-swarm load tester plus OpenTelemetry (Running at Scale).

Placement is a maturing subsystem — the load-aware rebalancer is shipped, but a few edges remain:

Rebalancing is automatic and load-aware. Each shard heartbeats per-zone occupancy; the leader director plans a player-weighted spread and issues rebalance-drain directives that the owning shard’s executor drains via the handoff, with a 5-minute per-zone cooldown and locality-aware colocation. It stays an optimizer — if the director is down, zones are still claimed and served, just possibly unbalanced.
Drain-target selection is director-owned and serialized — a peer is chosen against a soft occupancy ceiling and its headroom is atomically reserved in the directory, so concurrent drains don’t pile onto one peer. Each reservation carries its own expiry (the clock is the Redis server’s, not the caller’s) sized to outlast the whole drain — the drain deadline plus one presence-reflect window — since the hold is not refreshed during the wait for zones to empty (though it is rebased by each further reserve onto the same target), so a slow-but-alive drain keeps it while players migrate and only a crashed drainer’s hold lingers that long before it’s excluded from the reserved sum and pruned. (Refreshing one whole-key expiry per reserve was the opposite: under exactly the concurrent fleet rollout this guard exists for, a crashed drainer’s stale hold survived indefinitely as long as other drainers kept reserving onto the same hot target, inflating the sum and spilling everyone to the fallback.) If every peer is genuinely reservation-full it admits over the soft ceiling rather than stall — a dropped connection is worse than the transient overload the rebalancer then corrects. The reservation is an admission hint, not a durable lock.
Orphan-zone failover is not instantaneous — it relies on lease expiry (~15 s) plus a re-claiming shard on boot; there is no running-standby auto-adopt loop, so recovery is ~lease-TTL + reclaim.