Persistence and Durability

Audience: Engine Developer Status: ✅ Ready

Player state lives authoritatively in shard memory (on the zone goroutine) and is written down a three-tier durability ladder — memory → Redis → Postgres — trading latency for safety at each rung. Two distinct guards protect the durable row: state_version is contention control (somebody wrote since you read), and owner_epoch is the ownership fence that stops a zombie copy of a character from writing at all. Conflating them once shipped a live duplication bug — see Two guards. The whole ladder is optional: with no store configured, characters are ephemeral and the engine still boots.

The durability ladder

shard memory (authoritative, zone goroutine)
   │  ~10s   → Redis checkpoint   (shrinks the crash window)
   ▼  ~60s / logout / drain → Postgres CAS (durable record)

Tier	Keyed by	Holds	Role
Shard memory	—	live entity state	authoritative while online; holds `state_version`
Redis checkpoint	character name	full `CharSnapshot` JSON (carries `state_version` + `owner_epoch`)	crash-window shrinker; name-keyed so any shard can rehydrate a player it never saw. TTL 1h. Guarded by a Lua ownership check (below)
Postgres	`id`	identity/location columns + one `state` JSONB + a `chargen` column	the durable record. No per-stat column — the pillar

newSaver(store, ckpt) accepts either or both as nil; both-nil makes every save a no-op (ephemeral, storeless boot).

When state is written

Two cadences are pulse-scheduled per zone (the 250 ms pulse): a checkpoint every ~10 s (saveCheckpointPulses = 40) and a flush every ~60 s (saveFlushPulses = 240) — both package vars, so tests can shrink them. The callback registers lazily the first time a persisted player joins; an ephemeral shard never registers it. saveAll dumps every live, persisted, non-frozen, non-pending player — a player mid-handoff is skipped because its durable record belongs to the other shard.

`saveReason`	Redis	Postgres	Trigger
`saveCheckpoint`	✓	—	~10 s tick
`saveFlush`	✓	✓	~60 s tick / drain / live reconcile
`saveFinal`	✓	✓	logout / leave

Produce/consume split: the zone goroutine produces a CharSnapshot (dumpCharacter, zone-owned reads only) and hands it to the saver over a buffered channel (depth 256). enqueue is non-blocking — a full queue drops the request rather than stall the actor loop. The saver drains on a single background goroutine, doing all blocking Redis+PG I/O off every zone goroutine, bounded by a 5 s I/O timeout.

Shutdown waits for the queue. Cancelling the saver’s context would return the drainer without emptying its buffer — and the drain’s reclaimed stragglers enqueue their flush last, microseconds before shutdown, so the one cohort whose only durability path is that flush was exactly the cohort most likely to lose it. (Redirected players were never at risk: their state crosses in the handoff snapshot.) So shutdown enqueues a sentinel whose channel the drainer closes on dequeue: because a single goroutine drains a FIFO, dequeuing the sentinel proves every earlier request finished its I/O. The barrier sits between the drain and the world stop, is bounded so a wedged store delays shutdown rather than hanging it, and watches the shard’s run context so a lease fence that already killed the drainer returns at once.

The placement writer (the background goroutine that applies placement-record updates) gets the same barrier, for the same reason: a player who quits during a graceful shutdown enqueues their clean-logout tombstone on it, and if the world context cancelled before that drained, the record would keep naming a shard that is exiting — leaving the tell/mail existence oracle reporting the player as hosted on a dead shard until their next login. FlushPlacement mirrors the saver’s barrier; a hard context cancel mid-drain (a lease fence) returns without closing the barriers, and a re-check after the write loop stops a cancel during the last write from reporting a false success.

The character audit trail

Separate from the mutable durable record, an append-only character_audit table records the permanent things that happen to a character or account — creation, death, a permanent attribute-base grant, an advancement-track step, a cross-character item transfer (below), and account tier changes — so staff can answer “what happened to this character, and who did it.” One shared table is written by two services through the shared store package, each with the durability posture its source demands:

telos-world emits died / attribute / track through an async per-shard auditor — a saver twin: off-zone-goroutine writes that drain on graceful shutdown, so a death record survives a clean drain but never blocks the actor loop. The drainer coalesces: instead of one single-row INSERT per event, it batches up to 64 into one pipelined round-trip (pgx.Batch), so a death-storm against a slow Postgres empties the 256-deep queue in ~4 round-trips rather than 256 — shrinking the window in which a full queue would drop. A batch error falls back to per-row appends under the same bounded context, so one poison event’s blast radius is a single row, not the whole batch, and a wedged DB still can’t reintroduce slow-drain.
telos-account emits create / tier_changed in the same transaction as the change itself — atomic, so an account mutation is never applied-but-unrecorded.

Writes are idempotent on (subject_id, event_kind, dedup_key) with ON CONFLICT DO NOTHING, so a retried enqueue or an at-least-once path can’t double-record. The dedup key must be durably unique: died mints a fresh per-death UUID (the l.dying latch is the re-entrancy guard) — an earlier design keyed it on the transient deaths counter, which restarts at 0 every relog and so collided across sessions, dropping every death after the first; tier_changed keys on the account_role_audit row id, track on <track>\x1f<step>, create is one-per-subject. The read surface is the audit command: a self-view scoped by the caller’s stable pid (never the mutable, reusable name — a name-reuse leak), and a staff-only audit <name> refused before any store call and rate-limited. Newest-first reads order by (at DESC, seq DESC) — a seq identity column, not the random UUID id — which is true insertion order and, crucially, orders within a batch correctly even though its rows share one transaction’s now(). The staff audit <name> view also folds in the target account’s tier_changed rows (subject_type=account, so reachable by neither the pid self-view nor the by-name read on their own); the lookup resolves name → account_id → that account’s tier rows, staff-gated and scoped to the owning account so there is no cross-account leak. A nil sink disables auditing entirely (the bare-engine invariant), and a storeless shard degrades cleanly.

Detecting cross-character item transfers

Items are prototype-ref flyweights inside characters.state JSONB with no global identity, so the ownership fence — which protects a row — cannot answer “did this item cross a character boundary?” when a zombie session drops wealth that another character picks up (the write lands in a different row, under that row’s epoch). The transfer audit makes that question answerable, reusing the same character_audit trail:

A transient Released marker records who dropped or put an item down.
When a different saved player picks it up, an item_transferred row is emitted (subject = the acquirer, actor = the releaser, payload carries the item ref, stack, names, and room). It covers both drop → get and put → get-from-container.
A self-pickup, a never-player-released floor item, and a mob/engine move record nothing; the marker is cleared on pickup (a most-recent-release invariant, so there is no stale mis-attribution). Bound items can’t cross an owner boundary at all, so they never reach here.

This is detection only — no prevention, no conservation invariant. It converts “we cannot know” into “we can know,” which is the precondition for the durable-item-identity work deferred to Launch. Like the async auditor it is best-effort (a runtime marker across the live double-own window), not a guarantee.

Two guards: `state_version` and `owner_epoch`

They answer different questions, and conflating them shipped a live duplication bug.

state_version is contention control. The CAS lives in SaveCharacter: the UPDATE applies only WHERE id = $id AND state_version = $expected, bumps state_version + 1, and RETURNING state_version. Zero rows updated (pgx.ErrNoRows) means the writer lost the race. It reports only that somebody wrote since you read — and every caller answered that report the same way: re-read, rebase, write again.

It was never an ownership fence, though this page once said it was. Because a rebase is the documented response to a loss, a stale writer answers the CAS exactly as a legitimate one does. finalizeFlush rebased explicitly (snap.StateVersion = cur.StateVersion), and the guard meant to prevent that — zonePresent — probes this process’s z.players, so a live session on another shard was structurally invisible to it and both shards concluded their own map was authoritative. The result was a rollback: a stale shard’s 60-second-old logout snapshot force-wrote over the live owner’s state. Externalize wealth on the live copy, let the stale copy roll the character back, repeat — an attacker-timed, repeatable primitive that needed only patience.

owner_epoch is the ownership fence, and it is enforced at the sink:

Every ownership claim mints the next epoch atomically from that one column (ClaimCharacter) — a fresh login and a cross-shard handoff. Two claimants can never receive the same value, so an epoch names exactly one live copy. A read-then-bump would have restored the bug in a shape that looks fixed.
SaveCharacter applies only WHERE owner_epoch <= $k. A rebase moves state_version and cannot reach a separate conjunct — which is exactly why the fence had to be a second column rather than stricter handling of the first.
The result carries three outcomes, not a bool. ok bool became SaveResult/SaveOutcome, because an epoch loss and a version loss demand opposite handling, and a bool that already meant three things is how the force-write shipped. Its zero value is invalid, so a store double that forgets to set it fails loudly rather than reading as success.
An epoch loss is terminal on both save paths. The live path used to answer a CAS loss by re-reading and re-enqueuing immediately; routing an ownership loss there would spin the shard-wide saver drainer forever on a write that can never land. The zone is told instead, and evicts the zombie.

The mint floor is max(directory, row) and only ever raises. The directory is evictable Redis, so deriving a durability fence’s high-water mark from it would wedge legitimate players whose every save was then refused for the life of the session. Login fails closed if it cannot claim, or cannot read the row at all: a session that cannot prove ownership cannot save, and admitting it means hours of silently unpersistable play.

Who reconciles a version miss still depends on the reason:

saveFlush (live player): post a saveConflictMsg back to the zone; the zone re-reads the current version off-goroutine and re-dumps current in-memory state at it. The zone goroutine is the authority while the player is present.
saveFinal (logout): the session is gone, so there’s no one to bounce to. finalizeFlush probes the zone (zonePresent, a single-writer round-trip) and, if the player is truly gone, rebases state_version onto the current row and retries (bounded: 8 retries / 2 s budget). On exhaustion it logs a greppable event=final_flush_dropped — the last durable flush still recovers on next login, so it’s “logout delta lost,” not “character lost.” That rebase is safe only because owner_epoch fences it independently: a stale shard’s rebase cannot reach the epoch conjunct.

What the fence does not close. Wealth a zombie externalized before detection landed in another character’s row, under that character’s own epoch — and an epoch on row X cannot fence a write to row Y. This removes the rollback and with it the repeatability: an unbounded, attacker-timed primitive becomes a one-shot, alerted, crash-equivalent discard. Closing the remainder needs durable item identity plus paired transfer-audit rows — the transfer audit below is the first half of that (detection); the conservation invariant and durable identity are Launch work.

Load & the freshness check

The checkpoint tier was a complete bypass one rung up. It carried no CAS at all: one key per character, both copies of a double-owned character pulsing it every ~10 s, and the login read preferring the checkpoint on a state_version tie — which is the normal state between a login and its first flush. It now carries a Lua ownership guard and reports a refusal as ErrCheckpointNotOwner. Because this tier detects a double-own roughly 6× sooner than Postgres does, that signal is routed rather than swallowed.

loadCharacterSnapshot runs off the zone goroutine and picks the freshest of {Postgres row, Redis checkpoint} ordered by (owner_epoch, state_version) — ownership first, so a stale copy’s checkpoint can never outrank the live owner’s row — then breaking a state_version tie toward the checkpoint (>=). That tie-break is what makes the crash window actually the ~10 s checkpoint cadence rather than the ~60 s flush: state_version only advances on a durable Postgres CAS, and each checkpoint is dumped at the pre-bump version, so between two flushes the ~10 s checkpoints carry newer content (the position the player just walked to) at the same version as the row. Under a strict > a present row always won and the checkpoint tier was inert for every already-flushed player — it paid the Redis traffic but never shortened recovery, silently leaving the crash-loss window at the ~60 s Postgres cadence. Preferring the checkpoint on a tie is safe, not merely convenient: at an equal version the checkpoint’s content is by construction at least as recent as the row’s (both dumped from the same on-goroutine state, the checkpoint rewritten every pulse); a genuinely newer row bumps the version and still wins on >, and a genuinely stale checkpoint sits at a strictly lower version and is still rejected. (The rejected alternative — bumping state_version on a checkpoint write — would desync the in-memory version from the row’s optimistic-CAS token and make every flush conflict.) loadCharacter then applies on the zone goroutine via the shared applier applyStateComponents — the same path the cross-shard handoff rehydrate uses.

Two correctness rules in the applier:

Derived vs base: only base overrides and current pool values are stored; derived values recompute on load. Attributes store base overrides only; resources store cur only (max is derived); affects/cooldowns store remaining duration in pulses (conserved, never reset to full).
Resource-current installs last — after attributes, affects, and gear land — so a wounded cur clamps to the genuinely-final (possibly buffed) max. The ordering is the correctness.

A fresh login forces appliedSeq = 0 (restoring the saved value would drop the returning player’s first inputs). The chargen column (content-driven chargen, Model A) is applied on first spawn and nulled in the same CAS write that persists the built state — application and clear are atomic.

Where a returning player lands

characters.zone_ref is the durable record of which zone to rehydrate into, and it is the login routing key. Zone selection therefore happens after the snapshot load — the durable record is what names the zone:

An instance is never written here. Because the recorded zone must be the zone that actually holds the session, a shard-local ephemeral copy can’t be the durable location — and projecting its template would be worse, naming a zone another shard may own and letting a reconnect fresh-log a second copy of a live character. Instead the player carries an anchor (the zone + room they entered from): registerPlacement skips an instance entirely so the last good record — the entrance — stands, and clearPlacement still tombstones but drops the zone, which would otherwise dangle at a reaped id. zone_ref preservation is enforced at the sink with COALESCE rather than by writing "", since an empty ref maps to SQL NULL and would clobber the anchor while room_ref kept the instance’s authored room — an internally inconsistent row that loses the location outright.

Case	Attaches to
handoff re-dial (carries a handoff token)	whichever zone holds the matching pending player
rehydrating login	the zone named by the durable `zone_ref`
brand-new character	the shard’s home zone
a `zone_ref` this shard does not host	the home zone, with a warning (see below)

This was once wrong in a way that quietly destroyed player location. Every non-handoff login attached to the shard’s home zone, and the room resolver then silently fell back to that zone’s start room whenever the saved room_ref named a room some other zone hosts. Because a single shard hosts many zones (the demo pack ships three, and midgaard’s market exits north into darkwood), an ordinary intra-shard walk followed by a logout lost your location — no rebalance and no cross-shard hop required. Fixing it also fixed link-dead reconnect, which used to route home, find no player there, and spawn a duplicate while the detached copy waited in the other zone for the reap.

The room resolver now warns when a non-empty room_ref names no room in the zone (an empty ref is the intended new-character/respawn path and stays quiet) — that silent teleport-to-start-room is precisely how the data loss stayed invisible. Relatedly, the Redis checkpoint leg hand-maps CharSnapshot into its own struct and back, so a reflect-based drift guard now fails the moment a new snapshot field isn’t threaded through: zone_ref is the routing key, and dropping it there would send every crash-rehydrated player home on the one path a Postgres-only test can’t see.

The other half of this — getting the player to the right shard in the first place — is handled by the directory’s placement record, which now carries the player’s zone. The gate resolves that zone through ShardForZone to whichever shard currently owns it, so a rebalance that moved the zone while the player was offline is transparent, and a shard no longer receives logins for a zone it doesn’t host. The world still registers a placement on every residency change — fresh login, link-dead resume, cross-shard arrival, and the intra-shard zone walk — and a clean logout leaves a fenced tombstone rather than deleting the record.

A live reconnect routes by in-memory residency, not the durable zone. The durable zone_ref lags an intra-shard zone walk: transferIn never flushes a durable record, and detach’s save drains asynchronously. So a link-dead resume that beat the flush would read the stale pre-walk zone, find no session there (the character was removed from the old zone on the walk), fall into the fresh-login branch, and spawn a second live copy while the detached one still sat in the zone it had walked to — a double-owned character. The shard therefore keeps a live, in-memory character → zone residency index, maintained from the same two z.players mutators (so it mirrors residency exactly), and a token == "" reconnect consults it before the durable zone_ref: a still-held session routes to its actual zone, and the durable record is only the fallback for a genuine miss.

Flyweight item storage

Items are flyweights: a shared *Prototype plus a per-instance delta. Persistence stores only the proto ref + the delta, never the resolved item. ItemJSON is {ProtoRef, Delta, Contents}; loadItem re-spawns from the ref (“persistence only chooses what to spawn and where”). An unknown prototype (content stripped or renamed) is skipped with a warning — the character loads lighter, never crashes.

The delta (itemDeltaJSON) carries Quality (rolled loot level/affixes), Bound, Stack (partial material stack), and Kept. A populated delta must be owned bytes, never aliasing a live COW buffer, because the saver reads the snapshot off-goroutine. Carry-tree guards bound depth (16), width (512 nodes), and size (256 KB).

Ephemeral spawns and zone resets

Most entities are ephemeral — spawned by zone resets (repop) and never stored. The same runResets runs at zone boot and on the repop pulse timer, both on the zone goroutine with no I/O (resets come from already-loaded content, so a storeless zone repops identically). The top-up semantic prevents leak and duplication: each op declares a max, the interpreter counts the live instances it owns (room-scoped), and spawns only the difference — idempotent on a full zone.

Reserved: the durable world-object write side. A reset op flagged persistent routes to a load-once-at-boot path, but the ObjectLoader is read-only and nothing wires it today (z.objects == nil), so the persistent gate degrades to a logged no-op. Saving object deltas back is deferred; the demo pack flags no persistent op.

Cross-shard movement bypasses storage

A handoff carries the fat snapshot directly over the wire, not through the store — dumpStateJSON reuses the same dumpStateComponents as the durable save (byte-identical, one serialization). A handed-off character is removed without a save, so the destination shard’s epoch/state_version is never raced by a stale flush from the source. saveAll skipping pending/frozen players is the complementary fence. See Cross-Shard Handoff.

Degradation

Redis down: each checkpoint failure is non-fatal (a missed checkpoint widens the crash window by one tick); load falls back to Postgres.
Postgres down: a flush failure is non-fatal and retried next cadence; a read failure is treated as “no durable row.” A nil store means ephemeral.
Saver overload: a wedged saver (full 256-deep queue) drops requests rather than blocking a zone; the 5 s I/O timeout keeps a hung call from wedging the drainer.