Cross-Shard Handoff

Audience: Engine Developer Status: ✅ Ready

A zone — and the players in it — can move between world servers with zero dropped messages and exactly-once input semantics. This is the backbone of scaling in and out and of rolling upgrades: a draining shard hands its zones off to a peer while players keep their connection. The mechanism is a fenced two-phase protocol (Prepare/Commit) plus a per-player ownership epoch and a signed snapshot, all resting on the single-writer zone model.

Related: Distributed Systems Model, Edge & Protocol (the gate-side redirect/replay), RPC & Protobuf (the Handoff service), Persistence & Durability.

Why it exists

Each zone is owned by exactly one goroutine on one world server, arbitrated by a time-fenced Redis CAS lease. Two things must be true through a migration: no state is lost, and there is never a moment with two writers or an ownerless zone. The handoff carries the authoritative in-memory snapshot directly over the wire (never through the datastore), and flips the lease atomically so ShardForZone never observes a gap.

This is the one path traced at 100 % sampling. beginHandoff and handoverZoneTo each emit a root span with per-hop children (prepare/directory_claim/abort; rpc/lease_flip) and error-distinguishing failure events, and the carve-out sampler records them regardless of the head ratio — because this neither-owns/both-own window is exactly where the hardest bugs have lived. Span attributes are bounded to the zone template, shard ids, epoch, and lease gen — never the player-mintable instance id.

The player handoff (a cross-zone walk)

When a player walks across a zone boundary whose destination is on another shard, the source freezes and snapshots the player and the world redirects the gate:

sequenceDiagram
    participant W1 as source shard
    participant W2 as dest shard
    participant G as telos-gate
    W1->>W1: freeze player, build signed PlayerSnapshot
    W1->>W2: Handoff.Prepare(snapshot, epoch, target, snapshot_sig)
    W2->>W2: verify sig, rehydrate PENDING (applyStateComponents)
    W2-->>W1: {handoff_token, target_shard_addr}
    W1-->>G: ServerFrame{Redirect: addr, token}
    G->>W2: re-dial Play stream + Attach{handoff_token}
    W2->>W1: Handoff.Commit(token+sig) → activate
    W2-->>G: Attached (ack_input_seq = resume point)
    G->>W2: replay buffered input seq > ack (exactly-once)

The PlayerSnapshot (RPC & Protobuf) is the authoritative state so the destination resumes with zero DB round-trips: stats, vitals, inventory/equipment, affects, flags, command aliases, plus state_version (the CAS base), applied_seq (the freeze-point input high-water that seeds the destination’s dedup watermark), persist_id (so the destination CASes the same durable row), and tier. Player-defined aliases ride their own signature-bound field and are re-sanitized on arrival (re-bounded, control-stripped, reserved names dropped) as defense-in-depth against a forged snapshot.

The gate side buffers un-acked input, freezes on the Redirect, re-dials the stream (the TCP socket never moves), and replays from the destination’s ack_input_seq. The world dedups by seq, so a replayed line applies exactly once. See Edge & Protocol.
Storage is bypassed: dumpStateJSON reuses the same serializer as the durable save (byte-identical), and a handed-off character is removed without a save so the destination’s epoch/state_version is never raced by a stale flush from the source (Persistence & Durability).

Abort rolls the pending state back on failure/timeout; the pending record has a TTL. Both phase-2 messages — Commit and Abort — are themselves Ed25519-signed under the shared cluster handoff keypair, a credential distinct from the snapshot signature below. The handoff_token stays deterministic (sha256(character/epoch), so a retried Prepare converges and the token indexes the pending record), but its inputs are public — a name via who, the epoch a small monotonic int — so the token alone is forgeable: anyone with network reach to a world port could recompute it and Abort a player mid-handoff, or call Commit. The signature closes that. Its digest binds the token and the destination shard id under a per-operation domain separator, and the receiver checks that bound to_shard against its own id. Destination binding is load-bearing: the handoff wire is plaintext, so a captured signed Abort for one destination would otherwise be replayable against a second destination in a split-brain same-epoch race — both derive the identical token — to discard the winner’s live pending. A keyless dev/test shard fails closed (it never hands off), so there is no unsigned phase-2 path.

Snapshot signing and the tier

The snapshot is Ed25519-signed over a canonical digest binding character, epoch, target, and state; the destination verifies it when it has a key. Optional claims are written unconditionally, not appended-if-non-empty. tier was safe under append-if-non-empty because it was the only one — presence was unambiguous. Adding a second optional (account) on the same terms made single-non-empty digests collide: digest(tier="", account=X) and digest(tier=X, account="") were byte-identical, so on the plaintext handoff wire an on-path attacker could rewrite an ordinary player’s account into the tier field and keep the signature valid. Writing both unconditionally fixes field position. (Length prefixes solve field boundaries, not presence — a lesson worth carrying to any signed envelope that grows a second optional.) The change alters the digest for existing tier-only snapshots, so a rolling deploy’s skew is fail-closed: a handoff is refused, never mis-trusted.

A handoff mints a durable ownership epoch. Like a fresh login, beginHandoff claims the character through ClaimCharacter, which atomically mints the next owner_epoch — so the destination’s copy can write the durable row and the source’s can no longer reach it. Two details are load-bearing:

The mint happens after destination resolution. Minting at the top of beginHandoff raised the row while the source session still held its old epoch, so a cadence save enqueued before the move came back not-owner naming an epoch this very shard had just minted — and the eviction path then kicked a legitimate player mid-move and discarded their delta. ownershipLost also bails on a frozen/pending session and compares against the epoch that won, not the one the refused snapshot carried.
An unreadable store fails the handoff closed. Resolving the character’s PersistID by name once swallowed a LoadCharacter error identically to a genuine miss, so the handoff proceeded at an unminted epoch+1 — silently unfenced, with no log line, precisely when the store is sick and the fence matters most. A miss is legitimately unfenceable (no row exists yet — the async-create window, or an ephemeral shard) and still falls back with a Debug line; an error now refuses.

This also closes the shared-pending race structurally: because concurrent claims of one character receive distinct epochs, and the handoff_token is derived from sha256(character/epoch), they derive distinct tokens — so two claimants can never collide on one pending record wherever the durable tier is live.

Instanced zones are never a handoff destination: they take no lease, sit outside the placement pool, and Prepare / AdoptZone / the durable ZoneRef read all reject an instance-shaped id outright. This is why the account trust tier rides the snapshot directly rather than the reserved capability flags: carrying the flags would be a forgeable escalation surface (a malicious snapshot could inject admin), so the destination re-derives the reserved flags from the signed tier via applyTierFlags on arrival. An admin or builder therefore keeps elevation across a shard walk — see Trust Tier Model. Two documented caveats: wizinvis is a session concealment (never tier-grantable), so it clears on arrival — a deliberate one-time presence “flicker” across the boundary; and elevation survives only on a signed path — a keyless dev/test shard that skips verification fails closed to baseline (those are single-shard and never hand off).

The zone handoff (graceful drain)

BeginDrain (on SIGTERM) moves whole zones off a shard while they’re still live — the signal context is deliberately separate from the zone-lifetime context so flush + handoff precede loop teardown:

Set draining — reject new fresh logins, still accept inbound handoff binds.
For each hosted zone, choose a peer and atomically flip the lease via HandoverZone (a fenced CAS that flips owner only if the source is still the live owner and sets a fresh TTL in the same script — closing the ownerless-gap window), then post a drain message.
Each zone fans its players off in place — same zone id, same room, now owned by the peer — via the shared handoff path. The socket stays open; the player is redirected (zero drop).
Wait until every zone empties or the deadline; stragglers are durably flushed and left to resume from durable state on reconnect (counted as reclaimed, not zero-drop).

A gate-wedged player is deliberately not handed off, and not counted zero-drop. The Redirect frame travels to the gate over the player’s out channel, so if the gate has stopped reading the Play stream (its buffer is full — the same “wedged” threshold that reclaims a slow client, see Edge & Protocol), the Redirect would drop like any other frame: the gate would never re-dial, and the player would actually drop and reconnect from durable state while the drain reported them as a clean redirect. So the drain skips a wedged player, leaving it resident to be reclaimed at the deadline (a clean reconnect) and classified as a client-fault straggler — the same shape as a link-dead player holding the drain to its deadline. (The complete fix is a gate ack on the Redirect; this is the interim heuristic.)

An unexpected lease loss takes a different path: the fence stops the world without a drain (you can’t hand off zones you no longer own). See Running at Scale for what an operator triggers.

Zone adoption: build, confirm, or un-adopt

A rebalance moves a whole zone between shards, and the two ends are separated in time: AdoptZone makes the destination build and run the zone (rooms, resets, mob spawns, an actor goroutine, a scope subscription), and the source’s HandoverZone lease flip, several steps later, is what actually transfers ownership. Three guards keep that window from leaking a permanently orphaned “zombie” zone:

The destination requires from_shard_id to be the zone’s live owner and refuses before doing any state work. This isn’t a security barrier — owner and gen come from the same directory read, so anyone who can satisfy the generation fence already knows the owner — it enforces at the destination the precondition the source’s flip asserts anyway, so a misnamed source (desynced, lagging, mid-partition, or buggy) can’t make the destination build a zone it will never own.
Adoption is confirmed by observing the flip land, not by the RPC returning. The adopting shard’s lease-renewal loop tries to ClaimZone; a landed flip — from the source, or from any sibling that wins the CAS — makes that claim succeed and marks the adoption confirmed. The gate is “did the flip land?”, observed via lease acquisition, never “is this RPC’s context still alive?” (tearing down on ctx.Err() would delete a zone a concurrent sibling handoff had legitimately just flipped to us).
An unconfirmed adoption is un-adopted when its confirm deadline expires. If the flip never lands — the source’s drain deadline elapsed while the AdoptZone was in flight, or the source died mid-drain — the renewer runs the teardown (Zone Runtime & Actor Model). It is gated on an adopted flag, not merely on “unconfirmed”: a boot zone that genuinely loses its lease must fence, not delete itself — only a zone built at runtime for an in-flight handoff is ever un-adopted. Teardown is best-effort: a refusal (the zone has since acquired a player, or owes a durable write) keeps the zone, because a recoverable leak beats a dropped player.

Drain-target selection is director-owned and serialized. Rather than “first live peer,” the selector picks a target against a soft occupancy ceiling and atomically reserves its headroom in the directory, so two concurrent drains don’t pile onto the same peer. Each reservation carries its own expiry (on the Redis server’s clock), sized to outlast the entire drain — the graceful-drain deadline plus one presence-reflect window — because it is not refreshed during the wait for zones to empty: a slow-but-alive drain must keep its reservation while its players are still in flight, and a straggler landing right at the deadline needs one more presence heartbeat before the target’s weight reflects it. On a clean handover the hold is retired early; only a crashed drainer’s hold lingers the full TTL, after which the per-field expiry prunes it rather than inflating the reserved sum and starving the target. (A too-short TTL that expired mid-drain would let a concurrent drainer read the target’s stale, pre-migration load and over-commit onto it.) If every peer is genuinely reservation-full it admits over the soft ceiling rather than stall (a dropped connection is worse than transient overload, which the rebalancer then corrects). See Running at Scale.

Why that margin isn’t bound by how long step 1 takes. A reasonable worry is that the TTL covers the step-1 offset only while that loop’s own duration stays under the presence-reflect window, leaving an exposure gap at large zone counts. Two properties rule it out. First, the hold is rebased on every accumulate — reserveDrainTarget packs count and expiry into one field and writes now+ttl on each reserve onto the same (target, drainer), so a target’s exposure is measured from its last reserve, not from the start of the loop. Second, the hold is anchored to the same instant as the redirect it covers: handoverZoneTo is immediately followed by that zone’s drain post, so the zone begins fanning players off within milliseconds of its reservation being stamped, and the bridge it must span runs from there to +PresenceReflectWindow — 42 s covering a ~12 s bridge with 30 s to spare, whatever the loop’s total length. Besides, a zone still fanning off 30 s after being told to drain is by construction one whose players are reclaimed rather than redirected, so headroom a lapsed hold stopped reserving is headroom for arrivals that never come — against a soft ceiling that force-proceeds anyway. Refreshing mid-wait is not the easy fix it looks like: it needs an “extend expiry” primitive that inverts the monotonic-shortening invariant ExpireDrainTargetSoon relies on, which would hand a crashed drainer an unbounded hold. Since this is reasoning rather than measurement, retireDrainTargets WARNs when the hold was already gone while this drain was still sending to that target — the precise signature of a reservation lapsing early. If it never fires, the bound held; if it does, the question reopens with a measured duration attached.