Content Pack Operations

Audience: Sysadmin Status: ✅ Ready

Operating content on a running fleet: publishing content from an external versioned store, installing a version with telos-pull, driving a director-coordinated in-game pull/reload across shards, and understanding which packs a world actually loads. The model is seed/pull + enabled set + reload — there is no per-pack on/off toggle. For the pack format see Pack Authoring; for the builder-facing overview see Content Packs Intro.

This is how a deployed fleet gets its world. The published images ship core-only (the demo fixture is stripped), so a real deployment pulls its content pack from the external store into Postgres — it does not run an embedded seed. telos-pull is a published image for exactly this. The live staging environment, for instance, pulls the reference pack from telosMUD-content rather than seeding the demo.

The source-of-truth model

Canonical content lives in an external, versioned repository, not in the game — there is no in-game editing of canonical content. A published content repo has, at its root:

manifest.yaml       # version (required), content_hash (required), packs[] (required),
                    #   optional created_at, ci_run, engine_min
packs/              # the pack trees, exactly as authored
  <name>/...

version — the logical version label (a git tag/SHA).
content_hash — computed over the packs/ subtree and verified on import, so a tampered or truncated tree fails closed.
packs[] — the enabled set this version publishes.

The world process never runs a pull. It always reads content rows from Postgres; the pull machinery updates those rows and the world serves from them (Model 1 — Postgres stays the serving source).

Installing a version: `telos-pull` → `ImportVersion`

telos-pull resolves a git tag/SHA from the configured content store (content.url / content.version), reads the manifest.yaml, verifies the tree hash, and imports exactly the packs the manifest names — atomically. In one serialized transaction, ImportVersion:

prunes packs that the new version dropped,
strip-replaces each named pack’s rows,
overwrites content_pack_registry (the enabled-set authority — below),
bumps the monotonic content_version.

Re-importing the same content hash is idempotent — no version bump. This is the versioned, ops/CI-grade path; contrast telos-seed → ImportPack, the dev path that imports the embedded demo pack with a wall-clock version (see Pack Authoring → ingestion paths).

Switching a seeded deployment to `pull`

Definition refs are global, not per-pack. zones.ref, rooms.ref and most *_defs tables are ref TEXT PRIMARY KEY, with pack a plain column. That is normally invisible, because step 1 above prunes the packs a version drops before importing the ones it adds — but the prune diff is driven by content_pack_registry, and telos-seed does not register (it calls ImportPacks, which never touches content_version or the registry). So after a make seed the demo pack’s rows exist while the registry is empty, and the first pulled pack shipping the same zone ref collides with rows the importer cannot see:

import version: store: insert zone midgaard: duplicate key value violates
unique constraint "zones_pkey" (SQLSTATE 23505)

That is now detected before the inserts and refused with a typed error naming the owning pack, a sample colliding ref, and the remedy — telos-pull --purge-pack <name>. The check sits inside ImportVersion’s transaction, after the prune loop and before the inserts, which is the only place it can be authoritative: that section already holds SELECT … FOR UPDATE on the content_version singleton, so a pre-flight check in the puller would read outside the lock and race. Running it after the prune also means a pack this version legitimately drops has already gone, so an ordinary pack rename cannot false-positive — and it covers the director-coordinated pull for free, which hits the same collision as the CLI.

It deliberately does not “prune anything unregistered.” A seed-imported pack with disjoint refs is harmless — the world’s enabled set comes from the registry, so those rows are inert — and stripping it automatically would be a destructive change to environments that work today. The check fires only on an actual collision.

Two hazards worth knowing, both found in review. PurgePack takes the same FOR UPDATE lock, because a check and the action it guards must share a lock, not merely a transaction: under READ COMMITTED a purge could read “not registered”, an import could commit a version installing and registering that pack, and the purge’s delete would then remove the rows the import just wrote — both reporting success, leaving the registry describing content that is gone. And the error message now carries the registered subset and names the correct remedy per case, because it previously asserted a registry fact it had never checked (“almost certainly a telos-seed import”) — true for ImportVersion, false for ImportPacks, so re-seeding against a pulled, registered pack printed a wrong diagnosis plus a command PurgePack is guaranteed to refuse. A remedy that dead-ends is worse than no remedy.

A prune can be refused by a live zone

Step 1 is guarded: a director-coordinated pull will refuse to prune a pack whose zones are live in the fleet. Two ways a zone counts as live, and they have different remedies:

A shard holds its lease — drain that zone (or the shard) and retry.
It has live instanced copies — a dungeon template usually sits in no shard’s zone pool and so holds no lease at all, but parties may be standing inside private copies of it right now. Shards heartbeat a TTL’d in-use claim per template for exactly this case. Wait for the parties to finish (the claim clears ~45 s after the last copy is reaped) or drain the shard to eject them. The refusal names the reason:

prune guard: zone has no lease but is LIVE as an instance template (parties are inside copies of it right now); the pull is refused until they leave

The guard fails closed — an unreachable directory refuses the prune rather than assuming nothing is using the zone. Operator detail in Running at Scale.

Overriding the veto: `pull <version> force`

An advisory check with no override is really a veto — and since the guard learned about instance templates, one character logged in inside a dungeon copy and issuing no commands could block every content deploy indefinitely (an occupied instance is never reaped). force overrides it.

The guard still runs. The blocked list is computed, logged, returned, and reported back to the operator. Force downgrades the veto to a report; it does not skip the check — whoever overrode it needs to know exactly what they overrode, and so does whoever reads the log afterwards.
It requires admin, on top of the builder flag pull already needs. Installing content is a builder job and this guard is precisely what makes delegating that safe, so if the holder of the power could waive its own gate the gate would be decorative. Force also changes the kind of operation — it strips content the fleet may be hosting — so its blast radius scales with the fleet. A builder without admin is refused and enqueues nothing.
A stale override is downgraded, not applied. The request rides a durable at-least-once stream that NAKs on a non-leader, so a failover can delay delivery arbitrarily — long enough for the one idle player who was waived to have become a raid. Past forceMaxAge the request reverts to an ordinary guarded pull, because the override waived a guard whose entire value is that it reflects current occupancy.

The remedy the tooling prints leads with “redirect those characters first.” The obvious wording — players keep playing, then roll a reboot — is actively misleading, because the reboot is the harm event: a drain hands each zone to a peer, the peer cannot build a zone whose content was stripped, the handover fails, and those players are reclaimed from durable state — dropped and routed to their home start room rather than where they were. Both the request ack and the result line therefore lead with redirecting the affected characters, and both name the pinned-pack hazard: a shard pinning a stripped pack via TELOS_CONTENT_PACKS refuses to boot, so the prescribed remedy could otherwise take the fleet down.

Note that an unconditional force already existed and was worse: telos-pull/CI sets no guard at all, so anyone with CLI access had a silent, unlogged, unreported bypass. This replaces it for the in-game path with one that computes the blocked list, records it, reports it, and requires admin.

Cross-pack constraints enforced at import (so a multi-pack batch stays consistent): refs must be globally disjoint across the batch, pack names must be unique, and exits cannot cross packs (packs are self-contained worlds). Inserts are phased FK-safe (zones → rooms → exits → prototypes/resets → global defs).

Which packs a world loads: the enabled set

A world resolves its enabled pack set by precedence (ResolveEnabledPacks):

Explicit operator override — TELOS_CONTENT_PACKS (comma-separated).
else the content_pack_registry — what a pull/director last imported.
else ["demo"] — the bare-dev default.

Fail-closed consistency guard. If an explicit TELOS_CONTENT_PACKS override diverges from the published registry set (order-sensitive), the world/account refuses to boot — unless TELOS_ALLOW_INSECURE=1. This prevents a shard from silently serving a different pack set than the rest of the fleet. Production does not set the insecure flag; it makes the override match the registry, or drops the override and lets the registry drive.

So changing what is live means: pull a version (updates the registry + rows) and, if you use an explicit override, keep it consistent — then reload or reboot so running shards pick it up. There is no command that toggles an already-loaded pack on or off.

In-game `pull` and `reload` (director-coordinated)

Two staff verbs (world verbs, MinRank = 1 — see Builder Commands) drive content operations across the whole fleet, coordinated by the director:

pull <version> [force] — request a director-coordinated install of a published version. The director is leader-only and single-flights the pull; if a non-leader receives the request it NAKs so the work redelivers to the promoted leader. Because ImportVersion is idempotent by content hash, a duplicate delivery is safe. force (admin-only) downgrades the live-zone prune veto to a report — read that first.
reload [<pack>] — propagate a director-coordinated content hot-reload across the fleet so running shards pick up the new rows without a restart. Content is validated before the broadcast, and the live room-graph is reconciled under a version guard.

Note that telos-seed also publishes a hot-reload invalidation over NATS after it writes rows (optional and non-fatal), so a dev re-seed can refresh a running shard too; on a real fleet the director-coordinated pull/reload is the path that keeps all shards consistent.

Some changes can’t be hot-reloaded cleanly and need a rolling restart — chiefly changes to shared/global defs that entities are already built against. When in doubt, roll the world tier (the zero-drop drain in Running at Scale makes that non-disruptive).

The CI pipeline

telos-pull doubles as the CI toolchain for a self-managed content repository:

telos-pull --emit-manifest computes the content_hash and pack list over a local tree and writes the manifest.yaml — run it to publish a new version.
telos-pull --check dry-runs the import pre-flight (hash verify, constraint checks) without writing anything — wire it as the merge gate so a bad content change can’t land.

A typical flow: author in the content repo → CI runs --check on the PR → on merge, CI runs --emit-manifest and tags a version → an operator (or an automated job) runs telos-pull / in-game pull <version> to install it → reload to make it live.

Content Packs Intro — the built-in packs and the enable model, for a builder audience.
Pack Authoring — the pack format and the three ingestion paths.
Deployment — the fail-closed boot gates and the TELOS_ALLOW_INSECURE posture.

Content Pack Operations

The source-of-truth model

Installing a version: telos-pull → ImportVersion

Switching a seeded deployment to pull