Documentation · Releases · Marsyas

Marsyas — Pythia node pool

v.Chronos.Talos.3 — legacy v.H.1.65

Why “Marsyas”

Marsyas was the satyr who picked up the aulos Athena had thrown away and played it so masterfully that he was heard across the land — a ready-made instrument taken up and played through by another.

For this release, “Marsyas” names the shape of the node pool: the hub does not build anything new to run — it lends the chainweb fleet it already runs to an outside consumer. Pythia is that player, reading the fleet as a pool of slots and routing its reads through them, while the hub records every read and keeps each slot healthy.

What changed for the operator

• The hub lends its chainweb fleet to Pythia. A single POST-to-read node listing at /api/pythia/nodes/ surfaces the fleet to Pythia as IP-keyed slots — a bare-IP slot_id per container — with a LIVE-operator join so only genuinely reachable, live containers are ever offered.
• Reads are metered as they arrive. /api/pythia/usage/ records each read Pythia routes through a slot the moment it arrives, with a clear split between a malformed usage report and a report against a slot the hub does not know.
• Slots heal themselves. When the node backing a slot goes unreachable the slot fails over to another reachable node on its own, so a consumer never routes into a dead slot — no operator action required.
• A per-container Eye badge. /hub/nodes marks which of your containers are enrolled in the Pythia pool with a per-container Eye badge.
• The operator-reward scaffold is inert until armed. The Stoicism / PythXP reward for operators whose nodes serve Pythia is wired end-to-end but stays OFF behind an arm-flag — nothing pays out on its own until an Ancient admin arms the pool.

Operator notes

• No reward moves until you arm it. The reward accrual is modelled but inert by design — no Stoicism / PythXP is minted for pool-serving operators before an Ancient admin arms the pool.
• Only live, reachable containers are offered. The LIVE-operator join means an offline or non-live container is never listed to Pythia as a slot.
• Stale reads are forfeited, not accrued late. A usage report older than the retry window is forfeited rather than credited out of time, so the metering stays honest.

Marsyas.1–.5 — PythXP, PythLevels, and the earnings surfaces

Five follow-up patches completed the reward economics the Marsyas.0 scaffold staged:

• PythXP is usage (Marsyas.1). 1 PythXP = 1 keyed read your node served. XP settles from the usage ledger on every tick — armed or not — and the first settle retroactively granted the entire recorded history.
• PythLevels (Marsyas.2). An Ancient-configured curve of brackets maps XP to levels, each level carrying its own Stoicism-per-XP rate. Inert until the curve is set.
• The level mint engine (Marsyas.3). With a curve stored, rewards mint the exact bracket integral over your XP span — retroactively over your whole history on the first armed run, and forward with every new read. XP and Stoicism reconcile one-to-one.
• The surfaces (Marsyas.4). Your own standing lives at Earnings → PythXP (XP, level, progress, per-slot contribution, recent settles); Ancients see the fleet at Pythia → Pythia Node Pool.
• Earnings flow back to Pythia (Marsyas.5).The nodes feed now carries optional per-slot economics (operator XP/level, rewarded requests, Stoicism earned) so Pythia’s Observation Pool can rank the hub’s nodes by what they earn.

Marsyas.27 — the “everything is STALLED” fix

The Nodes registry was showing a red STALLEDtip on nearly every node at once. It was a measurement artifact, not a real chain stall: each node’s cached tip is refreshed by a full-fleet sweep, while the networktip it’s compared against refreshes on a faster, independent cadence. When the sweep fell behind, every node’s stale-but-fine height read as “100+ blocks behind” and lit up red. Two changes remove it:

• The sweep can’t be starved.Each node’s tip probe is now bounded by an 8-second ceiling. Previously a single unreachable host could hang a probe for minutes and, with only a handful of concurrent slots, stretch a 30-second sweep to ~12 minutes — ageing every node’s cached tip past its freshness window. A host that overruns the ceiling is simply marked offline for that cycle and the sweep moves on.
• A stale reading never flags red. A node is no longer shown as lagging when its own tip probe is older than 120 seconds — comparing a minutes-old height to a seconds-old network tip proves nothing. The amber stale badge still tells you the reading is old, and a genuine desync (a fresh probe that truly trails the fleet) still flags red.

Marsyas.28 — the “everything is STALE” fix

Marsyas.27 removed the false red STALLED badge, but the registry then showed an amber STALE tip on every node instead. Same underlying cause, one step further: the full-fleet tip sweep was still taking about four minutes to come back around to each node — far longer than the 90-second freshness window — so the heights were correct but perpetually labelled old. The sweep was being dragged down by long-dead machines:

• Archived nodes are no longer probed.A node that has been archived (offline for over a week, or retired by hand) is already excluded from scoring and the Pythia pool — but the sweep was still SSH-probing it every cycle, burning the 8-second ceiling on each dead box. On this fleet ~18 archived machines (offline ~10 days) dominated every sweep. Skipping them brings a full pass back under its cadence, so live nodes’ heights stay fresh and the amber badge clears.
• Orphaned tip rows are pruned. When a node is deleted its cached tip row was left behind forever. Those dangling rows are now cleaned up on each sweep, keeping the cache in step with the live fleet.
• Pythia gets fed again.The same staleness that turned the badge amber was also dropping otherwise-usable nodes out of the feed the hub serves to Pythia (a stale height reads as “lagging” against the live network tip). With sweeps fast again, the hub reliably advertises its healthy nodes to Pythia.

Marsyas.29 — the last of the “STALE” fix

Marsyas.28 made the tip sweep fast, but the amber STALE badge still lingered because a second background pass — the Pythia eligibilitypoller (which checks each node’s public HTTPS endpoint) — had the exact same blind spot:

• The eligibility poller now skips archived nodes too. It was still SSH-probing every archived box each cycle, and with no per-probe ceiling those dead hosts hung for up to ~20 seconds each — about 21 seconds of every sweep spent on machines that are gone. That congested the worker enough to stretch the whole loop, so the now-fast tip sweep only got to run every ~90–150 seconds instead of every 30 — just long enough to push live nodes back past the freshness window. Skipping archived nodes here lets both sweeps keep their intended 30-second cadence, so the heights stay fresh and the amber badge finally clears.

Marsyas.30 — making it stay fixed as the fleet grows

Skipping archived nodes made each sweep fast, but the amber STALEbadge still flickered because the worker’s overall loop cadence was being dragged out by SSH probes to the handful of live nodes that happen to be offline right now. Several background passes (scoring, provisioning checks, failover) each waited a full 20 seconds to discover such a node was unreachable — and that shared wait stretched as the fleet grew. Two changes make it robust:

• Offline nodes are detected ~3× faster. The SSH connect (handshake) timeout drops from 20 s to 8 s. A healthy node answers in a fraction of a second, so this only shortens how long the hub waits on a dead one — long-running jobs like benchmarks are governed by a separate timeout and are unaffected. Because the cost of probing offline nodes is what grows as more nodes join, this keeps the sweep cadence healthy at scale.
• The badge now means genuinely stale. The freshness window moves from 90 s to 5 minutes, so a node is only marked stale when its data is actually old (the hub or the node is down) — not merely because the sweep is a beat behind. Correct, up-to-date heights no longer get mislabelled.

Marsyas.31 — pool payouts resume (stuck-leg recovery)

Daily pool payouts silently stopped for eight days. The daily job kept firing and reported success but paid nothing, so the miss stayed invisible until noticed. The cause was a single wedged step in the cross-chain fund consolidation that precedes every payout:

• What happened.Consolidating the pool’s balances onto one chain is a two-step cross-chain transfer (a burn on the source chain, then a continuation on the destination). The pool records the burn beforeconfirming it, so it can safely resume after a crash. On 2026-07-23 a burn was recorded but never actually landed on-chain (a lost submission / the worker restarted mid-run by a deploy). No funds moved — the coin stayed put — but the recorded step could never be “continued,” and the consolidation refused to complete, so the payout gate opened each day and then paid nothing.
• The fix — it now self-heals. When a recorded burn is confirmed absent on-chain and older than the 8-hour window after which it can never land, the pool re-does that step cleanly instead of waiting on it forever. This is safe against paying twice: it only re-burns when the coin provably never moved; a burn that didland is resumed exactly as before, never repeated. Payouts resume on the next daily run, and a future one-off hiccup can no longer wedge the pool.

Marsyas.34 — Pythia earnings, checked in once a day

Pythia earnings now have a proper “hot vs flushed” model. Every keyed read your node serves earns a tiny Stoicism reward — dozens of times a day, each a fraction of a cent. Until now every one of those landed straight in your spendable pool and posted its own row in the earnings feed, which buried the meaningful events under a wall of “+0.00” lines. Now:

• A hot bucket. Per-read rewards accumulate in a hot bucket that is not yet in the mintable pool. On /hub/pythxpthe big green “Pythia earnings · B.UNA” is now that hot amount — what you’ve earned but not yet checked in.
• One daily flush.Once a day, at 00:00 UTC, the day’s hot total is moved into your cumulating pool, the hot number resets to zero, and a single “daily flush · checked into pool” row is posted to the earnings feed. The pythxp page now also shows your lifetime Pythia earnings beneath the hot figure.
• A calmer feed. The /hub/earnings“Recent events” stream collapses the old per-read rows into one rollup per day, so warmup, bonus, and benchmark events are legible again.
• Same money, exposed both ways. The login/earnings API now returns both the hot (un-flushed) and lifetime figures. Nothing already in your pool moved — existing balances are treated as already checked in.

Marsyas.35 — earnings API speaks B.UNA

A small follow-up to the earnings model. The login/earnings API already returned your hot and lifetime Pythia earnings as exact Stoicism values; a client had to multiply by a million to show B.UNA. Now the API also returns those figures pre-computed in B.UNA (exact, = Stoicism × 1,000,000), so apps like OuronetUI can display billion-UNA directly without converting. This is additive only — no earnings behaviour changed.

Marsyas.36 — no more “everything is red, then green”

A cosmetic fix to the earnings + nodes pages. A node row shows “Not earning · Offline” when the hub hasn’t successfully ticked it within a staleness window. That window was 3 minutes, but the worker’s full-fleet loop runs about every 2.3 minutes and its SSH probes occasionally time out under load — so a single slow cycle could push a whole batch of healthy nodes past the line and flash them red on page load, recovering moments later. The window is now 5 minutes (matching the tip badge), which absorbs those transient misses. Your earnings were never affected — this only governs when the offline label appears — and the two pages now share one threshold so they can’t disagree.

Marsyas.37 — earnings that tick up live

Earnings can now be streamed in real time. Alongside the existing /api/oidc/earnings endpoint (which a client polls), the hub adds /api/oidc/earnings/stream — a Server-Sent Events stream that stays open and pushes the same earnings payload the moment it changes. A consumer like OuronetUI sees the numbers climb per read, and sees the daily 00:00 UTC hot→pool flush the instant it happens (a dedicated flush event marks it, so the reset can be animated). Same login token and earningsscope as the REST endpoint; it’s advertised in the discovery document so clients can detect it and fall back to polling when it isn’t there. Read-only — nothing about how earnings are calculated changed.

Marsyas.38 — staying logged in without staying vulnerable

Access tokens are deliberately short-lived (5 minutes) so a leaked one expires fast. That’s good for security but means a long-running consumer had to send the operator back through login every few minutes. The hub now supports the standard OAuth refresh-token grant: an app that requested the new offline_access scope gets a long-lived refresh token it can exchange for a fresh access token silently, so the live earnings panel keeps running without interrupting the operator.

The refresh tokens are hardened the way OAuth recommends: opaque and stored only as a hash, rotated on every use, with reuse detection — if a stolen, already-rotated token is replayed, the entire chain is revoked. They last 60 days (sliding), can be revoked via a standard revocation endpoint, and are killed when the operator logs out of the hub. The 5-minute access-token lifetime is unchanged.