Incident process
Who is paged, what they do, and what gets published when the machinery fails — as distinct from a wrong value, which is the data-error playbook.
Status: published 2026-08-21 as part of oracle build item O1.7.
Scope: the governed benchmark family — WETGRI, WETFED, WETX, WETFRAG — and the jobs that
publish them. Slates are the tactical class and are out of scope here.
Companions: data-errors.md (a wrong input), restatement.md
(a wrong published value), determination-and-cessation.md (a
licensee's entitlements), cessation.md (the benchmark stops).
This document covers the fourth case those four do not: the machinery failed. A value that is wrong is a data error. A value that never arrived is an incident, and until now the desk had a policy for the first and a habit for the second.
1 · On-call
Administrator: Corbin King. Sole on-call.
That is the honest state and it is stated rather than dressed up: there is one person, there is no
rotation, and there is no escalation path past them. The independent committee seat is vacant
(committee-independence.md), so there is not even a second party to
escalate a judgement call to.
What that means in practice, said plainly: an incident beginning at 03:00 is handled when the
administrator wakes up. The publication window and the staleness tolerance published in
data-licence.md are written to be survivable on those terms — which is why they
are a measured target rather than an SLA, and why no settlement-reference licence may be granted
while this is the shape of the rota.
This section is the first thing to change when a second person exists.
2 · What pages, and what merely reports
From scripts/oracle-monitor.ts, which is the only thing that pages.
| Alarm | Severity | Meaning |
|---|---|---|
benchmark-absent |
PAGE | A close ran and published no row for a benchmark. A hole, not a refusal. |
no-publication |
PAGE | No close at all by its deadline (30h after the settlement day ends). |
signature-invalid |
PAGE | A signature is present and does not verify. Tampering, or an unrecorded key rotation. |
chain-divergence |
PAGE | The on-chain record disagrees with the published ledger. |
backstop-silent |
notice | No evidence the dead-man has run. May be correct; may be switched off. |
monitor-silent |
PAGE | The monitor itself stopped running. |
| delivery failed | PAGE | A channel is configured and the page did not arrive. See §5. |
A refusal is not an alarm and never will be. WETX printed insufficient_data on five
consecutive days in August; that is the coverage floor working. Paging on it would train the
administrator to ignore the pager, which is the failure mode every graded exit code on this desk
exists to prevent.
3 · The first fifteen minutes
Read the alarm. Every one names the benchmark, the date and the rule. None requires inference.
Classify it — and this is the decision that matters:
Refusal A row exists with status: insufficient_dataand a rule. Not an incident. The methodology worked. Note it and stop.Absence No row at all. Incident. §4. Wrong value A row exists and is wrong. Not this document — data-errors.md, thenrestatement.md.Publication failure The value exists locally and did not reach origin/master. §4.3.Do not re-run the close to "fix" it. The close computes from live market state; re-running it hours later reads a different book and prints a different number. That is not a repair, it is a silent restatement, and
settlement-project.ymlalready refuses to do it for the same reason.Do not fill the gap. Not with the prior value, not with an indicative, not with an estimate.
cessation.md: "Filling a hole to keep a chart continuous is a fabricated print, and it is the single most tempting corrosive act available to an index."
4 · Handling, by cause
4.1 · A benchmark was absent from a close that ran
The 2026-08-20 case: the close ran at 21:05:20Z and published WETGRI only; WETFED, WETX and WETFRAG got no row.
- Find the close note (
content/indices/close/<date>.md) and readpublishedCount/absentCount. The desk records this honestly even when nothing alarms. - Read the run's exit code against
cadence.json'srefusalAuthorityforindex-settle: 2 = venue data did not arrive, 3 = unreadable ledger, 4 =--require-sentinelwith no sentinel, 1 = engine fixtures failed. 0 including a refusal. - The day stays absent. It cannot be recovered — see §3.3. Record why in the next close note or in a dated note under this directory.
- If the cause is a code fault, fix it forward. The next close is the remedy.
4.2 · No close at all
Check whether the dead-man took the run: .github/workflows/index-settle.yml fires at 22:00 UTC and
stands down only when berry has delivered a ledger artifact. If neither plane produced a close, the
day is a hole, and the same no-backfill rule applies.
4.3 · The value exists but did not reach master
The 18-of-314 failure. npm run index:reconcile names the orphan. Recover the commit from berry's
reflog if it is still there — three mark tapes were saved that way — and push it. A mark tape
cannot be recomputed, because no venue serves historical order-book depth for a closed market.
4.4 · A signature does not verify
Stop publishing before investigating. An invalid signature is either tampering or a key rotation
nobody recorded, and continuing to publish under a key in question compounds whichever it is.
Verify with npm run index:oracle:verify, which needs no credentials and no network.
5 · When the alarm itself fails
A configured channel that does not deliver a page fails the run, and it is treated as an incident in its own right — because someone asserted this desk is reachable, it is not, and every future page is also going nowhere.
lib/indices/alerting.ts keeps three outcomes apart on purpose: delivered, not-configured (a
gap, printed every run so it cannot become ambient) and failed (an incident). Collapsing the last
two would cry wolf while nothing is wired and hide the day a wired channel breaks.
And the alarm on the alarm: monitor-silent fires if the monitor stops running, and the workflow
carries no continue-on-error and no || true. A monitor that can fail silently is not a monitor.
6 · What gets published
| A refusal | Already published, on the row, with its rule. Nothing further. |
| An absence | Named in the next close note, or a dated note here if it is not obvious from the ledger. An unexplained hole is the thing this section exists to prevent. |
| A restatement | restatement.md, plus direct notice to any settlement-reference licensee within one working day. |
| A publication failure that lost data | Published here with what was lost and whether it was recovered. |
| An incident lasting beyond one close | A dated note under docs/index-governance/, naming the cause where known. |
We publish the incident even when nobody asked. The availability numbers in
data-licence.md are measured and published rather than promised, and an
availability figure without the incidents behind it is a number nobody can check.
7 · Status page
Publication state and known incidents are served at /status — free,
unauthenticated, and reading the same committed ledger everything else reads. It is deliberately not
a dashboard: it answers did today's close publish, for which benchmarks, and if not what is known,
and nothing else.
8 · What this process does not have
Named rather than implied, so a reader is not misled by the existence of a document:
- No rotation and no escalation. One person (§1).
- No paging SLA. There is a channel; there is no commitment about response time, and there will not be one until there is a rota to back it.
- No automated remediation. Every alarm here ends in a human decision, on purpose — the failures that produced this document were mostly automation that "recovered" while reporting success.