Benchmark governance

The loop architecture

The nine scheduled agents that cultivate indices, which one may publish, and the six mechanisms that assure accuracy.

How W.E.T. cultivates indices — what the machines do, what they are forbidden from doing, and how a reader checks any of it.

This document is public on purpose. The cultivation engine is a competitive asset, and the asset is not that the machinery is secret — it is that the reasoning is legible. An index desk whose picks cannot be interrogated is an oracle; one whose every nomination, refusal and weight is published is a method. This page is the method.


1 · What is actually automated

Ten scheduled agents run on a single machine, each with a written brief, a fixed cadence, and a model chosen for what that job needs.

The briefs are the source of truth for this table, not the other way round: where a loop's own loopany/<loop>/README.md states a cadence, that is what this row says. A public page describing a method has to describe the method that runs, and this table did not. It put the cultivation panel at Sun 20:00 and the QA judge on Monday morning; the panel runs 17:00 Sunday and the judge 19:30 Sunday, before the insider preview reads the gate that same night. It also counted nine loops and omitted the manipulation sentinel, which is the one loop standing between a venue print and a checksum-chained close. Corrected below.

Loop Cadence (ET) Model Publishes?
Emergent event scout 07:00, 15:00 daily Opus 5 No — proposes
Index maintenance 07:30 daily Sonnet 5 No — proposes
WET desk health 08:00 daily none (deterministic) No — alerts
Category desk 13:00 weekdays Opus 5 No — proposes
Manipulation sentinel daily, after index:mark, before the WET Close none (deterministic) No — quarantines
Daily performance learning after the WET Close, daily Sonnet 5 No — records
Index cultivation panel Sun 17:00 Opus 5 No — ranks
QA judge Sun 19:30 Opus 5 No — gates
Index of the Week Mon 09:30 Sonnet 5 Yes — the only publisher
Weekly recap + blog draft Mon, shortly after 09:30 Opus 5 Draft only

Exactly one loop publishes. Everything else proposes, records, or refuses. That is the single most important property of this design: a system where several agents can publish is a system where nobody can say afterwards which one published what.

The week label is one clock, and a mismatch refuses. The panel, the judge and the release all belong to the same weekly ritual, and it straddles a date boundary: the judge runs Sunday evening ET — already Monday UTC for half the year — for a release the next morning. So the label is defined once, in America/New_York, as the Monday of the release the artifact belongs to, and a Sunday-evening run belongs to tomorrow's release. scripts/index-qa.ts and scripts/index-of-the-week.ts each carry the same cadence table as a fixture, so editing either implementation fails that file's own self-check. If the gate's label and the release's label disagree, the release publishes nothing — a gate written for another week reads as clearance in every artifact that follows it, and there is no way to tell afterwards which of its verdicts still applied.

The sentinel's ordering is not a preference. It runs after the day's mark tape and before the close, and the reason is the same reason a close is quotable at all: the WET Close is checksum-chained, so a quarantine that arrives after it is a note about a number already published. When the sentinel has not run, the settlement does not invent a clean board — it writes quarantinedCount: null and says in the row that no input was assessed. The ordering, and the command that checks it against the published ledger rather than against the cron script, are in docs/BERRY_CRON.md in the repository.


2 · The weekly ritual, end to end

flowchart TD
    subgraph venues["Venues"]
        K["Kalshi"]
        P["Polymarket"]
        G["Gemini"]
    end

    K --> POOL["Nomination pool<br/><i>lib/indices/pool.ts</i>"]
    P --> POOL
    G --> POOL

    POOL --> SCREEN["Trackability screen<br/><i>positions.ts</i><br/>unmarkable members removed<br/>BEFORE assembly"]
    SCREEN --> ASM["Assembler<br/><i>gate.ts</i><br/>8–20 members, capped at<br/>what the publisher can ship"]

    ASM --> PANEL

    subgraph PANEL["The specialist panel — six lenses, fixed published weights"]
        direction LR
        L1["catalyst<br/>0.20"]
        L2["divergence<br/>0.25"]
        L3["coherence<br/>0.20"]
        L4["liquidity<br/>0.15"]
        L5["novelty<br/>0.15"]
        L6["calibration<br/>0.05"]
    end

    PANEL --> GATE["Mixture gate<br/>weight x confidence,<br/>renormalised over<br/>reporting lenses only"]
    GATE --> ORIENT["Orientation<br/><i>orientation.ts</i><br/>side per member by<br/>thesis concordance"]
    ORIENT --> JUDGE["Judge<br/><i>judge.ts</i><br/>refuses + records<br/>which rule fired"]

    JUDGE -->|admitted| FOLD["Intra-run novelty<br/><i>gate.ts</i><br/>each candidate re-scored<br/>against the siblings<br/>ranked ABOVE it"]
    JUDGE -->|refused| RET["Retired<br/><i>published too</i>"]
    FOLD --> CAND["Ranked candidates<br/>+ JSON sidecar"]

    CAND --> QA["QA judge<br/>Sun 19:30<br/>accept / note / DECLINE"]
    QA --> GATEFILE["qa-gate.json<br/><i>stamped with the release week</i>"]

    GATEFILE --> REL["Index of the Week<br/>Mon 09:30"]
    GATEFILE -.->|"label ≠ release week"| NONE
    CAND --> REL
    VETO["Owner veto<br/><i>veto: true</i> or <i>VETOED</i>"] --> REL

    REL -->|"rank #1 unless<br/>vetoed or declined"| PUB["Published SLATE"]
    REL -->|nothing clears| NONE["Nothing published<br/><i>a blank week is a signal</i>"]

    PUB --> LOCK["Inception lock<br/><i>slate-lifecycle.ts --open</i><br/>the ONLY caller"]
    LOCK --> RECAP["Recap next Monday<br/>losses as prominent as wins"]
    RECAP --> CAL["Calibration record"]
    CAL -.->|"when n is large enough"| PROPOSE["Proposes a weight change<br/><i>never applies one</i>"]

3 · The mixture-of-experts layer, stated precisely

It is a mixture-of-experts pattern: several specialist scorers with narrow published lenses, combined by a gate that weights them, then filtered by a judge.

It is not a neural mixture-of-experts, and nothing in it is trained. The gate weights are fixed constants, published on every candidate, and versioned (GATE_VERSION) so any past decision can be re-derived from the same inputs. GateOutcome.adaptive is typed as the literal false so it cannot drift true without a deliberate code change.

We say this plainly because the alternative is an overclaim with a long half-life. A company whose public claim is "our numbers can be checked" cannot afford a description of its own machinery that cannot be.

What each lens measures

Lens Weight Measures Abstains when
catalyst 0.20 how tightly members resolve in time under 80% of members carry a date
divergence 0.25 cross-venue price gap on the same structural claim no member parses into a claim
coherence 0.20 whether the members share a nameable theme
liquidity 0.15 whether the basket is actually markable never (an abstention would be free)
novelty 0.15 distance from all 33 published indices and from the higher-ranked candidates in the same run the roster is empty
calibration 0.05 how well-calibrated this market family has been the sample is below the floor

Novelty is measured twice, and the second measurement is the newer one. Scored against the live roster alone, two candidates cut on adjacent axes out of the same board in the same run cannot see each other, and both report as original — the run on 2026-08-05 ranked Party — Elections, inside six months third, holding 43% of the composition of the Seat — Elections basket ranked first, and House — Elections fifth at 67% of Party. Whichever ships, the runners-up sit in the candidate set as distinct views when they are the same view. So each candidate is now also scored against the siblings ranked above it in its own run, on the larger of two overlaps: market refs, and — valid only within a run, where both baskets come from one pool — venue event keys, which catches two strikes of one question that share no refs at all. The rule is min(roster distance, sibling distance); no new threshold is introduced, and the demoted candidate's own artifact names the sibling and the percentage. Rank #1 is never adjusted, and a candidate is never marked down against one that finishes below it.

An abstention is not a score of zero. The gate renormalises the abstaining lens away and the candidate is scored on the lenses that reported — and the candidate artifact says which those were. A candidate carried by a single reporting lens is a single scorer wearing a panel's clothes, and the QA judge blocks it.

Divergence is the lens with genuinely proprietary input. identity.ts matches the structured claim across venues — subject, relation, strike, settlement day — where title similarity found zero cross-venue pairs in ~8,000 contracts. That is why the same question listed three different ways on three venues can be priced as one event.


4 · How accuracy is assured

Six mechanisms, each of which exists because the alternative is a specific, nameable failure.

4.1 The engine refuses rather than approximates

Every lens can abstain, every judge rule can refuse, and every refusal records which rule fired. Refused candidates are published alongside the accepted one — the rejections are the evidence the method did work, and a ranked list nobody can interrogate is an oracle.

4.2 Nothing is fabricated — including causes

A missing number is published as missing. This extends with full force to prose: a sentence naming a cause the engine cannot evidence is treated as the same defect as a fabricated price. The composers run a lexical guard that throws on causal vocabulary, and there is deliberately no field a human can type an unchecked reason into.

Two carve-outs, both narrow and both stated in code:

  • quoted() — a venue's own market title is not our claim. A market titled "Will Bitcoin be above $60,000 on August 7?" is not a profit-and-loss figure, and letting a venue's wording fail our lint would have barred the desk from indexing the crypto ladders entirely.
  • explains() — prose describing this engine's own design is exempt from the causal rule only. "The spread screen is deferred to the lock because the pool records no two-sided quote" is a statement about our implementation. It remains fully subject to the wagering, currency and advice rules, because those are our words.

4.3 No rate over a sample that cannot carry one

scorecard.ts refuses a rate below n=30 per bucket. recap.ts refuses below 12 weeks. The QA judge's own calibration follows the same floor and states the count in words until it clears.

"We are right 100% of the time" over n=1 is arithmetically true and completely dishonest.

4.4 The record is never filtered

trackRecord() publishes every week it was handed, unscored ones included, and asserts the published count against the input count. Survivorship is the failure mode that makes every fabricated record look superb, and it arrives as a .filter() nobody remembers writing — which is exactly how it arrived here once, and was caught.

Losses get the identical template, placement and colour ramp as wins. headline() has no branch on the sign of the score, because a highlight reel is not built by deciding to hide the losses; it is built by giving them a shorter sentence.

4.5 The machines propose; the gates dispose

Three independent stops sit between a proposal and a published index:

  1. the judge (deterministic rules — trackability, novelty, coherence, orientation)
  2. the QA judge (a reviewer that must name the rule behind any decline)
  3. the owner veto (veto: true in front-matter, or a VETOED line in the body)

An absent QA verdict is logged as unreviewed, not accepted. And when nothing clears, nothing publishes — a blank week is a real signal about the board, and substituting a candidate the methodology did not choose would be a release the methodology did not make.

4.6 Predictions are recorded before outcomes

The QA judge records its confidence before the week runs; scoring only ever reads. A judgement written after the fact is not a prediction, and a track record built from those is worthless in precisely the way that is hardest to detect later.

Two things are scored separately and never conflated: directional (did the index move as the thesis implied, Brier against a constant-50% forecaster) and attention (did it get read). Where no engagement signal exists, attention records unavailable with the reason — an "interest score" synthesised from whatever else is on disk would be a fabricated number with a decimal point.

Nothing in the loop layer edits GATE_WEIGHTS. Evidence accumulates; when the sample can carry a change, a change is proposed with its impact stated. Applying one is a methodology change under the methodology-change policy.

(That link used to point at index-governance/methodology-change.md — a repo-relative path, which this page renders verbatim, so on the web it resolved to /governance/index-governance/… and 404ed. A governance page whose link to the governing policy is dead is the same defect class as a cadence table that describes a cadence nobody runs.)


5 · Where the class line sits

flowchart LR
    subgraph B["BENCHMARKS — governed"]
        B1["WETGRI · WETFED · WETX · WETFRAG"]
        B2["versioned methodology<br/>14-day consultation<br/>checksum-chained ledger<br/>restatement policy<br/>daily WET Close"]
    end
    subgraph S["SLATES — agile"]
        S1["~29 curated views<br/>+ the weekly Index of the Week"]
        S2["horizon-tagged, finite-lived<br/>light governance<br/>equal-notional, size-blind score"]
    end
    CULT["The cultivation engine"] -->|"produces ONLY"| S
    CULT -.->|"never"| B

The weekly index is always a Slate. This is forced, not preferred: a benchmark carries versioned methodology and a consultation window, so one cannot be minted weekly without destroying the property that makes the benchmark class licensable. IndexCandidate.indexClass is the literal 'slate' — no other value is representable in the type.


6 · What this system does not do

Stated because a transparency document that only lists strengths is marketing.

  • It does not learn yet. The gate weights are fixed. The calibration lens abstains on nearly every real basket because the graded-outcome set is thin, and the QA judge has no measured accuracy at all. Adaptation is gated on data that does not exist.
  • It cannot orient most baskets. Side selection works from thesis concordance, but only two framing detectors exist (severity, price-threshold). Elections, culture and sports are largely unframed, so the majority of legs are held as the venue framed them — recorded as undetermined, not disguised as a choice.
  • It never claims a side improves returns. No code path consults price to pick one. That claim cannot be evidenced here, and it would be advice.
  • Its declines can never be scored. A declined index does not run, so the QA judge's refusals are permanently unfalsifiable. That is a real limit on how well its accuracy can ever be known.
  • Sports is tertiary. It gets a specialist day because its structure differs, not because it is the desk's centre of gravity.

7 · Checking any of it yourself

Claim Where to check
The gate weights published on every candidate artifact, and in gate.ts
Why a candidate won its markdown: every lens, score, confidence, effective weight, reason
Why a candidate lost the retired artifacts, with the rule that refused it
Why a candidate was demoted for resembling another its novelty verdict names the sibling, the percentage, and both overlap measures
Why a leg is YES or NO the sidecar's orientationRule and orientationReason
What the QA judge decided loopany/index-qa/qa-gate.json, stamped with the release week it gates
That a WET Close was screened before it printed the ledger row's sentinelVersionnull means nobody looked
What it predicted beforehand loopany/index-qa/predictions.json, stamped before the week
Whether a benchmark reproduces npm run index:verify — recomputes each published level from its own stored constituents
The full engine npm run cultivation:verify, index:qa -- --self-check

Benchmark and slate readings are derived from prediction-market prices. Not a forecast, not advice, and never a performance claim. Slate figures are hypothetical, equal-notional and size-blind — percentages only, before fees, spread and slippage.

The rest of the set
Methodology change & consultation
What counts as a material change, the 14-day consultation window, and why changes are prospective.
Data licence & rights
What the free tier grants, what attribution it requires, and what commercial use needs.
The confirm gate
What human confirmation covers — merges only — and how to read the provisional and single-venue counts.
Restatement policy
The three narrow conditions under which a published value may be corrected — and the many under which it may not.
Cessation & fallback
What happens when a benchmark cannot be computed, and why the history stays published forever.
Conflicts & the conflict wall
The wall between index operations and the trading circuit, plus standing disclosures.
Data-error playbook
Detection, triage, quarantine — and the commitment to disclose errors nobody noticed.
Complaints & challenges
How to challenge a value, a rule, or a constituent. No account required.
Settlement-reference licensing
What licensing a benchmark to settle against requires — and why publication alone does not grant it. Reading a value is free; referencing it in something that settles is not the same act.
Determination, disruption & cessation
Written for contracts rather than charts: who determines a final value, the disruption fallback ladder, and why a refusal is not a cessation.
The independent committee seat
The seven-part independence test, published before a candidate exists; the power to block rather than comment; and what the administrator may do while the seat is vacant.
Incident process
Who is paged, what they do, and what gets published when the machinery fails — as distinct from a wrong value, which is the data-error playbook.
Finding — the WETX coverage breach, August 2026
Why WETX refused for five consecutive days, the change that caused it, and what is still not established. Published because an availability figure without the incidents behind it is a number nobody can check.