No unversioned data
Every run pins a content-addressed dataset. A missing or mismatched hash is an error, not a warning.
dataset_hash required
One evaluation contract · any predictor
Proof,
not vibes.
ASSAY compares version-pinned model runs, measures uncertainty, names every changed sample, and gives CI a verdict it can defend.
Keyless replay · standard-library Python · deterministic fixture
Specimen A-0012
MERGE BLOCKED
Baseline1.000
Candidate0.880
Δ / 95% CI−.120[−.184, −.056]
12
specific regressions retained for review
BLOCKED100paired samples
12seeded regressions
2content hashes
$0keyless replay cost
01The problem
Aggregate green is not evidence
A model can improve on average
and still break the case that matters.
Model changes are probabilistic, scorer definitions drift, and test datasets quietly mutate. A single score hides all three. Reviewers need the exact before/after specimens, the uncertainty around the change, and proof that both runs used the same measuring instrument.
What a useful harness refuses to do
Three hard stops.
No cross-scorer diffs
A score only means something under a stable contract. Change the scorer and ASSAY refuses the comparison.
scorer hash mismatch
No aggregate-only gates
The verdict retains the paired examples that caused it, so a reviewer can investigate before acting.
12 samples attached
02The thesis
Different modalities · same experimental object
An LLM answer and a world-model rollout are both predictions of what comes next.
Contextxprompt / world state
+
Actionainstruction / control
→
Predictionŷanswer / future state
↔
Referenceyexpected / observed future
→
Scoresversioned contract
Language model
“Which tool should the agent call?”
- Context
- User request + available tools
- Action
- Prompt template
- Prediction
search- Reference
search
World model
“What state follows this control?”
- Context
- Observed trajectory
- Action
- Control sequence
- Prediction
- Generated future state
- Reference
- Held-out trajectory
The modality changes. The infrastructure—version, run, score, compare, gate—does not.
03The evidence
Bundled regression fixture · reproducible now
Twelve failures.
All twelve found.
The current repository proves its core mechanics on a deterministic 100-sample fixture. The candidate deliberately flips the first twelve tool selections. ASSAY finds exactly those twelve, adds zero false positives, computes the paired interval, and blocks.
Paired accuracy
raw JSON ↗
Baseline
1.000
Candidate
0.880
−.20−.100
−.120
95% CI [−0.184, −0.056] · paired normal approximation
100-sample plate12 regressed
held regressed
Detection recordfixture v1
100%seeded regression detection
12/12detected
0false positives
This is a software regression test, not evidence of performance on naturally occurring failures.
Same machinery · different predictor
Four world-model scorer contracts run on the CPU fixture.
The reference stack separates compounding drift, action grounding, memory through occlusion, and task utility. These are contract tests, not public-checkpoint measurements.
Open reproducible world-model result →.4.20
H0H1H2H3H4
wm/drift · error by horizon
04The people
One artifact · four decisions
Built for the whole review chain.
Engineer shipping a change
Runs the suite before a pull request, sees whether the candidate regressed, and opens the exact changed samples.
Success → an actionable list, not a score mysteryReviewer deciding merge
Checks the confidence interval, tolerance, hashes, and specimen diffs before accepting the automated gate.
Success → evidence survives skeptical reviewEval author maintaining quality
Versions the scorer, validates deterministic behavior, and treats a scorer change as a new instrument—not a comparable result.
Success → measurement drift cannot masquerade as model driftGovernance and infrastructure
Exports a compact chain of custody: run IDs, hashes, method, uncertainty, verdict, changed samples, and documented limits.
Success → a repeatable go / no-go record05The method
From immutable inputs to a reviewable decision
Five operations. No hidden leap.
-
01
Pin
Hash the dataset and scorer contract. Refuse incomplete provenance.
-
02
Replay
Run baseline and candidate against the same sample IDs and references.
-
03
Score
Apply one versioned scorer to every paired prediction.
-
04
Infer
Compute the paired delta, interval, and multiple-comparison correction.
-
05
Gate
Compare the interval to the declared tolerance and retain every changed specimen.
Small enough to inspect
Dependency-free reference architecture
The shipped P0 implementation is standard-library Python with a SQLite-first run store, content-addressed contracts, paired comparison engine, structured JSON, a local HTTP API, and a packaged operator UI. Both CLI and API invoke the same working comparison service.
Read the source on GitHub ↗01Datasethash + samples
→
02Runbaseline / candidate
→
03Scorerversion contract
→
04CompareCI + sample diff
→
05Gateexit + evidence
06The operating path
Replay in public · operate locally
One contract.
Two honest surfaces.
The public bench is a deterministic, zero-keyway to understand the interaction. Install ASSAY to ingest your own already-scored runs, execute the real comparison engine, and retain evidence in SQLite.
GitHub Pages replay
Embedded fixture data, browser-only controls, and no persistence. Every replay state is labeled; it never claims to contact a model or the installed service.
Explore the fixture →Local evaluation service
Verified run ingestion, immutable contracts, durable comparisons, health/readiness probes, a same-origin operator UI, and structured API errors.
assay serve --seed-demo --db .assay/runs.sqlite
Open the operations guide ↗
01POST runsverify hashes
→
02POST comparisonrun paired engine
→
03GET evidencepersist + inspect
→
04Gateexit 0 / 1 / 2
07Honest limits
Trust requires a boundary
What this build does
and does not prove.
The public interface labels synthetic evidence as synthetic. The installed service really ingests, compares, and persists user-supplied scored runs, but it does not execute providers. Measured claims remain limited to checked-in, deterministic artifacts.
- 100-sample keyless replay and blocking exit code
- Content-addressed datasets and scorers
- Hard refusal of cross-scorer comparisons
- Paired confidence interval and correction hook
- Per-sample regressions and structured report
- Four CPU-scale world-model scorer contracts
- Local API, SQLite evidence, probes, and operator UI
- No live Gemini, Groq, or Ollama provider runner
- No public world-model checkpoint measurement
- No external benchmark adapter or GPU notebook
- No public safe-horizon result
- No provider cost, latency, or nondeterminism claim
- No research-grade perceptual metric claim
About ASSAY
A CI-native evaluation harness for predictors.
ASSAY exists to make model evaluation behave more like serious engineering: fixed inputs, versioned instruments, uncertainty in the open, and decisions traceable to specimens. It does not invent another benchmark. It gives existing benchmarks one reproducible execution and gating contract.
Ready specimen / A-0012
Don’t read another aggregate.
Run the fixture, change the tolerance, step the protocol, and inspect the twelve samples behind the block.
Enter the evaluation bench →