bonsai eval runs source-controlled fixture tasks through the normal agent
and tool loop, grades the resulting worktree, and writes a JSON report. It is
the release-gating correctness and efficiency instrument: mock mode is fully
deterministic and needs no credentials, so it runs in CI on every change.
This page is the orientation; eval/README.md is the
authoritative reference for the suite schema, graders, budgets, baselines, and
adapters — keep that file the single source of truth when they disagree.
Run cargo run --locked -- eval --suite eval/suites/language_acceptance.toml --qualification --mode mock --out target/qualification
for the eight language/LSP cells. This executes real native tests and scripted
independent review, but uses fake available LSP servers and makes no paid calls.
The qualification workflow uploads only qualification.json; raw databases,
worktrees and private stage evidence must not be published.
Live qualification and baseline freezing remain disabled pending completion of provenance, promotion and strict comparison. See the qualification status and remaining #13 work. Deterministic mock success is not a live model release qualification.
bonsai eval # default: mock mode, default suite
bonsai eval --mode mock --json # print report JSON to stdout
bonsai eval --suite eval/suites/m2_10.toml --task t01_readme_agentic
bonsai eval --mode live --provider anthropic --out target/eval-live
--provider, then BONSAI_PROVIDER,
then the saved current provider (the same selection order as
headless mode). --model and --effort are live-only
overrides, recorded in the report.--fail-on-task-failure makes task, grader, and budget failures exit
nonzero — use it in CI and release gates. Harness, suite, CLI, fixture, or
auth/config errors always exit nonzero regardless.--baseline <path> compares the run against a versioned baseline profile
(exact suite/provider/model/effort identity) and fails below the
correctness floor or above any configured efficiency tolerance.Each run writes target/eval/<run-id>/report.json plus a CI-friendly
summary.md, and keeps an isolated storage/bonsai.db for forensics —
eval/README.md includes ready-made SQL for cache and read-efficiency
regressions.
Suites are TOML files: an id, a seed, optional suite-wide repetitions, and
[[tasks]] entries with a fixture directory, a prompt, an optional mock script,
and graders. Beyond the simple
read/write shorthand, tasks can exercise release scenarios: raw tool-call
batches with malformed arguments, deterministic cancellation and resumption,
truncated provider responses, forced compaction with a mock context window,
prompt-cache simulation with forbidden request substrings, and
shared-workspace tasks that run a second real agent session against the same
worktree to verify the workspace lock and change attribution. Queued human
follow-ups exercise additive, superseding, and status-request steering through
the same in-flight queue as the TUI.
Five grader types verify outcomes:
| Grader | Checks |
|---|---|
test-pass |
A trusted command exits 0 in the copied worktree within timeout_secs |
file-state |
A relative path’s existence/absence, required/forbidden text, or exact equality with an expected file |
assertion |
Required/forbidden text in the final assistant output |
changed-files |
Exact, required, or forbidden paths observed changing during the agent run |
tool-effects |
Required or forbidden successful tool-effect classes |
Suite-level [budgets] (overridable per task) bound logical turns, provider
attempts, duration, prompt/completion/uncached tokens, cost, per-turn
completion size, missing usage rows, re-reads of unchanged paths, and
post-warmup cache percentage.
eval/suites/release_gating.toml — transport, cancellation, batching,
permissions, compaction, and shared-workspace safety.eval/suites/language_acceptance.toml — dependency-free Rust, TypeScript,
Python, and Go repositories whose initial implementations fail their native
tests; the agent must inspect, fix, and pass a clean re-run.eval/suites/intent_continuity.toml — deterministic queued-steering,
pressure-compaction, and episode-recall continuity checks.eval/suites/intent_authority.toml — repeated live-model checks across full
and SMOL explanation, review, verification, monitoring, implementation, and
established-pattern engineering tasks. Run it on two model families for a
release qualification.eval/suites/task_todo.toml and eval/suites/read_efficiency.toml — the
end-to-end efficiency benchmarks. Run both on Codex and at least one
non-Codex transport before changing read reuse, cache shaping, retry, or
batching behavior.Baseline files (eval/baselines/*.toml, schema_version = 1) pin a
suite/provider/model/effort profile with a score_percent correctness floor
and optional efficiency tolerances (token growth, cost growth, duration
growth, cache-reuse drop, repair-turn increase). Omitted tolerances stay
diagnostic; configured gates fail closed when a measurement is missing.
bonsai eval adapter run executes the normal structured headless surface
inside a workspace prepared by an external harness (SWE-bench, Harbor,
Terminal-Bench 2), with every policy and budget explicit in a versioned JSON
request. Adapters never install harnesses or download datasets, and benchmark
content enters bonsai as untrusted data, not instructions. Pinned upstream
commits and the result schemas live in
eval/README.md.
eval/suites/fixtures/.[[tasks]] entry with a unique id and prompt.cargo run --locked -- eval --mode mock --task <id> --json.