Benchmark Mode

Benchmark Mode answers a question that’s otherwise hard to answer: given this specific prompt, which model or agent configuration actually performs best? It fans the prompt out across rows you define, runs each in isolation, and gives you a structured comparison before you commit to anything.

API mode vs. agentic mode

Benchmark Mode supports two execution styles.

API mode sends the prompt directly to provider APIs — one call per model row — and applies each output to its own per-run worktree. It reports pass@k, cost, token counts, and timing for each row, then surfaces a Pareto verdict ranking the runs across quality and cost.

Agentic mode spawns real coding-agent runs — Claude, Codex, or another configured runner — each in its own worktree. Agentic runs support tier and effort settings so you can benchmark the same agent at different effort levels, not just different models.

Context modes

Every agentic benchmark job records explicit context-mode settings for each run:

Workspace context — the agent sees your full current project context, as it would in a normal queue run.
Clean context — the agent starts with no project-specific context, giving you a baseline read on model capability without accumulated memory.

Alongside context mode, each run records whether Memtrace was enabled or disabled, allowing direct Memtrace-on vs. Memtrace-off comparisons within the same benchmark job.

Review and apply winner

After runs complete, Benchmark Mode enters a review phase where you inspect per-run diffs and the Pareto verdict. When you’ve identified the winning run, Benchmark Mode can turn that run’s reviewed diff into a concrete project change — writing it back to your project — after a final user review step. Nothing applies automatically.

Prose benchmarks with rubrics

Benchmark Mode → New benchmark → Advanced. Not every job is code. A prose benchmark compares plain-text model responses against an ordered rubric you write yourself, and shows criterion-level pass/fail evidence instead of forcing a code-only metric onto a writing or planning task.

Model and provider coverage

The model/provider picker includes xAI’s Grok models as an OpenAI-compatible provider alongside the existing options, and benchmark jobs accept image attachments so a run can include visual context, not just text.

Verdict prompt and corpus contribution

Every verdict keeps a short title but exposes the complete prompt behind it — escaping preserved, scrollable for long content — via View full prompt on the verdict header. Separately, a completed run can be reduced to a metrics-only contribution to the shared benchmark corpus, but only after explicit consent (per-run or set globally under Benchmark Mode → Data); you can see the complete shared/never lists and revoke consent before anything is actually shared.