Benchmark Mode
Benchmark Mode answers a question that’s otherwise hard to answer: given this specific prompt, which model or agent configuration actually performs best? It fans the prompt out across rows you define, runs each in isolation, and gives you a structured comparison before you commit to anything.
API mode vs. agentic mode
Section titled “API mode vs. agentic mode”Benchmark Mode supports two execution styles.
API mode sends the prompt directly to provider APIs — one call per model row — and applies each output to its own per-run worktree. It reports pass@k, cost, token counts, and timing for each row, then surfaces a Pareto verdict ranking the runs across quality and cost.
Agentic mode spawns real coding-agent runs — Claude, Codex, or another configured runner — each in its own worktree. Agentic runs support tier and effort settings so you can benchmark the same agent at different effort levels, not just different models.
Context modes
Section titled “Context modes”Every agentic benchmark job records explicit context-mode settings for each run:
- Workspace context — the agent sees your full current project context, as it would in a normal queue run.
- Clean context — the agent starts with no project-specific context, giving you a baseline read on model capability without accumulated memory.
Alongside context mode, each run records whether Memtrace was enabled or disabled, allowing direct Memtrace-on vs. Memtrace-off comparisons within the same benchmark job.
Review and apply winner
Section titled “Review and apply winner”After runs complete, Benchmark Mode enters a review phase where you inspect per-run diffs and the Pareto verdict. When you’ve identified the winning run, Benchmark Mode can turn that run’s reviewed diff into a concrete project change — writing it back to your project — after a final user review step. Nothing applies automatically.