Skip to content
Orbit Docs

Run a Benchmark Job

Benchmark Mode runs the same prompt against multiple model configurations simultaneously, measures pass rate, cost, and timing, and helps you pick the best option before committing to it.

API-mode benchmark

Section titled “API-mode benchmark”

Use API-mode when you want to compare pure text-generation output — the models answer without running a coding agent in a worktree.

  1. Open Benchmark Mode from the center-mode panel selector.
  2. Click New Job and choose API mode.
  3. Add model rows — each row is one configuration to compare. For each row, set the provider, model, and optional effort level.
  4. Write the prompt body and set any shared verification criteria.
  5. Click Run. Orbit fans out one request per row in parallel, each in its own per-run worktree.
  6. Monitor progress in the live run state view. You can cancel individual rows or the whole job from here.
  7. When all rows finish, open the Results tab. Orbit shows a Pareto verdict that weighs pass@k, cost, tokens, and timing.

Agentic-mode benchmark

Section titled “Agentic-mode benchmark”

Use agentic mode when you want to compare full coding-agent runs — Claude Code or Codex executing the prompt inside a worktree and making real commits.

  1. Create a new job and choose Agentic mode.
  2. Add agent rows. Each row specifies:
    • Runner: Claude or Codex
    • Tier / Effort: the complexity tier and effort level to pass to that runner
    • Context mode: Workspace (agent sees your real working tree) or Clean (fresh clone)
    • Memtrace: On or Off — lets you compare how much Memtrace context injection affects output quality
  3. Queue and run the job. Each row gets its own isolated worktree so agents cannot interfere with each other.
  4. Review results in the Results tab. Pass/fail, commit diffs, and timing appear per row.

Applying the winner

Section titled “Applying the winner”

After the review phase completes:

  1. Go to the Results tab and identify the winning run.
  2. Click Apply Winner on that row.
  3. Orbit presents the winning run’s diff for your review.
  4. Confirm — Orbit turns the diff into a concrete project change.

Using MCP to drive a benchmark

Section titled “Using MCP to drive a benchmark”

You can create and manage benchmark jobs from any MCP client (Claude.ai, Claude Code, etc.):

  • orbit_benchmark_create — create and auto-launch a new job
  • orbit_benchmark_results — list jobs or read a specific job’s results
  • orbit_benchmark_review — trigger the review phase on a completed job

These tools follow the same REST lifecycle that powers the Benchmark Mode UI.

Section titled “Related”