Measure with bench #
On this page
maple-harness bench answers one question on your own work: does a setting change what a task costs, and does the
result stay as good? It replays the same tasks under different settings ("arms"), several times each, in fresh
directories, and compares the arms task by task with the uncertainty shown.
It is an advanced command. It runs the real agent, so a run takes real time and, against a metered provider, real money. It never spends anything unless you confirm, and it stops at the budget you set.
In plain words #
- What it is. A command that runs the same tasks several times with and without a setting, and compares what they cost and whether the result stayed as good.
- What to do. Try it with
--dry-runfirst (it spends nothing), then set a budget and run it for real. There is no setup page step.
Try it without spending #
maple-harness bench init-demo ./demo # writes two tiny tasks you can read and edit
maple-harness bench run --suite demo --arms baseline,lsp-off --proxy 127.0.0.1:11535
With no --yes the command prints the plan (how many runs, in what arms) and an estimate of the cost from earlier
runs of the same suite, or says the cost is unknown. Nothing is run and nothing is spent.
Run it for real #
maple-harness bench run --suite ./mytasks --arms baseline,auto-model \
--proxy 127.0.0.1:11535 --model gpt-5-mini --k 3 --budget-ai-units 40 --yes
--k 3runs every task three times under every arm. Three is the least that says anything about variation.--budget-ai-units(or--budget-tokens) is required with--yes. The bench stops before the run that would pass it, judged by the average cost so far. One run that costs far more than the average can still pass the cap, so set it with room.- The order of the runs is shuffled with a seed that is recorded, so an arm is not always first and time of day does not favour one setting.
- The proxy must already be running: a bench never starts one.
- Press Ctrl-C to stop. What was done so far is recorded and can be reported, and the report says it is partial.
maple-harness bench arms lists the arms that ship with it. Some change the harness (for example turning language
server feedback off, or letting the agent choose the model). Some change the proxy (trimming old tool output); for those
run a second proxy with that setting and pass its address with --proxy-for trim-on=127.0.0.1:12345.
Write your own tasks #
A suite is a folder:
mytasks/
suite.yaml optional: a name, and arms of your own
tasks/
fix-parser/
task.yaml
files/ copied into a fresh directory before every run
task.yaml:
prompt: >-
The tests in parser_test.go fail. Fix parser.go so they pass. Do not change the tests.
verify: go test ./... # exit status 0 means the result is acceptable
timeout_seconds: 600 # optional
expect_files: [parser.go] # optional: must exist afterwards
Instead of files/ a task can name a repository and a commit to check out (repo: and commit:). Source files that
a build would otherwise pick up as part of its own project can carry a .tmpl suffix, which is removed on copy. A task
without verify has no check, and its quality is then only "did it finish".
An arm of your own goes in suite.yaml:
arms:
- name: no-diagnostics
description: language server feedback off
config: {harness: {lsp: off}}
config is merged into the harness's config.yaml for that arm.
Compare another tool on the same model #
An arm can be another program instead of a change to this one. It gets the same tasks, in the same fresh directories, with the same check, so the comparison is like for like. The example below runs GitHub Copilot CLI against a model you host yourself, next to Maple Harness on the same model.
arms:
- name: maple
description: this program, through a proxy in front of the same model
config:
harness: {default_model: "${env:MAPLE_BENCH_LOCAL_MODEL}"}
- name: copilot-cli
kind: command
command: ["copilot", "-p", "{prompt}", "--allow-all-tools", "--allow-all-paths", "-s"]
version_command: ["copilot", "--version"]
env:
COPILOT_PROVIDER_BASE_URL: "${env:MAPLE_BENCH_LOCAL_URL}"
COPILOT_PROVIDER_TYPE: openai
COPILOT_MODEL: "${env:MAPLE_BENCH_LOCAL_MODEL}"
COPILOT_PROVIDER_MAX_PROMPT_TOKENS: "222822" # window, minus the longest reply, minus a margin
COPILOT_PROVIDER_MAX_OUTPUT_TOKENS: "32768"
- No shell. The command is a list of arguments;
{prompt}is replaced by the task (or useprompt_via: stdin).${env:NAME}is replaced from your environment, and a name that is not set stops the bench before it spends anything, naming the variable. - A fresh
HOMEfor every run, so the tool cannot read your settings or keep state between runs. It also gets the bench's own temporary configuration folders. - Success is the task's check and nothing else. The tool's exit status is recorded but decides nothing: a tool that exits cleanly without doing the work fails, and one that exits with an error after doing the work passes.
- Give both arms the same limits. In the example the model window is 262,144 tokens and a reply may be 32,768, so the
prompt limit for the other tool is 222,822 (262,144 - 32,768 - a margin). On the Maple side, set the same two numbers
on the model in the proxy's provider entry (
context_window: 262144,max_output_tokens: 32768). A difference in limits, or in sampling, is a difference in results. - Settings from a program. If a small program of yours prints the settings for the tool,
env_fromreads them once:- name: copilot-cli kind: command command: ["copilot", "-p", "{prompt}", "--allow-all-tools", "--allow-all-paths", "-s"] env_from: ["rtx", "env", "copilot"] # prints lines such as export COPILOT_MODEL=...Only names starting with
COPILOT_,OPENAI_,LLM_orANTHROPIC_are taken (add more withenv_prefixes:); everything else the program prints is thrown away unread. Entries inenv:win over imported ones. The values are used and never written down: the results record the imported names and a short hash of the values, and names that contain KEY, TOKEN or SECRET are marked as hidden on screen. For a Maple arm, the imported names can be used in itsconfig:the same way ("${env:MAPLE_BENCH_LOCAL_MODEL}"). - Sampling. The temperature and related settings each arm ran with are recorded, and the plan warns when arms differ.
Counting on the model server #
Measured in AI units or the tool's own tokens, two programs cannot be compared, because one may report nothing. If your model server is vLLM, point the bench at its metrics and it reads the server's own counters around every run:
maple-harness bench run --suite mytasks --arms maple,copilot-cli --k 3 \
--server-metrics "$MAPLE_BENCH_LOCAL_URL" --budget-minutes 120 --yes
Before each run it checks that the server is idle (nothing running or waiting). If it is not idle after the wait, the run is marked contaminated: it stays in the raw files, and is left out of the statistics. Per run it records the change in prompt tokens, how many were computed and how many were already in the server's cache, generated tokens and requests. The report shows these as server-counted (local model) and never adds them to AI units. Nothing else may use the server while the bench runs.
A check for a person, too #
Tests can pass on something that does not work. A task may add verify_browser: check.py (a script in the task folder;
.js runs with node, .py with python, .sh with sh). It runs after the tests, gets the working directory and the
machine's numeric network address in MAPLE_BENCH_LAN_IP, and is typically a Playwright script that opens the app the way
a person would. The report gives two pass rates, one for the tests and one for the browser, and does not blend them.
Read the results #
maple-harness bench report results/mytasks/20260927T120000Z
maple-harness bench report results/mytasks/20260927T120000Z --json report.json
The report shows, for each arm, the median with a 95% interval; then each arm against the baseline, task by task: the change (negative means it used less), its interval, a p-value, and how many tasks it won, lost and tied; then the pass rate, so that a cheaper arm that fails more is visible; then the total each arm cost including failed runs and retries. With few tasks the intervals are wide. A difference inside its interval is not a finding.
What is recorded #
Each run records tokens, cache reads and writes, AI units, requests, retries, tool calls, wall time, files changed,
and whether the check passed. No prompt text, file contents or credentials are recorded, and paths are removed from
error messages unless you pass --anonymize=false.
Related: how to read the cost of a single session is in What /cost tells you; the checklist for comparing settings on your own tasks is in Check it yourself; and the settings to compare are in Settings by goal.