Isolation is not engineering
A function-level benchmark hands the model a signature, a docstring, and the intent. Real work supplies none of those. The first task is finding which of four hundred thousand lines is wrong.
DoraBench — Benchmarks for software-engineering agentsEdition 01 · Autumn 2026
We measurehow wellAI writessoftware.
Real repositories. Real engineering tasks. Reproducible evaluation.
01The problemWhy benchmark quality matters
The problemisn’t writing code.It’s writingthe right code.
Evaluating a coding model means giving it work that resembles the work. Three things make that hard — and all three are design decisions, not accidents.
Saturated scores stop separating
When every frontier system clears ninety percent, the benchmark has stopped measuring and started congratulating. Difficulty has to be admitted deliberately, not discovered by accident.
A plausible diff is not a fix
Reviewers approve changes that read correctly and do nothing. Only a suite that ran red before the patch and green after it can tell you the behaviour actually moved.
02The benchmarkEdition 01 · 2026.09
MeetTerminus.
A benchmark assembled from merged pull requests in projects people depend on. Each instance is a repository at a commit, an issue written by someone who hit it, and the tests that prove the fix.
The model gets what an engineer gets on their first morning: the codebase, the report, and nothing else.
- Languages
- 8
- Source
- Merged pull requests
- Harness
- Local or CI
- Format
- JSONL + container
- instance_id
- string
- owner__repo-PR
- repo
- string
- Source repository
- language
- string
- Primary language of the fix
- base_commit
- string
- Where the agent starts
- problem_statement
- string
- The issue, as filed
- patch
- string
- Reference fix, withheld at run time
- FAIL_TO_PASS
- string[]
- Must turn red to green
- PASS_TO_PASS
- string[]
- Must stay green
- environment
- object
- Image digest, build and test commands
Handed to the agent
The repository at its base commit and the problem statement. Not the patch, not the test names, not the files the fix touches — those are held back and used to score it.
Signals scored
- Build
- Tests
- Patch
- Semantic
- Environment
- Pinned container per repository
- Reproducibility
- Deterministic seeds, recorded runtimes
The published record, field by field
03Coverage8 shipping · 6 in review
One benchmark. Many languages.
- Rust.rs
Lifetimes, trait resolution, and failures the compiler refuses to let you fake.
Rust
.rsLifetimes, trait resolution, and failures the compiler refuses to let you fake.
Sample repositories
- tokio-rs/tokio
- rust-lang/cargo
- serde-rs/serde
- hyperium/hyper
- BurntSushi/ripgrep
- clap-rs/clap
In review for Edition 02Kotlin · Swift · Scala · C# · Elixir · Zig
04EvaluationBuild · Test · Measure · Compare
From patch to verdict.
Four stages, every one of them executed. Nothing about a run is inferred from the diff.
01
Build
The repository is restored to its base commit inside a pinned container, then built from scratch. A patch that does not compile stops here.
Tracebuild
- pull <image digest from the record>
- reset --hard <base_commit>
- apply candidate.patch
- build --all-targets
BuildPASS
02
Test
Two suites run. The failing tests attached to the issue must go green, and every test that passed before must still pass.
Tracetest
- run FAIL_TO_PASS → must turn green
- run PASS_TO_PASS → must stay green
- no test is skipped, none is inferred from the diff
TestsPASS
03
Measure
The patch is compared against the reference fix: scope, file overlap, and whether the change addresses the cause or masks the symptom.
Tracemeasure
- diff candidate ↔ reference
- files in scope
- out-of-scope edits
- cause addressed, not masked
ScopeIN SCOPE
04
Compare
Results are written to a run record with the seed, image digest, and timings, so any number on a leaderboard can be reproduced exactly.
Tracecompare
- record run
- seed
- image digest
- wall clock
- resolved
ResolvedTRUE
05Why TerminusFour decisions
Real, not plausible.
Four decisions, each of which costs something to make and shows up in every score the benchmark produces.
01 — Provenance
Real repositories
Every task comes from a merged pull request in a project people actually depend on. No generated bugs, no toy repositories, no puzzles written to be solvable.
Projects of the kind we draw from
Rejected at review
- Passes without the patch applied
- Fails intermittently across runs
- Resolves in a single line
- Issue names the file to change
- Fix is mechanical or generated
02 — Difficulty
Real engineering problems
A task is only admitted if the reference fix required reading code the issue never mentions. Single-line typo fixes are filtered out before review.
03 — Verification
Real tests
Each task ships the tests that failed before the fix and the suite that must stay green after it. Passing means the behaviour changed, not that the diff looked plausible.
Suites per taskExecuted, not inferred
- FAIL_TO_PASSRed before the patch, green after it
- PASS_TO_PASSGreen before, and still green after
04 — Reproducibility
Reproducible signal
Pinned images, recorded seeds, and a run record for every evaluation. A score you publish today can be re-derived by someone else next year.
Run recordWritten per evaluation
- image digest
- seed
- base commit
- fail → pass
- pass → pass
- wall clock
Same record, same inputs, same result — whoever re-runs it, whenever they do.
06AccessEdition 01 · Open
TerminusSWE
Benchmark
The whole edition, delivered once. No seats, no metering, and no call with anybody.
Full benchmark datasetJSONL, one object per instance
Task metadataRepository, commit, issue, test manifests
Evaluation harnessContainer definitions · local or CI
DocumentationSchema reference and run guide
Commercial licenceInternal use and published results
Edition 02Included when it ships
Availability
Early
access
Open now
The edition is still growing. Write to us and you will be sent what exists today, the harness that runs it, and the licence that covers it.
Request access
- Delivery
- Signed download
- Updates
- Edition 02 included
- Licence
- Commercial, per-repo attribution
Evaluating at scale or need a custom split? Write to hello@dorabench.in.
07QuestionsAnswered in full
Before you ask.
Everything a reviewer usually emails about, answered before they have to.
01What is DoraBench?
DoraBench is a benchmark company that builds evaluation datasets and harnesses for AI software-engineering agents. It publishes Terminus, a benchmark of real engineering tasks drawn from merged pull requests in open-source repositories across eight programming languages.
02What is Terminus?
Terminus is DoraBench's software-engineering benchmark. Each task pairs a real repository at a specific commit with a real issue and the tests that verify the fix, so a coding agent is scored on whether its patch makes failing tests pass without breaking the existing suite.
03Which programming languages does Terminus cover?
Terminus covers eight languages: Python, TypeScript, JavaScript, Java, Go, Rust, C++, and Ruby. Kotlin, Swift, Scala, C#, Elixir, and Zig are in review for the next edition.
04How does Terminus evaluate an AI coding agent?
Evaluation runs in four stages. The repository is restored to its base commit in a pinned container and built; the failing tests attached to the issue must go green while the existing suite stays green; the patch is compared against the reference fix for scope; and the result is written to a run record with the seed, image digest, and timings.
05How are Terminus tasks verified?
Every task is executed before admission. The reference patch must build, turn its FAIL_TO_PASS tests from red to green, and leave the PASS_TO_PASS suite untouched. Tasks that pass without the patch, fail intermittently, or resolve in a single line are rejected at review.
06How does Terminus handle training-data contamination?
Terminus ships a held-out split built only from commits merged after the published cutoff of current frontier models, so those tasks cannot appear in existing training corpora. Each task records its merge date, letting you filter the set against any model's cutoff yourself.
07How do I get access to Terminus?
Terminus is in early access. Write to hello@dorabench.in and you will be sent the dataset, the evaluation harness, the container definitions, the schema reference, and a commercial-use licence.
08What format is the Terminus dataset delivered in?
Tasks ship as JSONL, one object per instance, with the repository, base commit, problem statement, reference patch, test manifests, and container digest. A Docker context accompanies each repository, and the evaluation harness runs locally or in CI without calling a DoraBench service.
09Can Terminus be used commercially?
Yes. The early-access licence permits commercial evaluation, including internal model development and published results. Source repositories keep their original licences, and per-repository attribution ships with the dataset.
How goodis your AI?
Test it against reality — real repositories, real issues, real suites.
Edition 01 · Autumn 2026Open for early access