Isolation is not engineering
A function-level benchmark hands the model a signature, a docstring, and the intent. Real work supplies none of those. The first task is finding which of four hundred thousand lines is wrong.
DoraBench — Benchmarks for software-engineering agentsEdition 01 · Autumn 2026
We measurehow wellAI writessoftware.
Real repositories. Real engineering tasks. Reproducible evaluation.
Specimen01/08
TASK_019284
- Repository
- tokio-rs/tokio
- Language
- Rust
- Difficulty
- hard
- Tests
- 4 → pass
StatusEvaluated
01The problemWhy benchmark quality matters
The problemisn’t writing code.It’s writingthe right code.
Evaluating a coding model means giving it work that resembles the work. Three things make that hard — and all three are design decisions, not accidents.
Saturated scores stop separating
When every frontier system clears ninety percent, the benchmark has stopped measuring and started congratulating. Difficulty has to be admitted deliberately, not discovered by accident.
A plausible diff is not a fix
Reviewers approve changes that read correctly and do nothing. Only a suite that ran red before the patch and green after it can tell you the behaviour actually moved.
02The benchmarkEdition 01 · 2026.09
MeetTerminus.
A benchmark assembled from merged pull requests in projects people depend on. Each instance is a repository at a commit, an issue written by someone who hit it, and the tests that prove the fix.
The model gets what an engineer gets on their first morning: the codebase, the report, and nothing else.
- Instances
- 48,291
- Languages
- 8
- Harness
- Local or CI
- Format
- JSONL + container
- Repository
- tokio-rs/tokio
- Language
- Rust
- Difficulty
- hard
- Base commit
- a7c1e04
Issue#6482
Connections are dropped without flush when the runtime shuts down
Shutting down a multi-threaded runtime while a TcpStream still has buffered writes drops the connection before the write half is flushed. The shutdown path releases driver resources before parked I/O tasks get a final poll, so the peer observes a truncated response instead of a clean close.
Files in scope3
- src/runtime/scheduler/multi_thread/mod.rs
- src/runtime/driver.rs
- src/net/tcp/stream.rs
Evaluation
- Build
- Tests
- Patch
- Semantic
Fail → pass4Pass → pass1,842Runtime4m 12s
One instance of 48,291
03Coverage8 shipping · 6 in review
One benchmark. Many languages.
- Rust.rs3,914Engineering tasks
- 31
- Repos
- 8,112
- Tests
- 8.7
- Complexity
Lifetimes, trait resolution, and failures the compiler refuses to let you fake.
Rust
.rs3,914Engineering tasks
- 31
- Repos
- 8,112
- Tests
- 8.7
- Complexity
Lifetimes, trait resolution, and failures the compiler refuses to let you fake.
Sample repositories
- tokio-rs/tokio
- rust-lang/cargo
- serde-rs/serde
- hyperium/hyper
- BurntSushi/ripgrep
- clap-rs/clap
In review for Edition 02Kotlin · Swift · Scala · C# · Elixir · Zig
04ScaleEdition 01 · 2026.09
48,291
Engineering tasks
Every one derived from a merged fix in a public repository.PY29.4%RS8.1%CPP6.5%RB4.3%JS13.5%TS18.5%GO9.1%JV10.6%
8+
Languages
Six more in review for the next edition.
341+
Repositories
Median age eight years, median 12k stars.
98,204+
Verified tests
Red-to-green transitions, checked on clean containers.
- Provenance
- Merged pull requests, human-authored
- Contamination
- Held-out split, post-cutoff commits only
- Environment
- Pinned container per repository
- Reproducibility
- Deterministic seeds, recorded runtimes
- Licence
- Per-repo attribution, commercial use permitted
- Format
- JSONL + per-task Docker context
05EvaluationBuild · Test · Measure · Compare
From patch to verdict.
Four stages, every one of them executed. Nothing about a run is inferred from the diff.
01
Build
The repository is restored to its base commit inside a pinned container, then built from scratch. A patch that does not compile stops here.
Tracebuild
- pull ghcr.io/dorabench/tokio@sha256:9f2c…
- reset --hard a7c1e04
- apply candidate.patch → 3 files changed
- cargo build --all-targets → ok (41.2s)
BuildPASS
02
Test
Two suites run. The failing tests attached to the issue must go green, and every test that passed before must still pass.
Tracetest
- run FAIL_TO_PASS · 4 tests
- shutdown::flushes_pending_writes ok
- shutdown::drains_before_driver_release ok
- run PASS_TO_PASS · 1842 tests → 1842 ok
Tests4 / 4
03
Measure
The patch is compared against the reference fix: scope, file overlap, and whether the change addresses the cause or masks the symptom.
Tracemeasure
- diff candidate ↔ reference
- files in scope 3 / 3
- out-of-scope edits 0
- cause addressed yes
- patch size 71+ / 23−
Scope3 / 3
04
Compare
Results are written to a run record with the seed, image digest, and timings, so any number on a leaderboard can be reproduced exactly.
Tracecompare
- record run_2026_09_11T14_22Z
- seed 41
- image sha256:9f2c…
- wall clock 4m 12s
- resolved true
ResolvedTRUE
06ExploreSample of the published set
Look at the tasks.
A slice of the set, with the fields the harness actually reads. Open a row to see the record an agent receives.
36 tasks matching
TaskIssueLanguageDifficulty
Sample rendered from the published schema. The full set ships as JSONL with one container context per repository.
07Why TerminusFour decisions
Real, not plausible.
Four decisions, each of which costs something to make and shows up in every score the benchmark produces.
01 — Provenance
Real repositories
Every task comes from a merged pull request in a project people actually depend on. No generated bugs, no toy repositories, no puzzles written to be solvable.
- 341
- Repositories
- 8 years
- Median age
- 0
- Synthetic tasks
100 candidates reviewed38 admitted
02 — Difficulty
Real engineering problems
A task is only admitted if the reference fix required reading code the issue never mentions. Single-line typo fixes are filtered out before review.
- 3.1
- Mean files touched
- 46%
- Cross-module
- 62%
- Rejected at review
03 — Verification
Real tests
Each task ships the tests that failed before the fix and the suite that must stay green after it. Passing means the behaviour changed, not that the diff looked plausible.
Fail → passtokio-rs/tokio · #6482
- shutdown::flushes_pending_writes
- shutdown::drains_before_driver_release
- net::tcp_stream_close_is_graceful
- runtime::driver_release_ordering
- 98,204
- Verified tests
- 5.4
- Mean fail-to-pass
- 1,180
- Flaky, removed
04 — Reproducibility
Reproducible signal
Pinned images, recorded seeds, and a run record for every evaluation. A score you publish today can be re-derived by someone else next year.
- Pinned images
- 341
- Re-run variance
- < 0.4%
- Run records
- Public
Run · 2026.09.11resolved
- image
- sha256:9f2c1a…
- seed
- 41
- fail→pass
- 4 / 4
- by
- dorabench/ci
Run · 2027.03.02resolved
- image
- sha256:9f2c1a…
- seed
- 41
- fail→pass
- 4 / 4
- by
- independent replication
08Early accessEdition 01 · Open
TerminusSWE
Benchmark
One purchase, the whole edition. No seats, no metering, and no call with anybody.
Full benchmark dataset48,291 instances · JSONL
Task metadataRepository, commit, issue, test manifests
Evaluation harnessContainer definitions · local or CI
DocumentationSchema reference and run guide
Commercial licenceInternal use and published results
Edition 02Included when it ships
One-time
$49
Launch pricing
Early pricing while the benchmark grows. It buys the edition outright — nothing expires and nothing renews.
Get early access
- Delivery
- Signed download, immediate
- Updates
- Edition 02 included
- Licence
- Commercial, per-repo attribution
Evaluating at scale or need a custom split? Write to hello@dorabench.in.
09QuestionsAnswered in full
Before you ask.
Everything a reviewer usually emails about, with the numbers attached.
01What is DoraBench?
DoraBench is a benchmark company that builds evaluation datasets for AI software-engineering agents. It publishes Terminus, a benchmark of 48,291 real engineering tasks drawn from merged pull requests across 341 open-source repositories in eight programming languages.
02What is Terminus?
Terminus is DoraBench's flagship software-engineering benchmark. Each of its 48,291 tasks pairs a real repository at a specific commit with a real issue and the tests that verify the fix, so a coding agent is scored on whether its patch makes failing tests pass without breaking the existing suite.
03Which programming languages does Terminus cover?
Terminus covers eight languages: Python, TypeScript, JavaScript, Java, Go, Rust, C++, and Ruby. Python is the largest slice at 14,208 tasks and Ruby the smallest at 2,100. Kotlin, Swift, Scala, C#, Elixir, and Zig are in review for the next edition.
04How is Terminus different from SWE-bench?
SWE-bench is drawn almost entirely from Python repositories. Terminus spans eight languages including compiled and memory-managed ones, admits a task only when the reference fix required reading code the issue never names, and ships a pinned container image and recorded seed per task so any published score can be re-derived exactly.
05How does Terminus evaluate an AI coding agent?
Evaluation runs in four stages. The repository is restored to its base commit in a pinned container and built; the failing tests attached to the issue must go green while the existing suite stays green; the patch is compared against the reference fix for scope; and the result is written to a run record with the seed, image digest, and timings.
06How are Terminus tasks verified?
Every task is executed before admission. The reference patch must build, turn its FAIL_TO_PASS tests from red to green, and leave 98,204 PASS_TO_PASS tests untouched. Tasks that pass without the patch, fail intermittently, or resolve in a single line are rejected — 62% of candidates are discarded at review.
07How does Terminus handle training-data contamination?
Terminus ships a held-out split built only from commits merged after the published cutoff of current frontier models, so those tasks cannot appear in existing training corpora. Each task records its merge date, letting you filter the set against any model's cutoff yourself.
08How much does Terminus cost?
Terminus early access is $49 as a one-time purchase. It includes the full benchmark dataset, per-task metadata, the evaluation harness, container definitions, documentation, and a commercial-use licence. Pricing is held low while the benchmark grows toward its next edition.
09What format is the Terminus dataset delivered in?
Tasks ship as JSONL, one object per instance, with the repository, base commit, problem statement, reference patch, test manifests, and container digest. A Docker context accompanies each repository, and the evaluation harness runs locally or in CI without calling a DoraBench service.
10Can Terminus be used commercially?
Yes. The early-access licence permits commercial evaluation, including internal model development and published results. Source repositories keep their original licences, and per-repository attribution ships with the dataset.
How goodis your AI?
Test it against reality — real repositories, real issues, real suites.
Edition 01 · Autumn 2026Open for early access