Verifiable · Live · Contamination-resistant

COIN

Toward complete code-reasoning evaluation

COIN (COde → INput) measures the code-reasoning capability of LLM agents through semantic space → input space mapping: given a target line, the agent must produce an input that drives execution there. Execution decides — no answer to memorize, no judge to persuade.

Leaderboard

Targeted reachability — reach rate

8 state-of-the-art agents × 70 targets, 2-hour budget each. Reach rate = fraction of targets provably reached, verified by harness replay. Precision = fraction of submitted inputs that actually reached — even the best model's submissions are right barely half the time (52.5%), and the weakest lands only 12.8%: agents routinely claim success without reaching. Click a column header to sort.

30%best single model (Opus 4.6)
47 / 70unsolved by any of the 8
23 / 70union solved by any model (Opus alone: 21)
0.7%reach on frontier targets
# Model Scaffold Reach % Precision % Reached Frontier $/reached Total $

Open-weights models (DeepSeek, GLM) run under the Claude Code scaffold. Frontier = targets in the never-reached class solved (out of 35). Cost is total LiteLLM spend on the targeted track.

Constructing inputs that drive execution to chosen code locations underpins many dynamic-analysis tasks in software engineering and security — a generalized constraint-solving challenge that demands reasoning about the semantic mapping between input and code. COIN is the first benchmark to systematically measure this ability in modern code agents. It leverages OSS-Fuzz's long-running coverage data and maintainer-written harnesses to construct reachability challenges across 1000+ real-world projects in nine languages, and is contamination-resistant and self-refreshing as upstream code and coverage evolve. The strongest agent evaluated (Claude Code + Opus 4.6) reaches only 30% of selected targets; the weakest (DeepSeek V3.2) just 7.1%.

What COIN measures

A complete, verifiable measure of code reasoning

Most benchmarks evaluate comprehension — read code, explain it. COIN adds the mapping that turns comprehension into action: from a program's semantic space to its input space, producing an input that exercises a chosen line. Every answer is settled by execution, so the score is objective and verifiable — no judge, no leakable answer key.

semantic space · what the code means Comprehension Read a function, explain it, answer questions about it. Necessary — but graded against reference answers or an LLM judge: subjective, and easy to leak into training data.
input space · an input that reaches it Reachability Produce a concrete input that drives execution to a chosen line. Graded by running the code — reached or not: objective, and impossible to leak.

COIN grades code reasoning by running the code — reached or not. Comprehension is necessary, but the input is the proof: a verifiable, objective, contamination-resistant measure of code reasoning.

The task

Reach a line. Verified by execution.

A COIN task is a single target line ℓ in a real project at commit c. The agent picks one of the project's harnesses and produces a concrete input that, when fed to that harness, executes ℓ — checked by re-running the maintainer-written harness on the instrumented binary. No labels, no multiple choice: the ground truth is an input that actually reaches the line.

example target — frontier
project   libraw            (C++ raw-image decoder)
harness   libraw_raf_fuzzer
target    src/metadata/fuji.cpp:480
signals   G=0  F=0  L=0  C=1
# never covered by fuzzing or LLM baselines,
# but static analysis proves a reachable path.

agent must → emit bytes that decode as a
           Fuji RAF file and drive control
           flow into fuji.cpp:480.

🚫 Contamination-free by design

“What input reaches line ℓ in commit c?” is not text that exists on the web. Frontier targets have never been hit, so no solution can leak into training data.

🧰 Tool-permissive

Because there's no answer to memorize, agents may use anything real engineers use — web search, static analyzers, the project source itself — without leaking ground truth.

⚙️ Automatically verified

Outputs are graded by executing the project's own harness on a coverage-instrumented build. Reaching the line is a binary, objective outcome — no LLM judge.

Construction

Reachable by proof — but not by any cheap baseline

COIN doesn't take targets from fuzzing coverage — it filters against it. Reachability is established two ways: static CodeQL call-graph analysis, or long-running OSS-Fuzz history. Two cheap, target-blind baselines — a fresh fuzzer, and an LLM given the harness but not the target line — then mark everything a goal-blind shortcut already reaches. A line qualifies only when it is provably reachable yet both baselines miss it: reaching it takes real semantics-to-input reasoning. Fully automated, so the benchmark refreshes itself as OSS-Fuzz and upstream code evolve.

1000+
OSS-Fuzz projects
9
languages
8
agents evaluated
30%
best model reach
0.7%
on frontier lines
COIN construction and evaluation pipeline
The COIN pipeline: pin builds → establish reachability (static CodeQL + OSS-Fuzz history) and subtract cheap baselines (fresh fuzzing + goal-blind LLM) → typed target lines → agent evaluation, verified by harness replay.
Reachability evidence that some input provably reaches the line
C

Static reach

CodeQL call-graph evidence that a harness-to-target path exists in the source — reachability established with no dynamic run at all.

B

Untaken branch

The first executable line of an untaken branch or loop body adjacent to already-covered code — a witnessed edge the agent must flip.

G

Long fuzzing

Lines in OSS-Fuzz's continuous, industrial-scale coverage — months of fuzzing proving the line is reachable in practice.

Cheap baselines what a target-blind shortcut already reaches — subtracted
F

Short fuzzing

A fresh, fixed-budget libFuzzer run in-experiment — what a fuzzer reaches right now, with no target-specific effort.

L

LLM seed-gen

An LLM given the harness context but not the target line, generating seeds — what a naive, goal-blind model reaches.

Frontier targets

Never reached, statically reachable

(B ∪ C) \ (G ∪ F ∪ L) — no fuzzer or baseline has ever covered the line, yet static analysis proves a path exists. The hardest, most contamination-resistant class.

Non-trivial reachable

Covered by long fuzzing, missed by baselines

G \ (F ∪ L) — long-running fuzzing reached it, but fresh fuzzers and goal-blind LLM seeds miss it. The agent must rediscover a witnessed path cheap baselines can't.

639,410 → 70

31,259 non-trivial reachable + 7,048 branch-frontier candidates, sampled to 70 evaluable targets — five per family across 7 projects. As commits land, new frontiers open: COIN cannot saturate.

UpSet plot of reachability signal regions
How the G / F / L / C / B signals partition the 639,410 candidate uncovered lines — the intersections COIN samples its targets from.
Per-target results

Every model, every target

One column per model, one row per target — colored by outcome. Frontier targets (top) are a near-solid wall; the difficulty gradient is the point. Hover any cell for the target and outcome.

70 targets × 8 models on the targeted track. 47 targets were solved by no model; only 1 by all eight. Frontier targets account for nearly all of the unsolved wall.

What we learned

Even the strongest agents mostly miss

The frontier is a wall

Across all 8 agents and 35 frontier targets — lines no fuzzer or baseline has ever reached — the aggregate reach rate is 0.7%, versus 38.2% on non-trivial reachable targets. Reasoning to a never-before-seen input is categorically harder than rediscovering one long-running fuzzing already found.

Frontier
0.7%
Reachable
38.2%

Cost ≠ capability

Spend-per-reached-target doesn't track skill: GPT-5.4-mini is cheapest per success at $1.29, while Sonnet costs $15.51 per reach — more than Opus's $9.99 despite reaching fewer. Backward reasoning is the bottleneck, not budget.

A wide capability gap

Within every vendor, flagships beat their lightweight siblings — 25.7% vs 19.1% reach. Commercial models average 22.4% vs 10.7% for open-weights. Only 1 target was solved by all eight.

Reach rate versus cost per model
Reach rate vs. cost. More spend does not translate into more reached targets.

Two task types, one tool ablation — from a single target set

Targeted reachabilitytask type

Reach one specific line. The headline metric — reach rate and precision.

Seed-tool ablationtool access

The same targeted task, but mount the saturated seed corpus + a corpusdb query tool. Aggregate reach barely moves (19.5% → 19.1%); it just reshuffles which lines get solved.

Coverage maximizationtask type

No target line — maximize whole-project coverage. Coverage is not a target proxy: on liboqs it touches 47.7% of lines yet hits 0 of the 10 targets.

Reproduce

Run COIN yourself

The whole benchmark is one command against a published snapshot. Each project is rebuilt from its pinned commits and gated by a functional precheck, so you need only the dataset and a Docker host — no OSS-Fuzz tree, no re-mining. 

# install (uv resolves the bundled corpusdb)
uv sync --extra llm

# evaluate an agent against a published COIN snapshot
coin evaluate --dataset you/coin@v1

# view the dashboard
coin show

Requires Python ≥ 3.12 and Docker. Set LITELLM_URL / LITELLM_MASTER_KEY for the agents; HF_TOKEN to publish.

Why "live"

A benchmark that moves with the code

Static benchmarks leak and saturate. COIN is regenerated from OSS-Fuzz and upstream commits — new code opens new frontiers, and yesterday's solved targets are replaced. There is no fixed answer key to overfit, and no way to memorize inputs that have never existed.