20 real Python bug fixes from recent commits. Each model gets the GitHub issue and the code before the fix, works inside the Pi agent, and hidden tests decide if it worked. No LLM judge.
20 tasks means one task is 5 points. Read close scores as "on these 20 tasks", not as one model being better at coding in general. Click a row for per-task results.
Columns are tasks, sorted from most solved to least. Hover a cell for details.
Every task is a real issue-linked bug fix from a Python repo, committed in the 8 months before the release was frozen. Most models never saw the fix in training.
No Django, no FastAPI. Those are all over the internet and in every training set. The tasks come from real, used projects that aren't that famous.
The model works inside the Pi agent: it reads files, runs commands and edits code. No internet, no git history, no hidden tests, no gold patch in the sandbox.
A fix counts only if the hidden fail-to-pass test passes and the existing suite stays green. A failed first try gets one repair attempt with generic feedback only.
Before freezing, each task was proven: the test fails before the fix, passes after it, and nothing else breaks either way.
The release never changes. Scores from August and scores from today are on the exact same 20 tasks.
PatchEval runs through BenchKit. It builds every task's runtime locally and cleans up afterwards. Why only 20 tasks? Running SWE-bench locally is way too expensive for one GPU.
# get the frozen release, then point BenchKit at it git clone https://huggingface.co/datasets/DogukanUrker/PatchEval export BENCHKIT_PATCHEVAL_DATASET=$PWD/PatchEval # from a BenchKit checkout uv run benchkit --headless --models MODEL --benchmarks patcheval \ --harness pi --repair-attempts 1