Even when an agent fixes a bug, maintainers can be left with a large patch to read. Parsimony parses the code before and after patches published for DeepSWE, SWE-bench Verified and SWE-bench Live, and counts the “coding units” added and removed.

Tests are not rerun; the pass/fail results each benchmark published are used as is. The repository says anyone can recompute every number from the published data.

Coding units count syntax elements, not formatting

A coding unit is one syntax element, such as a statement, call, comparison, name or literal. Each statement block also ends with one EndBlock unit, so nesting and moving code into or out of a block are counted.

Comments, formatting, punctuation, brackets and name length are not counted. In the method doc's example, `if x.count(y) > 0: foo(a, b)` is 12 units.

For Python, only .py implementation files count, and test, doc and example directories and generated files are excluded. Each record lists the touched and excluded files.

Net units (net_units) are additions minus deletions, and churn is their sum. A small net can still hide a large rewrite.

Rewrites needing more than 500 edits use a line-anchored approximation and are flagged. A file that fails to parse on either side is recorded as unknown (null), never zero.

What was measured comes before the ranking

The website ranks models by mean net coding units over attempts that solved the task, smallest first. Next to it are the same mean over all measured attempts including failures, the benchmark solve rate and how many attempts could be measured.

The two mean columns answer different questions. A failed patch that deletes a lot of code does not affect the solved-attempt ranking.

Attempts that cannot be measured because of missing files or parse errors are left out, and the Measured column shows how many were measured. Even with the same coverage, each model may have solved a different set of tasks.

Caveats: the command-line leaderboard differs from the website ranking

The command-line leaderboard command sorts by medians, not means. The --shared option narrows comparison to the intersection of tasks every selected agent solved and that could be analyzed.

The method doc says --shared avoids task-mix confounding but can favor easy tasks. There is no basis for assuming the website's default ranking compares only shared tasks.

The README's limits section says models that write more code also tend to solve more tasks. On boards with 26 or more models, the Spearman correlation was 0.33–0.60.

Each benchmark and language has its own board, and unit counts cannot be compared across boards. Analyzer versions also differ by board, from 0.5.2-beta to 0.8.0-beta.

The agent matters, not only the model. DeepSWE and Verified use mini-SWE-agent for every model except GLM-5.1 on Verified, and each Live configuration uses its own agent.

The method doc notes that measurements depend on the interpreter's tokenizer, so one pinned Python version should be used for anything compared.

Turning code footprint into review questions

For maintainers, code footprint is a starting point for questions, not a verdict. Looking at net and churn together, one asks what comparable solved attempts added and removed.

Then the actual patch is checked for needed validation, readability and behavior the benchmark did not cover. A small patch may be easier to review, but that is a judgment made by reading it.

Where code size ends and technical debt begins

The README says units measure size, not readability or design. The public boards use beta analyzer versions, and pass/fail evidence comes from the original benchmarks.

Parsimony's net coding units are a supporting metric beside correctness and human review, and fewer units do not make a better patch.