Harness²: Automatic harness improvement, higher benchmark scores

Results

Evaluation covered WorkBuddy-Bench for office, web, and coding outputs, JobBench for end-to-end professional tasks, and LAB for legal work. The paper reports improvements for all four combinations across all three benchmarks. Gains ranged from 1.2–4.5 points in professional task scores and 2.0–11.8 percentage points in legal rubric pass rates. The table shows baseline harness → scores after 1 stage of Harness².

Table: Benchmark scores before and after 1 stage of Harness² (researcher-reported values, paper Table 1 and Table 2)

Conditions: By execution model and initial harness; baseline harness → Harness² stage 1 (k=1); harness editor: Claude Opus 4.8

Units: Points (WorkBuddy-Bench Avg., JobBench Score), % (LAB Partial absolute values), percentage points (LAB Partial changes in parentheses)

Comparing the same initial harnesses in the reported results, GLM-5 gained more than Gemini 3.5 Flash on office, web, coding, and legal tasks. With just 1 stage of improvement, GLM-5 exceeded Gemini 3.5 Flash’s baseline office, web, and coding average. Gemini 3.5 Flash gained more on professional tasks, so which model benefited more depended on the task.

Why the results differ

Comparing methods within the same model–harness setup clarifies where Harness² stands (GLM-5 + OpenCode, by domain).

Table: Methods compared by WorkBuddy-Bench domain for GLM-5 + OpenCode (researcher-reported values, paper Table 1)

Conditions: Scores by domain

Unit: Points

In the reported results, Harness² led competing methods by a wide margin in coding and web tasks, but was close to Harness Scaling on office tasks. The paper notes that ReasoningBank could fall up to 6.6 points below baseline in some coding task combinations. Harness Scaling fell below baseline in 7 of 12 paired combinations. According to the paper, Harness² led the comparison methods on WorkBuddy-Bench averages and JobBench scores in all four combinations. For legal work with Gemini 3.5 Flash + Codex, the offline search method Meta-Harness scored highest.

Harness² centers on proposal, testing, and assembly steps that revise the harness without a ground-truth verification tool. It proposes candidate edits, tests them on the same task, and assembles an improved harness from changes supported by differences in execution results. Harness-level recursion stores adopted harnesses and evidence-backed lessons in a library to refine subsequent tasks.

In a JobBench task (Gemini 3.5 Flash + OpenCode) reported in paper Figure 9, numerical cross-checks between input files and reinspection of output files were missing. The researchers added input inventory checks, numerical source tracing, output file reinspection, and task-specific reconciliation methods to the harness. They report that the rerun cross-checked figures and restored missing parts of a memo, raising the score from 43.1 to 77.8. This case fixed procedural flaws rather than adding knowledge.

k=1 denotes 1 stage of this improvement process; the +Recursion rows in the original paper tables use parallel recursion with k=5. In parallel recursion, k counts candidates and assembled results per scope, not stages. The paper finds that larger recursion budgets yield greater gains. For Gemini 3.5 Flash + OpenCode, raising k from 1 to 5 increased the reported gain over baseline from 5.1→8.4 points on the WorkBuddy-Bench average and 4.5→7.5 points on JobBench. Compute comparisons count all tokens before final output submission, including both task execution and harness improvement. The extra budget funds candidate testing and assembly as well as final answer generation. The offline search methods compared performed worse even with more compute than k=5.

The paper reports that all three components contribute to scores (ablation study using Gemini 3.5 Flash + OpenCode, k=1). Removing editor specialization means replacing the shared-procedure and task-specific editors with two general editors. Removing the library lowered WorkBuddy-Bench and JobBench scores by 3.1 and 1.8 points, respectively. Removing editor specialization lowered them by 2.9 and 3.7 points; removing assembly lowered them by 1.6 and 1.9 points.

Table: Benchmark scores in the ablation study (researcher-reported values, paper Table 3)

Conditions: Gemini 3.5 Flash + OpenCode, k=1

Unit: Points

How scores were measured

Each combination was run 3 times with shuffled task order to calculate means and standard deviations. The test sets comprised 40, 35, and 25 WorkBuddy tasks (coding, web, and office), 34 JobBench tasks, and 100 LAB tasks.

WorkBuddy uses a composite verifier for scoring, while Claude Opus 4.8 evaluates JobBench against rubrics. LAB reports per-criterion pass rates (Partial) evaluated by Claude Sonnet 4.6.

Limitations

During release validation, a Codex run on JobBench downloaded the task’s grading rubric without being instructed to do so. To prevent evaluation contamination, the execution agent’s external access must be restricted at the network level. Codex runs without a sandbox, a protection mechanism that isolates execution, so tool settings alone cannot be relied on to block access.

Installation guide

Installation targets Linux, Python 3.12, and uv. Follow the repository installation guide for model- and benchmark-specific dependencies and settings. The example below sets up LAB, runs a small test, and scores it separately. --max-tasks limits the number of tasks per domain.

harness2-setup lab
harness2-check --bench lab
harness2 --bench lab --k 1 --mode parallel \
  --only-domains antitrust-competition --max-tasks 1
harness2-judge --bench lab --k 1 --mode parallel

Selection criteria

Based on the authors’ experiments, teams able to modify their own harnesses (prompts, guardrails, skills, and scripts) can start with a single k=1 run. Use parallel recursion (k=5) if you have room to explore candidates, or sequential recursion to refine each stage. The paper compares both approaches at k=1–5.

The researchers report that improving the surrounding setup raised performance without replacing the model, supported by reusing experience across tasks.