All posts

A Self-Improving Agent Harness Still Needs a Grader You Own

Google's Harness² tunes an agent's harness per task without a grader. What it changes, what the results show, and the gate you need before shipping its edits.

Misha Druzhinin
Diagram: a Propose, Probe, Compose loop generates candidate agent harnesses, which must pass a capability floor gate and a sandbox boundary before a versioned harness ships.

Most teams design their agent harness once and run every task through it. The harness is everything around the model: system prompt, tools, memory, skills, subagents. Google Research’s Harness² (also written Harness2) lets the agent rewrite that harness for each new task. It uses no grader and no correct answer.

It works on the paper’s benchmarks. One round of edits lifted GLM-5 from 64.2 to 75.7 on WorkBuddy-Bench, and more rounds pushed it further (Xu et al., arXiv:2610.14224).

Our read: grader-free tuning is a good way to generate harness changes. It isn’t a way to decide which ones ship. The paper says so itself. Its run comparisons “inform subsequent composition rather than verify task correctness.” So keep the loop, and put your own capability floor, version control, and sandbox around it.

Key Takeaways

  • Harness² edits eight harness parts per task: prompt, guardrails, memory, tool guidance, subagents, skills, scripts, and plugins. Model weights stay fixed.
  • It picks edits by comparing how candidate runs behaved, judged by an editor model. No rubric or score is ever shown to it.
  • On two of three benchmarks, the weakest starting harnesses gained most: up to 11.8 points from one step, against 2-3 for the strongest.
  • Another automated method in the same paper scored below its own baseline in 7 of 12 pairs. Automated tuning can make an agent worse.
  • Treat its output as a proposed diff. Gate it on a fixed eval set, store it in version control, and run the tuning loop where it can’t cause side effects.

What does Harness² actually do?

Harness² runs a three-step loop on each task, called Propose, Probe, Compose. It keeps what it learns in a library for later tasks. Nothing touches the model’s weights.

Propose. The agent runs the task once with the current harness, as a reference. An editor session then proposes two edits. One targets general agent weaknesses: planning, tool use, context handling, delegation. The other targets the task’s domain knowledge and constraints.

Probe. Each candidate harness runs the same task, with the same model and environment. The system compares each run’s trajectory and output against the reference. The paper calls this a behavioral contrast.

Compose. The editor reads the reference run, both candidates, their diffs, and the execution evidence. It can merge edits, revise conflicting ones, adopt one candidate, or keep the current harness unchanged.

The budget is a knob, k. In parallel mode the system runs k sets of candidates and an agent picks the best composition. In sequential mode it refines one harness k times in a row.

Across tasks, a library stores each adopted harness with an EXPERIENCE.md file. That file records where the lesson applies, the lesson, and the evidence behind it. Lessons without supporting evidence aren’t stored. Each new task still starts from the base harness and pulls library components in through Propose. Nothing is inherited automatically.

If you’ve read our post on building the harness before a long agent run, this is the same layer. The difference is who edits it, and how often.

What do the results show?

Every Harness² row in the paper’s main results beat its base harness. The benchmarks cover long professional job tasks (JobBench), Harvey’s legal-work benchmark (LAB), and office, web, and code tasks (WorkBuddy-Bench).

Benchmark, model / executor Base Harness², 1 step Harness², k=5
WorkBuddy avg, GLM-5 / OpenCode 64.2 75.7 79.6
WorkBuddy avg, Gemini 3.5 Flash / Codex 74.5 77.5 80.4
LAB partial pass, GLM-5 / OpenCode 59.6 71.4 78.1
LAB partial pass, Gemini 3.5 Flash / OpenCode 79.9 81.9 84.5
JobBench, GLM-5 / OpenCode 20.6 24.2 28.4

Selected rows from the paper’s main results. k=5 values are the base plus the paper’s reported gains.

Three things in the results matter more than the headline numbers.

Weak starting harnesses gained most, mostly. This is our reading of the table, not a claim the paper makes. GLM-5 on LAB gained 11.8 points from one step. Gemini on the same benchmark gained 2.0. WorkBuddy shows the same split. JobBench doesn’t: the weaker GLM-5 setup gained 3.6, Gemini 4.5. Still, when the base harness is already good, there’s less for an editor to find. Recursion also didn’t close the gap between weak and strong starting harnesses. On WorkBuddy with GLM-5, that gap grew.

Automated tuning can lose. The paper compares against other self-improvement methods. One, Harness Scaling, scored below its own base harness in 7 of 12 pairs. Another, ReasoningBank, fell 6.6 points below base on one coding subset. “The agent improved its own setup” is not a safe default assumption.

Cross-task learning is real but small so far. A library built on JobBench added 0.6, 0.5, and 2.9 points to one-step WorkBuddy results on code, web, and office tasks. A separate leave-one-out rerun lifted WorkBuddy from 77.5 to 79.4. The authors call that an offline reuse analysis, and say repeated execution may explain part of it.

The paper also doesn’t report token or dollar cost per task. It says each step adds executions and editor sessions, and parallel runs cut latency but not compute. At k=5, you run the task many times before you get one answer.

Why isn’t a grader-free signal enough to ship on?

Because the signal is the editor’s judgment of two runs, not a check against what you need. The authors name the limits plainly. Run-to-run randomness and biases the models share limit how reliable that evidence is. It doesn’t guarantee correctness or transfer to other tasks.

That’s a fair trade for a research system meant for unseen tasks. It’s the wrong trade for a production agent where you know what “correct” means.

Compare it with NVIDIA’s SoL-Pi work, which we covered in cutting agent token spend. SoL-Pi also searched for harness changes automatically. But every change had to keep each capability metric within a tolerance set before the search began. About one in forty ideas survived.

Harness² is a better idea generator. SoL-Pi has the better shipping gate. You want both.

There’s a second reason. A harness that changes per task is hard to operate. When a production run goes wrong, you need to know which prompt, tools, and guardrails were active. You also need to rerun them. A per-task harness that lives only in a run directory makes both harder. Harness² helps here: its output is files, and the library writes its reasoning into EXPERIENCE.md. That’s reviewable. It just needs someone, or something you trust, to review it.

What can go wrong when the tuning loop runs?

The Probe step runs the same task several times. In a benchmark, that’s free of consequences. In your systems, it may not be.

Side effects multiply. If the task sends email, files a ticket, or writes to a database, each probe does it again. Run tuning against recorded traces with writes intercepted, as we describe in replaying production agent traces. Don’t run it against live tools.

The sandbox is off by default. The Harness² README says executor network access is unrestricted, and Codex runs with its sandbox disabled. It warns that public benchmark rubrics may be reachable online. It tells you to restrict network access before evaluation.

The paper does guard the editor. It refuses the editor any directory holding a rubric, score, or grader verdict. But the executor has the open network. An agent that can search for the grading criteria can score well without getting better. Keep credentials and network policy outside the agent’s reach, as in our post on agent credentials.

Guardrails are editable. Guardrails are one of the eight components Harness² can change. The editor prompt pushes back on overfitting and tells it not to run scripts found in candidate harnesses. Still, a policy your security team wrote shouldn’t be in the editable set. Split the harness into two parts: one the loop may edit, and one it may only read.

What does a shippable version look like?

Production repair loops already show the shape. Akshay Pachaar’s walkthrough of Comet’s Opik describes one. A failing trace gets a diagnosis and a proposed diff. Nothing changes without human approval. The fix reruns on the exact failing inputs, and the failure becomes a regression test. It’s a promotional walkthrough, but the shape is right.

Put Harness² inside that shape:

  1. Keep the harness in git. Prompt, tool definitions, skills, subagent configs, guardrails. Every production run logs the harness version it used.
  2. Fix the capability floor first. A held-out task set, pass criteria, and the regression you’ll tolerate. Our guide to designing evals for AI agents covers how to build one.
  3. Let the loop propose, not merge. Harness² output becomes a branch. The EXPERIENCE.md notes become the pull request description.
  4. Gate on the floor. The candidate must hold every capability metric on the held-out set, not just the task it was tuned on.
  5. Review the diff like code. Guardrail and tool-permission changes get a named owner. Our post on reviewing AI-written code by risk applies directly.

When is Harness² the wrong tool?

  • Your harness is already good. The paper’s strongest baselines gained 2-3 points from one step. Check whether your agent fails because of the harness before you pay for many extra runs.
  • Tasks are short or cheap to fix by hand. The loop’s cost is multiple full executions. A five-minute task with obvious failures is faster to fix directly.
  • You can’t isolate side effects. If you can’t replay a task without touching real systems, don’t probe it.
  • You have no eval. Without a floor, you can’t tell a better harness from one that does less. Build the eval first.

How to start this week

Pick one agent workflow whose failures you think come from its setup, not the model.

  1. Export its current harness into files in a repo: system prompt, tool descriptions, skills, subagent definitions. Tag it as version one.
  2. Write down 10 to 20 tasks with pass criteria. That’s your floor. Keep five aside that no tuning ever sees.
  3. Run one manual Propose-Probe-Compose round. Write two harness edits, run both on the same task, and diff the trajectories against the original.
  4. Run the winner on the held-out five. Ship it only if it holds there.

If one manual round finds an improvement, an automated loop will likely find more. The Harness² repo includes a local demo that runs without model credentials. If the manual round finds nothing, your problem is probably the model or the task design. Either way, you’ll have a versioned harness and an eval set, which many teams skip.

Frequently Asked Questions

What is Harness²?

Harness² is an open-source framework from Google Research, released under Apache 2.0. The README notes it isn’t an officially supported Google product. It refines an agent’s harness for each task through a Propose, Probe, Compose loop, without a ground-truth grader. It stores adopted harnesses and supported lessons in a library for later tasks.

Does Harness² change the model?

No. It edits only the harness: system prompt, guardrails, memory, tool guidance, subagents, skills, scripts, and plugins. Model weights stay fixed. It runs on top of OpenCode or Codex as the executor.

How does it compare runs without a grader?

An editor session compares trajectories and outputs from candidate harnesses against a reference run of the same task. The editor judges which behaviors to keep. It never sees rubrics or scores. The authors note this signal is sensitive to run-to-run variability and shared model biases.

Is it safe to run Harness² on production agents?

Not as shipped. The README says executor network access is unrestricted and Codex runs with its sandbox disabled. Run it against replayed traces with writes intercepted. Restrict the network, and gate every adopted harness on your own eval set.

If your agent’s harness has grown by accretion and nobody can say which parts still help, our AI cleanup work starts with exactly that audit.

Sources