How to Replay Production Agent Traces Offline

A customer writes to your scheduling agent on a Tuesday: “Can you move my install to Thursday?” The agent checks Thursday, finds no open slots, and says it can’t help. It never offers Friday. Online grading flags the run, and you promote the trace into an eval case. A week later you run that case in CI. It doesn’t fail the same way. It doesn’t fail at all.

Jake Klaristenfeld’s guide to eval-driven development at Tenex builds its loop on that promotion step. Alex Lieberman shared it on X, calling evals “the #1 thing keeping companies from becoming AI-native.” The guide ends by admitting the catch. For agents that read from or write to real systems, “you often can’t” recreate the trace. The guide leaves that problem for two future parts. This post fills that gap: how to make a promoted trace replay faithfully, and how to run it without touching production.

Key Takeaways

• A promoted trace drifts for four reasons: world state, the clock, model sampling, and writes. Running k trials only handles sampling.
• Freeze reads by replaying recorded tool outputs, seeding a fixture from the trace, or simulating tools. Replay is cheapest when traces store full outputs; fixtures survive changed paths.
• When the agent makes a read the recording doesn’t cover, call it diverged, not failed. Never fall through to a live system.
• In eval mode, write tools record what they would have done. Grade those intents with plain code, and treat any unclassified tool as a write.

Why doesn’t a promoted trace reproduce?

A trace records what the agent did. It doesn’t record the world the agent did it in. Rerun the input later and four things have moved.

1. World state. Tools return today’s data, not last week’s. A cancellation has since opened a Thursday slot, so the agent books it. The case passes for the wrong reason.
2. The clock. “Thursday” meant October 1 when the customer wrote. Run the case a week later and the model resolves it to October 8. Every relative date in the input points somewhere new.
3. Model sampling. The same input can take a different path on the next run. The Tenex guide covers this one with repeated trials and an explicit pass bar. Our eight-step eval method covers pass^k in depth.
4. Writes. reschedule_appointment moves a real technician’s day. You can’t run that a few hundred times a week in CI.

Only the third is a statistics problem. The other three are engineering problems at the tool boundary, and more trials won’t fix them. You fix them by controlling what each tool returns and what each tool is allowed to do.

Freeze the reads: replay, fixture, or simulator

You have three ways to give the agent the world it saw in production. They differ in what they cost to build and in what happens when the agent takes a different path from the recorded one.

Approach	What answers a read	Setup cost	When the agent takes a new path	Best for
Replay	The output recorded in the trace, matched by tool name and arguments	Low, if traces store full tool outputs	It has no answer	Reproducing a specific failure
Fixture	A small seeded datastore built from what the trace’s reads returned	Medium	Answers consistently from seeded state	Regression cases that should survive prompt changes
Simulator	A model, or hand-written code, that invents plausible responses	Low to start, high to trust	Answers, possibly wrongly	Exploring paths no one has recorded yet

Replay is the cassette pattern from HTTP testing. Python’s vcrpy records each request and response to a file, then serves them back on later runs. By default it matches on method, scheme, host, port, path, and query. Do the same at the tool boundary: key each recorded output by tool name and arguments. Replay works only if your tracing captured each tool call’s arguments and its full output, untruncated. Many setups log a preview or drop large payloads. Check that first, because no replay strategy recovers data you didn’t store.

A fixture turns the recorded reads into state. The trace showed one appointment and an empty Thursday, so you seed exactly that, plus the Friday slots you need for the correct answer. Now the agent can query in any order and get consistent answers. τ-bench builds on the same idea: it compares the database state at the end of a conversation with an annotated goal state. Our eval method’s outcome step covers that style of grading. Fixtures cost more to write, but a well-chosen fixture outlives many prompt changes.

A simulator answers any query, which is both its strength and its risk. An LLM-backed tool simulator is another model whose output you now have to evaluate. Use it to explore, then turn the paths that matter into replay or fixture cases.

We recommend pairing them. Replay reproduces the failure exactly, once, to confirm you can see it offline. Then you convert the case into a fixture so it keeps testing the behavior after your fix changes the agent’s path.

Decide what happens when the agent leaves the recording

Your fix will change the agent’s path, and that’s the point. The fixed agent checks Thursday, then calls list_open_slots for Friday. The recording has no Friday call, so replay has nothing to return.

You need a policy for that moment before it happens. There are three sensible choices:

• Fail closed. Raise an error and mark the run diverged. This is the right default for replay.
• Fall back to a fixture. Answer from seeded state, if the case has one.
• Fall back to a simulator. Only for exploratory runs, and label the result as simulated.

Two rules hold whichever policy you choose. First, a cache miss must never fall through to a live system. That’s how an eval run ends up booking real technicians. Second, diverged is a third outcome, not a failure. A diverged run tells you the case can’t judge this version of the agent. Count divergences separately from pass/fail and watch the rate. When a case starts diverging after a deliberate fix, convert it to a fixture case. When it diverges after an unrelated change, look at why the agent’s path moved.

The HTTP-testing world reached the same default. vcrpy’s none record mode raises an error on any request the cassette doesn’t hold. The pytest-recording plugin uses that mode by default “to prevent unintentional network requests.”

Don’t mistake framework replay for frozen replay, either. LangGraph’s time travel resumes a run from a checkpoint, but it “re-executes nodes—it doesn’t just read from cache.” Model and API calls after the checkpoint fire again and can return different results.

Intercept writes at the tool boundary

In eval mode, a write tool shouldn’t write. It should record what it would have written and return a plausible success response. Those recorded write intents are the most gradable thing an agent produces.

Grade them with plain code. Count the reschedule_appointment intents: there should be one. Check that it names the appointment from the trace and the Friday slot the customer accepted. Check that the confirmation follows the reschedule, not the other way round. These are the deterministic checks the Tenex guide recommends using “wherever you can,” and writes are where they matter most.

With a fixture, go one step further and apply each intercepted write to the seeded state. Then later reads see a consistent world: the agent books Friday, and its follow-up get_appointment call returns Friday. That’s a fake in Martin Fowler’s sense: a working implementation that “usually take[s] some shortcut.” Grading the final fixture state also gives you an outcome check. Anthropic’s guide to agent evals defines the outcome as “the final state in the environment at the end of the trial.” AppWorld adds a useful twist: its state-based tests also check for collateral damage, changes the agent shouldn’t have made.

Three more practices keep writes safe.

• Classify every tool as read or write, and default to write. A tool you haven’t classified gets intercepted. If you expose tools through MCP, the spec’s readOnlyHint and destructiveHint tool annotations are a useful starting point. The spec itself says clients must treat them as untrusted unless they come from trusted servers. Your harness should own the list.
• Use provider sandboxes for integration tests, not for every eval. Stripe’s sandboxes, for example, are isolated test environments where payments “aren’t processed by card networks.” They’re still network calls against shared state, so run them in a smaller suite before releases.
• Carry idempotency keys through the tool layer. Stripe’s idempotent requests show the pattern: the API saves the first result for a key and returns it on retries. Keys protect production from retried writes, and in evals they let you assert the agent didn’t issue the same write twice. Our retry storm analysis covers why retried writes hurt at scale.

Build the tool layer so this is possible

None of this works if each tool calls its API directly. The agent should reach every external system through one tool layer with a swappable backend: live in production, replay or fixture in evals. The clock goes through the same layer. Java’s Clock docs call this best practice: pass a clock into any method that needs the current time. In Python, freezegun can freeze time from outside when you can’t inject it. Braintrust’s agent eval guidance recommends the same split for dependencies: snapshot enough production state to stub databases and APIs.

Here is the core of a replay backend. It’s deliberately small: about 40 lines of Python, tested with pytest. Note that it lists read tools, not write tools, so a tool nobody classified is intercepted by default.

import json
from dataclasses import dataclass, field
from datetime import datetime

# Anything not listed here is treated as a write and intercepted.
READ_TOOLS = {"get_appointment", "list_open_slots"}


class Diverged(Exception):
    """The agent made a read the recording doesn't cover."""


def key(name, args):
    return name + ":" + json.dumps(args, sort_keys=True)


@dataclass
class ReplayBackend:
    """Answers reads from a recorded trace; records writes instead of running them."""
    recorded: dict                      # key(name, args) -> recorded tool output
    writes: list = field(default_factory=list)
    misses: list = field(default_factory=list)

    @classmethod
    def from_trace(cls, spans):
        return cls({key(s["tool"], s["args"]): s["output"]
                    for s in spans if s["tool"] in READ_TOOLS})

    def call(self, name, args):
        if name not in READ_TOOLS:
            self.writes.append({"tool": name, "args": args})
            return {"ok": True, "simulated": True}
        k = key(name, args)
        if k not in self.recorded:
            self.misses.append(k)
            raise Diverged(k)           # fail closed: never fall through to live
        return self.recorded[k]


@dataclass
class FrozenClock:
    at: datetime

    def now(self):
        return self.at

Build it from the promoted trace and pin the clock to the trace’s start time. Your harness must also write clock.now() into the prompt. The model resolves “Thursday” from the date it’s given, not from your Python clock.

Now the customer’s “Thursday” resolves to October 1 again, and the empty Thursday comes back from the recording. The unfixed agent gives up, exactly as it did in production. After your fix, its new Friday lookup raises Diverged instead of reaching a live system. Any write either version attempts lands in backend.writes, not in your scheduling system.

A production version needs more. It needs argument normalization, so equivalent calls match. It needs call-order keys, so a read repeated after a write returns its second recording. And it needs a fixture fallback. The shape stays the same.

The tool layer is only as good as what your traces record. For each tool span, log:

• The tool name and the full arguments, after any defaults are applied.
• The full output, or a pointer to it in blob storage when it’s large.
• Whether the tool is a read or a write.
• The wall-clock time and timezone the agent was given.
• The model, prompt version, and tool versions in use.

The Tenex guide already warns you to scrub sensitive data from promoted traces. Replay adds a constraint: scrub consistently, so the same customer ID maps to the same placeholder in the input and in every recorded output. Otherwise replay matching breaks.

Evals don’t replace bounded authority

A replayable eval suite tells you how the agent behaves on the cases you have. It can’t promise how it behaves on the next one. The Tenex guide makes the point precisely: passing k of k “doesn’t prove it can’t happen,” but failing once proves it can.

So the controls that limit damage belong in production, outside the model. Cap what a single run can do, such as one reschedule per conversation. Require approval above a threshold. Make writes idempotent. Agent frameworks and providers support this directly. The OpenAI Agents SDK’s needs_approval pauses a run until someone approves the tool call. Stripe’s restricted API keys for agents are subject to approval rules, and refunds are among the actions that need approval by default. Then the evals tell you how often the agent is right, and the limits decide what happens when it isn’t. Reliable agents need both, which is the argument of our post on why reliability takes design.

Frequently asked questions

Can’t I just rerun the input against a staging environment?

You can, but staging has its own state, which drifts too. A staging run tests whether the agent works against staging today, not whether it would have fixed last week’s failure. Use staging for integration tests, and use replay or fixtures for the regression suite.

Should I mock the model too?

Not for behavior evals, because the model’s decisions are what you’re grading. Scripted models are useful for testing the harness and graders themselves. Pydantic AI’s testing guide, for example, provides TestModel and FunctionModel for this, plus a switch that blocks real model requests in tests.

How many promoted cases should use fixtures?

Every case you plan to keep in the regression suite. Replay-only cases are fine while you reproduce and fix a failure. Once a case diverges after a deliberate fix, it has done its job as a replay case.

Where to start this week

Pick one flagged production trace that involved at least one write. Check two things in your tracing: are full tool outputs stored, and is the agent’s clock time recorded? If either is missing, fix the logging first, because nothing else in this post works without it.

Then make one change in your eval harness: every tool not on an explicit read list gets intercepted and recorded. That single default prevents the worst outcome: an eval run that books real technicians or moves real money.

If you’d like help making your agent evals replayable, or a review of how your agents touch production systems, talk to Moonlight AI.