Testing & replay

Which to use

Recording a run

agentloom run workflow.yaml --record recordings/run1.json

The run calls its providers as usual and saves each answer to the file, creating its folders if needed:

• Every model call, fallbacks included: plain calls, streamed calls once the stream ends, and embeddings. A stream that fails or is cut short isn’t recorded.
• The run’s event log, under _inter_agent_events.
• Tool results, with --record-tools. See Recording tool results.

A step that was retried keeps the calls of its last attempt only. A recording made with --stream replays without it, and the other way round.

The file is written after every call, so a run cut short keeps the calls it made, but not its event log, which is written when the run ends. --record refuses a workflow whose provider is mock.

Recording into an existing file replaces it: nothing from an earlier run stays. The file is cleared just before the run starts, so a run that can’t start leaves it alone.

What a recording holds

A recording is a JSON object with an entry per step:

{
  "_version": 2,
  "summarize": {
    "content": "Revenue grew 12% on subscription renewals.",
    "model": "gpt-4o-mini",
    "provider": "openai",
    "usage": {"prompt_tokens": 412, "completion_tokens": 18, "total_tokens": 430, "reasoning_tokens": 0},
    "cost_usd": 0.0000726,
    "latency_ms": 812.4,
    "finish_reason": "stop",
    "request_hash": "6054bd…"
  }
}

No field is required: an entry written by hand can hold content alone. Keys starting with _ aren’t steps: _version, _tools and _inter_agent_events. A file without _version loads; one with a version below 2 is refused.

• Entries are keyed by step id. A conversation’s summary call is recorded under <step>::summarize, and the calls of an agent used as a tool under <step>::<tool>#<n>. A subworkflow’s steps are keyed by their path, <subworkflow step>/<step>, and so are their tool results.
• A step that calls the model several times, in a tool loop or to re-ask for a valid structured reply, gets a list of turns, in call order.
• A call made from Python without a step id is keyed by a hash of its messages, model, temperature, max_tokens and other arguments.

Replaying a run

agentloom replay workflow.yaml -r recordings/run1.json

Every model call is answered from the recording, and no provider is called, even with its key set. The model’s tool calls still run, unless you replay their recorded results. Each replay writes a run history record, and OpenTelemetry stays off unless you pass --observability.

The CLI reference lists every option.

Strict replay

replay is strict: a call the recording can’t answer fails its step at once, without retries, and the run exits with 1.

• A step missing from the recording fails with No recorded response for step 'review'.
• A changed request fails with the replayed request (hash …) does not match the recording (hash …). Any change counts: the prompt, the state it renders, the model, the temperature, the tools or the schema.

The hash covers the model, so a recording made with --provider or --model replays with the same options. Only entries with a request_hash can catch a change. run --record writes one on every call, embeddings included; an entry written by hand can leave it out, and then matches by step id alone. Of the example recordings, only recordings/tool_calling.json has none: the request of its second turn holds a live tool result. An embed step fails at once when the number of inputs isn’t the number of recorded vectors, with a hash or without.

In a list of turns, strict replay picks the turn whose hash matches the call: a retried step starts its turns again. A list without hashes is served in order, one turn per call, and the call after its last turn fails with the recording holds N turns.

The hash is taken on the real request. A request that holds a masked value, from a redacted tool call replayed earlier in the step, can’t match it, so that turn is served in order, unchecked.

Lenient replay

Lenient replay answers what it can and never fails a call: a missing step gets Mock response, or {} for a step with a response_schema, with a warning, and a changed request gets its recorded answer, with a warning too. Lists are served in order, and repeat their last turn once they run out.

For CI, use replay or verify-determinism: a lenient run passes on a stale recording.

The mock provider in YAML

A workflow can replay a recording with plain agentloom run:

config:
  provider: mock
  model: mock-model
  responses_file: recordings/run1.json
  latency_model: constant
  latency_ms: 0

• The replay is lenient. For a strict one, run agentloom replay on the same recording.
• responses_file is relative to the working directory, not to the workflow file. Run from the folder the path assumes: from anywhere else, the file isn’t found.
• A recording that doesn’t exist or doesn’t parse stops the run before its first step, with an error and exit 1.
• Unlike the other providers the CLI sets up, the mock has no rate limit.

Examples 35 to 41 run this way against the recordings in recordings/. examples/32_yaml_mock.yaml names recordings/byzantine.json, which isn’t in the repository: make it first with example 31, or the run stops with an error.

Latency

Only a provider: mock workflow and a MockProvider built in Python wait before answering. replay, run --mock-responses and verify-determinism answer at once, whatever the workflow sets.

Recording tool results

By default a replay runs the model’s tool calls again. To replay them too, record their results:

agentloom run workflow.yaml --record recordings/run1.json --record-tools
agentloom replay workflow.yaml -r recordings/run1.json --tools replay

--record-tools saves, under _tools, each tool call the model made, per step and in order: its name, its arguments, the result the model saw, and whether it succeeded. Only model tool calls are recorded, from an llm_call or agent step’s tools:. A tool step always runs, in every mode.

• A call matches the next recorded call of its step with the same name and arguments; recorded calls skipped on the way aren’t served. A retried step starts its recorded calls again.
• With --allow-default-fallback, a missing or different call logs a warning and uses the live result.
• verify-determinism takes the same --tools option. With live, tools run once per replay.

Replaying up to a cut point

Replay a run up to a step and call the providers from there, to try a new prompt or model on the end of a workflow without paying for its start:

agentloom replay workflow.yaml -r recordings/run1.json --live-from summarize

• --live-from STEP runs that step and every step that depends on it live. --force-live STEP runs one step live, and can be repeated.
• Other steps replay until one of their calls misses the recording, and run live from that call on.
• Only the steps that run live need a provider and its key. A step replayed from the recording needs neither; a step that has to run live on a model with no provider fails, naming the model.
• --record FILE saves the run to a new recording: replayed calls as recorded, live calls as made. With --record-tools, tools too. Into a file that exists, it replaces the entries of the steps it called and keeps the rest, unlike run --record, so give each cut a file of its own when you mean to verify it.
• --json adds a calls list: for each call, its step, whether it was replayed or live, and why (hit, miss or forced).

A cut can’t be combined with --allow-default-fallback.

Injecting faults

An entry, or a turn of a list, can hold a fault instead of an answer. Every replay then fails that call the same way, which tests retries and fallbacks without waiting for a real outage. examples/40_fault_replay.yaml replays this recording:

{
  "_version": 2,
  "draft": [
    {"fault": {"type": "http_error", "status_code": 429, "retry_after_s": 0.1}},
    {"fault": {"type": "http_error", "status_code": 503, "body": "upstream overloaded"}},
    {"content": "Release 1.4 is in final testing and ships Thursday."}
  ]
}

The first two attempts of draft fail, and the third gets the answer: the step’s retry allows two retries.

Each failure is retried or not as a real one would be: see What is retried. A malformed fault is refused when the recording loads.

• A list without hashes, as written by hand, is served in order, one turn per call, and a retry carries on from the next turn. So a retry doesn’t go back to the start of a tool loop: with [a turn that calls a tool, a fault, an answer], the retried step gets the answer first and never calls the tool.
• A single entry with a fault fails every attempt: retries can’t recover.
• In a recorded list, with hashes, under strict replay, a fault turn fires when the call reaches it, and the retry starts the step’s turns again.
• A 429’s retry_after_s is honoured like a provider’s Retry-After: see Retries.
• Words a stream_truncate fault delivered are already printed when the retry streams its answer.
• embed steps ignore faults.
• Faults fire in the replayed steps of a cut too, without sending them live.

Inter-agent events

run --record saves the run’s event log under _inter_agent_events: what each step received, which tools it called, what it answered and what it wrote to the state. It is the same in every run that does the same thing, so verify-determinism compares it across replays.

For each attempt of a step, the log has:

• a task_sent from each finished step it depends on, and its own task_received;
• a tool_call and a tool_return for each tool call, a tool step’s included;
• when it succeeds, a response_sent, and a state_transition for each state key it wrote.

An agent used as a tool adds a task_sent from its caller, then its own task_received, tool events and response_sent. A subworkflow’s steps are named <step>.<child step>. A retried step keeps the events of its last attempt only.

Events are ordered by their causes, then by step and type, so steps that run in parallel give the same log whichever finishes first.

From Python, agentloom.contracts.experimental exports the event models, InterAgentEvent among them, and SCHEMA_PATH, the event log’s JSON Schema. Importing it warns once that the module may change.

Verifying a recording

agentloom verify-determinism workflow.yaml -r recordings/run1.json --runs 3

It replays the recording strictly --runs times and compares the runs: the status, each step’s status, output and error, the final state and the event log, which must also match the recorded one. --events-out FILE writes the first replay’s log as JSON Lines. A difference prints a diff and exits with 1, and so does a replay that doesn’t end in success. To accept a stable failure, pass --allow-failed. See the CLI reference.

Testing from Python

AgentLoom ships no pytest plugin or fixtures. A test builds the engine with a strict MockProvider:

import anyio
import pytest

from agentloom.core.engine import WorkflowEngine
from agentloom.core.parser import WorkflowParser
from agentloom.core.results import WorkflowStatus
from agentloom.core.state import StateManager
from agentloom.providers.gateway import ProviderGateway
from agentloom.providers.mock import MockProvider


@pytest.fixture(autouse=True)
def _history_in_tmp(tmp_path, monkeypatch):
    # Every run writes a history record
    monkeypatch.setenv("AGENTLOOM_RUNS_DIR", str(tmp_path / "runs"))


async def replay(workflow, recording, **state):
    wf = WorkflowParser.from_yaml(workflow)
    gateway = ProviderGateway()
    gateway.register(MockProvider(responses_file=recording, strict=True))
    engine = WorkflowEngine(
        wf,
        state_manager=StateManager(initial_state={**wf.state, **state}),
        provider_gateway=gateway,
    )
    try:
        return await engine.run()
    finally:
        await gateway.close()


def test_replays():
    result = anyio.run(replay, "workflows/report.yaml", "recordings/report.json")
    assert result.status is WorkflowStatus.SUCCESS


def test_a_changed_topic_misses_the_recording():
    result = anyio.run(lambda: replay("workflows/report.yaml", "recordings/report.json", topic="dogs"))
    assert "does not match the recording" in result.error

• Pass strict=True. A MockProvider is lenient by default, and a test then passes on Mock response with only a logged warning.
• Assert on result.final_state, where keys marked redact read as <REDACTED:sha256=…>: check those through a step’s output instead.
• Paths are relative to the working directory: responses_file, sandbox paths and agent files. monkeypatch.chdir sets it for a test.
• @pytest.mark.anyio on an async def test works too, with the plugin that comes with anyio.

MockProvider takes latency_model and latency_ms as the YAML does, plus seed, which makes normal waits repeat, and default_response, the placeholder text. Its calls list records each call it answered and how: step_id, prompt_hash or default.

To drive the CLI from a test, typer.testing.CliRunner invokes agentloom.cli.main.app. With --json, result.stdout holds only the JSON.

Metrics

With OpenTelemetry on, replays and recordings report:

replay reports them only with --observability, and verify-determinism never does. The bundled Grafana dashboard has a “Mock & Replay” row. See Observability.

Limits

• Two processes recording to the same file at once can lose each other’s entries.
• A changed request goes unnoticed in entries without a request_hash, and only warns in a lenient replay.
• A recording made before subworkflow steps had paths holds them under their bare id: they replay from it, and share the entry with a top-level step of the same id.
• A masked tool argument replays as its <REDACTED:…> sentinel: a tool run live gets that as its argument, so a masked call only replays faithfully with --tools replay.

What’s next

• CLI

  Every option of run --record, replay and verify-determinism.

• Reliability

  The retries and fallbacks that injected faults exercise.

• Observability

  Traces and metrics of live and replayed runs.