Engineering
Ayman S.
How Chronicle uses portable Linux microVMs, immutable world artifacts, and locked-down networking to run thousands of deterministic agent evaluations.
Evaluating an agent against one production trace is straightforward. The problem changes when a backtest includes several agent versions, hundreds of traces, retries, and fault scenarios. A single job quickly becomes thousands of independent trials.
An evaluation score is only meaningful if the world around the agent stays fixed. Every trial must begin from the same state, remain isolated from every other workspace, and be unable to reach services the test did not declare. It also has to be cheap enough to create for one run and destroy immediately afterward.
Shared staging environments are a poor fit. State accumulates between runs, failures become difficult to reproduce, and results drift as the environment changes underneath them.
At Chronicle, we now run these trials in portable Linux microVMs with smolvm. Each trial starts from a prebuilt .smolmachine artifact, boots in under 200 milliseconds behind a hypervisor boundary, and records everything the agent does as an ordinary Chronicle trace.
This is how the system fits together, and why moving the environment into an immutable artifact changed the way we run evaluations.
Why the trial matrix grows
Chronicle captures the activity around AI agents from integrations, SDKs, and production traces.
Teams curate production traces into Datasets and register the agent versions they want to compare, and a backtest runs every selected version against every selected case inside a deterministic world, then sends the resulting traces to a trusted evaluator for scoring.
Three versions across a few hundred cases, with retries and fault scenarios, quickly fan out into thousands of trials. Each one needs to be:
Deterministic: the same agent should encounter the same initial state and recorded behavior every time.
Isolated: one run must never see another workspace’s data or mutate another trial’s state.
Disposable: creating a clean environment should be cheap enough to do once per trial.
Shared staging infrastructure breaks these properties almost immediately. State changes between runs, concurrent trials interfere with one another, and a score that looked reliable last Tuesday becomes difficult to reproduce.
We also treat the agent under test as untrusted. Increasingly, its code may itself be AI-generated. We wanted every trial behind a hypervisor boundary rather than inside a shared-kernel container, with no network access as the physical default, not as a firewall rule we hope was applied correctly.
A world in a file
We bake the bundle and its runtime into a persistent machine, initialize it once, seed its datastores, install its packages, stop it, and publish the result as a .smolmachine artifact through an OCI registry. Because persistent smolvm machines retain their installed state across stop and restart, the artifact is a snapshot of a world that is ready to run.
Forking that world means rehydrating the same artifact as many times as the job requires. Every trial begins from byte-identical state. Determinism stops being something we reconstruct for each run and becomes a property of the artifact itself.
That gives us a clean counterfactual comparison. A baseline agent and a candidate agent start from the same world and receive the same recorded traffic; only the agent changes. Some versions take different routes through a vendor API, so we record companion captures in advance to keep those alternate code paths inside the world rather than letting them fall through to a live service.
A machine is configured with a Smolfile: declarative TOML describing its image, resources, networking, mounts, and initialization commands. The values below are illustrative.
The first 200 milliseconds
A trial begins when the orchestrator selects the next case from the pinned Dataset Version and asks the sandbox driver for an environment.
smolvm rehydrates the corresponding .smolmachine artifact. libkrun starts a Linux guest on the host’s native hypervisor, and in under 200 milliseconds the trial has its own running kernel—one that has never seen another workspace’s data and never will.
Inside the guest, worldctl, our runtime lifecycle tool, brings up worldd, the world server. worldd serves the compiled bundle: seeded Postgres state and the interception layer between the agent and everything it believes is the internet.
The agent makes a tool call. From its point of view, it is calling a vendor API. In reality, worldd answers from captured production traffic. As the trial runs, it appends every SQL and tool interaction it serves to an append-only interaction ledger. The ledger’s hash makes the run auditable.
If the agent tries to reach an authority the environment did not declare, the request is blocked. If it attempts a write the world cannot support, worldd refuses it.
When the trial finishes, the machine is destroyed. The rest of the matrix is doing the same work in parallel, with each trial running behind its own kernel boundary.
The same environment in the fleet and on a laptop
Our orchestrator already drives trials through a sandbox-driver abstraction, with Daytona and Docker implementations behind it. smolvm is another driver behind the same interface, so adopting it did not require changing how the rest of the evaluation system schedules or observes a run.
The portability matters more than the abstraction alone. In the fleet, smolvm runs on KVM. On an engineer’s Mac, it runs on Hypervisor.framework. It does not require a daemon, and the same .smolmachine artifact can be rehydrated on either supported host when the host architecture matches.
Many microVM tools are designed primarily for Linux server fleets. smolvm’s laptop path means that reproducing a failed trial is a pull and a boot, not a request for access to a shared staging environment. Trial 47 can run locally from the same artifact that ran in the fleet.
At that point, an environment starts to feel less like infrastructure you manage and more like a function call.
Networking starts closed
Every Chronicle world carries one of three egress policies:
deny_all permits no network access.
mocked_only permits only the interception path, making the replay index the agent’s entire internet.
passthrough_record permits a specific set of vendor domains during an authorized recording run.
worldd enforces the policy inside the guest. smolvm adds a second, independent layer at the VM boundary: networking is disabled by default, and enabling it requires an explicit domain allowlist.
This defense in depth protects against the quiet failure modes that are hardest to notice. A trial can appear healthy—every call returns 200, the agent reaches a plausible answer—and still be invalid because it silently touched a real API. Once live traffic leaks into an evaluation, the resulting score is no longer trustworthy.
smolvm’s default machine size of 4 vCPUs and 8 GiB is generous for a single trial, so we size machines down in the Smolfile and use virtio-balloon’s elastic memory to keep trials dense. The boot cost is low enough that an environment only needs to exist for the lifetime of its trial.
The run becomes an ordinary trace
Isolation and speed are useful on their own. The more important property is that the microVM does not become a separate observability system.
Every tool call, model response, and intercepted exchange flows back through Chronicle’s normal ingest routes. The SDK—including framework wrappers such as @chroniclelabs/ai-sdk—instruments the agent loop under a scoped chr_ key, while OTLP carries the surrounding traces. Both converge on the same insert_events write path used for production traffic.
Deterministic event IDs and tenant scoping are enforced in the same place. Progress streams over SSE. When the run finishes, it reads back as an ordinary trace with stable (event_time, event_id) ordering, directly beside the production timeline it was replayed from.
This symmetry is the center of the architecture:
One artifact records what the environment is.
One trace records what the agent did.
A trusted evaluator, running outside the untrusted sandbox, applies rubric graders to the trace. Those scores can gate which agent version ships and feed RL-style improvement loops over the same frozen worlds.
Failed trials return to the data loop. Teams curate them into new Dataset revisions, turning today’s hidden production failure into tomorrow’s permanent evaluation case. As long as the Dataset Version exists, that case can be replayed inside an identical microVM.
When environments cost milliseconds instead of minutes, evaluation stops being something teams ration. You can backtest every version and replay every incident.
From replayed worlds to learned ones
We are moving more of the trial fleet onto .smolmachine distribution so that a Chronicle environment version and its runnable machine image become the same artifact.
We are also watching smolvm’s GPU path—Vulkan through virtio-gpu and CUDA API remoting—as a way to bring heavier scoring models inside the isolation boundary rather than running them beside it.
Beyond that, the same substrate can support learned worlds: simulators induced from production logs rather than compiled only from recorded traffic. The source of the world changes, but the operating model remains the same—snapshot, boot, run, observe.
The broader lesson is that production failures are already some of the most valuable evaluation cases an agent team can have. The hard part is replaying them inside a world that is identical, isolated, and cheap enough to use continuously.
smolvm makes those worlds cheap. Chronicle turns them into repeatable evaluation data.
smolvm is open source under Apache 2.0. Chronicle’s sandbox tier is free forever.
