Copy-on-Write Forking for Pre-Warmed Sandbox State

CoW forking beats every other approach to sandbox provisioning available today, and the reason comes down to one design choice: pay the boot cost once, on a parent virtual machine, then hand every task after that a private, hardware-isolated copy of already-warm state in tens of milliseconds instead of tens of seconds. The mechanism, copy-on-write paging at the kernel level, wrapped around Firecracker microVMs, isn't new computer science at all. What changed is the workload: AI agents run for hours, install code nobody's actually read, and fork constantly, and that combination turned an obscure kernel feature into load-bearing infrastructure almost overnight.

Why the cold-boot model breaks under AI agent workloads

Cursor alone produces an enormous volume of accepted code every day. Behind each of those completions sits a sandbox, and increasingly that sandbox is a full execution environment running tests, installing packages, or evaluating a patch. Somewhere in the last two years, sandbox infrastructure stopped being a nice-to-have and became the thing everything else depends on.

The scale of the shift is worth sitting with. One provider went from roughly 40,000 sandbox runs a month in March 2024 to 15 million a month by March 2025, a 375x jump in twelve months. At 40,000 runs, a cold-boot cost of a few seconds per run is an annoyance you shrug off. At 15 million runs, that same per-run cost becomes a structural bottleneck: whole fleets of compute spent booting an OS, starting an interpreter, installing dependencies, warming a cache, over and over, for work that often finishes in under a second.

The work itself changed shape too. OpenAI's Codex runs regularly stretch to six hours, per the Codex engineering team's own notes. Every restart not backed by a snapshot means replaying that entire setup sequence again, mid-task, which is a strange place to be paying a cold-boot tax. Reinforcement learning training makes the same problem worse in a different way: each training step spins up sandboxes from an identical warm starting state, runs a rollout, tears the sandbox down, and repeats, sometimes thousands of times per run. Per the DeltaBox paper (arXiv:2605.22781), fork latency at that scale directly caps training throughput. Slow forks don't just waste compute. They set the pace of the whole training loop.

None of this maps cleanly onto the serverless model the industry spent a decade building. AWS Lambda caps execution at 15 minutes, a limit inherited from a design built around short, stateless HTTP requests. Agents aren't stateless request handlers. They're mid-session workers that install dependencies, edit files across a project, and need to pick up exactly where they left off, sometimes hours later. Forcing that workload through infrastructure built for HTTP handlers is where the cold-boot model starts to crack, and no amount of tuning fixes a mismatch that fundamental.

So if pre-warming a single sandbox solves the startup cost, what does it cost to pre-warm N independent sandboxes instead of one shared parent? That's the question the rest of this piece tries to answer.

Why containers are not a safe substitute for isolated forked microVMs

A standard container shares the host's kernel. That single fact carries the whole security argument here: a kernel exploit inside a container doesn't stay inside the container, it reaches straight through the boundary and onto the host. For workloads where the code is known and vetted ahead of time, that's a trade worth making. For AI agent workloads, it usually isn't, and treating the two cases the same is where a lot of teams get this wrong.

Code an agent runs is untrusted by definition. It might come from a model's own output, a user's prompt, a cloned repository, or a package pulled in during a build step, and in none of those cases has anyone on the operating team actually read and approved it before it runs. This isn't hypothetical. In September 2025, a self-replicating npm worm known as Shai-Hulud backdoored more than 500 packages, according to LangChain's engineering blog. A second wave in November hit 796 more packages and over 25,000 GitHub repositories, within hours. The malicious code ran in a preinstall script, before any validation step had a chance to catch it, which is exactly the code path an agent walks every time it installs a dependency on its own.

Then there's Copy Fail, tracked as CVE-2026-31431: a 732-byte Python script capable of rooting essentially every major Linux distribution shipped since 2017, by way of the kernel's crypto API. Per LangChain's engineering blog, AI tooling found this exploit in roughly an hour of automated searching. A container's shared-kernel model wouldn't have stopped it once it ran, because the whole point of the exploit is to escalate out through the kernel the container depends on.

The industry has increasingly moved toward hardware-enforced isolation for untrusted workloads. Not because containers are broken for what they were designed for, but because arbitrary code execution demands a stronger wall between tenant and host: a hypervisor boundary, separate kernel, separate memory space, enforced by the CPU's virtualization extensions rather than by how much anyone trusts the code running inside.

CoW-forked microVMs give an operator both properties at once. Each forked sandbox gets its own kernel behind a hardware boundary, and the spawn cost sits close to a plain process fork rather than a full cold boot. Hardware isolation running at fork speed is exactly what a shared-kernel container cannot offer, no matter how it's configured. Anyone weighing containers against microVMs for untrusted agent code is really weighing convenience against a wall that either holds or doesn't.

The isolation technology spectrum and where CoW forking fits

Isolation for untrusted code sits on a spectrum, and CoW forking occupies one narrow slot on it. It doesn't replace the whole ladder, and treating it as a drop-in substitute for every rung below it misreads what it's actually for.

At one end, standard containers work fine for batch workloads on infrastructure the operator controls, where the code is known ahead of time and a shared kernel is an acceptable risk. gVisor sits a step further out: it interposes on system calls in user space, adding a layer of isolation without the overhead of running a full second kernel, and it runs in production at real scale. One platform reports over 10,000 teams running workloads on gVisor-based sandboxes, supporting more than 100,000 concurrent sandbox instances. One vendor's isolates are fast and dense but tied to a single programming language, a poor match for agent workloads that need to run Python, shell commands, and arbitrary tooling side by side. WebAssembly sits closer to the sweet spot: polyglot, with fine-grained control over what a piece of code can touch, and speed that competes with gVisor on a lot of real workloads.

At the far end sit microVMs, with Firecracker as the reference implementation. This is maximum isolation, a genuinely separate kernel per sandbox, enforced by the hardware's virtualization instructions rather than by software policy. It's also the foundation the whole CoW forking approach sits on top of, and Firecracker is the same open-source project already powering AWS Lambda's own isolation model, now repurposed for agent fan-out instead of HTTP request handling.

CoW forking, then, sits apart from this ladder rather than forming a sixth rung on it. Think of it as an optimization bolted onto the microVM rung, one that keeps the full hardware isolation guarantee while stripping out the cold-boot cost that would otherwise make microVMs too slow to use at agent scale.

Google's GKE Agent Sandbox shows a related but genuinely different approach: a warm-pool model rather than true CoW forking. A developer defines a SandboxTemplate, a reusable blueprint, and a SandboxWarmPool, a set number of pre-warmed Pods kept running at all times. When a SandboxClaim comes in, an available Pod gets pulled from the pool, and the pool refills itself in the background. That gets fast claim times, but it does so by brute force, keeping real, fully independent instances warm and idle, waiting on standby for work that may never come. That's a more expensive tradeoff than CoW, since every Pod in the pool is a fully separate running environment with no memory shared between instances at all. Warm pools hold N independent environments running in parallel and pay for all N regardless of use. CoW forking holds one parent and only pays for the pages each child actually changes, and that gap only widens as N grows.

How the forking mechanism plays out in measured benchmarks

The mechanism itself is straightforward once it's laid out. Each forked child is its own Firecracker process, and it maps the parent's memory image into its address space using mmap with MAP_PRIVATE. The kernel handles copy-on-write at the page level from there: children share every one of the parent's resident memory pages, read-only, until one of them writes to a page. At that point the kernel allocates a private copy just for that page, just for that child. Everything untouched stays shared, so a child that never touches its Python interpreter's core pages never pays to duplicate them.

The published numbers back up how cheap this gets in practice. forkd, an open-source Firecracker-native runtime, reports forking 100 VMs from a warmed parent snapshot in 101 milliseconds, with a 100 out of 100 success rate across its benchmark runs. CubeSandbox reports a comparable fast-path figure on bare metal: 1.06 seconds wall-clock for 100 forks (1,056 milliseconds plus or minus 14ms across five runs), all succeeding, with the overhead sitting in the surrounding infrastructure rather than the CoW mechanism itself.

What's more interesting is forking a VM that's already mid-task, not just mid-warm-up. forkd's BRANCH operation, which pauses a running sandbox and forks from that exact live state, clocks in at 56 milliseconds at the 50th percentile and 64 milliseconds at the 90th, on a 1.5 GiB source, in forkd's v0.4 live mode. That's forking mid-thought, not just at the starting line.

Production systems accumulate scar tissue, and forkd's own changelog has an honest example of it. A regression in v0.3.4 let repeated BRANCH operations against the same parent balloon from 150 milliseconds up to 2.7 seconds, before a fix landed. The cause traced back to how diff snapshots accumulated against the same parent, and it's a useful reminder that CoW's elegance on paper doesn't excuse an implementation from careful bookkeeping.

That bookkeeping shows up clearly in forkd's v0.5 diff-snapshot benchmarks, run on a 512 MiB base image over ext4 on an i7-12700. A flat spawn at depth zero costs 59 milliseconds. Add a numpy layer (depth one) and it jumps to 751 milliseconds, a 692-millisecond tax for that one link in the chain. Add pandas on top (depth two) and it's 1,222 milliseconds, another 471 milliseconds. Add scikit-learn (depth three) and it's 1,668 milliseconds, 446 milliseconds more. A flat-equivalent build with all three packages baked into a single diff comes in at 1,746 milliseconds, roughly the same as the chained version, which says the chained approach trades a bit of spawn latency for real storage savings. It doesn't get either for free, and correctness across those runs held at 90 out of 90 probe passes regardless.

Morph Cloud's Infinibranch reports snapshotting an entire running VM and branching or restoring it in under 250 milliseconds, with each branch picking up from the exact parent state at the moment of the fork, nothing to reinstall or reconfigure. And even outside true CoW memory sharing, pre-warming alone moves the needle by orders of magnitude: one production deployment reports cold starts as low as 27 milliseconds using pre-warmed pools, through the discipline of keeping instances ready to go.

The research frontier: incremental CoW for checkpoint and rollback during execution

Provisioning speed solves half the problem. It doesn't solve what happens once an agent is deep into a long session and needs to checkpoint its state, or roll back after a bad decision, without redoing all that setup work again. That's the gap DeltaBox, a paper out of Shanghai Jiao Tong University and Huawei published in May 2026 (arXiv:2605.22781), sets out to close.

The insight is almost embarrassingly simple once stated: two consecutive checkpoints of the same agent session usually differ by very little. A handful of new files, a small number of modified memory pages. Duplicating the entire sandbox state at every checkpoint, the way naive snapshotting does, means paying to copy megabytes of data that hasn't actually changed since the last one.

DeltaBox splits the fix into two co-designed pieces. DeltaFS handles the filesystem side: at each checkpoint, it freezes the current writable layer and slides a fresh, empty layer on top. Every file modification after that becomes a copy-on-write operation against the new layer, and rolling back just means switching which layer is active. DeltaCR handles process state the same way, using CRIU dumps that capture process state at each checkpoint, while also forking off a frozen template process at checkpoint time that later restores fork from directly.

The measured numbers, from SWE-bench runs and a set of RL micro-benchmarks, land at roughly 10.83 milliseconds for a checkpoint (short enough to hide entirely under normal model inference time) and about 1.86 milliseconds for a rollback, against approaches needing hundreds of milliseconds to multiple seconds for a full-state checkpoint or restore.

Why does 1.86 milliseconds matter this much? Consider what tree search actually does. Search-and-backtrack reasoning backtracks constantly: testing a branch, discarding it, trying another. If rollback sits on the critical path and costs a few hundred milliseconds or more, it directly caps how many nodes an agent can explore inside a fixed time budget, since every backtrack eats into that budget. At 1.86 milliseconds, rollback effectively stops being a cost worth budgeting for at all. DeltaBox is a signal of where this area is heading: CoW not as a one-time trick applied at boot, but as a continuous primitive running underneath the agent for the entire length of its session.

What CoW forking enables that changes how agent workflows are designed

Once forking gets cheap and safe, certain workflow patterns stop being theoretical and start being things teams actually ship. Search-and-backtrack reasoning can branch an agent at a real decision point, run several trajectories in parallel from that exact state, and keep whichever one produces the best outcome, without rebuilding the surrounding environment separately for each branch. A/B experimentation on agent behavior can fork from one known-good state, apply a different intervention to each fork, and compare results with no risk of one branch's side effects leaking into another's. RL rollouts can spin up sandboxes from the same warm starting state over and over, run, tear down, repeat, without fork latency setting the pace of the whole training loop, which is exactly the bottleneck DeltaBox and forkd's benchmarks both target.

LangSmith Sandboxes, part of LangChain, build snapshotting and forking in as first-class operations rather than an afterthought. Per LangSmith's engineering blog, a session can be captured mid-run and used to boot new sandboxes from that exact point, and because forks use copy-on-write under the hood, spinning up ten parallel branches costs roughly what spinning up one does. Blueprints define a pre-warmed base environment, repository already cloned, dependencies already installed, so new sandboxes boot in seconds instead of minutes even before a fork happens.

forkd's own demo makes the mid-session case concrete. A LangGraph agent runs a ReAct loop, gets BRANCHed mid-thought, and three children spin off from that exact point, each given a different steering hint, producing independent downstream outputs from the same prior state. That's the detail worth sitting with: the forked children inherit the parent's full memory state at the moment of the fork.

That mid-execution BRANCH capability is really what separates CoW forking from a warm pool of pre-started instances. A warm pool gets a task started fast. BRANCH lets an agent fork at any arbitrary point deep into a session that's already run for hours, which matters given that Codex sessions regularly run six hours long. If an agent heads down a bad path two hours in, forking from a mid-session snapshot and trying a different direction costs milliseconds. Restarting from scratch costs the two hours, and that gap is really the whole argument for building on BRANCH instead of settling for a pool.

There's a GPU angle worth naming too, per LangSmith's engineering blog: when sandbox provisioning is near-instant, GPU capacity doesn't sit idle while CPU-side setup work grinds through a boot sequence. Fast forks turn into a GPU utilization multiplier once an operation runs at real scale, since the expensive hardware stops waiting around for the cheap hardware to catch up.

How current platforms implement pre-warmed CoW forking and where they differ

forkd is the clearest open-source, Firecracker-native implementation of this whole mechanism, and its benchmark numbers have been threaded through this piece already: 101 milliseconds to fork 100 VMs, 56 milliseconds p50 to BRANCH a live VM, and diff-snapshot chains that stack dependency layers on top of a base image without duplicating that base each time. It comes with real operational requirements, though. It needs Linux kernel 5.7 or newer, either CAP_SYS_PTRACE or the vm.unprivileged_userfaultfd=1 sysctl set, and a vendored fork of Firecracker rather than stock upstream. It ships SDKs for Python, TypeScript, and MCP, gives each child its own network namespace, and is meant to be self-hosted rather than consumed as a managed service. Its own v0.5 changelog is upfront about a known cost too: chain-spawn's per-link tax, roughly 460 milliseconds tied to SHA-256 verification on a 512 MiB image, is a real, documented tradeoff of the layered-diff approach, not something buried in fine print.

Where forkd, CubeSandbox, and Morph Cloud's Infinibranch actually differ isn't whether copy-on-write forking works. The benchmarks across them tell the same story: sub-second, often sub-100-millisecond forking from a warm parent is achievable and repeatable, full stop. The real differences show up a layer above the mechanism: how chains of diffs get managed as dependencies stack up, how mid-session BRANCH gets exposed to a developer building an agent, and how much Firecracker plumbing an operator has to manage directly versus consume through a managed API. That's the part still being actively shaped, not the kernel trick underneath it, and it's worth watching closely over the next year. Page-level copy-on-write already proved itself. What's still getting worked out is how to wrap it so a team building an agent can reach for it without needing to understand Firecracker internals first.