VM Snapshot and Restore for Sandbox State Reuse

The default pattern in early agent infrastructure was simple and wasteful: spin up a fresh sandbox for every turn, or at best every session, and rebuild the environment from nothing each time. That means reinstalling packages, re-cloning repositories, reloading datasets, and re-establishing whatever process state the agent had built up. Cold boot itself takes around 3 seconds, and even a "warm" re-initialization on every turn burns 200 to 500 milliseconds on environment setup alone. Multiply that across an agent working a Python project over ten turns, one that's installed dependencies, written files, and accumulated intermediate outputs, and the tax turns from a latency annoyance into a design flaw. Agent tasks are also getting longer: sessions spanning many turns are increasingly common, and a re-initialization cost that scales with session depth is not something you can outrun by throwing more compute at it.

So the real question this raises is what it takes to make a sandbox remember where it left off, and how fast that can actually happen. That's what the rest of this piece works through, mechanism by mechanism.

What a VM snapshot captures and what "state" means at the OS level

A snapshot is a capture of the full execution state of a virtual machine, not a filesystem backup with a fancier name, and getting that right means preserving three layers together, not separately. It is a capture of the full execution state of a virtual machine, and getting that right means preserving three layers together, not separately.

The first layer is the filesystem: files written, packages installed, repositories cloned, whatever intermediate outputs the agent produced along the way. The second is process state: memory contents, open file descriptors, execution context, register state, the stuff that lives in volatile memory and would vanish the instant power cut out. The third is what you might call runtime residue: background processes the agent spawned, dependencies it installed mid-session, environment variables it modified without telling anyone. Miss any one of these and the restore is incomplete, even if it looks fine on the surface.

This is why application-level recovery, the kind that saves conversation history or git state, falls short. It misses everything happening on the OS side: a pip install that succeeded, a background process still running, a config file a shell command quietly edited. The Crab paper (arXiv 2604.28138, HKUST, April 2026) makes this distinction directly, and it matters because most early "recovery" systems in agent frameworks operate entirely at the application layer, produced by application-layer logic that cannot see the OS effects it depends on.

The goal is well understood: a snapshot should contain the complete virtual machine state such that, after restoration, processes and network connections carry on as if nothing happened. Firecracker's mechanism for this is pause, dump, restore. The microVM pauses, memory and filesystem state get dumped and serialized to storage, and restore reverses the process, streaming that state back and resuming execution from exactly where it stopped.

The filesystem side leans on copy-on-write as the load-bearing primitive. DeltaBox's DeltaFS, for instance, organizes file state into layers and freezes the writable layer at checkpoint time, inserting a fresh one on top. Rollback then becomes a layer switch rather than a full copy operation, which is a meaningfully different cost profile. DeltaBox's process-state counterpart, DeltaCR, uses incremental dumps and speeds up rollback by forking directly from a frozen template process, bypassing the traditional checkpoint/restore pipeline entirely, only falling back to CRIU's lazy-pages restore on a cache miss.

Consecutive checkpoints in agent workloads look a lot alike, and that's the insight driving all of this. Full duplication on every checkpoint is wasteful when only a sliver of state actually changed between two checkpoints (DeltaBox, arXiv 2605.22781). These terms get used loosely elsewhere, so precision about the vocabulary matters here. A full VM snapshot is a complete memory and filesystem dump, correct but expensive. An incremental or delta snapshot captures only the pages and filesystem layers that changed since the last checkpoint. A copy-on-write branch is something else again: an instant fork where nothing gets copied on the critical path, and pages only get duplicated lazily, at the moment something actually writes to them.

Restore latency versus cold boot across the main isolation technologies

Here's the number that reframes the whole problem. Cold boot runs around 3 seconds. Firecracker's snapshot-restore path runs 5 to 30 milliseconds. That's not an incremental improvement; it's a different category of operation. PandaStack's end-to-end VM creation via the restore path is p50 179ms and p99 around 203ms, with the restore step itself contributing roughly 49ms of that total.

Cold boot only happens once, on the first spawn of a template. Every VM created after that is a restore, not a boot. That distinction is easy to gloss over but it's the whole point: once a template exists, creating a new sandbox instance amounts to copying state rather than booting one, and state-copying is much cheaper.

DeltaBox pushes further into this territory. Its delta-based checkpoint/restore approach reports checkpoint time of 10.83ms, restore (rollback) time of 1.86ms, and metadata overhead on the order of 4KB per checkpoint, evaluated across SWE-bench tasks and RL micro-benchmarks. The paper benchmarks against three comparison points: Firecracker paired with dm-snapshot for copy-on-write filesystem handling, CRIU paired with plain file copy via shutil.copytree, and CRIU paired with Docker commit for the filesystem layer. Against all three, the delta-based approach comes out meaningfully faster, which is the paper's central argument: full-state duplication on every checkpoint is doing far more work than the workload actually requires.

TClone (arXiv 2605.17320) reports a different kind of number, aimed at computer-use workloads specifically: a 1.9x reduction in total task latency compared to KVM, and 1.5x compared to CRIU.

What "restore latency" includes and excludes matters, because vendor numbers get compared sloppily. Does the figure include network round-trip? Storage I/O to stream state back from wherever it's persisted? Is the sandbox pre-warmed in storage or genuinely cold? Blaxel has claimed sub-25ms resume from standby, measured under specific conditions, and workload composition changes what's comparable across vendors. None of this is a knock on any particular number; it is a reason to read benchmark claims with the methodology attached rather than the headline figure alone.

The architectural upshot, though, holds regardless of which exact number you trust most: restore latency has dropped low enough that creating a VM via restore is now viable as a per-turn primitive, not just something you reach for at the session boundary.

The agent–OS semantic gap: why most checkpoints capture state that doesn't need saving

Here's the gap the Crab paper (arXiv 2604.28138, HKUST, April 2026) names directly. Agent frameworks see tool calls, but not their OS-level effects. The OS sees state changes, but has no idea which turn produced them or whether that state matters for recovery. Neither side has the full picture, and that mismatch is expensive: the paper's findings show over 75% of agent turns produce no recovery-relevant state. Under a naive policy that checkpoints every turn, the overwhelming majority of that work is wasted.

Two extremes fail in opposite directions. Application-level recovery, the kind coding assistants and frameworks like LangChain or LlamaIndex tend to implement, is cheap to run but blind to OS-side effects: installed packages, spawned background processes, files a shell command modified outside the application's awareness. Full per-turn OS or VM checkpointing (CRIU, Firecracker, a full Docker commit on every turn) is correct in the sense that it captures everything, but it's semantics-oblivious. It treats a turn where the agent read a file and returned a string the same as a turn where it forked a background daemon, checkpointing both identically regardless of whether either one produced anything worth saving.

The cost of getting this wrong is not abstract. The Crab paper finds that chat-only recovery methods recover correctly on only about 8% of realistic agent workloads. Chat-only recovery methods recover correctly on only about 8% of realistic agent workloads, a system that fails to recover in the overwhelming majority of cases where recovery actually gets tested.

What counts as "recovery-relevant" is actually a fairly narrow, definable thing once you name it. A turn that reads a file and returns a string changes nothing an OS needs to remember. A turn that runs pip install or forks a background process changes state that every subsequent turn now depends on. The sparsity here, the fact that so few turns actually matter, is the finding. Most checkpoint traffic in a naive system is overhead dressed up as safety. It's overhead dressed up as safety.

How Crab and DeltaBox solve selective, low-overhead checkpointing

Crab's answer to the semantic gap is a three-part design, and each piece solves a distinct part of the problem.

An eBPF-based inspector runs at the host level and classifies each turn's OS-visible effects as they happen, deciding whether that particular turn warrants a checkpoint and at what granularity, all without touching the agent code or the checkpoint/restore backend underneath. A coordinator aligns checkpoint operations with turn boundaries and, critically, overlaps the checkpoint I/O with the time the agent is already waiting on an LLM response, so the checkpoint work never sits on the critical path. A host-scoped engine schedules checkpoint traffic across sandboxes running on the same host, which matters once you're running dense multi-tenant deployments where I/O contention becomes the bottleneck rather than any single sandbox's overhead.

The results, measured on shell-intensive and code-repair workloads: recovery correctness jumps from 8% under the chat-only baseline to 100%. Checkpoint traffic drops by up to 87%. And end-to-end latency overhead compared to fault-free execution stays within 1.9%. That last number is arguably the most important one. It says the selectivity isn't just cheaper, it's cheap enough to disappear into the noise of normal execution.

DeltaBox (arXiv 2605.22781, Shanghai Jiao Tong University and Huawei, May 2026) attacks the same underlying waste from a different angle, at the OS mechanism level rather than the scheduling level. DeltaFS handles the filesystem with layered copy-on-write: checkpointing freezes the current writable layer and drops a new one on top, so rollback is a layer swap rather than a copy operation. DeltaCR handles process state incrementally, and rollback works by forking from a frozen template process rather than replaying a full CRIU restore pipeline.

DeltaBox aims this specifically at two workload types: MCTS-style tree search at inference time, where you checkpoint at a parent node and restore repeatedly to explore each new leaf without re-executing the shared prefix, and RL training fan-out, where a single training step forks many parallel rollout sandboxes off one warm template. Its reported numbers, a 14ms checkpoint and 5ms rollback on SWE-bench and RL micro-benchmarks, translate directly into agents exploring more of the search tree under the same time budget.

Put the two papers side by side and the shared conclusion is hard to miss: full-VM checkpointing was never wrong, exactly, just wasteful, and both systems get to near-zero overhead by making the checkpoint mechanism aware of what actually changed rather than treating every checkpoint as a fresh, total capture. For anyone building this kind of infrastructure, the lesson is that the mechanism matters as much as the policy. A naive per-turn full snapshot can end up slower and pricier than the failure it's meant to protect against.

Snapshot for branching, speculative execution, and RL rollouts, not just fault tolerance

The conventional way to think about snapshots is defensive: snapshot as backup, restore as disaster recovery, something you reach for when a turn goes wrong. That framing is incomplete, and treating it as the whole story misses where a lot of the recent engineering effort has actually gone.

The other framing, increasingly the dominant one in 2026 agent infrastructure: a snapshot is a fork point, and a restore is how you create a branch. Three use cases fall out of this, and all three reduce to the identical underlying primitive.

Fault tolerance is the familiar case: checkpoint before a risky action or ahead of a preemption event, restore to a known-good state without re-running every prior turn. That's Crab's primary motivation. RL rollout fan-out is a second case, and a genuinely different one: at each training step, fork N sandbox branches off a single warm checkpoint. Tree-based RL algorithms, Tree GRPO among them, depend on being able to do this cheaply, because re-executing a shared prefix across every branch would be prohibitively expensive once N gets large. Speculative or MCTS-style execution is the third: checkpoint at each parent node, restore to explore a child branch, then commit or discard depending on what the outcome tells you. That's a key use case DeltaBox targets, and it only works if restore is fast enough that exploring a branch costs less than the information gained from exploring it.

TClone (arXiv 2605.17320, May 2026) takes this idea into computer-use agents operating live desktop environments, where a "branch" is not a lightweight sandbox but an entire running workspace: processes, memory, open files, GUI state, authenticated sessions, all of it. TClone's design separates online branch creation from durable checkpointing. A branch becomes runnable through copy-on-write sharing without a single page copied on the critical path, while serialization to disk happens asynchronously, off to the side. It's built on a modified Linux kernel paired with an extended version of CRIU.

Exposing this as a first-class capability, rollback as something the agent itself can reason about and invoke, opens a category of agent behavior that simply doesn't exist without a fast underlying runtime. An agent that can say "let me try this, and if it doesn't work, roll back and try the other thing" needs sub-20-millisecond fork and restore latency to make that decision cheap enough to be worth making. Above that threshold, the overhead of exploring a branch swamps whatever you'd gain from exploring it, and the whole idea collapses back into "just pick one path and hope." Which means any runtime that only supports linear checkpoint-restore, one timeline, one history, cannot serve tree search or RL fan-out workloads no matter how fast its single-path restore gets. Branching has to be designed in from the start, not bolted on after.

A decision framework for when to checkpoint in agent sessions

So where does that leave someone actually building this? Checkpoint granularity is fundamentally a cost-correctness tradeoff, and the right point on that tradeoff depends entirely on what the workload looks like.

Turn-boundary checkpointing, the naive default of checkpointing after every single turn, is correct but wasteful. Recall the earlier number: over 75% of turns produce nothing worth saving. It still makes sense in specific circumstances, though, namely when the agent's behavior is unpredictable enough that any turn could turn out to be the one that mattered, or when the cost of re-executing from an earlier point would exceed the checkpoint overhead anyway.

Effect-triggered checkpointing, the approach Crab takes, only fires when something OS-visible changes that a future turn will actually depend on: a package install, an apt-get, a file write to a path some later tool will read, a background process spawning. This is the more surgical option, but it comes with a requirement most teams don't have sitting around: a runtime that can observe OS-level effects directly, via eBPF or an equivalent mechanism. You cannot get this visibility from inside the agent framework alone, because the framework, by definition, doesn't see what the OS sees.

For teams without that kind of infrastructure, there's a coarser but still useful heuristic: action-class checkpointing. Checkpoint before anything destructive or hard to undo, a file delete, a database write, an external API call with real side effects. Checkpoint after any step that installs something or mutates the environment. And let the agent itself declare explicit safe points before it enters a subtask it flags as risky. None of this requires eBPF-level introspection, just a bit of discipline about which actions get treated as checkpoint triggers.

One mechanism ties all of this together and deserves its own mention: overlapping checkpoint I/O with the gap where the agent is already sitting idle, waiting on an LLM response. Crab's coordinator does exactly this, and it's the reason the system stays within 1.9% of fault-free execution overhead despite checkpointing selectively rather than never. The LLM call was going to take that time regardless. Spending it on checkpoint I/O instead of wasting it is free, since it never adds to the critical path.

None of these approaches is universally correct, and that's really the point of laying them out side by side rather than picking a winner. The right answer depends on how much OS-level visibility the runtime actually has, how expensive re-execution is if something breaks, and how deep the agent session runs. What all three approaches share, though, is the underlying premise this piece has been building toward: checkpointing is a primitive to be used well, not a tax to be minimized by avoiding it, and using it well means matching the granularity to what the workload can actually justify.