Sandbox Support for AI Model Inference at Runtime

Sandbox support for AI model inference at runtime is about executing code that no developer has reviewed before it runs. That makes the sandbox a mandatory runtime layer. Its design must combine hardware-level isolation, fast provisioning, and a session lifecycle that preserves long-running work without leaving persistent credentials within the guest’s reach.

Why inference-time code execution is a different problem than pre-written code

Traditional application code has a fixed, auditable instruction set determined at compile or deploy time. Autonomous agents instead write new code from natural language inputs and run it during inference. The review boundary disappears before execution begins. A developer may review the application that hosts the model, but that does not amount to reviewing each script the model will produce.

This changes what the runtime must assume about its guest. An agent that can write scripts, install packages, and work with file descriptors can reach many parts of the operating system. Standard containers keep processes apart but share the host kernel, so they still depend on that kernel no matter how they are configured. The architectural constraint holds regardless of code volume. Sandboxing must contain potentially hostile behavior at execution time, because advance inspection is no longer the gate through which every instruction passes.

What isolation actually means across the technology stack

Docker and Podman use Linux namespaces and cgroups to separate processes that share a host kernel. They are fast and work well for trusted code written in-house. By contrast, gVisor intercepts and re-implements syscalls in user space, placing a software layer between the application and the real kernel. That keeps workloads further apart than ordinary containers do, with less overhead than a full VM, though some Linux software runs into compatibility edge cases.

Firecracker, Kata Containers, and libkrun use hardware virtualization through KVM so that each workload has its own kernel. Firecracker boots in roughly 125ms with very low memory overhead, while Kata Containers add a VMM layer and boot in roughly 150–300ms depending on the VMM. Here, the guest kernel is itself inside the boundary. Confidential computing also protects memory through technologies such as AMD SEV-SNP and Intel TDX, so even the host OS and hypervisor cannot read sandbox data, though this tier is still experimental and not yet ready for production in this map.

V8 isolates draw a different kind of boundary. Cloudflare Workers virtualizes the JavaScript runtime rather than hardware or a kernel, with microsecond startup. Its execution model is stateless by design and native to JavaScript and TypeScript. That suits short-lived edge tasks rather than general-purpose, multilingual agents needing persistent state.

Why microVM isolation is the minimum viable tier for inference-time code execution

For unreviewed inference-time code, the guest kernel must sit inside the untrusted environment. That makes hardware-enforced kernel separation the least that general-purpose execution needs. A sufficiently determined exploit can bypass gVisor’s software boundary, whereas a microVM places hardware virtualization between the guest kernel and the host. With that hardware boundary in place, taking over a guest kernel does not by itself give an attacker access to the host or nearby sandboxes.

Cloud-provider designs provide the strongest evidence for this choice, although their technologies are not identical. AWS built Firecracker for Lambda and Fargate, while Azure uses Hyper-V for hardware-enforced isolation. Google built gVisor for App Engine, Cloud Run, and Cloud Functions, but its agent-specific infrastructure also adopts hardware separation. Google Cloud announced Agent Compute Engine at an industry conference on October 22, 2026, with lightweight hardware-isolated microVMs. Its Titanium Infrastructure Processing Unit holds attestation keys outside the guest entirely, making the hardware boundary relevant to both execution and trust.

Firecracker and Kata Containers boot measurably slower than containers or gVisor at millisecond scale, and cold-start costs accumulate at high concurrency. That is a genuine deployment constraint. The answer in the runtime designs described here is warm pools and snapshot/restore, rather than relaxing the boundary around untrusted code.

How provisioning speed determines whether sandboxing is practical at inference scale

A secure sandbox that takes seconds to start on every invocation can still undermine an agent’s responsiveness. Raw VM boot time is only part of that delay, because environment creation includes the full orchestration cycle. The relevant measurement is time until usable execution.

Google’s GKE Agent Sandbox, generally available May 20, 2026, uses a warm pool to allocate sandboxes at high throughput, with the large majority of allocations completing within sub-second latency. AWS Lambda MicroVMs, launched June 22, 2026, takes a snapshot-based approach. Lambda initializes the application once and snapshots memory and disk, then resumes subsequent Firecracker VMs from that snapshot. With these methods, hardware isolation is practical because each invocation can avoid the full cold-start delay.

Agent Compute Engine similarly combines high activation throughput with low suspend/resume latency, validated in a large-scale benchmark. Agent Substrate, its open-source sibling native to a container orchestration platform, achieves significantly higher compute density than standard container runtimes while retaining per-agent kernel isolation. Density belongs in the same operational calculation as latency. Warm pools and snapshots are therefore not merely refinements to an otherwise complete runtime. They are the mechanisms that make its isolation model usable and economically viable at inference scale.

Why stateful sandboxes are required for long-running model workloads

Fast allocation solves the beginning of a session. Agents that run pipelines or debug iteratively need files and execution state to survive between steps, through idle periods, and across restarts. Ephemeral environments cannot support that work by themselves. A longer timeout postpones destruction, but it does not create an architecture for suspending and restoring a working session.

Execution caps expose this distinction clearly. Lambda’s 15-minute hard limit and Vercel’s execution cap constrain workloads built around bounded invocations, while long-horizon agents need a lifecycle that is not tied to a single synchronous request. Lambda MicroVMs preserves state for a multi-hour window, but that duration still constrains the longest tasks. Microsoft Foundry Agent Service gives each hosted agent session its own VM-isolated sandbox, retaining files between turns and resuming after idle periods. Northflank supports both ephemeral and persistent environments with no forced time limits.

Snapshot/restore separates continuity from continuously running compute. GKE Agent Sandbox’s Pod Snapshots suspend idle workloads and resume them in seconds, avoiding a choice between paying for idle execution and discarding state. Persistence also supports branching: cloning from a snapshot creates a memory checkpoint and warm-restores filesystem, memory, and running processes into a new sandbox. An agent can then explore parallel hypotheses from a known-good state, making statefulness an execution primitive.

Credential security and what happens when a sandbox is compromised

Credential design must account for what a compromised guest can obtain without escaping its sandbox. If long-lived credentials live inside the guest, hardware isolation does not prevent an attacker with guest-level access from taking them. The boundary can hold while secrets are lost.

Agent Compute Engine keeps those trust anchors outside the sandbox. Its attestation keys are in the Titanium IPU, while long-lived service credentials stay off-host in a central authorization service. When an agent authenticates, the Agent Gateway substitutes the actual backend credential in-flight as the request leaves the sandbox. The guest never holds that credential itself. Sandboxes should hold zero persistent credentials, while ephemeral, scoped tokens used for each operation limit what an attacker can gain by taking over a guest.

For engineering teams, this shifts evaluation from whether secrets are supported to where they reside and how requests acquire authority. The guest’s access and its possession of credentials are distinct concerns. At procurement, teams should also confirm that security certification covers the sandbox product specifically. A SOC 2 Type II certificate for a vendor’s hosting platform does not automatically cover its sandbox execution environment.

Computer-use agents and the distinct sandboxing requirements of rendered UI environments

Rendered interfaces give agents more places to take in input beyond generated scripts and API calls. A browser page or desktop document can contain adversarial instructions that steer an agent toward actions the user did not intend. The display becomes part of the attack surface. Because these agents inherit user-level session privileges, isolation must account for the environment they navigate as well as the code they execute.

Browser-only sandboxing commonly uses an isolated Playwright browser context with network egress restricted to an allowlist. Each task gets a fresh context that shares no cookies, localStorage, or session state. Full desktop sandboxing instead provisions a fresh environment with a virtual display, using Xvfb plus VNC or RDP, and destroys it after the task. Though the usual desktop setup uses containers, untrusted inference-time execution needs that desktop to run within microVM isolation rather than depend on a shared-kernel boundary.

Isolation does not, however, make an agent’s choices reliable. Web task success rates are workable with human-in-the-loop escalation, while fully autonomous desktop workflows succeed on only a minority of complex tasks in the supplied evidence. Containment and task success are separate properties. The practical implication is that unsupervised deployment remains inadvisable for high-stakes desktop processes, even when the underlying sandbox has the required hardware separation.

Enterprise compliance as a runtime selection criterion, not a checkbox

Enterprise requirements narrow the set of viable runtimes before a performance comparison can settle the choice. SOC 2 Type II is a baseline for winning enterprise contracts, while HIPAA and GDPR requirements also help determine which execution environments can be used. The isolation model and compliance scope interact. Strong hardware separation does not by itself establish that the environment processing regulated data falls within the required controls.

Data residency can further restrict deployment location. Bring Your Own Cloud, or BYOC, is a must for many regulated workloads that have to run inside the enterprise’s own VPC. That changes which runtime offerings are viable. It is therefore insufficient to compare managed services first and treat deployment location as a detail to resolve later.

Persistent sandboxes can be as compliant as ephemeral ones, because the risk comes from uncontrolled state. Duration alone is not the decisive property. But compliance frameworks often assume short-lived compute, so teams that use long-running sessions must document how isolation and monitoring keep working for the whole session. Runtime selection consequently includes the ability to establish those controls at startup and sustain them throughout the session.

Choosing the right sandbox tier for the inference workload you are actually running

The selection decision combines isolation tier, provisioning model, and session lifecycle. For general-purpose inference-time code execution, the default is Firecracker or Kata Containers, warm-pool or snapshot-based provisioning, and stateful or indefinite session support. With a different trust model or workload, gVisor can work for multi-tenant SaaS and CI/CD when code written in-house is reviewed, latency must stay low, and the team accepts the shared-kernel risk that remains.

Northflank, Amazon Bedrock AgentCore, and GKE Agent Sandbox are options when agents need to share a control plane with persistent services, databases, background workers, and GPUs. Northflank offers Kata Containers and Firecracker for microVM isolation, plus gVisor as a distinct option, with BYOC across AWS, GCP, Azure, Oracle, CoreWeave, on-premises, and bare-metal. Vercel Sandbox provides Firecracker-based sandboxes on Fluid compute for frontend-adjacent AI workloads already on Vercel.

Short-lived edge tasks occupy a different category. Cloudflare Workers fits stateless JavaScript and TypeScript execution. Runtime selection must match the boundary, startup path, and lifecycle the actual execution requires, with even the shortest boot time or longest timeout evaluated in that context.