Sandbox API Testing for AI Code Interpreter Integrations

This piece maps out what it actually takes to test a sandbox API before you let an AI code interpreter run untrusted, model-generated code against it. The central claim is simple: sandbox testing differs from ordinary API testing, because the input isn't a known payload but code the model wrote at runtime, and the sandbox is simultaneously the execution surface and the only thing standing between that code and your production infrastructure.

Most teams treat the sandbox as a convenience layer, something that keeps stray print statements from cluttering logs. That's a mistake worth correcting early. A 2025 Veracode report found that 45% of AI-generated code fails security tests outright. Read that number again and sit with what it means for anyone wiring a code interpreter into an agent pipeline: nearly half the code your model produces would fail a basic security check if a human wrote it and submitted it for review. Except no human is reviewing it. The sandbox is catching it, or it isn't, and you won't know which until something breaks.

Conventional API testing checks known inputs against expected outputs. You send a request, you know the shape of the response, you assert on it. That entire model collapses here, because the "input" is code the model generated on the fly, and its behavior can't be fully predicted or reviewed before it runs. So what do you test instead? You test the boundary itself: whether it holds, whether it forgets what it should forget, whether it runs what it claims to run, and whether it fails in a way you can actually observe and respond to. Those are the four layers this piece works through, in that order, because that's roughly the order in which they tend to break in production after they looked fine in a demo.

What a sandbox API actually exposes and what sits behind it

Strip away the branding and most sandbox APIs expose the same five primitives: create a session, execute code (either synchronously or streamed back to you), move files in and out, capture output, and tear the session down. Simple enough on paper. The design decision that actually matters, though, is whether the API spins up a fresh isolated environment on every call or reuses a session that's already active. That one choice determines almost everything you'll need to test later around state and isolation, so it's worth understanding before you write a single test case.

Take OpenAI's Containers API as an example. Passing "container": {"type": "auto"} either creates a new container or reuses one already active in the model's context, and which one happens is not something you control directly. Together AI takes a more explicit route: sessions last 60 minutes, can be called multiple times within that window, and cost $0.03 per session. That bounded window is itself a constraint worth testing against, not just a pricing detail. AWS Bedrock AgentCore Runtime goes further still, assigning each user session its own dedicated microVM with isolated CPU, memory, and filesystem, and it offers two very different lifespans: sessions on microVMs run up to 8 hours, while Instances can persist for up to 14 days. Those aren't interchangeable modes. They need separate test plans, because what breaks after 8 hours of continuous use is not what breaks after 14 days of intermittent reconnects.

Underneath all of this sits an isolation primitive, and the primitive determines your actual threat surface. Firecracker microVMs give each workload a dedicated guest kernel, boot in roughly 125 milliseconds, and carry less than 5 MiB of overhead per VM; that's a hardware boundary between tenants, about as strong as isolation gets in this space. Hardened containers using NsJail and seccomp offer real defense in depth, but they share the host kernel, which means a kernel vulnerability discovered in one container can, in principle, expose every other tenant on that host. gVisor sits somewhere in between: it intercepts syscalls in user space, which is stronger than a plain container, but it's still not a dedicated kernel per workload.

Here's the part that gets overlooked constantly. None of the isolation primitives above matter if the control plane leaks. If any execution path inside the sandbox can reach the host's Docker daemon, or if a Docker socket is accidentally mounted into the sandboxed environment, the code running inside can spin up new containers with host mounts and walk straight past whatever isolation the platform advertised. This is a common misconfiguration, and one of the most common ways "isolated" environments turn out to have gaps. Know which primitive you're relying on, because that choice tells you which attack classes your tests are obligated to cover.

Testing isolation: verifying the boundary actually holds

This is the adversarial core of the whole exercise. Isolation testing exists to answer one question: can code running inside the sandbox escape, pivot toward the host, or read state belonging to another tenant? The threat model driving these tests is uncomfortable but necessary to state plainly. The code wasn't reviewed by a person. It was generated by a model, and it may attempt something destructive, resource-hungry, or deliberately built to escalate privilege, whether the model "intended" that or was manipulated into producing it. A failure here doesn't stay contained; it can escalate to full remote code execution on the host.

Start with the filesystem. Try reads and writes that reach outside the sandbox root, and confirm the response is a clean permission error rather than a silent success. Check whether code executing in one sandbox session can see files left behind by a sibling session running on the same physical host. And after a session tears down, verify the root filesystem is actually wiped or discarded; residual state surviving into the next session is a cross-tenant data leak waiting to be discovered by whoever's unlucky enough to get assigned that slot next.

Network isolation deserves equal scrutiny. Attempt outbound connections from inside the sandbox and confirm that egress is restricted to whatever allowlist the platform declares, if one exists at all. This isn't academic: supply chain attacks have specifically targeted AI agent workflows, and an agent that installs packages inside a sandbox with unrestricted egress will happily execute whatever payload rides along. Test DNS resolution too, since an open resolver functions as a covert channel even when direct TCP connections are blocked. It's an easy thing to miss because it doesn't look like an obvious hole.

Process and syscall isolation round out the picture. Try forking processes, mounting filesystems, or invoking syscalls that seccomp or the hypervisor should restrict, and confirm the failure is the correct error class rather than a silent degradation that leaves the sandbox in some undefined state. Check whether a sandbox can observe or signal processes in a sibling sandbox on the same host. And check, specifically, whether the Docker socket or any container control-plane endpoint is reachable from inside. If it is, you likely have a full isolation bypass, not a minor gap.

One caveat worth stating directly: kernel-level boundary tests only make sense when the underlying primitive actually provides a dedicated kernel, as Firecracker and Kata Containers do. Against a shared-kernel runtime, don't pretend the container boundary is equivalent to a hardware boundary in your test documentation. Write down which attack classes you're accepting as risk instead of quietly assuming they're covered. CyLab research found that manipulating as little as 0.1% of a model's pre-training dataset is enough to launch an effective data poisoning attack, which is a good reminder that adversarially crafted code isn't a paranoid edge case in your test suite. It's a realistic scenario you should be running today.

Testing state handling: what persists, what resets, and what should never cross sessions

Isolation testing asks whether code can escape. State handling testing asks something different: does the environment remember what it should remember, and forget what it should forget, at the right moments? These are related concerns but distinct failure modes, and conflating them in your test plan is how gaps slip through.

The session reuse pattern is the main source of hazard here. When an API reuses an active container, as OpenAI's auto mode does, in-memory variables, loaded libraries, and cached data from a prior turn are still sitting there when the next call comes in. Agents depend on exactly this behavior to install a package in one turn and import it in the next. But it cuts both ways, so your tests need to confirm two things simultaneously: that the persistence agents rely on actually works, and that nothing sensitive survives when it shouldn't.

On the persistence side, write a variable or a file in one execution call and confirm it's readable in a later call within the same session. Test that packages installed mid-session stay installed and importable for the rest of that session, since agents lean on this constantly. For platforms supporting genuinely long-running sessions, like AgentCore's 14-day Instances, test what happens across an idle gap and a reconnect. Does the environment come back exactly as it was, or does something quietly get dropped?

On the reset side, the standard is stricter. After a session ends, whether through Together AI's 60-minute expiration or an explicit teardown call, confirm the next session starts genuinely clean: no leftover variables, no stray files, no credentials that belonged to whoever used that slot before. Test what happens when a session gets forcibly killed, and check whether residual state is accessible to the next session assigned to that same underlying VM or container. Environment variables injected for one tenant showing up in another tenant's session is about as bad as this category gets. Credential bleed between sessions turns a testing gap into an incident report.

Memory limits need their own test pass too. OpenAI's container default sits at 1 GB, so push execution right up against that ceiling and past it. Does the sandbox terminate cleanly and raise something the calling code can catch, or does it produce undefined behavior that leaves your agent guessing? The answer shapes how you handle memory-intensive workloads in production, so it's not a detail to leave undocumented.

Where a platform supports snapshotting, that's one more surface to check. Take a snapshot mid-execution and confirm the resumed environment comes back with file handles, in-progress computation, and open network connections all intact. And separately, confirm that resuming a snapshot never surfaces state belonging to a different agent's session. Snapshot isolation is its own risk category, distinct from live-session isolation, and it's easy to assume one implies the other when it doesn't.

Testing execution fidelity: confirming the sandbox runs what the agent expects

This layer sits closest to conventional API testing, but it carries its own wrinkles. Code is generated at runtime, the output has to be something an agent can actually parse downstream, and streaming behavior affects how responsive the whole interaction feels.

Start with output capture. Submit code that produces stdout, stderr, structured return values, and richer outputs like plots or dataframes, and confirm each type comes back in a format your agent's logic actually expects. OpenAI's Code Interpreter streams output, so test that partial chunks arrive in order and that the final assembled result matches what streamed in along the way. Streaming bugs tend to hide; they show up as truncation or out-of-order fragments only once you push real load through the system, not during a quiet demo run. Also confirm stderr is distinguishable from stdout in the response. An agent that can't tell the two apart will make confidently wrong decisions about whether its code actually succeeded.

Package and runtime environment checks matter more than they sound like they should. Confirm the pre-installed packages actually match what the platform documents, because the model is generating import statements based on assumptions baked into its training data, and a mismatch there causes failures that look like model errors but are really environment errors. Test pip install (or whatever the platform's equivalent is) mid-session, then confirm the newly installed package imports cleanly in a later call within that same session. And test what happens when code reaches for system calls, subprocesses, or shell commands: either the platform permits it with the behavior you'd expect, or it blocks it with a clear, catchable error. Anything in between is a problem.

Concurrency is where a lot of theoretical soundness falls apart. Agentica's integration with Together AI's Code Interpreter ran 1,024 code executions in parallel during training for DeepCoder-14B-Preview. Parallel execution at scale is a normal production pattern, not a rare edge case you can defer testing on. Confirm that concurrent sessions on the same account don't bleed into each other's execution or output; cross-session contamination has a way of staying invisible until you're running enough sessions simultaneously to expose it. Test rate limiting too. What error class comes back when concurrent session limits are exceeded, and how fast can the caller retry without making things worse?

Finally, timeouts. Submit code that runs indefinitely, or that simply outlasts the session window, and confirm termination is clean, the error surfaces to the caller, and no partial output vanishes silently. And keep the two timeout types distinct in your test plan: a session timeout means the whole window expired, while an execution timeout means a single call ran too long. Agents need to handle these two situations differently, so your tests should treat them as different failures, not variations on the same theme.

Testing failure modes: what happens when execution goes wrong at the boundary

Here's where most teams have the biggest blind spot. They test the happy path thoroughly against the sandbox API, watch it work, and assume errors will surface just as cleanly when something goes sideways. They often don't, and that gap is exactly where production incidents live.

Resource exhaustion deserves a dedicated test pass, not an afterthought. OWASP's Top 10 for LLMs 2025 classifies unbounded resource consumption as LLM10:2025, which tells you the industry already recognizes this as a first-class risk rather than an edge case. The sandbox API is your enforcement point for this risk, so test it directly: submit code that aggressively allocates memory, spawns a large number of subprocesses, or writes oversized files, and confirm the sandbox actually terminates the execution and returns a structured error, rather than hanging or dragging down neighboring sessions on the same host. Worth checking too: does sustained high CPU usage in one session measurably affect latency in concurrent sessions sharing that host? If it does, that's a resource isolation gap dressed up as a performance issue.

Malformed and adversarial code needs its own category of tests. Submit code with plain syntax errors and confirm the sandbox returns a parse error before attempting execution at all, rather than crashing mid-run. Submit code that tries privilege escalation, attempts to read something like /etc/shadow, or calls a system command with destructive arguments, and confirm each attempt is either blocked outright by the isolation layer or fails with an error that gives away nothing about the host underneath. Research examining LLM-generated code patches found that even while fixing the original bug, the patches introduced new security vulnerabilities in 9.5% of cases. That's a useful number to keep in mind when building your adversarial test set: pull from realistic patterns an agent might actually produce, rather than synthetic worst-case scripts that look scary but don't resemble anything the model would generate on its own.

Session lifecycle failures round this section out, and they're easy to underweight because they don't feel like "security" testing in the traditional sense. Test what happens when a session dies mid-execution, whether from a timeout, an explicit teardown call, or a simulated infrastructure failure. Does the API return a partial result, a clean error, or nothing at all? Test reconnection to a long-running session after a network blip, since for platforms supporting multi-hour or multi-day sessions, reconnection fidelity is really a reliability requirement dressed up as an edge case. And test what happens with auto-mode session reuse when the underlying container has been evicted between model turns; the API may quietly provision a fresh container, and the agent, having assumed continuity, loses state it never knew it was at risk of losing.

Error observability ties the whole section together. Every distinct failure needs a distinct, machine-readable error code, because an agent staring at an ambiguous error can't retry intelligently, and it certainly can't decide when to escalate to a human. OWASP's MCP Top 10 flags missing telemetry as a real risk category, and testing this means confirming the sandbox produces execution traces detailed enough to reconstruct what ran, what failed, and why, after the fact. This is also increasingly a compliance requirement, showing up in SOC 2 audits and GDPR-related logging expectations. Both your security team and your legal team will care about it equally.

Choosing a sandbox platform whose API design supports systematic testing

Everything above implies a set of concrete requirements a sandbox API needs to meet before systematic testing is even possible: clean session lifecycle semantics, structured and distinguishable error codes, observable execution traces, isolation guarantees stated plainly rather than implied, and resource limits that are documented rather than discovered the hard way in production.

OpenAI's Containers API is a managed, opaque environment. That has real advantages: you're not maintaining infrastructure, and the API surface is simple. But it also means isolation testing is limited to behavioral probing from the outside; you can't inspect the host, so you're inferring boundaries rather than verifying them directly. The auto-mode session reuse is genuinely convenient for building agents, but it demands explicit testing to confirm exactly what persists between calls and what residue, if any, carries forward unintentionally. And because output streams back, chunk ordering and completeness need their own dedicated test cases, since that's precisely the kind of thing that looks fine at low volume and breaks under real load.

AWS Bedrock AgentCore Runtime takes a different approach, and it shows in what you can verify. The dedicated microVM per session gives you the hardware isolation boundary that the strongest isolation tests actually require, rather than something you have to take on faith. But the dual session model, up to 8 hours on a microVM versus up to 14 days on an Instance, means you're not writing one test plan, you're writing two. State persistence looks different across those windows, and so do the failure modes; what degrades gracefully after 8 hours might behave completely differently after two weeks of intermittent use. Treating those as the same testing surface because they share an API is the kind of assumption that feels reasonable right up until it isn't.

Whichever platform a team lands on, the underlying question doesn't change: does the API's design let you actually verify isolation, state, execution fidelity, and failure behavior, or does it ask you to trust documentation you can't independently confirm? That question is worth asking before the integration ships, not after an agent finds the gap for you.