Sandbox Warm Pool Sizing for Bursty Agent Workloads

Warm pool sizing for bursty agent workloads is not a knob you set once and walk away from. It rests on four measurable inputs: burst shape, provisioning latency, sandbox cost, and drain rate. Teams that size pools around those numbers absorb traffic spikes without paying the cold-start tax or the idle-compute tax, and most teams get this wrong in one of two directions. They either overprovision out of fear and eat the idle cost forever, or they let users eat cold starts and call it "good enough" until the complaints start piling up. Neither approach is a strategy; it's an abdication. This piece walks through the four inputs, how to read them, and what the infrastructure has to do to keep pace with them.

Early generative AI treated inference as a transient function call: prompt in, response out, process dies. That fifty-millisecond, call-and-die model no longer describes how agents actually run. Autonomous agents run long, stay alive across hours or weeks, write code, call tools, and coordinate with other agents mid-task. A single user request can fan out into a dozen parallel sandbox executions, so concurrency spikes multiply instead of adding. Reactive autoscaling was built for the old call-and-die model, and it adds capacity only after a spike is already detected, and the gap in between is exactly the delay a real user feels. Once sandboxes take multiple seconds to boot, the slow boot itself becomes the bottleneck rather than the agent's reasoning, and no amount of clever prompting fixes a slow boot.

The economics shifted too. AWS changed Lambda billing in August 2025 so cold-start initialization now gets charged at invocation-duration rates, which raises the per-cold-start cost specifically for the bursty, spiky traffic agents generate. Warm pool sizing becomes a cost argument as much as a latency one at that point. Without a deliberate strategy, teams end up stuck between two bad options: pay for compute sitting idle most of the day, or let users eat cold starts every time traffic spikes. Sizing, treated as an actual engineering discipline instead of a guess, is what breaks that binary.

Agent Sandboxes vs. Generic Containers and Serverless Functions

Agent-grade sandboxes have to clear four bars that generic serverless compute never had to worry about. Startup latency needs to stay low enough that an agent doesn't stall mid-task between tool calls. Isolation needs to be strong enough to run untrusted, AI-generated code without one tenant's state leaking into another's. State has to persist across invocations, so an agent resuming a task isn't re-cloning a repo or reloading a dataset it already had loaded ten seconds earlier. Scaling has to absorb bursts on its own, without an on-call engineer getting paged at 2am to add nodes by hand.

Isolation level is where the cost structure actually gets decided, and there are five tiers worth knowing. Standard containers are the bottom: shared kernel, fastest possible startup, but nowhere near a strong enough boundary for code an LLM wrote on the fly. One step up is the gVisor user-space kernel, which intercepts syscalls for a tighter boundary and is what GKE Agent Sandbox runs on. Above that sits Firecracker, running microVMs that boot in 125 milliseconds, carry less than 5 MiB of overhead each, and can spin up 150 VMs per second on a single host. Firecracker is the current standard for running code nobody trusts, full stop. Microsoft's LiteBox, a Rust-based Library OS, was still experimental as of February 2026. At the top sits confidential computing, which encrypts memory at the hardware level and matters most for healthcare, finance, and anything touching PII.

That hierarchy carries a sizing consequence most teams underweight: heavier isolation buys stronger guarantees but raises the cold-start floor, and no amount of pool tuning moves that floor afterward. A microVM-backed warm pool buys back the 125-millisecond floor on every allocation, but it's a fixed cost the warm pool absorbs, not a number that shrinks with a smarter reconcile loop.

Statefulness complicates things differently, and this is where most sizing models quietly fall apart. A coding agent installing a package, writing a file, running the test suite, editing, and rerunning is carrying state forward the entire time. A pool member holding that filesystem and memory state is not interchangeable with a plain ephemeral one. If it is treated as such, the sizing math breaks the first time real load hits it. Blaxel's sandboxes resume with full filesystem and memory state in under 25 milliseconds, and that only works because the platform was built around that resume path from day one rather than bolted onto a stateless container model after the fact. That's the real argument for purpose-built agent runtimes over retrofitted developer containers or repurposed Lambda functions: the mismatch between a stateless primitive and a stateful workload doesn't disappear at deploy time. It appears later as pool-sizing complexity somebody has to untangle by hand, usually during an incident, because the mismatch only becomes visible once the pool has to be resized under real load.

The four concrete inputs that determine warm pool size

Burst shape comes first, and it has to come from production traces, not synthetic load tests. Synthetic benchmarks tend to miss the exact bursty arrival patterns and long-tail latency spikes that cause real problems. Pull p50, p90, and p99 concurrency from actual agent traffic. Fan-out makes the tail fat here: one user request turning into ten parallel sandbox claims means the tail of the distribution is where the real capacity pressure sits, so the p99 number reflects that pressure directly. Model that distribution at the sandbox level, not the user-request level, or the fan-out multiplier gets lost. And don't blend agent types together. A bursty coding-agent workload drains a pool at a different rate than a slow background summarization job, and averaging the two into one metric hides whichever one is actually causing the cold starts.

Provisioning latency comes second, and the number that matters is claim-to-ready time, measured per agent type. GKE Agent Sandbox's warm pool handles 300 sandbox allocations per second per cluster, with 90% of those completing inside 200 milliseconds, a useful benchmark for what "fast enough" looks like at scale. Blaxel's standby resume, under 25 milliseconds with full state intact, sets a different bar for cases where session continuity matters as much as raw speed. The gap between how fast a platform can provision and how long a user will actually wait is what sets pool depth. If provisioning takes longer than users will tolerate, the pool has to be deep enough to absorb the whole burst before new capacity even starts coming online.

Sandbox cost per unit is where the tiering argument comes from. An always-on warm member costs full compute at all times, whether it's serving anything or not, and that's the trap teams fall into when they overprovision out of fear. A suspended or snapshotted member costs a fraction of that. GKE Agent Sandbox treats suspended VMs as a cold buffer that replenishes the warm pool as needed, and since agent workloads tend to run in short bursty cycles followed by long idle stretches, snapshotting idle agents and resuming them in seconds fits that pattern well. GKE Agent Sandbox also claims up to 30% better price-performance on Axion processors compared to other cloud providers, a real input if the cost model includes a cross-provider comparison. The right pool size minimizes idle warm compute plus the cost of whatever cold starts do happen, and the August 2025 Lambda billing change pushes that second number up hard for anyone running bursty workloads on Lambda specifically.

Drain rate closes the loop: how fast do agents claiming sandboxes eat through the pre-provisioned pool during a burst? At GKE's 300 allocations per second, a large burst can drain even a deep pool in a matter of seconds. That makes pool depth a time-bounded number that changes rather than a static headcount someone sets once and forgets. Bounding conversation length and step count, set from observed p95 and p99 distributions of runtime and cost, limits how long any one agent holds onto a sandbox, and that bound caps how fast the pool can drain. As with burst shape, sizing to a blended average across agent types underserves whichever type burns through the pool fastest, and that type is usually the one causing the cold starts to begin with.

Reading the signals: observability primitives that make sizing a feedback loop

None of the four inputs mean anything without instrumentation to measure them continuously. A pool sized correctly today drifts out of tune as traffic patterns shift, sometimes within weeks. Warm Pool Efficiency, the ratio of allocations served from the warm pool versus allocations that hit a cold start, is the clearest single health signal available. Claim-to-ready latency needs logging per agent type rather than rolled into one aggregate number, since that's the level at which sizing decisions actually get made. Tracking creation latency metrics per agent type is a natural complement to monitoring control-plane lag before it turns into a user-facing problem.

Measure all of it at p99. Averages are close to useless here, because trigger sensitivity for autoscaling alarms depends on how the alarm periods and evaluation windows get configured, and average-based alarms can miss burst-induced degradation. A pool that looks perfectly healthy on average can still fail every single request during the two seconds a burst actually lands.

Distributed tracing, however it is structured, becomes genuinely useful when every trace is tagged with whether it hit a cold start or a warm resume. That tagging turns a vague sense of "the pool feels underprovisioned" into an actual distribution someone can size against, instead of a hunch that gets argued about in a standup.

When available drops below desired, the pool is at risk right now, and tracking available replicas against desired replicas is another useful ratio. When available drops below desired, the pool is at risk right now. When the gap between the two keeps widening over time, that's a sign the burst-shape assumptions built into the sizing model are wrong, or the drain rate is running hotter than modeled. Either way, the fix is going back to find where p99 claim-to-ready latency crosses the acceptable threshold, and using that crossing point, together with the observed drain rate, to set pool depth for the next planning cycle. Sizing, done properly, works closer to a control loop than a one-time calculation. It never really finishes.

Tuning the Kubernetes control plane for warm pool throughput without creating pod churn

The SandboxWarmPool custom resource is the native Kubernetes primitive built for exactly this problem. It keeps a set of pods pre-booted and ready, pushing cold-start latency down to sub-second territory without anyone hand-rolling custom autoscaling logic. SandboxTemplate defines the environment blueprint those warm pods build from, and SandboxClaim gives agent frameworks like LangChain or Google's ADK a way to declaratively request an execution environment instead of reaching into the cluster directly. This landed via the Kubernetes blog in March 2026 as a SIG Apps subproject, housed at kubernetes-sigs/agent-sandbox. It's a SIG Apps subproject, not a standalone CNCF project, and it did not launch at KubeCon NA 2025, whatever the conference-circuit rumor mill claimed.

The controller behind the warm pool exposes tuning parameters that determine how fast the reconcile loop can move. Setting --sandbox-concurrent-workers=25 lets the controller process 25 reconciles at once. API rate-limit flags raise the ceiling on how fast the controller can talk to the Kubernetes API. These settings exist for one reason: making warm pool replenishment keep pace with drain rate during a burst.

There's a failure mode on the other side of that dial, and it's the one teams tend to discover the hard way, usually mid-incident. If concurrent-workers is pushed too high, pod churn starts competing with actual agent traffic for the same node resources, undermining the very purpose of tuning the flag. Where the safe upper bound sits before that competition kicks in is workload-specific, and it stays an open question rather than a solved one. The practical move is treating concurrent-workers as a dial to raise gradually while watching pod churn metrics the whole time, not something set once from a blog post and forgotten.

At a different order of scale, Kubernetes itself starts to become the bottleneck. Standard Kubernetes control planes are built around thousands of long-running services, and agent workloads look nothing like that: millions of sub-second tool calls hitting the API in bursts is a pattern the standard control plane was never designed to absorb. Agent Substrate, an open-source project announced alongside GKE Agent Sandbox's general availability in May 2026, introduces a minimal control plane meant to bypass some of those Kubernetes limitations without replacing the rest of the stack. At sufficient scale, the control plane itself becomes the density ceiling, and that's the specific problem Agent Substrate targets.

Tiered pool architecture: warm, cold, and suspended layers working together

No single pool tier carries the whole load efficiently, and pretending one can is where a lot of sizing plans go wrong. The architecture that actually works splits into three layers. The warm tier holds fully provisioned, immediately claimable sandboxes, sized to absorb the expected p99 burst with zero cold starts, and it's the most expensive tier per unit since every member sits fully powered whether it's in use or not. The cold tier holds suspended VMs, pre-provisioned but powered down, ready to replenish the warm tier at a fraction of the running cost, which is what GKE Agent Sandbox's standby capacity buffers do. The on-demand tier sits beneath both and only gets touched once warm and cold are both exhausted. It carries the full cold-start penalty and works as a safety valve, not a tier anyone wants serving regular traffic.

Snapshotting is the mechanism tying the tiers together. Pod snapshots let an idle agent workload suspend and resume within seconds, so a sandbox sitting in the cold tier is a snapshot waiting to wake up, not a machine booting cold from nothing. Blaxel's DeltaBox model pushes this further: it keeps a persistent worker process alive inside the sandbox and checkpoints or restores full state, filesystem and memory both, atomically. Every node in an agent's reasoning tree becomes a joint state that can be saved and restored on demand, which is what makes resuming from the cold tier functionally equal to continuing in the warm tier, rather than the degraded fallback most architectures settle for.

The ratio between warm and cold tiers comes down to one question: how fast can a cold member get promoted to warm? If a suspended VM wakes up and joins the warm tier in under a second, the warm tier itself can stay shallow, since the cold tier backs it up fast enough to matter. If promotion takes longer than that, the warm tier needs more depth to cover the gap. Treating pool sizing as a binary choice between all-warm and no-warm-at-all means paying full isolation cost during the long idle valleys between bursts, when a cheaper suspended tier would cover the same ground for a fraction of the price, and that binary framing is exactly the mistake driving most overprovisioned pools today. The step-count and runtime bounds mentioned earlier do double duty here: capping how long any one agent can hold a sandbox caps drain rate directly, which shrinks how much warm-tier depth the whole system needs to carry.

Egress, isolation, and security properties that warm pool design must preserve

Warm pools introduce a specific risk a purely cold-start architecture doesn't have to think about nearly as hard. A pre-provisioned sandbox that isn't torn down deterministically can carry residual state from whoever used it last. Isolation isn't just something that happens at boot, it has to hold at teardown too, every single time a sandbox goes back into the pool, no exceptions.

Five properties matter for any sandbox sitting in a warm pool. Isolation means agent execution can't touch host resources it wasn't explicitly granted access to. Resource limits on CPU, memory, network, and execution time stop a runaway agent from eating into capacity the rest of the pool needs. Capability scoping controls, at a fine grain, exactly which APIs, files, and network endpoints a given sandbox can reach. Auditability means every action inside the sandbox gets logged and stays observable, which matters most after something has already gone wrong and someone needs to reconstruct what happened. Deterministic teardown means the sandbox gets destroyed completely between uses, leaving nothing behind that the next tenant pulling from the same warm pool could inherit.

That last property is the one warm pools put the most pressure on, and it's also the one most likely to get cut when a deadline is looming. A cold-start architecture destroys the whole environment after every use by default, more or less as a side effect of how it works anyway. A warm pool exists specifically to avoid that destroy-and-rebuild cycle, so teardown has to be engineered on purpose instead of happening automatically as a byproduct. Getting warm pool sizing right, in the end, is as much a security guarantee that has to hold on every single reuse as it is a performance and cost question. If that guarantee is skipped to save a few milliseconds of resume time, the latency and cost savings the architecture was built to capture stop being worth the risk they were taken to buy.