NVIDIA MIG for Sandboxed AI Inference Workloads

NVIDIA Multi-Instance GPU (MIG) makes sandboxed AI inference possible by partitioning GPU hardware. Its central promise is efficient multi-tenant inference without surrendering the GPU isolation boundary. A secure, useful production system requires profile sizing, scheduling, memory sanitization, and stateful runtimes alongside hardware separation.

Why GPU utilization is the central problem in multi-tenant AI inference

A dedicated GPU can be busy serving requests while leaving most of its resources unused. A single H100 provides far more compute than a typical small-model inference request needs, so reserving it for one workload can strand both compute and VRAM. Agentic workloads deepen this mismatch. Their chains of LLM calls arrive irregularly, leaving a reserved GPU idle between reasoning steps even while an agent remains active.

The obvious response is to consolidate workloads. Yet higher occupancy alone does not establish that consolidation works for production inference, where one tenant's activity must not make another tenant's latency unpredictable. Time-slicing and CUDA MPS offer software-based ways to share GPU capacity, but they do not create the hardware separation needed between untrusted tenants. How can workloads share the device without also sharing the consequences of each other's behavior?

What MIG actually does at the hardware level

MIG divides the hardware into instances. Each instance has its own dedicated streaming multiprocessors, memory controllers, isolated L2 cache banks, and exclusive DRAM address buses. The separation extends through the memory subsystem. Because this division is spatial, workloads on different instances execute in parallel. A tenant cannot consume a neighbor's assigned compute or contend for its isolated cache banks.

NVIDIA's MIG inference demonstration made that distinction tangible with a ResNet50 flower-classification workload on an A100. Across seven 1g.5gb instances, throughput grew linearly as more instances were added, and the others kept running correctly when one instance had a fault. An instance-local memory error or workload fault could be contained without interrupting neighboring inference workloads, allowing each partition to maintain its own quality of service.

That result explains why MIG is more than a scheduling feature. With time-slicing, workloads take turns using the same memory and cache resources, which leaves room for noisy-neighbor interference. MIG removes that contention across its hardware partitions. The distinction is specific, however, rather than a claim that every part of an inference service becomes independent. The hardware boundary governs the resources assigned to each instance.

How MIG profiles shape what inference workloads can run where

A MIG profile translates that hardware boundary into an allocation decision. On the A100 40 GB variant, profiles range from 1g.5gb, with one compute slice and 5 GB of memory, to the full-device 7g.40gb profile. The smallest configuration permits up to seven instances. The 80 GB variant works the same way, with profiles that include 1g.10gb and the full-device 7g.80gb. Profile choice therefore sets both the resources available to a workload and the number of workloads the GPU can host.

Smaller profiles work well for smaller models, while 3g.20gb and 4g.20gb allocations have room for mid-size models or workloads that need sustained memory bandwidth. Mixed allocations need not be uniform. For example, an A100 can combine one 3g.20gb instance with three 1g.10gb instances to reflect a varied model portfolio. The H100 extends the profile approach with higher per-instance memory bandwidth through HBM3 on SXM5, compared with HBM2e on the A100.

Will the allocation still suit its demand at peak traffic? Reconfiguration requires stopping all workloads on the GPU and destroying existing instances, even though the operation takes seconds. Enabling MIG mode also requires a GPU reset on Ampere, but not on Hopper/H100 and later. Operators must plan for the mix of workloads because these static partitions cannot grow while serving requests.

How the MIG hardware boundary makes multi-tenant inference genuinely safe

Under software sharing, workloads use shared GPU resources without MIG's dedicated memory boundary, creating risks around residual weights, activations, and inference inputs. MIG assigns separate memory paths and cache banks. That closes the cross-partition read path described by the shared-resource model, rather than depending on a tenant's willingness to respect a software convention.

Fault containment strengthens the same boundary. A tenant-local runaway kernel or out-of-memory failure does not give that tenant control over the resources assigned to a neighboring instance. This is especially relevant when agents execute model-generated code. The operator cannot treat generated output as a trusted promise that cross-tenant access will never be attempted. Hardware separation makes GPU sharing defensible without requiring that trust, although it does not sandbox every other resource the agent can reach.

What if privileged infrastructure operators are part of the threat? H100 Confidential Computing, paired with CPU trusted execution environments and confidential GPU modes, addresses protection of model weights and inference inputs from that different adversary. NVIDIA vGPU 19.0 also adds Virtualization-Based Security support with Microsoft Azure Local and Windows Server hypervisors. These protections extend the isolation stack for regulated environments such as financial services and healthcare. They address different boundaries rather than making MIG alone a complete security system.

Extending MIG isolation into the virtualization and Kubernetes layers

A MIG-backed vGPU connects a hardware instance to a virtual machine. In the dedicated-instance arrangement, the vGPU has exclusive access to its partition, so the VM inherits hardware isolation. NVIDIA AI Enterprise documentation describes this arrangement as providing hardware-level isolation and true parallel execution. Placement still has an operational constraint. Live migration between different MIG profiles is unsupported, and the destination host needs a matching MIG configuration.

Kubernetes exposes the same partitioning through typed resources. The NVIDIA device plugin makes instances available under resource names such as nvidia.com/mig-1g.10gb, while MIG Manager handles node-level configuration. Pods request a profile, and scheduling assigns the corresponding partition. NVIDIA's MIG announcement established compatibility with Kubernetes and containers without changes to the CUDA programming model.

Isolation and additional sharing can also be layered. CUDA MPS lets lightweight processes share one MIG instance and keeps the outer hardware boundary in place, but those processes share a MIG boundary with one another. The same distinction applies to layered virtualization. On the RTX PRO 6000 Blackwell Server Edition, four MIG slices can each host three time-sliced vGPUs, supporting 12 VMs with hardware separation between slices and a shared hardware boundary for VMs within each slice. Blackwell's Universal MIG also extends partitioning to mixed compute and graphics workloads.

Where MIG delivers the largest utilization gains for inference workloads

MIG has its clearest advantage when workloads are small, numerous, and sensitive to latency interference. A language model, image classifier, and recommendation engine can occupy separate instances rather than require separate physical GPUs. Each receives a defined resource allocation. This uses capacity that would otherwise sit idle and gives each model predictable access to compute and memory, while each model still needs the same resources.

The A100 demonstration gives this argument a concrete scale. Its throughput grew linearly with added instances at batch size 1, a configuration relevant to interactive inference where large batches are not always available. In this workload-specific comparison, two instances already exceeded a standalone V100 16GB or T4 on the same task. It shows why consolidating small inference jobs can outperform assigning them to older discrete devices.

Production and development can benefit from the same separation. A larger partition can serve an endpoint with strict latency SLAs while smaller partitions support development and testing on real hardware, without separate staging boxes. Tenant hosting works the same way across customers, with multiple independent workloads on A100 80GB 1g.10gb instances that each have their own compute and memory bandwidth. Bursty agents likewise cannot consume neighboring partitions' allocations. But some capacity can sit unused within the partition layout because idle resources in one instance do not automatically pass to a neighbor that is at capacity.

How stateful agent workloads interact with MIG's static partition model

An agent may run for minutes or hours, invoke external tools, and retain conversation state across multiple LLM calls. Its compute demand changes throughout that sequence. A fixed GPU partition can isolate those calls, but it cannot adjust its size to each phase or preserve the agent's execution state by itself.

Loading larger models back into VRAM takes time, so scaling a serving runtime to zero exchanges idle-resource savings for a reload penalty. NVIDIA Dynamo Snapshot, published in May 2026, addresses this through checkpointing and restoring loaded model state for single-GPU workloads. The prototype cuts startup delay, though some delay is still there. Its faster restoration leaves MIG's fixed partition boundaries unchanged.

Reuse also raises a different security question. What happens when a new tenant receives the same partition that a previous tenant occupied? Spatial isolation between active instances does not remove residual data within a reused instance, so explicit VRAM clearing is required before reassignment. To keep the agent's work going, its state must also be saved outside VRAM through checkpoints to durable storage and restored on the next allocation. A stateful runtime therefore complements MIG by preserving progress across interruptions, while MIG governs the hardware boundary during GPU execution.

What a production-ready MIG-backed inference architecture looks like end to end

The design needs profiles that fit the models' VRAM requirements and a mix of partitions that suits the workloads in the cluster. For multi-tenant or multi-agent clusters, the recommended default isolation mechanism is MIG. MPS can fit more workloads within a slice. Untrusted workloads also need explicit namespace boundaries, RBAC, and network policy, because GPU isolation does not govern the rest of the application environment.

NVIDIA Triton Inference Server suits multi-model deployments through concurrent model execution, dynamic batching, and model ensembling within a single server process, while vLLM serves LLM inference workloads on MIG slices. Serving stages can also receive different allocations. A production deployment pattern separates prefill and decode workers onto differently sized profiles on the same GPU, using larger slices for prefill and smaller ones for decode.

Pods request profiles through the device plugin, MIG Manager configures nodes, and Kubernetes Dynamic Resource Allocation, GA in 1.34+, supports finer-grained GPU allocation within the scheduler. Scheduling cannot substitute for capacity planning. The architecture still has to accommodate profile reconfiguration and matching-profile migration constraints, while the stateful runtime handles durable checkpoints and restoration when agent execution outlasts an allocation.

For regulated deployments, the security layer pairs MIG with H100 Confidential Computing per instance and adds vGPU VBS support where hypervisor-enforced protection is needed. Tenant turnover requires explicit VRAM sanitization. Audit logging and access controls cover both the infrastructure and the runtime, so the policy boundary reaches beyond the assigned GPU resources. This division of responsibilities is what turns partitioning into a production sandbox, with hardware enforcing separation, orchestration assigning resources, and the runtime preserving state without confusing any one layer for the whole system.