How GPU Memory Is Actually Partitioned Between Concurrent CUDA Tenants

GPU memory partitioning between concurrent CUDA tenants is a three-way tradeoff. Isolation strength, performance predictability, and tenant density pull in different directions. Understanding where time-slicing, MPS, and MIG enforce boundaries (and where they leave resources shared) is what lets engineers choose a mechanism that fits their workload rather than merely packs more processes onto a device.

Why a single GPU can no longer serve a single tenant

Modern datacenter GPUs often provide more compute and memory than an individual inference workload can use. Single-tenant assignment leaves much of that capacity idle. MISO’s characterization shows that even complicated AI/ML models leave GPU resources unused because they have different bottlenecks and differ in how much each resource affects them. A workload that cannot use more compute does not become faster simply because more compute is reserved for it. Paying for capacity that a workload cannot use creates pressure to share.

Technical University of Munich’s hierarchical partitioning research connects this difficulty to growing GPU resources, Amdahl’s law, and memory bandwidth limiting performance. For AI agents, the mismatch is also temporal, because agents dispatch short, bursty GPU calls. Reserving a whole device therefore leaves capacity unused both during underutilized execution and between calls. Sharing addresses that waste, but introduces a harder question about what each tenant can be promised.

The three-way tradeoff that governs every partitioning decision

Isolation strength describes whether one tenant’s memory and compute can be observed or disrupted by another. Performance predictability concerns whether a tenant receives a deterministic share of compute and memory bandwidth or competes for those resources at runtime. Tenant density measures how many independent workloads can share one physical GPU concurrently. These properties are distinct, but tightly coupled. A memory allocation limit, for example, does not by itself promise memory bandwidth or contain a failure.

The three mechanisms occupy different positions on these axes. Time-slicing favors density while offering the weakest isolation and predictability. MPS supports concurrent execution with logical resource controls, while MIG favors hardware isolation and predictable resource ownership at the cost of flexibility and density. None provides all three properties without compromise. Stronger physical boundaries reduce the pool of freely assignable resources, while fixed compute allocations cannot flex when tenants are idle. Choose the mechanism whose compromise the workload can tolerate.

Time-slicing: maximum density, zero memory isolation

Time-slicing lets processes take turns on the full GPU. The scheduler alternates workloads along the time axis, giving each roughly its share when all consume their slices and redistributing unused time. That flexibility makes it attractive for cooperative batch work, where completion time is more important than tail latency. Switching CUDA contexts has a cost, however. The resulting jitter makes time-slicing a poor match for latency-sensitive inference.

Memory does not follow the same division of time. Historically, time-slicing has imposed no per-tenant memory allocation boundary, so a process can allocate up to the device’s full memory rather than a fraction corresponding to its scheduled share. One team’s pod can thus use up memory that another pod on the same physical die needs. A scheduling share is not a memory reservation. CUDA 13.4 adds enforcement discussed later, but ordinary time-slicing structurally provides neither dedicated memory resources nor predictable execution. Its defensible use remains trusted, cooperative workloads in which density takes priority over isolation.

MPS: concurrent execution with logical partitioning and its limits

MPS replaces alternating CUDA contexts with a client-server architecture. A control daemon accepts client connections and funnels their CUDA work through a shared server process, allowing concurrent execution and spatial sharing of streaming multiprocessors, or SMs. This avoids time-slicing’s context-switch overhead. NVIDIA’s MPS documentation, Release 620, describes soft and hard memory limits, SM partition assignments, locality domains, and weighted timeslicing. These controls make MPS a resource-allocation mechanism.

MPS v3 can assign fixed SM chunks on Ampere and newer GPUs, giving each client deterministic compute capacity. The memory hierarchy remains shared, however. High-bandwidth memory, caches, and communication buses are not partitioned, so a bandwidth-intensive tenant can degrade other clients despite their SM limits. On select Blackwell GPUs, Memory Locality Optimized Partition, or MLOPart, improves memory locality by exposing logical CUDA devices. The documentation calls this feature locality domains in MPS v3, but better locality does not give it MIG-style physical isolation.

A crash or hang in one client can affect the shared server and other clients. MPS’s finer logical controls serve a different allocation need from MIG, making it a strong fit for cooperative serving where concurrent execution and reduced scheduling jitter are valuable and hardware isolation is optional.

MIG: hardware-enforced isolation at the cost of rigidity

MIG divides a GPU into physically separated GPU Instances, each with its own compute, high-bandwidth memory slice, and cache. Each GPU Instance can then contain Compute Instances that share its memory resources while receiving separate compute at GPC granularity. The distinction between these levels is important. Across isolated MIG instances, kernels cannot preempt or interfere with one another, and an inference hang in one does not stop the others.

That protection comes with fixed allocation choices. MIG profiles are set at partition time rather than resized as demand changes, and a GPU supports at most seven MIG instances. Small workloads can therefore run out of instance slots before exhausting useful capacity. Models requiring full GPU memory or multi-GPU parallelism cannot use MIG under the constraints described here. Reconfiguration interrupts instances and requires workloads to be drained, with reset requirements depending on the hardware and operation.

Amazon Web Services announced NVIDIA Multi-Instance GPU support for Amazon SageMaker Inference on October 30, 2026, allowing hardware-isolated models to share accelerators on real-time endpoints. SUSE Virtualization introduced native NVIDIA MIG vGPU partitioning at KubeCon EU 2026, exposing isolated instances to virtual machines through Harvester. These integrations reduce deployment work while leaving profiles rigid. MISO’s research shows that finding the right partition for co-located jobs still takes a lot of testing.

Hierarchical partitioning: composing MIG and MPS to escape the binary

MIG and MPS need not be competing choices. MIG can establish an outer isolation boundary, while MPS subdivides compute within a Compute Instance. This keeps protection between tenants separate from packing work into a space where the workloads already trust one another. Green Contexts use a similar setup, with slots set up in advance that each have a fixed fraction of SMs. The hierarchy answers two different questions, namely which workloads must be separated and how work inside each boundary should share capacity.

TUM’s research applies MPS inside MIG Compute Instances and uses reinforcement learning to co-optimize partition configuration and co-scheduling groups. Its approach achieves a maximum throughput improvement by a factor of 1.87 compared with time-sharing scheduling. The result comes from optimizing the combined structure. Practical constraints, expensive configuration exploration, and partition-switching overhead make this more than a manual packing exercise. MISO takes a complementary approach by running jobs at different MPS resource-sharing levels and training a performance predictor to find a MIG allocation close to the best one. This flexible co-location lets it skip the costly work of trying every physical configuration.

What CUDA 13.4 and the GPU Fractioning Operator change about software-layer enforcement

CUDA 13.4 adds a memory boundary that works without changes to the application in time-slicing and SM-sharing modes. An allocation above the enforced amount fails as it would on a physically smaller GPU. The application needs no special allocation protocol. With this boundary, each container on a shared device is held to its assigned amount of memory. It should be distinguished from MPS’s per-client controls, rather than read as a claim that MPS previously had no memory limits.

The GPU Fractioning Operator applies that boundary in Kubernetes. This open-source operator runs MPS on every GPU node, sets the memory limits and compute mode for each workload before its process starts, and measures how much memory and compute each fraction uses. A scheduler communicates the allocation through four per-container Pod annotations. The operator reads those annotations and configures the assigned GPU, allowing any Kubernetes scheduler that can write them to use the contract. KAI Scheduler v0.18.0 provides the first integration through NvFractions, enabled with one Helm flag that installs the operator as a dependency.

What does this enforce, and what remains shared? The boundary prevents a workload from allocating beyond its assigned amount, making software-layer sharing more viable when accidental memory exhaustion is the primary concern. It does not partition bandwidth, caches, or failure domains. Memory accountability improves without becoming hardware isolation. The advance narrows one part of the tradeoff while leaving contention and fault contagion unresolved.

Where the tradeoff bites hardest: untrusted code, compliance, and AI agent workloads

For agents executing unreviewed, autonomously generated code, isolation becomes a security requirement. Tenants in that setting cannot be counted on to cooperate. Time-slicing provides no hardware memory boundary, while MPS exposes clients to a shared server’s fault domain. An allocation cap does not resolve either concern. A tenant’s behavior must stay within the intended protection boundary even when its resource use is billed correctly.

Guardian’s research design uses a custom allocator that initially reserves all GPU memory and divides it into contiguous partitions assigned exclusively to tenant-clients. Checks then enforce that memory accesses remain within those boundaries. An allocation quota alone cannot do the work of those checks. The design demonstrates why “memory partitioning” needs a precise definition whenever it is used as a security claim.

When PHI enters a GPU workload, HIPAA obligations extend to the hardware layer, while SOC 2 Type II, HIPAA, or FedRAMP Moderate requirements apply according to the data and environment. GPU memory sanitization and audit logging become requirements. MIG provides the strongest hardware guarantee among these mechanisms as one part of a broader compliance program, while its fixed profiles conflict with agents’ unpredictable memory footprints, leaving an operational tension even after the isolation choice is settled.

How to match workload characteristics to the right partitioning mechanism

The decision framework starts with three questions about the workload. How much isolation does the trust model require, how predictable must latency be, and how many tenants must share each GPU? For untrusted tenants, third-party code, or regulated data, the guidance is to treat MIG as the minimum viable option among these mechanisms. MPS and time-slicing belong inside a trust boundary put in place by other means. If that boundary must contain several workloads, hierarchical MIG and MPS can provide intra-instance packing without abandoning the outer physical separation.

Latency requirements then distinguish compute allocation from opportunistic sharing. Explicit inference bounds or frame deadlines call for MPS v3 static SM partitioning or MIG’s dedicated compute rather than time-slicing’s jitter, while MPS still requires attention to shared memory bandwidth. VUDA’s driving-perception results show why controlled partitioning deserves that attention. Partitioned co-execution reduced inference-budget miss rates from 80.1% to zero while sustaining rendering throughput. That finding demonstrates the consequence of resource control for the tested workload.

Density determines which remaining arrangements are practical. Trusted tenants with high density requirements and moderate latency needs fit MPS v3 with static SM partitioning and CUDA 13.4 memory enforcement. Isolated tenants with predictable latency needs and moderate density fit MIG, while trusted cooperative batch workloads favor time-slicing. More independent tenants than MIG’s instance limit permits require software-layer sharing and conscious acceptance of its weaker boundaries. Hierarchical packing helps only when workloads inside each physical boundary can share the required trust relationship.

Teams using MIG should drain workloads before reconfiguration, since a CUDA process holding the device can obstruct the operation. MIG mode persists across reboots on Ampere GPUs but must be re-enabled after reboot on Hopper and newer GPUs. These operational constraints belong in the allocation decision. Better enforcement and optimization expand the useful choices, but they do not make isolation, predictable performance, and unrestricted density simultaneously available. The durable design principle is to choose which resource can remain shared before choosing how aggressively to share the GPU.