Home / Docs / GPU sandboxes / Pause semantics

GPU pause semantics — leases that run graphics/child processes

What pause does depends on the lease's suspend_mode. The rules on this page come from measurements on an A10 GPU host (2026-10-09) with graphics, simulation and game-engine workloads.

What those measurements showed:

• cuda-checkpoint toggles only the worker pid. Job children (a simulator runtime, two game-engine players and a 3D editor) were never frozen: they kept rendering/stepping and kept their VRAM while the lease read "suspended".
• On resume the worker re-reserved its whole slice on top of children that were still resident: a 3D editor's render child was OOM-killed by the VRAM cap; a simulator's lease exceeded the watchdog limit (14712 > 13108 MiB) and the tenant key was flagged for abuse — for an overshoot the gateway itself produced.
• docker pause (cgroup freezer) froze all four apps and they continued correctly after unpause; VRAM stayed resident (no spill).
• cuda-checkpoint on the simulator's child process: rc=1 and the child died — children cannot be checkpointed.

Semantics

• Nothing runs while paused. There is no silent half-pause: a transparent suspend that cannot freeze a lease process (e.g. EPERM) SIGCONTs what it stopped and refuses the suspend with an error — the alternative the measured data already justifies (checkpointing children kills them).
• On a hard-cap host, a transparent suspend whose non-worker processes hold VRAM is freeze-only: it freezes the whole lease and keeps its seat instead of spilling, because freeing the seat while frozen children still hold VRAM would let the next lease take the slice — oversubscription the gateway itself produced. A host without a hard cap has no VRAM reading, so its transparent suspend always spills.
• Freeze is not a checkpoint: no toggle, no release/reserve control traffic, no re-reserve storm on resume. The phase stays resident with frozen=true (wire phase reports suspended) and freeze_reason names the freeze-only path of a hard-cap host (non-worker VRAM resident or VRAM probe unavailable), so the seat is still accounted — VRAM never left the card.
• Pause/resume are idempotent. A busy lease is refused as before. Exec or resume on a frozen lease thaws it first, then proceeds. Destroy and the watchdog kill work on a frozen lease (docker rm -f and SIGKILL reap frozen processes; a SIGSTOPped process group is SIGCONTed before SIGTERM so the graceful path still lands).
• One thaw at a time: the thaw claim marks the lease as thawing, so a second exec/resume/pause that finds the thaw already in flight waits for its commit instead of thawing twice. After the wait the state is re-checked under the lock: a second exec of a lease the first thaw's job now holds is refused (already has a job in flight), a pause of that lease is refused the same way a pause of a busy resident lease is, and a resume whose lease was destroyed mid-thaw raises instead of committing thawed state on the dead record.

Billing — frozen-resident is billed in full

A frozen lease — a none pause, or a freeze-only transparent suspend — keeps accruing billed_s == resident_s through the freeze. Justification: the lease still exclusively holds its VRAM slice and its seat — resident GPU memory is not free capacity, and freezing changes nothing the operator could sell to someone else. A free pause would be a billing-dodge vector (hold the card, stop the meter). A freeze-only transparent suspend bills only its actual frozen intervals in full (frozen_billed_s); the freeze_only flag clears on thaw, so the rest of the residency bills like a transparent lease (busy + idle + resume) — one freeze-only interval must not reclassify the lease's whole lifetime residency as billed. New observability meters: frozen_s, freezes. Settled freeze-only intervals are emitted as a billed busy usage row (the same treatment a none pause's residency gets in memory): the store charges only busy/resume rows, so a frozen interval that emitted nothing — or an analytics running row — would be billed in memory but free after a restart. On the front+node topology the front settles the node's frozen_billed_s counter into its own gpu_usage as that same charged busy component (one frontier per kind shared with the exec's busy seconds, so the interval lands exactly once: a reply re-read adds 0 and a front restart re-seeds from the durable trail instead of billing it twice). A none lease is the exception there: the front charges its residency (running_s) as that busy component, in place of the exec and idle seconds inside it, which is what the node bills.

A freeze-only interval open on a lease that restores dead — adoption fails, the row cannot continue the freeze — settles at the previous process's last sign of life as a billed busy row and closes. Left open it would accrue on a dead row forever and a later destroy would flush the whole outage as one inflated busy row (a purge drops it instead).

A spilled transparent lease bills 0 while it is paused, as before. On a hard-cap host a pause spills only when the usage probe can read the lease's VRAM and no non-worker VRAM is resident; a host without a hard cap has no probe and always spills. On a hard-cap host a transparent suspend with resident non-worker VRAM (or an unreadable probe) does not spill at all, whether or not the worker has child processes: it is freeze-only, holds its seat, and bills its frozen time in full.

The figures above are the lease's meters, and they hold on every deployment. The credit bucket is drawn for them on a single GPU host and on a node fleet alike: on a node fleet the front settles busy and resume seconds (a freeze-only transparent lease's frozen interval counts as busy, and so does a none lease's whole residency, its frozen time included), and on a single GPU host the ledger charges the same. See Billing.

Watchdog + resume accounting (hard-cap)

• resume_grace_s (default 5 s, operator-configurable with GATEWAY_GPU_HARD_CAP_RESUME_GRACE_S; the default is a conservative choice, not a measured optimum): for that long after a resume commit, the watchdog's kill test counts only the lease's non-worker pids. The worker's own slice — including our own re-reserve — is excluded; children still count in full, so a real runaway child still gets killed and flagged. After the grace the full attribution applies. A lease must never be abuse-flagged for an overshoot the gateway's re-reserve produced.
• On resume the gateway re-reserves the worker's slice minus the VRAM the frozen children still hold (cap_mib = max(0, slice_target − measured non-worker VRAM)), so the lease's total settles at about its slice, not slice plus children. The worker's reserve target becomes min(slice_target, cap_mib), measured against the ORIGINAL slice target — never against an already-shrunk one, so a capped resume does not shrink the lease's slice for its whole life — and a reserve without a cap restores the full target. The cap is sticky for the top-ups after each job. When host usage is unreadable the cap is 0: holding too little beats an overshoot that kills an innocent tenant (the blind watchdog is already a loud degraded state).

Failure modes

Idle auto-pause (accel.idle_timeout_s)

A lease that sits idle holds a seat on the card. Create takes an optional accel.idle_timeout_s: 0 disables auto-pause, otherwise a number of seconds in [60, 86400]; anything else is a 400 naming the field. A free-tier key keeps auto-pause on: 0 or more than 600 s is a 403 gpu_free_tier_limit (param accel.idle_timeout_s). Omitted:

The sweep runs in reap_idle (same timer as the existing reaper) and calls pause() — the POST /pause path — for a resident, non-frozen lease with no job in flight whose last activity (create, exec end, resume or thaw) is older than its timeout. Locks, freeze-only rules and billing are the manual verb's; a sweep that races an exec is refused like a manual pause and retried next tick. Exec or resume on the paused lease thaws it exactly as after a manual pause. A longer value does not stop the plane's existing seat-demand, memory-pressure and max_idle_hold_s parking of transparent leases.

GET /sandboxes/{id} meters carry idle_timeout_s and auto_paused (pauses the sweep fired). Kill switch: GATEWAY_GPU_IDLE_AUTOPAUSE=0 (flag on by default; the sweep only parks leases the plane was already parking at 600 s unless the tenant names a timeout).

Next: GPU sandboxes · errors