NVIDIA announced Snapshot, an open-source checkpoint and restore system for GPU workloads on Kubernetes. It captures fully initialized CPU and GPU memory state, enabling rapid recovery of GPU workloads. This speeds up startup and recovery times, reducing delays when inference demand spikes or replicas fail.
Key facts
| Fact | Detail | The source says |
|---|---|---|
| Restore Time | 3.6 to 42.7 seconds | “Restoring from a Snapshot artifact brings inference replicas online in 3.6 to 42.7 seconds across tested configurations, delivering 8x to…” |
| Checkpoint Creation Time | 15 seconds to 5.5 minutes | “Checkpoint creation takes 15 seconds to 5.5 minutes depending on model size, and the combined checkpoint-plus-restore time is shorter than…” |
| Supported Workloads | Multi-GPU workloads with up to 1 TB checkpoints | “The system supports multi-GPU workloads using tensor parallelism, data and expert parallelism, and decode context parallelism for large…” |
What happened
NVIDIA introduced Snapshot, an open-source checkpoint and restore system for GPU workloads on Kubernetes. It captures fully initialized CPU and GPU memory state, enabling rapid recovery of GPU workloads. This speeds up startup and recovery times, reducing delays when inference demand spikes or replicas fail. Restoring from a Snapshot artifact brings inference replicas online in 3.6 to 42.7 seconds across tested configurations, delivering 8x to 36x speedups over cold starts that take minutes. The system supports multi-GPU workloads using tensor parallelism, data and expert parallelism, and decode context parallelism for large mixture-of-experts models with checkpoints up to 1 TB across eight GPUs. Benchmark results on NVIDIA B200 GPUs with VAST NFS storage show that restoring GPU state dominates large restores at 18 to 29 seconds, while CRIU-based CPU process restore takes 4.2 to 10.8 seconds.
What to weigh
- Restoring GPU state takes longer than CPU process restore.
- Checkpoint creation time varies with model size.
FAQ
What is the restore time for Snapshot?
Restoring from a Snapshot artifact brings inference replicas online in 3.6 to 42.7 seconds across tested configurations, delivering 8x to 36x speedups over cold starts that take minutes.
What workloads does Snapshot support?
The system supports multi-GPU workloads using tensor parallelism, data and expert parallelism, and decode context parallelism for large mixture-of-experts models with checkpoints up to 1 TB across eight GPUs.
How long does it take to create a checkpoint?
Checkpoint creation takes 15 seconds to 5.5 minutes depending on model size, and the combined checkpoint-plus-restore time is shorter than a single cold start in every tested configuration.
Source: NVIDIA
