研究人员对 NVIDIA 闭源驱动中的 cuda-checkpoint 功能进行了深入分析,揭示了其冻结和恢复 CUDA 进程状态的内部机制 [1]。该功能通过将 GPU 状态序列化至主机内存来保存运行中进程的上下文,并在需要时完全重建以恢复执行 [1]。逆向工程发现,checkpoint(检查点)与 restore(还原)操作的时间消耗主要集中在 staging buffer(暂存缓冲区)的分配与释放上,而非 PCIe 数据传输本身 [1]。
研究指出,由于未启用透明大页(Transparent Huge Pages, THP),页面零初始化带来的开销构成了性能瓶颈 [1]。通过优化内核设置并绕过部分驱动调用以直接操作管道协议,研究人员成功规避了 libcuda 的上下文初始化流程中的 lock/unlock 步骤,将其耗时从约 200ms 降低至毫秒级 [1]。启用 THP 后,页面分配开销显著减少,进一步加速了整个 checkpoint/restore 周期 [1]。
测试数据显示,优化后的冷启动速度达到显著提升:checkpoint 速率约为 15.5 GiB/s,restore 速率约为 12.7 GiB/s [1]。整体而言,这些改进措施将 CUDA 进程的冷启动时间缩短了约四倍 [1]。
Researchers have analyzed the closed-source NVIDIA driver to reveal the internal mechanisms of its cuda-checkpoint feature, which freezes and restores running CUDA process states [1]. The study found that this functionality serializes GPU state into host memory before resuming execution [1]. Analysis indicates that the time consumed during checkpointing and restoration is primarily attributed to staging buffer allocation and deallocation rather than PCIe data transfer itself [1]. Enabling Transparent Huge Pages (THP) significantly reduces page zero-initialization overhead, thereby accelerating the checkpoint/restore cycle [1]. By directly manipulating pipe protocols to bypass context initialization within libcuda, researchers reduced lock/unlock steps from 200ms to milliseconds [1]. These optimizations resulted in a cold start speedup of approximately four times, achieving checkpoint speeds of roughly 15.5 GiB/s and restore speeds of about 12.7 GiB/s [1].