PyTorch 2.15 tackles the ‘one bad GPU’ problem

PyTorch 2.15 introduces a rewritten backend for Nvidia’s NCCL communications library, aiming to make large-scale GPU training easier to diagnose and recover when a GPU fails or slows down. This matters because a single faulty or stalled GPU can disrupt a distributed training job, while the new tools are intended to help orchestrators remove or replace a lagging worker.

The previous backend could create many communicators per process group, contributing to deadlocks and unpredictable performance. The new version limits each process group to one communicator and adds a “flight recorder” that saves operation snapshots for investigating failures. The old backend remains available as a rollback in 2.15; if the replacement works as expected, it is due to be removed in 2.16, planned for early next year. The release also includes asynchronous data movement, Apple parallel-processing improvements and support for Nvidia’s next-generation Blackwell GPUs, across 3,117 commits from 583 contributors.