# PyTorch 2.15 tackles the ‘one bad GPU’ problem

- URL: https://talkfuse.com/story/pytorch-2-15-tackles-the-one-bad-gpu-problem/
- Published: 2026-11-06T18:36:42+00:00
- Updated: 2026-11-06T21:03:35+00:00
- Original publisher: The Register
- Original article: https://www.theregister.com/devops/2026/11/06/pytorch-215-tackles-the-one-bad-gpu-problem/5308227
- Topics: AI, Technology

## Key points

- PyTorch 2.15 aims to make GPU training failures easier to recover from.
- Its new NCCL backend limits each process group to one communicator.
- A flight recorder captures snapshots to help diagnose stalls and failures.

## Summary

PyTorch 2.15 introduces a rewritten backend for Nvidia’s NCCL communications library, aiming to make large-scale GPU training easier to diagnose and recover when a GPU fails or slows down. This matters because a single faulty or stalled GPU can disrupt a distributed training job, while the new tools are intended to help orchestrators remove or replace a lagging worker.

The previous backend could create many communicators per process group, contributing to deadlocks and unpredictable performance. The new version limits each process group to one communicator and adds a “flight recorder” that saves operation snapshots for investigating failures. The old backend remains available as a rollback in 2.15; if the replacement works as expected, it is due to be removed in 2.16, planned for early next year. The release also includes asynchronous data movement, Apple parallel-processing improvements and support for Nvidia’s next-generation Blackwell GPUs, across 3,117 commits from 583 contributors.

---
Summarised by TalkFuse (https://talkfuse.com/). Quote the original publisher for the reporting and link this page for the summary and discussion.