GPUs could explode to multiple TB with new storage-inspired memory tech
Sandisk and SK Hynix are developing high-bandwidth flash (HBF), a NAND-based memory technology intended to give AI accelerators far more local capacity than conventional high-bandwidth memory (HBM). It could allow GPUs to combine fast HBM with terabytes of lower-cost, high-capacity flash memory, potentially reducing the need to spread large AI models across several processors.
Sandisk says first-generation HBF could deliver read speeds of up to 1.6 TB/s and 512 GB per 16-layer module, more than 14 times the capacity of current HBM4 stacks; later versions are intended to exceed 2 TB/s and eventually reach 3.2 TB/s. However, NAND has microsecond-scale latency and limited write endurance compared with DRAM, so HBF is not expected to replace HBM outright. Instead, it would support read-heavy AI inference work, while HBM handles write-intensive operations such as generating key-value caches.
- HBF could add terabytes of fast flash memory to AI GPUs.
- It offers much higher capacity than HBM, with important compromises.
- The technology is aimed at cheaper, read-heavy AI inference.
New here? Start with this
Sandisk and SK Hynix are working on a new type of computer memory called high-bandwidth flash that could store far more data on graphics processing units (GPUs) – the specialised chips used for artificial intelligence work. Unlike existing memory used in GPUs, which is expensive and limited in capacity, this new flash-based memory would be cheaper and could hold much more information.
Modern AI systems like large language models require enormous amounts of memory to run. Currently, GPUs are constrained by the amount of high-speed memory they can physically hold, which means large AI models often need to be split across multiple separate processors. This approach is costly and complicated.
The new flash memory technology would sit alongside a GPU's existing memory rather than replace it, handling different tasks based on what each type does best. High-bandwidth flash would excel at quickly reading stored information, whilst the existing memory would handle the more demanding work of writing and updating data during AI processing.
Both sides, in good faith
The strongest fair case each way — we don't pick a winner.
The case for
High-bandwidth flash could give AI accelerators much more local memory at lower cost, letting them run larger models without spreading work across as many processors. If its read speeds prove sufficient for inference, pairing it with HBM could improve capacity and efficiency while preserving fast memory for write-intensive tasks.
The case against
The proposed speeds and capacities are development targets, and flash has higher latency and lower write endurance than DRAM, so its practical value for demanding AI workloads remains uncertain. Adding another memory tier may also bring design and software complexity, while leaving HBM necessary for the operations where flash performs poorly.