Nvidia and Cerebras are selling performance their customers will (probably) never see
Nvidia and Cerebras are promoting exceptionally high single-user AI inference speeds, but the article argues that most customers are unlikely to experience them in real-world services. Such figures are useful marketing benchmarks, yet commercial operators normally batch many requests together to make inference economical, which reduces the relevance of peak per-user token-generation rates.
Nvidia says its Groq-3-based LPX racks reached 3,400 tokens per second on Gemma 4 31B, around four times Cerebras’ earlier result, while Cerebras claims near-equivalent performance from its new CS-4 systems. Both architectures rely heavily on fast on-chip SRAM, but limited memory restricts batching: the article estimates that a 256-LPU Nvidia rack and a 132GB Cerebras CS-4 rack could each support only about 12 simultaneous 100,000-token sequences for this model before memory is exhausted.
- Peak inference speeds may not translate into practical customer performance.
- Limited memory constrains batch sizes on both systems.
- Commercial AI services prioritise efficient throughput over single-user speed.
New here? Start with this
AI inference is the stage when a trained AI model responds to a user, for example by generating text a token at a time. Nvidia and Cerebras make specialist systems designed to run these models, and both promote how quickly one user’s request can be answered.
Their systems use fast memory built into the processors, which can help generate text quickly but is limited in size. In a paid service, operators often handle many users’ requests together to make better use of their equipment; that can affect the speed each user sees.
The figures therefore describe performance under particular test conditions, rather than necessarily the experience of people using a commercial AI service. The gap matters to customers and operators comparing systems for real-world workloads.
Both sides, in good faith
The strongest fair case each way — we don't pick a winner.
The case for
Peak single-user inference speeds can show what specialised hardware is capable of and may matter for applications where responsiveness is crucial, such as interactive assistants or coding tools. Public benchmarks also give buyers a way to compare systems and may encourage competition to make fast inference more widely available.
The case against
Those headline rates may say little about the experience or cost most customers will get from a commercial service, where operators typically batch requests to use hardware efficiently. If limited on-chip memory constrains how many long sequences can run at once, real-world throughput and economics may matter more than the fastest result for one user.