How much memory a PC needs to run local AI models

Local AI chatbots can run on ordinary PCs, offering offline use without subscription fees and keeping data on the device. However, the size and speed of models are limited by available memory, so a typical 16GB computer can handle smaller models but not the most capable ones.

Quantised models use about four bits per parameter: an 8B model such as Llama 3.1 takes about 4.9GB, while a 70B model takes 43GB. Faster graphics memory helps because generating text depends heavily on memory bandwidth; the article cites speeds from 4.42 tokens per second for a 70B model on Nvidia’s $3,999 DGX Spark to 205 for a 20B model on an RTX 5090. Some smaller open models approach older frontier systems in benchmark scores, but a trillion-parameter model may need around 600GB of combined memory.