Dev proves LLMs will run on anything – even a $10 microcontroller
A developer known as SlvDev has demonstrated that even a large language model can run on an ESP32-S3 microcontroller costing less than $10, a device normally used for basic sensors and IoT tasks rather than generative AI. The feat, documented on GitHub and showcased on the Better Stack YouTube channel, shows how aggressive memory-saving techniques can squeeze AI workloads onto hardware far too small and cheap to handle them by conventional means.
To achieve this, SlvDev used TinyStories, a 28.9-million-parameter model from Microsoft Research, roughly 10,000 times smaller than modern frontier models. Quantisation cut the model's memory footprint from 60MB to 14.9MB, and a technique borrowed from Google's Gemma models, called per-layer embedding, offloaded around 12MB of weights to flash storage, leaving only about 2MB needed in active memory. The result was a working chatbot-style model running at 9.88 tokens per second on the ESP32-S3's 520KB of SRAM and 8MB of PSRAM, faster than the average person can read.
In short
- Developer ran a small LLM on a sub-$10 ESP32 microcontroller
- Used quantisation and flash offloading to shrink memory needs drastically
- Achieved nearly 10 tokens per second, faster than human reading speed
Explained simply
Large language models are software trained on text to generate responses. Most need substantial memory and computing power, so they usually run on servers or capable computers; an ESP32-S3 is a small, inexpensive microcontroller more commonly used in sensors and connected devices.
SlvDev’s demonstration uses TinyStories, a much smaller model than the systems behind widely used AI chatbots. Techniques that compress the model and keep some of its data in flash storage reduce how much needs to be held in working memory.
The result shows that some language-model tasks can run on very limited hardware. It does not mean that a low-cost microcontroller can run the much larger models used in today’s leading chatbots.
AI Geopolitics Politics Technology