World news and discussion on TalkFuse.
Kog is going deeper to squeeze more inference out of GPUs
French startup Kog is developing software intended to make existing data-centre GPUs run AI inference much faster, arguing that better low-level optimisation can reduce a major cost and speed bottleneck without requiring specialised chips. Its early demonstration attracted significant business interest, particularly from companies using AI for professional software engineering and prompt-based app or game creation, where faster results could improve productivity and revenue.
Kog says its technology achieved 3,000 tokens per second for a single request using its open-source, two-billion-parameter Laneformer model on AMD MI300X and Nvidia H200 GPUs, but it must prove the approach works at scale with much larger language models to support its claim of up to 30 times faster inference. Founder Gaël Delalleau believes modern GPUs’ growing memory bandwidth is underused, and says his background in physics and cybersecurity informs Kog’s focus on understanding and optimising GPU behaviour at a very low level.
In short
- Kog aims to accelerate AI inference on existing data-centre GPUs.
- Its early results use a small model, not large commercial LLMs.
- Faster inference could lower costs and improve AI workflow responsiveness.
Explained simply
AI inference is the process of using a trained model to produce answers, code or other results. It runs on data-centre GPUs, which are powerful processors also used to train AI, and the time and computing power it takes can affect how much AI services cost.
Kog is a French startup founded by Gaël Delalleau. It is developing software to make existing GPUs handle inference faster, by changing how the processors’ capabilities are used; other approaches include designing specialised chips.
Faster inference could matter to businesses that rely on AI tools, including coding assistants and services that create apps or games from prompts. Kog’s early results use a relatively small model, so testing with larger models will help show whether the approach can deliver similar gains in broader use.
Americas Art Business Celebrity Companies Culture Entertainment World