Technology news and discussion on TalkFuse.
Google’s C++ XSE engine serves machine-learning models without Python runtime
Google has open-sourced XSE, a C++ engine that serves machine-learning models without running a Python interpreter for each request. It is designed for smaller, task-focused models such as ranking and recommendations, where high request volumes and tight latency requirements make serving speed important.
Google says XSE handles more than six million queries per second across YouTube Shorts, Google Ads and Search. In tests on short requests, it achieved two to five times the throughput of Google’s best optimised Python server while using a quarter of the memory. Models can be written in Python with JAX, exported with their computations and trained weights, then compiled by XLA for CPUs or GPUs; Google also plans support for exporting PyTorch models to serve on TPUs.
In short
- Google has open-sourced its C++ XSE model-serving engine.
- It handles over six million queries per second across Google services.
- Tests showed up to five times the throughput using a quarter of the memory.
Steelman: both sides, in good faith
The strongest fair case each way — we don't pick a winner.
The case for
XSE could make serving smaller machine-learning models faster and less resource-intensive, which matters for services handling millions of requests under tight latency limits. Its reported gains in throughput and memory use, alongside support for models developed in familiar tools, may help more teams deploy these systems efficiently.
The case against
The reported performance figures come from Google’s own tests and deployments, so independent results would help establish how well they generalise to other workloads and organisations. A specialised serving engine may also add migration and maintenance costs, and its value depends on how broadly it supports teams’ existing models and hardware.
Explained simply
Machine-learning models find patterns in data to make predictions, such as which videos a person may want to watch or which search results are most relevant. Once trained, a model needs to be served: a software system runs it on incoming requests and returns predictions.
Python is widely used to build and run these systems, but starting a Python process for every request can use time and memory. XSE is a C++ engine for running models without that per-request Python runtime, aimed at small, specialised tasks that receive many requests and need quick responses.
Google’s services use ranking and recommendation models to select or order content and adverts. The speed and computing resources needed to serve these models can affect how quickly such services respond and how much infrastructure they require.
AI Technology Trending Weird & Viral
Read the full article at the source →
Originally published by The Register as “Google gooses Python AI performance by… removing Python”.