Prefill throughput
Prefill throughput is how fast an AI model processes its input before it starts producing output, measured in tokens per second, and it sets the pace of agents that read a lot and write little. AGI-RUN, AGI's on-device inference engine, averages about 5x the prefill throughput of each model's own native engine.
Last updated
Related terms
On-device inference
On-device inference means running a trained AI model on the device itself, such as a phone or PC, instead of on a server.
Inference engine
An inference engine is the software that runs a trained AI model on the chips a device has.
NPU (neural processing unit)
An NPU (neural processing unit) is the part of a chip built to run AI models quickly and at low power.
Measured in | Tokens per second |
|---|---|
Phase | Prefill, before decode |
Why it matters | Agents read much more than they write |
AGI-RUN result | About 5x each model's native engine on average, up to 6.9x |
What is the difference between prefill and decode?
Every response has two phases, prefill and decode.
Phase | What happens | Measured as |
|---|---|---|
Prefill | The model processes the whole input | Prefill throughput, in tokens per second |
Decode | The model writes its output one token at a time | Decode speed, in tokens per second |
Why does prefill set the pace for agents?
An agent's input is long, often the current screen and its recent steps, and its output is a short action. Most of the wait comes from reading that input, so a faster prefill lets the agent take in more context in the same time, or run a larger model.
How does AGI-RUN perform on prefill?
In benchmarks published in July 2026 on a Samsung Galaxy S25 Ultra, AGI-RUN averaged about 5x the prefill throughput of each model's own native engine across six small vision-language models, up to 6.9x. Against the full field of open inference frameworks, it was faster in 58 of 60 cases.
FAQ
What is a token?
A token is a small unit of input or output that a model processes, often part of a word or a patch of an image. Prefill throughput counts how many input tokens a model processes per second.
Is prefill throughput the same as latency?
No. Latency is the total time a person waits for a response. Prefill throughput is one part of it, and for agents with long inputs it is often the largest part.
Why measure prefill on a phone?
Phones have far less compute than servers, so slow prefill shows up as a visible wait. AGI-RUN was benchmarked on a Samsung Galaxy S25 Ultra to measure the speed people get on a real phone.
