On-device inference
On-device inference means running a trained AI model on the device itself, such as a phone or a PC, instead of sending the input to a server. AGI, Inc. runs its AGI-ACT action models on the device through AGI-RUN, its own inference engine, and uses a hybrid setup when a task needs more capacity.
Last updated
Related terms
On-device action model
An on-device action model is an action model small enough to run on a phone's or PC's own chip.
Hybrid inference
Hybrid inference runs an AI model on the device where the hardware allows and sends heavier steps to the cloud or on-prem servers.
Inference engine
An inference engine is the software that runs a trained AI model on the chips a device has.
NPU (neural processing unit)
An NPU (neural processing unit) is the part of a chip built to run AI models quickly and at low power.
Also called | Edge inference, local inference |
|---|---|
Runs on | The device's CPU, GPU or NPU |
AGI product | AGI-RUN, an on-device inference engine |
Alternatives | Cloud inference, hybrid inference |
Why run inference on the device?
For the tasks a device can handle, on-device inference keeps the input on the device and answers without a round trip to a server. It also keeps working with no connection. For an agent, the input is often what is on the screen, which can include messages and account details.
Setup | Where the model runs | Works with no connection | Input leaves the device |
|---|---|---|---|
On-device | The device's own chip | Yes | No |
Hybrid | The device, with heavier steps sent to the cloud | For steps the device handles | Only for steps sent off the device |
Cloud | A server | No | Yes |
What limits on-device inference?
Memory and power set the limits. A phone has far less memory than a server, so on-device models are smaller, and long tasks draw on the battery. Many products use a hybrid setup that sends heavier steps to the cloud.
How does AGI, Inc. use on-device inference?
AGI-ACT can run on the device through AGI-RUN, AGI's inference engine for phones and edge hardware. In benchmarks published in July 2026 on a Samsung Galaxy S25 Ultra, AGI-RUN processed input about 5x faster than each model's own native engine on average.
FAQ
Is on-device inference more private?
For the steps that run on the device, yes, because their input does not leave it. In a hybrid setup, steps sent to the cloud do leave the device, so it matters which steps run where.
Does on-device inference work offline?
The model can run with no connection. At Snapdragon Summit 2026, AGI's action model completed a task on a phone with no connection, running on the Snapdragon NPU. Apps that need the internet still need it.
What does it take to run a model on a phone?
It takes a model small enough for the phone's memory and an inference engine that runs it fast on the phone's chips. AGI, Inc. builds both, the AGI-ACT models and the AGI-RUN engine.
