Skip to content
Fathom Layer
GlossaryGlossary6 MIN READ

What is on-device AI?

Opened camera housing showing its lens module connected to a local processor board
AI-generated editorial illustration · Conceptual scene

On-device AI means running a model's inference on the phone, laptop or other device in your hands, instead of sending the input to a model on a remote server. The model's weights are stored on the device, and the prompt, image or audio is processed by the device's own CPU, GPU or NPU. "On-device inference" is the same idea stated more precisely: training still happens in a data centre; only the step that turns an input into an output moves to the device.

How it works

Three pieces have to line up.

  • A model small enough to fit. In its June 2025 model update, Apple said the on-device language model behind its Foundation Models framework had about 3 billion parameters, and described it as good at summarization, entity extraction, text understanding, refinement, short dialog and creative content, but "not designed to be a chatbot for general world knowledge". That is the typical profile of a small language model.
  • Compression. In the same June 2025 update, Apple said it compressed that model to 2 bits per weight using quantization-aware training, and that "the KV cache was quantized to 8 bits per weight". Treat these figures as a June 2025 snapshot; later Apple models may differ. Quantization is one of the main ways to shrink a model to fit a phone's memory.
  • An accelerator and a runtime. Chip and platform vendors each ship their own; Qualcomm, AMD and Microsoft are examples, not a complete list. Qualcomm describes its Hexagon NPU as combining scalar, vector and tensor accelerators with a large shared memory, and its AI Hub says it offers 300+ models optimized for its devices. AMD lists its Ryzen AI Max+ 395 processor with an NPU of up to 50 TOPS, and up to 126 TOPS across the whole chip. Microsoft's developer guidance for Copilot+ PCs says many new Windows AI features require an NPU that can run at 40+ TOPS, and recommends Windows ML as the way to run models on the NPU.

On Android the model is often supplied by the operating system rather than the app. Google says AICore, an Android system service, manages the distribution and updates of its Gemini Nano model, so an app does not ship the model in its own download.

What the platform vendors say it buys you

Each claim below is the vendor's own, about its own platform.

Claim Who says it What they say
Works without a network Google Gemini Nano delivers generative AI "without needing a network connection or sending data to the cloud"
Data stays local Google AICore has no direct internet access and does not store a record of inputs or outputs after processing
Privacy by design Apple It protects privacy "with powerful on-device processing" and Private Cloud Compute
Lower cost Google On-device execution "reduces inference costs"
Power efficiency Qualcomm Its micro-tile inferencing is aimed at AI performance "at ultra-low power"

Latency is the benefit most often assumed: removing the network round trip should make responses start sooner. None of the sources above publishes a latency figure for it, so treat that as an inference, not a measured result. On-device inference still takes time, and a small model on a phone is not automatically faster than a large model on a data-centre GPU.

The limit, and the hybrid answer

Memory and power set a ceiling on model size, which is why on-device models are smaller than frontier cloud models. The vendors' own products show the result is usually hybrid. Apple's Foundation Models documentation tells developers that when they need more reasoning capability and context size, they can use Private Cloud Compute or another server model provider. Apple's security team describes Private Cloud Compute as designed for stateless processing, where personal data must not be retained after the response, and for verifiable transparency, where security researchers can check its guarantees. That is Apple's design claim about its own service.

So the practical question is rarely "on-device or cloud". It is which requests stay local and what happens to the ones that do not.

When on-device AI fits

  • Good fit: short, frequent tasks on personal data, such as summarizing a message, extracting a date from a flyer, transcription, image classification, or offline features.
  • Poor fit: open-ended questions that depend on broad world knowledge, long documents that exceed a small model's context, or tasks where quality matters more than where the data goes.
  • Check first: device support. Apple says Foundation Models needs a device that supports Apple Intelligence; Microsoft's 40+ TOPS requirement rules out many older PCs. An on-device feature can only reach users whose hardware qualifies.

Honest caveats

  • "On-device" describes where inference runs, not everything an app does. An app can process a prompt locally and still send analytics, logs or results elsewhere. Check what the app sends, not only where the model runs.
  • A hybrid system sends some requests off the device by design. Its privacy depends on the server side too.
  • TOPS measures peak arithmetic throughput. It does not say how much memory the device has, which limits model size, or how well a specific model runs.

In the index

Hardware for running models locally, from mini PCs to desktop GPUs, is listed under local AI workstations. Related terms: NPU, small language model, quantization, unified memory and local AI appliance.

Sources

END OF ANALYSIS

Related Intelligence