It's Not the Compute. It's the Bandwidth.
Almost every conversation about on-device AI begins and ends with compute. The spec sheets quote TOPS, the press releases lead with benchmark scores, and the comparison charts rank chips by how many trillions of operations per second they can theoretically perform. It is a clean way to talk about a chip, and a misleading way to predict whether that chip can run the AI workload you have in mind. The number that decides that is rarely on the spec sheet at all. It is memory bandwidth.
A model running on a phone is, mechanically, a giant file of weights being streamed from memory into the matrix-multiply units of an accelerator, one tile at a time, for every token or every diffusion step. The compute units do real work, but they cannot do it without weights, and the weights live in main memory because there is far too much of them to keep on-chip. That stream has a maximum speed, set by the memory subsystem of the SoC, and on a phone the memory subsystem is shared. The CPU, the GPU, the NPU, the display engine, and the OS itself all pull from the same pool of LPDDR over the same bus. There is no separate VRAM, no dedicated channel for AI work. Whatever bandwidth the chip is advertised as having is the total budget for everything happening on the device at that moment.
Why Concurrent Models Are a Bandwidth Problem, Not a Compute Problem
If you only run one model at a time, this rarely shows up as a hard limit. A single model has a known weight footprint and a known step rate, and modern NPUs are fast enough to consume weights at roughly the rate the memory bus can deliver them. The chip is busy, the bandwidth is busy, the user gets an answer. Nothing about that experience tells you what happens when a second model joins.
Real on-device AI products almost never run a single model. A useful pipeline is a stack of components running in overlap: a speech model transcribing while a language model interprets and a generative model renders the result, or any other combination where two or more networks share the device at once. The instant that second model starts streaming its weights, both models are contending for the same memory bus. The compute units on each accelerator are still capable of doing their work. They just spend more of their time waiting for data that has to wait its turn in line. Bandwidth, not compute, is what gets divided.
The Counterintuitive Part
The consequence surprises people the first time they see it on a profiler. The NPU graph shows headroom, the GPU is not pegged, the CPU has cores sitting idle. By every compute-side metric the chip should be able to do more. And yet adding a second concurrent inference makes both run slower than either would alone. Nothing is saturated except the memory bus. Two models reading weights at the same time means each one waits while the other's data moves, so throughput drops and latency rises while the device looks underutilized. The compute headroom is real. It is also unreachable, because the path between the weights and the math units has already been spoken for.
How to Read a Hardware Claim
This reframes how you should look at any on-device AI announcement. A chip rated at a higher TOPS number than its predecessor is not necessarily a chip that can run more concurrent AI work. If the memory bandwidth has not scaled in step, the additional compute is hypothetical for any workload that involves more than one model at a time. The benchmarks that get quoted are almost always single-model benchmarks, run in isolation with the rest of the device quiet. That is the easiest scenario for the chip, and the one that resembles real product usage least.
The harder and more honest measurement is sustained, concurrent inference: two models running together long enough that the bandwidth-bound behavior stabilizes, on a device with the OS and the display engine doing what they normally do, inside the thermal envelope of a phone rather than a benchtop. That measurement rarely makes the press release. It is the one that predicts whether the next generation of on-device AI products is actually possible on the hardware being shipped.
What This Means for Where On-Device AI Is Going
The ceiling on what a phone can do with AI is not where the TOPS chart suggests it is. It is wherever the memory bus runs out of room for one more concurrent model. Until the industry publishes bandwidth figures with the same prominence it gives compute figures, the picture of on-device capability that reaches the public will keep overstating what real multi-model workloads can actually do. The chips will get faster at the math. Whether the products built on them get more capable depends almost entirely on whether the memory subsystem keeps pace.