Quantization: What Actually Happens When You Try to Run AI on a Phone
Most people in ML have heard the word "quantization." Most of them glaze over the second you say "4-bit versus 8-bit." That's a problem, because quantization is the single most important tool you have for deploying models outside of a datacenter. And almost nobody understands it well enough to use it.
I built Foxleaf, a mobile app that runs Stable Diffusion on-device using Qualcomm's NPU. Not in the cloud. On the phone. That meant I had to quantize every component of a diffusion pipeline to fit inside the memory and thermal constraints of hardware as old as a Snapdragon Gen 1. Here's what I learned.
What Quantization Actually Is
Large AI models are, at their core, giant files full of floating-point numbers. Millions to billions of them. Each number represents a weight or parameter the model uses to make predictions.
The precision of those numbers, how many decimal places they carry, is what quantization targets. A model stored in FP32 (32-bit floating point) uses 4 bytes per parameter. FP16 cuts that in half. INT8 cuts it in half again. INT4, half again.
That's it. Quantization is reducing the numerical precision of a model's parameters to make it smaller and cheaper to run.
The math is straightforward. A 1-billion-parameter model at FP32 is roughly 4GB. At FP16, 2GB. At INT8, 1GB. At INT4, 500MB. Double the precision, double the size. Every time.
Why It Matters More Than You Think
Here's the part most cloud-native ML engineers miss: quantization isn't primarily about disk space. Storage is cheap. The real constraint is memory.
When a model runs inference, it loads into RAM (or VRAM, or shared memory on a mobile SoC). Every parameter has to be accessible. If your model needs 4GB of memory and your device has 8GB total, with the operating system, your app, and background processes all competing for that same pool, you have a problem.
The constraints that drive quantization decisions come in a few flavors, and all of them are hard walls.
Budget constraints. You have X dollars. You need to run inference on some model. Your budget determines your hardware, and your hardware determines your memory ceiling. In most cases, you cannot architect your way around this. The money is the limit.
Here's a real example. A 32-billion-parameter Qwen 3 model at FP16 eats 48-60GB of VRAM. Quantize it to INT8 and you're hovering around 30GB. That's the difference between needing a high-end instance and running comfortably on a g6 or g6e at half the cost. Same model. Same capabilities, roughly. Half the infrastructure bill. That's what quantization buys you in the cloud, before you even start talking about edge deployment.
Hardware constraints. This is the mobile world. A flagship phone in 2024 has 8-12GB of RAM shared between the CPU, GPU, and NPU. There's no adding more. There's no spinning up a bigger instance. What the silicon gives you is what you get.
Thermal constraints. This is the one cloud engineers never think about. Mobile chips throttle when they get hot, and running inference on higher-precision floats generates more heat. Quantized models don't give you a clean 2x reduction in thermal output just because the data is half the size, but the improvement is significant. Roughly 30% less thermal load in my testing. On a chipset that already runs hot, like the Snapdragon Gen 1, that 30% is the difference between sustained inference and the SoC throttling itself into uselessness. Every idle millisecond on the CPU matters, and lower precision means fewer cycles burning through your thermal budget.
In all three scenarios, quantization lets you fit a bigger model into a smaller box. You trade a small amount of numerical precision for a massive reduction in memory footprint and compute overhead. In practice, the quality loss from going FP32 to INT8 is often negligible for inference tasks. Going to INT4 starts to get noticeable, but it's still usable for many applications.
Quantization is how you right-size your model to your deployment target. It's arguably the most powerful optimization tool in the edge ML toolkit.
Where It Gets Ugly
Everything above sounds clean. Manageable. In theory, it is.
In practice, quantization is a pain in the ass.
I run Foxleaf's Stable Diffusion pipeline across Snapdragon Gen 1, Gen 2, Gen 3, and Snapdragon Elite. Each chipset has different NPU capabilities, different supported operations, and different runtime requirements. The Snapdragon Gen 1 is the worst-case device. It technically works, and works well, but it's old enough that many of the modern automated quantization tools don't fully support it.
Here's what nobody tells you about real-world quantization for on-device deployment.
Every component has to match. A diffusion model isn't one model. It's a pipeline: text encoder, UNet (or transformer), VAE decoder, safety checker. Each component gets quantized separately. Each component produces a QNN context binary tuned to a specific chipset. And every single one of those binaries has to be built against the same QAIRT SDK version using compatible quantization encodings. Mix and match, and you get garbage output. Not "slightly degraded" output. Garbage.
Deprecated formats will burn you. Older quantization encodings like w16a8 were supported in earlier QAIRT versions but are now deprecated. If your target device needs an older SDK version because the newer ones dropped support for its chipset, you're stuck hunting for a quantization path that works with outdated tooling. The automated pipelines on Qualcomm AI Hub help with some of this, but they don't cover every edge case, especially the ones involving hardware that's fallen off the support matrix.
Each quantization attempt is a 30-minute blind bet. Quantizing a model component ties up your CPU for roughly 30 minutes. You don't know if the output will work until it's done and you try to run it. When you're iterating across four chipsets, multiple pipeline components, and different quantization strategies, the time adds up fast. I've burned entire days on quantization runs that produced unusable artifacts.
Numerical divergence between chipsets is real. The same model quantized with the same settings can produce visually different results on Gen 1 versus Gen 3. The NPU architectures handle fixed-point arithmetic differently. Calibration data matters, and what calibrates well for one chipset might not for another.
The Calibration Problem
Quantization isn't just "make the numbers smaller." To quantize well, you need calibration data: representative inputs that the quantization tool uses to determine the optimal range for each layer's parameters. Bad calibration data means the quantizer clips values in the wrong places, and your model's output quality tanks.
For image generation, this means running a representative set of prompts through the model and capturing the activation ranges. The quality of your calibration set directly determines the quality of your quantized model. It's not something you can skip or half-ass.
What I Actually Ship
For Foxleaf, the pipeline looks like this: ONNX export from the source model, calibration runs against curated prompt sets, quantization to architecture-specific QNN context binaries for each supported chipset, integration testing across all target devices. For the Whisper speech model (tiny.en), I use GGUF quantization at q5_1, which hits a good balance between size and transcription accuracy.
I also experimented extensively with Vulkan compute as an alternative runtime path. Didn't ship it. The QNN/NPU path was consistently faster and more power-efficient. But the exploration informed how I think about fallback strategies for devices where NPU acceleration isn't available.
The result is a diffusion pipeline that runs entirely on-device, generates images synchronized to audiobook narration in real time, and works on hardware going back three generations.
The Takeaway
Quantization is how you make AI models work in the real world. The world where hardware has limits, budgets exist, and you can't just throw more GPUs at the problem. It's a powerful tool. It's also an unforgiving one.
If you're evaluating edge deployment for your organization, or you're an ML engineer who's never had to think about what happens when your model leaves the cloud, understand this: the model is the easy part. Getting it to run efficiently on constrained hardware, across multiple device generations, with consistent quality, that's where the engineering actually happens.