Your On-Device AI Works Great for 20 Minutes. Then It Melts.
I went into building Foxleaf expecting memory to be the main fight. Fitting Stable Diffusion and Whisper into the RAM constraints of a mobile device, managing shared memory on the SoC, keeping the OS from killing my process. Those were the problems I prepared for.
Thermals were harder. And I didn't even think about them until they nearly killed the project.
Why Most Developers Have Never Thought About This
If your app is a UI layer over a database and an API, thermal management is irrelevant. You're not generating enough sustained compute load to matter. The phone handles it. You never notice.
AI inference is a different category of work. It's computationally intense in a way that most mobile apps simply aren't. Running one model is hot. Running two models concurrently (which is what Foxleaf does with Whisper for real-time transcription and Stable Diffusion 1.5 for image generation) multiplies thermal output in ways that aren't linear. And if your app is just a wrapper around a cloud API, none of this applies to you. The heat is being generated in a datacenter somewhere. Congratulations on dodging the problem.
For everyone else running inference on the device itself, thermals are a constraint as hard as memory. Maybe harder.
The Burn Strategy Question
There's a fundamental strategic question in on-device AI: is it better to burn slow and steady, or go full blast and maximize idle cooling time between inference runs?
There's no universal answer. It depends on your workload pattern.
If you're doing batch processing (the user hits a button, something runs, it's done) you might be able to blast through on performance cores and let the device cool before the next request. Short bursts with recovery windows.
If you're doing streaming work, like Foxleaf, that strategy will destroy you. Foxleaf has to deliver AI-generated images synchronized to audiobook narration in real time. Transcription and image generation run in overlapping batches on a continuous schedule. We can't pause inference to let the device cool down. Delayed images mean a broken user experience. The schedule is the schedule.
For sustained workloads, you need thermal sustainability. Not peak performance.
How Snapdragon Chips Handle Heat (And Why the Defaults Will Hurt You)
Snapdragon SoCs use a big.LITTLE-style architecture with distinct core clusters. Performance cores run fast and run hot. Efficiency cores run slower but generate significantly less heat. Less electricity going in, less processing per cycle, less thermal output. There's a battery dimension here too: more electricity means faster drain, and nobody wants an app that kills their phone in 45 minutes.
By default, Android's scheduler pushes compute-heavy threads onto performance cores. This makes sense for most apps. You want your UI thread running as fast as possible, and the workload is usually short enough that heat doesn't accumulate. But for sustained AI inference, the default is wrong. If you don't explicitly pin your work to a specific core type, you're running on performance cores whether that's a good idea or not.
This is the kind of thing that works perfectly in testing (short sessions, fresh device, room temperature) and then falls apart in production when real users run the app for 30 minutes in a warm room.
How Foxleaf Splits the Work
Foxleaf runs two AI workloads simultaneously, and we deliberately split them across different hardware.
Image generation runs on the NPU via QNN. The NPU's performance is solid for diffusion models, and it generates a fraction of the heat the GPU would for the same work. This was a deliberate architectural choice. The GPU could do it, but the thermal cost wasn't worth it.
Whisper (speech-to-text) runs on the CPU. We split this from the NPU for several reasons. First, concurrency. Transcription and image generation often run at the same time. We don't want them competing for the same processing resources. Second, Whisper's GPU acceleration path on Android is limited enough that rebuilding it for GPU would mean significant effort to potentially run hotter, on a task that already completes in 6.6 seconds for 30 seconds of audio on CPU. Not worth reimplementing.
This split (NPU for vision, CPU for audio) gives us independent thermal budgets and eliminates resource contention between the two workloads.
The Death Spiral
Before we solved the thermal problem, here's what happened during a longer Foxleaf session. Anything over 20 minutes.
Whisper starts transcribing audio chunks on performance cores. It's fast. About 6.6 seconds per 30-second chunk. Plenty of idle time for cooling between runs.
But the heat accumulates. After several cycles, Android's thermal throttling kicks in. The OS starts capping clock speeds to protect the hardware. Transcription slows down. 10 seconds. 15. 20.
Now the idle window between transcription runs is shrinking. Less cooling time means more heat retention going into the next run. Throttling gets worse. 25 seconds. 30.
Eventually you're at 30-40 seconds to transcribe a 30-second audio chunk. Your idle cooling window has collapsed from 30-plus seconds to almost nothing. The spiral accelerates from here. The device overheats. Functionality degrades. The app black-screens.
Session over.
This isn't a theoretical failure mode. This is what happened on Snapdragon Gen 1 devices in real usage before we fixed it.
The Fix: Slower Is Faster
The turning point was pinning Whisper to efficiency cores.
Transcription time went from about 6.6 seconds to roughly 12 seconds per chunk. On paper, that's almost twice as slow. But thermal output dropped by 45%.
That 45% changes everything. At 12 seconds per chunk, the device still has an 18-second idle window per 30-second audio segment. The efficiency cores generate so much less heat that the device stays within its thermal envelope indefinitely. No throttling. No spiral. No black screen.
A 12-second transcription time that stays at 12 seconds for a two-hour session beats a 6.6-second time that degrades to 40 seconds after 20 minutes. That's not a tradeoff. That's the only viable path.
Inefficiency Is Heat
One thing I didn't appreciate going in: every inefficiency in your app contributes to your thermal budget. Not just inference. Everything.
Poor UI design that triggers unnecessary recomposition. Background threads doing work they don't need to. Logging that's too verbose in production. Anything that touches the CPU (and CPU is significantly worse for thermal management than GPU or NPU) adds heat to a system that's already running near its limits.
When your AI workloads are pushing the thermal ceiling, there's no room for sloppy code elsewhere in the app. Every unnecessary CPU cycle is heat you can't afford.
What This Means If You're Planning On-Device AI
If you're evaluating edge AI deployment and your planning conversations are entirely about model size and memory footprint, you're solving the wrong problem first.
Memory is a hard constraint, yes. But it's a static one. You can calculate it upfront. You quantize your model, measure the footprint, and either it fits or it doesn't.
Thermals are a dynamic constraint that only reveals itself under sustained load. Your benchmarks will look great. Your demos will run perfectly. And then your users will run the app for 25 minutes and it will die.
The questions you need to be asking: What's the sustained thermal output of your inference workload? What's the thermal ceiling of your target hardware? How long can you run before throttling starts? What's your throttling degradation curve? Can you architect your workload to stay below the thermal ceiling indefinitely, not just for the length of a demo?
If you can't answer those questions, you're not ready to ship.