Technology & privacy
Why Apple Silicon changed local speech AI
It is not raw speed. It is that the CPU and GPU share the same memory, which removes the expensive step that makes running a model on a laptop slow everywhere else.
Last updated
The copy that used to dominate
On a traditional computer the CPU has its memory and the graphics processor has its own, separate memory. Fast parallel arithmetic — which is what running a model is — happens on the GPU. So the model's weights have to be moved there.
For a model of a couple of gigabytes, that transfer is not free, and on a laptop with a shared bus it can cost more time than the actual computation. It is why local AI on laptops was a curiosity for years while the same models ran comfortably in data centres with hardware designed around the problem.
Apple Silicon uses unified memory: one pool, addressed by the CPU, the GPU and the Neural Engine. The model is loaded once and everything works on it in place. The copy does not happen because there is nowhere to copy it to.
| Separate memory | Unified memory | |
|---|---|---|
| Where the model lives | System memory, copied to GPU memory | One pool, shared |
| Cost of using the GPU | A transfer proportional to model size | None |
| Effect on a 2 GB model | Significant, on every load | Nothing to pay |
| Ceiling on model size | The GPU's own memory | The machine's total memory |
| Practical result for dictation | Latency you notice | Latency you do not |
The Neural Engine
Alongside the CPU and GPU, Apple Silicon includes hardware built specifically for the arithmetic neural networks do. It is efficient rather than merely fast, which matters on a laptop running from a battery.
Not every workload uses it — it suits some operations and model shapes better than others — but where it applies, the power cost of local inference drops enough to change what is reasonable to run continuously on a machine with no power socket.
Why several apps refuse Intel Macs
This is the practical consequence, and it is worth stating plainly rather than hiding in a requirements list.
A local speech model on an Intel Mac is slow enough that dictation stops feeling like an input method and starts feeling like a request. Superwhisper's own FAQ puts it directly: Intel Macs work best with Cloud models. Offline models only run really well on Apple Silicon macs.
Given that, a developer has three options. Ship to Intel with a quietly bad experience and absorb the support load. Ship a cloud fallback and abandon the architectural claim. Or require Apple Silicon and lose those customers. VV takes the third — Apple Silicon, macOS 14 Sonoma or later — which is a smaller market and a product that works.
What this means in practice
- Latency low enough to be an input method. A dictation that returns in a fraction of a second is typing. One that takes three seconds is a request you are waiting on, and people stop using it.
- Memory is the real constraint, not speed. An 8 GB Mac can run dictation-grade models comfortably; the question is whether the app releases the memory when idle or holds a gigabyte all day.
- Battery cost is real but modest. Local inference uses your own silicon. On Apple Silicon that is a few percent over a long writing session rather than something you plan around.
- Bigger models became reasonable. When the ceiling is the machine's total memory rather than a graphics card's, a 16 GB Mac can hold models that would not have fitted at all.
The thing this does not mean
Questions
Why does VV require Apple Silicon?
Local speech models run well on Apple Silicon and poorly on Intel, because of unified memory and the Neural Engine. Shipping to Intel would mean a dictation experience slow enough that people would stop using it, so we require M1 or later and say so on the system requirements page.
How much memory do I need?
A dictation-grade model is typically a few hundred megabytes to a couple of gigabytes while loaded. That is comfortable on an 8 GB Mac if the app releases it when idle. Watch Activity Monitor for an hour before committing to any local app.
Does local AI drain the battery?
It uses your own silicon rather than a server's, so yes, somewhat. On Apple Silicon it is modest — a few percent over a long session — but it is not nothing on a flight with no power.
Is the Neural Engine always used?
Not for every operation. It suits some model shapes better than others, and implementations vary in how much they use it versus the GPU. Where it applies, the efficiency gain is the part that matters on a laptop.
Will local models catch up with cloud models?
The gap has narrowed considerably and is now small on clean audio. It is unlikely to close entirely, because a data centre can always run something larger. What changed is that the remaining gap stopped being the deciding factor for most everyday writing.