ONNX: the reason a model trained elsewhere runs on your Mac

Last updated 11 September 2026

The problem it solves

A model is trained in a research framework — usually PyTorch — on large machines, by people whose job ends when the model is good. Running it in a desktop app is a different problem with different constraints, and carrying the entire training framework into a Mac app is not a reasonable answer.

ONNX separates the two. The trained model is exported into a portable description of its structure and weights. A runtime then executes that description on whatever hardware is available, using whatever acceleration that hardware offers.

What a runtime actually does

Reads the model description and executes the operations it defines, in order, on the input you give it. The interesting part is that it chooses how.

• Execution providers. The same model can run on the CPU, on the GPU via Metal, or on Apple's Neural Engine via Core ML, depending on what is available and what suits each operation.
• Graph optimisation. Operations are fused and reordered before execution, so the model runs faster than a naive walk through its structure.
• Quantisation support. Running a model in lower precision — 8-bit instead of 32-bit — for a large speed and memory saving at a small accuracy cost. This is often what makes a model fit on a laptop at all.

Where it fits in a dictation app

In VV, ONNX Runtime executes several of the models involved: the voice activity detector that trims silence, and ONNX-format speech models such as the Parakeet family. Other models use different runtimes — GGML-format models run through their own, and the local writing step uses llama.cpp.

An app typically ends up with more than one runtime, because the model families it wants to ship do not agree on a format. That is a normal engineering reality rather than a sign of anything wrong.

Should you care?

For developers

If you are building something in this area, three practical notes from doing it:

• Execution provider selection is where the surprises are. A model that runs beautifully on one provider can be slower on what looks like a faster one, because not every operation is supported everywhere and the fallbacks are silent.
• Quantised models need testing on real audio, not benchmarks. The accuracy cost is usually small and it is not uniform — it tends to show up on exactly the unusual words you care about.
• Verify the model file before loading it. A model is downloaded at runtime and is executable input. Checksum it against a pinned revision, and never load one you have not verified.