AI Models for Frigate Enrichments
Frigate uses a handful of AI models — the object detector, plus the “enrichment” features that build on it (semantic search, face recognition, license plates, bird and audio detection, and more). This page lists every model, grouped by feature, and where each runs — the Neural Engine (ANE), the GPU, or the CPU — on Fregata and, for comparison, in Docker.
The short version: object detection — the one thing that runs on every single frame — gets the Neural Engine to itself. For everything else, Fregata uses the GPU for the models Frigate accelerates that Apple’s CoreML can run, and the CPU for the rest. A few models that upstream Frigate accelerates on a GPU use dynamic tensor shapes the Mac’s CoreML GPU path can’t take yet, so they run on the CPU for now — none of them a hot path (see “A few models run on the CPU for now”).
You don’t configure any of this — Fregata picks the right place automatically.
The only related knob is model_size on the features that offer one (for
example face recognition’s small vs large).
Where each model runs
Section titled “Where each model runs”The full picture — every model, the feature it powers, and where it runs across the three ways you might run Frigate:
| Feature | Model (variant) | macOS Docker¹ | Linux / Windows Docker² | Fregata (macOS) |
|---|---|---|---|---|
| Object detection³ | YOLOv9-t (the detector) | CPU | GPU / NPU | Neural Engine |
| Semantic search | jina-clip — vision (images) | CPU | GPU | GPU |
| jina-clip — text (query) | CPU | GPU | CPU † | |
| Face recognition | FaceDet — finding faces | CPU | GPU | GPU |
FaceNet — small embedding (default) |
CPU | CPU | CPU | |
ArcFace — large embedding |
CPU | GPU | GPU | |
| License plate | YOLOv9-256 — plate detection | CPU | GPU | CPU † |
| PaddleOCR — orientation | CPU | GPU | GPU | |
| PaddleOCR — recognition_v4 (OCR) | CPU | GPU | CPU † | |
| Bird classification | MobileNetV2 (iNaturalist) | CPU | CPU | CPU |
| Audio detection | YAMNet | CPU | CPU | CPU |
| Audio transcription | sherpa-onnx — small (default) |
CPU | CPU | CPU |
Whisper — large |
CPU | GPU | CPU † | |
| Custom classification | MobileNetV2 — training | CPU | GPU | CPU |
trained model (.tflite) — inference |
CPU | CPU | CPU |
¹ Docker Desktop on a Mac runs everything on the CPU. Docker’s Linux VM can’t reach the Mac’s Neural Engine or GPU, so nothing is accelerated — the main reason Fregata exists.
² With a supported GPU or accelerator (NVIDIA/TensorRT, Intel/OpenVINO, a Coral, etc.). Without one, Linux/Windows Docker also runs everything on the CPU — same as the macOS Docker column.
³ The detector model differs by platform: Fregata bundles YOLOv9-t and runs it on the ANE; Docker images default to a CPU SSD/MobileDet model (or an accelerator-specific one).
† GPU on a GPU-equipped Frigate, but CPU on Fregata today. These four use dynamic tensor shapes Apple’s CoreML GPU path can’t take yet. None is a hot path (they run per search query, per plate, and per spoken phrase — not per frame), so the impact is small; GPU support is a planned optimization in a future release. See below.
The takeaway: Fregata is the only way to optimally hardware-accelerate Frigate on a Mac. macOS Docker is CPU-only across the board; Fregata puts object detection on the Neural Engine and most enrichment models on the GPU — matching a GPU-equipped Linux/Windows box except for four dynamic-shape models that fall back to the CPU for now.
Why object detection owns the Neural Engine
Section titled “Why object detection owns the Neural Engine”The ANE is Apple’s dedicated, extremely power-efficient inference chip — but it works best running one model at a time. Object detection runs multiple times on every frame with motion from every camera, so it’s by far the most demanding and latency-sensitive job Fregata does. Pinning detection to the ANE — and routing everything else to the GPU or CPU — means a burst of face recognition or search indexing can never stall the detector. It’s also why Fregata needs Apple Silicon: Intel Macs have no Neural Engine, and the CPU fallback (~50 ms/frame versus ~2 ms on the ANE) is too slow to be a real NVR.
Why some models use the GPU and others the CPU
Section titled “Why some models use the GPU and others the CPU”Fregata is a faithful port of Frigate and keeps the same accuracy. Frigate already decides, per model, where it runs; Fregata honors those choices on Apple Silicon wherever the Mac’s GPU path can:
- GPU (Apple’s CoreML, via ONNX Runtime) — the models Frigate accelerates
that have fixed input shapes CoreML can compile: image search (vision),
largeface recognition, and license-plate orientation. - CPU, same as upstream — the models Frigate runs on the CPU everywhere:
bird, audio detection,
smallface recognition, and the small speech-to-text model. These are small / TFLite models with no GPU path upstream either, so they run on the CPU like in all version of Frigate.
A few models run on the CPU for now
Section titled “A few models run on the CPU for now”Four models that upstream Frigate accelerates on a GPU use dynamic tensor
shapes the Mac’s CoreML GPU path can’t take yet, so today they run on the CPU:
the text side of semantic search, license-plate detection and OCR, and
the large (Whisper) transcription model. On a GPU-equipped Frigate those steps use the
GPU, so they’re slower on Fregata for now — but none is a hot path (they run
per search query, per plate, and per spoken phrase, not per frame), so the
real-world impact is small. GPU support for them is a planned optimization in a future release.
When models download
Section titled “When models download”Object detection and audio detection ship inside the app, so they work offline the moment you install Fregata. The other models download automatically the first time you turn on the feature that needs them, into Fregata’s model cache — a one-time download per model. Custom-model training additionally fetches TensorFlow on first use (a larger, one-time download); the model you train then runs on the CPU like any other.