Running Gemma 4 on an iPhone with Apple MLX: what we learned

Two years ago, running a capable multimodal language model on a phone was a research demo. Today it's our production architecture: the EchoSense Companion runs Gemma 4 on the iPhone through Apple MLX, handling text, voice, and images with zero server inference.

The stack

Apple MLX is Apple's array framework for Apple Silicon, exploiting unified memory so weights don't shuffle between CPU and GPU. We run Gemma 4 quantized to 4-bit through mlx-vlm for multimodal support, with PLE-safe handling, inside a Flutter app via platform channels. Speech synthesis is Kokoro TTS, also local.

The pipeline around the model matters more than the model

• ContextBuilder — assembles what Echo knows: memories, stress tier, time of day, recent conversation.
• 7-layer prompt — persona, safety rules, memory, vitals context, tools, history, and the user's message, in a fixed order that survives long chats.
• Sanitizer — strips artifacts and malformed spans from raw model output before anyone sees it.
• Tool parser & validator — extracts tool calls (like SOS or memory writes) and validates them against schemas before execution. An invalid call is dropped, never guessed at.

Hard-won lessons

• Quantization is a UX decision — 4-bit made the difference between a model that fits alongside a real app and one that doesn't.
• Sanitize everything — small local models emit artifacts at the worst times; the sanitizer is not optional plumbing.
• Validate tool calls like user input — a hallucinated tool call must fail closed, especially when one of the tools dispatches SOS.
• Thermals are a feature constraint — sustained generation heats a phone; short, grounded replies aren't just better UX, they're better thermals.