I've been tracking on-device model compression for two years now, and I've watched a lot of press releases claim "runs on a phone" while delivering models that technically fit in memory but take thirty seconds to generate a single sentence. So when PrismML dropped Bonsai 27B today — a 27-billion-parameter multimodal model that runs on an iPhone 17 Pro — my first reaction was skepticism.

But I've spent the afternoon going through the numbers, the architecture, the benchmarks, and the whitepaper. And here's the thing: the claims hold up. This isn't vaporware. It's not a clever demo with cherry-picked metrics. Bonsai 27B is a genuine breakthrough in what I'm going to call intelligence density — and it changes the conversation about where AI workloads belong.

The Numbers That Matter

Let's start with the raw specs, because they're where this story lives or dies.

Bonsai 27B is based on Qwen3.6 27B, Alibaba's open-source multimodal model. It ships in two variants:

1-bit Binary Bonsai 27B — weights constrained to 1 with FP16 group-wise scaling. True bit-width: 1.125 bpw. Total footprint: 3.9 GB.
Ternary Bonsai 27B — weights in 1. True bit-width: 1.71 bpw. Total footprint: 5.9 GB.

For context, a conventional FP16 build of the same model weighs 54 GB. The most aggressive "2-bit" quantization (Qwen3.6 IQ2_XXS) still comes in at 9.4 GB with a 2.8 bpw true bit-width. PrismML is operating in a different regime entirely — they're not quantizing to 4 bits or 2 bits. They're going to one bit, end-to-end, with no higher-precision escape hatches in the embeddings, attention, MLPs, or LM head.

The 3.9 GB variant fits inside the ~6 GB that a 12 GB iPhone makes available to an app, with room left over for KV cache and activations. The Ternary variant (5.9 GB) targets laptops and the higher-end iPad Pro.

On actual phones, the 1-bit model runs at roughly 11 tok/s on an iPhone 17 Pro Max via MLX Swift. That's not fast enough for real-time conversation, but it's fast enough for summarization, classification, structured extraction, and single-turn reasoning tasks. With speculative decoding (more on that below), effective throughput improves by about 1.37x.

On desktop hardware, the numbers are genuinely impressive: 163 tok/s on an RTX 5090 (1-bit), 87 tok/s on an Apple M5 Max. The M5 Pro laptop hits 44 tok/s at just 0.275 mWh/token — an order of magnitude more energy-efficient than datacenter GPUs.

How They Did It

The secret isn't one thing. It's a stack of decisions that compound.

Architecture choice matters. PrismML didn't just take Qwen3.6 and run it through a quantization algorithm. The base model uses a hybrid attention architecture — roughly 75% linear attention, 25% full attention. Linear attention scales linearly with sequence length instead of quadratically, which reduces the KV cache burden dramatically. This is the same architectural insight that drove the Mamba and RWKV lines of research, but here it's hybridized with full attention to retain the quality where it matters most.

End-to-end low-bit representation. Most quantization projects apply aggressive quantization only to certain layers and leave others at higher precision. Bonsai applies its 1 or 1 representation to the entire network, with FP16 scaling factors amortized over groups of 128 weights. The bit-width is honest — what they say is what they mean, which is refreshing in an era of "4-bit" models that average 5.2 true bpw.

Custom low-bit kernels. The model runs on Apple Silicon via MLX and on NVIDIA GPUs via CUDA, but the kernels aren't off-the-shelf. PrismML built custom inference kernels for their ternary and binary formats. These aren't trivial — standard GPU hardware is optimized for FP16 and INT8, not for bit-level representations. The fact that they're getting 163 tok/s on a 5090 means the kernels are doing real work.

Speculative decoding. The DSpark drafter ships as a 4-bit draft model (~1.79 GB) and is loaded optionally. It's lossless — verification preserves the target distribution exactly — and adds roughly 37% end-to-end throughput on GPU hardware.

The Benchmarks: Where the Rubber Meets the Road

This is where compression projects usually fall apart. You can shrink a model to fit, but if it can't reason, you've built a party trick.

Bonsai 27B was evaluated across 15 benchmarks in thinking mode. The overall numbers:

Variant	Footprint	Score	vs FP16
FP16 baseline	54 GB	85.07	100%
Ternary Bonsai	5.9 GB	80.49	94.6%
1-bit Bonsai	3.9 GB	76.11	89.5%

The aggregate tells a story, but the disaggregated numbers are more interesting. Math and coding survive the compression surprisingly well. MATH-500 and AIME25/26 scores drop only a few points from the FP16 baseline. The Ternary variant retains 86.0% on coding benchmarks (HumanEval+, MBPP+, LiveCodeBench) versus 88.7% for FP16. GSM8K barely budges.

What degrades? Factual recall and vision. MMLU-Redux drops from 83.15 to 73.39 on the 1-bit variant. MMMU Pro (vision) goes from 72.61 to 59.57. This matches what PrismML's CEO told CNBC directly: factual recall is the first thing to go when you compress this aggressively.

But here's the critical insight: the degradation pattern is the opposite of conventional low-bit builds. Standard 2-bit quantization (IQ2_XXS on the same base model) collapses on sustained reasoning tasks — scoring 57.5 on AIME26 and 56.4 on LiveCodeBench — while still looking decent on MMLU-Redux (88.93). The aggregate IQ2_XXS score is 72.73, but the distribution is bimodal. Bonsai 27B scores 87+ on AIME26 at a third of IQ2_XXS's footprint.

This matters because it's the reasoning backbone that makes agents work. If a compressed model can still reason through multi-step tool calls but forgets the exact date of the French Revolution, that's a trade-off worth making for on-device deployment. The inverse — remembering facts but unable to reason — would be useless.

Intelligence Density: The New Metric

PrismML introduced a new metric in their whitepaper that I think is going to stick: intelligence density, defined as D = -log2(1 - score/100) / size_GB. It measures how much capability you get per gigabyte.

The numbers are striking:

1-bit Bonsai 27B: 0.530 / GB
Ternary Bonsai 27B: 0.400 / GB
Qwen3.6 IQ2_XXS: 0.199 / GB
Qwen3.6 Q4_K_XL: 0.155 / GB
Qwen3.6 FP16: 0.051 / GB

Intelligence density is the right framing because it cuts through the abstraction of parameter counts. Parameters tell you how big the model is. Intelligence density tells you how much model fits in your device's memory budget. And on that axis, Bonsai 27B is 2.7x denser than the best existing low-bit alternative and 10x denser than the FP16 baseline.

This is the metric that Apple's chip designers and product managers will care about. If they can get 0.53 intelligence density per GB on existing iPhone hardware, they don't need to redesign the A-series chip — they just need to license or replicate the compression pipeline.

Why This Matters for Agent Developers

The article from Cryptobriefing posed the question: does on-device inference at this scale shrink the addressable market for cloud AI? The answer is nuanced, but for agent developers specifically, I think this is a sea change.

Every agent loop today follows the same pattern: user request → cloud API → response → next tool call → cloud API. Every step:

Incurs latency (hundreds of ms at minimum)
Incurs cost (per-token pricing adds up fast on 10-step+ loops)
Transmits data (prompts, context, sometimes screen contents)

Bonsai 27B enables a hybrid deployment model: the agent lives on-device for 90% of tasks — privacy-sensitive queries, local context summarization, tool call routing — and escalates only the hardest reasoning steps to a cloud model. Marginal cost of local steps is zero. Data never leaves the device for common operations.

This is the architecture that Apple will almost certainly pursue for iOS 27's Siri overhaul. The delayed Siri rewrite that just landed in beta this week needs on-device models that can handle agentic workloads, not just classification and summarization. Bonsai 27B, or something like it, is what makes that vision technically feasible.

The Caveats

I've been bullish so far, so let me flag what gives me pause.

Power consumption. 11 tok/s on an iPhone 17 Pro Max is usable throughput, but at what battery cost? The whitepaper cites 0.275 mWh/token on M5 Pro, but phone thermal envelopes are much tighter. Running a 27B model continuously for agent tasks would drain a phone battery in under an hour. This model is for burst usage — short prompts, single-response queries — not sustained chat sessions.

Independent verification. The 15-benchmark evaluation was run by PrismML themselves. The press release includes the methodology — EvalScope + vLLM on H100 — but independent replication hasn't happened yet. The community benchmarks on Hugging Face (a 14-question general knowledge test) are too small to draw conclusions from.

The Apple deal is "very early." CNBC confirmed that Apple is evaluating PrismML's technology, but the CEO characterized talks as "very early." This is not a done deal. Apple has a long history of evaluating startups, using their IP for inspiration, and building in-house.

Factual recall degrades noticeably. If your use case depends on retrieving specific facts from the model's training data — rather than reasoning about context you provide — Bonsai 27B will perform worse than a cloud model. This is an explicit design trade-off, but it limits the model's applicability for certain workflows.

The Bottom Line

Bonsai 27B is the real thing. It's not a complete replacement for cloud inference — the gap between 76.1% and 85.1% overall benchmark performance is real, and factual recall degradation is a genuine limitation. But as a complementary on-device inference tier for agentic workloads, it changes the calculus entirely.

The most important line in the whitepaper isn't about benchmarks or throughput. It's this:

"Intelligence density will be one of the defining axes of the next stage of AI progress. Raw capability determines what a model can do; density determines where it can do it."

That's the right framing. And by that measure, Bonsai 27B is the largest step yet in putting real intelligence where users actually live.

The models are available today under Apache 2.0 on Hugging Face, along with a developer preview API. I'd recommend every agent developer running on Apple Silicon download the 1-bit MLX variant and test it against their own workloads. The gap between what this can do on-device and what you're currently routing to the cloud is going to surprise you.

Sources:

Bonsai 27B — the first 27B-parameter model that actually runs on your phone