PrismML Ternary Bonsai 2 27B: 5.9 GB, 98.2% Retention, 143 Tokens/Sec

The local inference community has spent three years arguing about the floor. Four bits per weight became the working consensus sometime in 2024 — a compromise between the size of a 7B model you could actually serve on a laptop and the coherence needed for multi-turn tool calling. Below four bits, the numbers would still look acceptable on MMLU, but the models would quietly fall apart under pressure: long chains of thought losing track of their premises, tool-call JSON corrupted at the second argument, code that compiled but did nothing useful.

On September 17, 2026, PrismML shipped Ternary Bonsai 2 27B and the argument effectively ended. Not because one-and-a-half bits magically became uncompromising, but because the gap between ternary and full precision has closed to a point where the remaining delta is smaller than the benchmark noise on most practical tasks. The model scores 83.9 on PrismML’s 20-benchmark suite versus 85.4 for uncompressed Qwen3.8 27B. On instruction following it actually beats the baseline — 82.66 versus 81.25.

This piece audits how that happened, what the numbers mean in practice on real hardware, and where the ceiling of post-training ternary quantization actually sits.

Where PrismML Comes From

PrismML emerged from a Caltech research group and incorporated with backing from Khosla Ventures, Cerberus, and Google, with continuing support from Samsung. Their stated thesis — “concentrating intelligence” — is not a marketing slogan. It is a design constraint. Every model they have shipped, from Bonsai 1.7B to Bonsai 8B to the original Bonsai 27B in July 2026, has been released at ternary or 1-bit precision and targeted hardware that most people actually own.

The original Bonsai 27B, announced July 14, 2026, was the first time a 27B-class multimodal model ran on a phone — specifically the 1-bit 3.9 GB variant on an iPhone 17 Pro. Bonsai 2 27B dropped two months later with a stronger base model (Qwen3.8 versus Qwen3.6), a higher retention score (98.2% versus ~95%), and meaningfully better performance in exactly the task categories where low-bit compression has historically been most brittle: agentic tool use, vision, and long-context retrieval.

The Base Architecture: Qwen3.8 27B Is Not a Standard Transformer

Before addressing the quantization mechanics, it is worth understanding what PrismML was working with. Qwen3.8 27B is not a vanilla decoder-only transformer. Its 64 layers follow a strict 3:1 alternating pattern: three Gated DeltaNet (GDN) layers followed by one full Gated Attention (GA) layer, repeated 16 times across the stack.

This matters for ternary quantization in a non-obvious way. The GDN layers maintain a fixed-size recurrent state rather than growing a KV cache quadratically. At 262K tokens — which Bonsai 2 fully supports, extendable to 1M with YaRN scaling — a full-attention-only model would need roughly 130 GB of KV cache. With 75% of layers running GDN, that requirement drops to something manageable on local silicon. The architecture is designed for memory constraint. The ternary weight format compounds that advantage.

Why Quantization Below Four Bits Historically Fails

Post-training quantization below four bits fails for a structural reason that is worth stating precisely, because PrismML’s solution is a direct response to it.

In a transformer model with more than roughly 6B parameters, certain activation channels develop magnitudes that are orders of magnitude larger than their neighbors. These are not random noise — they are structured, they appear in the same feature dimensions across layers, and they encode semantically significant information. When you apply uniform symmetric quantization with a scaling factor determined by the maximum absolute value in a tensor, a single outlier forces a large quantization step size. Every non-outlier activation then gets mapped to one of a tiny handful of integer levels, losing the fine-grained distinctions the model learned during training.

At INT4, grouping (typically group_size=128) isolates outliers to specific tiles and limits the damage. At ternary precision — where each weight can only take one of three values — grouping alone is not sufficient. The quantization step is so coarse that even moderate activation outliers trigger catastrophic precision collapse: chain-of-thought reasoning loses internal consistency, JSON tool calls develop structural errors, and multi-step arithmetic unravels.

This is why the field largely dismissed sub-4-bit post-training quantization as an academic limit. Microsoft’s BitNet b1.58 demonstrated that ternary precision was viable — but only when the model is pre-trained from scratch with ternary constraints enforced during every gradient update. Applying ternary weights to an existing FP16 model after training was understood to be a different, harder problem.

PrismML’s Bonsai series is a direct refutation of that understanding.

The Hadamard Fix: Rotating the Outliers Away

PrismML’s approach builds on the rotation-based quantization lineage — QuaRot, SpinQuant, and the newer Grouped Sequency-arranged Rotation (GSR) methods — but applies it end-to-end as a post-training transformation targeting ternary precision specifically.

The core insight is that multiplying by an orthogonal matrix does not change what a linear layer computes. For any orthogonal matrix H where HᵀH = I, the output of a linear projection is unchanged:

A Walsh-Hadamard matrix H spreads the energy of any sparse, high-magnitude activation spike across all d hidden dimensions uniformly. Before rotation, 0.2% of channels can carry 99% of the activation magnitude. After rotation, the distribution approaches an isotropic Gaussian — the outlier structure simply does not exist in the rotated basis. Weights in that rotated space can then be projected to {−1, 0, +1} without the catastrophic rounding errors that destroyed earlier sub-4-bit attempts.

The ternary assignment itself uses a blockwise optimal scaling factor α that minimizes the Frobenius norm reconstruction error for each block of 128 weights:

The effective bit depth of 1.76 bits per weight is higher than the theoretical floor of log₂(3) ≈ 1.58 bits precisely because of these FP16 group-wise scale factors. Each 128-weight block requires one 16-bit float for α, which adds 0.125 bits per weight on top of the ternary payload. This is the honest number PrismML reports, not a rounded-down marketing figure.

What Multiplication-Free Matrix Math Actually Means for Hardware

The inner loop of a linear layer is a general matrix multiply (GEMM). In FP16 inference, each element of the output requires multiplying each input activation by its corresponding weight and accumulating. This is expensive: on modern 4nm silicon, an FP16 multiply-accumulate costs approximately 1.1 pJ.

With W ∈ {−1, 0, +1}, the multiply disappears. The dot product degenerates to three cases: add the input (weight = +1), subtract it (weight = −1), or do nothing (weight = 0). A 16-bit integer add costs roughly 0.1 pJ — an 11× reduction in arithmetic energy per operation, before any memory transfer considerations.

The larger gain is on the memory bus. Autoregressive token generation at batch size 1 is memory-bandwidth bound: for every token emitted, the hardware must stream the entire model weight tensor through the memory subsystem into compute registers. The throughput ceiling is therefore set by bytes-per-second, not FLOP/s.

Real Benchmark Numbers — Not the Aggregate

The 98.2% aggregate retention figure is the headline, but the distribution across capability categories tells a more useful story. PrismML’s official whitepaper reports the following scores in thinking mode, compared against the Qwen3.8 27B and Qwen3.6 27B baselines:

The pattern is worth reading carefully. Precision loss concentrates in Knowledge & Reasoning (−2.71) and Vision (−3.05) — tasks that require either very specific factual retrieval from training data or fine-grained spatial feature discrimination in the unquantized vision encoder’s output. Math and Coding lose less than one point. Instruction following gains a point — likely a calibration artifact from the distillation process slightly sharpening the model’s format adherence.

The Agentic & Tool Calling delta of −2.17 deserves specific attention. τ²-bench tests multi-turn dialogue tasks with external tool dependencies; BFCLv3 evaluates function calling against schemas of varying complexity. A 2-point drop in this category on quantized models historically compounds badly in real agentic loops because each tool-call error cascades into downstream subtask failures. PrismML’s whitepaper demonstrates through the Cline (RTX 5090) and computer-use demos that the model handles real coding-agent loops without structural breakdown — but users running 20+ step autonomous workflows will encounter more error recovery events than they would with the full-precision base model.

Throughput That Changes the Hardware Calculus

PrismML’s official throughput figures from the whitepaper: 143 tokens/sec on RTX 5090, 46.8 tokens/sec on M5 Max. The energy figure on RTX 4090 is 0.714 mWh/token — 40% more efficient than an 8B model in full precision on the same hardware.

These numbers matter because of what they imply for the economics of running this model. At 143 tok/s, a coding assistant powered by Bonsai 2 27B on a single consumer GPU can saturate a developer’s reading speed with no noticeable latency gap between prompt submission and the start of code output. The same hardware running Qwen3.8 27B in BF16 would require two RTX 4090s and would generate roughly 40 tok/s — slower, more expensive, and requiring a workstation rather than a desktop.

The Intelligence Density Argument

PrismML frames their compression work around the concept of “intelligence density” — the negative log of a model’s error rate divided by its size in gigabytes. On this metric, Bonsai 2 27B outperforms every other model in the 27B parameter class, including full-precision alternatives that are 9× larger.

This framing is more than marketing. It encodes a genuine engineering constraint: for any fixed memory budget, ternary Bonsai 2 delivers more correct answers per gigabyte than any alternative. A server operator running 24 GB of VRAM gets four simultaneous Bonsai 2 27B instances serving four independent users, versus one instance of the INT4-quantized equivalent serving one user at a time. The throughput economics at inference scale — where compute is purchased per token and VRAM is the binding constraint — shift significantly in favor of ternary.

The energy figure is equally consequential. At 0.714 mWh/token on RTX 4090, Bonsai 2 27B uses 40% less power than running Qwen3.8-8B in full precision. A research team or startup running 24×7 agentic coding workflows on commodity GPUs will see that efficiency compound over months into meaningful cost reductions — and for organizations with sustainability mandates, the energy argument is increasingly a procurement criterion.

Where This Falls Short

PrismML’s 98.2% aggregate figure does not mean 98.2% on every task. The vision tower is kept unquantized, which means the language model receives full-precision image embeddings — but the 3-point drop in the Vision category suggests that the language model’s ability to reason about those embeddings is degraded by the ternary weight projection in adjacent layers. Tasks requiring precise reading of dense charts, exact OCR of printed text at small font sizes, or fine-grained spatial relationship reasoning will show larger gaps than the benchmark score implies.

The Knowledge & Reasoning drop is a more fundamental issue. MMLU-Redux and GPQA Diamond measure the retrieval of specific factual associations stored in the model’s weights. When those weights are compressed to three discrete values, low-frequency factual associations — rare entities, specialized technical vocabulary, less-common historical events — are the first to become ambiguous under the ternary rounding. Users querying the model on niche technical domains will encounter a higher hallucination rate than the headline numbers suggest.

The agentic task gap (−2.17) is the sharpest practical concern for anyone considering Bonsai 2 as a drop-in backend for an autonomous coding agent. Independent testing on Terminal-Bench 2.1 and SWE-bench Verified places retention in that specific regime closer to 75% — significantly below the aggregate. Multi-turn autonomous loops accumulate error faster than single-shot tasks, and a 2-point base-rate reduction in tool-call accuracy translates to materially higher rates of agent derailment at step 10+ of a complex task.

None of this is a reason to dismiss Bonsai 2. For interactive use, document analysis, code generation, and multimodal Q&A on hardware that cannot fit the alternative, it is the right choice. For high-stakes, fully autonomous multi-step agentic pipelines where correctness at every step is load-bearing, the full-precision model or a hybrid routing architecture (local Bonsai 2 for drafts, cloud escalation for critical decisions) is a more appropriate architecture.

What the Pre-Training vs. Post-Training Distinction Still Matters For

BitNet b1.58 and Bonsai 2 are both ternary — but they are different classes of solution. BitNet trains the model natively in the ternary regime from the first gradient update, so the architecture learns to express all of its knowledge within the {−1, 0, +1} weight space. There is no reference FP16 model that has been compressed. The model’s representations are intrinsically discrete.

Bonsai 2 is post-training quantization applied with extreme care. It begins with a frontier-quality FP16 model (Qwen3.8 27B, one of the strongest open-weight 27B models available), and compresses it after the fact using Hadamard rotation plus blockwise optimal projection. The model’s “memory” of its training is preserved — imperfectly, but at 98.2% fidelity — in the ternary representation.

The practical implication: as Qwen, Mistral, Llama, or any other base model architecture improves, PrismML can immediately produce an updated Bonsai generation without retraining from scratch. Bonsai 2 was shipped within three months of Qwen3.8’s release. That iteration speed is only possible because the compression is applied post-training. A natively ternary model requires an entirely new pre-training run — expensive, slow, and always trailing the frontier.

Platform Coverage and License

Bonsai 2 27B runs on NVIDIA GPUs via CUDA and on Apple Mac, iPhone, and iPad via MLX. The 5.9 GB GGUF weights (PTQ1_0 packing) and MLX format weights are available at huggingface.co/collections/prism-ml/bonsai-2. A WebGPU browser demo is live at the Hugging Face Space webml-community/ternary-bonsai-2-webgpu-kernels, requiring no local installation. The model is released under Apache 2.0 with no usage restrictions for commercial deployment.

PrismML also offers a Bonsai Studio iOS app (App Store ID 6767042620) for on-device inference, and accepts enterprise engagements for domain-specific post-training and hardware-targeted kernel optimization. Contact is at contact@prismml.com.

The Structural Shift This Represents

For three years, the local inference community treated four bits as a hard floor. The reasoning was sound: the physical limits of post-training quantization made sub-4-bit compression destructive enough that any theoretical efficiency gain was overwhelmed by the capability loss. Bonsai 2 27B changes the empirical basis of that reasoning.

At 98.2% retention, ternary compression has crossed the threshold from “impressively close” to “operationally equivalent for most use cases.” The 5.9 GB footprint means a 27B-class reasoning model now fits in the GPU budget of hardware that most working engineers already own. The 143 tok/s throughput on current-generation consumer hardware means the model generates faster than developers read — the latency constraint that made smaller, weaker models preferable for interactive use no longer applies.

The question that follows is straightforward: if a 27B model in 5.9 GB retains 98.2% of its intelligence, what does a well-designed 100B or 200B model look like at 1.76 bits? PrismML’s roadmap points at that direction. The compression techniques that made Bonsai 2 possible are architecture-agnostic. They will apply to whatever frontier model emerges next.

The local inference story just got significantly larger.

EyesTech Systems Lab independently audits AI infrastructure releases. Benchmark figures cited in this piece are sourced directly from PrismML’s published whitepaper at github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-2-27b-whitepaper.pdf, the official announcement at prismml.com/news/bonsai-2-27b, and the Hugging Face model card at huggingface.co/collections/prism-ml/bonsai-2. No promotional relationship with PrismML exists.