Bonsai 2 27B
A 27B AI Model That Fits in 5.9 GB

§ Abstract & How to Read

What this paper claims, what it does not, and how the evidence is organized.

On September 17, 2026, the Pasadena- and San Francisco-based company Prism ML released Ternary Bonsai 2 27B, a compressed build of the Qwen3.8-27B reasoning model in which essentially every language-model weight is stored as one of three values — {−1, 0, +1} — with a single FP16 scale shared by each group of 128 weights. The result occupies 5.93 GB for the language model as shipped, against 53.8 GB for the FP16 baseline, and retains 98.2% of the base model’s average score across the company’s 20-benchmark thinking-mode suite — 83.9 versus 85.4. That average also slightly exceeds the 83.6 posted by the previous-generation Qwen3.6-27B in full precision, at roughly a ninth of either model’s size. The company’s framing is direct: at this level of retention, compression becomes a deployment unlock rather than a quality compromise.

This paper is a comprehensive technical deep dive on that release and the six-month arc that produced it — five public launches from March 31 to September 17, 2026, spanning 1-bit and ternary language models at 1.7B through 27B scale, an image-generation model, and the 27B flagship that first ran a model of its class on a phone. It reconstructs, from primary sources only, what the Bonsai representation actually is (Chapter 8), how the rotated Hadamard basis works and why it is folded offline into the stored weights (Chapter 9), what the custom hybrid-attention kernels do and why stock runtimes cannot execute these files (Chapter 10), how the benchmarks were run and what the full per-benchmark tables say (Chapter 11), and how the model performs across twelve hardware platforms from an L4 to an M5 Max laptop (Chapter 12).

It is equally a paper about evidence quality. Bonsai 2 27B arrives with an unusually detailed 14-page technical report, but also with two distinct benchmark suites (a 20-benchmark whitepaper suite and a 14-benchmark Hugging Face companion), two generations of throughput measurements as the stack matured, and several numbers that vary between documents by small, explainable amounts. Where sources disagree, this paper reports each figure with its attribution rather than silently picking one. The accompanying verification pass re-checked every quantitative claim in this document against the whitepaper, both model cards, the announcement posts, and third-party measurements; the fact-check results are summarized in Chapter 16.

1 The Memory Wall

Why a 27B model needs 54 GB, why decode is bandwidth-bound, and why the conventional quantization ladder breaks below four bits.

The defining constraint of local AI deployment is not compute — it is memory. A 27-billion-parameter model stored in the industry-standard 16-bit floating-point format occupies roughly 54 GB before a single token of context is loaded. That number alone disqualifies the model from every consumer device on the market: an everyday laptop ships with 16–32 GB of unified memory, a phone offers far less usable headroom, and even a 24 GB single-GPU workstation cannot hold the weights. The problem compounds at inference time, because generating text is dominated not by arithmetic but by movement: at batch size one, every decoding step streams the entire set of resident weights through the memory system. The arithmetic each weight performs is trivial; the cost of fetching it is everything.

This is why Prism ML’s whitepaper describes batch-1 decoding as “memory-bandwidth-bound” and why the entire Bonsai program is built around reducing bytes per weight rather than reducing the parameter count. The company’s own measurement makes the point concretely: on an Apple M5 Pro laptop, Bonsai 2 27B sustains 27.7 tokens per second while streaming approximately 201 GB/s of weight traffic — the decode path is, to first order, a bandwidth benchmark. Cut the bits per weight by 9× and, on a bandwidth-limited device, tokens per second rise accordingly; the same 27B model that needed a datacenter-class memory system becomes interactive on hardware people already own.

The conventional ladder — and where it breaks

The industry’s standard answer to the memory problem is post-training quantization (PTQ): take a trained FP16 model and round its weights to a coarser grid. The ladder is well understood. Eight-bit builds cost little quality. Four-bit builds — the current default for local deployment — lose roughly a point of aggregate benchmark score at a quarter of the footprint; Prism’s own data shows Qwen3.8-27B’s Unsloth UD-Q4_K_XL build scoring 85.18 against the FP16 baseline’s 86.32 on their 14-benchmark suite, 98.7% retention at 17.6 GB. Below four bits, the ladder historically collapses: the widely-used IQ2_XXS “2-bit” build of the same model scores 72.59 on that suite — 84.1% retention — and, more damaging, fails selectively, a pattern examined in detail in Chapter 11.

Two structural facts about conventional low-bit builds matter for everything that follows. First, their advertised labels understate their true bit-width. A GGUF “2-bit” build is a mixture: attention projections, embeddings, and sensitive tensors are quietly kept at higher precision, so the build Prism cites — at 9.4 GB for a 27B model — is really 2.8 bits per weight on average, not 2. Second, the mixture is where the quality hides: the high-precision escape hatches carry much of the reasoning load, so a “2-bit” label describes the cheapest tensors, not the model. Bonsai’s central design commitment is the opposite: the low-bit representation runs end to end across the language model — embeddings, attention projections, MLP projections, and the LM head — with the advertised bit-width matching the shipped artifact, and only 26.2M parameters (0.0976% of the language model, the recurrent-state path and normalization weights that linear attention requires) held above it, explicitly counted into the 1.72-bits-per-weight figure.

Why 1.58 bits is the natural floor

Ternary storage — weights restricted to {−1, 0, +1} — is not an arbitrary cruelty. A ternary value carries log₂(3) ≈ 1.585 bits of information, and adding a single FP16 scale per 128-weight group costs 16/128 = 0.125 bits per weight, for an information-theoretic floor of about 1.71 bits per weight. That is a 9.3× reduction against FP16 — large enough to move a 27B model across device-class boundaries, and small enough that the arithmetic (multiply-by-scale, accumulate, with zero contributions costing nothing) stays trivially cheap. Binary weights {−1, +1} go further still — 1 bit plus scale overhead lands at 1.125 bits per weight, a 14.2× reduction — which is exactly the operating point Prism used to put a 27B model on an iPhone. The question that defined the field for a decade was never whether ternary storage was compact; it was whether a network’s behavior could survive it. Chapter 2 traces how the answer evolved.

2 The Ternary Lineage

From Trained Ternary Quantization (2016) through BitNet and b1.58 to 2026’s post-training ternarization — and the two schools that emerged.

Ternary weights are older than the LLM era. Trained Ternary Quantization (Zhu et al., 2016) showed, in the pre-transformer world, that neural-network weights could be constrained to three values during training with tolerable accuracy loss — work that has since accumulated roughly 1,550 citations and established the basic recipe: pick per-layer scales, discretize, and retrain so the network adapts to its own coarseness. The idea then went quiet for half a decade, because dense FP16 training was cheap enough that nobody needed the savings.

The LLM-scale revival came from Microsoft Research. BitNet (October 2023) proposed training transformers with binary weights from scratch; its follow-up, “The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits” (Ma et al., February 2024, arXiv 2402.17764), argued that ternary {−1, 0, +1} weights, trained natively, match FP16 baselines at matched scale — Wikipedia’s summary of the line of work puts it as “comparable in performance to the 16-bit Llama 2” — and defined a new scaling-law framing for what the paper’s title called the 1-bit era. The proof that the recipe scaled to a genuinely useful model arrived in April 2025 with BitNet b1.58 2B4T, Microsoft’s 2.4-billion-parameter model trained on 4 trillion tokens and released as “the first open-source, native 1-bit LLM,” with 1.58-bit weights and 8-bit activations, plus its own inference framework (bitnet.cpp).

Two schools: train ternary, or compress after

BitNet’s school is quantization-aware training (QAT): the model is born ternary, spending its full training budget inside the low-bit regime. Its strength is quality; its cost is that every ternary model must be trained from scratch, so the method only reaches the base models someone is willing to spend a pretraining run on. The whitepaper’s reference table makes the contrast explicit in 2026 terms: among Prism’s comparison points, the QAT entry is Gemma-4-31B QAT at 6.0 true bits per weight, 23.3 GB, retaining 98.0% — excellent retention, at a footprint four times ternary’s and a bit-width four times higher, because that QAT program chose 4-bit, not 1.58-bit, as its target.

The second school is post-training ternarization: take an existing strong model — with its vision tower, its tool-calling tuning, its agentic post-training already paid for — and convert its weights to ternary without retraining. This is the harder scientific problem, and the one Prism ML chose: the company’s first-generation 1-bit Bonsai 8B card states plainly that the methodology is applied to existing bases rather than baked into training, and third-party coverage from the launch window (regolo.ai among others) describes Bonsai explicitly as “a compact representation of a much larger dense model,” not a small model trained from scratch. The 2026 research literature crowded into exactly this gap — academic work on cost-efficient ternary quantization (CAT-Q, June 2026) and on training-free ternary optimization via asymmetric quantizers refined iteratively (ATQ) — which is the context in which Prism’s claims of 90–98% retention from post-training conversion should be read: the company is claiming results at the aggressive end of what the field believed possible.

Why a 2026 base model changes the calculus

One more piece of context is needed before the Bonsai story makes sense: the base model itself. Qwen3.8-27B is a hybrid-attention architecture — 75% of its layers use linear attention (Gated DeltaNet) rather than full attention — a design inherited from the Qwen3.5 generation that dramatically shrinks the KV cache and makes a 262K-token context practical on-device. Chapter 8 details the layout. For compression, the hybrid backbone is a gift twice over: its recurrent-state path is small enough to keep in full precision at negligible cost (0.0976% of parameters), and its cache grows on only 16 of 64 layers, so long-context deployment is not memory-starved. The Bonsai program did not compress a 2023-style transformer; it compressed an architecture already engineered for efficiency, which is part of why the end-to-end numbers look the way they do.

3 Prism ML, the Company

A Caltech spinoff, an information-theory pedigree, and a thesis: intelligence per bit over parameter count.

Prism ML incorporated as Prism ML, Inc. and, per its own July 2026 post, “emerged from a team of Caltech researchers and was founded with support from Khosla Ventures, Cerberus, and Google, with continuing support from Samsung.” The team page names Babak Hassibi — a Caltech computer scientist and mathematician, and the company’s co-founder and CEO — alongside co-founders Sahin Lale and Omead Pooladzandi (co-heads of research) and Reza Sadri (VP Strategy), with Shayan Ilbagian as CFO, Karim Mattar as VP Engineering, and Tushar Bansal as Director of Product. The advisory board is short and heavyweight: Ion Stoica — the Berkeley professor behind much of modern serving infrastructure, from vLLM’s PagedAttention lineage (which the Bonsai whitepaper itself cites as reference [5]) to Anyscale — plus Buno Pati and Julie Schoenfeld. Press coverage from the March 31, 2026 stealth emergence consistently framed the company as a Caltech research bet: “founded by Caltech mathematicians including Babak Hassibi,” in one April 2026 write-up’s words.

The company’s framing of its own mission is unusually precise for an AI startup, and it matters for reading their engineering choices. The homepage is “Concentrating intelligence”, with the two-line problem statement: “Large models can’t fit on smartphones. Datacenters can’t sustain them. Prism ML is building ultra dense intelligence to solve both.” The About page casts the program as “fundamental research problems around the decades-old quandary: can we massively multiply intelligence in models without increasing their size or complexity?” And the CEO’s launch-week quotation, given across the stealth-emergence press cycle, states the bet directly: “We spent years developing the mathematical theory required to compress a neural network without losing its [reasoning ability].”

That orientation — compression as a mathematical problem in the information-theory tradition, rather than a systems-engineering problem — shows up in the artifacts. Prism publishes formal technical reports with reference lists that reach back to 1976 (the fast Walsh–Hadamard transform paper of Fino and Algazi is reference [29] of the Bonsai 2 whitepaper). It invented and repeatedly uses a quantitative framing device, intelligence density (Chapter 15), defined with a logarithmic error term straight out of the channel-capacity playbook. And it ships its claims with more measurement disclosure than the local-model scene is used to: idle-power floors stated rather than silently subtracted, run-to-run variance quoted (up to ~11% on prefill), and cross-platform energy comparability explicitly refused where instrumentation encloses different components.

The business model around the open weights is still forming. The models are Apache 2.0; the company operates a free, limited-time developer-preview API announced with the July release; a consumer iOS app (Bonsai Studio) is on the App Store; and by September 18, 2026, an inference provider called Darkbloom was serving the model through OpenRouter at $0.075 per million input tokens. The careers page’s open roles — kernel optimization, post-training platform, edge & consumer AI, large-scale systems — read as a roadmap in themselves: the company is hiring for exactly the surfaces where ternary deployment is hardest.

4 1-bit Bonsai (March 2026)

The stealth launch: binary 8B/4B/1.7B models, “commercially viable 1-bit LLMs,” and the first intelligence-density numbers.

Prism ML emerged from stealth on March 31, 2026 with 1-bit Bonsai, a family of binary-weight language models the announcement called “the first commercially viable 1-bit LLMs.” The release comprised three sizes — 8B, 4B, and 1.7B parameters — each storing every language-model weight as a sign bit: {−1, +1}, one FP16 scale per group of 128 weights, for an effective 1.125 bits per weight. The 8B flagship’s deployed footprint was 1.15 GB (1.16 GB in the documentation’s lineup table) against roughly 16 GB for the same class in FP16 — the 14× reduction that let the model run on phones outright. The Hugging Face card for the 8B lists its architecture as qwen3 with a packed Bonsai-8B-unpacked base — the family was built on the Qwen3 line of open models, a fact consistent with everything Prism later compressed.

The proof point was a benchmark table that put 1-bit Bonsai 8B at a 70.5 average across its reported suite — competitive with full-precision 8B-class instruct models at one-fourteenth the size. Among the peers listed in the Ternary Bonsai comparison a fortnight later, the 1-bit model’s 70.5 sat above LFM2 8B (69.6), Llama 3.1 8B (67.1), GLM 4 9B (65.7), Hermes 3 8B (65.4), and a tail of smaller models, and behind only Qwen3 8B (79.3) among same-class full-precision peers. Per-benchmark, the card reports GSM8K at 88.0, HumanEval+ at 73.8, IFEval at 79.8, and BFCLv3 at 65.7 — numbers that made the “commercially viable” claim arguable rather than absurd for chat and summarization workloads, while leaving agentic tool use visibly weaker, a pattern that would define the family’s reviews.

The launch also introduced the metric that Prism has used to frame every release since: intelligence density, benchmark score per deployed gigabyte under a logarithmic error transform (formally, D = −log₂(1 − score/100) / size_GB). By that measure, 1-bit Bonsai 8B posted 1.062 per GB against 0.098 for Qwen3 8B — “10.8× higher intelligence density than full-precision Qwen 3 8B” in the card’s words. Whether one accepts the metric’s framing or not (Chapter 15 examines it), it supplied the company’s public vocabulary: every release since is positioned as a leftward shift of the capability-per-gigabyte frontier.

5 Ternary Bonsai (April 2026)

1.58 bits as the quality tier: the 8B model that trailed only Qwen3 8B, at 1.75 GB.

Two weeks later, on April 16, 2026, Prism released Ternary Bonsai — the same three sizes in ternary {−1, 0, +1} weights with group-128 FP16 scales, “a modest increase in size for a meaningful gain in performance.” The announcement positioned the two families as deliberate operating points rather than a replacement: 1-bit where footprint is everything, ternary where quality is. The trade was quantified cleanly: Ternary Bonsai 8B costs 600 MB more than its 1-bit sibling (1.75 GB versus 1.15 GB) and scores 5 points higher (75.5 versus 70.5 average) — a exchange rate of roughly 0.12 GB per benchmark point that, at small scale, most observers found compelling.

Ternary Bonsai 8B versus same-class peers, as published in Prism’s April 16, 2026 announcement (thinking-mode evaluation). The ternary 8B trails only Qwen3 8B — at 1/9th of its footprint — and beats every other listed full-precision 8B-class model.

The announcement’s throughput claims established the mobile story early: 82 tokens/second on an Apple M4 Pro for the ternary 8B (“roughly 5× faster than a 16-bit 8B model”) and 27 tokens/second on an iPhone 17 Pro Max, with energy consumption of 0.105 mWh/token on the M4 Pro and 0.132 mWh/token on the phone — “roughly 3–4× better energy efficiency” than full-precision counterparts. Platform coverage at this stage was Apple-first: the models ran natively via MLX on Mac, iPhone, and iPad, with weights on Hugging Face under Apache 2.0. The Hacker News thread (April 18) drew a modest 225 points, but the more important audience was elsewhere: by June, the 1-bit 8B’s GGUF pack was averaging 123,876 downloads per month on Hugging Face, with 14 community Spaces built on top of it.

6 Bonsai Image 4B (May 2026)

The same recipe applied to diffusion: 1-bit and ternary image generation on FLUX.2 Klein 4B.

The third release, on May 26, 2026, extended the compression thesis from language to images: Bonsai Image 4B, a pair of 1-bit and ternary diffusion models built on FLUX.2 Klein 4B that run image generation on laptops and phones. The numbers Prism published were, by the family’s standards, conservative-sounding but striking in effect: packs up to 8.3× smaller than the original, generating a 512×512 image on an iPhone in about 9.4 seconds. The documentation’s lineup table lists the binary variant at 0.93 GB and the ternary at 1.21 GB — image generation in a footprint comparable to the 8B language model’s ternary pack.

For the purposes of this paper, Bonsai Image 4B matters less for its own numbers than for what it demonstrated about the methodology’s reach. The compression pipeline was claimed to be architecture-agnostic — a transformer language model, a hybrid-attention reasoning model, and a diffusion transformer all passed through it — which lent credibility to the central bet that the technique was a general property of neural-network weights rather than a trick specific to one model family. The release also drew the family’s second-largest Hacker News audience (464 points, 201 comments, May 31), and the App Store presence that commenters noticed around Prism’s demos dates from this era: an iOS image tool built on the model, followed by the Bonsai Studio app.

7 Bonsai 27B (July 2026)

The first 27B-class model on a phone: Qwen3.6-27B base, two operating points, DSpark speculative decoding, and the phone-gate arithmetic.

On July 14, 2026, Prism announced Bonsai 27B, “the new multimodal flagship of the Bonsai family and the first model of its capability class to run on a phone.” The base was Qwen3.6-27B — the strongest 27B open-weight reasoning model of its generation, itself a hybrid-attention architecture — and the release shipped two deliberate operating points of the same compressed model:

• Ternary Bonsai 27B — ternary {−1, 0, +1} weights with FP16 group-wise scaling at a true 1.71 bits per weight, occupying 5.9 GB. The quality-oriented variant: “it runs on an everyday laptop with the full reasoning, tool-calling, and agentic capability.”
• 1-bit Bonsai 27B — binary {−1, +1} weights with the same scaling at 1.125 bits per weight, occupying 3.9 GB. The footprint-oriented variant, sized to fit “within the memory budget of an iPhone 17 Pro” — the first 27B-class model to do so.

Both variants kept the end-to-end discipline that defines the program — “the low-bit representation runs end to end across the language network, embeddings, attention, MLPs, and the LM head, with no higher-precision escape hatches” — and both shipped with the vision tower in a compact 4-bit form (HQQ inside a Q8_0 container, ~0.63 GB, loaded only when an image arrives), a full 262K-token context, and speculative decoding support. Everything was Apache 2.0, with a free limited-time developer-preview API alongside the weights.

The retention claim: 95% and 90%

Across a 15-benchmark thinking-mode suite, the announcement reported that Ternary Bonsai 27B retained 95% of the full-precision Qwen3.6-27B baseline, and 1-bit Bonsai 27B retained 90%. The category-level table:

Bonsai 27B first-generation results as published July 14, 2026 (thinking mode). The model cards’ own tables give the precise averages: 85.07 FP16 baseline, 80.49 ternary (94.6%), 76.11 one-bit (89.5%).

The model cards expanded this into a cross-family comparison that remains one of the most informative tables in the entire Bonsai corpus, because it includes a second base model — Gemma-4-31B — with both its QAT and conventional low-bit builds, demonstrating that the sub-4-bit collapse is a property of the methods, not of one base:

First-generation cross-family comparison (15-benchmark suite, thinking mode), from the Bonsai-27B-gguf model card. Both families’ conventional “2-bit” builds collapse into the low 70s while occupying 2.4× the ternary model’s footprint; the Bonsai operating points sit well above them.

Throughput, memory, and the phone

The launch’s throughput figures: 163 tokens/second in 1-bit and 134 in ternary on an RTX 5090, and on an M5 Max laptop, 87 in 1-bit and 58 in ternary. The 1-bit GGUF card’s llama-bench table adds the mid-tier Apple hardware: 66.4 tok/s on M5 Max, 44.2 on M5 Pro, 26.0 on M4 Pro for the binary pack — and a revealing datacenter row: on an H100 at batch 1, binary and ternary converge (104.8 vs 98 tok/s) “despite their ~1.9× difference in bytes per step,” because a datacenter GPU at batch one is limited by kernel-launch and synchronization latency rather than weight bandwidth. On the phone itself, the MLX Swift runtime for the 1-bit model reported about 11 tokens/second on an iPhone 17 Pro Max.

The card’s most practically useful contribution may be its peak-memory table — what a device must actually accommodate is weights plus KV cache plus activations and runtime buffers (~1.3 GB across backends), not the headline weight size:

Peak memory in decimal GB, language model only, KV cache left at FP16. With the near-lossless 4-bit KV quantization enabled, the 1-bit build’s 100K peak drops to ~6.8 GB and the full 262K window fits in ~9.4 GB peak. The hybrid backbone grows its full-attention cache on only 16 of 64 layers (~4.3 GB at the full 262K window in FP16).

DSpark: speculative decoding for a ternary target

Both first-generation packs shipped with DSpark, a speculative-decoding drafter layer trained against the low-bit target — the detail that made a 3.9 GB phone-class model feel fast on GPUs. The card’s description is specific: a semi-autoregressive drafter with confidence-scheduled verification; a compact six-layer block-parallel transformer conditioned on hidden states tapped from five evenly spaced layers of the target model; drafter-unique weights of roughly 0.5 GB at serving precision (embeddings and output head shared with the resident target); a diffusion-flavored block-denoising objective with survival-probability-weighted distillation; and a draft block size chosen from a measured verify-cost model of the serving stack. The drafter ships 4-bit quantized as a 1.79 GB Q4_1 pack by default (a 7.29 GB bf16 reference also exists).

Because verification preserves the target distribution exactly, the acceleration is lossless — drafter precision affects speed, never output. On the CUDA serving path, an accepted length of τ ≈ 3.6 at draft depth k = 4 turned into a 1.37× end-to-end decode speedup on H100 for the binary model (104.8 → 143.8 tok/s) and 1.34× for the ternary. On Apple Silicon, batch-1 verification does not yet amortize, so the drafter is not enabled by default on-device. The card also quotes decode energy on the M5 Pro of 0.275 mWh/token with the drafter enabled — against 0.63–1.32 mWh/token across the datacenter GPU classes it measured.

8 Anatomy of Bonsai 2 27B

A Qwen3.8-27B backbone left architecturally untouched, a ternary grid applied end to end, and two new packing decisions that define the release.

Bonsai 2 27B is, before everything else, a faithful copy of a specific strong model. The base is Qwen3.8-27B, the most capable generation of the Qwen open-model family at the 27B tier: a native vision-language model “designed to carry complex, multi-step tasks through to completion with greater reliability,” in its own card’s words, with thinking mode on by default, tunable reasoning effort, and preserved reasoning context. The whitepaper is explicit that the Bonsai pipeline left the architecture unchanged — every structural number below is inherited, not designed, by Prism:

System specification, consolidated from the Bonsai 2 27B whitepaper (Table 1) and both Hugging Face model cards. The Qwen3.8-27B card supplies the base-layout row: 48 linear-attention V heads / 16 QK heads at head dimension 128 for Gated DeltaNet; 24 Q / 4 KV heads at head dimension 256 with RoPE dimension 64 for the gated full-attention layers.

The choice of base deserves a moment, because it is where the release’s quality story starts. Qwen3.8-27B’s own card reports results like 90.3 on LiveCodeBench v6, 89.2 on GPQA Diamond, 73.0 on Terminal-Bench 2.1 (Terminus), 84.3 on OSWorld-Verified, and 91.1 on OmniDocBench 1.5 — a compact, deployment-friendly frontier-tier agentic model, not merely a chat model. Compressing a model of that caliber to 98.2% retention is a categorically different claim than compressing a mid-tier one; and the choice of the 27B tier specifically — small enough for consumer hardware, large enough to carry real agentic capability — is the release’s strategic center of gravity.

The ternary grid, precisely

The whitepaper’s formal statement of the format is compact enough to quote almost in full. Each weight takes a value in {−1, 0, +1} with one shared FP16 scale per group of 128 weights:

A ternary value carries log₂(3) ≈ 1.585 bits of information, so with one FP16 scale amortized over 128 weights the effective storage cost of the ternary tensors is b_eff ≈ 1.71 bits/weight. That covers the ternary tensors only; the release’s second-generation refinement is a short, explicitly enumerated list of tensors held above the low-bit representation — the previous generation ternarized these alongside everything else, and Bonsai 2 does not. The list is the recurrent-state path of the linear-attention layers (the two in_proj matrices at 11.8M parameters each, the short conv1d, and the small A_log/dt_bias/state-norm tensors) plus ordinary normalization weights — 26.2M parameters, 0.0976% of the language model, 52 MB at bf16. Counting them in, the model as a whole is 1.72 bits per weight, a ~9.3× reduction in principle against FP16. The practical figure, as shipped in the dense-trit GGUF packing with its container overhead, is 1.76 bits per weight and 5.93 GB against the FP16 baseline’s 53.8 GB — ~9.1×.

Two packings, one representation

Practical deployment needs formats that efficient kernels can consume, and Bonsai 2 ships two — a pairing introduced with this release, replacing the first generation’s single Q2_0_g128 layout:

• PTQ1_0 packs trits densely and “lands essentially on the information-theoretic target” — 1.75–1.76 bits per weight, 5.93–5.95 GB, ~9.0–9.1× smaller than FP16. The catch: unpacking dense trits costs arithmetic, so it is not uniformly faster despite moving 17–18% less weight data per step.
• PQ2_0 stores each trit in a 2-bit slot — 2.13–2.16 bits per weight, 7.21–7.25 GB, ~7.4–7.5× smaller — trading footprint for cheaper unpacking. It is the faster decode on Hopper- and Blackwell-class parts and the faster prompt-processing everywhere.

Storage footprint of the language model, whitepaper Table 3 (decimal GB). The model cards round the same rows to 5.95/7.21 GB, and the Ollama mirror lists 5.53/6.70 GiB — the same packs in binary units. The vision tower adds 0.63 GB (GGUF, 4-bit) or 0.92 GB (MLX, FP16) when image input is needed; the MLX pack on disk is 8.60 GB including it.

The MLX packaging deserves its own note because it is the release’s most instructive example of container overhead. MLX’s grouped low-bit format stores both a scale and a bias per group; Bonsai’s ternary levels {−s, 0, +s} need only the scale, so the bias is set to −s and is formally redundant — the 2-bit codes {0, 1, 2} decode to −s, 0, +s exactly. But the container stores two FP16 values per 128-weight group where the native format stores one, so the block costs 36 rather than 34 bytes and the packed rate rises from 2.125 to 2.250 bits per weight. The whitepaper is careful to call this “a container property, not a different representation”: the packed weights decode to exactly the same ternary values as the GGUF bands, “verified by comparing the group scales bit for bit.”

Serving parameters

The recommended sampling configuration is inherited verbatim from the base model’s generation_config.json and carried in the GGUF metadata (general.sampling.*): thinking mode at temperature 1.0, top-p 0.95, top-k 20, min-p 0, no presence or repetition penalty; instruct mode at temperature 0.7, top-p 0.80, top-k 20, presence penalty 1.5. The model thinks by default at xhigh reasoning effort; medium trades some quality for shorter responses (both effort levels are fully reported in the whitepaper’s appendix — 79.3 versus 83.9 average on the 20-suite); low is not supported. The system prompt can be as simple as “You are a helpful assistant.”

9 The Rotated Basis

Why ternarizing in a Hadamard-rotated coordinate system works, why the rotation costs nothing at inference, and the runtime contract that enforces it.

The single most consequential idea in the Bonsai pipeline, after the ternary grid itself, is the basis in which the grid is applied. Ternarizing raw FP16 weights directly fails for a now well-understood reason: real weight matrices are badly conditioned for rounding — outlier dimensions concentrate variance in a few directions, and a uniform grid wastes most of its resolution on directions that barely matter while shredding the ones that do. The fix, descending from the QuIP/QuIP# line and from SpinQuant (Liu et al., 2024 — the whitepaper’s reference [28]), is to rotate the weight matrix into a basis where no direction is special before the grid is applied.

The whitepaper’s formalization: ternary matrix weights W are stored transformed blockwise by

where H_n is a Walsh–Hadamard matrix and S a fixed diagonal of ±1 signs. Inference at each layer then computes f(x) = W(Rx) — the activation is first transformed by the same rotation, then multiplied by the ternarized matrix. An orthogonal rotation preserves lengths and angles exactly, so the function computed is mathematically identical to the un-rotated one; what changes is only which coordinate system the ternary rounding error lands in. The Hadamard structure is what makes this practical: H₁₀₂₄ is a fixed matrix of ±1 entries, and multiplying by it costs O(n log n) via the fast Walsh–Hadamard transform (FWHT) — the 1976 Fino–Algazi result the whitepaper cites as reference [29] — which is negligible next to the matrix multiply it precedes.

The fold, and the zero-bit rotation

The release’s cleverest engineering decision is that the rotation costs no extra bits and no extra weight traffic. Rather than storing W and R separately, the pipeline folds the rotation into the stored weights offline: what is ternarized and packed is the rotated matrix itself, so the packed file contains one ternary tensor per projection, exactly as an un-rotated build would. At runtime only the activation-side transform remains — a fixed sign flip followed by a blockwise FWHT before each folded matrix multiply. The card’s phrasing: “the rotation is folded into the stored weights offline, so it costs no extra bits and no extra weight traffic.”

That activation-side transform is not free, and the whitepaper is unusually candid about where it hurts. At batch size one it lies “on the critical path of every projection” — Appendix A names it “one of the larger non-matrix-multiply costs of a decode step” and lists it among the release’s open engineering targets. The mitigation is per-backend: on Metal, the sign flip is fused into the transform’s load path, “eliminating a separate activation pass” — a substantial saving when the transform accounts for a substantial fraction of the operations dispatched per generated token; on CUDA, the transform is parallelized across a full thread block rather than a single warp, “reducing serialization across the Hadamard stages.” Prompt processing is largely insensitive, since the transform amortizes across many tokens in the compute-bound regime.

Why the rotated grid holds reasoning together

It is tempting to treat the rotation as an implementation detail, but it is the most plausible mechanical explanation for the release’s signature result — that reasoning survives ternarization while conventional sub-4-bit builds lose it first. The mechanism, in outline: an un-rotated grid’s rounding error is concentrated in the directions of largest weight variance, which in transformer matrices correlate with the residual stream’s functionally loaded directions; the compounding, multi-step character of reasoning benchmarks then amplifies small per-step corruption into task failure. The rotated grid’s error, by contrast, is spread uniformly across directions — closer to isotropic noise, which deep networks tolerate far better and which does not systematically target the computations a chain of reasoning depends on. The whitepaper itself frames the observed pattern in exactly these terms: conventional low-bit failure “is easy to miss in casual testing and expensive to discover in production” because surface fluency survives while “long context reasoning and coding deteriorate first.”

The independent evidence for this framing is the cross-family replication in Chapter 7’s table — the same selective collapse appears in IQ2_XXS builds of both Qwen3.6-27B and Gemma-4-31B, so the phenomenon is a property of the methods, not of one base model — and the per-benchmark pattern of Chapter 11, where the conventional 2-bit build’s worst losses land precisely on AIME and LiveCodeBench while its MMLU-Redux score barely moves. A skeptic should still note what this chapter has not established: no public ablation isolates the rotation’s contribution from the rest of Prism’s pipeline (the training-free conversion, the group scaling, the full-precision state path). The rotation is the best-supported explanation available, not a measured decomposition.

10 Low-Bit Kernels

Custom hybrid-attention kernels for MLX and CUDA, the bandwidth profile they exploit, and the PTQ1_0/PQ2_0 trade across GPU generations.

A compact weight representation is useful only if the runtime can execute it directly, and here Bonsai 2 hit a genuine infrastructure gap: existing low-bit inference paths target standard full-attention transformers and simply do not cover Qwen3.8-27B’s structure, which interleaves 48 Gated DeltaNet linear-attention layers with 16 full-attention layers in a repeating 3:1 pattern. The whitepaper’s summary of the problem: “Existing low-bit inference paths primarily target standard full-attention transformers and do not cover the hybrid-attention structure of Qwen3.8-27B.” The release therefore ships custom kernels for the hybrid backbone on Apple Silicon and CUDA, operating directly on the packed ternary weights — “avoiding materialization of dense FP16 weights,” which the cards call essential to realizing the representation’s bandwidth advantage.

Three runtimes carry the kernels: the MLX fork (Python, Apple Silicon) and mlx-swift fork (iOS/macOS) for Apple platforms, and the llama.cpp fork (CUDA + Metal, with CPU paths) for everything else. The CUDA backend’s description in the whitepaper’s Appendix A is terse but complete: fused low-bit GEMM kernels that unpack ternary codes and apply group-wise scales inside the matrix multiply, with activations and numerically sensitive operations kept in higher precision. The Apple side adds the MLX-specific loading contract — the pack declares model_type: prism_hadamard_qwen35 and requires the loader bundled in its runtime/ directory, because “ordinary MLX loaders skip the activation transform and the inverse embedding lookup, so they return wrong output rather than an error.”

The bandwidth profile, measured

Chapter 1 claimed decode is bandwidth-bound; the release’s own measurements close the loop. On the M5 Pro laptop, the 27.7 tok/s decode corresponds to ≈201 GB/s of weight traffic — the model is, functionally, a bandwidth benchmark with an LLM attached. The whitepaper adds the operational details that make the number trustworthy: each configuration is measured three times after a warm-up pass, on the release artifact and backend build of September 16, 2026, with NVIDIA hosts first passing a memory-bandwidth health check (reaching 76–91% of rated bandwidth), and prefill acknowledged as the noisier measurement (run-to-run variation up to ~11% on the fastest devices).

The packing trade, across architectures

The two packings are “a genuine trade rather than a strict ordering,” and the whitepaper’s architecture-specific analysis is the most instructive part of the throughput story. PTQ1_0 moves 17–18% less weight data per step than PQ2_0, but unpacking dense trits costs arithmetic. Which cost dominates depends on what limits the hardware:

• Ada-generation cards and the L4 — memory-constrained parts — favor PTQ1_0: RTX 6000 Ada (92.0 vs 84.8 tok/s), RTX 4090 (96.7 vs 90.9), L4 (32.1 vs 29.7). Bytes saved translate directly into tokens.
• Ampere, Hopper, and Blackwell — where batch-1 decode is limited by instruction throughput and launch overhead rather than bandwidth — favor PQ2_0: H100 SXM (103.2 vs 77.3), A100 (74.0 vs 54.6), RTX 5090 (142.5 vs 134.4), RTX PRO 6000 Blackwell (140.6 vs 136.8).
• Prompt processing favors PQ2_0 everywhere — prefill is compute-bound and the cheaper unpacking matters more than the extra bytes: 4121 vs 1901 tok/s on the RTX 5090.

The GGUF card compresses the guidance into a rule of thumb: PQ2_0 is “the faster decode on H100, A100, and the Blackwell cards, faster at prompt processing everywhere, and is the pack measured on Apple Silicon”; PTQ1_0 is “the faster decode on the Ada-generation cards and the L4, and the pick wherever memory is tightest.” Making PTQ1_0 as fast as PQ2_0 everywhere — on top of its lower footprint — is named in the whitepaper’s limitations section as an active engineering target.

11 The Benchmarks

Two suites — twenty benchmarks plus a fourteen-benchmark companion — every per-benchmark table, the methodology behind them, where conventional 2-bit collapses, and how far the agentic claims reach.

Bonsai 2 27B’s evaluation is documented in more depth than any prior Bonsai release, and it comes in two parts that must be kept distinct: the 20-benchmark suite of the technical report — the release’s primary evidence, including the agentic long-horizon results — and a 14-benchmark companion suite carried on the Hugging Face model cards, which adds the UD-Q4_K_XL four-bit comparison point and per-benchmark detail against it. Both are thinking-mode evaluations run through the same infrastructure: EvalScope with the vLLM backend on NVIDIA H100 GPUs (the whitepaper specifies tensor-parallel 2, data-parallel 4), reasoning effort xhigh, with a server-side reasoning parser so that “extractors and judges score only the final answer.”

The headline table

Thinking-mode average across the 20-benchmark suite, whitepaper Table 7. Bits-per-weight are averages over all model weights for the packed checkpoint. The two sentences the table supports: Bonsai 2 retains 98.2% of the current-generation base at 1/9th its size, and its average slightly exceeds the previous generation’s full-precision model — 83.9 versus 83.6 — while the conventional 2-bit build falls 8.7 points short of it despite being 1.23× larger than Bonsai’s pack.

Category-level results

Whitepaper Table 8. Note the shape of the gap: the reasoning core holds — math within half a point of full precision (96.57 vs 97.06), coding within 0.6 (81.58 vs 82.17), instruction following above both baselines — while the remaining distance concentrates in knowledge & reasoning (−2.7) and vision (−3.1), the two most demanding categories.

The full 20-benchmark table

Whitepaper Table 10, thinking mode with reasoning-effort xhigh. Sizes: 53.8 / 53.8 / 7.35 / 5.93 GB (language-model weights, decimal GB); bits/weight 16.0 / 16.0 / 2.2 / 1.76. On 9 of 20 benchmarks Bonsai 2 lands within a point of the current-generation FP16 baseline or ahead of it — including LiveCodeBench (90.07 vs 90.05) and IFBench (74.00 vs 71.00), where it is ahead; its largest deficits are GPQA Diamond (−4.8), OCR Bench v2 (−4.1), BigCodeBench (−3.4), and the vision pair OmniDocBench/RealWorldQA (−3.3 each).

The selective collapse of conventional 2-bit

The aggregate gap “understates where quality is preserved,” and the whitepaper’s chosen example is the release’s sharpest evidence. On the 20-suite, IQ2_XXS falls to 78.6 on AIME26 and 70.05 on LiveCodeBench while still posting 85.79 on MMLU-Redux — “which is why casual testing misses the collapse.” Bonsai 2 scores 95.83 and 90.07 on the same two benchmarks — the latter actually a hair above the FP16 baseline’s 90.05 — and 89.09 on MMLU-Redux. The 14-benchmark companion suite sharpens the contrast further with a different IQ2_XXS build (2.8 true bpw at 9.4 GB): there the conventional build’s AIME26 falls to 57.5 and LiveCodeBench to 56.4, versus Bonsai’s 95.83 and 90.07. Either way, the pattern is the same: knowledge survives, sustained reasoning dies.

The agentic long-horizon results

For the first time, the evaluation extends to end-to-end software engineering, and here the retention story is more sober — honestly so. On Terminal-Bench 2.1 (run with the Terminus-2 agent on all 89 tasks, single attempt, eight concurrent trials, 32 CPUs and 48 GB of memory per task container, a 262,144-token context and an 80,000-token output budget per turn), Bonsai 2 reaches 52.8 against the FP16 baseline’s 69.7. On SWE-bench Verified (the mini-swe-agent scaffold on all 500 instances, resolved rate), it reaches 60.8 against 80.6. That is roughly three-quarters retention on both — materially below the suite’s 98.2%, and the whitepaper says so directly: “long-horizon agentic evaluations are unforgiving under aggressive compression,” while noting that “models at this footprint often fail to make any meaningful progress” where Bonsai 2 “remains capable of sustained tool use and end-to-end software-engineering workflows.” The shorter-horizon agentic results are stronger: τ²-Bench reaches 80.2 (up from 73.6 in the previous generation), BFCL v3 holds at 74.9, and AA-LCR lands within a point of full precision at 77.0.

A methodological note for cross-source readers: Qwen’s own card reports Terminal-Bench 2.1 at 73.0 for the FP16 model against Prism’s 69.7 — the same model, different harness protocol. Prism’s numbers are internally consistent (both models through the identical Terminus-2 configuration), which is what a compression comparison requires, but the absolute figure is protocol-dependent. The medium-effort appendix completes the picture: at reasoning effort medium, the FP16 baseline averages 82.6 and Bonsai 2 averages 79.3 — a slightly wider relative gap than at xhigh, with AIME25 the largest single drop (86.25 → 74.58).

Methodology, in enough detail to audit

The whitepaper’s Appendix B is the most auditable evaluation description the local-model field has produced in some time, and its key commitments deserve listing, because they are what separates this release’s benchmark claims from a marketing table:

• One harness, all models. Every variant — FP16 baselines, conventional low-bit references, Bonsai — runs through the same EvalScope pipeline against the same vLLM serving path with identical dataset versions and scoring.
• Single-sample discipline. Every item is answered once: pass@1 at temperature 1.0, not greedy decoding — except where a benchmark’s convention demands sampling: GPQA Diamond five samples per question (990 completions, mean accuracy), AIME25/AIME26 eight samples per problem (240 completions each, avg@8).
• Reasoning stripped before grading; generous output budgets (81,920 tokens for AIME/MATH-500/GPQA/LiveCodeBench/IFBench; 49,152 for HumanEval+/BigCodeBench/OmniDocBench/OCRBench/τ²-Bench; 32,768 for the rest).
• Rule-based scoring wherever possible — sandboxed execution for code, programmatic verifiers for instruction following, parsing and state checks for tool calling, reference matching for OCR — with a rule-first extractor and a fixed LLM-judge fallback (Gemini 3 Flash, temperature 0, identical across models) for knowledge, reasoning, and math.
• Explicit benchmark-version choices: MMLU-Redux 2.0 (57 subjects, 5,700 questions); LiveCodeBench release_v6 (1,055 problems); BigCodeBench complete split over all 1,140 problems (not the 148-problem Hard subset), v0.1.4 with calibrated pass@1 in a Docker sandbox; IFEval/IFBench reported at prompt-level strict accuracy; τ²-Bench’s airline, retail, and telecom domains (278 tasks) scored as single-trial pass@1 with deepseek-v4-flash at temperature 0 as the user simulator; BFCL v3 in prompt-based function-calling mode over all 17 subsets including multi-turn (4,441 items, sample-weighted mean).
• Agentic parity: all agentic benchmarks serve models identically and drive them through the Harbor framework.

The 14-benchmark companion, in one paragraph

The model cards’ companion suite (thinking mode, same infrastructure) replaces the suite’s GPQA/AA-LCR/BigCodeBench/τ²-Bench/CharXiv/A-OKVQA/OmniDocBench/RealWorldQA entries with MuSR, MMMU-Pro, and a slightly different benchmark mix, and adds the four-bit comparison point that makes the deployment decision concrete: UD-Q4_K_XL at 5.2 true bpw and 17.6 GB scores 85.18 (98.7%), against Bonsai 2’s 84.78 (98.2%) at 5.9 GB — “within 0.4 points of UD-Q4_K_XL at three times the footprint.” Per-benchmark on the overlapping entries the two suites agree on Bonsai’s numbers exactly (AIME26 95.83, LiveCodeBench 90.07, IFEval 91.31, BFCL v3 74.92), which is itself a small integrity signal: the companion was clearly re-run, not copy-edited. The full 14-benchmark table, including the IQ2_XXS collapse figures cited above, appears in the cards; the whitepaper’s 20-suite remains the release’s canonical record.

12 Throughput & Energy

Twelve platforms, two packings, and an energy-measurement discipline unusual enough to be its own finding.

The whitepaper’s throughput table is the release’s most complete hardware survey — twelve platforms from a 72-watt L4 to an H200, plus three Apple laptops — measured on the current release stack (artifact and backend build of September 16, 2026) with the standardized tg128/pp512 protocol: token-generation throughput over 128 generated tokens (the bandwidth-bound, interactive phase) and prompt-processing throughput over 512 input tokens (the compute-bound phase), each configuration run three times after warm-up, batch size one, depth zero, vision tower excluded.

Whitepaper Table 5: cross-platform throughput and energy for the two packings, tokens/second and mWh/token (board power, no idle subtraction). Bold marks the faster decode per row — note the Ada/L4 rows flipping to PTQ1_0 exactly as Chapter 10’s analysis predicts. The announcement’s “up to 143 tokens/second on RTX 5090” is this table’s 142.5 rounded; its “0.714 mWh/token on an RTX 4090” is the PQ2_0 energy cell of the same row.

Whitepaper Table 6. The M4 Pro row comes from the earlier pre-rotation build and should be read as a scaling indicator rather than a release-stack comparison; on M4 Pro, prefill (~125 tok/s) rather than decode is the practical constraint for very long prompts. The announcement’s “46.8 tokens/second on M5 Max” is this table’s top row.

What the numbers mean in practice

Three practical readings of the tables are worth spelling out. First, the laptop row is qualitative, not a speedup. On a 16-class GB laptop, the FP16 baseline (54 GB) and even the conventional 4-bit build (17.6 GB) do not fit at all — “the meaningful statement is not a speedup ratio but that a 27B model runs interactively on an everyday laptop,” at 27.7 tok/s on an M5 Pro and 46.8 on an M5 Max. Second, the consumer-GPU row is the value proposition. ~130–143 tok/s of 27B-class decode from a single RTX 5090, ~91–97 tok/s from a 24 GB RTX 4090, and ~30 tok/s from a 72-watt L4 — with the card’s reminder that there is “headroom for larger batches, longer contexts, or co-resident models” since the weights leave most of a 24–32 GB card free. Third, prefill is the constraint to watch on-device. Prompt processing runs far faster than decode on GPUs — 468–4,520 tok/s of PP512 across the measured cards, against 30–143 tok/s of decode — but on Apple laptops it sits at 125–765 tok/s, which for very long prompts — the 262K-context workloads the model is otherwise built for — makes ingestion time, not generation, the binding limit.

The energy discipline

The most unusual feature of the throughput work is what the paper refuses to report. NVIDIA energy is board power via nvidia-smi, “total, with no idle subtraction”; Apple’s powermetrics reports CPU, GPU, and ANE with no DRAM rail. Because excluding memory power would systematically flatter the Apple figures in a bandwidth-bound workload, the whitepaper declines to publish a cross-platform per-token energy number at all and instead reports absolute draw: the M5 Pro sustains plain autoregressive PQ2_0 decode at 27.0 tok/s drawing 27.0 W on the GPU rail and 32.8 W across CPU and GPU, with the idle floor stated separately (0.17 W GPU, 1.47 W CPU) rather than silently subtracted. Against that, the NVIDIA cards’ board power runs 300–455 W under the same batch-1 decode.

The announcement’s headline energy claim — “40% more energy-efficient than an 8B model running in full-precision” on the RTX 4090, at 0.714 mWh/token — is a comparison against a model class one-third its size, which is the right framing for the claim being made: per unit of delivered capability, the ternary 27B beats an FP16 8B on energy. And the whitepaper adds a detail deployers should internalize: prefill is far cheaper per token than decode everywhere measured (0.012–0.060 mWh/token, e.g. 0.017 on the RTX PRO 6000), because prompt processing amortizes weight traffic across many tokens.

13 Deployment Field Guide

What actually runs where: the fork, the MLX runtime, the Ollama situation, the browser demo, the API, and the integrations.

Bonsai 2 27B’s deployment surface is broader than any prior Bonsai release but comes with one non-negotiable requirement that shapes everything else: the packed formats need a runtime that knows them. The model cards designate the PrismML-Eng/Bonsai-demo repository (2,924 stars) as “the source of truth for running these models — it carries the tested setup for every backend, pins a known-good binary release, and is kept current”, with the rule stated without diplomacy: “Where anything here disagrees with it, it is right.” The repo ships run scripts that pick correct flags per hardware, plus serving, benchmarking, and integration examples for tool calling, reasoning budgets, image input with the mmproj file, and speculative decoding.

llama.cpp (CUDA, Metal, CPU)

The GGUF packs run through Prism’s llama.cpp fork, which carries the ternary hybrid-attention kernels and the Hadamard activation runtime. Stock llama.cpp “will not run these files. It rejects PQ2_0 and PTQ1_0 as unknown types, and it loads Q2_0 without any warning and produces garbage, because it has no Hadamard activation runtime” — the first-generation format’s name collision with llama.cpp’s existing Q2_0 being the trap case. Prebuilt release archives exist (Simon Willison’s thread-tested pick was the prism-b10685-7dffb15 tag) alongside build-from-source instructions:

# Clone the PrismML fork of llama.cpp (ternary hybrid-attention kernels)
git clone https://github.com/PrismML-Eng/llama.cpp
cd llama.cpp

# Build with CUDA support (drop -DGGML_CUDA=ON on macOS; Metal is default)
cmake -B build -DGGML_CUDA=ON
cmake --build build -j

# Download the ternary GGUF weights (PQ2_0 packing shown)
hf download prism-ml/Ternary-Bonsai-2-27B-gguf Ternary-Bonsai-2-27B-PQ2_0.gguf --local-dir .

# Run inference: this is a reasoning model — it thinks by default
./build/bin/llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
  -ngl 99 -fa on -c 32768 \
  --temp 1.0 --top-p 0.95 --top-k 20 \
  -p "Explain quantum computing in simple terms." -n 256
# -ngl 99 offloads every layer (0 = CPU-only); -c sets context, up to 262144

Apple MLX (Python and Swift)

The MLX pack runs on stock mlx-lm/mlx-vlm packages — “no fork” — but requires the custom loader bundled in the pack’s runtime/ directory, because the model declares model_type: prism_hadamard_qwen35 and ordinary MLX loaders “skip the activation transform and the inverse embedding lookup, so they return wrong output rather than an error.” The vision flow:

# Get the pack (language model + vision tower, 8.60 GB on disk)
hf download prism-ml/Ternary-Bonsai-2-27B-mlx-2bit --local-dir bonsai2-27b-mlx

pip install -r bonsai2-27b-mlx/runtime/requirements.txt

# The pack's own loader applies the Hadamard transform + inverse embedding
import sys; sys.path.insert(0, "bonsai2-27b-mlx/runtime")
from vision_artifact import load_vl_model, chat_config
from mlx_vlm import generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor, config = load_vl_model("bonsai2-27b-mlx")
prompt = apply_chat_template(processor, chat_config(config),
                             "What is in this image?", num_images=1)
print(generate(model, processor, prompt, ["photo.jpg"],
                 max_tokens=256, temperature=1.0))
# Pass no images for text-only use. PACK-RUNTIME.md documents the contract.

Ollama, LM Studio, and the browser

Ollama cannot run these packs today. A community mirror exists — tobestyledintro/Ternary-Bonsai-2-27B, republishing Prism’s blobs with identical SHA-256 hashes, 7,961 downloads in its first days — but its own readme carries the warning in bold: “These files do not run in Ollama.” The mirror’s verification against Ollama 0.34.2 is specific: the runner “knows the architecture (qwen35) but has no pq2_0 or ptq1_0 anywhere in its library” and fails with unsupported tensor "output.weight" size overflows. The mirror’s model page nonetheless documents the application surface that wants this model — Claude Code, OpenCode, Hermes Agent, and OpenClaw can all be pointed at a compatible server via ollama launch-style flows once a runtime exists. LM Studio is listed among the pack’s supported local apps (a faster turnaround than the first generation, whose GGUF and MLX packs initially failed there outright). And the most accessible entry point of all requires no install: the WebGPU Space (webml-community/ternary-bonsai-2-webgpu-kernels, 98 upvotes in its first days) runs the full ternary model in the browser — with the community’s standing reminder to clear the downloaded weights afterward.

The API surface

For deployment without local hardware, OpenRouter listed prism-ml/ternary-bonsai-2-27b on September 18, 2026 at $0.075 per million input tokens and $0.50 per million output, served by a single provider (Darkbloom) with 99.99% uptime over the following three days, P50 latency of 0.66 seconds, and a tool-call error rate of 0.02% — alongside a structured-output error rate of 20.02% that early adopters should weigh before depending on strict JSON modes. Prism’s own free, limited-time developer-preview API has run since the July launch. The pricing is the interesting signal: at roughly a tenth of typical frontier-model output pricing, the ternary model’s serving economics — a 5.9 GB resident footprint on commodity GPUs — are being passed through to the API customer.

Docs coverage and the integration map

Prism’s documentation covers the fork’s prebuilt binaries across macOS, Apple Silicon, Windows x64, CPU, CUDA 12.4, Vulkan, and HIP/ROCm — the broadest backend matrix any low-bit release has shipped — and the model cards enumerate first-party integration recipes for agent harnesses that were evidently treated as the primary users from day one: Pi (the coding agent, via a local OpenAI-compatible server), Hermes Agent, OpenClaw, plus Cline and a computer-use stack driving the announcement’s two launch demos on an RTX 5090. The pattern across all of them is identical: run a local OpenAI-compatible endpoint (llama.cpp server or MLX server) and point the harness at it — the deployment model that made local models useful for agents in 2025–2026, now supplied with a 27B-class reasoning model at an 8B-class footprint.

14 Reception & Reception Risks

A 579-point launch thread, three independent write-ups, 2.23 million downloads in four days — and the criticisms that survived fact-checking.

The Bonsai 2 launch reached Hacker News’s front page within hours and finished its run at 579 points and 199 comments — strong, though below the first-generation 27B’s 706, a normal decay for a second act. The thread’s most practical contribution came early from Simon Willison, who flagged the deployment trap this paper’s Chapter 13 exists because of — the GGUFs require Prism’s llama.cpp fork, with a pinned release tag and a tested command sequence — and the thread’s most upvoted practical observations traced the deployment envelope: excitement that a sub-6 GB pack puts a 27B-class model on 16 GB graphics cards that could never hold Qwen3.8-27B’s FP16 (let alone its 4-bit), and confirmation that the full model runs in the browser via the WebGPU Space.

The criticisms, taken seriously

Four lines of criticism recur across both 27B launches, and they deserve precise statement because each one survived this paper’s fact-checking pass in some form:

• Long-horizon degradation. The most quoted community verdict, from the September thread: “Like the last model, it’s amazing they work as well as they do. Use it for any longer task and they fall apart spectacularly and in interesting ways.” This is consistent with the whitepaper’s own long-horizon numbers — 52.8 on Terminal-Bench 2.1 and 60.8 on SWE-bench Verified against 69.7/80.6 for FP16 — and with the first generation’s τ²-Bench collapse in the binary build (82.9 → 61.3). The honest reading: the 98.2% headline describes the 20-suite average, not the hardest agentic tail.
• The PTQ-rivalry question. From the July thread (the commenter liuliu): Unsloth’s post-training UD_Q2 variant “does only post-training” and gets numbers “pretty close to the 1-bit model here,” while the 5% tool-calling drop matters more in real use than averages suggest. Prism’s own 14-suite table partially answers this — Bonsai 2 at 84.78 beats the IQ2_XXS class at 72.59 on the same suite — but the criticism correctly notes that the right conventional comparison point is contested territory, and that the ternary model’s advantage is narrower against modern dynamic 2-bit recipes than the headline suggests.
• Knowledge, not reasoning, is the weak flank. The July thread’s summary judgment — “Where it falls apart is with knowledge. It has good prose/logic for a tiny model” — matches the category table: knowledge & reasoning is where Bonsai 2 gives up the most (86.66 → 83.95), with MuSR (79.63 → 70.63 on the 14-suite) and the OCR edge cases the visible symptoms. A model that reasons like a 27B but recalls like a compressed one is a specific product shape, not a universal substitute.
• Runtime friction and speed skepticism. The first generation launched with LM Studio incompatibility and early llama.cpp builds that ran slower than competing MoE quants with CPU offloading; the second generation still requires a fork. Third-party write-ups from the September window state the constraint plainly: running it “requires a custom llama.cpp fork from Prism ML’s model card, since stock llama.cpp cannot decode the ternary kernels or apply the” activation transform. Upstreaming is in active discussion — llama.cpp issues and PRs reference GGML types 142 (PQ2_0) and 143 (PTQ1_0), and a September 18 GitHub discussion on group-128 ternary support records the fork’s constraint that upstream Q2_0 uses per-row scales “so won’t be native support” — but at publication time, the ecosystem integration a casual user expects does not exist.

The independent record

Outside the aggregator threads, the third-party coverage converges on the same balanced picture. regolo.ai’s Alex Genovese (July 17) ran the first generation’s numbers and drew the line the community kept redrawing: tool calling and agentic capability “drop significantly in the 1-bit variant (80.0 → 66.0), making it 4.6× more affected than mathematical reasoning tasks” — so “for AI agents, coding assistants, and tool-using workflows, the ternary Bonsai 27B is the recommended choice, while the 1-bit version is best suited for chatbots, text generation, summarization.” mindstudio.ai’s September write-up restated the headline correctly (“Bonsai 2 27B shrinks a 27B model to 8.6GB using ternary quantization, keeping 98.2% of FP16 benchmark performance intact” — the 8.6 GB being the MLX pack with vision tower on disk). And benchlm.ai’s independent tracker offers the most sobering external datapoint: on its own public-score estimate, Qwen3.8-Flash-Next outscores Ternary Bonsai 2 27B (57.11 vs 51.45) — with the caveat, in benchlm’s own phrasing, that the 90% score intervals overlap, and that its estimate is a different instrument from Prism’s suite. A frontier-class smaller model from the same family can still beat the compressed 27B on capability-per-token; Bonsai’s claim is capability-per-gigabyte, not capability per parameter.

Adoption, measured

Whatever the caveats, the adoption signal is unambiguous and unusually fast. The GGUF pack crossed 2.23 million downloads within four days of release (against 36.7 thousand for the MLX pack and 4.5 thousand for the dev pack), the Hugging Face collection earned 135 upvotes, and the org passed 3.66k followers. The first generation’s packs had been compounding for months — 442k downloads/month for the 1-bit 27B, 124k for the 1-bit 8B — and the ecosystem’s build-out (23 community Spaces on the first-gen 1-bit model alone, WebGPU kernels, community mirrors, Docker serving recipes on ModelScope and beyond) shows a deployment community that treats the format as infrastructure, not a curiosity. A mid-August community thread on the first-gen ternary card asking for a “Qwen3.8-27B version” — answered with a discussion of the ternary container’s properties under the name TAARDIS — foreshadowed exactly what shipped a month later: the base-model treadmill, and the community waiting on it, are both real.

15 Intelligence Density

The metric Prism invented to price capability per gigabyte — what it measures, what it hides, and what it predicts.

Every Bonsai release has been framed through a single invented quantity. Intelligence density, introduced in the 1-bit Bonsai 8B report, is defined as benchmark performance per deployed gigabyte under a logarithmic error transform: with average score avg and error probability P_e = 1 − avg/100,

The logarithm is the load-bearing choice, and Prism is explicit about why: it “places greater weight on preserving high accuracy,” because −log₂(P_e) grows without bound as the error probability approaches zero, while the naive score-per-GB metric would treat the difference between 95% and 99% accuracy as worth the same 4 points at any size. On the 20-benchmark suite, the release’s densities:

Whitepaper Table 9, thinking mode, 20-benchmark suite. On the 14-benchmark companion suite the same framework yields 0.469 for Bonsai 2 against 0.416 for the previous-generation ternary (“a 12.5% gain”), 0.199 for the denser conventional build there, and 0.053 for FP16 — which is the source of the cards’ “over 2.3× the density of the densest conventional build and nearly 9× FP16.”

What the metric genuinely captures is the deployment question this whole program is about: for a fixed memory budget — a phone’s 6 GB, a laptop’s 16, a consumer card’s 24 — how much task capability fits? By that measure the answer has moved by an order of magnitude in six months: 1-bit Bonsai 8B posted 1.062/GB in March (10.8× the FP16 Qwen3 8B’s 0.098), the first-generation ternary 27B posted 0.416, and Bonsai 2 posts 0.444–0.469 — the small model’s advantage shrinking as the flagship’s quality rose, which is the density frontier’s shape one should expect as compression moves up the capability scale.

What the metric hides is equally worth stating, because no single number should steer a deployment. It is benchmark-indexed, so it inherits every limitation of Chapter 11’s suites — including the fact that the agentic long-horizon results sit well below the suite average that feeds the density numerator. It prices storage, not operation: KV cache, activations, and runtime overhead (the ~1.3 GB of buffers from the first generation’s peak-memory table) are outside N, and the vision tower’s 0.63–0.92 GB is optional and uncounted. And its logarithm rewards accuracy in a way that favors exactly the regime where Bonsai is strongest — high average scores — which is to say, it is a well-constructed metric for the company’s thesis rather than a neutral one. Used as intended — as a placement on a capability-per-gigabyte frontier, alongside the category table and the long-horizon results — it is genuinely informative. Used as a single ranking number, it will mislead.

16 Limitations, Roadmap, Verdict

What the release itself admits, what it leaves open, how this paper verified its claims, and what to conclude.

The stated limitations

The whitepaper’s own limitations section is short, specific, and worth reproducing in substance, because each item is a real engineering constraint rather than a disclaimer. First, the quality–footprint trade remains a trade: 98.2% retention means the gap to FP16 is “modest and predictable,” concentrated in knowledge & reasoning and vision, and “deployments that need the last few points of accuracy can still reach for the full-precision model where its footprint is not a constraint.” Second, the native kernels are unfinished: PTQ1_0’s dense-trit unpacking wins on Ada-class and smaller accelerators but loses on Ampere, Hopper, and Blackwell, “where batch-1 decode is not bandwidth-starved” — returning the footprint advantage as latency on those targets is “an active engineering target.” Third, the roadmap names its next two moves: KV compression into the sub-4-bit regime (“early results show that KV cache can be pushed toward the sub-4-bit regime while retaining benchmark performance” — a path to still longer contexts in fixed memory), and the rotation overhead, “a pure implementation cost with no bearing on output quality” that is “the most direct remaining lever on on-device decode latency.”

To the whitepaper’s list this paper appends the constraints the community established in Chapter 14: long-horizon agentic retention of roughly three-quarters rather than ninety-eight percent; the contested conventional-comparison question; the knowledge-flank weakness; and the runtime-friction problem — a fork requirement and no stock-Ollama path at publication time, with upstreaming (GGML types 142/143) under active but unresolved discussion.

Open questions

• Does the method scale past 27B? The community’s most-requested next step — compress a 100B+-class model into a single-card footprint — is unanswered. Nothing in the public record tests the pipeline above 27B, and the whitepaper’s roadmap does not promise it.
• What exactly is the conversion pipeline? The rotated ternary grid, the group scales, the enumerated full-precision tail, and the results are all documented; the algorithm that chooses scales and signs for a given FP16 model is not. Post-training ternarization is a competitive research field (CAT-Q, ATQ, and peers in 2026), and Prism’s exact recipe — training-free or lightly calibrated, iterative or closed-form — remains the program’s least-disclosed component.
• MoE bases? Thread commenters asked for a Qwen3.8-Next compression that would fit a single 5090; the hybrid-attention kernels cover a dense 27B, and nothing public covers sparse mixture-of-experts routing under ternary weights.
• Hardware. The first generation’s card said it outright: “no native 1-bit hardware exists yet.” Every number in this paper is a software-kernel result on general-purpose silicon.
• Will the benchmarks survive contact with users? The suite is unusually auditable, but vendor-run; the community’s long-task degradation reports and benchlm’s overlapping-intervals estimate against Qwen3.8-Flash-Next are the current external checks, and both suggest the average overstates the hardest use cases.

The verification pass

Per this paper’s remit, every quantitative claim above was re-checked after writing against the primary sources: the 14-page technical report (downloaded and fully extracted), both Hugging Face model cards, all five Prism ML announcement posts, the Qwen3.8-27B base-model card, the first-generation 27B cards, Prism’s documentation and homepage, the OpenRouter and Ollama listings, both 27B Hacker News threads via their full comment records, and the third-party write-ups. Derived arithmetic was recomputed rather than trusted: the 1.71 bits/weight figure (log₂3 + 16/128 = 1.585 + 0.125), the 98.2% retention (83.9/85.4 = 0.9824), the 14.2× binary reduction (16/1.125), the density ordering, and every average recomputed from the per-benchmark tables (the 20-suite averages check to 85.4/83.6/75.2/83.9 within rounding). Where documents disagree — 5.93 vs 5.95 GB for PTQ1_0, 8.49 vs 8.60 GB for the MLX package, a 0.46B vs 0.47B vision-tower parameter count between the cards and the whitepaper’s Table 1, 142.5 vs 129.9 tok/s on the RTX 5090 across measurement rounds, 2.2-bpw/7.3 GB vs 2.8-bpw/9.4 GB IQ2_XXS builds across suites — both figures are reported with attribution, and the discrepancies trace to decimal-rounding conventions, container accounting, kernel-build vintage, and genuinely different comparison builds respectively. No claim in this paper rests on an unattributed number.

Verdict

Bonsai 2 27B is the strongest evidence yet that extreme weight compression has crossed from research curiosity into deployment engineering. The specific achievement is narrow and real: a current-generation 27B reasoning model, compressed after training to 1.76 bits per weight with its architecture and 262K context intact, retaining 98.2% of a 20-benchmark thinking-mode average — and, more tellingly, holding the benchmarks that conventional sub-4-bit quantization provably destroys, at a footprint that fits phones-adjacent memory budgets, everyday laptops, and single consumer GPUs, with throughput and energy measured across twelve platforms and disclosed with unusual candor. The caveats are equally real: the hardest long-horizon agentic workloads retain three-quarters, not ninety-eight percent; knowledge and vision give up the most; the deployment story requires a fork; and every headline number is vendor-run, however auditable.

The right conclusion is not that ternary models have replaced full-precision ones — it is that the deployment decision has become a genuine two-front choice for the first time. A 5.9 GB pack that reasons like the 27B that generated it, at a tenth of the API price and zero network exposure, changes what a phone, a laptop, or a single GPU can be trusted to do alone; a frontier model kept in reserve for the hardest steps changes what the cloud is for. Prism’s own closing frame is the one this paper’s evidence supports: “Raw capability determines what a model can do; density determines where it can do it.” Six months and five releases in, the where has moved further than most observers expected — and the 2.23 million downloads in four days say the deployment world noticed.

Sources

Document-dates note: this paper was assembled September 21, 2026, four days after the Bonsai 2 27B release; adoption figures, OpenRouter telemetry, HN scores, and upstreaming status are current to that date and will age. Citation of Prism ML’s own materials follows the company’s requested format: @techreport{bonsai2_27b, title={Bonsai 2 27B: A 27B Ternary Reasoning Model}, author={Prism ML}, year={2026}, month={September}}.