How to Run Bonsai 2 9B: GGUF Formats and Memory

Bonsai 2 9B’s default PQ2_0 GGUF download is about 2.42GB. It needs PrismML’s compatible llama.cpp fork; the alternative Q2_0 file works with a recent upstream build containing Bonsai 2 support. Adding the recommended Q8 vision projector brings the default model’s combined downloads to approximately 3.04GB, before runtime memory.

The first installation decision is therefore which runtime you will use. A file ending in GGUF does not establish that an application can execute its tensor types and model transformations. Downloading the smallest file into an older bundled engine can leave you with a model that fails to load or produces incorrect output.

Choose the packing before downloading

PrismML’s backend support documentation distinguishes three packings of the same model weights. PTQ1_0 packs ternary values densely; PQ2_0 trades a larger file for execution convenience; Q2_0 uses an upstream-supported representation. Bonsai 2 also needs the matching Hadamard transforms and model integration. Recognizing the quantization type alone is insufficient.

The official repository’s file inventory, checked on October 17, gives these rounded decimal download sizes:

The PQ2_0 file contains 2,416,851,840 bytes. Its Q8 projector adds 624,229,664 bytes: 3,041,081,504 bytes, or 3.041GB, combined. This is storage arithmetic. Loading an image also requires execution buffers and image tokens; context caches and backend allocations add memory beyond these files. Reserve space for the runtime and operating system when choosing a machine.

Use the 9B launcher settings

The official demo defaults to 27B unless a model size is selected. For macOS or Linux, its documented selection mechanism can be applied to 9B as follows:

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo
export BONSAI_MODEL=9B
BONSAI_OPENWEBUI=0 BONSAI_CODE_INTERPRETER=0 ./setup.sh
BONSAI_CTX=8192 ./scripts/start_llama_server.sh -np 1

Open http://localhost:8080 after the server starts. The two setup flags skip optional Open WebUI and code-interpreter dependencies. The 8,192-token context is a conservative starting configuration, not a measured hardware minimum. Check the startup log for the selected model, backend and effective context before increasing it.

The demo pins release prism-b10777-e3de715 at the time of writing and downloads matching fork binaries. Its small-model instructions supply separate sampling settings for 2B, 4B and 9B. Use those profiles when moving to an API client: changing thinking mode alone does not automatically update every sampling parameter. The standalone chat/API route is the documented starting point; the optional agentic launcher still has small-model limitations.

These commands adapt the project’s documented setup; EyesTech has not run a local inference benchmark for this article. If setup fails, retain the model filename and build identifier in the error report. Switching an application interface without updating its embedded inference engine will not add missing format support.

Add vision and acceleration deliberately

The 9B model card recommends the Q8 vision projector and documents a crash affecting its BF16 alternative in the current fork’s vision-serving path. Its optional MTP checkpoint already contains the PQ2 target weights and draft head; it replaces the plain target file in that route. It is not another full target model to load alongside it.

Start with the base model before enabling speculative decoding. For a document workflow, compare extracted numbers against a fixed answer sheet, include at least one deliberately missing field, and record both first-response latency and total time to a correct result. This is a proposed application check, rather than a reported test.

PrismML’s launch evaluation shows why the task matters: 9B scores 98.60 against its full-precision baseline’s 99.00 on MATH500, but 73.02 against 85.25 on Tau2Bench. Those are developer-reported benchmark results, with different retention across workloads. They support trying the compressed model against your own acceptance criteria; they do not establish universal equivalence to the full-precision model.