NVIDIA Nemotron 3.5 Bolt 74B-A9B carries “24GB” in its name, but that figure describes a specific text-only memory estimate. NVIDIA’s accelerated GGUF instructions also offer recipes labelled 28GB and 32GB. Choosing hardware requires matching the model, runtime settings and optional components to the workload you intend to run.
The useful distinction is between an architecture designed around a budget and a complete serving configuration. A download size, a theoretical allocation and a measured peak on a particular GPU answer different questions. NVIDIA has published enough detail to separate them; none alone establishes how much context your machine can serve reliably.
What the 24GB estimate includes
In NVIDIA’s technical report, Table 2 and Section 8.2, the estimated text-only total is 23.83GB: 22.01GB of weights, roughly 0.87GB of runtime state and activations, and a 0.96GB speculative decoder. The assumed workload is one sequence, 131,072-token context and an 8,192-token Mamba prefill chunk. The separately listed 1.68GB vision encoder and projector are excluded.
That leaves only 0.17GB within the stated target. Adding the report’s vision component yields 25.51GB by arithmetic, before any extra overhead outside its accounting. This is a calculation from NVIDIA’s estimates, not an observed allocation or a guarantee of GPU fit. The report also rounds component totals, so hundredths of a gigabyte should not be treated as precision measurements.
“74B at two bits” is an incomplete sizing shortcut. NVIDIA’s release explanation describes mixed precision: expert weights use IQ2_XXS, while other components retain higher precision. The nine billion active parameters describe per-token computation; they do not mean the remaining experts disappear from storage.
The GGUF recipes choose different operating points
The official GGUF model card pairs the model with a vision projector and LiLiCorr speculative decoder. Its configuration choices include:
| Recipe label | Context / KV cache | Listed hardware |
|---|---|---|
| 28GB | 65,536 tokens; Q8_0 keys and values | RTX Spark 64GB or RTX discrete GPU 32GB+ |
| 32GB | 262,144 tokens | RTX Spark 128GB or RTX discrete GPU 48GB+ |
Both recipes use one parallel sequence. They also differ in batching and speculative draft length. These are recipe labels and hardware recommendations, rather than independent measurements of exact peak VRAM. A smaller context reduces cache demand; omitting vision changes the component set; disabling speculation changes the speed/memory trade-off. Those changes should be recorded separately when diagnosing a failed allocation.
Evaluate the completed interaction
NVIDIA reports 83.99 tokens per second for Bolt with LiLiCorr in its controlled SPEED-Bench-Coding comparison. The same release reports AA-AgentPerf Local replay durations of 600.80 seconds for Bolt and 453.97 seconds for Qwen3.6-35B-A3B with MTP. Measurements used llama.cpp on Windows 11 ARM x64 and an RTX Spark Surface Laptop Ultra with 128GB.
Bolt’s replay took 32.3% longer, calculated as 600.80 ÷ 453.97 − 1. That comparison does not establish task success, output quality or performance on another GPU. It does show why a decoding rate should not stand in for the time taken by a full agent interaction: processing incoming context also consumes time.
For a local deployment decision, first record the exact checkpoint, runtime build, context limit, cache precision and auxiliary models. Then measure peak allocated memory during the longest representative prompt and response. For an assistant, include tool results and accumulated conversation history in that prompt. Compare total elapsed time only after checking whether the answer met the task’s acceptance criteria.
That proposed evaluation has not been run by EyesTech. A published trace with those settings and a measured peak could establish a workable 24GB configuration. Until then, use NVIDIA’s larger documented recipes as the reproducible starting point and treat the report’s 23.83GB figure as its explicitly scoped estimate.
Publisher information: Prepared with AI assistance from the primary documentation linked above. Calculations are identified separately from developer measurements. Featured image: original AI-generated conceptual illustration.
