Bonsai 2 9B’s default GGUF is 2.42GB; adding its Q8 vision projector brings downloads to 3.04GB. Match PQ2, PTQ or Q2 files to the correct llama.cpp build.
Local LLMs & Homelabs
30 Articles in this Category
Running open-weights without cloud APIs: VRAM sizing, Ollama, vLLM, GGUF/EXL2 quantization, and Apple Silicon unified memory.
NVIDIA estimates 23.83GB for text-only Nemotron 3.5 Bolt, while its GGUF recipes use larger budgets. Check vision, context and draft-model requirements.
Liquid AI claims 184 tok/s on Apple M5 Max with 2.45B active compute, but LFM2.5-24B requires 15GB RAM, crushing 16GB laptops and lagging on SWE-bench.
Timescale pg_vectorsearch and PRISM replace HNSW graphs with RaBitQ-quantized partitions, serving 100M vectors at 21ms on 128GB RAM with 51x faster builds.
d1-omni-600M returns typed decisions without generated text. Image and audio scores need separate validation, with 30-second speech and input limits.
MiniCPM-V 4.7 1B needs a 672 MB language file plus a 1.11 GB vision projector. Get the Ollama tag, llama.cpp recipe and visual-compression settings.
pg-jev evaluates Postgres rows through a decision API. Batches of 20 and session caches change cost; LIMIT 3 does not guarantee three billed rows.
MiniCPM-V 4.7 reports 13+ tokens/s in 3GB phone memory. Expert loading, quantization and speculation explain why that is not its total weight size.
Muse Wonder costs $4/$20 per million input/output tokens. Full requires a Muse Code plan; max reasoning changes effort, not the model you access.
NVIDIA Research’s Low-Rank Gradient Sketches (LoGRA) algorithm reduces reinforcement learning optimizer VRAM consumption by 45.7% on 7-billion-parameter…