Ornith 1.5 122B Local Setup: GGUF and Memory Requirements

Ornith 1.5 122B is an open-weight mixture-of-experts model with official GGUF, FP8 and MLX distributions. The Q4_K_M GGUF file is 75.84 GB, or 70.64 GiB, before runtime allocations. Its BF16 tensor index lists 250.17 GB of weights. Choosing a format is therefore the first hardware decision.

The model card describes single-80GB-GPU serving, but its unquantized checkpoint does not fit entirely in that capacity. That claim needs a specific quantization or offload configuration to become a useful deployment recipe. It should not be read as a memory requirement for every download carrying the same model name.

Match the memory estimate to the actual artifact

The BF16 tensor index records 250,172,994,016 bytes, approximately 233.0 GiB. This is tensor storage, before cache, workspaces and the serving process. The model’s sparse expert selection does not make the full checkpoint disappear from memory.

The following figures were calculated from official repository file metadata checked October 21. GB uses one billion bytes; GiB uses 1,073,741,824 bytes. These are artifact sizes, not measured peak VRAM.

The GGUF figures come from the official GGUF files. The FP8 and MLX figures sum the published Safetensors shards in their respective FP8 and MLX 4-bit repositories. Tokenizers and other small configuration files are excluded.

An “FP8” label is insufficient for estimating the entire checkpoint at one byte per parameter. The FP8 configuration excludes several component groups from quantization, including visual, linear-attention and MTP modules. Its shard total is the more concrete download budget.

For Q4_K_M, an 80 GiB memory budget minus 70.64 GiB of model-file storage leaves 9.36 GiB before runtime costs. That subtraction is a planning illustration, not proof of a successful load. A 64 GiB machine cannot hold that entire file in memory while also accommodating its operating system and inference allocations. CPU offload or a different quantization changes the arrangement and needs its own performance check.

Begin with a short text context

The model card lists recent Transformers, vLLM and SGLang requirements, and separates reasoning parsing from tool-call parsing. Those paths require the runtime to understand the checkpoint’s architecture and formats. A familiar OpenAI-compatible URL does not supply that backend support by itself.

For a GGUF installation, use a current llama.cpp server. Once installed, a conservative text-only starting command is:

llama-server \
  -hf ornith-ai/Ornith-1.5-122B-GGUF:Q4_K_M \
  --host 127.0.0.1 \
  --port 8080 \
  --no-mmproj \
  -c 8192

This is a setup example, not a command EyesTech has benchmarked. Backend defaults determine device placement; inspect startup logs for the selected file, offloaded layers and actual allocations. The context has deliberately been reduced from the advertised maximum to make initial diagnosis simpler.

If the server starts, check the model listing before connecting a coding client:

curl http://127.0.0.1:8080/v1/models

Use the identifier returned by that server in subsequent requests. Confirm a short response first, then evaluate tool calls separately. Valid text generation does not demonstrate that a client’s tool schema, reasoning fields and streaming behavior all work as expected.

Context and vision need separate budgets

The checkpoint configuration sets 262,144 maximum positions and includes both linear- and full-attention layers. A context setting describes a capacity; the runtime still needs memory for its implementation of attention, recurrent state and temporary work.

Increase context only after recording a stable baseline. Keep prompt length, generated tokens, backend version and memory usage with each measurement. Report prefill and generation separately so a long document’s reading time is visible.

The official GGUF repository also contains a separate 0.912 GB BF16 vision projector. The command above explicitly disables the auto-downloaded projector with --no-mmproj. Remove that restriction only when configuring vision support. Follow the runtime’s multimodal instructions and include the projector and image-processing allocations when budgeting an image workload.

For a Mac, select a supported MLX runtime when using the MLX distribution. Its roughly 63.99 GiB shard total does not imply comfortable operation on a 64 GiB unified-memory system: the OS and active inference need space too. Artifact size should be the lower bound of a memory plan, rather than its final requirement.

Ornith’s performance claims remain vendor results until reproduced under a specified setup. The immediate decision is simpler: choose the exact artifact your machine can accommodate, establish a short-context run and expand only from recorded behavior. That produces a reviewable hardware answer without treating “122B” or “single GPU” as a complete specification.