Needle 3: Inside Cactus Compute’s 8MB On-Device Model

For three years, the edge artificial intelligence ecosystem operated under a flawed assumption: that the path to ubiquitous local intelligence lay in quantizing generative chatbots until they squeezed into mobile DRAM.

Engineering teams spent thousands of compute hours crushing 3-billion to 8-billion parameter decoder models into 4-bit and 3-bit GGUF binaries. Mobile developers watched devices burn through battery reserves and throttle thermal envelopes just to evaluate whether an incoming push notification required setting a calendar reminder. Worse, when these compressed conversational models were asked to format outputs into strictly typed JSON function calls, they routinely dropped closing brackets, hallucinated extraneous conversational preamble, or invented API parameters that did not exist.

The release of Needle 3 by Cactus Compute marks the formal abandonment of that approach.

Needle 3 is not a conversational assistant. It possesses no chat persona, cannot draft marketing emails, and outright refuses open-ended dialogue. Instead, Cactus Compute designed an open-weight foundation model dedicated entirely to three programmatic primitives: tool calling, structured schema extraction, and semantic vector embedding.

Shipping in binaries ranging from 8 MB to 29 MB compressed via 2-bit Cactus Quants (CQ2), Needle 3 runs locally across edge silicon—from ARM Cortex-M microcontrollers and Raspberry Pi 5 boards to Apple Silicon Neural Engines and Qualcomm Snapdragon NPUs—achieving execution speeds up to 4,000 tokens per second with sub-15-millisecond latency. Released under the Apache 2.0 open-source license, the model weights, configs, and C++ inference engine are publicly accessible on the Cactus Compute Needle repository.

Architectural Breakdown: The Laddered Simple Attention Network

Traditional neural architectures enforce a static topology: a 7B model requires loading all 32 layers into memory for every forward pass. Early exit heuristics or speculative early termination approaches degrade representations because intermediate layers were not trained to operate as terminal classification heads.

Needle 3 resolves this via the Laddered Simple Attention Network (SAN).

In a Laddered SAN, a single pre-trained checkpoint contains 20 layers. During deployment, the compiler or runtime loader slices the model tensor at layer k (where 2 ≤ k ≤ 20). Every layer index k possesses native projection weights directly into the vocabulary and parameter space. Multi-lane hyper-connections route intermediate representations around deeper blocks, ensuring that the 4-layer variant behaves as a fully converged, standalone subnetwork rather than a truncated model missing contextual depth.

Monarch Hadamard MLPs: Eliminating Dense Matrix Multiplications

In conventional transformer blocks, dense Feed-Forward Networks (FFNs) account for roughly 65% of the total parameter count and floating-point operations (FLOPs). A standard FFN projects an embedding dimension d to an expanded intermediate state 4d through dense weight matrices:

Needle 3 completely replaces dense FFNs with Monarch Hadamard MLPs. Rooted in the foundational Monarch matrices research by Dao et al., Monarch matrices factorize large linear operators into products of block-diagonal and permuted sub-matrices, capturing full expressive capacity while slashing parameter counts and compute operations from quadratic to sub-quadratic scaling:

The Engram Memory: Decoupling Arithmetic from Static Facts

In standard LLMs, world knowledge (e.g., API syntax, parameter names, date formatting conventions) is burned directly into floating-point weights within dense layers. Every time the model recalls a static syntax rule, it must execute billions of Multiply-Accumulate (MAC) cycles.

Needle 3 isolates static knowledge into an Engram N-Gram Memory system. The architecture maintains a hash-addressed table of n-gram embeddings. During token processing, input sequences generate hash indices into the engram table:

1. The model performs hardware-accelerated memory gather operations across the engram tables rather than floating-point matrix multiplications.
2. The retrieved static memory vectors are fused into the transformer hidden state via causal convolution taps and Grouped-Query Attention (GQA).
3. Because the heavy parameter mass is stored in compact hash tables rather than active compute layers, the 121M-parameter 20-layer model executes with the arithmetic workload of an ordinary 50M-parameter model.

Unlike server-bound autoregressive deployments requiring dedicated container warm-ups (such as our architecture for DiffusionGemma-Jev on Cloud Run), Needle 3 decouples static memory lookup from dynamic tensor math, permitting sub-15ms cold inference directly within bare-metal microcontroller memory budgets.

Cactus Quants (CQ2): 2.125-Bit Parameter Packing

To achieve an 8MB to 29MB footprint, Cactus Compute developed CQ2 (Cactus Quants). Standard 4-bit quantization (INT4 or FP4) leaves weights at 4.0 bits per parameter. Needle 3 compresses weight blocks down to an average of 2.125 bits per parameter through ternary-inspired sign-magnitude packing and structured codebooks:

• Weight distributions are quantized into non-uniform discrete centroids: {-1, -0.25, 0, +0.25, +1}.
• Non-zero values are encoded using 2-bit symbols, with a shared block-level scale factor stored every 32 elements.
• Dequantization is implemented via bit-shift and table lookup instructions that map directly into ARM NEON and Apple Silicon AMX vector registers, avoiding floating-point division at runtime.

Technical Specifications: Needle 3 vs. Competing Edge Engines

The following matrix compares Needle 3 across its operational configurations against leading mobile and on-device models.

Deterministic Execution: Byte-Level Grammars and Calibration

The critical barrier preventing enterprises from deploying language models directly to device hardware has never been parameter size alone; it has been non-deterministic output corruptions.

A general-purpose LLM prompted to return JSON can produce syntactically invalid strings when it encounters out-of-distribution input. In production, this causes unhandled exceptions in client code or triggers endless retry loops that drain device power.

Needle 3 attacks this vulnerability through two mechanical layers: Byte-Level Grammar Compilation and Mathematically Calibrated Confidence Scoring.

Byte-Level Grammar Masks and Finite State Machines

When an application invokes Needle 3 with a target function schema or JSON data contract, the runtime compiles the schema into a deterministic Finite State Machine (FSM).

At each decoding step: – The FSM calculates the exact set of valid UTF-8 byte continuations that conform to the target schema. – Invalid token logits are masked out with negative infinity (-inf) before the softmax projection. – If the schema specifies an integer field, token candidates containing alphabet characters, quotation marks, or syntax symbols receive zero probability.

Because the grammar operates at the byte level within the decoding loop, Needle 3 outputs are mathematically guaranteed to parse on the first attempt. Syntax errors are structurally impossible.

Calibrated Confidence Scoring and Failure Suppression

Standard language models are notoriously uncalibrated: cross-entropy loss trains models to be confident even when guessing. A 1B model might output a hallucinatory tool invocation with 0.98 softmax certainty.

Needle 3 outputs an explicit, calibrated confidence score c ∈ [0, 1] alongside every generated invocation. When the model outputs a confidence score of 0.85, empirical execution yields exactly 85% real-world correctness.

Crucially, when user input does not match any registered function schema, Needle 3 outputs an empty array [] with high confidence. It does not fabricate imaginary API endpoints or default to generic fallback hallucinations.

Forensic Critique: Architectural Trade-Offs and Boundary Conditions

While Needle 3 establishes a new efficiency frontier for edge devices, systems architects must evaluate the hard mechanical boundaries inherent to sub-100M parameter networks.

General Reasoning Absence and Symbolic Constraints

Needle 3 is completely incapable of multi-step logical deduction or semantic synthesis. If a tool call requires deducing implicit arithmetic across disparate paragraphs (e.g., “Calculate the net profit margin after tax from this quarterly filing and pass it to the reporting tool”), Needle 3 will either miscalculate the slot value or refuse the invocation.

Similar to the reflexive routing layers analyzed in Laya’s open-source System 1 decision engine, Needle 3 functions as an instinctive dispatcher, not a reflective thinker. Complex workflows require a hierarchical architecture: Needle 3 serves as the local real-time gatekeeper, routing complex reasoning anomalies to cloud frontier models (such as the frontier reasoning chains analyzed in our OpenAI secret RL freeze report) only when local confidence drops below calibrated safety margins.

Schema Saturation and Attention Collisions

Because Needle 3 compresses its representation space into Monarch factorized layers and an Engram hash table, presenting the model with more than 30 concurrent tool schemas causes vocabulary collision in the 2-layer and 4-layer configurations.

When tool schemas share overlapping argument names (e.g., three separate tools requesting target_id: string), the shallow layers exhibit attention confusion. Production systems must implement dynamic schema pruning: client code should pre-filter candidate tools using lightweight vector embeddings before feeding the top 5 to 10 schemas into Needle 3.

Fine-Tuning Elasticity and the DroidCall Benchmark

Cactus Compute’s empirical benchmarks reveal that base weights achieve 61% to 86% tool-calling precision out-of-the-box. However, fine-tuning on domain-specific datasets (such as DroidCall, a corpus of 12,000 mobile automation traces) produces non-linear accuracy jumps of 18 to 36 percentage points.

This characteristic indicates that while the base representation understands functional decomposition, the lightweight weights benefit disproportionately from task-specific alignment. In contrast to heavy browser agent loops (such as agentic web workflows with Kimi browser extension skills), on-device UI automation requires deterministic, low-latency token generation where every millisecond counts. For enterprise deployments, Needle 3 should be treated as a high-efficiency foundation template to be fine-tuned on company-specific API schemas rather than deployed purely on zero-shot inference.

Industry Architectural Bifurcation: Cloud Reasoning vs. Edge Instinct

The arrival of models like Needle 3, alongside TypeSafe’s Jev System 1 model and PrismML’s Bonsai-2, marks the definitive structural bifurcation of modern AI architecture:

For three years, the industry attempted to force cloud-scale conversational models onto battery-powered edge hardware. Needle 3 proves that on-device intelligence does not require multi-gigabyte models or generative chatter. By engineering for sub-quadratic arithmetic, grammar-constrained decoding, and dynamic ladder scaling, Cactus Compute has provided the missing execution engine for autonomous edge systems.

Frequently Asked Questions

What is Needle 3?

Needle 3 is an open-weight, on-device foundation model developed by Cactus Compute, released under the Apache 2.0 license. Ranging in size from 8 MB to 29 MB, it is specifically engineered for programmatic edge tasks—including tool calling, structured JSON extraction, and semantic embeddings—rather than open-ended conversational chat.

How does the Needle 3 Intelligence Ladder work?

The Intelligence Ladder allows a single set of trained weights (up to 20 layers and 121 million parameters) to be sliced dynamically at any depth between 2 and 20 layers. A developer can deploy a 2-layer subnetwork (29M parameters, 8 MB) onto microcontrollers or scale up to a 20-layer model for complex mobile routing, eliminating the need to train and maintain separate model checkpoints for different hardware tiers.

Why does Needle 3 refuse general conversational chat?

Generative conversational capabilities require vast associative memory and parameters that bloat model footprint and drive up latency. By eliminating chat, poetry, and open-ended text generation, Needle 3 focuses entirely on exact schema alignment, syntax validity, and deterministic parameter extraction, outperforming models ten times its size on tool-calling benchmarks.

How does Needle 3 prevent syntax hallucinations in JSON output?

Needle 3 incorporates a byte-level grammar compiler that maps target function schemas and Pydantic models into a finite state machine. During inference, invalid token continuations are masked with negative infinity before softmax calculation, making syntax violations structurally impossible and guaranteeing that all outputs parse cleanly on the first pass.

What are Monarch Hadamard MLPs?

Monarch Hadamard MLPs replace the traditional dense feed-forward networks (FFNs) found in transformers. By factorizing dense matrices into products of block-diagonal and permutation matrices combined with Hadamard tensor mixing, compute scaling drops from quadratic O(8d²) to sub-quadratic O(n√n), drastically lowering arithmetic requirements without sacrificing functional accuracy.