DiffusionGemma-Jev (djev) can now be deployed as a Jev API-compatible inference endpoint on Google Cloud Run using a single gcloud command — no dedicated GPU server required. The setup delivers ~35–60 ms single-step latency and batch@32 throughput of ~100–123 req/sec on an NVIDIA L4, running at roughly $3/hr under load and dropping to $0 when idle.
What changed?
The djev-run open-source project packages DiffusionGemma’s vLLM backend into a Cloud Run-ready container. One gcloud run deploy invocation pulls, builds, and serves a fully Jev-compatible REST endpoint — no Kubernetes, no persistent GPU reservation.
| Metric | Value |
|---|---|
| Single-step latency | ~35–60 ms |
| Throughput (batch@32) | ~100–123 req/sec |
| Active cost (NVIDIA L4) | ~$3/hr |
| Idle cost (scale-to-zero) | $0 |
Why DiffusionGemma Needs a Different Serving Stack
Standard autoregressive models generate text one token at a time — each forward pass is serialized through a memory-bandwidth bottleneck. DiffusionGemma breaks from that entirely. It operates on a fixed 256-token canvas, denoising all positions simultaneously across multiple iterative passes. The compute graph is shaped by diffusion steps, not sequence length, which fundamentally changes the performance profile at the serving layer.
The payoff: latency no longer scales linearly with output length for short-to-medium responses. A 60-token reply and a 240-token reply cost nearly the same wall-clock time, because the model denoises the entire canvas in parallel regardless. This property is what makes DiffusionGemma structurally well-suited to the Jev use case — Jev’s design contract calls for structured, single-pass decisions, exactly the regime where a fixed diffusion canvas incurs no marginal penalty.
What djev-run Actually Does
The djev-run repository by taeold does three things in one shot: it packages a vLLM server configured for DiffusionGemma’s diffusion sampler, wires a Jev-compatible REST interface on top, and wraps the entire thing into a Cloud Run deployment manifest. The deploy command triggers Cloud Build to build the container image, push it to Artifact Registry, and provision the Cloud Run service in a single pipeline — no intermediate steps, no manual image tagging.
Under the hood, vLLM’s diffusion sampler configuration is critical. DiffusionGemma’s denoising state per request is substantially larger than a standard KV cache entry — it holds the full canvas (length × vocab size) in memory simultaneously. This means the standard vLLM concurrency defaults will cause OOM on an L4. djev-run ships with conservative --max-num-seqs limits and appropriate --gpu-memory-utilization headroom baked in, so the deployment doesn’t require manual tuning to stay stable. For a deeper look at how quantization interacts with memory pressure at this scale, see our breakdown of NVFP4 vs FP8 on Blackwell.
Performance Numbers: What 35–60 ms Actually Measures
The reported 35–60 ms single-step latency refers to the time for one complete diffusion denoising step across the 256-token canvas — not time-to-first-token in the autoregressive sense, since that concept doesn’t directly apply here. A complete response may involve multiple denoising steps before the canvas converges, so end-to-end response latency will be a small integer multiple of that per-step figure depending on the configured step count.
| Workload | Metric | Result |
|---|---|---|
| Single request | Per-step latency | 35–60 ms |
| Batch size 32 | Throughput | 100–123 req/sec |
| GPU: NVIDIA L4 | Active billing rate | ~$3/hr |
Note: Single-step latency measures one denoising pass across the 256-token canvas. Complete response latency scales with the number of configured diffusion steps. The L4 is the current GPU type available on Cloud Run standard configurations.
The batch@32 throughput figure of 100–123 req/sec is the more operationally significant number for API serving. At that rate, a single L4 instance can comfortably handle burst traffic from developer integrations and prototyping workflows without queuing penalties — the target audience for this deployment template. Compare this against ternary-quantized local inference at 143 tok/sec, which operates under a completely different memory constraint profile.
The Economics: $3/hr Active, $0 Idle
Cloud Run’s billing model is the architectural reason this deployment pattern makes sense for experimentation. Unlike a dedicated Compute Engine instance with an attached GPU — which runs continuously whether or not a request is in flight — Cloud Run GPU instances scale to zero on inactivity. You pay only for wall-clock time your container is alive and processing. For a developer running sporadic experiments, five hours of active testing at ~$3/hr costs $15 total. A dedicated GPU VM for the same experimentation sprint costs $72+ regardless of utilization.
Scale-to-zero means cold starts. When traffic arrives after an idle period, Cloud Run must spin up a new instance, pull the container, load the model weights into GPU VRAM, and initialize the vLLM server — a process that can take 30–90 seconds for large model containers. For latency-sensitive production workloads, configure min-instances=1 to keep one warm instance alive (at the cost of ~$3/hr continuous billing).
The trade-off is clean: min-instances=0 for cost-zero idle (cold starts accepted), or min-instances=1 for always-warm serving at continuous GPU cost. For experimentation without latency SLAs, the default zero configuration is the obvious choice. This serverless economics model is part of a broader shift in how developers access inference compute in 2026 and beyond.
Deployment: The Single-Command Path
The deploy command follows the standard Cloud Run source-based deployment pattern. After cloning the djev-run repository, the invocation is:
gcloud run deploy djev-service \
--source . \
--region us-central1 \
--gpu 1 \
--gpu-type nvidia-l4 \
--memory 24Gi \
--cpu 8 \
--no-allow-unauthenticated \
--no-cpu-throttling \
--timeout=600 \
--min-instances=0Cloud Build handles image construction and pushes to Artifact Registry automatically. The resulting endpoint speaks the Jev API, meaning any client already pointed at a Jev-compatible inference service can switch over with a URL change and no code modifications. This ties directly into how Google’s managed agent harnesses route inference calls — djev endpoints slot into that ecosystem without custom adapters.
Where This Fits — and Where It Doesn’t
djev-run is explicitly an experimentation template. The L4 is a competent inference GPU, but it sits below the performance tier that high-QPS production APIs demand. At very high concurrency, DiffusionGemma’s large per-request memory footprint (the full diffusion state buffer per canvas) means you hit memory ceilings before compute ceilings — the opposite constraint from standard autoregressive models where KV cache growth is the limiting factor. The same memory wall dynamics appear in Gemini 4’s TPU 8t serving architecture, where activation memory per sequence is the binding constraint at scale.
| Scenario | Verdict |
|---|---|
| Research experimentation, prompt engineering | Excellent fit |
| Low-to-medium QPS developer APIs (<50 concurrent) | Good fit |
| Jev-compatible structured decision endpoints | Designed for this |
| High-concurrency production inference (>200 req/sec) | Not recommended |
| Fine-tuning or training runs | Wrong tool |
For teams that want to prototype Jev-style structured reasoning pipelines without standing up dedicated GPU infrastructure, this is a practical entry point. The $0 idle cost means you can leave the service deployed between sessions without accumulating charges — something impossible with reserved GPU instances where the meter runs continuously.
The Source
The deployment code, Dockerfile, and configuration reference are available at github.com/taeold/djev-run. The original deployment report was shared by @dylayed:
