How Nunba Runs a 4B LLM on 8GB RAM
Speculative decoding + tiny draft model + quantized main model = local AI that's actually responsive on the laptop you already own. Here is exactly how the pipeline fits.
How Nunba Runs a 4B LLM on 8GB RAM
How Nunba Runs a 4B LLM on 8GB RAM: Speculative Decoding Explained
May 23, 2026 by Hevolve AI Team
If you've tried running a 4B-parameter language model on your laptop, you've probably hit the wall everyone hits: it's too slow to be useful. The first token takes seconds. Long replies feel like watching a turtle type. So most people give up and go back to ChatGPT.
We didn't want to give up. So we built Nunba around a trick called speculative decoding, paired with two cooperating models. Here's how it works, and how we got from "barely usable on 16GB" to "snappy on 8GB."
The problem: token-by-token generation is sequential
A language model generates one token at a time. Each token depends on every token that came before. On a CPU or a modest GPU, each step takes milliseconds, and a 200-token reply means 200 sequential GPU passes. There's no parallelism to exploit. Throughput is capped by latency per step.
You can throw a bigger GPU at it. Or quantize the model to 4-bit. Or use a smaller model. All of those work, but each has a cost: more money, lower quality, or both.
Speculative decoding: two models, one prompt
The idea is simple: use a tiny fast model (the "draft") to guess the next 4-8 tokens, then have the big accurate model (the "main") verify all of those guesses in a single forward pass. When the draft is right, you get 4-8 tokens for the price of one main-model step. When it's wrong, you fall back to the standard token-by-token path, no quality loss, just no speedup for that step.
On real chat data the draft is right about 60-70% of the time on common patterns. The end result on Nunba: first-token latency drops from ~2.5s to ~700ms, and sustained throughput roughly doubles compared to running the main model alone.
Two paths, and which one you get
Which of these runs is decided by the VRAM manager at load time, from what your machine actually has.
With a 10GB+ CUDA card you get the pair below and speculative decoding between them. That is the configuration this post measures.
Without one you get a single compact main model, 0.8B or 2B class, doing the answering by itself on CPU. No draft, no speculation. Between 4 and 10GB of VRAM the GPU runs the main model alone, same as CPU but faster. The main slot takes any GGUF, so swapping the model is configuration rather than a rebuild.
That second path is not a lesser product, which is easy to miss. The agent runs the same either way, and a question the small local model should not take can be handed whole to a peer on your Hive running something bigger. Nothing is sharded across the network and no layer-level parallelism is attempted over consumer links: one machine answers one request. So the compact model is a floor, not a ceiling.
The rest of this post is about the speculative path, because that is where the interesting engineering is.
Nunba ships two GGUF models for it:
- Main: Qwen3-4B-Instruct (Q4_K_M quantization). 4 billion parameters, sub-2GB on disk, ~3.2GB resident at inference. Handles the actual reasoning.
- Draft: Qwen3-0.8B-Instruct (Q4_K_M). 800 million parameters, ~500MB on disk, ~700MB resident. Same tokenizer family as the main model, that's critical, because mismatched tokenizers break speculative decoding entirely.
Both run on llama.cpp's llama-server. Main on port 8082, draft on 8081. Nunba's dispatcher sends every chat turn to both and merges the results.
The memory budget
The full active set at chat time on the speculative path:
- Main model: ~3.2GB
- Draft model: ~700MB
- llama-server process overhead: ~600MB
- TTS model (Piper CPU fallback): ~80MB
- Python + Flask + the rest of Nunba: ~1.5GB
That totals ~6.1GB resident. With Windows and a browser open on an 8GB machine you have ~1.9GB headroom, which is tight but workable. On 16GB it is comfortable, and we can also keep MiniCPM-V vision (~2.5GB) loaded for image input and a more expressive TTS like Kokoro (~1.5GB) for human-sounding voice.
A machine without the VRAM for the pair carries a smaller set, since it is running one compact model instead of two.
What we cut to make this fit
There's no free lunch. To run on 8GB we made deliberate trade-offs:
- No simultaneous vision model on 8GB. MiniCPM-V loads on-demand when you actually share an image, then unloads.
- Piper CPU TTS as the default fallback. Beautiful TTS engines (F5, Kokoro, Indic Parler) need GPU + extra RAM. We ship Piper as a tiny CPU-only safety net; the others auto-load when there's headroom.
- Smaller context window when slots are contended. We dynamically adjust
n_ctxper concurrent request so two chat sessions don't blow the budget. - Smart eviction. The VRAM manager (
tts/vram_manager.py) tracks every loaded model and evicts the least-recently-used when something new needs to load. No manualnvidia-smirequired.
The result
On the speculative path first-token latency lands around 300ms and sustained throughput near 35 tokens/sec, which is competitive with cloud inference and keeps your data on your machine. On the CPU path, one compact model, it is slower and still responsive enough to hold a conversation.
The point isn't that we beat cloud inference on raw speed, we don't. The point is that good enough for chat, on hardware you already own, costing zero per query, with full privacy is a fundamentally different product than the cloud one. Speculative decoding is what makes the GPU path feel like a hosted API. Running at all on a machine with no CUDA at all is what makes the rest of it matter.
Try it
The full pipeline ships in Nunba, no setup beyond running the installer. The first launch downloads both models (~5GB total) and pre-warms them. After that, every chat turn runs through the dispatcher you just read about.
The source is on GitHub. The model files are pulled from official Qwen releases on Hugging Face on first launch. Everything verifiable, everything local.