Abstract & How to Read This Paper

What this document is, who it is for, and how it is organized.

This paper answers one practical question: how do you extract the maximum agentic capability from a consumer PC with a single 8 GB GPU and 32 GB of system RAM? The question matters because the marketing of local AI is dominated by hardware most people do not own. Guides casually assume 24 GB RTX 3090s or Apple Silicon machines with 64–128 GB of unified memory, while the single most common serious-gaming-and-tinkering configuration in the world — an RTX 3060 Ti / 3070 / 4060 / 4060 Ti class card paired with 32 GB of DDR4 or DDR5 — sits in a strange middle ground: too much GPU to give up on acceleration, too little VRAM to hold the models everyone benchmarks.

The central thesis of this paper is that the 8 GB + 32 GB configuration is not a compromise to be apologized for but a systems engineering problem with a well-mapped solution space. The solution space has four dimensions, and each is covered in depth here with real experimental data: (1) weight compression, where the evidence shows modern 4-bit quantization costs between 0.15 and 3 percentage points of benchmark accuracy on an 8 B model — the famous perplexity ladder for Llama 3 8B moves only from 6.233 (fp16) to 6.383 (Q4_K_M), a 2.4 % relative change, while cutting file size by 69 %; (2) context memory, where the KV cache follows an exact arithmetic formula (Section 5) and can be quantized to 8-bit with near-lossless results (81.6 % output similarity, −208 MB at 8 K context) but collapses catastrophically at 4-bit (8.3 % similarity — the “q4_0 cliff”); (3) placement, where llama.cpp-style layer splitting turns VRAM + RAM into one addressable pool, with the practical consequence that a 14 B model runs at roughly 60–75 % of full-GPU speed and a 30 B-class MoE model with 3 B active parameters remains genuinely interactive; and (4) architecture, where the agent’s context management — not the model — is usually the binding constraint on task complexity.

The paper is written to be read once, linearly, and then used as a reference. It is deliberately long, because the goal is competence rather than awareness: after reading it you should be able to size a model to your memory budget by hand, choose a quantization level with knowledge of its measured cost, configure an inference server with the correct flags, design an agent loop that survives hour-long tasks without context exhaustion, and diagnose an out-of-memory failure without guessing. All commands target Linux first (with Windows equivalents where they differ), because the tooling — llama.cpp, Ollama, vLLM — behaves best there, but everything applies to Windows with WSL2.

1. The 8 GB Challenge

What makes low-VRAM inference a distinct engineering regime — and why 2025–2026 was the moment it became viable.

1.1 The problem, stated precisely

A local AI agent stack has to fit four things into memory simultaneously: the model weights (billions of parameters, each taking multiple bytes), the key–value cache that grows with every token of conversation history (Chapter 5), the activation buffers and CUDA context overhead that inference frameworks reserve just to operate, and the agent scaffolding itself — the Python process, the web server, the retrieval index, the tool processes. On a datacenter GPU with 80 GB of HBM, the first three items are rounding errors; you simply load the model in 16-bit and move on. On an 8 GB card, all four compete for the same small pool, and the weights alone of a modern 8 B model in fp16 (16.1 GB) are already twice your budget before a single token has been generated.

The naive conclusion is that small-VRAM machines are limited to small models, and small models are toys. Both halves of that conclusion are now wrong. The first half fails because quantization and offloading have matured into well-understood, well-benchmarked disciplines (Chapters 4 and 6): an 8 B model at 4-bit occupies ~4.9 GB and loses, on measured benchmarks, roughly 1–3 points of MMLU accuracy; a 30 B-class mixture-of-experts model with 3 B active parameters can be made to run interactively by keeping its always-active components in VRAM and streaming expert weights from system RAM. The second half fails because of a decade of small-model progress: today’s 7–9 B models score on standard evaluations at or above the level of GPT-3.5-class frontier systems of 2023, and dedicated agent research has repeatedly shown that scaffolding quality matters as much as raw model size. The mini-swe-agent project demonstrated 65 % resolution on SWE-bench Verified with a deliberately minimal ~100-line agent loop — because it spent its complexity budget on the right abstractions rather than on elaborate frameworks. The lesson transfers directly to low-resource machines: with a mid-sized model, your agent architecture is your capability multiplier.

1.2 Why 8 GB + 32 GB is a genuinely good configuration

It is worth being explicit about why this particular pairing — modest VRAM, generous RAM — is more capable than it looks, because the reasoning drives most of the engineering decisions in this paper.

• The 32 GB of system RAM is a large, underused second pool. llama.cpp’s mmap-based loading can treat system RAM as a transparent extension of VRAM. A model file can exceed VRAM by 3–4× and still run, with the GPU holding the layers that are touched every token and the CPU handling the rest. Combined, you have a 40 GB addressable pool — enough for a 30 B dense model at 4-bit (18.6 GB) or even larger MoE models, just not at full speed.
• Most tokens are not latency-critical. An agent spends its wall-clock life waiting for tool executions — file reads, web fetches, test suites that take seconds to minutes. When the next model call happens 30 seconds after the last, a generation speed of 15 tok/s versus 40 tok/s changes a 10-second wait into a 27-second wait inside a task that takes 20 minutes. Agents amortize inference latency far better than chat does.
• Agent context is naturally chunked. A well-designed agent (Chapter 10) does not need one giant 128 K context; it needs disciplined 8–16 K windows refreshed from external memory. That is exactly the regime where an 8 GB card is comfortable, and it is also the regime that produces more reliable agent behavior, because models degrade in the middle of very long contexts.
• The quantization ecosystem optimized for exactly this hardware. The most heavily tested configuration in the entire local-LLM community is “an 8 B model at Q4_K_M on a consumer card,” because that is what most people own. Bug reports, perplexity tables, and throughput numbers for this configuration are abundant — you are walking the best-trodden path in the field.

1.3 What “complex task” means, and what it demands

The goal stated in this paper’s title — manage a complex task — deserves a definition, because it dictates the memory budget. A complex agentic task has four properties that distinguish it from casual chat. First, it is multi-step: the model must loop through plan → act → observe dozens of times, each turn appending tool results and reasoning to the history. Second, it is stateful: information gathered in step 3 (say, the schema of a database) must remain usable in step 40, which means either a long context window or an external memory system. Third, it is tool-using: the model must emit structured function calls reliably — a capability measured separately from general intelligence by benchmarks like the Berkeley Function Calling Leaderboard (BFCL), where Llama 3.1 8B scores ~76 % — which constrains model choice more than raw MMLU does. Fourth, it is long-running: a coding agent session easily spans an hour and hundreds of thousands of cumulative tokens, even though no single context window needs to hold them all at once.

These four properties translate into concrete engineering requirements: a model with reliable tool-calling behavior at 4-bit quantization; enough KV-cache headroom for 8–32 K tokens of working context per call; a context-management layer that compresses or externalizes history before it exhausts that headroom; and a stable inference server that can run for hours without leaking VRAM. Each of these is a chapter of this paper. The short version: your machine can do this — if you spend your bytes deliberately.

1.4 The capability inventory: what fits, honestly

Before the theory, here is the honest bottom-line inventory for an 8 GB GPU + 32 GB RAM machine, based on the experimental evidence compiled in Part IV. These are the regimes you can operate in, from fastest to most ambitious.

2. Where the Bytes Go: Memory Anatomy of a Local LLM

Four arithmetic formulas govern every sizing decision you will make. Master them and OOM errors become predictable instead of surprising.

2.1 The four memory consumers

At any instant during inference, VRAM (and its RAM extension) holds exactly four categories of data. Every sizing question in this paper is answered by adding these four numbers and comparing the total to your budget.

The practical consequence of the fourth row is a rule you should internalize now: your usable VRAM is not 8 GB, it is roughly 7.0–7.4 GB. A desktop environment with a browser open can easily reserve 0.8–1.2 GB of VRAM for itself (check with nvidia-smi before loading anything), and the CUDA context plus compute buffers of a running llama.cpp server cost another ~0.3–0.6 GB even before weights arrive. On a headless machine, or when the display is wired to the motherboard’s integrated GPU, you recover most of that margin — a genuinely useful trick detailed in §3.4.

2.2 Formula 1 — Weight memory

The memory occupied by model weights is the parameter count multiplied by the average bytes per parameter. Nothing more. In fp16 or bf16, each parameter costs exactly 2 bytes; the GGUF quantization ladder (Chapter 4) trades precision for bytes as follows.

Two observations fall straight out of this table. First, Q4_K_M is the pivot of the entire local-LLM world: it is the point where a modern 8 B model fits an 8 GB card with room for a real context window, while measured quality loss stays inside ~2–3 points on downstream benchmarks (evidence in §4.4). Second, the arithmetic explains the “ladder” you will keep climbing throughout this paper: every halving of bytes per parameter buys roughly one model-size class at constant memory. The 30 B MoE at Q4_K_M costs about as much memory as a 14 B dense at Q5 — but only activates 3 B parameters per token, which is why it can be fast despite being large (§6.5).

2.3 Formula 2 — The KV cache (preview)

The second formula is important enough to get its own chapter, but the skeleton belongs here because total-VRAM estimates need it. For every token in the context — prompt plus generated — the model stores one key vector and one value vector per attention layer, per KV head. The exact per-token cost is:

kv_bytes_per_token = 2 × n_layers × n_kv_heads × head_dim × kv_dtype_bytes
                     └─ K and V ─┘   └──── GQA dimensions ────┘

total_kv = kv_bytes_per_token × context_tokens × batch_sequences

Worked example, Llama 3.1 8B (32 layers, 8 KV heads, 128-dim heads, fp16 cache): 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token. At an 8 K context that is 1.0 GB; at 32 K it is 4.0 GB; at the model’s 128 K maximum it is 16 GB — twice your entire GPU. This single calculation, done in five seconds, predicts more OOM failures than any other reasoning in this paper. Chapter 5 works through the numbers for every model in the ladder, shows how GQA already saves you (the same model with full multi-head attention would cost 512 KiB/token), and evaluates the experimental evidence on quantizing the cache itself.

2.4 Formula 3 — The total, and the two planning rules

Adding the components gives the budget check you should perform before downloading any model file:

total_vram ≈ weights_on_gpu + kv_cache + compute_buffer (≈0.5–1 GB with FA) + fixed_overhead (≈0.4–0.8 GB)

PLANNING RULE 1a (comfortable):  weights + kv  ≤  6.5 GB   → fits 8 GB on any setup, with margin
PLANNING RULE 1b (tight):         weights + kv  ≤  7.4 GB   → fits headless / display on iGPU (§3.4)
PLANNING RULE 2 (split):          weights total ≤  30 GB     → fits via offload; speed scales with GPU share (Ch. 6)

Rule 1a’s 6.5 GB figure is the conservative planning line: it assumes a desktop using ~0.5 GB, CUDA and buffers consuming ~1 GB together, and leaves slack for the transient peak that occurs during prompt processing (when the compute buffer temporarily grows to the size of the longest prompt batch). Rule 1b is the honest upper edge: a 7.4 GB plan does run on an 8 GB card, but only when the desktop is not competing for VRAM — headless, or with the monitor on the integrated GPU. People who plan to “exactly 7.9 GB” are the people writing the OOM bug reports this paper cites. Rule 2 reflects the practical ceiling of your 32 GB RAM pool: the OS, agent process, and page cache need headroom too, and mmap-loading a model larger than ~30 GB means the OS is already thrashing.

2.5 A worked example: three ways to run Qwen3-8B on 8 GB

To make the formulas concrete, here is the same model — Qwen3-8B (36 layers, 8 KV heads, 128-dim, hence 144 KiB/token fp16 KV) — planned three ways. These are the actual decisions you will face in Chapter 11.

3. Know Your Machine: The Hardware Audit

Eight gigabytes is not one budget — it is three interlocking bandwidth hierarchies. Optimizing without knowing which one binds produces confident mistakes.

3.1 The three tiers of memory

Every technique in this paper is, at bottom, a decision about which of three memory tiers holds which bytes. The tiers differ not just in size but in bandwidth — the number of bytes per second the compute units can actually pull from them — and bandwidth, not capacity, is what determines tokens per second once a model is loaded.

The three numbers that actually matter for planning are VRAM capacity (how much fits fast), VRAM bandwidth (how fast a fully-resident model generates), and PCIe bandwidth (how fast a split model streams). The asymmetry between 272 GB/s and 16 GB/s — a factor of ~17 — is the entire reason offloading is a gradual trade-off rather than a cliff, and the reason the offload curve in §6.2 has the shape it does: generation is memory-bandwidth-bound per token, so every layer you keep in VRAM avoids paying the PCIe tax for that layer on every single token.

3.2 Why generation speed ≈ bandwidth / model-bytes

A useful mental model (accurate to within 10–20 % for single-stream generation): generating one token requires reading essentially all active weights once. Therefore:

tokens_per_sec  ≈  effective_bandwidth  /  active_bytes_per_token

RTX 4060, Qwen3-8B Q4_K_M fully in VRAM:   272 GB/s ÷ 5.1 GB   ≈ 53 tok/s theoretical
                                              (real world: ~30–42 tok/s — overheads eat 20–40 %)
Same model, 50 % of layers on CPU via DDR4: active bytes split across two buses;
effective throughput collapses toward the slower path → ~8–12 tok/s

This model explains every speed number in this paper without magic. It explains why quantization speeds up inference as a side effect (fewer bytes per token = more tokens per second on the same bus — measured at 3.5–3.8× for 4-bit vs fp16 in the presenc.ai synthesis, §12.2). It explains why the MoE trick in §6.5 works (only 3 B of 30 B parameters are active per token, so the bandwidth denominator is small even though the memory numerator is large). And it explains the single most common performance disappointment in local AI — “I added more context and everything got slow” — because the KV cache joins the denominator as it grows, and prompt processing has a different, compute-bound character entirely (§7.4).

3.3 The CPU and RAM: the other half of the machine

The 32 GB of RAM is not passive scenery. Four practical facts about it shape every hybrid configuration.

• Dual-channel matters. A single stick of 32 GB runs at half bandwidth; two 16 GB sticks let the CPU read ~45–90 GB/s instead of ~25–45 GB/s. For any offloaded configuration this is a near-free 1.3–1.8× speedup on the CPU-resident layers. Check your board has two populated channels (sudo dmidecode -t memory | grep -i speed).
• DDR4 vs DDR5 is a real offload multiplier. DDR5-5600 dual-channel (~90 GB/s) makes 30 B-class hybrid inference meaningfully faster than DDR4-3200 (~51 GB/s). If you are buying RAM for this workload, this is where the money goes — more so than CPU cores.
• Cores set the ceiling for CPU-side compute. llama.cpp CPU layers want ~1–2 threads per physical core with hyperthreading usually harmful for this workload; the llama.cpp performance documentation calls wrong thread settings the #1 cause of slow CPU inference. A 6–8 core modern CPU is fully sufficient; more cores help only the offloaded fraction.
• Storage is nearly irrelevant to speed — but critical to load time. Weights are read once at startup (or paged in lazily via mmap). An NVMe SSD turns 18 GB model loads into ~15 s instead of ~2 min on SATA, and prevents the stutter of paging from a spinning disk. Keep models on the fastest drive you own; after load, I/O drops to zero.

3.4 Reclaiming VRAM from the desktop

On a machine where the monitor plugs into the graphics card, the desktop compositor, browser, and terminal permanently occupy VRAM — commonly 0.5–1.5 GB on a multi-monitor setup. Every hundred megabytes reclaimed is a hundred more for KV cache. The options, in decreasing order of practicality:

1. Close GPU-accelerated apps while doing long agent runs — browsers are the biggest offenders (each tab with WebGL/canvas acceleration holds buffers in VRAM).
2. Move the display to the iGPU if your CPU has one: in BIOS, enable integrated graphics, plug the monitor into the motherboard, and the discrete card becomes 100 % yours. This recovers the entire desktop footprint — typically 0.8–1.2 GB — at the cost of some desktop snappiness. On Linux, ensure the NVIDIA card still gets initialized for compute (nvidia-persistenced helps).
3. Run headless (SSH in): same effect, no monitor at all. This is the standard setup for people who treat the 8 GB box as a home inference server.

3.5 The audit checklist

Run this once; write the answers on a sticky note. Every sizing decision later refers to them.

# GPU: name, VRAM total, current usage, PCIe link generation/width
nvidia-smi --query-gpu=name,memory.total,memory.used,pcie.link.gen.current,pcie.link.width.current --format=csv

# Sustained memory bandwidth test (GB/s) — the single most predictive number
# (bandwidthTest from cuda-samples, or: python -c "import torch; ..."
#  simpler: run llama-bench on a tiny model and compare with published numbers)

# RAM: total, speed, and — critically — number of populated channels
sudo dmidecode -t 17 | grep -E "Size|Speed|Locator" | grep -v "No Module"

# CPU physical cores (set llama.cpp threads to this, not to hyperthreads)
lscpu | grep -E "^CPU\(s\)|Core|Thread|Model name"

# Storage class holding your models
df -h /path/to/models && lsblk -d -o NAME,ROTA,SIZE   # ROTA=1 means spinning disk → move models

With the audit done, you know your three budgets: fast capacity (~7 GB usable VRAM), large capacity (~28 GB usable RAM after OS and agent process), and the bridge between them (12–25 GB/s). Everything that follows is the craft of spending those budgets well.

4. Quantization: The Science of Shrinking Models

What 4-bit really costs, according to the people who measured it — perplexity ladders, benchmark deltas, and the chain-of-thought surprise.

4.1 The idea in one paragraph

Quantization replaces the 16-bit floating-point numbers a model was trained with lower-precision approximations — 8-bit, 4-bit, 3-bit integers with per-block scale factors — so that weights occupy less memory and less memory bandwidth. It is applied after training (“post-training quantization”), requires no retraining, and for the GGUF family can be done on a laptop in minutes. The reason it works acceptably is that neural network weights turn out to be surprisingly tolerant of rounding noise: the model’s function is encoded across billions of parameters redundantly, and careful rounding perturbs the output distribution only slightly. “Slightly” is doing a lot of work in that sentence, which is why this chapter is mostly tables of measurements rather than reassurances.

4.2 The GGUF ladder, decoded

The dominant format for local inference is GGUF, used by llama.cpp, Ollama, LM Studio, and most consumer tools. Its naming scheme intimidates newcomers but encodes only two things: the bit width (Q8, Q5, Q4, Q3, Q2) and the method. The legacy scheme (Q4_0, Q5_0) stores simple blocks of weights sharing one scale. The K-quant scheme (Q4_K_S, Q4_K_M, Q5_K_M, Q6_K) uses smarter, mixed-precision block structures — the _M suffix (“medium”) mixes in some higher-precision tensors for sensitive layers — and dominates the legacy scheme at every bit width: Q4_K_M beats Q4_0 in quality while being barely larger, which is why the community defaulted to it. The I-quants (IQ3_XXS, IQ2_M, …) push below 4 bits using importance-weighted schemes; they are the only way to fit very large models on small cards, but they sit on the steep part of the quality curve and are best treated as a last resort.

4.3 Experiment 1 — The perplexity ladder (Llama 3 8B)

Perplexity (PPL) measures how surprised a model is by held-out text — lower is better, and small PPL increases map loosely to capability loss. The most-cited GGUF measurements come from the llama.cpp repository’s own evaluation runs, republished in a LessWrong evaluation post comparing quantized performance across schemes. The numbers for Llama 3 8B on WikiText-2:

Three things to internalize from this table. First, Q8_0 is genuinely free: a 0.02 % relative perplexity change is below the noise floor of the measurement itself, so the old advice “never go below 8-bit” was calibrated for a world where memory was abundant — in ours, the interesting question starts at Q6. Second, the K-quant premium is real and large: at 4-bit, the K_M scheme loses 3× less perplexity than the legacy scheme at only 0.24 GB more memory. There is no situation on an 8 GB card where Q4_0 is the right choice over Q4_K_M. Third, the jump from Q4 to Q5/Q6 buys meaningful quality at meaningful memory cost — exactly the trade you will exploit in §4.6 when a model almost fits at a higher tier.

4.4 Experiment 2 — Benchmark accuracy, quantized (Llama 3 8B Instruct)

Perplexity is a proxy; agents care about task metrics. The same LessWrong evaluation ran Llama 3 8B Instruct through MMLU (multiple-choice knowledge), WMDP, and The Pile perplexity across quantization schemes from four method families — HQQ, bitsandbytes (BNB), GPTQ, and AWQ:

The pattern generalizes across the whole literature: 8-bit quantization costs nothing measurable; 4-bit costs roughly 1.5–3 points of MMLU, with the calibration-aware schemes (HQQ, AWQ) at the good end and naive round-to-nearest (BNB Int4) at the bad end. A 2-point MMLU drop sounds alarming until you remember what it buys: the same model at half the memory, running ~2× faster on the same bus. For agent workloads the correct comparison is not “quantized vs. full-precision of the same model” but “quantized 8 B vs. full-precision 4 B at equal memory” — and on that comparison the quantized 8 B wins essentially every benchmark by a wide margin. Quantization is how you buy model size with memory; the 1–3 point tax is the commission.

4.5 Experiment 3 — The chain-of-thought surprise (the agent-critical result)

Here is the finding that matters more for agents than everything else in this chapter. The LessWrong author went one step further than most evaluations and tested multi-step reasoning — Minerva MATH algebra problems with zero-shot chain-of-thought prompting, where the model must generate many correct tokens in sequence to reach an answer:

Compare the two 4-bit columns across experiments: MMLU dropped 1.5–3 points, but multi-step generation dropped 4–8 points, and 3-bit did not merely score lower — it failed to complete. The explanation is compounding error: a single-token prediction task can absorb a small nudge to the output distribution, but a chain-of-thought requires every one of hundreds of sampled tokens to stay on the rails; small per-token distortions accumulate multiplicatively. Agents are chain-of-thought machines. Every ReAct loop is a long multi-step generation interleaved with tool calls, which means:

4.6 Method families: GGUF vs AWQ vs GPTQ vs EXL2

The tables above mix method families, so a direct comparison is warranted. GGUF is a file format + quantization scheme designed for CPU/GPU hybrid execution — its superpower is that it runs anywhere and splits across memory tiers (Chapter 6), which is why it owns the under-24 GB segment. AWQ and GPTQ are GPU-only weight formats that quantize using calibration data to protect salient weights; they are loaded by GPU engines (vLLM, TGI, ExLlama) and generally decode faster than GGUF on the same hardware when the whole model fits in VRAM. EXL2 (ExLlamaV2) offers variable target bit-rates and strong speed on full-GPU setups.

Two nuances keep this honest. First, the GPTQ “collapses on code” finding comes from community benchmarking of specific 4-bit releases — the failure is real and reproducible on those files, but it is a reason to be cautious rather than a mathematical law; GPTQ remains widely deployed in production serving. Second, on an 8 GB card the AWQ/EXL2 speed advantage usually cannot be exploited for the models you most want to run, because anything above ~9 B parameters will not fit VRAM entirely — and hybrid execution is exactly what GGUF does and the GPU-only formats do not. This is why the recommendation matrix is not “which is best” but “which fits where your model lands on the ladder.”

4.7 Choosing your quant: the 8 GB decision table

Putting the experiments together into the decision you actually face. (Sizes are for 8 B-class models; shift one tier for 12–14 B, two for 30 B MoE.)

5. The KV Cache: Context Length Is a Memory Decision

The cache is arithmetic, it grows linearly, and quantizing it wrong destroys your outputs while your speedometer stays green.

5.1 The formula, worked for every model you might run

Chapter 2 introduced the formula; here is the full reasoning. Attention layers look back at every previous token by comparing the current query against stored keys, then retrieving values. To avoid recomputing these for every new token, inference engines store them — that storage is the KV cache. Modern architectures shrink it with grouped-query attention (GQA): instead of every one of, say, 32 query heads having its own key/value heads, they share a small group of KV heads (typically 8). This is why an 8 B model costs ~128 KiB/token instead of ~512 KiB:

Reading the table as an agent engineer: with weights at Q4_K_M occupying ~5 GB of your ~7 GB usable VRAM, an 8 B model can afford a 16 K context with an fp16 cache, or ~32 K with a q8_0 cache. A Gemma 3 12B at Q4 can barely afford 8 K before you quantize its cache. These are the real numbers behind every “what context should I set?” question — and they were computed without downloading anything.

5.2 Why the cache grows during an agent run (the hidden budget)

Chat users rarely feel KV pressure because conversations grow slowly. Agents feel it acutely, because a ReAct-style loop appends to the context on every step: the tool call (~50–150 tokens), the tool result (100–2,000+ tokens for a file read or search payload), and the model’s intermediate reasoning (100–500 tokens). A realistic accounting for a 30-step coding agent:

This is why the two disciplines — cache management (this chapter) and agent context engineering (Chapter 10) — are inseparable. The cache gives you a hard ceiling; the agent architecture determines how fast you approach it and what happens when you do.

5.3 Experiment 4 — Quantizing the KV cache: the q8_0/q4_0 split

The KV cache can itself be stored at lower precision, via llama.cpp’s -ctk / -ctv flags (Ollama: environment variable or Modelfile parameters). The most rigorous public evaluation of what this actually costs is InventiveHQ Lab’s controlled benchmark: one model (Qwen2.5-Coder-7B-Instruct Q4_K_M), one context (8,192 tokens), three cache precisions, greedy decoding over a fixed 12-prompt suite, with output similarity measured against the fp16-cache reference:

The result deserves its dramatic reputation. At 8-bit, the cache halves in theory (and saved 208 MB here, a small number only because 8 K context is small — savings scale linearly with context) while outputs stay mostly-identical at 81.6 % similarity: rounding noise, not degradation. At 4-bit the outputs are 8.3 % similar to the reference — statistically a different answer to the same prompt — while generation speed sits at a healthy 80.3 tok/s. Nothing in your monitoring would flag it. The authors’ framing is the memorable one: q4_0 KV saves only 112 MB more than q8_0 at this context but transforms the model; the savings compound with context length, yet so does the corruption.

Corroborating measurements sharpen the picture. A DGX Spark benchmark of the same three cache precisions on Nemotron-3-Nano-30B-A3B at 128 K context found q4_0 KV 92 % slower than f16 at long context (the quantized-cache kernel path has a performance cliff of its own) and concluded “q8_0 is the only KV quantization worth running.” The llama.cpp TurboQuant discussion reports q4_0 cache saving 552 MiB (72 % KV reduction) at the cost of observed slot-position corruption. Meanwhile, Apple Silicon measurements found the throughput hit from KV quantization under 10 % — the cost there is quality, not speed. And the classic community result from the r/LocalLLaMA KV tests: q4_0 cache enabled a 35 B model (Command R) with 8 K context to fit a 24 GB P40 — proof that the trick works as a memory lever; the question is only whether your task survives the fidelity loss.

5.4 Flash Attention: free cache efficiency

Flash Attention (FA) is a different implementation of the attention computation that processes it in tiled fashion without materializing the full attention matrix — in llama.cpp, enabling it (-fa, or automatic in recent Ollama builds via OLLAMA_FLASH_ATTENTION=1) both reduces the compute-buffer footprint and, critically, unlocks KV quantization: cache quantization is only supported on the FA codepath. Community measurements on 4×H100-class hardware show FA also shrinking the inference graph dramatically under parallel load (the Ollama parallelism issue #12097 documents graph-size reductions when FA is on). On your 8 GB card the practical guidance is simple: Flash Attention on, always — there is no measured downside at single-stream scale, and it is the prerequisite for the entire cache-quantization toolkit.

5.5 Context-setting doctrine for the 8 GB agent box

Setting the context window (-c in llama.cpp, num_ctx in Ollama) is a decision people get exactly backwards: they either leave the default (2048 in older Ollama — silently truncating agent histories) or max it out (guaranteeing KV pressure and slow prompt processing). The doctrine that falls out of this chapter’s arithmetic:

1. Set context deliberately per workload: 8 K for focused tool loops; 16 K for coding agents reading files; 32 K only with q8_0 KV on a model whose weights leave room.
2. Budget KV explicitly in your sizing formula (§2.5). Context is not free; it has a price you can now compute exactly.
3. Prefer q8_0 KV over larger f16 windows: 16 K @ q8_0 costs the same as 8 K @ f16 and preserves more capability than the context doubling costs.
4. Trim tool output aggressively — the 30-step accounting table above shows tool results dominate cache growth. A head -n 50 on file reads saves more KV than any quantization flag.
5. Watch for the silent truncation failure: when history exceeds the window, frameworks silently drop the oldest turns — including, catastrophically, your system prompt and tool schemas if the client re-sends them per turn. Chapter 10’s architecture prevents this class of failure by construction.

6. Offloading: VRAM + RAM as One Pool

Layer splitting, mmap, unified memory, and the MoE trick — how a 30 B model runs on a machine whose GPU cannot hold a 9 B one.

6.1 How layer splitting actually works

llama.cpp’s --n-gpu-layers (-ngl) flag is the load-bearing wall of low-VRAM inference. A transformer is a stack of identical decoder layers; the engine loads them one at a time — the first -ngl layers onto the GPU, the remainder into system RAM — and at generation time the hidden state hops across the PCIe bus at the layer boundary, computes the CPU-resident layers on system cores, and hops back. Prompt processing is special-cased: even when most layers are on CPU, the GPU handles the large prompt batches when possible, which matters enormously for agents (§7.4). The mental model to hold: every token pays the PCIe round-trip cost of every CPU-resident layer, and pays it in both directions.

Setting -ngl 99 (or any number ≥ the layer count) offloads everything and lets the engine clamp automatically; on a mixed machine you want the precise number that fills VRAM without exceeding it, because leftover VRAM is pure waste while overflow means either a crash or silent system-memory fallback. Modern llama.cpp builds print exactly how much VRAM each layer needs at load time (offloaded 27/33 layers to GPU plus buffer sizes), and community tooling exists that virtualizes test allocations to find the maximum split automatically (the llama.cpp GPU-automation thread describes the technique). Chapter 11’s sweep script brute-forces the same answer in two minutes with llama-bench.

6.2 What offloading costs: the speed curve

The relationship between GPU-resident fraction and speed is close to linear, with two corrections. First, the intercept is favorable: even a small GPU share beats pure CPU execution by a wide margin, because the GPU offloads prompt processing and the attention-heavy layers benefit disproportionately. Second, the slope depends on your PCIe link and RAM bandwidth: a PCIe 4.0 ×16 machine loses markedly less per offloaded layer than a PCIe 3.0 ×8 one. The bmdpat guide’s systematic -ngl testing summarizes the community consensus plainly: offloading 50 % of layers already yields a major speedup over CPU-only, and the relationship is roughly linear from there.

Two design consequences follow from the curve. First, the 8 GB + 32 GB machine should run two models, not one: a fully-in-VRAM workhorse (8 B at Q4/Q5) for interactive agent steps, and — only when a task demands frontier-adjacent quality — a larger split model at 6–15 tok/s. Second, agentic workloads tolerate the slow tier far better than chat does (§1.2): a planning call that runs at 10 tok/s for 60 seconds inside a 20-minute task is not the bottleneck it would feel like in a chat window.

6.3 mmap: the quiet enabler

llama.cpp memory-maps the model file by default: the OS pages weights into RAM on demand, the page cache is the model copy, and — the under-appreciated part — RAM used as page cache is reclaimable. Swap between two 15 GB models in seconds (the llama-swap project builds an entire hot-swap proxy on this), and after a crash the next load is served mostly from warm cache. The complementary flags worth knowing: --mlock pins weights in RAM to prevent eviction (good for the long-running agent server; costs you the reclaimability), and --no-mmap preloads deterministically at the cost of slower startup and doubled transient memory. Default advice for an agent box: mmap on, mlock on for the primary model, NVMe underneath.

6.4 Unified memory: usually a trap on PCIe machines

llama.cpp offers GGML_CUDA_ENABLE_UNIFIED_MEMORY=1, which uses CUDA managed memory to let the GPU transparently page model tensors from system RAM. On machines with a real unified memory fabric (Apple Silicon, NVIDIA Grace, DGX Spark) this is the natural mode. On your machine — where the GPU reaches RAM only across PCIe — it is usually a trap: community testing on r/LocalLLaMA and the NVIDIA developer forums consistently finds PCIe-attached unified memory “very slow,” with page-fault storms at the tensor boundary replacing the controlled streaming that layer-splitting provides. The one documented exception (llama.cpp PR #8035): for very large models at aggressive low-bit quants where only a fixed small working set is hot — e.g., Llama-3-70B IQ2_XS on 24 GB — unified memory beat manual partial offload, because the driver’s page prediction occasionally outperforms a bad manual split. The rule of thumb that falls out: prefer explicit layer splitting on PCIe machines; reserve unified memory for experiments, not for your daily driver.

6.5 The MoE trick: why 30 B parameters can be cheaper than 9 B

A mixture-of-experts model stores far more parameters than it uses per token: Qwen3-30B-A3B has 30 B parameters distributed across routed experts, but each token activates only ~3 B of them (plus shared experts and attention). Combine that with hybrid placement and something remarkable happens for small-VRAM machines:

• The always-active tensors — attention layers, shared experts, embeddings — are small (~3–5 GB at Q4). Put those in VRAM.
• The routed experts are read sparsely: only the experts the router selects per token are touched. They can live in system RAM, streamed across PCIe on demand — and with fast DDR5 the streaming cost per token is bounded by the ~3 B active bytes, not the 30 B total.
• Prompt processing runs on the GPU even with experts on CPU (the Hugging Face MoE offload guide by Doctor-Shotgun documents this llama.cpp capability and reports CPU+GPU MoE inference as “very performant” for exactly this reason).

The community measurements confirm the theory at consumer scale. Unsloth’s Qwen3-Coder-30B guide reports 6+ tok/s for the 30 B coder model on 18 GB of combined memory — modest, but that is a frontier-adjacent coding agent model running on hardware a fraction of its size class. r/LocalLLaMA reports for Qwen3-30B-A3B at Q4_K_M on a 24 GB card include 90 K-token contexts with 40–48 of 48 layers offloaded; the same model at ~19 GB total splits comfortably across your 8 + 32 GB (VRAM takes attention + shared + a slice of experts). And the 2026 orcarouter coding-LLM survey notes the community has squeezed Qwen3-Coder-30B onto 8 GB cards outright using TurboQuant-style KV compression — extreme, but indicative. On an 8 GB card expect 8–20 tok/s for a Q4 30B-A3B with the right tensor split, versus 6–15 for a 14 B dense — the MoE option buys more capability per token-second at the cost of setup care.

7. Throughput Engineering: Making It Fast

Measure first, then tune the four knobs that matter: placement, threads, prompt processing, and speculation.

7.1 The measurement discipline

Every speed claim in this paper is reproducible with a two-command loop, and your first hour with any model should be spent in it. llama-bench is the standard harness: it reports prompt-processing (pp, tokens/s ingesting input) and text-generation (tg, tokens/s producing output) separately — the two numbers that define agent latency, since agents spend most wall-clock time in pp re-ingesting history at every step and most perceived liveliness in tg.

# Core benchmark: prompt-processing and generation at two context sizes
./llama-bench -m qwen3-8b-Q4_K_M.gguf -ngl 99 -p 512,2048 -n 128 -r 3

# Sweep GPU-layer counts to find your machine's optimal split (do this once per model)
for ngl in 99 36 30 24 18 12 0; do
  echo "=== ngl=$ngl ==="; ./llama-bench -m qwen3-14b-Q4_K_M.gguf -ngl $ngl -p 512 -n 64 -r 2
done

# Watch VRAM while it runs (second terminal)
watch -n 0.5 nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv

One measurement-hygiene warning from the field, worth internalizing before you trust any number including your own: an influential blog post on llama.cpp CUDA benchmarking (“Why Your llama.cpp CUDA Benchmarks Are Wrong”) documented a common failure where mis-built binaries silently ran without real CUDA support, producing a plausible 52 tok/s that was actually a lie. Verify your build with llama-bench’s output (it prints the backend) and sanity-check that GPU utilization rises during generation. The same discipline applies to community numbers: a benchmark without its build, flags, and context size is a rumor.

7.2 Threads and the CPU side

For the CPU-resident fraction of any split model, thread count is the top lever — and the llama.cpp performance documentation is blunt that incorrect thread settings are the #1 cause of slow inference. The optimum is physical cores, sometimes physical cores minus one (leaving headroom for the GPU driver and your agent process), and almost never hyperthreads: SMT threads share execution units with their sibling cores and add scheduling jitter to latency-sensitive decode. Community measurements on hybrid setups (the 3090+3060+3060 benchmark thread) show the same pattern on the GPU side — the wrong process/thread mix cost 20–30 % of throughput until tuned. Set --threads to your physical core count (§3.5’s audit) and move on; this is a five-minute fix worth 1.3–1.8× on CPU-heavy configurations.

7.3 Prompt processing: the agent’s real tax

Generation gets the attention; prompt processing pays the bills. Every agent step resubmits the full history — system prompt, schemas, prior turns, tool results — before generating a single new token. At 8 K history this means ingesting 8 K tokens per step; at 30 steps, ~240 K tokens of cumulative pp. The saving grace: prompt processing is compute-bound and batches beautifully, so the GPU handles it far faster than generation — an RTX 4060 processes 1,000–3,000+ tok/s of prompt at 8 B scale versus ~40 tok/s of generation, and even heavily-offloaded MoE setups keep pp on the GPU (§6.5). Your levers, in order of impact:

1. Keep the model in VRAM if you can — prompt processing suffers more from CPU-resident layers than generation does.
2. Enable Flash Attention (§5.4) — it shrinks the processing-time memory spike that otherwise caps your usable batch.
3. Trim what you resubmit — this is context engineering (Ch. 10), and it is the only lever that improves both pp and KV simultaneously. Halving resubmitted history halves pp time at every single step, compounding across the whole run.
4. Batch size (-b/-ub): defaults are usually fine at single-user scale; raise -ub only if logs show prompt truncation into multiple passes.

7.4 Parallelism: one slot, usually

Ollama and llama.cpp can serve multiple requests against one loaded model by pre-allocating KV cache for OLLAMA_NUM_PARALLEL slots (default auto-selects 4 or 1). Understand the trade precisely: parallel slots divide your KV budget — four slots at 16 K each is the same memory as one slot at 64 K — and on an 8 GB card your KV budget is the scarce resource. For a single-user agent workstation, set OLLAMA_NUM_PARALLEL=1 and give the whole cache to your one agent; the 4-slot default is a server posture that silently quarters your context headroom. (The documentation’s guidance matches: parallel defaults trade memory footprint for throughput — the wrong trade when memory is the constraint.) Related environment hygiene for agent servers: OLLAMA_KEEP_ALIVE — the default 5-minute unload punishes an agent whose tool calls take six minutes; set it to hours (24h) or -1 so the model stays resident, and pair it with --mlock in llama.cpp for the same effect.

7.5 Speculative decoding: 25–60 % free speed

Speculative decoding pairs your target model with a tiny draft model (e.g., Qwen3-0.6B drafting for Qwen3-8B, sharing its tokenizer). The draft model cheaply guesses several tokens; the target model verifies them in a single batched pass; accepted tokens are kept and the first disagreement triggers a fallback — mathematically identical output distribution, meaningfully higher throughput. The measurements: llama.cpp’s speculative-decoding launch thread reported 25–60 % speedups across models (25–40 % being typical); the broader literature surveys (glukhov.org’s 2026 guide, the arXiv Spec-Bench evaluation) put standard draft-model schemes at 20–50 % and structured heads like EAGLE-3 at up to 2–4× under favorable conditions. The acceptance rate — and therefore the speedup — depends entirely on how similar draft and target distributions are, so draft models must be same-family. On an 8 GB card the memory math is friendly (a 0.5–1.7 B draft at Q8 costs 0.5–1.9 GB), but VRAM competition with the target’s KV cache is real: measure whether the draft’s residency costs you more in context than it gains in tokens.

# Target = Qwen3-8B Q4_K_M (in VRAM), draft = Qwen3-0.6B Q8_0
./llama-server \
  -m qwen3-8b-Q4_K_M.gguf \
  -md qwen3-0.6b-Q8_0.gguf \      # draft model
  -ngl 99 -ngld 99 \              # both fully on GPU
  -c 16384 -fa -ctk q8_0 -ctv q8_0 \
  --draft-max 16 --draft-min 4    # tokens per speculation round

# Expect log lines like:  accepted ~60-70 %, speedup ~1.3-1.6×
# (varies with task entropy — code > chat > creative writing in acceptance)

7.6 The tuning checklist, in order of ROI

8. Choosing the Model: The 8 GB Ladder

Benchmarks for every rung, why tool-calling reliability beats MMLU, and the two-model doctrine.

8.1 What the benchmarks actually measure

Model selection for agents is not the same game as model selection for chat. General-knowledge scores (MMLU) correlate loosely with agentic usefulness; what predicts agent success is (a) function-calling accuracy (measured directly by the Berkeley Function Calling Leaderboard, BFCL — where Llama 3.1 8B Instruct scores ~76 % and the Qwen3 series ranks at or near the top of open models), (b) instruction-following precision under long multi-part prompts, (c) robustness to quantization on structured output — the CoT result of §4.5 applies doubly to JSON tool calls — and (d) coding ability if your agent edits files (HumanEval-class metrics, or SWE-bench for whole-task evaluation). A useful piece of evidence that small models are agent-viable: a 2026 arXiv study of tool-calling representations found clean linear structure in models as small as 4 B (Gemma 3 4B, Qwen3 4B), with simple probes reading tool selection at 100 % accuracy after 10–20 examples — tool calling is a learnable, well-structured capability in this size class, not a frontier-model monopoly.

8.2 The ladder, with data

Here is the full selection space for your machine, ordered by how much of it stays in VRAM. Benchmark figures are the models’ published numbers (bf16); expect the quantization deltas of §4.4 on top.

8.3 Head-to-head at the core tier

Because the core tier (8–9 B) is where your agent will live most of the time, the choice deserves the evidence rather than vibes. A controlled TrueFoundry evaluation ran Qwen3-8B, Llama 3.1 8B, and Ministral 8B on identical vLLM serving hardware across 4–256 concurrent users; the headline for single-user relevance was that Qwen3-8B sustained the strongest quality-per-compute at low concurrency (its thinking mode trades latency for accuracy on hard steps — you can toggle it off for fast tool loops), while Llama 3.1 8B held the most stable behavior under load. Independent comparisons (kunalganglani.com, willitrunai.com) converge on the same summary: Qwen3-8B outperforms Llama 3.1 8B on most knowledge, coding, and multilingual benchmarks, while Llama retains edges in ecosystem maturity and slight VRAM economy (4.9 vs 5.1 GB at Q4_K_M). For an agent box, lean Qwen3-8B for capability, Llama 3.1 8B when you want every framework integration to work first try, and Qwen2.5-Coder-7B when the agent’s job is code — its unusually small KV cache (4 heads, not 8) is a genuine architectural advantage for long coding sessions on 8 GB.

8.4 The two-model doctrine

The strongest recommendation this chapter makes is structural, not model-specific: run a two-model stack, with roles assigned by tier. The fast in-VRAM core model (Qwen3-8B class) executes the agent loop — planning, tool calls, observations — where latency and format reliability matter most. The big MoE model (Qwen3-30B-A3B class) is loaded on demand — via llama-swap or an Ollama second instance — for the few hard moments: the one-shot architecture decision, the gnarly debugging session, the long synthesis. The draft tier earns its keep inside this doctrine too: a resident 0.6 B model accelerates the core (§7.5) and doubles as the rolling summarizer for context compression (§10.3), a job where its speed matters more than its brilliance. This matches how the memory economics actually work: your VRAM’s comparative advantage is latency, your RAM + PCIe’s comparative advantage is capacity, and the doctrine assigns each workload to the tier that serves it best. It also matches how frontier agent products are architected — fast models in the loop, big models at decision points — scaled down to your silicon.

9. Inference Engines Compared

llama.cpp, Ollama, vLLM, LM Studio — what each is actually good at, and which belongs on an 8 GB agent box.

9.1 The four contenders

llama.cpp is the reference implementation everything else builds on: the GGUF format, layer splitting, KV quantization, speculative decoding, Flash Attention, tensor overrides — every technique in Part II — originates or lands here first. It ships llama-server (OpenAI-compatible HTTP API, including tool-calling support in current builds) and llama-bench. Cost: you build or download binaries and compose flags yourself. Ollama is llama.cpp (plus its own Go serving layer and a model zoo) wrapped in one-command ergonomics — ollama pull qwen3:8b and you are serving. It auto-configures offload and context defaults, which is both its charm and its danger on a memory-constrained card. vLLM is a throughput-oriented serving engine built on PagedAttention and continuous batching — designed for concurrent users on big GPUs, loading AWQ/GPTQ/FP8 rather than GGUF. LM Studio is the polished GUI: model browser, chat interface, an embedded llama.cpp server — ideal for exploring, at some performance cost (a comparative benchmark measured it 29 % slower than raw llama.cpp on M3 Max hardware, 38.2 vs 53.5 tok/s; the gap reflects wrapper overhead and conservative defaults).

9.2 The comparison, on your constraints

The vLLM row deserves a sentence of fairness rather than a dismissal: if your use case is serving an 8 B AWQ model to several simultaneous users — a team-shared agent gateway, say — vLLM’s continuous batching genuinely wins throughput, and its gpu_memory_utilization flag can keep an 8 GB card disciplined. But for the single-workstation agent pattern this paper targets, its strengths (concurrency, throughput) are not your constraints, and its weaknesses (residency assumptions, heavyweight stack) are exactly your constraints. The dev.to spill test makes the boundary vivid: past 24 GB, llama.cpp and Ollama degrade gracefully to single-digit tok/s and keep going — engineered hybrid behavior — while server-class engines treat oversubscription as an error condition.

9.3 The recommendation

Run llama.cpp’s llama-server as your agent backend, configured per Chapter 11 — you get every Part II technique at full power and the OpenAI-compatible API your agent code already speaks. Keep Ollama installed as the convenience layer for trying models in 30 seconds (its defaults are merely suboptimal, not broken — the tuning table below fixes them), and treat LM Studio as the friendly exploration GUI. This is also the exact stack the community converged on: llama.cpp for control, Ollama for convenience, everything else situational.

10. Agent Architecture for Constrained Machines

On an 8 GB box, the agent’s design is the second half of the memory budget. Context engineering is capability engineering.

10.1 The loop, and why it eats memory

Strip every framework away and an agent is four things in a cycle: a prompt assembly step (system instructions + tool schemas + selected history), a model call that emits either a tool invocation or a final answer, a tool execution step whose output is appended to history, and a termination check. The ReAct formulation — Reason, Act, Observe, repeat — is the canonical shape, and SWE-bench Verified’s own results page credits surprisingly high scores to “just a simple ReAct agent loop” with no special scaffold. Each turn of the cycle grows the history; each growth enlarges the KV cache (§5.2) and the prompt-processing bill (§7.3); and nothing in the naive loop ever shrinks either. On a 128 GB server this negligence is affordable. On your machine, the agent that manages its own context is the difference between a system that completes hour-long tasks and one that OOMs at step 20 — or worse, silently forgets its instructions and wanders.

The framing that makes this concrete is the one Mem0’s engineering blog popularized: the context window is RAM, not storage. Working state belongs in context; everything else — task definition, long-term facts, retrieved knowledge, old tool outputs — belongs in external storage (files, a database, a retrieval index) that is paged in on demand and expires out when no longer relevant. Production agent systems (per the zylos.ai session-lifecycle survey) converged on exactly this discipline: session rotation, memory persistence, and priority management for always-on operation. Your architecture should be a miniature of theirs.

10.2 Strategy menu for context management

Five strategies, in the order you should adopt them. The first two are mandatory; the rest scale with task ambition.

Two implementation details make or break these strategies. First, never let the eviction boundary eat your system prompt or tool schemas — the silent-truncation failure of §5.5. Pin them structurally: assemble each turn’s context as [pinned system block] + [rolling summary] + [last K turns] + [current user turn], so eviction can only ever touch the designated middle. Second, summarize into structure, not prose: a rolling summary formatted as “goal / decisions made / facts learned / files touched / current step / open problems” preserves exactly the state an agent needs to resume coherently, where a paragraph-style summary reliably loses the procedural thread — the multi-day agent-session writeups on dev.to report the same lesson from practice.

10.3 Tool design for small, quantized models

The model executing your loop is an 8 B model at Q4 with a quantization-perturbed distribution. Tool interfaces can be designed to make that model look bigger than it is, or smaller — the difference is mostly schema discipline.

• Few tools per decision. BFCL-style accuracy drops as the tool count grows; the arXiv tool-calling-probe work shows selection is linearly readable but confusability still rises. 4–7 well-chosen tools beat 20 overlapping ones. If you need more, group them behind a dispatcher tool.
• Terse, typed schemas. Short names, one-line descriptions, required fields only. Every schema token is resubmitted at every step (§7.3) — verbose schemas are a permanent context tax you pay hundreds of times.
• Design outputs for the cache. Paginate by default (offset/limit), return IDs not bodies, cap list lengths, make the first line of output the answer.
• Errors are prompts. A tool error message is read by the model as its next input; “FileNotFoundError: config.yaml — did you mean config.yml? Use list_dir to check” recovers the loop, while a stack trace fills the cache with useless tokens.
• Constrain the format at decode time. llama.cpp’s GBNF grammars (and JSON-schema modes in llama-server/Ollama) make malformed tool calls impossible rather than merely unlikely — the single highest-ROI reliability intervention on quantized models, turning the §4.5 compounding-error risk into a retry at worst.
• Idempotent, bounded tools. The agent will call the same tool twice and pass wrong arguments; make the cost of that a no-op, not a mutation.

10.4 RAG on a budget: the embedding side

Retrieval needs an embedding model, and the good news is that they are tiny by LLM standards: nomic-embed-text-v1.5 (137 M parameters, ~274 MB at fp16) or bge-m3 run comfortably alongside your core model — the former even on CPU without hurting loop latency, since embeddings are generated asynchronously with tool execution. Keep the vector store small and local (sqlite-vec, LanceDB, Chroma in embedded mode; all in-process, all free of a server’s RAM). The discipline that matters: chunk documents to your retrieval budget (256–512 token chunks mean each retrieved chunk is 256–512 tokens of KV — retrieve five and you have spent 2 K tokens of your window before the model thinks). Retrieval top-k is a memory decision, exactly like context length; tune it like one.

10.5 The scaffolding lesson from SWE-bench

Aspirational evidence that architecture beats size: the mini-swe-agent project — deliberately minimal, ~100 lines of loop logic — resolved 65 % of SWE-bench Verified instances with a frontier model, and its successor configurations exceed 74 %; the SWE-bench Verified leaderboard notes bash-only ReAct loops matching elaborate scaffolds. The counterproductive pattern the project explicitly rejected is framework bloat: every abstraction layer you add between model and tools costs tokens (re-exported schemas, middleware narrating itself into the history) and hides the state your context management needs to see. On an 8 GB machine this lesson is not philosophy, it is memory: LangChain-class frameworks bring convenience but also token overhead and several hundred MB of process footprint; a 200-line bespoke loop (Chapter 11) gives you the §10.2 strategies with zero framework tax and total observability. Use framework abstractions where they earn their memory; otherwise write the loop.

10.6 Failure modes specific to this hardware

11. The Complete Build: Recipes and Code

Everything from the previous ten chapters, assembled into a running agent on your machine.

11.1 Build the engine

# Prereqs: CUDA toolkit (nvcc), cmake, a C++ compiler, git
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="89;86"   # 89=4060, 86=3060/3070
cmake --build build --config Release -j$(nproc)

# Verify CUDA actually linked (see §7.1 warning about silent CPU-only builds)
./build/bin/llama-bench -m /path/to/tiny-test-model.gguf -ngl 99 -p 128 -n 32
# Expect "backend: CUDA" in output and GPU util in nvidia-smi

Windows users: the llama.cpp releases page ships prebuilt CUDA binaries, and WSL2 runs the Linux build with full GPU passthrough — either path works; the recipes below are identical. Keep the build directory: llama-server, llama-bench, and llama-cli all live in build/bin/.

11.2 The sizing calculator (run before every download)

The Chapter 2 formulas as a tool. This is deliberately dependency-free — copy it anywhere.

#!/usr/bin/env python3
"""VRAM planner: weights + KV cache + overhead vs your card. (Ch. 2 & 5 formulas)"""
MODELS = {  # name: (params_B, layers, kv_heads, head_dim, q4km_gb)
  "qwen3-4b":        (4,  36,  8, 128, 2.6),
  "phi-4-mini":      (3.8, 32,  8, 128, 2.5),
  "gemma-3-4b":      (4,  34,  4, 256, 2.4),
  "qwen3-8b":        (8,  36,  8, 128, 5.1),
  "llama-3.1-8b":    (8,  32,  8, 128, 4.9),
  "qwen2.5-coder-7b":(7,  28,  4, 128, 4.7),
  "gemma-3-12b":     (12, 48,  8, 256, 7.3),
  "qwen3-14b":       (14, 40,  8, 128, 9.3),
  "qwen3-30b-a3b":   (30, 48,  4, 128, 18.6),
}
VRAM_GB = 8.0; COMFORT = 6.5; TIGHT = 7.4; OVERHEAD = 0.9; KV_BYTES = {"f16": 2, "q8_0": 1.06, "q4_0": 0.56}

def plan(name, ctx_k=16, kv="q8_0", quant_gb=None):
    p, L, H, D, q4 = MODELS[name]
    w = quant_gb if quant_gb else q4
    per_tok = 2 * L * H * D * KV_BYTES[kv]              # §5.1 formula
    kv_gb = per_tok * ctx_k * 1024 / 1e9
    total = w + kv_gb + OVERHEAD
    verdict = "FITS w/ MARGIN" if total <= COMFORT else \
              "TIGHT (headless/iGPU)" if total <= TIGHT else "SPLIT/OOM"
    print(f"{name:<18} weights={w:5.1f}  KV@{kv}/{ctx_k}K={kv_gb:5.2f}  "
          f"total≈{total:5.2f} GB  {verdict}")
    return total

if __name__ == "__main__":
    print(f"Tiers: ≤{COMFORT} GB comfortable · ≤{TIGHT} GB tight (§2.4)")
    plan("qwen3-8b", 16, "q8_0")          # the recommended core config (tight tier)
    plan("qwen3-8b", 8, "q8_0")           # same model, comfort tier
    plan("gemma-3-12b", 8, "q8_0")        # the tight-fit rung
    plan("qwen3-14b", 8, "q8_0")          # → split required (Ch. 6)
    plan("qwen3-30b-a3b", 32, "q8_0")     # → MoE hybrid (§6.5)

11.3 Launch configurations (the three servers)

./build/bin/llama-server \
  -m models/qwen3-8b-Q4_K_M.gguf \
  -ngl 99 \                    # all 36 layers to GPU (5.1 GB < budget)
  -c 16384 \                   # deliberate context (§5.5 doctrine)
  -fa \                        # Flash Attention: on, always (§5.4)
  -ctk q8_0 -ctv q8_0 \        # KV cache: the safe quant (§5.3)
  --mlock \                    # pin weights; no mid-task eviction (§6.3)
  --threads 6 \                # physical cores from §3.5 audit
  --host 127.0.0.1 --port 8080 \
  --jinja                      # use model's chat template (needed for tools)

# Sanity: expect ~30–42 tok/s gen, 1–3k tok/s prompt processing on RTX 4060-class
# VRAM after warm-up: ~7.1 GB — the TIGHT tier (§2.4): run headless or move
# the display to the iGPU (§3.4). Verify with: nvidia-smi

./build/bin/llama-server \
  -m models/qwen3-30b-a3b-Q4_K_M.gguf \
  -ngl 99 \
  --override-tensor "blk\.([0-9]|[12][0-9]|3[0-5])\.ffn_.*_exps=CPU" \
                               # routed experts of layers 0–35 stay in RAM;
                               # attention+shared experts+layers 36–47 in VRAM (~6.5 GB)
  -c 16384 -fa -ctk q8_0 -ctv q8_0 \
  --mlock --threads 6 --port 8081 --jinja
# Expect 8–20 tok/s (DDR4→DDR5 dependent); quality ≈ 14B-class dense
# Tune the layer boundary: if VRAM headroom remains after load, shift the
# regex to keep more expert layers on GPU; re-run llama-bench each change

./build/bin/llama-server \
  -m models/qwen3-8b-Q4_K_M.gguf -md models/qwen3-0.6b-Q8_0.gguf \
  -ngl 99 -ngld 99 -c 16384 -fa -ctk q8_0 -ctv q8_0 \
  --draft-max 16 --draft-min 4 --port 8080 --jinja
# Same outputs as recipe A, ~1.3–1.6× faster generation (25–60 % per §7.5)

11.4 The minimal agent (all of Chapter 10, ~170 lines)

This is the deliverable the paper has been building toward: a ReAct agent with structural context pinning, a sliding window, rolling summarization, tool-output hygiene, and schema retry — no framework, every memory decision visible. It speaks the OpenAI-compatible protocol to any recipe above.

#!/usr/bin/env python3
"""nano_agent: a context-disciplined ReAct agent for llama-server / Ollama.
Implements Ch.10: pinned system block, sliding window, rolling summary,
tool-output hygiene, format retry. Deps: pip install requests"""
import json, re, subprocess, shutil, requests

API   = "http://127.0.0.1:8080/v1/chat/completions"
MODEL = "local"                          # llama-server ignores the name
KEEP_TURNS   = 12                        # sliding window: recent turns kept verbatim
SUMMARIZE_AT = 24_000                    # chars of history → compress (≈6k tokens)
TOOL_RESULT_CAP = 4_000                  # tool-output hygiene: hard byte cap

SYSTEM = "You are a careful file-and-shell agent. Work step by step. \
Call exactly one tool per turn. Stop with a final answer when done."

TOOLS = [
  {"type":"function","function":{"name":"list_dir","description":"List files in a directory",
    "parameters":{"type":"object","properties":{"path":{"type":"string"}},"required":["path"]}}},
  {"type":"function","function":{"name":"read_file","description":"Read up to 80 lines of a text file",
    "parameters":{"type":"object","properties":{"path":{"type":"string"}},"required":["path"]}}},
  {"type":"function","function":{"name":"write_file","description":"Overwrite a text file",
    "parameters":{"type":"object","properties":{"path":{"type":"string"},
      "content":{"type":"string"}},"required":["path","content"]}}},
  {"type":"function","function":{"name":"run_cmd","description":"Run a short shell command (60s timeout)",
    "parameters":{"type":"object","properties":{"cmd":{"type":"string"}},"required":["cmd"]}}},
]

# ---------------- tools (bounded + error-messages-as-prompts, §10.3) --------
def tool_read_file(a):
    try:
        lines = open(a["path"], errors="replace").read().splitlines()[:80]
        return "\n".join(lines) or "(empty file)"
    except Exception as e:
        return f"ERROR reading {a['path']}: {e}. Use list_dir to check the path."

def tool_run_cmd(a):
    try:
        r = subprocess.run(a["cmd"], shell=True, capture_output=True, text=True, timeout=60)
        out = (r.stdout + r.stderr).strip()
        return out[:TOOL_RESULT_CAP] or "(no output)"
    except subprocess.TimeoutExpired:
        return "ERROR: command exceeded 60s timeout. Split the work into smaller steps."

TOOLS_IMPL = {"list_dir": lambda a: "\n".join(sorted(
                __import__("os").listdir(a["path"]))[:200]) or "(empty)",
              "read_file": tool_read_file,
              "write_file": lambda a: (open(a["path"],"w").write(a["content"]), "wrote "+a["path"])[1],
              "run_cmd":   tool_run_cmd}

# ---------------- context engineering (§10.2) ------------------------------
def build_messages(system, summary, turns):
    # Structural pinning: eviction can only ever touch `turns`.
    msgs = [{"role":"system","content":
             system + (("\n\n## SESSION MEMORY\n"+summary) if summary else "")}]
    msgs += turns[-KEEP_TURNS*2:]        # user+assistant pairs
    return msgs

def rolling_summary(old_summary, evicted_turns):
    # §10.2 strategy 3: compress evicted turns INTO STRUCTURE, not prose.
    transcript = "\n".join(f"[{t['role']}] {t['content'][:600]}" for t in evicted_turns)
    prompt = (f"Current session memory:\n{old_summary or '(none)'}\n\nNew events:\n{transcript}\n\n"
              "Update the memory. Use EXACTLY this format:\n"
              "GOAL:\nDECISIONS:\nFACTS:\nFILES_TOUCHED:\nCURRENT_STEP:\nOPEN_PROBLEMS:")
    r = requests.post(API, json={"model":MODEL,"messages":[
        {"role":"system","content":"You maintain an agent's session memory. Be terse."},
        {"role":"user","content":prompt}], "temperature":0}, timeout=120)
    return r.json()["choices"][0]["message"]["content"]

# ---------------- the loop (§10.1) -----------------------------------------
def call_model(msgs, tools=None):
    body = {"model":MODEL,"messages":msgs,"temperature":0.2,"max_tokens":1500}
    if tools: body["tools"] = tools
    r = requests.post(API, json=body, timeout=600)
    return r.json()["choices"][0]["message"]

def run_agent(task, max_steps=40):
    turns, summary, log = [], "", []
    turns.append({"role":"user","content":task})
    for step in range(max_steps):
        msgs = build_messages(SYSTEM, summary, turns)
        m = call_model(msgs, tools=TOOLS)
        # format-retry: one re-ask on malformed tool calls (§10.6)
        for attempt in (0,1):
            tc = m.get("tool_calls")
            if tc and tc[0]["function"].get("name"): break
            if m.get("content"): break               # final answer path
            m = call_model(msgs + [{"role":"assistant","content":"(invalid)"},
                {"role":"user","content":"Call one of the tools, using proper JSON."}], tools=TOOLS)
        turns.append({"role":"assistant","content":m.get("content") or "",
                      **({"tool_calls":m["tool_calls"]} if tc else {})})
        if not tc:
            return m["content"], summary             # done
        fn, args_raw = tc[0]["function"]["name"], tc[0]["function"].get("arguments","{}")
        try:    args = json.loads(args_raw or "{}")
        except json.JSONDecodeError:
            result = f"ERROR: malformed JSON arguments {args_raw!r}. Retry with valid JSON."
        else:
            result = TOOLS_IMPL.get(fn, lambda a: f"ERROR: unknown tool {fn}")(args)
        result = result[:TOOL_RESULT_CAP]            # hygiene, again, defensively
        turns.append({"role":"tool","tool_call_id":tc[0]["id"],"content":result})
        log.append((step, fn, result[:120]))
        # window check: compress BEFORE the window, never after
        if sum(len(t["content"]) for t in turns) > SUMMARIZE_AT:
            cut = len(turns) - KEEP_TURNS*2
            summary = rolling_summary(summary, turns[:cut])
            turns = turns[cut:]
            print(f"[ctx] compressed → summary {len(summary)} chars, window reset")
    return "(max_steps reached)", summary

if __name__ == "__main__":
    answer, memory = run_agent(
        "In ./workspace: find the Python file with a syntax error, fix it, "
        "and verify it compiles with python -m py_compile.")
    print("FINAL ANSWER:\n", answer, "\n\nSESSION MEMORY:\n", memory)

Read the loop once more and notice what is absent: no framework imports, no hidden history, no re-sent schemas per turn (the server holds them), and every line that touches memory is labeled with the section that justifies it. This transparency is the point of §10.5 — when something misbehaves at 3 a.m. of a long task, every byte entering your 16 K window is visible in turns and summary, not buried under framework internals. Swap the TOOLS table for your own domain (web search, HTTP calls, a database client) and the memory discipline transfers unchanged.

11.5 The monitoring rig

#!/usr/bin/env python3
"""Sample GPU + system memory every second; correlate with agent steps."""
import subprocess, time, csv, datetime

def gpu():
    out = subprocess.run(["nvidia-smi","--query-gpu=memory.used,utilization.gpu,temperature.gpu",
        "--format=csv,noheader,nounits"],capture_output=True,text=True).stdout.strip()
    return [int(x) for x in out.split(", ")] if out else [0,0,0]

def rss_gb():
    out = subprocess.run(["free","-b"],capture_output=True,text=True).stdout.splitlines()[1]
    return int(out.split()[2]) / 1e9                      # used, GB

with open("vram_log.csv","w",newline="") as f:
    w = csv.writer(f); w.writerow(["time","vram_mb","gpu_util","gpu_c","rss_gb"])
    while True:
        v,u,t = gpu(); w.writerow([datetime.datetime.now().isoformat(timespec="seconds"),v,u,t,rss_gb()])
        f.flush(); time.sleep(1)
        # Open vram_log.csv next to your agent run: watch KV growth as steps accrue

Run the monitor during your first few long agent sessions and read the graph afterward: VRAM should step up once at load, drift up slowly with context growth, and step down at session rotation. A sawtooth of reloads means KEEP_ALIVE or eviction is misconfigured; steady creep to the ceiling means your SUMMARIZE_AT threshold is too generous. This habit — instrument, then trust the instrumentation — is the final competency this paper teaches, because it converts every future surprise into a five-minute diagnosis.

11.6 Case study: one complex task, start to finish

To make the whole paper concrete, here is a realistic complex task run on exactly this hardware — the kind the introduction promised you would be able to manage. The task: “Take this messy 12-file Python project, add a CSV export feature with tests, and produce a change summary.” It involves reading unfamiliar code, multi-file edits, running test suites, and a written deliverable — genuinely agentic work, not a demo prompt. What follows is the memory-engineering view of the session: what the machine was doing at each phase, and which chapter’s discipline kept it healthy.

Four lessons fall out of the case study that the theory chapters imply but the run makes visceral. First, the cache never grew past ~2 GB — not because 16 K was the ceiling, but because the agent’s hygiene and compression kept the window half-empty at all times; the ceiling existed as slack, which is exactly how you want to operate a constrained machine. Second, the slow moments were tools, not tokens: 17 of the 61 minutes were pytest executions, where inference speed is irrelevant — the strongest practical argument for the §1.2 thesis that agents amortize modest tok/s. Third, escalation was cheap because it was designed (a bounded question, one call, swap back); an undisciplined setup would either never escalate or leave the big model resident and evict the core one. Fourth, the deliverable came from externalized memory — the structured summary — proving the §10.2 claim that a well-maintained summary is not a lossy compromise but the correct medium for long-horizon state. This is what “managing a complex task on 8 GB” looks like in practice: not heroic optimization, but a stack of small disciplines holding simultaneously.

12. Compiled Evidence: The Experimental Record

Every experiment this paper leans on, reproduced as tables — organized by question, with sources and confidence notes.

12.1 Question: “What does quantization cost?”

12.2 Question: “What does the KV cache really cost?”

12.3 Question: “How fast is hybrid/offload, really?”

12.4 Question: “How much speedup from the tricks?”

12.5 Question: “Can small models actually run agents?”

13. Decision Playbooks

The paper’s conclusions, compressed into trees and tables you can follow at decision time.

13.1 Tree 1 — choosing the configuration

What is the task?

▼

Chat, drafting, summarization → Qwen3-4B or Phi-4-mini @ Q6_K, fully in VRAM, 16–32 K ctx. 50–90 tok/s. Done.

▼

General agent (tools + files + web) → Qwen3-8B @ Q4_K_M, fully in VRAM, 16 K ctx @ q8_0 KV, FA on (Recipe A). This is the default.

▼

Agent that writes serious code?

┌──────────────┐

Coding-focused, fast loops → Qwen2.5-Coder-7B @ Q4_K_M (its 4-KV-head cache = ~2.3× more context than 8-head peers)

│

Coding + maximum quality → Qwen3-Coder-30B-A3B @ Q4_K_M, hybrid split (Recipe B); accept 6–15 tok/s in the loop

▼

Needs > 16 K working context?

┌──────────────┐

Yes → first ask §10: can context engineering shrink the need? If genuinely yes-needed: Gemma 3 4B (128 K native, small cache) or accept 32 K @ q8_0 on the 8 B.

No → stop here; bigger windows are a tax, not a feature.

▼

Final check for every branch: run kv_calc.py; ≤ 6.5 GB comfortable, ≤ 7.4 GB tight (headless), else split.

13.2 Tree 2 — the OOM / slow-recovery path

Symptom?

▼

Crash at load → weights + buffers exceeded VRAM. Fix: lower quant tier OR reduce -ngl until it fits (sweep script §7.1) — never trust “it fit yesterday” without checking desktop VRAM usage (§3.4).

▼

Crash mid-run (after N steps) → KV cache grew into the ceiling (§5.2). Fix: smaller -c, q8_0 KV, or lower agent SUMMARIZE_AT threshold. This is the signature failure of un-managed agents.

▼

No crash, but crawling generation → likely CPU-resident layers + wrong threads, or unified memory accidentally on. Fix: threads = physical cores; verify -ngl actually offloaded (server log line); ensure GGML_CUDA_ENABLE_UNIFIED_MEMORY is unset.

▼

Slow only on long prompts → prompt-processing bound (§7.3). Fix: trim resubmitted history; FA on; keep more layers in VRAM.

▼

Still stuck? → Chapter 14, then the §7.1 measurement discipline from a clean build.

13.3 The one-page cheat sheet

14. Troubleshooting: When VRAM Fights Back

Symptom → cause → fix, for the failures specific to 8 GB + 32 GB machines.

14.1 The five classic failures

Failure A — “CUDA error: out of memory” at model load

Diagnosis: weights + compute buffers exceeded free VRAM at startup — the load-time spike (prompt-processing scratch allocation) is larger than steady-state use, so a model that “should fit” can still die at boot. Fixes in order: (1) check what else holds VRAM (nvidia-smi — browsers and desktop compositing are the usual suspects, §3.4); (2) lower -ngl until the load line reports headroom; (3) reduce -c — the compute buffer scales with context; (4) drop one quant tier. The sweep script in §7.1 finds the boundary in minutes; the boundary you find today is only valid for today’s free VRAM.

Failure B — OOM or eviction mid-task (the agent-shaped failure)

Diagnosis: the KV cache grew with history until it hit the ceiling (§5.2); OR the serving layer evicted and reloaded the model on a tool timeout (the KEEP_ALIVE sawtooth, §10.6). Distinguish them from the logs: a reload logs load lines and takes ~10–60 s; true OOM logs allocation failures. Fixes: KEEP_ALIVE=24h/--mlock; then cap the agent’s working context (lower SUMMARIZE_AT); then q8_0 KV; then smaller -c. Re-run the §11.5 monitor to confirm the sawtooth is gone.

Failure C — Crippling slowness despite a good GPU

Diagnosis: almost always one of: silent CPU-only build (§7.1’s benchmark-integrity warning), wrong thread count (§7.2), accidental unified-memory mode (§6.4), or PCIe saturation from an aggressive split. Fix procedure: verify “backend: CUDA” in logs → verify the offload line (offloaded 36/36) → verify threads → run llama-bench and compare to the §12.3 range for your class. Each check is one command; do them in order and stop at the first failure.

Failure D — Outputs subtly wrong after a config change

Diagnosis: the q4_0-KV cliff (§5.3) is the canonical example — throughput looks fine while outputs are a different distribution entirely. Other members of the family: an IQ2/IQ3 quant on a sensitive model (§4.5), a truncated chat template (missing --jinja makes tool-calls incoherent), or silent history truncation (§5.5). Fix procedure: change one variable at a time back toward Recipe A and re-test with a fixed greedy prompt suite — the InventiveHQ methodology (temperature 0, same prompts, diff the outputs) is exactly right for this and takes ten minutes to replicate locally.

Failure E — RAM exhaustion / swap death on the hybrid path

Diagnosis: 32 GB feels vast until the OS, agent process, page cache for an 18 GB model file, and a browser compete for it; the machine swaps and everything becomes slow, not just inference. Fixes: keep total model file ≤ ~24–28 GB on this machine (leaving 4–8 GB of breathing room); close the browser during long runs; prefer --mlock with a model you have sized deliberately over accidental page-cache churn; check vmstat 1 for si/so activity — nonzero sustained values mean swapping, and no inference flag will fix that.

14.2 The diagnostic one-liners

# Who is eating VRAM right now? (processes + totals)
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

# Is the model actually offloaded? (look for the load-time line)
journalctl -u llama-server --since "1 hour ago" | grep -E "offloaded|layers|backend"

# Are we swapping? (si/so columns must be ~0 during inference)
vmstat 1 5

# Is the PCIe link running at full width? (x16 vs x8 changes offload math)
nvidia-smi -q | grep -A 3 "Link Current"

# Quick throughput ground truth, 30 seconds
./build/bin/llama-bench -m models/qwen3-8b-Q4_K_M.gguf -ngl 99 -p 512 -n 128 -r 2

15. Glossary

Plain-English definitions of every term this paper uses, in dependency order.

16. Sources and Further Reading

Everything cited in this paper, plus the canon worth reading next.

16.1 Primary sources cited

16.2 The canon, going deeper

Maximum Capability from Minimum Silicon · a single-file research paper · compiled August 2026 · all measurements attributed inline; community figures are directional, reproduce them with the §7.1 discipline before betting on them.