1.1 The 8 + 32 regime

An 8 GB GPU paired with 32 GB of system RAM is the most common "serious but affordable" configuration of the last several hardware generations — RTX 3050/3060/3070-class and their 40-series successors on the NVIDIA side, and functionally similar tiers from AMD. It sits at an awkward midpoint. It is far too small to hold a modern frontier model's weights, even aggressively quantized — a 70B model at 4-bit is ~40 GB, five times the VRAM. But it is also far too capable to write off: 8 GB comfortably holds any 7–8B model at 4-bit with room for a real context window, and — the finding this archive returns to repeatedly — it can orchestrate models far larger than itself when those models are Mixture-of-Experts (MoE) architectures whose active parameters are small.

The defining feature of the regime is asymmetry, and it shows up in three dimensions at once:

• Capacity asymmetry. The fast pool (VRAM, 8 GB) is a quarter the size of the slow pool (RAM, 32 GB). Whatever you run must be split across the two, or must fit in the small one with room to breathe.
• Bandwidth asymmetry. The fast pool is 5–9× faster to read than the slow pool. An RTX 3070 moves ~448 GB/s; dual-channel DDR4-3200 moves ~51 GB/s. Generating one token requires reading essentially all active weights, so a model resident in the wrong pool becomes 5–9× slower, not slightly slower.
• Control asymmetry. The GPU cannot address system RAM directly (on discrete cards). Anything that lives in the slow pool must cross the PCIe bus — 16–32 GB/s in practice — which is another 15–30× slower than VRAM. This is the wire. Every offloading architecture in Chapter 4 is a strategy for sending the fewest possible bytes across it.

1.2 The bandwidth law

Autoregressive generation has a brutal, simple cost model: to emit one token, the hardware must read every weight that participates in that token's computation. Compute is nearly free by comparison — GPUs spend most of the decode phase idle, waiting on memory. This is why the single best predictor of tokens/sec on low-end hardware is bytes-per-token divided by memory bandwidth, and why the bandwidth ladder below is effectively a preview of every speed result in this paper.

The ladder explains several results that otherwise look surprising. Apple's M-series chips, often dismissed as "integrated graphics," sit high on it — an M4 Max offers ~546 GB/s of unified memory that the GPU can address directly, no PCIe wire involved. That is why official llama.cpp benchmarks (§5) show a MacBook generating gpt-oss-20b tokens at speeds an 8 GB discrete card cannot match unless the card keeps the whole model in VRAM. Conversely, the Raspberry Pi 5's 17 GB/s explains why its community results cluster around 2–8 tokens/sec for small models — and why a 4-board cluster running a 30B MoE model at 13 tokens/sec (§10.3) was received as an achievement rather than a curiosity.

1.3 Five laws that follow

Everything in the archive — every paper, every community tweak — is an application of one of five rules. We state them here because they recur, and because readers who internalize them can predict experimental results before looking them up.

One more framing device that the rest of the paper relies on: think of the 8 GB rig as a cache hierarchy with a GPU at the top, not as a small server. VRAM is L1 (8 GB, ~450 GB/s), system RAM is L2 (32 GB, ~50–90 GB/s), NVMe is L3 (terabytes, ~7 GB/s). Datacenter inference engines solve this same hierarchy with tensor parallelism across many GPUs; the low-resource community solved it with smarter placement of a single model across one GPU's two pools. That is the engineering problem, and Chapters 3–5 are its solution space.

2.1 Anatomy of a forward pass

A transformer layer, at generation time, performs four kinds of work, each with its own memory footprint. Understanding the split is what lets you read other people's experiments critically, because the four costs respond to different optimizations.

The practical consequence: when a community post says "I fit 30B on 8 GB," the claim decomposes into a claim about each row. Which quant (row 1)? What context (row 2)? FlashAttention on (row 3)? How much left over (row 4)? The next two sections do that decomposition explicitly.

2.2 Auditing an 8 GB budget

Here is the canonical "comfortable" configuration for this paper's reference rig, audited line by line. It is the configuration that the model-fit ladder in §6 calls Tier 1, and every number in it can be checked against a real llama.cpp load.

Three budgeting rules fall out of this audit, and they are worth internalizing before the archive chapters because every experiment in them assumed at least one:

1. Reserve overhead before anything else. Plan against ~6.5 GB of usable VRAM, not 8. Headroom is not waste; it is what stops the Windows VRAM-overcommit trap (Law 5) from silently destroying performance.
2. Size the context deliberately. The same model at 32K context needs 4.3 GB of KV cache at fp16 — instant OOM. At q8_0 KV it needs 2.1 GB — tight but survivable. The context window is a budget decision, not a default.
3. One model at a time. The 8 GB rig is a single-tenant machine. Agent stacks that want a "big reasoner + small utility" pair must swap models in and out (§9.3) or move the small one to CPU entirely.

2.3 The KV cache, precisely

Because agent workloads live and die by context, the KV formula deserves to be stated exactly rather than gestured at. For each token, every layer stores a key vector and a value vector for each KV head (with Grouped-Query Attention, far fewer than the total attention heads). In bytes:

The figure below plots the growth curve for three KV precisions, and it is the single most important chart for agent builders: it is the reason a rig that comfortably runs 8K-context chat falls over on an agent task that simply has more turns, and it is the reason KV quantization (§7.1) and eviction (§7.2) exist as research fields.

Two architectures change the constants, not the law. GQA (used by Llama 3, Qwen3, Gemma 3) is why modern 8B models cost 128 KB/token where a pre-GQA model like Mistral-7B's multi-head attention would cost ~512 KB/token at the same width — a 4× context discount already built into the models you would choose anyway. Sliding-window attention (Gemma 3, gpt-oss) caps the cache at the window size, trading exact long-range recall for a bounded footprint — a trade that matters for document QA and is catastrophic for "find the instruction from turn 3" agent behavior unless you keep a summary outside the window (§9.2).

The mechanics in one paragraph: a quantization format stores each weight in fewer bits than the 16 it was trained in — 8, 5, 4, 3, or fewer — using a scheme to choose the representable values (a plain grid, or one scaled per block of 32–128 weights) and optionally a small calibration step to place the grid where the model's weights actually cluster. The design space is real: which 4-bit you pick changes perplexity more than the jump from 8-bit to 4-bit does. That is what the datasets below are for.

3.1 The GGUF ladder: a community dataset

llama.cpp's GGUF format and its K-quant / I-quant families are the de facto standard on low-resource machines, and the community has measured their quality cost obsessively. The most-cited dataset is the perplexity ladder for Llama-3-8B on WikiText-2, reproduced here in full — it is the table people mean when they say "Q4_K_M is the sweet spot."

3.2 Across methods: GPTQ vs AWQ vs HQQ vs BitsAndBytes

GGUF dominates local inference, but the PyTorch-serving world (vLLM, TGI, transformers) quantizes with GPTQ, AWQ, HQQ, and BitsAndBytes — and the research community has run careful head-to-heads. Two datasets matter: the multi-model perplexity comparison from the HQQ paper, and an independent Llama-3-8B-Instruct benchmark spanning quality (MMLU), safety (WMDP), and speed.

3.3 QAT: quality at 4-bit by design

Post-training quantization (everything above) asks "how little does 4-bit hurt a model trained at 16-bit?" Quantization-aware training asks the opposite: "how good can a model be if it trains with 4-bit weights in the loop?" Google's Gemma 3 QAT releases in 2025 made this the standard answer for 8 GB machines, because they delivered 12B-class quality at a file size that fits — with the vendor explicitly blessing the 8 GB tier.

A definition before the experiments, because "offloading" hides the key design question. In every scheme, some tensors live in VRAM and some in system RAM, and the token's computation hops between pools. The four architectures differ in what they choose to demote and when they move it:

4.1 Architecture A — layer offload: the experiment everyone runs first

llama.cpp's --n-gpu-layers N puts the first N transformer layers on the GPU and runs the rest on CPU. It is the oldest trick in local inference and the one whose failure mode is most instructive: because every token must traverse every layer, offloaded layers put the token on the slow pool once per offloaded layer. Speed degrades smoothly toward CPU-only speed — a cliff in the making, exactly as Law 1 says.

4.2 Architecture B — expert offload: the discovery that changed the game

Mixture-of-Experts models changed the arithmetic of offloading in one specific way: only a small fraction of each MoE layer's weights — the experts the router selects — participate in any given token. If the rarely-routed experts live in system RAM, a token only pays the PCIe trip when the router actually calls for them. Offloading went from "pay on every layer" to "pay probabilistically on a fraction of layers," and the results were dramatic enough to reorganize the entire low-VRAM community around MoE models in 2024–25.

# gpt-oss-20b, full context, 22 MoE layers whose experts live in CPU RAM
llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048 --n-cpu-moe 22

# ...and the same card can even orchestrate the 120B model:
# gpt-oss-120b, 32k context, 35 expert layers on CPU
llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 32768 --jinja -ub 2048 -b 2048 --n-cpu-moe 35

4.3 Architectures C and D — the research systems

5.1 Same model, nine machines

When OpenAI released gpt-oss-20b in August 2025, the llama.cpp team published a guide with llama-bench numbers across their whole device lab — and, accidentally, produced the best controlled comparison of low-resource silicon in existence. Every run below is the same GGUF file, same build, same flags (-fa 1 -b 2048 -ub 2048), measuring generation (tg128):

5.2 Apple's unified memory: the accidental LLM machine

Apple never marketed the M-series as inference hardware, but unified memory — one pool, GPU-addressable, high bandwidth — happens to be exactly what the bandwidth law wants, and the gpt-oss guide's Mac results made that unmistakable. The guide's capacity notes are as instructive as its benchmarks: 16 GB Macs must offload some experts (--n-cpu-moe 12 -c 32768), while 8 GB Macs cannot run gpt-oss at all — the OS reserves too much of the pool. That last fact is worth a moment of attention from readers with 8 GB discrete cards: unified memory means GPU and OS compete for the same 8 GB, whereas your rig's 8 GB VRAM is dedicated and your 32 GB RAM absorbs the OS for free. On paper the same number, in practice different machines.

The Apple story also carries a software-lesson the community keeps relearning: the MLX framework (Apple's array library, with its own LLM stack) is typically 10–30% faster than llama.cpp's Metal backend on M-series for decode — with some independent measurements showing up to ~3× on specific models where llama.cpp's Metal kernels were immature — while llama.cpp sometimes wins at long context where its attention kernels are better tuned. Engine choice is a per-model, per-workload empirical question (Law 5 again), never a religious one.

5.3 The absolute edge: Raspberry Pi and friends

6.1 The ladder, rung by rung

Speeds are community-measured on 8 GB cards (RTX 3070/3070 Ti-class) from the experiments in §§4, 10; they will drift ±30% with CPU and RAM generation — a DDR5 machine offloads experts meaningfully faster than DDR4, exactly as the bandwidth law predicts. Note what the ladder is not: there is no rung for dense 30B+ or 70B models. At Q4 those need 17–40 GB of weights; architecture A (§4.1) makes them technically loadable but the 4–9 t/s price was judged unacceptable by nearly every experimenter who tried, and MoE rungs dominate them at equal or better speed and quality.

6.2 How the community actually chose

Reading the experiment threads end-to-end, a clear decision pattern emerges — the same one the model recommendations settled on across 2024–25:

1. Start at rung 3–4, not rung 1. The MoE rungs are why an 8 GB card is a viable agent machine in 2025. gpt-oss-20b and Qwen3-30B-A3B both activate ~3.3–3.6B parameters, so their decode cost resembles a 3B model's while their knowledge resembles a 30B model's. Every Poor GPU Club result (§10.1) is a rung 3–4 configuration.
2. Keep a rung-1 dense model loaded for utility calls. Agent frameworks need cheap structured outputs — reranking, schema validation, summaries. A Q4 8B at 80+ t/s fills that role, or run it fully on CPU to keep VRAM for the orchestrator (32 GB RAM runs a Q4 8B at a usable ~8–12 t/s with plenty left for the OS).
3. Match context to the rung. Rung 1–2 models leave 1.5–3 GB for KV — 8–32K tokens comfortably. Rung 3–4 models have almost nothing left after weights, which is why every real recipe in §10–11 pairs them with KV quantization (§7.1) and modest windows (16–32K).
4. Tool-calling support is a rung requirement, not a nice-to-have. An agent model must emit structured tool calls reliably — historically the failure mode of small models. gpt-oss ships a native tool-calling format (harmony), Qwen3 models were post-trained for it, and llama.cpp's --jinja flag enables their chat templates. The archive's agent recipes all live on rungs with proven tool-calling; community lore treats everything below rung 2 as "chat only."

6.3 A note on engines

Why this paper's recipes are llama.cpp-shaped: because the archive is. llama.cpp is where the 8 GB community measures things — its GGUF format, --n-cpu-moe, KV quantization flags, and llama-bench instrumentation are the substrate of nearly every result reproduced here. That said, the engine landscape has real alternatives, and the honest summary after reading the experiments is: Ollama is llama.cpp with a model manager (measurably 10–20% slower at the edge tiers per the Pi comparisons, but the easiest on-ramp); LM Studio is llama.cpp with a GUI and a good memory HUD for learning your own VRAM budget; MLX is the Mac-native pick (§5.2); vLLM/ExLlamaV2 serve AWQ/GPTQ at high throughput but generally want the model to fit in VRAM — which on 8 GB means small models, so they sit off this paper's critical path. For agents specifically, one further point decides it: your agent framework needs an OpenAI-compatible endpoint with tool-calling support, and llama.cpp's llama-server --jinja provides exactly that (§9.3).

7.1 KV quantization: the cheapest context doubler

llama.cpp exposes KV precision with --cache-type-k / --cache-type-v (q8_0 and q4_0 are the useful tiers). The community's hands-on finding is consistent across model families: q8_0 KV costs essentially nothing in quality while halving the cache, and q4_0 halves it again with visible-but-tolerable degradation on summarization-heavy tasks. The Poor GPU Club thread (§10.1) shows the flag in live use — one user's Qwen3-30B-A3B run carries type_k/type_v q8_0 in its benchmark table, exactly because rung-4 models have no KV budget to waste.

The one caveat the experiments surfaced: key and value tensors quantize differently well — the KIVI paper (below) traced this to per-channel vs per-token outlier structure — which is why llama.cpp lets you set K and V precision independently. Practical default: K at q8_0, V at q8_0 or q4_0.

7.2 The research record: StreamingLLM, H2O, KIVI, PagedAttention

8.1 Seven levers, ranked by community payoff

8.2 Measuring honestly: the llama-bench discipline

Every credible experiment in this archive used the same loop, and Law 5 says you should internalize it before touching a single flag. The loop, plus the two traps the community documents most often:

# 1. baseline: one change at a time, same prompt lengths both sides
llama-bench -m model-Q4_K_M.gguf -fa 1 -p 512 -n 128

# 2. sweep the placement parameter (the one that matters most, §4)
llama-bench -m model.gguf -ngl 99 -ncmoe 24 -fa 1
llama-bench -m model.gguf -ngl 99 -ncmoe 28 -fa 1   # repeat until t/s peaks

# 3. watch VRAM *while* it runs — loading ≠ fitting (Windows lies)
nvidia-smi -l 1          # Linux; on Windows: Task Manager > GPU > "GPU Memory"

# 4. write it down. The Poor GPU Club thread is literally this loop, in public.

9.1 The memory anatomy of an agent loop

An agent turn appends four things to context: the model's reasoning, the tool call it emitted, the tool's raw output, and the framework's bookkeeping (tool schemas ride along in the system prompt from turn 0). A representative loop — one tool call per turn, moderately verbose tools — adds ~0.8K tokens per turn, and that compounds:

Reading the figure against the rig's budgets produces the three numbers every 8 GB agent operator should know cold: 16K context is the comfortable default (q8 KV, ~1 GB, leaves headroom), 32K is the practical ceiling with q8 KV on rung-4 models, and past 32K you are trading model rungs for context — a smaller model with more context usually beats a bigger model that crashes. The Figure-3 curve is exponential in your patience: every extra turn costs the same tokens but progressively fills the only pool that matters.

9.2 Patterns that survived contact

Four agent-design patterns recur across the successful low-resource builds in this archive. They are ordered by leverage:

Pattern 1 — Context compaction (the load-bearing one)

When context crosses a threshold (the green line in Figure 7), summarize the oldest turns into a compact running brief, drop verbatim tool outputs, and keep four things verbatim: the original task statement, the attention-sink prefix (the system prompt's first tokens — §7.2's StreamingLLM finding), the latest two turns, and any facts the task depends on. Implemented in ~30 lines in the agent harness (not the model), it converts an exponential context curve into a sawtooth and is the difference between 15-turn tasks that finish and ones that die. Every serious agent framework of 2024–25 grew a compactor for exactly this reason.

Pattern 2 — Tool-result triage at the boundary

Never paste raw tool output into context. A web search returns 4K tokens; the model needs 200 of them. The archive's working pattern is a fixed triage at the tool boundary — head/tail truncation, structural extraction (parse the JSON, keep the fields the task needs), or a cheap summarize pass on a rung-1 model. This is the agent-side twin of KV quantization: it shrinks the bytes that ever enter the cache.

Pattern 3 — One orchestrator, cheap specialists

The rung-4 MoE orchestrates; a rung-0/1 dense model (often CPU-resident, §6.2) handles utility calls — rerank, schema-fix, summarize. The two-model split works on the 8 GB rig precisely because the specialist can live in the 32 GB RAM pool where its 5–10 t/s is irrelevant to latency-critical paths, and because the orchestrator's VRAM share is never contested mid-task.

Pattern 4 — Retry with degradation, never crash

OOM at turn 12 loses the task. The working pattern: catch the server's OOM/timeout error, halve the context window, compact harder, reload, and resume from the brief. llama.cpp's server returns clean errors on failed allocations, which makes this pattern mechanical — and it converts the rig's hard memory ceiling into a soft one.

9.3 A reference stack, end to end

Here is the stack the archive's evidence points to, in one diagram-shaped paragraph and one runnable artifact. Serving: llama.cpp llama-server with a rung-4 model, --jinja for tool-call templates, expert offload tuned per §8's sweep, q8_0 KV, 16–32K context, FlashAttention on. Protocol: the server's OpenAI-compatible endpoint, so any standard framework can drive it. Agent harness: whatever you like — the community's experiments span LangGraph, claude-code-style CLI loops, and 300-line custom harnesses — with Patterns 1–4 implemented at the harness level. Observability: nvidia-smi in a side terminal, always (Law 5).

The minimal version, complete and runnable, is short enough to be its own proof that "agent" need not mean "framework":

import json, requests

SERVER = "http://127.0.0.1:8080/v1/chat/completions"
TOOLS = [{
    "type": "function", "function": {
        "name": "read_file",
        "description": "Read a text file from disk",
        "parameters": {"type": "object",
            "properties": {"path": {"type": "string"}},
            "required": ["path"]}}}
# triage at the boundary: Pattern 2 — tools may return at most this many chars
MAX_TOOL_CHARS = 1500

def run_tool(name, args):
    if name == "read_file":
        text = open(args["path"], errors="replace").read()
        return text[:MAX_TOOL_CHARS] + ("...[truncated]" if len(text) > MAX_TOOL_CHARS else "")
    return "unknown tool"

def agent(task, max_turns=12):
    msgs = [{"role": "system", "content":
             "You are a precise agent. Use tools. Be terse."},
            {"role": "user", "content": task}]
    for turn in range(max_turns):
        r = requests.post(SERVER, json={"messages": msgs,
            "tools": TOOLS, "temperature": 0.3}).json()
        m = r["choices"][0]["message"]
        msgs.append(m)
        if not m.get("tool_calls"):
            return m["content"]          # final answer
        for tc in m["tool_calls"]:   # execute, append results
            out = run_tool(tc["function"]["name"],
                           json.loads(tc["function"]["arguments"]))
            msgs.append({"role": "tool",
                         "tool_call_id": tc["id"], "content": out})
    return "max turns reached"

if __name__ == "__main__":
    print(agent("Read config.yaml in this directory and list every API key name it defines."))

Swap in your task and tools; the loop's shape is the production pattern minus compaction (add Pattern 1 by summarizing msgs[2:-4] when len(json.dumps(msgs)) crosses your threshold — the exact checkpoint logic from Figure 7's green line).

10.1 The Poor GPU Club: 8 GB VRAM versus two MoE flagships

10.2 The official 8 GB recipe: llama.cpp's gpt-oss guide

The llama.cpp team's gpt-oss launch guide (source of §5's cross-silicon table) is the "official" counterpart to the Poor GPU Club: same model, same engine, sanctioned flags per hardware tier. Its 8 GB-relevant guidance, condensed:

One further warning from the guide deserves verbatim status, because it is the single most common way 8 GB users silently destroy their performance — Windows will over-commit VRAM and swap instead of failing:

10.3 Four Raspberry Pis: the floor of the experiment

11.1 The steps

1. Establish the baseline machine. Confirm 32 GB RAM, dual-channel populated, XMP/EXPO enabled (worth 5–20% on offload-heavy setups, §8.1 lever 7). Update the GPU driver. Close the browser. Open a terminal with nvidia-smi -l 1 and leave it running — this is your truth-teller for the rest of the session.
2. Install llama.cpp with CUDA. Prebuilt release binary, or build from source with -DGGML_CUDA=ON. Verify with llama-server --version and a CUDA device listing on startup.
3. Download a rung-4 model and its smaller sibling. From the ladder: Qwen3-30B-A3B IQ4_XS (or UD-Q4_K_XL) plus a Q4_K_M 8B for utilities. Use the official GGUF repos (ggml-org, Unsloth, bartowski all publish calibrated quants).
4. Sweep the placement. Run the §8.2 llama-bench loop: -ngl 99 -ncmoe 24, then 26, 28, 30… watch t/s climb then fall as VRAM pressure builds, and watch nvidia-smi to stay under ~7.4 GB. Keep the peak. This is the single highest-value hour in the playbook — the Poor GPU Club's 31→34 t/s came from exactly this step.
5. Start the server, tuned. The recipe below. Verify with one chat completion and one tool call from the §9.3 loop before building anything larger.
6. Wire the agent harness. The 60-line loop from §9.3, or your framework of choice pointed at http://127.0.0.1:8080/v1. Implement Patterns 1 and 2 (compaction + triage) before your first long task — retrofitting them after a mid-task OOM is how people learn Law 3 the hard way.
7. Run a real task end-to-end. Something honest: "read this repo, find the bug the test describes, fix it, run the tests." Watch the context sawtooth, watch VRAM, and record t/s at turn 1 vs turn 10. If turn 10 is much slower, your context discipline slipped — that diagnosis, not hardware, is the usual finding.

11.2 Copy-paste recipes, annotated

# Qwen3-30B-A3B as the orchestrating agent on 8 GB VRAM + 32 GB RAM
#   -ngl 99       : attention + shared layers fully on GPU
#   -ncmoe 28     : expert tensors for 28 of 48 MoE layers live in RAM (SWEEP THIS, §11.1 step 4)
#   -fa           : FlashAttention (lever 1)
#   --cache-type-k/v q8_0 : halves KV so 32K context fits (§7.1)
#   --jinja       : native tool-calling template for the agent loop
llama-server -m Qwen3-30B-A3B-IQ4_XS.gguf \
  -ngl 99 -ncmoe 28 -fa 1 \
  -c 32768 -ctk q8_0 -ctv q8_0 \
  -b 2048 -ub 2048 --jinja \
  --host 127.0.0.1 --port 8080

# Official-guide-derived 8 GB configuration (§10.2):
llama-server -hf ggml-org/gpt-oss-20b-GGUF \
  --ctx-size 32768 --jinja -ub 2048 -b 2048 \
  -ngl 99 --n-cpu-moe 16 -fa 1
# n-cpu-moe 16 buys speed (fewer experts on CPU) at the cost of context;
# n-cpu-moe 22 buys full context at ~2/3 of the speed. Pick per task.

# The rung-1 specialist on CPU only — 32 GB RAM runs it at ~8-12 t/s
# while the GPU stays fully committed to the orchestrator.
llama-server -m llama-3.1-8b-Q4_K_M.gguf -ngl 0 \
  -c 8192 -t $(nproc) --host 127.0.0.1 --port 8081

# Throughput, not latency: huge batches, everything prompt-processing-friendly
llama-server -m model.gguf -ngl 99 -ncmoe 28 -fa 1 \
  -c 32768 -b 4096 -ub 4096 --jinja
# pair with -np 2..4 parallel slots when the framework can use them (§8.1 lever 3)

11.3 The verification checklist

12.1 Out-of-memory and crash matrix

12.2 "Why is it slow?" — the decision path

Active parameters

The fraction of an MoE model's weights that participate in a given token's computation (gpt-oss-20b: 3.6B of 21B). Along with quantization, the master variable of the bandwidth law.

Attention sink

Early-sequence tokens that absorb disproportionate attention mass; evicting them from cache collapses quality (StreamingLLM). Reason your agent's system prompt prefix is sacred.

CPU offload (expert) — --n-cpu-moe

Architecture B: keep attention/shared weights in VRAM, expert tensors in system RAM. The 2024–25 answer to small VRAM.

CPU offload (layer) — --n-gpu-layers

Architecture A: the first N transformer layers on GPU, the rest on CPU. Simple, scales poorly (§4.1).

FlashAttention

Fused, memory-lean attention kernel; ~free speed and VRAM on every modern build. If your engine has a flag for it, it should be on.

GGUF / K-quants / I-quants

llama.cpp's model container and its quant families. K-quants (Q4_K_M…) are block-scaled grids; I-quants (IQ4_XS…) add importance-matrix calibration for better quality per bit at low widths.

GQA (grouped-query attention)

Sharing K/V heads across query heads — the architecture change that cut KV-per-token ~4× on modern 7–9B models. Assumed by every current model in the ladder.

KV cache

Per-token key/value memory saved so past tokens aren't recomputed; grows linearly with context (§2.3's formula). The agent-relevant memory cost.

MoE (mixture of experts)

Architecture with many specialist FFN blocks ("experts") and a router selecting a few per token. Large total knowledge, small per-token compute — the enabling trick of low-VRAM inference.

MXFP4

The 4-bit microscaling format gpt-oss models ship in natively (block-scaled, hardware-friendly). A vendor-grade answer to "which 4-bit."

PagedAttention

Treating KV memory like OS pages — on-demand, shareable, near-zero waste (vLLM). Why parallel agent slots are affordable.

pp / tg (prompt processing / token generation)

The two halves of inference latency: compute-bound prefill (the prompt read) and bandwidth-bound decode (the answer). Different bottlenecks, different medicine.

QAT (quantization-aware training)

Training with quantization in the loop so 4-bit weights are compensated at the source — Gemma-3-12B QAT's route onto 8 GB cards at 12B quality.

Speculative decoding

A small drafter proposes tokens; the big model verifies in one batched pass. Output-exact, 2–3× where it applies (§8.1).

Unified memory

One physical pool shared by CPU and GPU (Apple M-series, integrated GPUs). No PCIe wire — bandwidth is honest and capacity is shared with the OS.

VRAM over-commit

Windows allocating more GPU memory than exists and silently swapping — the "it loaded, why is it 2 t/s" trap. Diagnosed only by watching real usage.

Peer-reviewed systems papers

Official benchmarks and guides

Community experiments

A note on data hygiene: figures 1–4 and 6–7 are plotted from the numbers above; Figure 5 mixes measured points with one estimated anchor (the full-GPU tier of a 24 GB card), labeled as such in its caption. Where sources disagreed (e.g., MLX vs llama.cpp Mac speeds), the disagreement itself was reported rather than averaged away. Reader-measured numbers will drift from these as engines and drivers improve — the archive's claims are about the shape of the relationships, which has been stable since 2023.