From One Giant FFN to 384 Tiny Experts

§Abstract & How to Read

What this paper argues, how it is sourced, and how to navigate four parts and sixteen chapters.

Between November 2023 and September 2026, DeepSeek shipped eight generations of large language models, and the public conversation tracked their attention stacks almost exclusively — Multi-head Latent Attention in V2, DeepSeek Sparse Attention in V3.2, Compressed Sparse Attention in V4. This paper is about the other half of the Transformer. The feed-forward network (FFN) is where a language model keeps most of its knowledge and most of its parameters: in a conventional decoder block, roughly two-thirds of the non-embedding weight lives in the FFN, and roughly two-thirds of the per-token arithmetic is spent there. Whoever restructures the FFN restructures the cost base of the entire model. That is precisely what DeepSeek did, five times in a row.

The arc is easy to state and demanding to execute. DeepSeek LLM 7B/67B (2023) carried one giant SwiGLU FFN per layer, inherited almost verbatim from LLaMA. DeepSeekMoE (January 2024) replaced it with fine-grained routed experts plus always-on shared experts — a design whose 16.4B-parameter instantiation matched LLaMA2-7B with about 40% of the computation. DeepSeek-V2 (May 2024) scaled that design to 236B total parameters with 21B activated, paying for scale with three auxiliary balance losses and a device-limited routing scheme. DeepSeek-V3 (December 2024) industrialized it: 671B total, 37B activated, 256 routed experts and 1 shared expert per layer, load balanced by a bias term instead of an auxiliary loss, trained in FP8 on 14.8T tokens for 2.788M H800 GPU-hours, and extended with Multi-Token Prediction modules that later served as speculative-decoding draft heads with an 85–90% token acceptance rate. DeepSeek-V4 (April 2026) pushed the same paradigm to 1.6T total with 49B activated, changed the routing affinity to Sqrt(Softplus), converted the first layers to hash-routed MoE, clamped its SwiGLU gates to stabilize trillion-parameter training, and quantized expert weights to FP4. DeepSeek-V4.1-Flash (September 2026) made the feed-forward path itself asymmetric: a 552B multimodal MoE that activates 8B parameters per token during prefill and 16B during decode, with 384 routed experts and 1 shared expert per layer and a 196B conditional memory module — Engram — built from multi-head hashing and context-aware gating.

Every configuration number in this paper was verified against the primary technical reports and, where possible, against the config.json files published with each checkpoint. Where a number is derived arithmetic rather than a quotation (parameter shares, FLOP splits), the derivation is shown inline so it can be checked with a calculator. Chapter 14 compiles the full evidence tables; Chapter 15 places the design against Mixtral, Qwen3, Kimi K2, GLM-4.5 and Llama 4; Chapter 16 tells you how to actually serve one of these things. Readers who only want the narrative can read the chapter subtitles and the part dividers; readers who want to rebuild the argument can start at Chapter 14 and work backwards.

01The Two-Thirds Problem

What the FFN actually does, why it owns most of the parameters, and why 2023’s economics made conditional computation inevitable.

The decoder-only Transformer is made of two alternating machine families. The attention block moves information between positions — it is the only place where token i can read token j. The feed-forward network, by contrast, never looks across positions at all. It takes each token’s post-attention representation u and pushes it through a position-wise two-layer MLP, on the theory that the model’s factual and procedural knowledge — what a protocol header looks like, how a metaphor resolves, which digits follow which — can be stored in these big matrices and retrieved on demand. Attention is the retrieval index; the FFN is the library.

Since the GPT-3 era, the library has been built to a standard blueprint. A projection up from the model dimension d to an intermediate dimension d_ff, a nonlinearity, and a projection back down. The modern refinement is SwiGLU, standard since LLaMA and the direct ancestor of every DeepSeek FFN to date: instead of one up-projection, there are two parallel ones — a value branch and a gate branch — and the gate’s activation modulates the value element-wise before the down-projection:

u      : [d]                          # token representation entering the FFN
z_v    = W_up(u)          : [d_ff]     # value branch:  W_up   in R^{d_ff x d}
z_g    = W_gate(u)        : [d_ff]     # gate branch:   W_gate in R^{d_ff x d}
h      = SiLU(z_g) * z_v  : [d_ff]     # SiLU(x) = x * sigmoid(x)  (Shazeer, 2020)
y      = W_down(h)        : [d]        # down projection: W_down in R^{d x d_ff}

# parameter count: 3 * d * d_ff   (three matrices, not two)
# FLOPs per token: ~6 * d * d_ff  (multiply-accumulate both ways)

The three-matrix structure is the detail that quietly sets the budget of the whole field. An attention block with standard multi-head attention owns approximately 4d² parameters per layer (Q, K, V, O projections). A SwiGLU FFN with the conventional widening ratio of d_ff ≈ (8/3)d owns 3d·(8/3)d = 8d². Per layer, the FFN therefore owns roughly two of every three non-embedding parameters, and — because matrix–vector work costs the same per parameter everywhere — it also burns roughly two of every three multiply-accumulate operations at decode time. The library is twice the size of the index, and it is consulted in full on every single token.

That table is the whole story of why 2023 ended the way it did. In a dense model, knowledge capacity and per-token compute are the same number. Adding parameters to the library means adding arithmetic to every token, forever, because there is no mechanism for consulting part of the library. A dense 70B model must move all 70B weights (or their quantized shadows) through the GPU for every token generated; a hypothetical dense 700B model must move 700B. Capacity scales linearly with cost, and the industry’s appetite for capacity was scaling faster than anyone’s inference budget.

The escape hatch had existed on paper since Shazeer’s 2017 “Outrageously Large Neural Networks”: conditional computation. Split the FFN into many expert sub-networks, install a cheap router that inspects each token and activates only a few of them, and the two locked quantities come apart — total parameters (knowledge capacity) grows with the number of experts, while activated parameters (compute per token) grow with the top-k selection, which you can hold constant. GShard and Switch Transformer proved the concept at scale for machine translation and sparse language modeling, and Mixtral 8x7B would prove it commercially in December 2023. But the 2021-era recipe had known diseases: coarse experts that wasted capacity by learning redundant copies of common knowledge, auxiliary balance losses that fought the language-modeling objective, and routing collapse that left most of the library unread.

1.1 — The three diseases of 2021-style MoE

It is worth naming the three failure modes precisely, because every DeepSeek design decision from 2024 onward is a direct response to one of them. They are the reason the gap between “MoE works in a paper” and “MoE trains a frontier model” stayed open for three years.

Disease one: coarse-grained redundancy. A GShard-style layer splits the FFN into a handful of full-size experts (say 8 or 16) and activates the top 1–2. Tokens that share common linguistic material — which is most tokens, most of the time — arrive at different experts and drag the same common knowledge into each of them. The expert pool fills with near-duplicate encodings of syntax and function words while genuinely specialized knowledge fights for the remaining capacity. Capacity is spent on the average token instead of the unusual one, which is exactly backwards.

Disease two: the auxiliary-loss tax. A router that can learn anything will happily collapse onto a favorite expert, stranding the rest and wrecking expert-parallel throughput. The standard cure is a differentiable balance loss — reward load uniformity with a gradient term — but that gradient flows through the same parameters as the language-modeling loss, and the two objectives are simply not aligned. Turn the loss up and routing balances while model quality degrades; turn it down and quality rises until imbalance strangles throughput. The coefficient becomes a permanent, model-wide compromise.

Disease three: load imbalance in practice. Even with an auxiliary loss, per-batch balance is statistical, not guaranteed. Systems shipped with token-dropping — discarding the tokens that would overload an expert’s capacity factor — which silently deletes training signal and creates a train/inference discrepancy that has to be reasoned about at every deployment.

DeepSeek’s engineers did not invent the idea of curing these diseases; they invented the specific cures that survived contact with frontier scale. Fine-grained segmentation attacks disease one. Shared experts attack disease one from the other side. The auxiliary-loss-free bias attacks disease two. Node-limited routing and redundant expert placement attack disease three. The next three chapters take those cures in the order DeepSeek shipped them.

02The 2023 Baseline: Dense SwiGLU

DeepSeek LLM 7B/67B and DeepSeek Coder: LLaMA’s feed-forward block, adopted wholesale, then deepened instead of widened.

DeepSeek’s first two model families arrived within weeks of each other in November 2023. DeepSeek Coder (1.3B, 6.7B, 33B) targeted code; DeepSeek LLM (7B, 67B) targeted general language. Both were deliberately conventional. The DeepSeek LLM technical report describes the micro-architecture in one sentence’s worth of design: “the micro design of DeepSeek LLM largely follows the design of LLaMA, adopting a Pre-Norm structure with RMSNorm function and using SwiGLU as the activation function for the Feed-Forward Network (FFN), with an intermediate layer dimension of ⁸⁄₃d_model.” Rotary embeddings throughout; Grouped-Query Attention (8 KV heads) on the 67B to cut KV-cache traffic; plain MHA on the 7B. Nothing in the FFN was novel, and that was the point — the novelty budget of the 2023 papers went into data engineering and scaling-law methodology, not blocks.

The one place DeepSeek deliberately deviated was macroscopic shape, and it is a deviation with FFN consequences. The report is explicit: “we expanded the 67B model’s parameters in network depth rather than the common practice of widening the intermediate width of FFN layers, aiming for better performance.” Where a LLaMA-2-70B-class model spreads 80 layers across a wider trunk, DeepSeek LLM 67B stacks 95 layers on a narrower one, keeping d_ff at (8/3)·8192 = 22,016 rather than pushing it to LLaMA-2’s 28,672. Depth buys more sequential refinement steps per token at the same parameter count; width buys more parallel knowledge storage. DeepSeek’s bet on depth was a bet that the knowledge-per-parameter curve had not yet flattened at 67B — a bet the benchmarks (67B outperforming LLaMA-2 70B on code, mathematics and reasoning, with the chat model competing with GPT-3.5) largely vindicated.

Two details in that table deserve a second look, because they matter later. First, the (8/3)d rule is honored with rounding to multiples of 512: 7B’s 11,008 is exactly (8/3)·4,096 rounded up, and 67B’s 22,016 is (8/3)·8,192 rounded up — not LLaMA-2-70B’s 28,672, a number frequently mis-cited to DeepSeek in secondary sources. Second, the 7B and 6.7B coder share their FFN geometry exactly (11,008), which means every conclusion drawn about the 7B’s feed-forward arithmetic transfers to the coder that shipped alongside it.

Why did a lab that would, within eighteen months, ship the most-copied sparse architecture in open weights start with a by-the-book dense model? The DeepSeek LLM report frames the 7B/67B pair as a scaling-law instrument first and a product second — a carefully controlled two-point measurement of how batch size, learning rate and data allocation should co-scale, with the models as proof artifacts. That framing matters for the FFN story, because it meant the 67B dense model existed precisely as a calibrated baseline. When DeepSeekMoE arrived in January 2024 claiming to match “DeepSeek 67B” with a fraction of the computation, it was matching a baseline whose training recipe, tokenizer, corpus pipeline and evaluation harness were all published — an unusually clean comparison, and one DeepSeek would exploit deliberately in every MoE ablation that followed.

The economics, though, were already terminal. A 67B dense model costs roughly 134 GFLOPs per token at decode — and every one of those FLOPs was in service of the full 51.4B-parameter FFN stack whether the token needed it or not. Serving cost, KV-cache pressure aside, is a straight multiple of activated parameters, and DeepSeek’s own scaling-law projections said the next meaningful capability step lived well past 100B dense-equivalent parameters. Something in the two-thirds had to give. On January 5, 2024, something did.

03MoE Before DeepSeek

Seven years of conditional computation, from sparsely-gated LSTM layers to Mixtral — and the specific failure modes DeepSeek set out to fix.

The mixture-of-experts idea is older than the Transformer. Shazeer’s 2017 Outrageously Large Neural Networks inserted sparsely-gated MoE layers between LSTM layers: a noisy top-k gate over a pool of expert feed-forward networks, each expert seeing only the tokens it was selected for. The paper demonstrated models up to 137B parameters in an era when 137B dense was unthinkable, and it introduced the two artifacts that would haunt every MoE paper for the next seven years: a load-balancing auxiliary loss, and soft top-k gating whose batch statistics made deployment awkward.

The idea moved into the Transformer era through the machine-translation systems of 2021. GShard replaced every other FFN in an encoder–decoder translation model with a large pool of experts and top-2 routing, gated by softmax affinities, kept honest by the now-standard auxiliary loss L = α·Σ_i f_iP_i (the product of each expert’s token fraction f_i and average gate probability P_i, minimized at uniform load), and protected at serving time by capacity factors that drop tokens when an expert overfills. Switch Transformer took top-k to its minimal extreme — top-1 — and scaled sparse training to trillion-parameter territory on the T5 stack. And the Hash Layer (Roller et al., 2021) removed learning from routing entirely: the expert is chosen by a hash of the token ID, balanced by construction, training-free to route — and interestingly, this is the idea DeepSeek would revive for V4’s first layers in 2026.

What all of these systems shared was the disease profile of Chapter 1: coarse experts (8–16 full-size FFNs), balance kept by auxiliary losses whose gradient interference was tolerated as a cost of doing business, and deployment-time mitigations (capacity factors, token dropping, dropless oversubscription) that traded training signal for throughput. What none of them had was a theory of what an expert should be for — the pool was homogeneous, the routing was uniform over it, and common knowledge was everyone’s job, therefore duplicated everywhere.

DeepSeek’s January 2024 paper, DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models, opened with that exact framing: expert specialization was the goal, and the two named obstacles were knowledge redundancy and insufficient specialization. Its diagnosis was that both trace back to expert granularity and expert role. The cure came in two moves, and the paper’s abstract states them with unusual economy: “(1) finely segmenting the experts into mN ones and activating mK from them, allowing for a more flexible combination of activated experts; (2) isolating K_s experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts.” The next chapter takes those two sentences apart, because they are the load-bearing sentences of every DeepSeek model since.

04DeepSeekMoE: Fine & Shared

January 2024, arXiv:2401.06066 — split the experts small, quarantine the common knowledge, and let the combinatorics do the work.

4.1 — The two moves

Move one: fine-grained expert segmentation. Take a conventional MoE layer with N full-size experts and top-K routing. Holding total expert parameters and per-token computation approximately constant, split each expert’s FFN intermediate dimension by a factor m: the layer now has mN experts, each 1/m the size, and each token activates mK of them. Nothing about the layer’s budget changed — what changed is the number of distinct subsets of the library a token can consult. The paper’s own arithmetic makes the point vividly: a 16-expert top-2 layer offers C(16,2) = 120 possible expert combinations; split each expert four ways and route top-8 over the resulting 64, and the combination space becomes C(64,8) = 4,426,165,368. Knowledge can now be decomposed finely and learned precisely into different experts, each of which stays more specialized because it is small.

Move two: shared expert isolation. Dedicate K_s experts as always active, outside the routing entirely. Their job is the common knowledge — the syntax, the function words, the universal structure — that would otherwise be re-learned by every routed expert that receives a typical token. With common knowledge quarantined, the routed pool is freed to spend its capacity on the unusual, and redundancy among routed experts is mitigated by construction rather than by hope. The shared experts also serve as a floor of dense computation per token, which turns out to matter for training stability at small batch sizes.

u      : [d]                             # token representation entering the layer
e_i    : [d]  for i = 1..N_r             # centroid of routed expert i (learned)

s_{i,t} = Softmax_i( u_t . e_i )         # affinity of token t to each routed expert
g_{i,t} = s_{i,t}  if  s_{i,t} in Topk(s, K_r)  else  0      # zero out non-selected
g_{i,t} = g_{i,t} / sum_j g_{j,t}        # renormalize over selected experts

h_t' = u_t + sum_{i=1..N_s} FFN_s_i(u_t)              # shared experts: always on
             + sum_{i=1..N_r} g_{i,t} * FFN_r_i(u_t)  # routed: gated top-K
# N_s shared + N_r routed; K_r routed active per token
# (this is the softmax-affinity form; V3 later swaps Softmax for Sigmoid,
#  and V4 for Sqrt(Softplus) -- the rest of the recipe survives unchanged)

4.2 — The evidence ladder: 2B, 16B, 145B

DeepSeekMoE’s validation strategy is one of the cleanest in the sparse literature, because every experiment is a same-corpus, same-hyperparameter comparison against the dense 67B baseline calibrated a month earlier. Three rungs:

At 2B scale, the comparison set was the 2021 pantheon: Hash Layer, Switch (top-1), and GShard (top-2), all with the same total parameters, GShard matched on activated parameters too. DeepSeekMoE with 1 shared + 63 routed ran away with the comparison and nearly closed the gap to a same-size dense model — the theoretical ceiling for any sparse layer. At 16B, the stakes became commercial: a 16.4B-total / 2.8B-activated model, 28 layers of 2 shared + 64 routed experts with the first layer left dense (the authors observed load balance converging especially slowly there), trained on 2T tokens, trading blows with LLaMA2-7B — a model with roughly 2.5× the activated parameters. On the Open LLM Leaderboard the 16B sat above the fitted compute–quality line of every open model in its class.

The 145B rung is the one with the sharpest economic claim, and it deserves to be quoted precisely rather than paraphrased. The paper’s abstract: performance “comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.” That parenthetical is doing real work — it says the compute-matched frontier of fine-grained MoE had not been found yet, only bounded. DeepSeekMoE 145B was explicitly labeled “ongoing” in the paper; the release that finished the thought was DeepSeek-V2, four months later.

4.3 — Why it worked: the ablations

The paper’s ablation studies are where the two moves earn their keep separately. Isolating fine-grained segmentation from shared experts and vice versa, at 2B and 16B scale, produced three findings that have held up in every subsequent DeepSeek release:

• Shared experts are irreplaceable by routed experts. Removing the shared experts and adding equivalent routed capacity back degrades the model measurably; the always-on pool is not a routing gimmick but a distinct functional role. The paper’s probe: replace shared experts with more routed ones and the model loses, because common knowledge gets re-fragmented across the routing.
• Redundancy among routed experts drops. Measured by similarity of expert outputs on the same tokens, the routed pool under DeepSeekMoE is significantly more differentiated than a GShard pool — evidence that the fine grain really is buying specialization, not just parameter slack.
• Knowledge is acquired more accurately. On factuality-oriented probes, the fine-grained + shared combination beats coarse MoE with identical compute — the combination space is being used, not merely afforded.

The authors also drew one boundary worth remembering: they did not push segmentation finer at 16B, “due to the potential reduction in computational efficiency associated with excessively small expert sizes.” Expert granularity is a trade against kernel efficiency and communication fan-out, not a monotone good — a constraint that reappears every time DeepSeek changes expert counts (64 → 160 → 256 → 384) while keeping experts comfortably above the thousand-parameters-per-dimension floor. One more ratio study fixed the family’s shape for its first two years: holding expert count and activation constant, 1, 2, and 4 shared experts scored Pile losses of 1.808, 1.806, and 1.811 — a wash — so DeepSeek standardized on 1 shared : 3 activated routed for scale-ups, which is exactly what 16B (2+6), 145B (4+12) and V2 (2+6) ship. V3 would later depart from the ratio (1 shared : 8 routed), a quiet signal that by 2024 the shared pool’s job had narrowed.

One piece of housekeeping before scaling up: DeepSeekMoE’s 2B/16B experiments still used the 2021 balance machinery — an expert-level auxiliary loss, and (at scale) the prospect of token dropping. Scaling the design to 236B parameters with 8-way expert parallelism would force every one of those compromises to be paid in real GPU-hours. Chapter 5 is the bill.

05DeepSeek-V2: Scale & Its Taxes

May 2024, arXiv:2405.04434 — 236B total, 21B activated, and the first honest accounting of what sparse scale costs in communication and loss-function real estate.

DeepSeek-V2 shipped on May 7, 2024 (with revisions through June 19), and its abstract reads like a bid for a specific title: “a strong Mixture-of-Experts language model characterized by economical training and efficient inference… 236B total parameters, of which 21B are activated for each token.” The model that carried that claim was built from two named inventions — Multi-head Latent Attention on the attention side, DeepSeekMoE on the FFN side — and the headline economics were quoted against the dense 67B baseline of Chapter 2: 42.5% of training costs saved, 5.76× the maximum generation throughput, and (the MLA side of the ledger, out of scope here) a 93.3% smaller KV cache. The feed-forward half of that bid is the subject of this chapter.

5.1 — The configuration, verified

V2’s MoE layer is DeepSeekMoE’s 16B recipe scaled by a factor of 2.5 in expert count. Sixty Transformer layers at hidden width 5,120; the first layer left dense (intermediate width 12,288), the remaining fifty-nine converted to MoE, each holding 2 shared experts + 160 routed experts, every expert a narrow SwiGLU with intermediate width 1,536, and each token routed to 6 of the 160 (plus both shared experts). Affinities are softmax, exactly as in the DeepSeekMoE paper. The full specification, cross-checked against the published config.json:

Two structural notes. First, the residual-plus-shared-experts topology means every V2 token executes 8 expert FFNs (2 shared + 6 routed) of width 1,536 at every MoE layer — roughly 189M FFN parameters activated per layer (derived: 8 × 3 × 5,120 × 1,536), about 90% of what a dense 5,120-wide layer with (8/3) widening would cost — drawn from a pool of 162 experts holding 3.82B parameters per layer. The activated-FFN budget is deliberately slightly under the equivalent dense model’s; the pool it is drawn from is more than an order of magnitude larger. Second, the paper credits an early piece of systems engineering that later releases would build on: “we overlap the computation of shared experts with the expert parallel all-to-all communication” — the always-on experts are placed on the computation schedule as free filler for the routing network’s communication gaps, the first appearance of a pattern that culminates in DualPipe.

5.2 — Device-limited routing: taxing communication down to law

Fine-grained experts create a communication problem that coarse MoE never faces. With 160 experts striped across 8 devices per layer (D = 8) under expert parallelism, a token’s 6 selected experts can land on up to 6 different GPUs, and every distinct device multiplies the all-to-all traffic that token generates. V2’s answer is device-limited routing: after the affinity scores are computed, each token’s candidate set is restricted so that its experts span at most M devices, with M = 3 in production. Selection is still affinity-driven — the constraint just prunes which experts are eligible — and the paper reports the constraint barely touching model quality while capping each device’s outbound transmissions at MT hidden states.

5.3 — Three losses and a dropper

V2’s load-balancing machinery is where the 2021 inheritance is paid in full. Three auxiliary losses run alongside the language-modeling objective, each with its own coefficient, each fighting a different imbalance:

• Expert-level balance, L_ExpBal = α₁·Σ_i f_iP_i over the 160 experts, with α₁ = 0.003 — the classic GShard/Switch term, defending against routing collapse at the expert level.
• Device-level balance, L_DevBal = α₂·Σ_i f′_iP′_i over the 8 devices (α₂ = 0.05), computed on device-aggregated expert loads — keeping the physical placement of experts from going hot and cold.
• Communication balance, L_CommBal = α₃·Σ_i f″_iP″_i (α₃ = 0.02) — because device-limited routing bounds outbound traffic, this third loss pushes inbound traffic toward uniformity too, so each device both sends and receives around MT hidden states.

And for whatever imbalance survives the three losses, V2 trains with a device-level token-dropping strategy: each device’s compute budget is fixed at capacity factor 1.0 (the perfectly balanced load), tokens are dropped in order of ascending affinity score until the budget is met, and — a thoughtful detail — tokens belonging to roughly 10% of training sequences are marked undroppable so the model never fully unlearns how to process a full-length un-truncated batch. Evaluation runs dropless.

Read as a design document, V2’s balance stack is an admission: the price of 21B-activated sparsity in 2024 was three gradient terms fighting the LM objective, plus a training-time data-loss mechanism whose train/inference inconsistency had to be reasoned about. The V2 paper itself flags the tension obliquely by tuning α₂ an order of magnitude above α₁ — balance mattered more than quality loss at the device level, because throughput at 8-way EP was non-negotiable. Chapter 6 is about how DeepSeek deleted this entire stack.

5.4 — The family around V2

Three siblings completed the 2024 lineup, and all three are FFN-identical to V2, which is itself a datum: the architecture was considered settled at release.

V2-Lite is worth a paragraph of respect, because its appendix entry in the V2 paper quietly demonstrates how far the DeepSeekMoE design scales down: 2.4B activated parameters delivering usable 128K-context chat and coding, with the expert pool small enough (64) that expert parallelism is unnecessary and the whole 2021 loss stack collapses to a single α₁ = 0.001 term. A generation of local-first MoE serving stacks would later grow up around exactly this configuration class.

06Balancing Without the Damage

The auxiliary-loss problem, stated honestly — and the bias-update idea from Wang et al. (2024) that DeepSeek would ship as V3’s routing core and, eventually, as a single config field: noaux_tc.

Why exactly are auxiliary balance losses a problem, rather than a nuisance? Because they are a gradient-level parasite on the language-modeling objective. The balance term’s optimum — uniform expert load — is defined over batches, while the LM loss’s optimum is defined over the data distribution. Whenever the data genuinely wants concentrated routing (a code document wants code experts), the two objectives issue opposing gradient commands through the same router parameters, and the coefficient α becomes a permanent dial deciding which command wins on average. V2 ran three of these dials simultaneously (0.003, 0.05, 0.02), and the V3 technical report states the conclusion plainly in its motivation for change: “too large an auxiliary loss will impair the model performance.”

The alternative DeepSeek adopted had just been published by its own researchers as Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts (Wang et al., 2024). The idea is disarmingly simple once seen: stop using gradients to balance load, and use selection arithmetic instead. Attach a bias b_i to every routed expert. Add the bias to the affinity score only when deciding which experts make the top-K cut. Do not add it to the gate value that scales the expert’s output. Then, once per training step, look at which experts actually got overloaded across the batch and nudge their biases:

# selection (training and inference):
g_{i,t} = s_{i,t}  if ( s_{i,t} + b_i ) in Topk( s_{j,t} + b_j, K_r )  else 0
# NOTE: b_i perturbs ONLY the top-k cut. The gate value g_{i,t} uses the raw
# affinity s_{i,t}, so the model's outputs never see the bias.

# update (once per training step, over the whole batch):
for expert i:
    if   expert_load(i) > balanced_level:  b_i -= gamma   # overloaded: harder to select
    elif expert_load(i) < balanced_level:  b_i += gamma   # underloaded: easier to select

# gamma = "bias update speed". V3: 0.001 for the first 14.3T tokens, then 0.0.
# No loss term. No gradient. No interference with the LM objective.

The elegance is in what the bias cannot do. Because it never multiplies an expert’s output, it cannot distort the function the layer computes — it only changes which narrow SwiGLU blocks get executed for which tokens, within a model that is free to route anywhere it likes. And because the update is a signed nudge computed from batch statistics, it closes the loop at exactly the timescale (one step) where imbalance actually develops, rather than pulling a gradient average over an entire corpus. Load balancing becomes a control system instead of a taxation scheme.

One refinement completes the design. The bias operates per training step, across a whole batch — which leaves a pathological corner case: a single long sequence whose tokens pile into one expert, hidden inside a batch whose global load looks fine. DeepSeek’s answer, shipped with V3, is to keep a complementary sequence-wise balance loss with a deliberately tiny coefficient (V3: α = 0.0001; V4: 0.0001; V4’s paper calls its weight “slight”). The auxiliary-loss era thus ends not with abolition but with demotion: from three dominant gradient terms fighting the objective, to one ghost of a term guarding a corner case the control loop cannot see.

That is the full theory of the modern DeepSeek router. The remaining chapters of Part III are about what the theory bought at frontier scale: a 671B-parameter model trained without a single token dropped, an FFN stack whose 256-expert layers ran on 2048 GPUs in FP8, and a feed-forward side-effect — Multi-Token Prediction — that turned out to double as a speculative-decoding engine.

07DeepSeek-V3: 256 Experts, No Losses

December 26, 2024, arXiv:2412.19437 — 671B total, 37B activated, one shared expert, sigmoid affinities, and the first frontier-scale training run with no token ever dropped.

DeepSeek-V3’s abstract compresses its FFN story into one clause — “adopts… DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2” — and the compression is honest: the layer-level design is a direct maturation of V2’s, not a redesign. What changed is every number around the design, and the deletion of the 2021 inheritance. The specification, again verifiable field by field against the released config.json:

Each row of that comparison is a deliberate decision, and the paper documents them all. The shared-expert count halving from 2 to 1 is the fine-grained philosophy taken to its conclusion: with 256 narrow experts and 8 activated, common knowledge no longer needs two dedicated reservoirs. The first three layers staying dense (up from one) mirrors the DeepSeekMoE observation that early-layer routing is the slowest to reach balance — at 61 layers and 671B parameters, the risk is worth three dense layers of insurance. And the affinity switch from softmax to sigmoid is subtle but consequential, stated in exactly one sentence: “Slightly different from DeepSeek-V2, DeepSeek-V3 uses the sigmoid function to compute the affinity scores, and applies a normalization among all selected affinity scores to produce the gating values.” Softmax forces the 256 affinities to sum to one before selection — experts compete globally for probability mass. Sigmoid scores each expert independently, so a token can be genuinely hot for many experts (or none), and only the selected 8 are renormalized to become gate values. The router stops allocating a fixed budget and starts making 256 independent yes/no assessments — the scoring function that best matches a bias-perturbed top-K selection, which is exactly what the aux-loss-free router performs on it.

7.1 — The bias, run at scale

Chapter 6’s mechanism ships here, and the training recipe reveals its operating envelope: γ (bias update speed) set to 0.001 for the first 14.3T tokens, then to 0.0 for the final 500B. Read that schedule as an engineer: the control loop runs for 96.6% of pre-training, and then DeepSeek simply switches it off — once load is balanced, the equilibrium holds statically and the biases freeze into constants. The complementary sequence-wise loss runs throughout at α = 0.0001, three orders of magnitude below V2’s dominant device-level coefficient, and the paper’s stated role for it is only “to avoid extreme imbalance within any single sequence.”

The payoff is stated with unusual firmness for a technical report: “DeepSeek-V3 does not drop any tokens during training” — and not at inference either, with balance maintained instead by deployment-side expert placement (Chapter 9’s EPLB is the open-sourced form of that). The token-dropping era, with its 10% undroppable sequences and train/inference discrepancy, ends in this paragraph of the V3 paper.

7.2 — Routing under a communication budget, again

Node-limited routing is V2’s device-limited routing lifted one level of the hierarchy, because V3’s expert parallelism spans nodes: 64-way EP across 8 H800 nodes (2048 GPUs total, 16-way pipeline parallel, ZeRO-1 data parallel, no tensor parallelism). Each token’s 8 selected experts must span at most 4 nodes, with the 4 nodes chosen by the sum of each node’s top K_r/M (here, top-2) affinity scores among its hosted experts. The paper quantifies the headroom this leaves: tokens average 3.2 experts per visited node, so the model could scale to 13 routed experts per token at the same communication cost — a number worth remembering when V4 later drops the constraint entirely.

7.3 — The economics, in the paper’s own units

V3’s cost accounting became the most quoted paragraph in open-model history, and every line of it flows from the sparsity arithmetic of the FFN stack: 2.788M H800 GPU-hours for the complete training run — 2,664K hours of pre-training (about 180K hours per trillion tokens, 3.7 days per trillion on the 2048-GPU cluster), 119K hours of context extension, 5K hours of post-training — which the paper prices at $5.576M assuming $2 per GPU-hour. The training ran without a single irrecoverable loss spike or rollback. A 671B-parameter model whose per-token cost is that of a 37B model, trained for the price of a mid-size apartment: the two-thirds problem of Chapter 1, dissolved.

08MTP: The FFN Plans Ahead

Multi-Token Prediction in V3 — the training objective that densifies the gradient signal, the modules that implement it, and the 1.8× decode speedup they deliver for free at inference.

The second structural novelty V3 shipped is not an MoE mechanism at all, but it lives squarely in FFN territory and its fingerprints are on every later release. Multi-Token Prediction (MTP) extends the training objective from “predict token i+1 from the state after token i” to “also predict i+2, i+3, … i+1+D” — one shared backbone, several prediction depths. The declared motivations are denser training signal (each position teaches the model D+1 facts instead of one, improving data efficiency) and representational pre-planning: a hidden state that must serve two future predictions must encode more than a state serving one.

V3’s implementation is sequential rather than parallel, and the paper draws the design contrast itself. Where Gloeckle et al. (2024) attach D independent output heads predicting in parallel from the same trunk state, DeepSeek keeps “the complete causal chain for the prediction of each token at each depth”: depth k consumes the representation produced at depth k−1. The machinery per depth is deliberately cheap and deliberately shared-heavy:

for depth k = 1..D:                       # V3 ships D = 1: predict exactly 2 tokens per position
    # combine main-model state with the EMBEDDING of the token being predicted around
    h'_i^k  = M_k [ RMSNorm( h_i^(k-1) ) ; RMSNorm( Emb( t_{i+k} ) ) ]
              # M_k in R^{d x 2d}: concatenation [d ; d] projected back to d
              # k = 1 reads h_i^0 = the main model's output at position i

    h^k     = TRM_k( h'^(k) )            # one Transformer block per depth (own weights)
    P^k     = OutHead( h^k )             # SHARED with the main model's output head

L_MTP = (lambda / D) * sum_k CrossEntropy( P^k, t_{i+k+1} )   # averaged, weighted by lambda

Every MTP depth adds one Transformer block, one projection matrix M_k ∈ ℜ^d×2d, and nothing else — the embedding table and the output head, the two most parameter-expensive components of a vocabulary-129,280 model, are shared with the trunk. The loss joins the main objective scaled by λ/D. (V4 would publish its operating point: MTP loss weight 0.3 for most of training, 0.1 at learning-rate decay — a strong weight, not a garnish.)

8.1 — The ablation and the two-mode payoff

The V3 paper validates MTP with the same two-scale discipline as its load-balancing ablations: MoE baselines at 15.7B total (1.33T tokens) and 228.7B total (540B tokens), each retrained with a 1-depth MTP module appended, identical data and architecture otherwise. Both scales show the same direction: MTP “consistently enhances the model performance on most of the evaluation benchmarks” — and because the MTP modules can simply be discarded at inference, the comparison is exact: same inference cost, better model, the rare ablation with no asterisk.

The speculative-decoding mode is why MTP matters beyond training-quality folklore. A 1.8× TPS multiplier stacks multiplicatively with everything else in the serving stack: it compounds with the 5.5% activation ratio of Chapter 7 (compute saved), with the KV-cache compression on the attention side (memory saved), and with the FP4 expert weights of Chapter 11 (bandwidth saved). By the V4 era, MTP modules ship as standard equipment — the V4 report adopts the V3 strategy “without modification,” and every serious serving recipe (vLLM, SGLang, LMCache’s published V4-Flash configuration) treats the MTP draft head as a first-class component of the deployed model rather than an optional extra.

There is also a quieter reason MTP belongs in an FFN history. The projection M_k and the per-depth Transformer block are feed-forward machinery in the strict sense — position-wise transformation of concatenated representations — and the module’s design shows the DeepSeek house style applying to a new problem: share the expensive parts (embedding, head), minimize the new parameters, keep the causal chain complete, and make the whole thing optional at deployment. It is the same design grammar as one-shared-expert-plus-256-narrow-ones, transposed from “which FFNs run” to “which tokens get predicted.”

09The Expert-Parallel Machine

R1 freezes the weights while RL runs; DeepSeek open-sources the FFN’s circulatory system — DeepEP, EPLB, DualPipe — in a single February week.

January 20, 2025 brought DeepSeek-R1, and it belongs in this history only as a boundary marker: R1 is V3’s FFN stack with the architecture explicitly untouched — reinforcement learning (GRPO, distilled reasoning data, cold-start SFT) reshaping behavior while the 1-shared-plus-256-routed topology of December 2024 stayed byte-for-byte what it was. The same holds for the year’s interim checkpoints: V3-0324 (March 25, 2025) and R1-0528 (May 28, 2025) were post-training and data refreshes. In an architecture history, 2025’s first five months are a gap — and the gap is the point: the feed-forward design of V3 was considered finished enough to leave alone while the lab threw everything at training method and long context.

What DeepSeek did change in that window was the machine around the experts. February 2025’s “Open Source Week” released, one repo per day, the systems that make a 671B sparse model tractable. For the FFN story, three of the days matter:

It is worth pausing on what this release week says about where the FFN lives now. V3’s routed stack alone holds, by simple arithmetic, 58 MoE layers × 256 routed experts = 14,848 routed experts, each a 2,048-wide SwiGLU of roughly 44M parameters (3 × 7,168 × 2,048 = 44.04M — derived, checkable by hand), for about 654B of the 671B total. That is not a layer anymore; it is a fleet, striped across 8 nodes × 8 GPUs, each GPU hosting on the order of 232 experts, every token consulting 8 of 14,848 under a per-node cap of 4. The research problem has moved: routing quality and expert specialization are now inseparable from communication scheduling, placement, and replication. DeepSeek’s answer was to open-source all of it — training libraries, placement algorithms, profiler data — which is a large part of why the industry’s V3-class serving stacks (vLLM’s DeepSeek recipes, SGLang’s day-0 support, LMCache’s offload work) converged so quickly.

10V3.1 & V3.2: The Frozen Backbone

August 2025 to December 2025 — two flagship releases, one attention revolution, zero FFN changes. What sixteen months of deliberate silence says about a design.

DeepSeek-V3.1 (August 21, 2025) introduced the hybrid-thinking regime — one checkpoint producing both a thinking and a non-thinking mode, with a long-context curriculum stretching to 128K — and its architecture chapter is, from this paper’s standpoint, a null result: the released config is field-for-field identical to V3’s (n_routed_experts: 256, n_shared_experts: 1, num_experts_per_tok: 8, moe_intermediate_size: 2048, first_k_dense_replace: 3). All of V3.1’s budget went into training stages, data, and thinking-mode orchestration. The feed-forward stack had become infrastructure.

DeepSeek-V3.2-Exp (September 29, 2025), and the full V3.2 (December 1, 2025), then staged the year’s real revolution — on the attention side. DeepSeek Sparse Attention (DSA) replaced full attention with a two-stage pipeline: a lightning indexer scoring every cached token cheaply, then fine-grained top-k selection of the entries each query actually attends to. Prefill arithmetic at 128K-plus context collapsed, and the API price followed (DeepSeek’s launch announcement led with a 50%+ inference price cut). But the FFN fields in every V3.2-class checkpoint and serving recipe are the same 1 + 256, top-8, 2,048-wide experts: the vLLM recipe page for V3.2 describes it, in one line, as “MoE model with MLA attention, sparse attention” — the MoE half of that sentence unchanged since the previous December.

An architecture historian should read that table as a finding, not a lull. Between December 26, 2024 and April 24, 2026 — sixteen months — every capability headline DeepSeek shipped (R1’s reasoning, V3.1’s hybrid thinking, V3.2’s long-context economics) was delivered on top of a frozen feed-forward design. The lab’s iteration budget flowed to wherever the marginal inference dollar was being spent, and the V3.2-Exp report is explicit that at 128K context the primary bottleneck had become attention’s KV traffic, not the experts. Meanwhile the FFN side of the ledger matured quietly in deployment: FP8 serving of the 671B checkpoint became the standard recipe across vLLM and SGLang, and EPLB-style redundant-expert placement became routine at scale.

By the end of 2025, then, the DeepSeekMoE stack had survived its first durability test: three major releases, an RL revolution on top of it, and an attention overhaul beside it, with the 1+256 expert layer as the untouched constant. The next escalation — a trillion parameters and a million-token context window — would finally force the feed-forward network itself back onto the workbench. Four changes were waiting: a new affinity, a resurrected 2021 idea, a numerical clamp, and a 4-bit format.

11DeepSeek-V4: Trillion-Parameter Sparsity

April 24, 2026, arXiv:2606.19348 — the fine-grained recipe at 1.6T, five feed-forward changes big and small, and the training-stability folklore written down.

DeepSeek-V4 arrived as a preview pair: V4-Pro with 1.6T parameters, 49B activated and V4-Flash with 284B parameters, 13B activated, both with one-million-token context windows. The attention side (hybrid CSA/HCA) took the headlines; this chapter audits the four feed-forward changes that shipped alongside it, plus one piece of training folklore that finally got written into a technical report. The configurations first, both paper- and config-verified:

11.1 — Change one: no dense layers, hash-routed head

Every DeepSeek model since DeepSeekMoE 16B had kept at least one dense FFN layer at the stack’s entry, for the reason the original paper gave: load balance converges slowest in layer one, and a router that cannot yet route is a liability exactly where every token passes. V4 deletes the exception and converts the first three blocks to MoE with Hash routing (Roller et al., 2021) — the expert chosen by a predefined hash of the token ID. Hash routing needs no learning, is balanced by construction over a large enough vocabulary, and costs one function call instead of 256 dot products. The 2021 idea that DeepSeek’s own 2B ablations beat with learned routing returns, five years later, to solve the one problem learned routing was worst at.

11.2 — Change two: the affinity, once more

The affinity function mutates for the second time: “Different from DeepSeek-V3, we change the activation function that computes the affinity scores from Sigmoid(·) into Sqrt(Softplus(·)).” The paper states it in one sentence and does not argue for it, which is itself information — the scoring function is now treated as a tunable, with Softplus supplying an always-positive, smooth, non-saturating score and the square root compressing its dynamic range. In the released configs the lineage is explicit: V4.1-Flash ships "scoring_func": "sqrtsoftplus", the third and current entry in a sequence that began with Softmax (V2, 2024), passed through Sigmoid (V3, 2024–2025), and arrived here (V4, 2026).

11.3 — Change three: the constraint comes off

V2 capped each token at 3 devices; V3 capped it at 4 nodes; V4 removes the routing constraint entirely — “we remove the constraint on the number of routing target nodes, and carefully redesign the parallelism strategy to maintain training efficiency.” The redesign is the paper’s infrastructure section 3.1, “Fine-Grained Communication-Computation Overlap in Expert Parallelism,” which generalizes the V2-era shared-expert-overlap trick into a full overlap discipline for the all-to-all traffic. The trade documented in Chapter 5 — selection freedom vs communication fan-out, baked into the weights — was finally retired in favor of spending kernel engineering instead of model quality.

11.4 — Change four: clamping the gate

The most unexpectedly valuable paragraph in the V4 report, for anyone who trains large MoE, is §4.2.3, “Mitigating Training Instability.” Its diagnosis first: loss spikes at trillion scale were “consistently tied to outliers in the MoE layers, and the routing mechanism itself appears to exacerbate the emergence of these outliers” — a vicious cycle where outlier activations attract routing, which concentrates gradients, which grows outliers. Its two cures, offered “openly” and explicitly without theory:

• SwiGLU Clamping. Through the entire training of both V4 models, the linear (value) component of every SwiGLU was clamped to [−10, 10] and the gate component capped above at 10. Clamping as a numerical tool goes back years (the report cites Bello et al., 2017 and Riviere et al., 2024), and DeepSeek credits contemporary practice (OpenAI, 2025) — but this is the first frontier-scale public confirmation that it works on the full loss curve of a trillion-parameter MoE, and V4.1-Flash later ships it as the config field "swiglu_limit": 10.0.
• Anticipatory Routing. At step t, compute features with current parameters θ_t, but compute and apply the routing indices using historical parameters θ_t−Δt — fetched one step ahead and cached. Decoupling the router’s update from the backbone’s breaks the outlier cycle at its source. The overhead is bounded to roughly 20% wall-clock when active, and in practice the mechanism is triggered automatically on loss spikes (with a short rollback), run for a while, and switched off — a safety system, not a permanent tax.

11.5 — The supporting cast: mHC and Muon

Two V4 subsystems are not FFN mechanisms but change the FFN’s operating conditions. Manifold-Constrained Hyper-Connections (mHC) widen the residual stream that feeds every expert layer by a factor n_hc = 4, replacing the fixed skip connection with a per-layer mixing matrix B_l constrained to the Birkhoff polytope of doubly stochastic matrices — guaranteeing ‖B_l‖₂ ≤ 1, i.e., a non-expansive residual map that will not amplify activations across 61 stacked layers. (Unconstrained Hyper-Connections, the report notes, “frequently exhibit numerical instability when stacking multiple layers” — the constraint is what makes the depth safe. The projection onto the polytope uses 20 Sinkhorn–Knopp iterations.) Muon, the orthogonalized-momentum optimizer, takes the majority of parameters while AdamW keeps the embedding, the output head, and all RMSNorm weights; V4 is its first frontier-scale deployment, and the report credits it (with the MoE overlap engineering) for the throughput that made 32–33T-token pre-training affordable.

11.6 — FP4: the experts go to four bits

V4’s post-training stage introduced the change with the largest serving consequence: quantization-aware training to FP4, in the OCP MXFP4 format, applied to two components — MoE expert weights, “which are a major source of GPU memory occupancy,” and the attention indexer’s query–key path. The scheme is more elegant than a naive 4-bit cast: FP32 master weights are quantized to FP4 (E2M1) in 1×32 sub-blocks, then dequantized back to FP8 (E4M3) in 128×128 blocks for computation — and the report proves the round trip is lossless, because E4M3’s two extra exponent bits can absorb the sub-block scale ratios as long as those ratios stay within a verified bound. Gradients flow through the same FP8 weights via a straight-through estimator, the existing FP8 training framework is reused without modification, and during inference and RL rollouts the model runs native FP4 weights, halving memory traffic on the expert matrices relative to FP8. The released instruct checkpoints make the policy public: FP4 for expert weights, FP8 for everything else, in both Pro and Flash.

The economic close-out of the V4 chapter is attention-dominated (at one million tokens, V4-Pro runs at 27% of V3.2’s single-token FLOPs and 10% of its KV size; Flash, 10% and 7%) — but the FFN’s contribution is structural: a 49B-activated budget carrying 1.6T parameters of knowledge, with 4-bit expert weights making the fleet fit in a servable footprint. The pre-training recipe that produced it — Muon, mHC, clamped gates, anticipatory routing — is the accumulated folklore of Parts II and III, finally written down as configuration.

12V4-Pro GA & the Flash Lineage

August 2026 — the preview becomes a product line: general availability, a vision-capable Flash, and the pricing that reveals what 4.6% activation buys.

For four months after the April preview, the V4 series soaked in external deployment. On August 13, 2026, DeepSeek-V4-Pro went to general availability as the flagship: the 1.6T/49B checkpoint with million-token context, the FP4-expert instruct build, and the “Pro-Max” maximum-reasoning-effort mode first named in the preview paper. A week later, August 21, 2026, V4-Flash-Vision-Exp added native visual understanding to the 284B/13B line. The release cadence — preview, GA, vision variant — tracks the same maturation curve V3 followed, with one difference: this time the entire expert stack ships pre-quantized, because by 2026 FP4 experts stopped being a deployment option and became the release format.

Serving-side, the V4 generation entrenched two FFN-specific practices. First, MTP became standard deployment equipment: the published serving recipes for V4-Flash (vLLM and LMCache’s configuration guides) wire the MTP module in as the speculative-decoding draft head by default, treating the 1.8×-class decode multiplier as part of the model rather than an optimization. Second, expert-parallel serving became the assumed topology: with 23,424 experts in the Pro model, no single host holds the fleet, and the same DeepEP-style dispatch/combine patterns open-sourced in February 2025 graduated from training infrastructure to inference infrastructure.

The pricing tells the capacity-vs-compute story in the fewest words. At GA, V4-Pro’s API listed at roughly $1.32 per million input tokens ($0.044 on cache hits) and $3.96 per million output tokens — flagship pricing, not budget pricing, because 49B activated is still 49B. The Flash line carried the low end. And then, twenty-eight days after GA, DeepSeek shipped a model that made both of those price points look transitional: a 552B-parameter MoE that runs its prefill at 8B activated parameters — the per-token compute budget of DeepSeek’s own 2023 dense 7B — and its decode at 16B.

13V4.1-Flash: Asymmetry & Engram

September 10, 2026 — a 552B multimodal MoE whose feed-forward budget depends on which half of the network a token is in: 8B activated for prefill, 16B for decode, plus a 196B conditional memory called Engram.

DeepSeek-V4.1-Flash was announced as “the smallest model in our new architecture family” — a phrase worth reading twice, because 552B backbone parameters is only “smallest” inside a family whose next members are expected to scale up from this template, with native multimodality (a vision token vocabulary, image tokens at token id 129,264) and million-token context built in. What makes it a new architecture family rather than an incremental V4 is a single structural split with direct FFN consequences: the Causal Encoder–Decoder (CED) layout. The backbone’s 40 Transformer layers divide into a 20-layer encoder and a 20-layer decoder; the decoder’s global attention reads KV entries projected directly from the encoder’s final hidden state H_L/2 rather than recomputing them per decoder layer. The design cites its inspiration openly — You Only Cache Once: Decoder-decoder Architectures for Language Models (Sun et al.) — and its arithmetic is workload-aware: standard prefill costs O(N·L); CED reduces it to roughly O(N·L/2 + n_win·L/2), nearly halving the cost of processing long inputs.

For the feed-forward stack, the CED split enables the headline asymmetry, quoted from the launch materials in both formulations: the model “activates 16B parameters per token during decode but only 8B parameters during prefill” — or in the announcement’s phrasing, “just 8B active parameters for input, 16B for output.” The workload logic is agentic: an agent reads a million tokens and writes a few thousand, so the read side (encoder) gets the sparser budget — 8B activated, a per-token compute budget on par with the dense DeepSeek LLM 7B of 2023 — while the generation side (decoder), where each token’s quality is paramount and the batch is small, runs at 16B. Compute is allocated by phase, not just by expert; the sparse-activation principle of DeepSeekMoE, which allocates compute per token, gains a second axis that allocates it per workload stage.

Read down that table and the continuity is the story: the same one-shared-plus-many-routed shape DeepSeekMoE proposed in January 2024, the same aux-loss-free selection V3 pioneered, the same Sqrt(Softplus) affinity and SwiGLU clamp V4 introduced — each mechanism surviving its third or fourth generation, now expressed as fifteen lines of a JSON file. What is genuinely new sits outside that table, in a module the family is named for.

13.1 — Engram: conditional memory, 196B parameters

V4.1-Flash ships an Engram module: a 196B-parameter conditional memory. The technical report’s description is precise about mechanism and purpose: “multi-head hashing and context-aware gating to store and retrieve information efficiently without significantly increasing the activated parameter count per token… helps the model manage vast amounts of knowledge while maintaining a compact runtime footprint.” Its intellectual provenance is cited directly — Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models — and the training infrastructure adapted to it is documented too: the Engram’s (large) embedding tables train with Sinkhorn-Balanced Updates — momentum updates followed by Sinkhorn balancing to equalize magnitude across the update matrix, saving optimizer-state memory — while the backbone uses a head-wise Muon.

In the frame of this history, Engram is not a departure from the DeepSeekMoE program — it is the program’s completion. The 2024 insight was that knowledge capacity and per-token compute could be decoupled by routing tokens to small experts. Engram decouples them further, along two new axes: lookup (multi-head hashing selects which memory rows a token consults — the Hash Layer’s 2021 idea, promoted from a whole-layer router to a memory index) and context (gating decides how much of the retrieved memory actually flows into the residual stream). Two hundred billion parameters of knowledge that cost, per token, only what the hash and the gates select. The feed-forward network has finally stopped being a layer at all and become an addressable library.

13.2 — The rest of the generation

Three more V4.1 mechanisms complete the picture, each touching the FFN’s operating conditions. Single-Pass mHC revises the V4 residual-stream mixing so it can be kernel-fused — the “Mega-mHC” kernel reduces activation-memory traffic to (2n+2)d reads and writes, halving it; the residual path that feeds every expert layer got cheaper. Training scale reached 45T multimodal tokens across the pre-training curriculum, with RL post-training including the family’s novel Controllable Reasoning Effort: a scalar 1–100 in the system prompt trading cost for accuracy, enforced during RL by an exponential token-penalty reward k(b) = k₀·exp(−(b−b_min)/τ) — effort made a first-class, billable axis of the model. And quantization shipped as release format: FP8 storage with 32×32 blocks and UE8M0 scales, experts in FP4 — the V4 QAT policy, now the default rather than an instruct-branch option.

The reported results close the loop on the economics that opened this paper: benchmark performance ahead of V4-Pro (the launch cites independent tests placing it ahead on performance, cost, speed and total runtime), agentic coding results including DeepSWE 74.2% and Terminal-Bench 2.1 at 90.6 (V4-Pro: 82.7), and a Codeforces rating of 3471 — from a model whose prefill compute matches the 2023 dense 7B baseline this history started with. The two-thirds problem of Chapter 1, measured against that baseline, is now a 69×-capacity-per-prefill-FLOP separation: 552B of knowledge consulted at 8B of activation, gated by hashes, biases, and gates.

14Compiled Evidence & Timeline

Every feed-forward configuration in the DeepSeek line, one table at a time — then the complete release timeline, and the five generational moves in summary form.

14.1 — The master configuration table

14.2 — The economic claims, as published

14.3 — The complete timeline

14.4 — Five moves, one grammar

Compress the three years into the changes that mattered and the sequence is almost didactic:

1. Split and quarantine (2024). Many small experts for specialization, shared experts for common knowledge — the combinatorial library replaces the monolith.
2. Scale and pay (2024). 236B/21B buys frontier-adjacent quality, paid for in three auxiliary losses, device constraints, and dropped tokens.
3. Delete the taxes (2024–2025). The aux-loss-free bias replaces gradient interference; no tokens drop; sigmoid affinities fit the selector; MTP makes the FFN stack train better and serve faster.
4. Industrialize (2025). FP8, DeepEP, EPLB, DualPipe: the expert fleet becomes first-class systems infrastructure, open-sourced.
5. Push the axes (2026). More experts (384), fewer activated (6, then 4 phases of asymmetry), coarser precision (FP4), earlier conditional computation (hash routing), and conditional memory (Engram) — capacity per activated parameter, chased along every axis the architecture offers.

15The Rivals: A DeepSeek-fied World

What Mixtral, Qwen, Kimi, GLM and Llama learned from the lineage — and which of DeepSeek’s bets each of them copied or declined.

By 2025, the fine-grained-plus-shared recipe was no longer DeepSeek’s trade secret; it was the default grammar of open-weight frontier models. The clearest way to see it is a table — every entry below is verified against the vendor’s model card or technical report:

Three adoption patterns stand out. First, expert granularity converged upward — 8 → 16 → 64 → 128 → 384 — exactly the direction DeepSeekMoE’s combinatorics argument predicted, with Kimi K2 adopting DeepSeek’s literal 384+1 shape for its trillion-parameter model. Second, the shared expert spread selectively. Meta adopted it for Llama 4 (one shared expert, top-1 routed sparsity); Kimi kept it; Qwen and GLM declined it, relying instead on renormalized gates and routed-scaling factors in the same design space. Third, load balancing quietly de-taxed. The 2024-era papers still leaned on auxiliary losses; by 2025’s big releases, bias-based or otherwise loss-minimal balancing is the norm, and deployment-side replication (the EPLB pattern) is standard practice — Kimi’s own deployment notes describe redundant experts on decode nodes to balance routing, precisely the EPLB trick.

What has not been copied, so far, is the part of the lineage that postdates V3: hash-routed early layers, the CED encoder–decoder split with per-phase activation budgets, and Engram-style conditional memory remain DeepSeek-only as of this writing. (OpenAI’s gpt-oss releases did adopt MXFP4-quantized MoE weights, converging with DeepSeek’s FP4-expert policy from the other direction — a notable meeting of the roads on precision.) The honest reading of Table 15.1 is that the industry absorbed DeepSeek’s 2024 ideas within about eighteen months, and is still digesting 2026’s.

16Field Guide: Serving the Beast

Sizing, serving, and offloading a DeepSeek-class MoE — then the glossary and the sources.

16.1 — Sizing: the experts are the model

Every practical decision about serving a DeepSeek model follows from one arithmetic fact established in Chapter 9: roughly 97% of the parameters live in the routed expert stack. Weight the memory budget accordingly (derived arithmetic, BF16 = 2 bytes/param, FP8 = 1, FP4 = 0.5):

16.2 — The three serving patterns

Pattern one: expert parallelism on a cluster. The production pattern for the big checkpoints — vLLM and SGLang both expose it (--enable-expert-parallel in vLLM, combined with tensor parallelism across enough GPUs to hold the fleet; the official DeepSeek recipes pages document per-model topologies). The dispatch/combine traffic between GPUs is the DeepEP pattern, and redundant-expert placement (EPLB-style) is the standard answer to hot experts at decode. For V4-class models the published recipes (vLLM’s and LMCache’s V4-Flash guide) also wire in the MTP draft head for speculative decoding and the FP8 KV format — treat the model’s own recipe page as the source of truth for flags, since they move release by release.

# V3-class, FP8, 8-way (H200/B200 class node):
vllm serve deepseek-ai/DeepSeek-V3 \
    --tensor-parallel-size 8 \
    --enable-expert-parallel

# add MTP speculative decoding where the recipe supports it:
#   --speculative-config '{"method": "deepseek_mtp", "num_speculative_tokens": 1}'
# (acceptance ~85-90% on V3-family drafts; expect ~1.8x TPS)

# SGLang equivalent:
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3 --tp 8

Pattern two: CPU-offloaded experts. The DeepSeekMoE structure makes MoE models unusually friendly to hybrid CPU+GPU serving, because 6–8 active experts per token means the hot working set is small relative to the fleet. llama.cpp’s override-tensor mechanism lets you park the entire expert stack in host RAM and keep attention/shared/dense weights on the GPU with one regex:

# offload every MoE expert tensor to CPU, keep the rest on GPU:
./llama-server -m DeepSeek-V3-Q4_K_M.gguf \
    -ngl 99 -ot ".ffn_.*_exps.=CPU"

# the active 8-of-256 experts per token stream from RAM each step;
# shared experts + attention stay hot in VRAM. Works for V3/V3.1-class
# quantized builds on a single 24 GB GPU + large system RAM.

Pattern three: KTransformers. The Tsinghua MADSys project formalizes pattern two into a full inference system for ultra-large MoE — DeepSeek V3/R1-class models on a single GPU with the expert fleet in CPU memory, with amplitudes of speedup that made it the standard desktop recipe for 671B-class checkpoints, and follow-on support for fine-tuning MoE models under memory constraints (its documentation claims 6–12× faster offload-tuning than ZeRO-Offload-style baselines). SGLang’s hybrid CPU/GPU MoE path covers the same idea at server scale. The common thread is architectural: fine-grained sparsity is what makes offloading work — a dense 671B has no small working set to exploit.

16.3 — Glossary

16.4 — Sources

One closing observation, offered as engineering rather than sentiment. The dense FFN of 2023 was a monument: every parameter sacred, every token taxed for all of them, capacity and compute the same number. DeepSeek’s three-year project was to take that monument apart — first into 64 experts, then 160, then 256, then 384, each smaller, each more specialized, the common knowledge quarantined, the load balanced first by gradients and then by arithmetic, the weights shrinking from sixteen bits to eight to four, until by September 2026 the “feed-forward layer” is a 552B-parameter library entered through hashes and gates, consulted at the cost of a 2023 toy model. The pattern is the same one the KV-cache history revealed, aimed at a different two-thirds of the network: find the multiplication nobody is attacking, and attack it for three consecutive years. The experts are still multiplying.