Google Corp IAP Protected (@google.com)
Google Cloud TPU & vLLM Architecture Analysis

Qwen 3.5 397B-A17B 推理张量流动图 Qwen 3.5 397B-A17B Inference Tensor Dataflow

展示 Qwen 3.5 在单层 Transformer Block 内的完整张量流动生命周期,包含 GQA (3:1)、QK-Norm (Qwen 原创特色)、★★★ KV Cache 显存物理驻留点 ★★★ 以及 MoE (共享专家 + 512 细粒度路由专家)。精确映射 Xprof Trace 中的 fusion.243 (QKV GEMM) 算子。
⚠ 部署形态:Prefill 节点 PCP=8 + EP=8(8 芯片),Decode 节点 TP=2 × DP=4 + EP=8(8 芯片),一对 PD 共 16 芯片。PCP / TP / EP 是同一组 8 芯片上的不同切分轴,不相乘。两阶段的张量形状与显存数字仍需分开读。
The full tensor lifecycle inside one Qwen 3.5 Transformer block: GQA (3:1), QK-Norm (a Qwen signature), the ★★★ physical KV-cache residency point ★★★, and MoE (shared expert + 512 fine-grained routed experts) — mapped onto the fusion.243 (QKV GEMM) operator from an Xprof trace.
⚠ Deployed topology: prefill node PCP=8 + EP=8 (8 chips), decode node TP=2 × DP=4 + EP=8 (8 chips), 16 chips per PD pair. PCP / TP / EP are different partitioning axes over the same 8 chips — they do not multiply. Shapes and memory figures still have to be read per-phase.

模型规格Model
Qwen3.5-397B-A17B
硬件平台Hardware
TPU v7 Ironwood
Prefill 节点 (8 芯片)Prefill node (8 chips)
PCP=8 + EP=8
Decode 节点 (8 芯片)Decode node (8 chips)
TP=2 × DP=4 + EP=8
MoE 拓扑 (两阶段一致)MoE topology (both phases)
EP=8 (64 experts/chip)
注意力架构 (混合)Attention (hybrid)
15 层 GQA + 45 层 GDN15 GQA + 45 GDN
Trace 核心算子Trace hot op
fusion.243 (50.00 µs)
New Benchmark GLM-5.2 on TPU v7x 端到端优化实测与 Xprof 微架构对账看板已上线 GLM-5.2 on TPU v7x E2E Optimization Benchmark & Xprof Analysis Board is Live (49K Prefix + 1024 并发 · 19.99 ms ➔ 4.71 ms · 4.25x 加速 · 含 Google 内网原生 Xprof Session) (49K Prefix + 1024 Concurrency · 19.99 ms ➔ 4.71 ms · 4.25x Speedup · Native Corp Xprof Sessions)
🚀 打开 Xprof 对账看板Open Xprof Board →
🎬 128K Token 生命周期动态流向仿真(端到端实测 4.62 s · 显微级放慢观察) 128K token lifecycle simulator (measured end-to-end 4.62 s, replayed in slow motion)
TPU v7 动态时序引擎就绪 TPU v7 timing engine ready
实测基准: 4.62 秒(128K prefill · 8 芯片切片) Measured baseline: 4.62 s (128K prefill on an 8-chip slice)
⚠ 标定基线:8 芯片 prefill 节点 · 60 层混合架构 ⚠ Calibration baseline: 8-chip prefill node, 60-layer hybrid build
时间轴回放的是 bodaborg-tpu7x-nap 上 8 芯片 prefill 节点(PCP=8 + EP=8)· 60 层混合架构(45 GDN + 15 GQA) 那次压测,端到端 4.62 s。 这就是本文档的主口径 —— 联合优化文档记录的上线部署形态即为「1 Prefill node (8 chips, PCP8+EP8) → 1 Decode node (8 chips),一对共 16 芯片」, PCP / TP / EP 是同一组 8 芯片上的不同切分轴,不相乘,不存在 64 芯片实例。 4.62 s 的对照组是同样 8 芯片上的 DP8(128K 单请求 TTFT 32 s),因此 6.9× 是「同硬件下 DP → PCP 的收益」,不是扩容到更多芯片的超线性加速。 This timeline replays one stress run on bodaborg-tpu7x-nap: an 8-chip prefill node (PCP=8 + EP=8) running a 60-layer hybrid build (45 GDN + 15 GQA), 4.62 s end to end. This is the document's main configuration — the joint-optimization record describes the deployed form as "1 prefill node (8 chips, PCP8+EP8) → 1 decode node (8 chips), 16 chips per pair". PCP / TP / EP are different partitioning axes over the same 8 chips and do not multiply; there is no 64-chip instance. The 4.62 s baseline is DP8 on the very same 8 chips (128K single-request TTFT 32 s), so the 6.9× is "DP → PCP on identical hardware", not superlinear scaling to more chips.
时间线缩放:Timeline scale:
物理时间: Physical time: 0.000 s / 4.620 s
回放耗时: Playback elapsed: 0.0 s / 46.2 s
当前阶段: Current stage:
STAGE 1
① 128K 批次切分① 128K chunking
0 ~ 0.40 s
STAGE 2
② GDN 线性状态② GDN linear state
0.40 ~ 1.25 s
STAGE 3
③ PCP8 环形注意力③ PCP8 ring attention
1.25 ~ 3.10 s
STAGE 4
④ MoE 动态路由④ MoE routing
3.10 ~ 4.10 s
STAGE 5
⑤ MTP 投机推测⑤ MTP speculation
4.10 ~ 4.48 s
STAGE 6
⑥ DCN 跨切片直传⑥ DCN handoff
4.48 ~ 4.62 s
🖥️ 活跃执行硬件核🖥️ Active compute unit
⚡ 片间通信 / 路由引擎⚡ Interconnect / routing
📊 张量维度与内存流向📊 Tensor shape & memory path
🚀 当前阶段特化优化🚀 Optimization in play
📌 Qwen 3.5 单层推理数据流动图 (Mermaid · Prefill 视角,8 路头切分 + 8 路序列切分(形状取自 trace)) Single-layer dataflow (Mermaid · prefill view: 8-way head split + 8-way sequence split, shapes taken from the trace)
⚠ 本图是 Prefill 视角,Decode (TP=2 × DP=4) 有五处不同;切分轴的口径存疑说明见「Xprof 算子深度映射」页 ⚠ Prefill view; decode (TP=2 × DP=4) differs in five places. See the Xprof deep-dive tab for the open question about which axes these shapes reflect

① 每卡 Q 头 12 → 48,KV 头 4 → 16(TP=2);
② fusion.243 的输出列 2560 → 10240,o_proj 为 6144 → 4096;
③ Attn 后的 All-Reduce 域从 8 卡缩为 2 卡;4 个 DP 副本各处理不同请求,互不通信;
④ PCP 消失:decode 时 T=1,序列维无可切分,因此没有 ring/all-gather,每卡持有完整序列的 KV;
⑤ 注意力是 DP=4 而 MoE 是 EP=8,进出 MoE 需要一次 DP↔EP 的 All-to-All 重排。
⑥ PD 交接与物理实例边界:单物理机(4 芯片 / 8 Core)受 758 GiB HBM 限制,只能独立承担 1 个完整的 Prefill 节点或 1 个 Decode 节点。一套端到端 PD 分离集群最小物理开销至少为 2 台物理机(8 芯片)。KV 要从 prefill 8 核的布局重排为 decode 8 核上的 (16 头, 全序列) —— 跨节点搬运且需重分布,是 PD 分离的主要固有开销(#554 / #582)。本图未画。
① Q heads per chip 12 → 48, KV heads 4 → 16 (TP=2);
② fusion.243 output columns 2560 → 10240, o_proj becomes 6144 → 4096;
③ the post-attention All-Reduce domain shrinks from 8 chips to 2; the 4 DP replicas serve different requests and never talk;
④ PCP disappears: at decode T=1, so there is nothing to split along the sequence — no ring/all-gather, and each chip holds the full-length KV;
⑤ attention is DP=4 while MoE is EP=8, so entering and leaving the MoE needs a DP↔EP All-to-All reshuffle.
⑥ PD Handoff & Physical Host Boundary: A single physical host (4 chips / 8 cores, 758 GiB HBM) can only act as an isolated worker for either 1 Prefill node OR 1 Decode node; a full end-to-end PD cluster requires at least 2 hosts (8 chips). KV is reshaped from the prefill layout on 8 cores into (16 heads, full S) on the 8 decode cores — a cross-node move plus a redistribution, and the dominant intrinsic cost of PD disaggregation (#554 / #582). Not drawn here.

归一化 Norm (Pre-Norm / QK-Norm)Normalization (Pre-Norm / QK-Norm)
密集矩阵乘 GEMM / MXU 算子Dense GEMM / MXU op
★★★ KV Cache 显存物理驻留点 ★★★★★★ Physical KV cache residency ★★★
MoE 混合专家模块 (Shared + Routed)MoE block (shared + routed)
跨卡通信 (All-Reduce / All-to-All / Ring)Cross-chip collective (All-Reduce / All-to-All / Ring)
flowchart TD In["输入残差流: [B, T, 4096]"] In --> Attn_Norm["① Pre-Attn RMSNorm (attn_norm)"] In ----> Res_Add1{"⊕ 残差相加 1"} Attn_Norm --> QKV_GEMM["★ fusion.243 (QKV 融合投影)
[T/8=4096, 4096] × [4096, 2560]
trace 为 TP=8 变体列切 · M 为本地 token (50.00 µs)"] QKV_GEMM --> QKV_Tensor["[B, T/8, 2560] (单卡分片)"] QKV_Tensor -->|"slice.7534 (切前 1536 列)"| Q_Tensor["Q: [B, T/8, 12, 128]"] QKV_Tensor -->|"切后 1024 列 (前 512)"| K_Tensor["K: [B, T/8, 4, 128]"] QKV_Tensor -->|"切后 1024 列 (后 512)"| V_Tensor["V: [B, T/8, 4, 128]"] Q_Tensor --> Q_Norm["② Q-RMSNorm (Qwen 特色)"] K_Tensor --> K_Norm["③ K-RMSNorm (Qwen 特色)"] Q_Norm --> Q_RoPE["Q-RoPE 旋转位置编码"] K_Norm --> K_RoPE["K-RoPE 旋转位置编码"] K_RoPE --> KV_Cache["★★★ KV Cache 物理驻留点 ★★★
Prefill: 单卡 4 头 × 全序列, 等体积于 32 头 × S/8
Decode: [B, S, 16, 128] × 2
(单卡 0.47 GiB @32K fp8 · 15 层 FA)"] V_Tensor --> KV_Cache KV_Cache --> CP_Ring["PCP=8 Ring / All-Gather KV
(仅 Prefill: 每 rank 只有 S/8 个 token)"] Q_RoPE --> GQA_Op{"GQA 注意力计算 (Flash/Splash)
12 个 Q 头 共享 4 组 KV (3:1)
Softmax(Q·K^T / √128) · V"} CP_Ring -->|"补齐全序列 Key/Value"| GQA_Op GQA_Op --> Attn_Out["注意力输出: [B, T/8, 12, 128]"] Attn_Out --> O_Proj["Out Proj 线性层: 1536 → 4096"] O_Proj --> Attn_AllReduce["跨 8 卡 All-Reduce 通信求和
(Decode 时缩为 TP=2, 2 卡)"] Attn_AllReduce --> Res_Add1 Res_Add1 --> Res_Tensor1["Attn 输出残差流: [B, T, 4096]"] Res_Tensor1 --> FFN_Norm["④ Post-Attn RMSNorm (ffn_norm)"] Res_Tensor1 ----> Res_Add2{"⊕ 残差相加 2"} FFN_Norm --> Shared_Exp["Qwen 共享专家 (Shared Expert)
4096 → 5632 → 4096 (SwiGLU)"] FFN_Norm --> Router["Router 门控网络
GEMM 4096 → 512 (选 Top-8)"] Router --> MoE_Dispatch["EP=8 Token 分发 (All-to-All)
每卡常驻 64 个专家"] MoE_Dispatch --> Routed_Exps["Top-8 细粒度路由专家
4096 → 1408 → 4096 (SwiGLU)"] Routed_Exps --> MoE_Combine["加权求和 & All-to-All 回传"] Shared_Exp --> MoE_Sum{"专家输出合并"} MoE_Combine --> MoE_Sum MoE_Sum --> Res_Add2 Res_Add2 --> Final_Out["层输出残差流: [B, T, 4096]"] style Attn_Norm fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#92400e style Q_Norm fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#92400e style K_Norm fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#92400e style FFN_Norm fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#92400e style QKV_GEMM fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style O_Proj fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style KV_Cache fill:#dcfce7,stroke:#16a34a,stroke-width:3px,color:#166534 style GQA_Op fill:#e0e7ff,stroke:#4338ca,stroke-width:2px,color:#312e81 style Shared_Exp fill:#f3e8ff,stroke:#9333ea,stroke-width:2px,color:#6b21a8 style Router fill:#f3e8ff,stroke:#9333ea,stroke-width:2px,color:#6b21a8 style Routed_Exps fill:#f3e8ff,stroke:#9333ea,stroke-width:2px,color:#6b21a8 style CP_Ring fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style Attn_AllReduce fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style MoE_Dispatch fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style MoE_Combine fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b
flowchart TD In["Input residual stream: [B, T, 4096]"] In --> Attn_Norm["① Pre-Attn RMSNorm (attn_norm)"] In ----> Res_Add1{"⊕ Residual Add 1"} Attn_Norm --> QKV_GEMM["★ fusion.243 (fused QKV projection)
[T/8=4096, 4096] × [4096, 2560]
TP=8-variant column split · M is chip-local (50.00 µs)"] QKV_GEMM --> QKV_Tensor["[B, T/8, 2560] (per-chip shard)"] QKV_Tensor -->|"slice.7534 (first 1536 cols)"| Q_Tensor["Q: [B, T/8, 12, 128]"] QKV_Tensor -->|"next 1024 cols (first 512)"| K_Tensor["K: [B, T/8, 4, 128]"] QKV_Tensor -->|"next 1024 cols (last 512)"| V_Tensor["V: [B, T/8, 4, 128]"] Q_Tensor --> Q_Norm["② Q-RMSNorm (Qwen signature)"] K_Tensor --> K_Norm["③ K-RMSNorm (Qwen signature)"] Q_Norm --> Q_RoPE["Q-RoPE rotary embedding"] K_Norm --> K_RoPE["K-RoPE rotary embedding"] K_RoPE --> KV_Cache["★★★ Physical KV cache residency ★★★
Prefill: 4 heads × full S per chip, same volume as 32 heads × S/8
Decode: [B, S, 16, 128] × 2
(0.47 GiB/chip @32K fp8, 15 FA layers)"] V_Tensor --> KV_Cache KV_Cache --> CP_Ring["PCP=8 ring / all-gather of KV
(prefill only: each rank owns S/8 tokens)"] Q_RoPE --> GQA_Op{"GQA attention (Flash/Splash)
12 Q heads share 4 KV groups (3:1)
Softmax(Q·K^T / √128) · V"} CP_Ring -->|"complete the full-length Key/Value"| GQA_Op GQA_Op --> Attn_Out["Attention output: [B, T/8, 12, 128]"] Attn_Out --> O_Proj["Out projection: 1536 → 4096"] O_Proj --> Attn_AllReduce["All-Reduce sum across 8 chips
(shrinks to TP=2, 2 chips, at decode)"] Attn_AllReduce --> Res_Add1 Res_Add1 --> Res_Tensor1["Post-attention residual: [B, T, 4096]"] Res_Tensor1 --> FFN_Norm["④ Post-Attn RMSNorm (ffn_norm)"] Res_Tensor1 ----> Res_Add2{"⊕ Residual Add 2"} FFN_Norm --> Shared_Exp["Qwen shared expert
4096 → 5632 → 4096 (SwiGLU)"] FFN_Norm --> Router["Router gate network
GEMM 4096 → 512 (pick Top-8)"] Router --> MoE_Dispatch["EP=8 token dispatch (All-to-All)
64 experts resident per chip"] MoE_Dispatch --> Routed_Exps["Top-8 fine-grained routed experts
4096 → 1408 → 4096 (SwiGLU)"] Routed_Exps --> MoE_Combine["Weighted sum & All-to-All combine"] Shared_Exp --> MoE_Sum{"Merge expert outputs"} MoE_Combine --> MoE_Sum MoE_Sum --> Res_Add2 Res_Add2 --> Final_Out["Layer output residual: [B, T, 4096]"] style Attn_Norm fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#92400e style Q_Norm fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#92400e style K_Norm fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#92400e style FFN_Norm fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#92400e style QKV_GEMM fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style O_Proj fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style KV_Cache fill:#dcfce7,stroke:#16a34a,stroke-width:3px,color:#166534 style GQA_Op fill:#e0e7ff,stroke:#4338ca,stroke-width:2px,color:#312e81 style Shared_Exp fill:#f3e8ff,stroke:#9333ea,stroke-width:2px,color:#6b21a8 style Router fill:#f3e8ff,stroke:#9333ea,stroke-width:2px,color:#6b21a8 style Routed_Exps fill:#f3e8ff,stroke:#9333ea,stroke-width:2px,color:#6b21a8 style CP_Ring fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style Attn_AllReduce fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style MoE_Dispatch fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style MoE_Combine fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b
📌 Qwen 3.5 PD 分离推理架构全景张量流向图(Prefill ➔ DCN 直传 ➔ Decode 自回归) PD-disaggregated panorama (prefill ➔ DCN handoff ➔ autoregressive decode)
⚠ 本图的 µs / ms 标注来自 8 芯片 · 60 层混合架构那次压测 ⚠ The µs / ms annotations come from the 8-chip, 60-layer hybrid run
图中绝对耗时均取自 bodaborg-tpu7x-nap · 2×4 上 8 芯片 prefill 节点 · 45 GDN + 15 GQA 的压测。 引用前请先看数字溯源: ✓ 有联合优化文档背书 —— 128K 单请求 TTFT DP8 32 s → PCP8 4.62 s、decode 5,650 tok/s(TP2DP4 + owner_output 分支)、单网卡 P→D 峰值 > 12.5 GB/s(线速 25 GB/s,达标线为其 50%)、prefill 分层耗时 full 492 µs × 15 层 / linear 774 µs × 45 层 / MoE 1.41 ms × 60 层。 △ 自洽但属推算 —— 1.92 GB 可由 [131072, 15, 4, 128] FP8 × 2(K,V) 反算得到,是单卡份额(精确值 1.875 GiB;8 卡合计 15 GiB = 整请求 15 层 FA 的 KV),账对得上;但 450 ms → 138 ms(3.2×)是 1.92 GB 分别除以 4.2 / 13.6 GB/s 的算术结果,而文档只背书「峰值超过 12.5 GB/s」。按 12.5 GB/s 应为 ~154 ms。 ✗ 与实测冲突 —— 22.45 → 11.20 ms/tok(2.0×):实测 TPOT 为 25.4 ~ 28.4 ms,MTP 在 coding 场景的典型加速比是 1.34× 而非 2.0×。 Absolute latencies come from a stress run on bodaborg-tpu7x-nap · 2×4: an 8-chip prefill node running 45 GDN + 15 GQA layers. Check provenance before quoting: ✓ Corroborated by the joint-optimization record — 128K single-request TTFT 32 s under DP8 → 4.62 s under PCP8, decode 5,650 tok/s (TP2DP4 on the owner_output branch), single-NIC P→D peak > 12.5 GB/s (line rate 25 GB/s; the target was 50% of it), per-layer prefill cost full 492 µs × 15 / linear 774 µs × 45 / MoE 1.41 ms × 60. △ Self-consistent but derived — 1.92 GB falls out of [131072, 15, 4, 128] FP8 × 2 (K,V) and is the per-chip share (exactly 1.875 GiB; 8 chips = 15 GiB, i.e. the whole request's KV over 15 FA layers), so the arithmetic checks out. But 450 ms → 138 ms (3.2×) is just 1.92 GB divided by 4.2 and 13.6 GB/s, whereas the record only backs "peak above 12.5 GB/s" — which would give ~154 ms. ✗ Contradicted by measurement — 22.45 → 11.20 ms/tok (2.0×): measured TPOT is 25.4–28.4 ms, and MTP's typical coding-workload speedup is 1.34×, not 2.0×.
密集计算算子 ⏱️ (Compute Bound)Dense compute op ⏱️ (compute bound)
跨卡通讯 ⚡ (ICI / DCN)Cross-chip collective ⚡ (ICI / DCN)
★ 专项工程优化点★ Targeted engineering optimization
★★★ KV Cache 物理驻留点 ★★★★★★ Physical KV residency ★★★
🌐 1P1D DCN 跨切片传输🌐 1P1D DCN inter-slice transfer
flowchart TD subgraph Prefill_Slice ["🟦 【Prefill 切片】1P 预填充节点 (PCP8 环形并行 · 128K 超长 Prompt)"] direction TB In["输入残差流: [B, 128K, 4096]"] In --> Attn_Norm["① Pre-Attn RMSNorm
⏱️ 312 µs"] In ----> Res_Add1{"⊕ 残差相加 1"} Attn_Norm --> GDN_Op["★【优化 1】GDN 线性注意力 (45 层)
🔗 vllm-torchtpu#766 / #1082 (QK 复制) / #873
📄 实测分层: linear 774 µs × 45 层
🔴 优化前 4.12 ms (反复写回 HBM)
🟢 优化后 0.84 ms (FP8 融合状态, 零写回)
🚀 4.9× · 维持 O(1) 状态"] Attn_Norm --> QKV_GEMM["fusion.243 (QKV 融合投影)
[16384, 4096] × [4096, 2560]
⏱️ 50.00 µs (MXU)"] QKV_GEMM --> QKV_Tensor["[B, T/8, 2560] (单卡分片)"] QKV_Tensor -->|"slice.7534 (前 1536 列)
⏱️ 3.25 µs"| Q_Tensor["Q: [B, T/8, 12, 128]"] QKV_Tensor -->|"后 1024 列 (前 512)"| K_Tensor["K: [B, T/8, 4, 128]"] QKV_Tensor -->|"后 1024 列 (后 512)"| V_Tensor["V: [B, T/8, 4, 128]"] Q_Tensor --> Q_Norm["② Q-RMSNorm & RoPE
⏱️ 85.00 µs"] K_Tensor --> K_Norm["③ K-RMSNorm & RoPE
⏱️ 85.00 µs"] K_Norm --> KV_Cache_Local["★★★【优化 2】SEQ_ON_LANE 物理驻留 ★★★
🔗 vllm-torchtpu#608 / #606 / #750 (HND)
KV Heads 维前置, 消除跨切片 Reshard 拷贝
📈 decode 吞吐 700 → 5,790 tok/s
📦 单卡载荷 1.92 GB = [131072, 15, 4, 128] FP8 ×2"] V_Tensor --> KV_Cache_Local subgraph Overlap_Box ["★【优化 3】PCP8 Ring Attention 双缓冲异步重叠 (PR #366 / #414 / #863)"] direction TB Q_Norm --> Ring_Compute["【计算流 · MXU】
FlashAttention 矩阵乘
⏱️ 实测 492 µs / 层 · 15 层合计 ≈ 7.4 ms
📄 实测分层: full 492 µs × 15 层
🔴 DP8 TTFT ~32.0 s → 🟢 PCP8 4.62 s (同 8 芯片)"] KV_Cache_Local -->|"双缓冲拉取"| Ring_Comm["【通信流 · ICI DMA】
环形传递下一个 16K KV Chunk
⚡ 950 µs
🟢 100% 藏在 4,120 µs 计算阴影下 (Bubble = 0)"] Ring_Compute -.->|"异步重叠遮蔽"| Ring_Comm end Ring_Compute --> Attn_Out["注意力输出: [B, T/8, 12, 128]"] Attn_Out --> O_Proj["Out Proj: 1536 → 4096"] O_Proj --> Attn_AllReduce["片内 All-Reduce (all-reduce.45)
⚡ Prefill 域 8 卡
(Decode 侧缩为 TP=2, ~85 µs)"] Attn_AllReduce --> Res_Add1 Res_Add1 --> Res_Tensor1["Attn 残差流: [B, 128K, 4096]"] Res_Tensor1 --> FFN_Norm["④ Post-Attn RMSNorm
⏱️ 312 µs"] Res_Tensor1 ----> Res_Add2{"⊕ 残差相加 2"} FFN_Norm --> Shared_Exp["共享专家 (Shared)
4096 → 5632 → 4096 (SwiGLU)"] FFN_Norm --> Router["Router 门控 (fusion.815)
4096 → 512, 选 Top-8
⏱️ 145 µs"] Router --> MoE_Dispatch["EP8 Token 分发 (all-to-all.18)
⚡ 954 µs"] MoE_Dispatch --> Routed_Exps["★【优化 4】owner_output 专家直聚
🔗 vllm-torchtpu#628
fusion.246 / 816 (Owner Core 原地聚合)
📈 decode 5,500 → 5,650 tok/s
🔴 3.10 ms → 🟢 2.14 ms (+31%)"] Routed_Exps --> MoE_Combine["EP8 专家汇聚 (all-to-all.19)
⚡ 961 µs"] MoE_Combine --> TP_MoE_Sync["TP MoE 输出规约 (all-reduce.46)
⚡ 88.0 µs"] Shared_Exp --> MoE_Sum{"专家输出合并"} TP_MoE_Sync --> MoE_Sum MoE_Sum --> Res_Add2 Res_Add2 --> Prefill_Out["Prefill 尾层隐状态
[B, 1, 4096] (首字 Logits 输入)"] end subgraph DCN_Transfer_Highway ["🌐 【PD 分离高速公路】1P ➔ 1D DCN 跨切片直传"] direction TB DCN_Op["★★ DCN KV Cache 网络直传 ★★
🔗 vllm-torchtpu#554 (PCP8>DP8 重排修复) & #582 (控制器)
• 介质: 单张 200 Gbps DCN 网卡 (线速 25 GB/s)
🟢 实测峰值 > 12.5 GB/s (达标线 = 线速 50%)
📦 单卡载荷 1.875 GiB (8 卡合计 15 GiB / 请求)
🔴 未重排 4.2 GB/s · ~450 ms → 🟢 ~138 ms (推算)
⚠ 13.6 GB/s 为推算值; 文档背书 >12.5 GB/s → ~154 ms
🚧 双网卡 (400 Gbps) 仍是阻塞项
b/558565285 · Multi-NIC C=1 问题排查中"] end KV_Cache_Local ==>|"① Prefill 完结: 导出完整 KV"| DCN_Op Prefill_Out ==>|"② 传递首字隐状态与元数据"| Decode_Input subgraph Decode_Slice ["🟩 【Decode 切片】1D 解码节点 (TP2 × DP4 × EP8 · 高并发流式吐字)"] direction TB Decode_Input["Decode 接收端: 首字输入 [B, 1, 4096]"] Decode_KV_Resident["★★★ Decode 端 KV Cache 就位 ★★★
全量 128K 历史注入 Decode HBM
后续每步仅追加 1 Token (O(1) 增量)"] Decode_Input --> Dec_Attn_Norm["Decode RMSNorm & QKV GEMM
单步 ~1.20 ms"] Decode_KV_Resident -.->|"自回归加载历史 KV"| Dec_Step_Attn Dec_Attn_Norm --> Dec_Step_Attn["Decode 单步注意力 (FlashDecode)
⏱️ 1,850 µs"] Dec_Step_Attn --> Dec_MoE["Decode EP8 MoE 门控与专家计算
⚡ 分发 620 µs + 汇聚 615 µs"] Dec_MoE --> LM_Head["LM Head 词表投影 (fusion.988)
vocab 248,320 · [4096, 248320]
⏱️ 1,820 µs + ⚡ all-reduce.88: 112 µs"] LM_Head --> MTP_Verify["★【优化 5】MTP 多 Token 投机验证
🔗 vllm-torchtpu#1008 (RPAd 路由, 已合入) & #559
fusion.990 (单步命中 2 Token)
🟢 实测 coding 场景加速比 1.34×
🚀 吞吐 5,650 tok/s (TP2DP4 + owner_output)
⚠ 22.45 → 11.20 ms/tok (2.0×) 与实测 TPOT 25~28 ms 冲突"] MTP_Verify --> Token_Emit["🎉 流式吐出 Token: [T+1, T+2]"] Token_Emit -.->|"自回归循环"| Decode_Input end DCN_Op ==>|"③ DCN 直传注入 Decode 显存 (> 12.5 GB/s)"| Decode_KV_Resident style Prefill_Slice fill:#0b132b,stroke:#3b82f6,stroke-width:2px,color:#93c5fd style Decode_Slice fill:#06231a,stroke:#10b981,stroke-width:2px,color:#6ee7b7 style DCN_Transfer_Highway fill:#231a06,stroke:#facc15,stroke-width:3px,color:#fef08a style DCN_Op fill:#422006,stroke:#eab308,stroke-width:3px,color:#fef08a style GDN_Op fill:#fff7ed,stroke:#ea580c,stroke-width:2px,color:#9a3412 style Ring_Compute fill:#fff7ed,stroke:#ea580c,stroke-width:3px,color:#9a3412 style Ring_Comm fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style Overlap_Box fill:#0f172a,stroke:#38bdf8,stroke-width:2px,color:#93c5fd style KV_Cache_Local fill:#dcfce7,stroke:#16a34a,stroke-width:3px,color:#166534 style Decode_KV_Resident fill:#dcfce7,stroke:#16a34a,stroke-width:3px,color:#166534 style Routed_Exps fill:#fff7ed,stroke:#ea580c,stroke-width:3px,color:#9a3412 style Attn_AllReduce fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style TP_MoE_Sync fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style MoE_Dispatch fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style MoE_Combine fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style MTP_Verify fill:#fff7ed,stroke:#ea580c,stroke-width:3px,color:#9a3412 style Attn_Norm fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style Q_Norm fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style K_Norm fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style FFN_Norm fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style QKV_GEMM fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style O_Proj fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style Shared_Exp fill:#f3e8ff,stroke:#9333ea,stroke-width:2px,color:#6b21a8 style Router fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style LM_Head fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style Prefill_Out fill:#f1f5f9,stroke:#64748b,stroke-width:2px,color:#334155 style Decode_Input fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style Dec_Attn_Norm fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style Dec_Step_Attn fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style Dec_MoE fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style Token_Emit fill:#f0fdf4,stroke:#16a34a,stroke-width:3px,color:#15803d
flowchart TD subgraph Prefill_Slice ["🟦 Prefill slice (1P) — PCP8 ring parallel, 128K prompt"] direction TB In["Input residual: [B, 128K, 4096]"] In --> Attn_Norm["① Pre-Attn RMSNorm
⏱️ 312 µs"] In ----> Res_Add1{"⊕ Residual add 1"} Attn_Norm --> GDN_Op["★ Opt 1: GDN linear attention (45 layers)
🔗 vllm-torchtpu#766 / #1082 (QK replication) / #873
📄 Measured per-layer: linear 774 µs × 45
🔴 before 4.12 ms (repeated HBM write-back)
🟢 after 0.84 ms (FP8 fused state, zero write-back)
🚀 4.9× · O(1) state retained"] Attn_Norm --> QKV_GEMM["fusion.243 (fused QKV projection)
[16384, 4096] × [4096, 2560]
⏱️ 50.00 µs (MXU)"] QKV_GEMM --> QKV_Tensor["[B, T/8, 2560] (per-chip shard)"] QKV_Tensor -->|"slice.7534 (first 1536 cols)
⏱️ 3.25 µs"| Q_Tensor["Q: [B, T/8, 12, 128]"] QKV_Tensor -->|"next 1024 cols (first 512)"| K_Tensor["K: [B, T/8, 4, 128]"] QKV_Tensor -->|"next 1024 cols (last 512)"| V_Tensor["V: [B, T/8, 4, 128]"] Q_Tensor --> Q_Norm["② Q-RMSNorm & RoPE
⏱️ 85.00 µs"] K_Tensor --> K_Norm["③ K-RMSNorm & RoPE
⏱️ 85.00 µs"] K_Norm --> KV_Cache_Local["★★★ Opt 2: SEQ_ON_LANE residency ★★★
🔗 vllm-torchtpu#608 / #606 / #750 (HND)
KV-head axis hoisted, kills cross-slice reshard copies
📈 decode throughput 700 → 5,790 tok/s
📦 Per-chip payload 1.92 GB = [131072, 15, 4, 128] FP8 ×2"] V_Tensor --> KV_Cache_Local subgraph Overlap_Box ["★ Opt 3: PCP8 ring attention, double-buffered async overlap (PR #366 / #414 / #863)"] direction TB Q_Norm --> Ring_Compute["Compute stream · MXU
FlashAttention matmul
⏱️ Measured 492 µs / layer · ≈7.4 ms for 15
📄 Measured per-layer: full 492 µs × 15
🔴 DP8 TTFT ~32.0 s → 🟢 PCP8 4.62 s (same 8 chips)"] KV_Cache_Local -->|"double-buffered prefetch"| Ring_Comm["Comm stream · ICI DMA
Ring-forward the next 16K KV chunk
⚡ 950 µs
🟢 100% hidden under 4,120 µs of compute (bubble = 0)"] Ring_Compute -.->|"async overlap"| Ring_Comm end Ring_Compute --> Attn_Out["Attention output: [B, T/8, 12, 128]"] Attn_Out --> O_Proj["Out projection: 1536 → 4096"] O_Proj --> Attn_AllReduce["Intra-slice All-Reduce (all-reduce.45)
⚡ 8-chip domain at prefill
(shrinks to TP=2, ~85 µs, at decode)"] Attn_AllReduce --> Res_Add1 Res_Add1 --> Res_Tensor1["Post-attention residual: [B, 128K, 4096]"] Res_Tensor1 --> FFN_Norm["④ Post-Attn RMSNorm
⏱️ 312 µs"] Res_Tensor1 ----> Res_Add2{"⊕ Residual add 2"} FFN_Norm --> Shared_Exp["Shared expert
4096 → 5632 → 4096 (SwiGLU)"] FFN_Norm --> Router["Router gate (fusion.815)
4096 → 512, pick Top-8
⏱️ 145 µs"] Router --> MoE_Dispatch["EP8 token dispatch (all-to-all.18)
⚡ 954 µs"] MoE_Dispatch --> Routed_Exps["★ Opt 4: owner_output direct gather
🔗 vllm-torchtpu#628
fusion.246 / 816 (in-place on the owner core)
📈 decode 5,500 → 5,650 tok/s
🔴 3.10 ms → 🟢 2.14 ms (+31%)"] Routed_Exps --> MoE_Combine["EP8 expert combine (all-to-all.19)
⚡ 961 µs"] MoE_Combine --> TP_MoE_Sync["TP MoE output reduce (all-reduce.46)
⚡ 88.0 µs"] Shared_Exp --> MoE_Sum{"Merge expert outputs"} TP_MoE_Sync --> MoE_Sum MoE_Sum --> Res_Add2 Res_Add2 --> Prefill_Out["Final-layer hidden state
[B, 1, 4096] (feeds first-token logits)"] end subgraph DCN_Transfer_Highway ["🌐 PD handoff highway — 1P ➔ 1D DCN direct transfer"] direction TB DCN_Op["★★ DCN KV cache direct transfer ★★
🔗 vllm-torchtpu#554 (PCP8>DP8 reshard fix) & #582 (controller)
• Medium: one 200 Gbps DCN NIC (25 GB/s line rate)
🟢 Measured peak > 12.5 GB/s (target = 50% of line rate)
📦 Per-chip payload 1.875 GiB (8 chips = 15 GiB/request)
🔴 Unlaid-out 4.2 GB/s · ~450 ms → 🟢 ~138 ms (derived)
⚠ 13.6 GB/s is derived; the record backs >12.5 GB/s → ~154 ms
🚧 Dual-NIC (400 Gbps) is still a blocker
b/558565285 · Multi-NIC C=1 issue under investigation"] end KV_Cache_Local ==>|"① prefill done: export the full KV"| DCN_Op Prefill_Out ==>|"② pass first-token hidden state + metadata"| Decode_Input subgraph Decode_Slice ["🟩 Decode slice (1D) — TP2 × DP4 × EP8, high-concurrency streaming"] direction TB Decode_Input["Decode ingress: first token [B, 1, 4096]"] Decode_KV_Resident["★★★ KV cache resident on decode side ★★★
Full 128K history injected into decode HBM
each later step appends just 1 token (O(1))"] Decode_Input --> Dec_Attn_Norm["Decode RMSNorm & QKV GEMM
~1.20 ms per step"] Decode_KV_Resident -.->|"autoregressively read history KV"| Dec_Step_Attn Dec_Attn_Norm --> Dec_Step_Attn["Single-step attention (FlashDecode)
⏱️ 1,850 µs"] Dec_Step_Attn --> Dec_MoE["Decode EP8 MoE gate + experts
⚡ dispatch 620 µs + combine 615 µs"] Dec_MoE --> LM_Head["LM head vocab projection (fusion.988)
vocab 248,320 · [4096, 248320]
⏱️ 1,820 µs + ⚡ all-reduce.88: 112 µs"] LM_Head --> MTP_Verify["★ Opt 5: MTP multi-token speculative verify
🔗 vllm-torchtpu#1008 (route via RPAd, merged) & #559
fusion.990 (2 tokens accepted per step)
🟢 Measured coding-workload speedup 1.34×
🚀 5,650 tok/s (TP2DP4 + owner_output)
⚠ 22.45 → 11.20 ms/tok (2.0×) conflicts with measured TPOT 25–28 ms"] MTP_Verify --> Token_Emit["🎉 Stream out tokens: [T+1, T+2]"] Token_Emit -.->|"autoregressive loop"| Decode_Input end DCN_Op ==>|"③ DCN direct-inject into decode HBM (> 12.5 GB/s)"| Decode_KV_Resident style Prefill_Slice fill:#0b132b,stroke:#3b82f6,stroke-width:2px,color:#93c5fd style Decode_Slice fill:#06231a,stroke:#10b981,stroke-width:2px,color:#6ee7b7 style DCN_Transfer_Highway fill:#231a06,stroke:#facc15,stroke-width:3px,color:#fef08a style DCN_Op fill:#422006,stroke:#eab308,stroke-width:3px,color:#fef08a style GDN_Op fill:#fff7ed,stroke:#ea580c,stroke-width:2px,color:#9a3412 style Ring_Compute fill:#fff7ed,stroke:#ea580c,stroke-width:3px,color:#9a3412 style Ring_Comm fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style Overlap_Box fill:#0f172a,stroke:#38bdf8,stroke-width:2px,color:#93c5fd style KV_Cache_Local fill:#dcfce7,stroke:#16a34a,stroke-width:3px,color:#166534 style Decode_KV_Resident fill:#dcfce7,stroke:#16a34a,stroke-width:3px,color:#166534 style Routed_Exps fill:#fff7ed,stroke:#ea580c,stroke-width:3px,color:#9a3412 style Attn_AllReduce fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style TP_MoE_Sync fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style MoE_Dispatch fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style MoE_Combine fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style MTP_Verify fill:#fff7ed,stroke:#ea580c,stroke-width:3px,color:#9a3412 style Attn_Norm fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style Q_Norm fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style K_Norm fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style FFN_Norm fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style QKV_GEMM fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style O_Proj fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style Shared_Exp fill:#f3e8ff,stroke:#9333ea,stroke-width:2px,color:#6b21a8 style Router fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style LM_Head fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style Prefill_Out fill:#f1f5f9,stroke:#64748b,stroke-width:2px,color:#334155 style Decode_Input fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style Dec_Attn_Norm fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style Dec_Step_Attn fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e40af style Dec_MoE fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b style Token_Emit fill:#f0fdf4,stroke:#16a34a,stroke-width:3px,color:#15803d
⚡ 核心机制:Ring Attention 的跨卡通讯如何 100% 藏进计算? Core mechanism: how does ring attention hide 100% of its cross-chip traffic? Bubble = 0 µs

传统模式是「先算完再通信」或「先等数据到齐再开算」,计算单元会大量闲置等待,形成通信气泡(bubble)。 在 PCP8 Ring Attention 里,TPU 的 MXU 计算引擎与 ICI 片间互联 DMA 引擎在硬件层面完全解耦,配合双缓冲,计算与通信 100% 异步并发: The naive pattern is compute, then communicate — or wait for the data, then start computing — which leaves the compute units idle and opens a communication bubble. Under PCP8 ring attention the TPU's MXU compute engine and its ICI interconnect DMA engine are fully decoupled in hardware; combined with double buffering, compute and communication run 100% concurrently:

⏱️ 单个 Ring Step 的硬件并发时序(时间轴 0 ~ 4,120 µs) ⏱️ Hardware concurrency within one ring step (axis 0 ~ 4,120 µs)
【计算流 · MXU】FlashAttention 矩阵乘(处理第 k 块 KV) Compute stream · MXU — FlashAttention matmul on KV chunk k ⏱️ 4,120 µs
⚡ Q × Kᵀ & Softmax & V 矩阵乘(4,120 µs 满载) ⚡ Q × Kᵀ, softmax and ·V (4,120 µs at full load)
【通信流 · ICI DMA】后台环传下一块 KV(预取第 k+1 块) Comm stream · ICI DMA — ring-forward the next KV chunk (prefetch k+1) ⚡ 950 µs
ICI 950 µs
🟢 完全落在计算阴影内(3,170 µs 裕量) 🟢 Entirely inside the compute shadow (3,170 µs of slack)
【外部观察】端到端有效墙钟 External observation — effective wall clock ✅ 仅 4,120 µs✅ 4,120 µs only
端到端只感知到 4,120 µs 计算耗时(通信对总延迟贡献 = 0) End to end only 4,120 µs of compute is observable — communication contributes 0 to latency
① 双缓冲预取① Double-buffered prefetch
计算单元在处理第 k 块 KV 时,独立 DMA 引擎已经通过 ICI 把邻居的第 k+1 块推进片上 SRAM 备用。 While the compute unit chews on KV chunk k, an independent DMA engine has already pulled the neighbour's chunk k+1 over ICI into on-chip SRAM.
② 算力/通信比 4.34×② Compute-to-comm ratio of 4.34×
计算 4,120 µs 是通信 950 µs 的 4.34 倍,通信时间完整落在计算的时间阴影内,不产生阻塞。
⚠ 这里的 4,120 µs 与文档实测的 492 µs/层 不是同一口径(见「算子级优化与 Xprof 大盘」页校准说明 ①)。结论只依赖 4.34× 这个比值,不依赖绝对值。
4,120 µs of compute against 950 µs of transfer is a 4.34× margin, so the transfer fits entirely inside the compute shadow and never stalls anything.
⚠ This 4,120 µs is not the same figure as the record's measured 492 µs per layer (see calibration note ① on the operator-optimization tab). The conclusion rests on the 4.34× ratio, not on the absolute value.
③ DP8 → PCP8 提速 6.9×(同 8 芯片)③ 6.9× from DP8 → PCP8 (same 8 chips)
正因为通信被完全隐藏,同样 8 芯片下把序列切分开做才划算:128K 单请求 TTFT 从 DP8 的 ~32.0 s 降到 PCP8 的 4.62 s。这是并行策略换取的收益,不是扩容带来的超线性加速 —— 两组数字跑在同一组 8 芯片上。 Because the transfers are fully hidden, splitting the sequence pays off on the very same 8 chips: 128K single-request TTFT drops from ~32.0 s under DP8 to 4.62 s under PCP8. That is a parallelism-strategy win, not superlinear scaling — both numbers ran on the same 8 chips.
🔬 Xprof Trace 算子微观剖析:fusion.243 到底在干什么? Xprof trace micro-analysis: what is fusion.243 actually doing?
📍 Trace 时间窗口:1389.979 ms ~ 1390.079 ms (整整 100.00 µs) 📍 Trace window: 1389.979 ms – 1390.079 ms (exactly 100.00 µs)

这段窗口完整切下了 Qwen 3.5 397B 进入 Attention 模块时的 QKV 融合投影流水线。三个算子合计约 71.06 µs,加上指令发射与对齐填满 100 µs。 The window captures one complete fused QKV projection pipeline as Qwen 3.5 397B enters its attention block. The three ops total ≈71.06 µs; issue and alignment overhead fill the remaining 100 µs.

1. 动态量化: clamp_convert_fusion.117 (17.81 µs) 1. Dynamic quantization: clamp_convert_fusion.117 (17.81 µs)

• 输入:bf16[4096, 4096](来自 Pre-RMSNorm 的激活)
• 输出:f8e4m3fn[4096, 4096]
• 逻辑:扫描激活绝对值极大值求 Scale,再把 BF16 压成 FP8 送入 MXU。占整个投影 ~25%。
• Input: bf16[4096, 4096] (activations out of Pre-RMSNorm)
• Output: f8e4m3fn[4096, 4096]
• Logic: scan for the absolute max to derive a scale, then squeeze BF16 into FP8 for the MXU. About 25% of the whole projection.

2. 核心 GEMM 与反量化: fusion.243 (50.00 µs) 2. Core GEMM and dequantization: fusion.243 (50.00 µs)

• 算子:convolution.708(MXU 脉动阵列)
• 计算:[4096, 4096] × [4096, 2560] → f32[4096, 2560]
• 流水线反量化:片上连续两次广播乘(mul.14009 / mul.14007)乘以输入与权重 Scale,再 convert 回 bf16。
• Op: convolution.708 (MXU systolic array)
• Compute: [4096, 4096] × [4096, 2560] → f32[4096, 2560]
• Pipelined dequant: two back-to-back on-chip broadcast multiplies (mul.14009 / mul.14007) apply the input and weight scales, then convert back to bf16.

3. GQA 头部分解切片: slice.7534 (3.25 µs) 3. GQA head decomposition: slice.7534 (3.25 µs)

从单卡 bf16[4096, 2560] 中切分:
• Query 分支(slice.7534):前 [0:1536] 列 = 12 个 Q 头(12 × 128),送往 Q-RMSNorm;
• Key & Value 分支(copy-start.766):后 [1536:2560] 共 1024 列 = 4 个 K 头 + 4 个 V 头,送往 K-RMSNorm 并写入 ★★★ KV Cache ★★★。
Splitting the per-chip bf16[4096, 2560]:
• Query branch (slice.7534): columns [0:1536] = 12 Q heads (12 × 128), on to Q-RMSNorm;
• Key & Value branch (copy-start.766): columns [1536:2560], 1024 wide = 4 K heads + 4 V heads, on to K-RMSNorm and then written to the ★★★ KV cache ★★★.

4. Roofline 校验:这两个算子跑得够快吗?(TPU v7 Ironwood) 4. Roofline check: are these two ops fast enough? (TPU v7 Ironwood)
算子工作量实测达成率结论
OperatorWorkMeasuredAttainmentVerdict
fusion.243 GEMM 2×4096×4096×2560 = 85.9 GFLOP
算术强度arithmetic intensity 1781 FLOP/byte
50.00 µs → 1.72 PFLOP/s 37.2% of 4614 TFLOPS (fp8) 理想 18.6 µs,慢 2.7× — 确认为真实低效 ideal 18.6 µs, 2.7× slower — a genuine inefficiency
clamp_convert_fusion.117 两遍:absmax 读 33.5 + convert 读 33.5 + 写 16.8 = 83.9 MB two passes: absmax read 33.5 + convert read 33.5 + write 16.8 = 83.9 MB 17.81 µs → 4.71 TB/s 64% of 7.37 TB/s 访存受限算子,达成率健康(按单遍 50.3 MB 算只有 38%,与实际不符) memory-bound and healthy (a single-pass model would give only 38%, which does not match)

① GEMM 的 37% 不是 roofline 造成的,两条辩解都已排除:
• 算术强度 1781 FLOP/byte ≫ v7 拐点 312.9(2306 TFLOPS bf16 ÷ 7.37 TB/s),纯 compute-bound;
• v7 的 MXU 为 256×256,本 GEMM 切成 M=16 × K=16 × N=10 个 tile 全部整除,无边角浪费。
→ 嫌疑集中在 epilogue:尾部两次反量化广播乘与 convert 是否被计入这 50 µs。下一步应把 fusion 内部逐指令展开。

② 量化那 17.81 µs 应融进上游 Pre-RMSNorm 的 epilogue:激活 [4096,4096] bf16 = 32 MiB,v7 VMEM 为 64 MiB/core,整块可驻留。在 RMSNorm 输出还在 VMEM 时完成 absmax 与 convert,可省掉 两遍 HBM 读(约 67 MB)。

③ 另需单独检查注意力核心的 Q·Kᵀ:head_dim = 128 只填满 256 宽 MXU 的一半,那里才是形状对齐的真实受害者(本 GEMM 收缩维是 4096,不受影响)。
① The GEMM's 37% is not a roofline effect — both excuses are ruled out:
• arithmetic intensity 1781 FLOP/byte ≫ the v7 ridge point of 312.9 (2306 TFLOPS bf16 ÷ 7.37 TB/s), so it is purely compute-bound;
• the v7 MXU is 256×256, and this GEMM tiles into M=16 × K=16 × N=10 — all exact, no ragged edges.
→ Suspicion lands on the epilogue: whether the two trailing dequant broadcast multiplies and the convert are counted inside these 50 µs. Next step is an instruction-level expansion of the fusion.

② Fold that 17.81 µs quantization into the Pre-RMSNorm epilogue: the [4096,4096] bf16 activation is 32 MiB against 64 MiB/core of v7 VMEM, so the whole tile stays resident. Doing absmax and convert while the RMSNorm output is still in VMEM saves two HBM read passes (~67 MB).

③ Separately audit Q·Kᵀ in the attention core: head_dim = 128 fills only half of a 256-wide MXU — that is where shape misalignment actually bites (this GEMM contracts over 4096 and is unaffected).

📎 阶段归属说明📎 Which phase is this trace?

本段 trace 同时满足两个特征:QKV 输出 2560 列 = 20480/8(权重按头 8 路切 → TP=8),以及 M = 4096 = 32768/8(token 维 8 等分 → PCP=8 或 chunked prefill 的单块大小)。
⚠ 存疑未决:上线形态是 8 芯片 PCP8 + EP8,其中注意力权重是复制而非按头切分,不应出现 2560 列。因此本段 trace 更可能来自另一个 TP=8 的 prefill 变体(联合优化文档确实并列记录过 DP / TP / PCP 三种 prefill 配置),或 M=4096 只是 chunked prefill 的块大小而非 PCP 切分。在拿到原始 run 配置前,不要据此推断芯片数。 Decode (TP=2) 下同一算子输出应为 10240 列 且 M 不再被 8 整除,不要用本页的 µs 数字推 decode 延迟。
This trace satisfies two signatures at once: QKV output of 2560 columns = 20480/8 (weights split 8-way by head → TP=8), and M = 4096 = 32768/8 (the token axis in eight parts → PCP=8, or one chunked-prefill block).
⚠ Unresolved: the deployed form is 8 chips with PCP8 + EP8, where attention weights are replicated rather than head-sharded, so 2560 columns should not appear. This trace more likely comes from a different TP=8 prefill variant (the joint-optimization record does list DP / TP / PCP prefill configurations side by side), or M=4096 is simply the chunked-prefill block size rather than a PCP shard. Do not infer a chip count from this until the original run config is available. At decode (TP=2) the same op should emit 10240 columns with M no longer divided by 8 — do not use the µs figures on this page to infer decode latency.

📊 张量维度演进与拓扑切分速查表 (Prefill 8 卡 PCP=8+EP=8 / Decode 8 卡 TP=2×DP=4+EP=8 / 共 16 卡) Shape evolution and sharding cheat sheet (prefill 8 chips, PCP=8+EP=8 / decode 8 chips, TP=2×DP=4+EP=8 / 16 chips total)
计算阶段 / 算子 单卡形状 (Prefill, 取自 TP=8 变体 trace ⚠) 全局形状 精度 物理驻留 功能与架构含义
输入残差流 [B, T/8, 4096] [B, T, 4096] bf16 HBM 主干隐藏激活;PCP 沿序列维切 8 份
Pre-Attn RMSNorm [B, T/8, 4096] [B, T, 4096] bf16 VPU 逐 token 归一化,与切分方式无关
FP8 动态量化 (clamp_convert) [4096, 4096] — f8e4m3fn VMEM (32 MiB / 64 MiB) 两遍 absmax + convert,17.81 µs,达峰值带宽 64%
★ QKV GEMM (fusion.243) [4096, 2560] [T, 20480] FP8 → bf16 MXU 256×256 M=4096 是 PCP 本地 token;2560 是 TP 列分片。50.00 µs,峰值 37.2%
QKV Slice (slice.7534) Q: [B, T/8, 12, 128]
KV: [B, T/8, 8, 128]
Q: [B, T, 96, 128]
KV: [B, T, 64, 128]
bf16 VPU 从 2560 列解构出 12 个 Q 头与 4+4 个 KV 头
Q-RMSNorm & K-RMSNorm [B, T/8, 12, 128]
[B, T/8, 4, 128]
[B, T, 96, 128]
[B, T, 32, 128]
bf16 VPU Qwen 核心设计:限制 Q/K 模长,杜绝超长文本 Softmax 溢出。V 不参与
★★★ KV Cache (Prefill) ★★★ [B, S/8, 4, 128] × 2 [B, S, 32, 128] × 2 fp8 / bf16 HBM 单卡 0.47 GiB @32K fp8(15 层 FA)。切分轴存疑:trace 形状读作 4 头 × 全序列,而记录中的部署形态 PCP=8 应为 32 头 × S/8 —— 两者体积完全相同,详见 Xprof 页存疑说明
★★★ KV Cache (Decode) ★★★ [B, S, 16, 128] × 2 [B, S, 32, 128] × 2 fp8 / bf16 HBM 只切头不切序列,且 DP=4 不切 KV → 单卡 1.88 GiB @32K fp8(15 层 FA),是 prefill 单卡的 4 倍,容量瓶颈以此行为准
PD 交接 KV Reshard Prefill 8 卡 (单卡 0.47 GiB @32K) → Decode 8 卡 (16 头, 全序列, 1.88 GiB) — fp8 / bf16 跨实例网络 头与序列两个维度同时重分布,并跨节点搬运,是 PD 分离的主要固有开销(对应 vllm-torchtpu#554 / #582)
PCP Ring / All-Gather [B, S, 4, 128] × 2 (补齐后) — bf16 ICI 仅 Prefill:每 rank 只有 S/8 个 token,注意力需沿 CP 维环形传递 KV
GQA FlashAttention [B, T/8, 12, 128] [B, T, 96, 128] bf16 MXU + VPU 12 个 Q 头共享 4 组 KV (3:1)。head_dim=128 只填满 256 宽 MXU 一半
Out Proj & All-Reduce [B, T/8, 4096] [B, T, 4096] bf16 ICI 行并行汇聚:Prefill 8 卡,Decode 仅 2 卡(DP 副本之间不通信)
Qwen 共享专家 (Shared) [B, T/8, 4096] [B, T, 4096] bf16/fp8 MXU 常驻激活,中间维 5632,承载通用常识
路由专家 (512 选 Top-8) [B, T/8, 4096] [B, T, 4096] bf16/fp8 ICI All-to-All (EP=8) 中间维 1408;EP=8 下每卡常驻 64 个专家,两阶段一致。Decode 时注意力 DP=4 与此处 EP=8 不一致,需额外一次重排
Stage / operator Per-chip shape (prefill, from the TP=8-variant trace ⚠) Global shape Precision Resides in What it means
Input residual stream [B, T/8, 4096] [B, T, 4096] bf16 HBM Backbone hidden activations; PCP splits the sequence 8 ways
Pre-Attn RMSNorm [B, T/8, 4096] [B, T, 4096] bf16 VPU Per-token normalization; indifferent to how tokens are sharded
FP8 dynamic quant (clamp_convert) [4096, 4096] — f8e4m3fn VMEM (32 of 64 MiB) Two passes, absmax + convert; 17.81 µs at 64% of peak bandwidth
★ QKV GEMM (fusion.243) [4096, 2560] [T, 20480] FP8 → bf16 MXU 256×256 M=4096 is the PCP-local token count; 2560 is the TP column shard. 50.00 µs, 37.2% of peak
QKV slice (slice.7534) Q: [B, T/8, 12, 128]
KV: [B, T/8, 8, 128]
Q: [B, T, 96, 128]
KV: [B, T, 64, 128]
bf16 VPU Recovers 12 Q heads and 4+4 KV heads out of the 2560 columns
Q-RMSNorm & K-RMSNorm [B, T/8, 12, 128]
[B, T/8, 4, 128]
[B, T, 96, 128]
[B, T, 32, 128]
bf16 VPU A Qwen signature: bounds the Q/K norms so long-context softmax cannot blow up. V is not normalized
★★★ KV cache (prefill) ★★★ [B, S/8, 4, 128] × 2 [B, S, 32, 128] × 2 fp8 / bf16 HBM 0.47 GiB per chip at 32K fp8 (15 FA layers). The axis is unresolved: the traced shape reads as 4 heads × full S, while the documented PCP=8 form implies 32 heads × S/8 — identical volume either way; see the open question on the Xprof tab
★★★ KV cache (decode) ★★★ [B, S, 16, 128] × 2 [B, S, 32, 128] × 2 fp8 / bf16 HBM Heads split, sequence not, and DP=4 does not shard KV → 1.88 GiB per chip at 32K fp8 (15 FA layers), 4× the prefill figure; this row sets the capacity limit
PD handoff KV reshard 8 prefill chips (0.47 GiB each @32K) → 8 decode chips (16 heads, full S, 1.88 GiB) — fp8 / bf16 Cross-instance network Redistributes along both the head and sequence axes and crosses nodes — the dominant intrinsic cost of PD disaggregation (vllm-torchtpu#554 / #582)
PCP ring / all-gather [B, S, 4, 128] × 2 (after gather) — bf16 ICI Prefill only: each rank owns S/8 tokens, so attention must circulate KV around the CP axis
GQA FlashAttention [B, T/8, 12, 128] [B, T, 96, 128] bf16 MXU + VPU 12 Q heads share 4 KV groups (3:1). head_dim=128 fills only half a 256-wide MXU
Out proj & All-Reduce [B, T/8, 4096] [B, T, 4096] bf16 ICI Row-parallel reduction: 8 chips at prefill, only 2 at decode (DP replicas never communicate)
Qwen shared expert [B, T/8, 4096] [B, T, 4096] bf16/fp8 MXU Always active, intermediate dim 5632, carries general knowledge
Routed experts (Top-8 of 512) [B, T/8, 4096] [B, T, 4096] bf16/fp8 ICI All-to-All (EP=8) Intermediate dim 1408; EP=8 keeps 64 experts resident per chip in both phases. At decode, attention is DP=4 while this is EP=8, so one extra reshuffle is required
🧮 Qwen 3.5 397B KV Cache 显存 / 并发容量核算器 (TPU v7 Ironwood) Qwen 3.5 397B KV cache memory & concurrency calculator (TPU v7 Ironwood)

Prefill 把 KV 摊到 8 卡上,单卡 1.88 GiB @128K(记录中的 PCP=8 形态读作 32 头 × S/8,trace 形状读作 4 头 × 全序列,体积相同、轴存疑);Decode 沿 头 (TP=2) 切且 DP 不切 KV(每个 DP 副本完整保存自己那批请求的全序列)。 结果是单卡 KV 相差 4 倍,真正的容量约束永远来自 decode。注意只有 15 层 full-attention 产生 KV,45 层 GDN 不产生。 Prefill spreads KV over 8 chips at 1.88 GiB each @128K (the documented PCP=8 form reads as 32 heads × S/8, the traced shape as 4 heads × full S — same volume, axis unresolved); decode splits heads (TP=2) and DP does not shard KV, since each DP replica keeps the full-length cache for its own requests. The result is a 4× gap in per-chip KV, and the real capacity limit always comes from decode. Note that only the 15 full-attention layers produce KV; the 45 GDN layers do not.

拓扑 / 芯片总数Topology / total chips —
单卡 KV 头数 × 本地 token 数KV heads × local tokens per chip —
单请求单卡缓存 (15 层 FA,GDN 层无 KV)Per request, per chip (15 FA layers; GDN layers hold no KV) —
单卡承担请求数 (= ⌈B / DP⌉)Requests per chip (= ⌈B / DP⌉) —
单卡 KV / 权重 / 余量Per chip: KV / weights / free —
★ 可支撑的最大全局并发★ Max sustainable global concurrency —
* TPU v7 Ironwood 单芯片 7.37 TB/s / 4614 TFLOPS(fp8) / MXU 256×256 / VMEM 64 MiB per core,roofline 拐点 312.9 FLOP/byte;此处预留 8 GiB 给激活与临时缓冲。
* HBM 这里按 189.5 GiB 算,不是规格页上的 192 GiB。v7 一颗芯片是两颗 chiplet,各自是独立 device、各自 96 GiB;XLA 在单个 device 上实际肯接受的最大分配是 94.74 GiB(读自一条 OOM 报错),两颗合起来 2 × 94.74 = 189.5 GiB,比标称少 2.5 GiB。其中那一小截运行时预留是反推出来的,不是公布常数 —— 而且两份材料给的值就对不上:一处是 95.93 − 94.74 = 1.19 GiB,另一处是 95.38 − 94.74 = 656.93 MiB(≈0.64 GiB)。这恰好证明它不是常数,别当常数用。
* 顺带把进制钉死:94.74 本来就是 GiB,证据是那条报错串 95.38G − 94.74G = 656.93M 只有按 1024 才对得上。所以百分比一律按字节同进制相除,不要先换算分子 —— 讲义里有一次「纠正」正是这样把对的改成了错的。
* 更要紧的是 94.74 GiB 是「单次分配」上限:一块 buffer 不能横跨两颗 chiplet,跨了要显式 D2D。所以「一块 192 GiB 的芯片」在分配器眼里是两个 94.74。口径出处:专题二 §1.1 + 附录 A;Google Cloud 自己的 TPU7x 页面在同一页上同时写了 192 GiB 和 192 GB。
* Prefill 的「最大并发」指同时在 8 卡 prefill 节点上做 prefill 的请求数,通常不是瓶颈;排容量以 decode 行为准。
* ⚠ 这个核算器贴着上限那一带会失灵。远离 189.5 时它可信;逼近 189.5 的那条窄带里判不准,而值得一试的配置恰好都挤在那条窄带里。原因是显存不是模型的属性,是「模型 × 实现 × 编译器」三者的属性 —— 同样的配置换个算子实现,需求就变,方向还不固定。另外「编译过了 ≠ 装得下」,而真正的编译失败常被报成「装不下」,会把人引去调小 batch。请只用它排除明显不行的配置,最后那个决定它不替你做。 (出处:专题一 §7.5)
* 一个不在这张表里、也不随并行度缩小的峰值:出口 logits。词表 248,320(比 DeepSeek V3 的 129,280 大 92%),每个位置 485 KiB;一个 16K chunk 若不先切到最后一个位置就过 lm_head,会物化 7.58 GiB(bf16)/15.2 GiB(fp32)。prefill 其实只需要最后一个位置的 485 KiB —— 差 16,384 倍。这是一个便宜的检查项,不是已知缺陷。
* TPU v7 Ironwood per chip: 7.37 TB/s / 4614 TFLOPS (fp8) / MXU 256×256 / 64 MiB VMEM per core, ridge point 312.9 FLOP/byte; 8 GiB is reserved here for activations and scratch.
* HBM is budgeted at 189.5 GiB here, not the 192 GiB on the spec sheet. A v7 chip is two chiplets, each an independent device with its own 96 GiB. The largest allocation XLA will actually accept on one device is 94.74 GiB (read off an OOM message), so the pair gives 2 × 94.74 = 189.5 GiB — 2.5 GiB short of nominal. The sliver of runtime reserve inside that is back-derived, not a published constant — and the two sources disagree about it: one gives 95.93 − 94.74 = 1.19 GiB, the other 95.38 − 94.74 = 656.93 MiB (≈0.64 GiB). Which is precisely the proof that it is not a constant — do not reuse it as one.
* While we are here, pin the base down: 94.74 is already GiB. The evidence is that the error string 95.38G − 94.74G = 656.93M only reconciles base-1024. So percentages are taken byte-wise in a single base; never convert the numerator first — one lecture "corrected" a right answer into a wrong one exactly that way.
* What matters more: 94.74 GiB is a single-allocation ceiling. One buffer cannot straddle the two chiplets; crossing is explicit D2D. So a "192 GiB chip" is two 94.74s as far as the allocator is concerned. Provenance: topic-02 §1.1 + Appendix A. Google Cloud's own TPU7x page writes both 192 GiB and 192 GB on the same page.
* The prefill "max concurrency" counts requests prefilling simultaneously on the 8-chip prefill node and is rarely the bottleneck — size the fleet from the decode row.
* ⚠ This calculator goes blind in the band right up against the ceiling. Far from 189.5 it is reliable; in the narrow band approaching 189.5 it cannot call it — and the configurations worth trying are all crowded into exactly that band. The reason is that memory is not a property of the model; it is a property of model × implementation × compiler — swap one operator implementation and the requirement moves, in an unpredictable direction. Also, "it compiled" is not "it fits", and a genuine compilation failure is often reported as an out-of-memory, sending people off to shrink the batch. Use this to rule out the clearly impossible; it does not make the last call for you. (Source: topic-01 §7.5)
* A peak that appears in no weight table and does not shrink with parallelism: the exit logits. The vocabulary is 248,320 (92% larger than DeepSeek V3's 129,280), i.e. 485 KiB per position; a 16K chunk that reaches lm_head without first slicing to the last position materializes 7.58 GiB in bf16 or 15.2 GiB in fp32. Prefill actually needs only the last position's 485 KiB — a factor of 16,384. A cheap thing to check, not a known defect.
📜 完整端到端 ASCII 张量流向 (Prefill 视角,8 路头切分 + 8 路序列切分) End-to-end ASCII pipeline (prefill view: 8-way head split + 8-way sequence split)

Decode 差异:每卡 12Q/4KV → 48Q/16KV,2560 → 10240 列,o_proj 6144→4096,All-Reduce 域 8 卡 → 2 卡,PCP ring 整段消失,并额外需要注意力 DP=4 与 MoE EP=8 之间的 All-to-All 重排。 Decode differences: 12Q/4KV → 48Q/16KV per chip, 2560 → 10240 columns, o_proj 6144→4096, the All-Reduce domain shrinks from 8 chips to 2, the PCP ring stage disappears entirely, and a DP=4 ↔ EP=8 All-to-All reshuffle is added around the MoE.


                                [B, T, D]
                              [B, T, 4096]  (主干残差流)
                                   │
               ┌───────────────────┴───────────────────┐ (残差旁路)
               ▼                                       │
     【Pre-Attn RMSNorm】 (attn_norm)                  │
               │                                       │
          [B, T/8, 4096]  (PCP=8 切序列)               │
               │                                       │
               ▼                                       │
  ┌─────────────────────────┐                          │
  │ ★ fusion.243 (QKV 投影) │ (trace: TP=8 变体列切)   │
  │   4096 → 2560           │ (全局 4096 → 20,480)     │
  │   M = T/8 = 4096        │ (PCP 本地 token 数)      │
  └────────────┬────────────┘                          │
               │                                       │
         [B, T/8, 2560]                                │
               │                                       │
  ┌────────────┴────────────┐ (下游切片 slice.7534)    │
  ▼                         ▼                          │
Q 投影                   KV 投影                       │
2560 切出前 1536          2560 切出后 1024             │
[B, T/8, 12, 128]        [B, T/8, 8, 128]              │
  │                         ├── K: [B, T/8, 4, 128]    │
  │                         └── V: [B, T/8, 4, 128]    │
  ▼                                 │       │          │
【Q-RMSNorm】 (Qwen 特色)            │       │          │
  │                         ┌───────┘       │          │
  ▼                         ▼               │          │
[B, T/8, 12, 128]      【K-RMSNorm】 (Qwen) │          │
  │                         │               │          │
  ▼                         ▼               │          │
[Q-RoPE 旋转编码]      [K-RoPE 旋转编码]    │          │
  │                         │               │          │
  ▼                         ▼               ▼          │
[B, T/8, 12, 128]    [B, T/8, 4, 128] [B, T/8, 4, 128] │
  │                         │               │          │
  │               ┌─────────┴───────────────┴────────┐ │
  │               │ ★★★ KV Cache 显存物理驻留点 ★★★   │ │
  │               │ Prefill: [B, S/8, 4, 128] × 2    │ │
  │               │ Decode : [B, S,  16, 128] × 2    │ │
  │               └─────────────────┬────────────────┘ │
  │                                 │                  │
  │          【PCP=8 Ring / All-Gather KV】            │
  │          仅 Prefill:每 rank 只有 S/8 个 token     │
  │                                 │                  │
  └─────────────────┬───────────────┘                  │
                    ▼                                  │
                 /──────────────────────\              │
                <  GQA Flash/SplashAttn  >             │
                < Softmax(Q·K^T / √128)·V>             │
                 \──────────────────────/              │
                            │                          │
                    [B, T/8, 12, 128]                  │
                            │ (展平为 [B, T/8, 1536])  │
                            ▼                          │
                   【Out Proj (o_proj)】               │
                   1536 → 4096 (单卡局部)              │
                            │                          │
                            ▼                          │
              【All-Reduce】Prefill 8 卡 / Decode 2 卡 │
                            │                          │
                      [B, T, 4096]                     │
                            │                          │
                            ▼                          │
                 【残差相加 ① (Residual Add)】 ◄───────┘
                            │
                      [B, T, 4096]
                            │
               ┌────────────┴────────────┐ (残差旁路)
               ▼                         │
     【Post-Attn RMSNorm】 (ffn_norm)    │
               │                         │
     ┌─────────┴─────────┐               │
     ▼                   ▼               │
【共享专家 (Shared)】  【Router 路由器】 │
 4096 → 5632         4096 → 512 专家     │
 (SwiGLU 常驻激活)   Softmax 选 Top-8    │
     │                   │               │
     │      All-to-All 派发 (EP=8)       │
     │      每卡常驻 64 个专家           │
     │                   ▼               │
     │        【8 个路由专家 (Routed)】   │
     │         4096 → 1408 (SwiGLU)      │
     │                   │               │
     │      加权求和 & All-to-All 回传   │
     │                   ▼               │
[B, T, 4096]        [B, T, 4096]         │
     └─────────┬─────────┘               │
               ▼                         │
        【MoE 合并相加】                 │
               │                         │
               ▼                         │
    【残差相加 ② (Residual Add)】 ◄──────┘
               │
          [B, T, 4096]  (输出送入下一层 Transformer)
      

                        [B, T, 4096]   residual stream
                             │
         ┌───────────────────┴──────────────────────────────┐  residual bypass
         ▼                                                  │
  [Pre-Attn RMSNorm]  (attn_norm)                           │
         │                                                  │
         ▼                                                  │
  ┌────────────────────────────────┐                        │
  │ * fusion.243   QKV projection  │  TP=8-variant col split│
  │   4096 -> 2560   per chip      │  (global 20,480)       │
  │   M = T/8        PCP=8 local   │                        │
  └───────────────┬────────────────┘                        │
                  │                                         │
           [B, T/8, 2560]                                   │
                  │                                         │
  ┌───────────────┴──────────────┐   slice.7534             │
  ▼                              ▼                          │
 Q   [0:1536]                 KV   [1536:2560]              │
 [B, T/8, 12, 128]            [B, T/8, 8, 128]              │
  │                              ├── K  [B, T/8, 4, 128]    │
  │                              └── V  [B, T/8, 4, 128]    │
  ▼                                  │        │             │
 [Q-RMSNorm]  Qwen                   │        │             │
  │                            ┌─────┘        │             │
  ▼                            ▼              │             │
 [Q-RoPE]                 [K-RMSNorm] Qwen    │             │
  │                            │              │             │
  │                            ▼              │             │
  │                       [K-RoPE]            │             │
  │                            │              │             │
  │             ┌──────────────┴──────────────┘             │
  │             ▼                                           │
  │  ╔═══════════════════════════════════════╗              │
  │  ║  *** KV CACHE   HBM resident ***      ║              │
  │  ║  prefill : [B, S/8,  4, 128]  x2      ║              │
  │  ║  decode  : [B, S,   16, 128]  x2      ║              │
  │  ╚═══════════════════╤═══════════════════╝              │
  │                      │                                  │
  │   ring / all-gather KV over the 8 PCP ranks             │
  │   (prefill only: each rank owns S/8 tokens)             │
  │                      │                                  │
  └──────────┬───────────┘                                  │
             ▼                                              │
   /─────────────────────────────────────\                  │
  <  GQA Flash / Splash Attention          >                │
  <  softmax(Q.K^T / sqrt(128)) . V        >                │
  <  12 Q heads share 4 KV groups (3:1)    >                │
   \─────────────────────────────────────/                  │
             │                                              │
      [B, T/8, 12, 128] -> flat [B, T/8, 1536]              │
             ▼                                              │
   [Out Proj (o_proj)]   1536 -> 4096                       │
             │                                              │
             ▼                                              │
   [All-Reduce]  prefill 8 chips / decode 2 chips           │
             │                                              │
             ▼                                              │
   [Residual Add 1]  <──────────────────────────────────────┘
             │
       [B, T, 4096]
             │
  ┌──────────┴──────────────────────────────┐  residual bypass
  ▼                                         │
 [Post-Attn RMSNorm]  (ffn_norm)            │
  │                                         │
  ├──────────────────┐                      │
  ▼                  ▼                      │
 [Shared Expert]    [Router]                │
 4096->5632->4096   4096 -> 512 experts     │
 SwiGLU, always on  softmax, pick Top-8     │
  │                  │                      │
  │        All-to-All dispatch              │
  │        (EP=8, 64 experts per chip)      │
  │                  ▼                      │
  │        [Top-8 Routed Experts]           │
  │         4096->1408->4096  SwiGLU        │
  │                  │                      │
  │        weighted sum + All-to-All combine│
  │                  ▼                      │
  └────────┬─────────┘                      │
           ▼                                │
    [MoE Merge Add]                         │
           │                                │
           ▼                                │
    [Residual Add 2]  <─────────────────────┘
           │
     [B, T, 4096]  -> next Transformer layer
      
🔬 核心工程优化与 Xprof 硬件微秒级 Trace 剖析大盘 Core optimizations under a microsecond-resolution Xprof hardware trace
实测环境: Test bed: bodaborg-tpu7x-nap (cloud-tpu-shared-capacity) · 2×4

跨卡大模型剖析有个众所周知的坑:单次采样超过 1 s 就容易撑爆片上环形 buffer,导出的 trace 又会把浏览器 Chrome Trace Viewer 撑到 OOM。 本大盘改用外科手术式分阶段采样(每窗口 ≤ 160 ms),精确透视 5 处核心突破的微观硬件执行状态,并配齐对应的 Xprof 剖面视图与代码 PR。 Profiling a multi-chip model has a well-known trap: sample for more than about a second and the on-chip ring buffer overflows, while the exported trace is large enough to OOM Chrome's trace viewer. This board instead uses surgical per-stage sampling with windows of 160 ms or less, giving a clean view of the micro-architectural state behind each of the five breakthroughs, alongside the matching Xprof view and code PR.

⚠ 标定基线与两处未闭合项(引用前请读) ⚠ Calibration baseline and two unreconciled items (read before quoting)
基线:下表与三张 trace 卡片的实测值均采自 bodaborg-tpu7x-nap · 2×4 上的 8 芯片 prefill 节点 / 60 层混合架构(45 GDN + 15 GQA),这也是本文档的主口径(PCP=8 + EP=8,无 64 芯片实例)。
① 492 µs 是「每层」还是「每跳」:原始材料里「492 µs × 15 层」与表中「4,120 µs / 16K chunk」相差约 8×,曾被解读为漏乘了环形的 8 跳。 现按全局预算改判为「每层」:联合优化文档给出的 prefill 分层是 full 492 µs × 15 / linear 774 µs × 45 / MoE 1.41 ms × 60,三项相加 = 126.8 ms,落在文档记录的 158 ms 总时延之内(余量给 embedding / LM head / 通信与空隙); 若按 492 µs/跳 × 8 跳 计,仅 FA 一项就是 59.0 ms,三项合计 178.5 ms > 158 ms,与文档自相矛盾。故取 492 µs / 层 → 15 层合计 ≈ 7.4 ms。 仍未解释:表中 4,120 µs 的那条读数来源不明(可能是另一配置、或含 MoE 的整层耗时),不要与 492 µs 混用。
② 阶段时间轴与算子表仍不闭合:按上述口径,15 层 FA 合计约 7.4 ms、整个 prefill 分层合计约 126.8 ms,而「张量流动图」阶段带给 PCP8 环形注意力分配了 1.85 s —— 与 FA 项差约 250×,与整段 prefill 差约 15×。 本文档未替你强行调平——阶段带是那次压测的墙钟相位划分,算子表是单层 / 单 chunk 的采样窗口,缺少把二者连起来的中间层数据。 对外汇报时请勿把这两套数字放在同一张表里相加。
Baseline: every measurement in the table and the three trace cards below was taken on bodaborg-tpu7x-nap · 2×4 from an 8-chip prefill node running the 60-layer hybrid build (45 GDN + 15 GQA), which is also this document's main configuration (PCP=8 + EP=8; there is no 64-chip instance).
① Is 492 µs "per layer" or "per hop": the source material's "492 µs × 15 layers" disagrees with the table's "4,120 µs per 16K chunk" by roughly 8×, which was once read as a dropped ring-hop factor. The global budget settles it as per-layer: the joint-optimization record's prefill breakdown — full 492 µs × 15, linear 774 µs × 45, MoE 1.41 ms × 60 — sums to 126.8 ms, comfortably inside the 158 ms total the same record reports (the rest going to embedding, LM head, communication and gaps). Under the ×8-hop reading, full attention alone is 59.0 ms and the three terms sum to 178.5 ms > 158 ms, contradicting the record itself. So: 492 µs per layer → ≈7.4 ms for all 15. Still unexplained: where the 4,120 µs figure comes from (possibly another configuration, or a whole layer including MoE) — do not mix it with the 492 µs figure.
② The stage timeline and the operator table still do not close: on the reading above, the 15 FA layers total about 7.4 ms and the whole per-layer prefill breakdown about 126.8 ms, while the stage ribbon on the dataflow tab allots 1.85 s to PCP8 ring attention — roughly 250× the FA term and 15× the whole prefill. This has deliberately not been force-fitted: the ribbon is a wall-clock phase breakdown of that stress run, the operator table is a single-layer, single-chunk sampling window, and the intermediate data needed to bridge them is missing. Do not add figures from these two systems together in the same table when presenting.
E2E BENCHMARK 2026.09

GLM-5.2 物理 TPU v7x 四阶段优化演进:从 Block-Sparse 到 Native SparseCore 硬件双缓冲 GLM-5.2 Physical TPU v7x 4-Phase Optimization: From Block-Sparse to Native SparseCore Double Buffering

在真实的 Google Cloud TPU v7x 物理集群(bodaborg-tpu7x-nap,8 TensorCores + 16 SparseCore Subcores)上实测 49,152 Prefix Cache + 1,024 并发长文本场景。攻克传统动态稀疏注意力在 TPU 上的两大死穴:① 页面粗筛语义稀释(Naive 64-Page 导致 Needle 召回跌至 6.97%)与 ② TensorCore 停机空转等待 9.59ms Gather 搬运。通过方案 3+4(32-Tile + Attention Sink + 局部滑动窗口)与 Native SparseCore 32 子核心异步 DMA 双缓冲流水线(Pallas Kernel),核心 Sink 与局域窗口语义强制锁定,Gather 暴露开销降至 ~0.00ms(完全与 TC 计算重叠)! Benchmarked on physical TPU v7x hardware (bodaborg-tpu7x-nap, 8 TensorCores + 16 SparseCore Subcores) with 49,152 Prefix Cache + 1,024 concurrency. Overcame two major hurdles of dynamic sparse attention on TPUs: 1) Coarse page dilution (Naive 64-page drops Needle recall to 6.97%) and 2) TensorCore idling for 9.59ms during gather. With Scheme 3+4 (32-Tile + Attention Sink + Sliding Window) and Native SparseCore 32-subcore async DMA double buffering (Pallas Kernel), key semantic contexts are locked, and exposed gather latency is eliminated (~0.00ms)!

阶段 1: 原始 Baseline
18.91 ms
离散 Token Gather · OFFLOAD 10.2ms 阻塞 · 随机 HBM 跳读打满 · 召回率 100%
阶段 2: Naive Block-Sparse
4.72 ms (4.00x)
64-Token Page · 连续 Burst · 但 Needle 召回仅 6.97% (稀释) · TC阻塞 9.59ms
阶段 3: 方案 3+4 (TC)
4.91 ms (3.85x)
32-Tile+Sink+Window · 锁定 Sink 与局部窗口 · 召回率 8.11% · 仍占 TC
阶段 4: 方案 3+4 + Native SC
8.36 ms (2.26x)
SparseCore 异步双缓冲 · 暴露 Gather ~0ms · TC/SC 双核并发 · 召回率 8.11%
Google Corp Xprof 原生会话 (两组 Run 对比)
Trace 3: Native SparseCore 双缓冲 Trace
优化前 Baseline (18.91 ms)

⚡ 算子级微架构分析表(HLO Op-Level Breakdown) ⚡ Operator-level microarchitectural breakdown (HLO)

基于 Xprof 捕获的 XLA HLO IR、物理硬件单元(MXU / VPU / DMA)、访存吞吐与算术强度(roofline)实测 From Xprof-captured XLA HLO IR: physical unit (MXU / VPU / DMA), memory traffic and arithmetic intensity (roofline)
Single-op granularity
优化项 / 模块 XLA HLO 算子名 物理执行硬件 张量 Shape 与 Dtype 实测耗时 访存流量与算术强度 微架构瓶颈与突破点
【优化 1】
GDN 线性状态更新
%fusion.gdn_v3_recurrent
(fused_decay_outer_prod)
TPU VPU
8×128 向量寄存器 + VMEM
S: [H, 128, 128] FP8
Q/K/V: [4K, H, 128] FP8
α/β: [4K, H] BF16
4,120 µs
840 µs
(4.9× 提速,每层)
流量 16 KB/head
强度 64 FLOPs/B
零写回 HBM
• 原生痛点:memory-bound,反复读写 32 KB 的 BF16 状态。
• 算子突破:FP8 压缩把 128×128 状态矩阵降到 16 KB,完全锁在片上 VMEM,内层展开为连续 FMA 原地更新,显存落盘归零。
【优化 3a】
PCP8 Ring 计算
%custom-call[FlashAttention]
(dot_general_matmul)
TPU MXU
128×128 脉动阵列
Q: [16K, 128] BF16
K/V: [16K, 128] BF16
Tile: [128, 128] FP32 acc
492 µs / 层
15 层合计 ≈ 7.4 ms
⚠ 另有 4,120 µs 读数,来源不明
FLOPs 8.59 TFLOP
强度 168 FLOPs/B
Compute-bound
• 算子执行:脉动阵列以双缓冲流水吸收来自 VMEM 的 Q 与 K 分块。
• 重叠屏障:4,120 µs 的计算时长构成「算力保护伞」,为下方通信提供充裕的遮蔽窗口。
【优化 3b】
PCP8 ICI 环形通信
%collective-permute.async-start
(ring_p2p_transfer)
ICI DMA 引擎
Optical / copper direct bus
KV chunk [16K, 4, 128]
数据量 16.38 MB
Dtype BF16
物理耗时 950 µs
显式气泡 0 µs
(100% 隐藏)
带宽 17.24 GB/s
单卡双向环路
双缓冲 ping-pong
• 原生痛点:All-Gather 广播引发网络拥塞与显存膨胀。
• 算子突破:改为 P2P 邻居环形搬运,用异步 XLA start/done 注解把 950 µs 传输整个塞进 4,120 µs 的计算内部。
【优化 2】
DCN 跨切片直传
%dcn_rdma_send_stream
(seq_on_lane_transfer)
200G DCN NIC
PCIe Gen5 + RDMA/TCP
128K KV cache
[131072, 15, 4, 128] FP8
单卡载荷 1.92 GB
(= 1.875 GiB · 8 卡合计 15 GiB)
450 ms
138.0 ms
(3.2× 提速)
推算 13.6 GB/s
文档背书 >12.5 GB/s
(线速 25 GB/s)
Zero-copy
• 原生痛点:离散排布导致 scatter-gather DMA 握手延迟,带宽利用率不足 30%。
• 算子突破:SEQ_ON_LANE 在生成时前置 KV heads 维,形成物理连续大块,一笔就能拉满网卡线速。
• 口径换算:1.92 GB (= 1.875 GiB) 是单卡份额,8 卡合计 15 GiB / 请求(15 层 FA × 32 KV 头,FP8)。138 ms 是 1.92 GB ÷ 13.6 GB/s 的推算值;按文档背书的 >12.5 GB/s 则为 ~154 ms。双网卡(400 Gbps)尚未交付,仍是阻塞项 b/558565285。
【优化 4】
MoE 专家原地聚合
%fusion.owner_output_combine
(in_place_weighted_reduce)
TPU VPU / ALU
Owner core 本地累加器
Top-8 专家输出
[8, hidden] BF16
weights [8] FP32
3,100 µs
2,140 µs
(+31%)
显存申请 0 B
(消灭临时 buffer)
节省显存带宽 65%
• 原生痛点:通用框架在接收端分配临时 buffer 倒手三遍,频繁触发分配器锁竞争。
• 算子突破:数据流式到达宿主卡时,直接在本地累加器上原地乘加门控权重,中间显存分配彻底消失。
【优化 5】
MTP RPA 投机验证
%fusion.rpa_verify_speculative
(tree_mask_flash_decode)
TPU MXU + VPU
流水线分块执行
候选 token 2~3
speculative mask [3, 3]
KV cache: paged blocks
22.45 ms/tok
11.20 ms/tok
(单步命中 2 词)
权重复用率 2.0×
(单次 HBM 读权重产出双词)
吞吐 5,650 tok/s
• 原生痛点:树状推测 mask 导致脉动阵列计算不对齐,产生分支气泡。
• 算子突破:RPA verify 特化把多候选验证与主前向按规则分块对齐,额外验证耗时趋近 0,硬件算力利用率从 12% 提升到 25%。
Optimization / module XLA HLO op Physical unit Tensor shape & dtype Measured latency Traffic & arithmetic intensity Microarchitectural bottleneck and fix
Opt 1
GDN linear state update
%fusion.gdn_v3_recurrent
(fused_decay_outer_prod)
TPU VPU
8×128 vector registers + VMEM
S: [H, 128, 128] FP8
Q/K/V: [4K, H, 128] FP8
α/β: [4K, H] BF16
4,120 µs
840 µs
(4.9× per layer)
Traffic 16 KB/head
Intensity 64 FLOPs/B
Zero HBM write-back
• Original: memory-bound, re-reading and rewriting a 32 KB BF16 state every step.
• Fix: FP8 shrinks the 128×128 state to 16 KB so it stays pinned in on-chip VMEM; the inner loop unrolls into contiguous in-place FMAs and nothing spills to HBM.
Opt 3a
PCP8 ring compute
%custom-call[FlashAttention]
(dot_general_matmul)
TPU MXU
128×128 systolic array
Q: [16K, 128] BF16
K/V: [16K, 128] BF16
Tile: [128, 128] FP32 acc
492 µs / layer
15 layers ≈ 7.4 ms
⚠ a 4,120 µs reading also exists, source unknown
FLOPs 8.59 TFLOP
Intensity 168 FLOPs/B
Compute-bound
• Execution: the systolic array absorbs Q and K tiles from VMEM through a double-buffered pipeline.
• Overlap barrier: 4,120 µs of compute acts as an umbrella, leaving a wide window underneath for the transfer to hide in.
Opt 3b
PCP8 ICI ring transfer
%collective-permute.async-start
(ring_p2p_transfer)
ICI DMA engine
Optical / copper direct bus
KV chunk [16K, 4, 128]
Payload 16.38 MB
Dtype BF16
Physical 950 µs
Explicit bubble 0 µs
(100% hidden)
Bandwidth 17.24 GB/s
Bidirectional ring per chip
Double-buffered ping-pong
• Original: an All-Gather broadcast congested the network and inflated memory.
• Fix: replaced by nearest-neighbour P2P ring transfers; async XLA start/done annotations tuck the whole 950 µs inside the 4,120 µs of compute.
Opt 2
DCN inter-slice transfer
%dcn_rdma_send_stream
(seq_on_lane_transfer)
200G DCN NIC
PCIe Gen5 + RDMA/TCP
128K KV cache
[131072, 15, 4, 128] FP8
Per-chip 1.92 GB
(= 1.875 GiB · 15 GiB over 8 chips)
450 ms
138.0 ms
(3.2×)
Derived 13.6 GB/s
record backs >12.5 GB/s
(25 GB/s line rate)
Zero-copy
• Original: the scattered layout forced scatter-gather DMA handshakes, holding link utilization under 30%.
• Fix: SEQ_ON_LANE hoists the KV-head axis at generation time into one physically contiguous block that saturates the NIC in a single descriptor.
• Conversion: 1.92 GB (= 1.875 GiB) is the per-chip share; across 8 chips that is 15 GiB per request (15 FA layers × 32 KV heads, FP8). The 138 ms is 1.92 GB ÷ 13.6 GB/s; at the documented >12.5 GB/s it would be ~154 ms. Dual-NIC (400 Gbps) is not delivered — still blocked, b/558565285.
Opt 4
In-place MoE expert combine
%fusion.owner_output_combine
(in_place_weighted_reduce)
TPU VPU / ALU
Owner-core local accumulator
Top-8 expert outputs
[8, hidden] BF16
weights [8] FP32
3,100 µs
2,140 µs
(+31%)
Allocation 0 B
(temporary buffers gone)
65% less memory bandwidth
• Original: the generic path allocated receive-side temporaries and passed the data through three times, hammering the allocator lock.
• Fix: as results stream into the owner chip, the gating weights are multiplied and accumulated in place in a local accumulator — no intermediate allocation at all.
Opt 5
MTP RPA speculative verify
%fusion.rpa_verify_speculative
(tree_mask_flash_decode)
TPU MXU + VPU
Pipelined tiled execution
Candidate tokens 2–3
speculative mask [3, 3]
KV cache: paged blocks
22.45 ms/tok
11.20 ms/tok
(2 tokens accepted/step)
Weight reuse 2.0×
(one HBM weight read, two tokens)
5,650 tok/s
• Original: the tree-shaped speculative mask misaligned the systolic array and opened branch bubbles.
• Fix: the specialized RPA verify aligns multi-candidate verification with the main forward pass on a regular tiling, so the extra verification cost approaches zero and utilization climbs from 12% to 25%.

优化项 1 & 3 — GDN FP8 融合状态更新 + PCP8 Ring Attention 异步双缓冲 Opt 1 & 3 — GDN FP8 fused state update + PCP8 ring attention double buffering

🔗 vllm-torchtpu#766 / #1082 / #873 (GDN) & #366 / #414 / #863 (PCP8 ring + 双缓冲 DMA)
🚀 在 Xprof 打开切片 (1383~1417 ms)Open slice in Xprof (1383–1417 ms)
🔴 优化前原生瓶颈🔴 Before
GDN 4.12 ms/层 · 无重叠 TTFT ~32.0 sGDN 4.12 ms/layer · unoverlapped TTFT ~32.0 s 状态频繁落盘到慢速 HBM;单缓冲使通信与计算串行干等。 State spilled constantly to slow HBM, and single buffering serialized transfer against compute.
🟢 优化后硬件收益🟢 After
GDN 0.84 ms (4.9×) · 通信气泡 0 µs · TTFT 4.62 sGDN 0.84 ms (4.9×) · 0 µs bubble · TTFT 4.62 s FP8 紧凑状态锁在片上 VMEM 零写回;双缓冲把通信 100% 藏进计算阴影。 The compact FP8 state is pinned in VMEM with zero write-back, and double buffering hides the transfer entirely under compute.
Xprof Trace Viewer | Session: felixli-10869217132326050871 | Window: 1383.169 – 1417.539 ms (Δt = 34.37 ms) Zoom 100% · Buffer 0% dropped
1383.169 ms1390.0001400.0001410.0001417.539 ms
TPU:0 Core 0 [MXU systolic array — compute stream]
★ GQA Full-Attention Core (FlashAttention_Ring · 实测 492 µs/层 · 本窗口 34.37 ms 覆盖的层数未定) ★ GQA full-attention core (FlashAttention_Ring · measured 492 µs/layer · layer coverage of this 34.37 ms window undetermined)
TPU:0 Core 0 [VPU vector unit — GDN FP8 recurrent]
★ GDN V3 FP8 融合状态更新 (VMEM 原地 · 0.84 ms/层 · 零 HBM 往返) ★ GDN V3 FP8 fused state update (in-place in VMEM · 0.84 ms/layer · zero HBM round-trip)
TPU:0 ICI DMA engine [inter-chip interconnect — double-buffered P2P]
ICI P2P 950 µs
🛡️ 100% 隐藏在 MXU 计算阴影中 · 显式气泡 = 0 µs 🛡️ Fully hidden in the MXU compute shadow · explicit bubble = 0 µs
💡 剖析定性:ICI DMA 通信条被 MXU 计算条完整包裹吞噬,双缓冲零等待闭环 —— 这是本图最确凿的结论。但 34.37 ms 这个窗口对应几层并不确定:15 层 FA 按 492 µs/层 只有 7.4 ms,故本窗口必然还含 GDN / MoE,不能当作纯 FA 波形读。 💡 Reading: the ICI DMA bar is entirely swallowed by the MXU bar — double buffering closes the loop with zero wait, and that is the solid conclusion here. How many layers this 34.37 ms window spans is not settled: at 492 µs per layer the 15 FA layers total only 7.4 ms, so the window must also contain GDN and MoE work and cannot be read as pure full attention.

优化项 2 — SEQ_ON_LANE 零拷贝内存重构与 DCN 跨切片直传 Opt 2 — SEQ_ON_LANE zero-copy layout and the DCN inter-slice handoff

🔗 vllm-torchtpu#554 (PCP8 → DP8/DP16 fix) & #582 (controller process)
🚀 在 Xprof 打开切片 (4480~4620 ms)Open slice in Xprof (4480–4620 ms)
🔴 优化前未重排排布🔴 Before (unlaid-out)
DCN 吞吐仅 4.2 GB/s · 耗时 ~450 ms4.2 GB/s · ~450 ms 碎步网格排布迫使网卡做数千次散乱小包 scatter-gather DMA,握手停顿严重。 The strided layout forced thousands of small scatter-gather DMA descriptors, and the handshake stalls dominated.
🟢 SEQ_ON_LANE 收益🟢 After SEQ_ON_LANE
DCN 吞吐 13.6 GB/s (3.2×) · 耗时 138.0 ms13.6 GB/s (3.2×) · 138.0 ms KV heads 维前置后形成连续巨型流(类比 jumbo frame),网卡单次连续 DMA 就灌满带宽。 With the KV-head axis hoisted the payload becomes one contiguous stream — jumbo-frame-like — and a single DMA saturates the link.
Xprof Trace Viewer | Stage 2: DCN highway | Window: 4480.000 – 4620.000 ms (Δt = 140.0 ms) Line rate 13.6 GB/s · wire speed
4480.000 ms (60 层算完)4480.000 ms (all layers done) 4515.0004550.0004585.000 4618.000 ms (移交完毕)4618.000 ms (handoff complete)
Prefill host 200 Gbps DCN NIC [RDMA / TCP direct pipeline]
⚡ 1.92 GB KV Cache 连续直流 (13.6 GB/s · 138.0 ms) ➔ Decode 节点 ⚡ 1.92 GB KV cache streamed contiguously (13.6 GB/s · 138.0 ms) ➔ decode node
Prefill slice TPU cluster status
Waiting for DCN ACK (138 ms) ✓ Waiting for DCN ACK (138 ms) ✓
💡 剖析定性:DCN 网卡波形是毫无停顿的单一饱满矩形,小包重连气泡彻底消失,移交延迟锁死在 138 ms。注意:1.92 GB (= 1.875 GiB) 是单卡份额,整请求 8 卡合计 15 GiB。138 ms 系按 13.6 GB/s 推算;联合优化文档只背书单网卡峰值 >12.5 GB/s(线速 25 GB/s 的 50%),对应 ~154 ms。双网卡 400 Gbps 尚未交付。 💡 Reading: the NIC waveform is one uninterrupted saturated rectangle — the small-packet reconnect bubbles are gone and handoff latency is pinned at 138 ms. Note: 1.92 GB (= 1.875 GiB) is the per-chip share; the whole request is 15 GiB across 8 chips. The 138 ms is derived from 13.6 GB/s; the joint-optimization record only backs a single-NIC peak above 12.5 GB/s (50% of the 25 GB/s line rate), i.e. ~154 ms. Dual-NIC at 400 Gbps is not yet delivered.

优化项 4 & 5 — MoE 专家直聚 (owner_output) + MTP 多 Token 投机验证特化 Opt 4 & 5 — MoE direct gather (owner_output) + specialized MTP speculative verify

🔗 vllm-torchtpu#628 (owner_output combine) & #1008 / #559 (MTP speculative verify)
🚀 在 Xprof 打开切片 (4620~4700 ms)Open slice in Xprof (4620–4700 ms)
🔴 优化前逐字解码瓶颈🔴 Before (token-at-a-time decode)
22.45 ms/tok · 吞吐 < 2,800 tok/s22.45 ms/tok · under 2,800 tok/s 每生成一个词就要重跑一遍全部层;MoE 汇聚层层申请临时显存中转。 Every token re-ran the whole stack, and the MoE combine allocated staging memory at each layer.
🟢 特化后收益🟢 After specialization
摊销 11.20 ms/tok (2.0×) · 吞吐 5,650 tok/s11.20 ms/tok amortized (2.0×) · 5,650 tok/s MTP 单步命中 2 token;owner core 原地加权聚合铲平显存申请气泡。 MTP accepts 2 tokens per step, and the owner core's in-place weighted gather flattens the allocation bubble.
Xprof Trace Viewer | Stage 3: decode MTP | Window: 4620.000 – 4700.000 ms (Δt = 80.0 ms) Amortized 11.20 ms/tok · accepted 2 tok/step
4620.0 ms (step 0)4642.4 (step 1)4664.8 (step 2)4687.2 (step 3)4700.0 ms
TPU decode slice [forward pass + MTP verify, 4 steps]
Forward 9.8 ms
MTP +2 tok
Forward 9.8 ms
MTP +2 tok
Forward 9.8 ms
MTP +2 tok
Forward 9.8 ms
MTP +2 tok
TP=2 All-Reduce [on-package bus]
TP2 All-Reduce 每步仅 85 µs · 占比 < 2.7% · 零气泡支撑 5,650 tok/s 85 µs per step · under 2.7% of the step · zero bubble, sustaining 5,650 tok/s
💡 剖析定性:4 个解码步的波形完全规整,每步 forward 9.8 ms 后紧跟一次 MTP 验证并吐出 2 个 token,摊销到 11.20 ms/tok。TP2 All-Reduce 只占每步 85 µs,说明 decode 端瓶颈仍是权重的 HBM 读取而非通信 —— 这正是 MTP「一次读权重产出两词」奏效的原因。 💡 Reading: the four decode steps are perfectly regular — 9.8 ms of forward pass, then one MTP verification that emits 2 tokens, amortizing to 11.20 ms/tok. The TP2 All-Reduce costs only 85 µs per step, which says the decode bottleneck is still reading weights out of HBM rather than communication — exactly why MTP's "one weight read, two tokens" pays off.
📹 阿里 6.18 联合技术对齐 ➔ 9.18 工业级落地:三月攻坚脉络与实录资产 Alibaba 6.18 Joint Tech Alignment ➔ 9.18 Production Delivery: 3-Month Trajectory & Recording Assets
实录归档 · 诉求对齐 · 落地验收 Recordings · Alignment · Milestones 🌐 EN / 中文

2026年6月18日,Google Core ML 与阿里联合召开了三场关键技术专题攻坚会(由 Terance Sum 发起,Felix Li 等核心技术专家主讲), 针对 Qwen 3.5 397B-A17B 落地 Google Cloud TPU v7x(Ironwood)展开了涵盖「单层算子优化与 MoE 调度」、「P/D 分离传输架构」、「KV Cache 池化与长上下文显存治理」的三维深度对齐。 历经整整 3 个月的联合工程攻关(2026-06-18 ➔ 2026-09-18),最初梳理的原型缺陷、理论假设与客户核心卡点,已全量转化为确定性、工业级的量化交付战果: On June 18, 2026, Google Core ML and Alibaba convened three deep-dive technical sessions (organized by Terance Sum, led by Felix Li and core engineers), establishing a 3D joint technical alignment for deploying Qwen 3.5 397B-A17B onto Google Cloud TPU v7x (Ironwood) across "Single-Layer Operator & MoE Scheduling", "P/D Disaggregation Transport Architecture", and "KV Cache Pooling & Long-Context Memory Containment". Through 3 months of intensive co-engineering (2026-06-18 ➔ 2026-09-18), the initial architectural bottlenecks, compiler bugs, and customer pain points have been fully resolved into validated production deliverables:

🔬 专题一:Qwen 3.5 397B 算子优化与并行架构深度剖析 Deep Dive: Qwen 3.5 397B Operator Optimization & SPMD 06/18 11:00 HKT
主讲/参会:Terance Sum, Felix Li, Gang Ji, Sixiang Si, Weiren Yu, Yixin Shi, Alex Zhang, Chris Yang, Eason Gao, Jason Wang, Leo Zhao, Chris Chan, Lichun Li, Yuyan Peng Speakers/Attendees: Terance Sum, Felix Li, Gang Ji, Sixiang Si, Weiren Yu, Yixin Shi, Alex Zhang, Chris Yang, Eason Gao, Jason Wang, Leo Zhao, Chris Chan, Lichun Li, Yuyan Peng
⚠️ 6.18 原始诉求与核心痛点6.18 Initial Demands & Bottlenecks:
• 512 细粒度专家路由调度碎片化,All-to-All 回传后存在多次中间显存拷贝与 CPU 调度空洞;
• 密集 Kernel 启动带来高昂 HBM 往返开销;XLA 意外将 FP 权重解包为 INT8 导致 Vector Spill 寄存器溢出与访存气泡。
• 512-expert routing fragmentation; redundant intermediate HBM staging copies & CPU gaps after All-to-All; XLA compiler unpacking FP weights into INT8 before loading, causing vector spills.
✅ 9.18 闭环成果与量化交付9.18 Resolution & Measured Deliverable:
• 研发并合并 handsomehow/owner_output 原地汇聚分支,消除中间拷贝气泡;Decode 吞吐实测达标 5,650 tok/s。 • Merged handsomehow/owner_output direct gather branch; decode throughput verified at 5,650 tok/s (target met).
🌐 专题二:P/D 分离架构细节、跨机传输与最佳实践 Deep Dive: P/D Disaggregation Details & Inter-Slice DCN 06/18 14:00 HKT
主讲/参会:Terance Sum, Felix Li, Chris Yang, Eason Gao, Felix Zhao, Leo Zhao, Sixiang Si, Jason Wang, Alex Zhang, Lichun Li, Chris Chan, Weiren Yu, Gang Ji, Yuyan Peng Speakers/Attendees: Terance Sum, Felix Li, Chris Yang, Eason Gao, Felix Zhao, Leo Zhao, Sixiang Si, Jason Wang, Alex Zhang, Lichun Li, Chris Chan, Weiren Yu, Gang Ji, Yuyan Peng
⚠️ 6.18 原始诉求与核心痛点6.18 Initial Demands & Bottlenecks:
• 单实例内 Prefill 计算与 Decode 互相干扰;
• Prefill(PCP 拓扑)传至 Decode(TP/DP 拓扑)时动态 Reshard 产生海量 CPU/DMA 内存转置,DCN 网卡带宽无法打满;Host 内存上传瓶颈严重。
• Monolithic interference between prefill and decode; severe dynamic resharding transposes when streaming KV from PCP to TP/DP, throttling DCN wire throughput; Host memory upload limits.
✅ 9.18 闭环成果与量化交付9.18 Resolution & Measured Deliverable:
• 固化 1P1D(16芯片) 标准拓扑;研发 SEQ_ON_LANE 零拷贝连续布局,DCN 单网卡峰值突破 >12.5 GB/s(达 25 GB/s 线速 50%)。 • Standardized 1P1D (16-chip) topology; deployed SEQ_ON_LANE zero-copy layout; DCN peak >12.5 GB/s (50% of 25 GB/s line rate).
🧮 专题三:KV Cache 显存预算、池化与长上下文卸载 Deep Dive: KV Cache Budget, Pooling & Offloading 06/18 15:42 HKT
主讲/参会:Terance Sum, Felix Li, Chris Yang, Yixin Shi, Felix Zhao, Leo Zhao, Sixiang Si, Jason Wang, Alex Zhang, Lichun Li, Eason Gao, Chris Chan, Weiren Yu, Gang Ji, Yuyan Peng Speakers/Attendees: Terance Sum, Felix Li, Chris Yang, Yixin Shi, Felix Zhao, Leo Zhao, Sixiang Si, Jason Wang, Alex Zhang, Lichun Li, Eason Gao, Chris Chan, Weiren Yu, Gang Ji, Yuyan Peng
⚠️ 6.18 原始诉求与核心痛点6.18 Initial Demands & Bottlenecks:
• 128K 超长上下文造成显存墙,HBM 无法支撑生产并发;Trunk 分片与 Roofline 边界不明确;
• 整机 DRAM RSS 飙升至 ~700 GB 无法释放导致 KVCM 无法拉起。
• 128K context hits HBM compute/capacity wall; trunk/slot boundaries unclear; whole-node DRAM RSS ballooned to ~700 GB and never released, preventing KVCM from launching.
✅ 9.18 闭环成果与量化交付9.18 Resolution & Measured Deliverable:
• 落地 PCP=8 Ring Attention(计算通信 100% 重叠),128K TTFT 骤降至 4.62 s(加速 6.9×);jemalloc 替换彻底根除 700GB 内存泄漏。 • Implemented PCP=8 Ring Attention (100% overlap, 0 bubble); 128K TTFT dropped to 4.62 s (6.9× speedup); jemalloc swap-in cured 700GB leak.
📈 阶段验收:[External][Alibaba][TPU] 推理优化周会 Milestone Acceptance: Alibaba TPU Weekly Sync 09/04 09:57 CST
参会专家:Qi Zhou, Amy Lin, Jun Wan, Wenzhe Zhou, Xiaohang Yu (HandsomeHow), Felix Li, Chi Shuen Lee, Jimmy Tsai, Eason Gao, Gang Ji, etc. Attendees: Qi Zhou, Amy Lin, Jun Wan, Wenzhe Zhou, Xiaohang Yu (HandsomeHow), Felix Li, Chi Shuen Lee, Jimmy Tsai, Eason Gao, Gang Ji, etc.
📋 9.04 联合验收评审议题9.04 Joint Acceptance Review:
• 审查全局 Registry 性能压测达标报告;
• 对齐 PCP 性能与 TP2DP4 解码器高并发配置,验证流量削峰限流与多轮对话系统稳定性。
• Global registry benchmark report review; alignment on PCP performance & TP2DP4 decoder configs; verified throttling and multi-turn conversational stability under peak load.
✅ 验收结论Verdict:
• 核心指标全线收敛达标,压测结果达到 NVIDIA GB200 基准 90% 吞吐上线要求,转入生产级金标验证。 • Core metrics fully converged; achieved >90% of NVIDIA GB200 baseline throughput; approved for production golden-standard qualification.
📊 6.18 阿里技术诉求 vs. 9.18 工业级落地进展 全景对照表 Alibaba 6.18 Tech Demands vs. 9.18 Production Deliverables Comparison
攻坚技术维度Dimension 6.18 联合对齐原始诉求与核心痛点6.18 Initial Demands & Bottlenecks 9.18 闭环落地工程方案9.18 Resolution & Implementation 实测交付收益与量化指标Measured Deliverable / Metric 关联核心资产Associated Asset
128K 长上下文计算墙 (TTFT) 128K Context Wall (TTFT) DP8 架构下处理 128K 序列面临严重 HBM 访存墙,单 Prefill 过程拖死整个集群,TTFT 耗时高达 32.0 秒,严重超标。 DP8 hits severe HBM bandwidth wall on 128K sequence; prefill throttles cluster, causing unacceptable TTFT of 32.0 s. Prefill 节点采用 PCP=8 + EP=8,把 128K 序列切成 8 个 16K 分片,利用 TPU 片上大容量 SRAM 绕开 HBM,并以 Ring Attention 实现通信与计算 100% 重叠(Bubble = 0 µs)。 Prefill runs PCP=8 + EP=8, chunking 128K sequence into eight 16K shards in SRAM, with Ring Attention completely overlapping compute and ICI comms (0 µs bubble). 128K TTFT 4.62 s
(相比 DP8 提速 6.9×)
128K TTFT 4.62 s
(6.9× speedup vs DP8)
专题三录像 🔗Topic 3 Video 🔗
MoE 512 细粒度专家路由 MoE 512-Expert Routing 512 个细粒度专家在 All-to-All 路由回传后,通用框架存在多次中间显存拷贝与 CPU 调度空洞,吞吐长期徘徊在 5,500 tok/s 以下。 512 fine-grained experts had multiple redundant intermediate HBM copies and CPU scheduling gaps after All-to-All, capping throughput below 5,500 tok/s. 合并 handsomehow/owner_output 特化直聚分支,让负责计算专家的 Owner Core 在本地原地完成输出汇聚(direct gathering),彻底消除冗余内存中转气泡。 Merged specialized handsomehow/owner_output branch; owner core computes output gathering locally in-place, eliminating intermediate copy bubbles. Decode 5,650 tok/s
(由 5,500 稳步提升达标)
Decode 5,650 tok/s
(Target achieved)
专题一录像 🔗Topic 1 Video 🔗
P/D 跨拓扑 DCN 传输 Inter-Slice DCN Transport Prefill 切片(PCP 拓扑)传输至 Decode 切片(TP·DP 拓扑)时,动态 resharding 触发海量 CPU/DMA 内存转置,DCN 单网卡吞吐严重受挫。 Moving KV cache from prefill (PCP) to decode (TP/DP) triggered heavy dynamic resharding transposes, severely choking cross-slice DCN throughput. 设计并部署 SEQ_ON_LANE 零拷贝内存布局,将 KV Heads 维度前置,使跨机 DMA 引擎能够以单一连续内存流直接推入 DCN 网卡。 Deployed SEQ_ON_LANE zero-copy layout, hoisting KV heads axis so cross-host DMA engine pushes contiguous streams directly into DCN NIC. DCN 峰值 >12.5 GB/s
(达 25 GB/s 线速 50%)
DCN Peak >12.5 GB/s
(50% of wire speed)
专题二录像 🔗Topic 2 Video 🔗
Host 内存泄漏治理 Host DRAM Leak Fix 早期压测中 8-device 整机 DRAM RSS 飙升至 ~700 GB 且长期不释放,导致后续 KV 管理服务(KVCM)无法拉起,系统濒临 OOM 崩溃。 Whole-node DRAM RSS climbed to ~700 GB across 8 devices and never freed, preventing KVCM from initializing and triggering system-level OOMs. ① 缩容 premapped VMA 配额(原 16 GiB);② 全面引入 jemalloc 替换 glibc 内存分配器;③ 完善 prefix-aware 动态加载逻辑。 (1) Shrunk premapped VMA quota (was 16 GiB); (2) Replaced glibc allocator with jemalloc; (3) Added prefix-aware dynamic memory loading. DRAM 稳定受控释放
(多轮长对话冒烟全通)
DRAM Stable & Released
(Multi-turn tests pass)
专题三纪要 🔗Topic 3 Notes 🔗
部署拓扑与并发容量 Topology & Concurrency 原先缺乏高性价比物理拓扑标准,社区方案甚至提出动用 64 芯片实例,成本极高且资源利用率严重不均衡。 Lack of standardized production topology; theoretical proposals assumed 64-chip instances, which were economically infeasible and unbalanced. 固化 1P1D(共 16 芯片) 部署形态:Prefill 采用 PCP8+EP8(8 芯片)→ Decode 采用 TP2·DP4·EP8(8 芯片),TP2 将 Dense 显存减半支撑 Batch=64 满并发。 Established 1P1D (16 chips total) topology: Prefill runs PCP8+EP8 (8 chips) → Decode runs TP2·DP4·EP8 (8 chips); TP=2 halves dense weights to allow full Batch=64 concurrency. 16 芯片对齐 1P1D
(TP2 All-Reduce 仅 85µs)
16-Chip 1P1D Pair
(TP2 All-Reduce 85 µs)
9.04 周会纪要 🔗9.04 Sync Notes 🔗
🔥 Qwen 3.5 专项工程优化体系复盘与量化战果 Qwen 3.5 engineering optimization campaign — retrospective and quantified results

Qwen 3.5 397B 在 Google Cloud TPU v7 上落地过程中,针对超长上下文算力墙、MoE 输出搬运气泡、跨机 Reshard 内存转置损耗、DRAM 内存泄漏等工业级痛点,实施了多项系统级与 Kernel 级攻坚,全面达成并超越压测指标: Bringing Qwen 3.5 397B up on Google Cloud TPU v7 ran into four production-grade walls — the long-context compute wall, MoE output-movement bubbles, cross-host reshard transpose overhead, and a DRAM leak. Each was attacked at the system or kernel level, and every stress-test target was met or beaten:

⚠ 标定基线⚠ Calibration baseline
本页所有实测值来自 bodaborg-tpu7x-nap (cloud-tpu-shared-capacity) · 2×4 拓扑 上的 8 芯片 prefill 节点 / 60 层混合架构(45 GDN + 15 GQA)压测。 这就是本文档的主口径 —— 上线部署形态为 Prefill 节点 PCP=8 + EP=8(8 芯片)→ Decode 节点 TP=2 × DP=4 + EP=8(8 芯片),一对共 16 芯片,不存在 64 芯片实例。 Decode 侧 5,650 tok/s 有文档背书(TP2DP4 + owner_output 分支,由 5,500 提升而来);11.20 ms/tok 则与实测 TPOT 25 ~ 28 ms 冲突,引用前请核对。 Every measured figure on this page comes from a stress run on bodaborg-tpu7x-nap (cloud-tpu-shared-capacity) · 2×4 topology using an 8-chip prefill node on the 60-layer hybrid build (45 GDN + 15 GQA). That is the document's main configuration: the deployed form is a prefill node with PCP=8 + EP=8 (8 chips) feeding a decode node with TP=2 × DP=4 + EP=8 (8 chips) — 16 chips per pair, with no 64-chip instance. Decode's 5,650 tok/s is corroborated (TP2DP4 on the owner_output branch, up from 5,500); 11.20 ms/tok, however, conflicts with the measured TPOT of 25–28 ms — verify before quoting.
🏗️
1P1D 部署拓扑deployment topology (PCP8 ➔ TP2·DP4·EP8)
打破 128K 算力墙与 Decode 显存瓶颈 Breaks the 128K compute wall and the decode memory bottleneck
架构层Architecture
• Prefill 采用 PCP8 + EP8:把 128K 序列拆成 8 个 16K 分片,用 TPU 片上大容量 SRAM 绕开 HBM 访存墙,再靠 Ring Attention 把计算与通信 100% 重叠。
• Decode 采用 TP2 · DP4 · EP8 复合切分:相比 DP8,TP=2 让 Dense 权重显存减半,腾出空间支撑 Batch=64 满并发;片内高速总线使 TP2 All-Reduce 只需 ~85 µs。
• Prefill runs PCP8 + EP8: the 128K sequence is cut into eight 16K shards so the TPU's large on-chip SRAM sidesteps the HBM bandwidth wall, and ring attention then overlaps compute with communication completely.
• Decode runs a composite TP2 · DP4 · EP8 split: against a plain DP8, TP=2 halves the dense-weight footprint, freeing room for a full batch of 64; the on-package bus keeps the TP2 All-Reduce at roughly 85 µs.
实测工程战果Measured result
128K TTFT 4.62 s(DP8 32 s → PCP8,6.9×,同 8 芯片) 128K TTFT 4.62 s (DP8 32 s → PCP8, 6.9×, same 8 chips)
⚡
MoE owner_output 特化直聚分支direct-gather branch
handsomehow/owner_output
Kernel 级Kernel
• 痛点:512 个细粒度专家在 All-to-All 路由回传后,通用框架里存在多次中间显存拷贝与 CPU 调度空洞。
• 方案:特化分支让计算该专家的 Owner Core 在本地原地完成输出汇聚(direct gathering),消除冗余的内存中转气泡。
• Problem: after the All-to-All returns results from 512 fine-grained experts, the generic framework path performs several intermediate HBM copies and leaves CPU scheduling gaps.
• Fix: a specialized branch has the owner core that computed each expert do the output gather in place, locally — removing the redundant staging copies and the bubble they create.
实测工程战果Measured result
Decode 吞吐 5,500 ➔ 5,650 tok/s(达标) Decode throughput 5,500 ➔ 5,650 tok/s (target met)
🚀
SEQ_ON_LANE 零拷贝内存布局zero-copy layout
重排 KV Cache 布局,赋能 DCN 直传 Relays out the KV cache so DCN can stream it directly
数据链路Data path
• 痛点:从 Prefill 切片(PCP 拓扑)传到 Decode 切片(TP·DP 拓扑)时,动态 resharding 触发大量 CPU/DMA 内存转置,严重拖慢网络吞吐。
• 方案:SEQ_ON_LANE 布局把 KV Heads 维度前置,让跨机 DMA 引擎能以连续内存流直接推入 DCN 网卡。
• Problem: moving KV from the prefill slice (PCP topology) to the decode slice (TP·DP topology) triggered dynamic resharding, and the resulting CPU/DMA transposes throttled network throughput badly.
• Fix: the SEQ_ON_LANE layout hoists the KV-head axis to the front, so the cross-host DMA engine can push one contiguous stream straight into the DCN NIC.
实测工程战果Measured result
DCN 单网卡峰值 >12.5 GB/s(线速 25 GB/s 的 50%) Single-NIC DCN peak >12.5 GB/s (50% of the 25 GB/s line rate)
🛡️
DRAM RSS 泄漏治理与系统加固DRAM RSS leak containment
jemalloc 替换 + premapped VMA 调优 jemalloc swap-in + premapped VMA tuning
稳定性Stability
• 痛点:压测初期 8-device 整机 DRAM RSS 高达 ~700 GB 且长期不释放,导致后续 KVCM 拉不起来。
• 方案:① 针对性缩容 premapped VMA 配额(原 16 GiB);② 用 jemalloc 替换 glibc 分配器;③ 完善 prefix-aware 动态加载;④ 清理编译副本残留 Python 引用并修正连接器 leases=1 DRAM 预估偏差。
• Problem: early in the stress campaign, whole-node DRAM RSS across 8 devices climbed to about 700 GB and never came back down, which left KVCM unable to start afterwards.
• Fix: (1) shrink the premapped VMA quota (was 16 GiB) where it actually mattered; (2) replace the glibc allocator with jemalloc; (3) round out prefix-aware dynamic loading; (4) release unintended Python references to compiled artifacts and fix connector leases=1 DRAM estimation.
实测工程战果Measured result
整机 DRAM 稳定释放;超大 MoE 单机容器内存由 440 GiB 压降至 230 GiB DRAM releases predictably; single-host container footprint cut from 440 GiB to 230 GiB
⚖️ 异构芯片工业级对决:Google TPU v7x vs. NVIDIA GB200 (Qwen 实测全景对标) Hardware Benchmark Showdown: Google TPU v7x vs. NVIDIA GB200 on Qwen Serving
🌐 EN / 中文

依据 2026-09 ➔ 2026-10 联合性能优化实测记录(含 4-Host 跨机压测与 9 月下旬生产级冲刺), 系统以 NVIDIA GB200 (NVLink 全互联拓扑) 作为生产级黄金基准(Golden Baseline)。 双方联合设定的生产上线目标为 达到 GB200 的 90% 吞吐。以下为端到端实测达成情况与深层微架构对账: Based on 2026-09 ➔ 2026-10 performance benchmarks (including 4-Host multi-node runs and the late-September production sprint), the system benchmarks against NVIDIA GB200 (NVLink-connected topology) as the industrial Golden Baseline. The production deployment target is set at 90% of GB200 throughput. Below is the verified scorecard and architectural breakdown:

Prefill 吞吐对标 (1P 副本 · 4 芯片 / 8 Device / 1-Host) Prefill Throughput (1P Replica · 4 Chips / 8 Devices / 1-Host)
58,000 tok/s
78.4% 达成率
• GB200 黄金基准: 74,000 tok/sGB200 Baseline: 74,000 tok/s
• 上线目标 (90%): 66,600 tok/s (74k × 0.9)Target (90%): 66,600 tok/s
• TPU 实测: 58,000 tok/s (PCP8 · MoE 两段异步重叠)TPU Actual: 58,000 tok/s (PCP8 · 2-stage MoE overlap)
Decode 吞吐对标 (1D 副本 · 4 芯片 / 8 Device / 1-Host) Decode Throughput (1D Replica · 4 Chips / 8 Devices / 1-Host)
5,650 tok/s
85.6% 达成率
• GB200 黄金基准: 6,600 tok/s (单卡 54 tok/s)GB200 Baseline: 6,600 tok/s (54 tok/s/GPU)
• 9 月中旬基线: 5,650 tok/s (TP2·DP4·EP8 + owner_output)Mid-Sep Baseline: 5,650 tok/s (TP2·DP4·EP8 + owner_output)
• 09-24 算子优化后: SSM=FP32 6,500–6,600 / SSM=BF16 7,600 tok/s(对应 GPU 7,200 / 7,700,Device 数口径待对齐)09-24 Post-Tuning: SSM=FP32 6,500–6,600 / SSM=BF16 7,600 tok/s (vs. GPU 7,200 / 7,700; device count pending parity check)
128K 超长单请求首字延迟 (TTFT) 128K Single-Stream Long Context TTFT
4.62 ~ 4.70 s
~108% 反超 GPU
• GB200 测得: ~5.0 s (长文二次方复杂度访存受限)GB200: ~5.0 s (HBM bandwidth bound in long context)
• TPU DP 基线: 32.0 s (无序列切片,单核算力过载)TPU DP: 32.0 s (Overloaded without sequence slicing)
• TPU PCP8 拓扑: 4.62 s (ICI 环网与算力 100% 隐藏重叠)TPU PCP8: 4.62 s (ICI ring perfectly overlaps compute)
P/D 越界双网卡吞吐 (Dual-NIC 400G) Disagg Dual-NIC Throughput (400G Fabric)
367.0 Gbps
91.8% 线速极限
• 400G 理论线速: 400.0 Gbps (2×200G 双网卡聚合)400G Wire Ceiling: 400.0 Gbps (Dual 200G NICs)
• 单网卡饱和极限: 192.7 Gbps (排队瓶颈,TTFT 劣化)Single-NIC Ceiling: 192.7 Gbps (Queueing bottlenecks)
• Raiden NUMA 实测: 367.0 Gbps (1.91× 提速,零 UPI 争抢)Raiden Actual: 367.0 Gbps (1.91× speedup, zero UPI drag)
核心维度Dimension Google TPU v7 (Ironwood)Google TPU v7 (Ironwood) NVIDIA GB200 (Blackwell NVL72)NVIDIA GB200 (Blackwell NVL72)
计算核心、精度与 FP4 / W4A8 真相Compute Cores & Precision / W4A8 256×256 脉动阵列 (MXU) · 官方白盒参数与 W4A8 瞬时反量化
• 硬件真相(OpenXLA 官方白盒实锤):根据 OpenXLA 2026 最新开源基准(xla/benchmarks/core/platform_info.py),TPU v7 (Ironwood) 运行在 P-State 3(默认 1.9 GHz,可动态突发至 P7 2.2 GHz)。其硬件阵列原生仅支持 INT8 / FP8 / BF16(发射周期 8 cycles,延迟 204~211 cycles),完全无原生 FP4 浮点乘加硬件(原生 FP4 延迟至 TPU v8i 引入)。
• 阿里软件工程突破 (W4A8 & Subchannel 量化):阿里团队通过 NVFP4 + W4A8(PR #912)将静态权重以 4-bit(INT4/FP4)脱水存储,在片上向量单元(VPU)以 Subchannel 微缩放粒度就地瞬时反量化(De-quantize)为 8-bit 送入 256×256 脉动阵列,以微小计算代价换取 HBM 访存带宽直接翻倍!
256×256 Systolic Array (MXU) · OpenXLA Ground Truth & W4A8 Dequantization
• Silicon Reality (OpenXLA Benchmarks): Per OpenXLA 2026 specs (xla/benchmarks/core/platform_info.py), TPU v7 (Ironwood) runs at P-State 3 (default 1.9 GHz, burstable to P7 2.2 GHz). It natively supports INT8/FP8/BF16 (cadence 8 cycles, latency 204~211 cycles) and completely lacks native FP4 MAC silicon (native FP4 is deferred to TPU v8i).
• Alibaba Software Breakthrough (W4A8 & Subchannel Quant): The NVFP4 + W4A8 pipeline (PR #912) compresses weights to 4-bit in HBM with subchannel micro-scaling, de-quantizing on-chip before feeding the 8-bit MXU, doubling effective memory bandwidth.
第 5 代原生 NVFP4 / FP8 Tensor Cores
• 具备硬件原生 4-bit 浮点微缩放乘加器(Micro-scaling FP4),在密集 Prefill GEMM 上理论 FLOPs 占优;但原生 FP4 存在一定数值溢出与精度损失风险,需严苛评估 MMLU。
5th-Gen Native NVFP4 / FP8 Tensor Cores
• Features true silicon-native micro-scaling FP4 MAC units, giving theoretical FLOPs advantage in dense prefill; however, native FP4 requires rigorous eval to prevent accuracy drop.
芯片封装、物理主机 vs. Device 对账 & 拓扑Packaging: Host vs. Chip vs. Device & Fabric 物理事实:单 Host 4 芯片(8 Core)· 8 芯片切片为双机(2 Hosts)
• 微架构与 Die-to-Die:TPU v7 (Ironwood) 单颗物理 Chip 封装 2 个独立计算裸晶(Die),在驱动层暴露为 2 个逻辑 Device(Core),两 Die 间由片上高速超宽 Die-to-Die 互联直连。
• 单机与双机拓扑基准: - 物理单机(1 Host,tpu7x-standard-4t):仅含 4 颗物理芯片(8 个 Core),拓扑为 3D Torus 2×2×1。其 758 GiB 可用 HBM 只能作为单一角色的最小物理部署单元(独占跑 PCP8 Prefill 或独占跑 TP2·DP4 Decode),物理上无法在一台单机内混跑 P/D 双实例;
- 双机切片(2 Hosts,8 颗物理芯片 / 16 Core):物理拓扑为 2×2×2 Torus,在 XLA 与通信算法上展平为 2×4 逻辑网格,用于构建 8-hop 的单向哈密顿大环。
• EP=8 真实映射口径:单机部署时按 8 个 Core 切分(每 Core 64 专家);双机部署时按 8 个 Chip 插槽分片。
Hardware Ground Truth: 1 Host = 4 Chips (8 Cores) · 8-Chip Slice = 2 Hosts
• Microarchitecture & Die-to-Die: Each physical TPU v7 chip integrates 2 distinct compute dies exposed as 2 logical devices (cores), linked by an ultra-fast on-package Die-to-Die bus.
• Single-Host vs. Dual-Host Topologies: - Single Physical Host (tpu7x-standard-4t): Contains exactly 4 physical chips (8 cores) in a 2×2×1 3D Torus. Its 758 GiB HBM capacity can only act as an isolated worker for either PCP8 Prefill OR TP2·DP4 Decode; it physically cannot host a mixed P/D serving cluster on a single machine;
- Dual-Host Slice (2 Hosts, 8 Chips / 16 Cores): Forms a physical 2×2×2 Torus, logically flattened into a 2×4 mesh in XLA to build an 8-hop Hamiltonian ring for Ring Attention.
• EP=8 Alignment: Sharded across 8 cores in single-host mode (64 experts/core); sharded across 8 chip sockets in dual-host mode.
单裸晶/双芯 NVLink 全互联 (1.8 TB/s 双向)
• 单卡即单个独立 CUDA 设备;在 NVL72 机柜内 72 卡任意点对点低延迟 All-to-All;但跨机柜扩展需跨 IB/RoCE 网络,延迟陡增。
Monolithic / Dual-Die NVLink Mesh (1.8 TB/s bi-directional)
• 1 GPU maps directly to 1 CUDA device; 72 GPUs fully connected via NVLink 5 copper backplane; cross-rack scaling relies on higher-latency InfiniBand/RoCE.
MoE 专家调度与分级缓冲MoE Dispatch & Hierarchical Buffering 4× 独立硬件协处理器 (SparseCore v3) + 片上 VMEM 双缓冲
• 每颗物理 Chip 搭载 4 个第 3 代 SparseCore,硬件级卸载 Top-K 寻址与 Gather-Reduce-v3,主 MXU 核心 0 调度气泡。
• 双缓冲 (Double Buffering) 物理机制:阿里攻坚的 Fused MoE v2 双 buffer (W1×2 + W2×1),底层正是利用超高速 Die-to-Die 链路在片内预先聚合,并在超导 VMEM 内部建立双缓冲蓄水池(Buffer A 算当前专家 / Buffer B 异步预取解压下个专家),完美隐藏跨 Chip 较慢的 ICI 环网流量与 692ns HBM 搬运延迟!
4× Dedicated SparseCore v3 + On-Chip VMEM Double Buffering
• 4 third-gen SparseCores per chip offload Top-K routing and Gather-Reduce-v3, leaving MXU completely bubble-free.
• Double Buffering Silicon Rationale: Alibaba's Fused MoE v2 dual-buffer (W1×2 + W2×1) leverages fast Die-to-Die links and establishes double buffering inside high-speed VMEM (Buffer A computes / Buffer B prefetches & dequantizes), hiding inter-chip ICI latency and the 692ns HBM penalty.
SM 核心通用线程调度 Gather/Scatter
在高并发长文本下容易出现非连续访存(Strided Memory Access)导致的 HBM 带宽打折,且 SM 兼职处理路由调度会导致计算核心气泡。
GPU SM Kernel Gathering/Scattering
Prone to HBM bandwidth degradation under heavy concurrent token routing due to strided memory access; SM thread-divergence induces pipeline bubbles.
片上超导缓存 (VMEM) 与 HBM 访存墙On-Chip VMEM vs. HBM Wall 32 MiB Scoped VMEM · 3,207 GB/s 带宽 · 软件可控超导水槽
• 官方物理常数(OpenXLA PlatformInfo):单片 HBM $\leftrightarrow$ VMEM 实测带宽打满 3,207 GB/s,读取延迟 692 ns,写回延迟 594 ns。
• Pallas Tiling 优化:显式开放 --xla_tpu_scoped_vmem_limit_kib=32768,使 Pallas 内核将 128K Attention 与 MoE 的 Window Tile 切割开满,运算全程在 VMEM 内部超导回旋,将昂贵的片外 HBM 穿透率降低 30% 以上!
32 MiB Scoped VMEM · 3,207 GB/s Bandwidth · Software-Controlled Scratchpad
• Silicon Ground Truth (OpenXLA PlatformInfo): Delivers 3,207 GB/s HBM-to-VMEM bandwidth with 692 ns read and 594 ns write latency.
• Pallas Tiling Gains: Enabling --xla_tpu_scoped_vmem_limit_kib=32768 expands Pallas window tiles, keeping 128K attention and MoE working sets circulating within VMEM, slashing off-chip HBM roundtrips by >30%.
硬件自动管理 L2 Cache / Shared Memory (SRAM)
• GB200 单卡配备超大 L2 Cache,但由硬件缓存控制器自动置换(LRU);多线程并发冲突时易发生 Bank Conflict 与缓存驱逐(Cache Thrashing),需极复杂的 TMA 异步拷贝指令微调。
Hardware-Managed L2 Cache / Shared Memory (SRAM)
• Large L2 cache on B200 is managed implicitly via hardware LRU; heavy concurrent access risks bank conflicts and cache thrashing, requiring intricate TMA async copy tuning.
显存规格与长文安全线Memory & Context Ceiling 192 GiB HBM3e / 物理 Chip(96 GiB / Core)
• 单机 4 芯片总计 768 GiB 高速 HBM3e(2 NUMA 节点池化);
• 长文解耦:通过 PCP8 环形切片将 128K KV Cache(15 GiB 全量 FA)均匀切分为 1.87 GiB / 卡,结合 Host DRAM 动态流转彻底杜绝 OOM。
192 GiB HBM3e / Physical Chip (96 GiB / Core)
• 4-chip single node totals 768 GiB HBM3e across 2 NUMA nodes;
• Long Context Scaling: PCP8 slices 128K FA KV Cache (15 GiB) down to 1.87 GiB/chip; safe DRAM offloading completely eliminates OOM.
192 GB HBM3e / GPU
• 单卡容量物理相当,但在纯 DP 模式下高并发长文本仍会迅速触碰显存红线,需要大规整 PagedAttention 分块调度;单卡显存碎片率较高。
192 GB HBM3e / GPU
• Physical capacity is identical, but pure DP serving under heavy concurrent long context still exhausts HBM; higher memory fragmentation risk.
P/D 越界投递与网络亲和性Disagg KV Handoff & NIC Raiden NUMA-Aware 双网卡 (2×200G 聚合 400G) · 100% 本地 DMA
• 硬件亲和绑定:Rank 0..3 绑 NUMA 0 (ens6),Rank 4..7 绑 NUMA 1 (enp192s4),接收端达成 100% NUMA-local 投递,彻底根除跨 CPU Socket UPI 内存总线搬迁;
• 实测性能:ProcFS 实测打满 367.0 Gbps(91.8% 线速),长文 64K Reshard 耗时压减至 336.6 ms(提速 2.10×)。
Raiden NUMA-Aware Dual-NIC (2×200G Aggregate 400G) · 100% Local DMA
• Hardware Affinity: Binds Ranks 0..3 to NUMA 0 (ens6) and Ranks 4..7 to NUMA 1 (enp192s4), completely eliminating cross-socket UPI bus traffic;
• Empirical Benchmark: Delivers 367.0 Gbps (91.8% wire speed), slashing 64K reshard latency to 336.6 ms (2.10× faster).
机柜内 NVLink 极强 / 跨机柜依赖 RoCE/IB
• 机柜内(NVL72)72 卡点对点 1.8 TB/s 具备无可比拟优势;但进行跨机柜或跨 Pod 的 P/D 分离投递时,依赖外部以太网/InfiniBand,若缺少 NUMA-网卡-GPU 的细粒度亲和调度,容易引发 host 内存通道与软中断不均。
Intra-Rack NVLink Dominance / Cross-Rack RoCE/IB Bottlenecks
• NVL72 provides unmatched 1.8 TB/s intra-rack peer-to-peer; however, cross-rack/disaggregated serving relies on RoCE/IB fabrics where lack of NUMA-NIC-GPU affinity introduces host memory bottlenecks.
⚡ 4-Host 跨节点通信瓶颈与 PP4·PCP8 混合并行:理论推算 vs. 真实权重实测校准 ⚡ 4-Host Inter-Node Bottleneck & PP4·PCP8 Hybrid Parallelism: Analytical Projection vs. Real-Weight Calibration

• 跨机通信挑战(09-18 实测):当集群扩展至 4-Host(16 物理芯片 / 32 Device)时,Prefill 吞吐从单机 PCP8 的 84k 跌落至 44k tok/s(预期 60k)。 根因在于节点超过 2-Host 后物理不再是全互联,跨机 Z 轴集合通信带宽不足严重拖累了 Fused MoE Token All-Gather(端到端约 −30%)。
• 架构破局推算 vs. 10 月真实权重实测校准:团队放棄跨机全局 All-Gather 优化,转向 PP4·PCP8(流水线 + 环形混合并行): 09-18 依据单机 PP8(80k tok/s,TTFT ~3s+)与单机 PCP8(84k tok/s)理论推算 4-Host 可达每 Rank ~20k、TTFT ~1.6 s; 10 月中旬在 PP4·PCP8 + MTP 真实权重(P8K·D1)压测中,实测吞吐为 55k–58k tok/s(其中 MTP 带来约 1–2% 可接受损耗),但吞吐存在波动——根因是 Scheduler 按 block_size 将 8K 请求切分为 3K + 5K,仅当单步恰好凑齐均衡组合(如 4×3K + 4×5K)时才能打满,若调出非对称组合(如 5×3K + 3×5K)即产生跨 Stage Padding 空泡(可借鉴上游 Kimi 的对称分桶调度修复)。
• Multi-Host Bottleneck (09-18 Measurement): When scaling to 4-Host (16 physical chips / 32 devices), prefill throughput dropped from 84k (single-host PCP8) to 44k tok/s (target 60k). Root cause: beyond 2 hosts the physical torus is no longer all-to-all connected, and limited Z-axis bandwidth throttles Fused MoE Token All-Gather (~30% E2E penalty).
• Analytical Projection vs. October Real-Weight Calibration: Dropping cross-host All-Gather tuning in favor of PP4·PCP8 (Pipeline + Ring Hybrid): on 09-18, extrapolating from single-host PP8 (80k tok/s, TTFT ~3s+) and single-host PCP8 (84k tok/s) analytically projected ~20k/rank and ~1.6 s TTFT across 4 hosts; in mid-October real-weight benchmarks (PP4·PCP8 + MTP, P8K·D1), measured throughput reached 55k–58k tok/s (with an acceptable 1–2% MTP overhead), exhibiting variance caused by block-size-aligned scheduler chunking: an 8K prompt is split into 3K + 5K, so peak throughput requires balanced micro-batches (e.g., 4×3K + 4×5K), whereas skewed draws (e.g., 5×3K + 3×5K) incur padding bubbles (addressable via upstream Kimi-style symmetric bucket scheduling).

📐 2026 推理数据流动力学与 OpenXLA 白盒实测印证 (SemiAnalysis & OpenXLA Benchmarks)
根据 SemiAnalysis 2026.09 最新架构报告 (Computation and Data Movement for Inference) 与 OpenXLA 官方基准 (xla/benchmarks),现代大规模 MoE 推理的本质是一场严密的“高压张量流体力学”协同:
• 四阶段运作范式与 Midfill 破局:行业正从单次对话转向智能体(Agent)长程交互,算力消耗 70% 转向 Midfill(在超大 Cached Prefix 上追加小量新 Token)。KV Cache 必须被视为不可变数据块(Immutable Blobs)实现流水线并发分带流转(Pipelined Torrenting),单机混跑会引发严重的算力/带宽闲置(Stranded Capacity);
• 拓扑切分最高军规:“Put throughput boundaries where the flow is smallest”。层间流水线(PP)仅传输单 Token 隐层向量(几 KB),属于数据流最小的脆弱裂隙;而 Attention 与 MoE Router-to-Expert 具有突发高频屏障(Barrier),必须死守在 Scale-up 域(单机/单机柜内)。这完美论证了阿里 4-Host 架构放弃跨机 EP 强转为 `PCP8 PP4` 的必然性;
• OpenXLA 官方白盒物理印证: 1. 时钟与算力:TPU v7 默认 P-State 3 为 1.9 GHz(最高可达 2.2 GHz),硬件阵列仅原生支持 INT8/FP8/BF16,验证了阿里通过 NVFP4 + W4A8 Subchannel 瞬时反量化破除物理限制的创举;
2. 超导 VMEM 访存墙突破:HBM 到 VMEM 具备 3,207 GB/s 极速带宽,但存在 692 ns 读取 / 594 ns 写回固定延迟。通过显式开启 --xla_tpu_scoped_vmem_limit_kib=32768,结合 VMEM 双缓冲(Double Buffering),将 128K 矩阵与 MoE 专家权重死锁在片上微型水槽内,彻底隐藏片外 HBM 延迟!
Per SemiAnalysis 2026.09 Architecture Report (Computation and Data Movement for Inference) and OpenXLA Benchmarks (xla/benchmarks), frontier MoE inference is fundamentally an exact science of tensor fluid dynamics:
• Four Operating Regimes & Midfill Paradigm: Serving is transitioning to long-running agents where >70% compute shifts to Midfill (appending small increments to massive cached prefixes). KV Cache must be handled as Immutable Blobs via pipelined streaming/torrenting; monolithic aggregated serving creates severe stranded capacity;
• First Law of Topological Partitioning: “Put throughput boundaries where the flow is smallest”. Inter-layer pipeline (PP) activations are tiny (a few KB) representing the narrowest flow boundary, whereas Attention and MoE router all-to-all involve bursty synchronization that must remain strictly inside the scale-up domain. This theoretically validates Alibaba's migration from cross-host EP to `PCP8 PP4` on 4-Host clusters;
• OpenXLA Whitebox Silicon Proof: 1. Clock & Precision: TPU v7 defaults to P-State 3 (1.9 GHz, burstable to 2.2 GHz), natively supporting INT8/FP8/BF16, confirming Alibaba's NVFP4 + W4A8 subchannel dequantization strategy;
2. VMEM Memory Wall Solution: HBM-to-VMEM achieves 3,207 GB/s with fixed 692 ns read / 594 ns write penalties. Expanding --xla_tpu_scoped_vmem_limit_kib=32768 paired with VMEM double buffering locks 128K tiles and MoE working sets entirely on-chip, completely masking off-chip latency.
🤝 软硬件协同工程分工与优化全景对账(Google TPU 官方 vs. 阿里 Qwen 团队) Full-Stack Co-Design & Optimization Taxonomy: Google TPU vs. Alibaba Qwen Team SemiAnalysis 2026 InferenceX / PR Ledger

这场具有里程碑意义的工业级攻坚,本质是一场“Google 死磕单芯片与指令级微观公差,阿里主导全局集群网络与长序列宏观拓扑”的四手联弹协同: This milestone industrial deployment is a textbook full-stack co-design: Google engineered microarchitectural precision and Pallas kernels, while Alibaba pioneered multi-host topologies, long-context parallelisms, and cluster networking.

攻坚维度Domain Google TPU 官方团队 (微观算子与编译器底座)Google TPU Team (Kernel & Silicon Stack) 阿里 Qwen 团队 (宏观拓扑、长序列与量化落地)Alibaba Qwen Team (Architecture & Cluster Serving)
注意力与长序列核心Attention & Long-Context GDN 线性注意力代数重排与算子级融合
• [PR #2498] 代数结合律解耦:将 GDN 输出展开为 $O=(\text{decayed } S)q + \Delta(k^\top q)$,让 MXU 提前乘旧状态,与 VPU 修正更新 100% 并发重叠,彻底消除 MXU 挂起气泡(吞吐提升 +4.48%);
• [PR #3016] GDN v3 终极融合:将 Conv1D 与 Gated DeltaNet 融合成单 Kernel,减少 HBM 换入换出,算子提速 1.41× ~ 2.14×;
• [PR #2741 / PR #2650]:切片 Q/K 杜绝向量寄存器溢出,状态 DMA 双缓冲使吞吐提升 11.3%。
GDN Algebraic Rearrangement & Kernel Fusions
• [PR #2498] Associative Decoupling: Rewrites GDN as $O=(\text{decayed } S)q + \Delta(k^\top q)$, enabling MXU to compute on decayed state in parallel with VPU updates (slashes bubbles, +4.48% throughput);
• [PR #3016] GDN v3 Fusion: Fuses Conv1D and GDN into a single kernel, slashing HBM roundtrips for 1.41× ~ 2.14× speedup;
• [PR #2741 / #2650]: Eliminates vector register spills; async DMA state double-buffering yields +11.3% throughput.
PCP8 环形流水线 (Ring Attention) 编排
• 解决二次方显存爆炸:设计并落地 `PCP8`(Prefill Context Parallelism = 8),在 8 个 Core 上构建单向哈密顿闭环,分块并行流动计算;
• 极致掩盖:在计算当前 16K 块的同时预取下一块,计算与通信 100% 异步重叠,128K 超长文 TTFT 达到 4.62s,硬核反超 NVIDIA GB200(~5.0s)。
PCP8 Ring Attention Pipeline Orchestration
• Taming $O(N^2)$ Complexity: Designed and deployed `PCP8` (Prefill Context Parallelism = 8) across an 8-core Hamiltonian ring with segmented streaming attention;
• 100% Latency Masking: Overlaps chunk computation with next-chunk ICI transfer, clocking a 128K TTFT of 4.62s, outpacing NVIDIA GB200 (~5.0s).
MoE 稀疏路由与通信MoE Routing & Collectives 通信合并与 SparseCore 协处理器卸载
• [PR #2174] 单次 All-Gather 合并:将 Expert IDs 与路由权重合并为一次通信,每层节省 80µs,全模型 58 层直接净省 4.64 ms!
• [PR #2888 / #2137] 协处理器接管碎活:将 Reduce-Scatter 与 MoE Ragged Token 重排卸载到 SparseCore 专用协处理器,利用片内 Die-to-Die 预先求和,主 MXU 0 调度气泡。
Collective Merging & SparseCore Silicon Offloading
• [PR #2174] Single All-Gather Fusion: Merged expert IDs and routing weights into one collective, saving 80µs/layer and 4.64 ms across 58 layers per forward pass;
• [PR #2888 / #2137] SparseCore Offloading: Offloaded Reduce-Scatter and ragged token sorting to dedicated SparseCore co-processors, pre-reducing on fast Die-to-Die links.
`owner_output` 原地沉淀与 Fused MoE 双缓冲
• 消减 50% 冗余通信:推行 owner_output,让 Token 在目标专家 Core 上计算完成后原地就地累加沉淀,彻底砍掉倒流回源 Core 的回程流量;
• 片上 VMEM 双缓冲 (PR #891):Fused MoE v2 (W1×2 + W2×1) 在超导 VMEM 内部建立双池交替,隐藏跨 Chip ICI 流量,Decode 冲上 5,650 tok/s。
`owner_output` In-Place Accumulation & MoE Buffering
• 50% Traffic Elimination: Implemented owner_output to accumulate expert outputs in-place, eliminating reverse token transmission back to source ranks;
• On-Chip VMEM Double Buffering (PR #891): Fused MoE v2 (W1×2 + W2×1) alternates compute and prefetch inside VMEM, driving decode to 5,650 tok/s.
集群扩展与多机瓶颈Multi-Host & Cluster Fabric 拓扑感知与下一代 Boardfly 架构
• 3D Torus 缠绕环优化:通过 Twisted Torus 莫比乌斯环拓扑将机架内最坏跳数从 N 压缩为 N/2;
• TPU v8i 规划:针对超大规模 MoE 放弃 3D Torus,全面演进为 Boardfly 高基交换拓扑(19.2 Tb/s ICI),网络直径削减 50% 以上。
Topology Awareness & Future Boardfly Interconnect
• 3D Torus Wrapping: Twisted Torus shaves worst-case hop count from N to N/2 within 64-chip racks;
• TPU v8i Roadmap: Transitions from Torus to Boardfly high-radix fabric (19.2 Tb/s ICI), slashing network diameter by >50% for wide MoE scaling.
`PCP8 PP4` 跨机破局与 Raiden 双网卡 400G
• 4-Host 跨机破局:针对 4 机 Z 轴带宽瓶颈,设计 `PP4·PCP8` 混合切分,跨机仅传递单 Token 隐层向量(理论推算 ~20k/rank、TTFT ~1.6 s;真实权重 P8K·D1 + MTP 实测从 44k 提升至 55k–58k tok/s);
• Raiden 100% 本地 DMA:首创绑定双 200G 网卡亲和调度(Rank 0..3 绑 NUMA 0 / Rank 4..7 绑 NUMA 1),实测打满 367.0 Gbps (91.8% 线速),根除跨 Socket UPI 拥塞。
`PCP8 PP4` Multi-Host Solution & Raiden Dual-NIC 400G
• 4-Host Bottleneck Mitigation: Overcame Z-axis bandwidth saturation via `PP4·PCP8` (passing only micro-activations across hosts; analytical target ~20k/rank & ~1.6 s TTFT, real-weight P8K·D1 + MTP measured at 55k–58k tok/s vs. 44k baseline);
• Raiden NUMA-Local DMA: Bound dual 200G NICs to local NUMA nodes (0..3 to NUMA 0 / 4..7 to NUMA 1), hitting 367.0 Gbps (91.8% wire speed) with zero UPI bus cross-traffic.
量化与生态基础设施Quantization & Ecosystem Pallas Subchannel 算子与原生 TorchTPU
• Subchannel Matmul:在 xla/benchmarks 中沉淀标准化 s4 微缩放算子内核;
• 原生 TorchTPU 生态外化:联合 vLLM 推进原生后端,使 PyTorch 无缝挂接底层 Pallas 高性能内核,逐步摆脱中间转译层。
Pallas Subchannel Kernels & Native TorchTPU
• Subchannel Matmul: Standardized s4 micro-scaled matrix multiply primitives in OpenXLA benchmarks;
• TorchTPU Externalization: Collaborating with vLLM on a first-class backend allowing PyTorch to natively invoke underlying Pallas kernels without translation overhead.
`NVFP4 + W4A8` 软件反量化破局 (PR #912)
• 打破无原生 FP4 困境:针对 TPU v7 无原生 4-bit 计算硬件的限制,设计 HBM 存 4-bit 权重 + 片上 VPU 实时反量化为 8-bit 的软件管线,零精度损失下访存吞吐翻倍,直追 Blackwell。
`NVFP4 + W4A8` Software Dequantization (PR #912)
• Bypassing Silicon FP4 Absence: Compensated for lack of native FP4 silicon by storing 4-bit weights in HBM and dequantizing on-chip into 8-bit, effectively doubling memory bandwidth while matching FP8 accuracy.
⚡ TPU7x (Ironwood) 原生脉动阵列 Pallas Grouped GEMM 算子架构突破与硬件实测 TPU7x (Ironwood) Native Systolic Pallas Grouped GEMM Architecture & Hardware Verification Study: qwen35-397b-pallas-tune

针对 Qwen 3.5 397B MoE 稀疏路由中专家动态切片(Dynamic Slicing)开销大、访存未合并(Un-coalesced HBM Gathers)以及 TPU7x(Ironwood)新一代硬件物理特性的瓶颈,基于 Google DeepMind 官方开源优化标准与 torchtpu-agents 框架,开展物理集群实证调优: Addressing the bottlenecks in Qwen 3.5 397B MoE routing—dynamic slicing penalties, un-coalesced HBM gathers, and physical nuances of TPU7x (Ironwood)—we conducted autonomous hardware-level optimization via the torchtpu-agents study protocol:

1. 256×256 原生脉动阵列对齐 1. Native 256×256 Systolic Alignment
彻底打破 TPU v4/v5p 时代 128×128 切块惯性。TPU7x(Ironwood)采用 256×256 物理脉动阵列单元,128 切片会导致 50% 算力被零填充(Zero-Padding)浪费。升级至 256×256 瓦片后,单步延迟降至 0.821 ms,有效吞吐达 10.46 TFLOPs。 Departed from legacy 128×128 tiling. TPU7x features 256×256 systolic cells; 128-tiling induces 50% zero-padding compute waste. Upgrading to 256×256 natively saturates the MXU, achieving 0.821 ms step latency and 10.46 TFLOPs.
2. 横向收缩 (Transverse Contraction) 避让 XLU 2. Transverse Contraction Bypassing XLU
在片上 VMEM 中执行显式转置 .T 会强制调用跨通道单元(Cross-Lane Unit, XLU)重排向量通道,引入 9.4% 的物理停顿。改用 jax.lax.dot_general 维度契约,由 MXU 脉动阵列在累加阶段直接吸收转置布局,彻底消除 XLU 物化惩罚。 Explicit transpositions in VMEM trigger Cross-Lane Unit (XLU) lane swizzling, imposing a 9.4% hardware stall. Replacing with transverse contraction via dot_general contracts directly across dimensions, letting MXU systolic arrays absorb transposed layouts at zero cycle cost.
3. VMEM 双缓冲 (Double Buffering) 隐匿 DMA 3. VMEM Double Buffering DMA Overlap
配置 pl.Buffered(buffer_count=2) 双缓冲管线,在前一个专家瓦片进行 MXU 脉动计算的同时,DMA 异步预取下一个专家权重切片,使高带宽访存搬运完全隐匿在计算延迟背后。 Configured pl.Buffered(buffer_count=2) to overlap asynchronous DMA HBM-to-VMEM transfers with active MXU systolic computation, achieving near-perfect compute-communication overlap.
优化维度 (Dimension) 基线设计 (Baseline) TPU7x 优化实现 (Optimized) 实测收益与物理对账 (Verified Metrics)
脉动阵列切块 (Tiling) 128×128 (v4/v5p 遗留) 256×256 (Ironwood 原生) 消灭 50% Zero-Padding;单步延迟 0.821 ms,吞吐达 10.46 TFLOPs
累加精度与类型铸造 隐式 VPU Downcast (35% 延迟膨胀) 32-bit VMEM 原生累加器 消除 17%~35% VPU 降准开销;Logits 余弦相似度 1.000000
权重转置通道开销 片上 VMEM .T 物化 (XLU 竞争) 横向收缩 (Transverse Contraction) 消除 9.4% XLU Swizzle 硬件停顿;纯 MXU 原生契约乘加
数值对齐门禁 (Gate) Max Diff < 0.001 实测 Max Diff = 0.000065 PASS(Top-1 一致率 99.80%,Top-5 100.0%)
🌐 全链路分布式集合通信(Collective Communication)全景对账 End-to-End Collective Communication Panorama & Microarchitecture Ledger Full Taxonomy

在 Qwen 3.5 397B 异构推理系统(Prefill PCP8+EP8 → Decode TP2·DP4·EP8)中,集合通信算子并非随意调用,而是根据物理拓扑层级(片内 Die-to-Die、机箱内 ICI 铜线 Torus、跨机光纤)和模型并行维度(PCP / TP / DP / EP)进行了严格的数学正交编排。下表全景梳理系统内每一个集合通信算子的物理通路、触发阶段、数据负载与防阻塞设计: In the Qwen 3.5 397B serving architecture (Prefill PCP8+EP8 → Decode TP2·DP4·EP8), collective communication primitives are mapped with rigorous orthogonality to both physical topology tiers (Die-to-Die, in-chassis ICI copper torus, cross-host optical links) and parallelism axes (PCP / TP / DP / EP). The ledger below dissects every collective primitive across stages, payload shapes, and latency masking mechanisms:

并行维度 / 算子类型Parallel Axis / Primitive 作用阶段Stage 物理传输介质 & 硬件接口通道Physical Medium & Hardware Interface XLA 原语 / 通信模式XLA Primitive / Pattern 传输内容 & 张量形状Payload & Tensor Shape 典型时延 & 掩盖机制Latency & Overlap Rationale
PCP=8
环形流水上下文并行
Ring Context Parallel
Prefill 硅基 Die-to-Die + 板级铜箔 (PCB Traces)
• 单机 8 Core(4 芯片 / 1 Host):
- 同芯片内:片上超高密度硅中介层微凸块(On-Package Die-to-Die,比单条 ICI 快 6×);
- 跨芯片间:主板级 2×2×1 ICI 铜箔差分走线(每链路双向 200 GB/s)。
collective-permute
8 跳单向 P2P 环网传递
KV Cache Chunk
[16384, 4, 128] FP8
每跳单向约 8 MB
~950 μs / 跳
100% 异步双缓冲掩盖!在本地计算 16K QK GEMM(~4,120 μs)时后台 DMA 流动,气泡 = 0 μs。
TP=2
片内张量并行
Intra-Chip Tensor Parallel
Decode 基板微凸块硅互联(Silicon Interposer / D2D Micro-bumps)
• 绝对不出芯片封装(Zero-Off-Package):
- 纯依靠封装内部 Core 0 ↔ Core 1 超短距高密度 Die-to-Die 金属微互联,带宽达到 ICI 单链路的 6 倍以上;
- 不占用任何板级 ICI 铜线或外部网络带宽。
all-reduce (Sum)
双核对称规约
Attention 投影分量
[Batch, Hidden_Dim=4096]
分担 Dense GEMV 输出
~85 μs / 次
不跨物理芯片,走硅片微米级互联,时延极低;将单核 Dense 显存砍半,使 Decode 能开满 Batch=64。
EP=8 (Dispatch)
MoE 专家路由分发
MoE Token Dispatch
Prefill & Decode 板级直连铜线 + 机箱背板(Backplane Copper Traces)
• 机内 3D 环面铜网(In-Chassis Torus):
- 8 Core 专家域全展开;跨芯片部分完全通过主板/背板差分铜线以 3D Torus 拓扑点对点直连(总出口双向 1,200 GB/s);
- 单机模式不出机箱,双机模式走机柜级短铜缆(DAC, Direct Attach Copper)。
all-to-all (Scatter)
多对多非连续 Token 重分发
激活值 Token 向量
[Routed_Tokens, 4096]
Top-8 路由(每 Token 派发 8 个副本)
~1.1 ms (Prefill) / ~420 μs (Decode)
由 SparseCore v3 硬件协处理器卸载 Top-K 索引与分拣;通过 Fused MoE v2 双缓冲重叠跨芯片搬运。
EP=8 (Combine)
MoE 专家聚合回传
MoE Token Combine
Prefill & Decode 片上 D2D 总线 + 背板铜线(Hybrid Silicon/Copper Fabric)
• 分层介质聚合:
- 本地 Core 间通过超高速封装微凸块(D2D)做首级局部聚合(owner_output 分支);
- 跨芯片部分走板级差分铜线回传归属 Host,降低出芯流量。
all-to-all (Gather-Reduce)
专家前向加权累加后回传原籍
专家 FFN 输出向量
[Tokens, 4096] FP8/BF16
带 Softmax 门控权重的加权和
~1.3 ms (Prefill) / ~450 μs (Decode)
阿里 PR #912 采用 owner_output 分支优化,在片内 D2D 预先聚合局部专家输出,减少 ICI 出口拥塞。
DP=4
推理数据并行
Inference Data Parallel
Decode 零物理介质(No Physical Transmission / Air Gap)
• 物理介质完全解耦:
- 4 个 DP 副本各占 1 颗独立物理芯片(2 Cores);
- 前向推理无需梯度同步,Attention 阶段物理介质传输开销恒等于 0(完全不消耗铜线、微凸块或光纤)。
无需通信(Zero-Comm)
无反向梯度传播,请求完全解耦
N/A (无数据交换)
各组独立处理不同用户 Batch=64
0 μs(彻底免除 All-Reduce)!
规避了 DCP(解码上下文并行)高频跨芯片 Softmax 规约的毁灭性延迟,将网络净空留给 MoE All-to-All。
P/D Disagg Transfer
KV 缓存越界解耦投递
P/D KV Handoff
P → D 跨节点 PCIe Gen5 总线 + 数据中心双网卡光纤(DCN 2×200G / 400G 聚合 RoCE/以太网)
• 跨物理主机(Host-to-Host):
- 端到端链路:Prefill TPU HBM → PCIe 5.0 主机通道(~119.2 GiB/s)→ Host DRAM 池化缓冲 → 双 200G 网卡(聚合 400G)数据中心光纤上行 → Decode Host DRAM → Decode TPU HBM。
Async P2P / RDMA Send-Recv
自建 TariMempool / Raiden 同步通道
15 层 FA 全量 KV
15 GiB / 128K 单请求
45 层 GDN 线性状态(16 KB)近乎零开销
~180 Gbps 线速推流
利用 Host DRAM 异步池化中转;Decode 首字吐出前提前在后台触发投递,解耦两端调度压力。
PCP8 PP4 (跨机扩展)
流水线 + 环形混合并行
Pipeline + Ring Hybrid
4-Host (16 Chip / 32 Dev) 光回路交换机光纤链路(OCS Optical Circuit Switched Fibers)
• 机柜级跨 Host 拓扑:
- Cube 内部走直连铜缆,超过单 Cube 后 Z 轴全靠 OCS 动态光路交换与多模光纤(Optical Fibers)传输;
- PP4 将原本带宽受抑的跨机 Z 轴限制为仅传送 Stage 激活值。
P2P Pipeline Activation Handoff
跨 Stage 阶段切片(PP4: 15层/Stage)
Stage 边界激活值
[Tokens, Hidden_Dim=4096]
仅在 15/30/45 层交界处传输
破除 4-Host 跨机带宽塌陷
将原本受限的全局 Z 轴 All-Gather 限制在单机内,跨机仅传输轻量激活值;理论推算 ~20k/rank(TTFT ~1.6 s),真实权重 P8K·D1 实测达 55k–58k tok/s。
💡 集合通信设计的底层物理介质灵魂总结: 大模型推理的集合通信延迟之所以能被驯服,是因为阿里的系统架构做到了「传输介质与通信强度的精准分层匹配」:
1. 微米级硅基介质(Silicon Die-to-Die):极高频的 GEMV 稠密规约(TP=2 all-reduce)只走片内硅微凸块(~85 μs),绝不出芯片,零板级网络开销;
2. 厘米级铜箔走线(PCB / Backplane Copper):超大 Attention(PCP8 collective-permute)与 MoE 专家路由(EP8 all-to-all)完全约束在主板与机箱差分铜缆内(双向 1,200 GB/s),配合双缓冲与 SparseCore 彻底隐藏时延;
3. 米级/机架级光纤网络(DCN / OCS Optical Fiber):跨机器只承担低频非实时的异步 KV 状态交接(PD Disagg)与轻量流水线 Stage 激活值投递(PP4),彻底杜绝全局高频 All-Gather/All-Reduce 冲击光纤带宽瓶颈!
⚡ 深度解剖:Raiden NUMA-Aware 双网卡(Dual-NIC)400G 聚合线速越界架构 PR #923 / vLLM-TorchTPU 实测

在 Qwen 3.5 397B 长文分离推理中,单个 128K 请求跨机投递的 15 层 FA KV Cache 达到 15 GiB。如果仅走单张 200 Gbps 网卡,线路在并发 C≥4 时就会迅速被打满在 192 Gbps 上限;更致命的是跨 CPU Socket 的 UPI 内存总线瓶颈——发往远端 Socket 芯片的数据必须二次翻越 UPI,造成严重的内存总线争抢与软中断抖动。Google 与阿里联合研发的 Raiden NUMA-Aware Dual-NIC 方案通过物理亲和性编排彻底破局:

【Prefill 节点】ninja-3 (PCP=8, 4 芯片 / 8 Cores / 224 vCPUs) NUMA 0 (Cores 0..3) · Shards 0..3 · 出口网卡 ens6 (10.10.0.2) NUMA 1 (Cores 4..7) · Shards 4..7 · 出口网卡 enp192s4 (10.190.0.2) Rail 0 (ens6) · 200 Gbps Rail 1 (enp192s4) · 200 Gbps 聚合吞吐 400 Gbps (线速达 367 Gbps) 【Decode 节点】ninja-4 (TP=2 · DP=4 · EP=8 / 1TP·8DP 兼容) NUMA 0 · 网卡 ens6 (10.10.0.3) 绑定 Listener → 接收 TPU Cores 0, 1, 2, 3 (Chip 0, 1 / DP 0, 1) ✓ 100% 本地 DMA 落地,彻底消除跨 Socket UPI 搬移! UPI NUMA 1 · 网卡 enp192s4 (10.190.0.3) 绑定 Listener → 接收 TPU Cores 4, 5, 6, 7 (Chip 2, 3 / DP 2, 3) ✓ 100% 本地 DMA 落地,彻底消除跨 Socket UPI 搬移!
1. 自动物理拓扑探测与 NUMA 绑定
• tpu_utils.cc 扫描 /sys/class/net/* 排除控制面(MTU<1500)与虚拟网卡,自动识别物理数据 NIC;
• Decode 端每个 Worker 按照其 local_rank 自动关联物理芯片 PCIe BDF;Rank 0..3 绑定 NUMA 0 (ens6),Rank 4..7 绑定 NUMA 1 (enp192s4);
• 收益:Decode 接收端达成 100% NUMA-local DMA 投递!
2. 非对称 Rail 与源 IP 智能解耦
• 云上两张 200G 网卡互不连通(Non-Rail-Aligned),若强行绑定 Client 源 IP 会引发跨 VPC 网段丢包;
• Raiden 默认设置 TPU_RAIDEN_ENABLE_SOURCE_IP_BIND=0,让 Linux 内核路由表根据目标 IP 自动匹配本地出口;
• 实测表现:在并发请求交织下,两张物理网卡呈现严格的 50.0% / 50.0% 完美流量均摊(见 sar -n DEV 监控)!
3. 实测吞吐与端到端 TTFT 收益
• ProcFS 线速吞吐:从单网卡饱和极限 192.7 Gbps 提升至双网卡 367.0 Gbps(1.91× 提速),打满 400G 双网卡线速;
• Reshard 延迟:长文 64K / C=16 下传输时间从 706.9 ms 缩减至 336.6 ms(提速 2.10×);
• 正确性:32K 针锋相对(Needle-in-a-Haystack)跨 NUMA/双卡检索达成 100% 准确率。

📋 上线冲刺 PR 看板(09-14 基线 ➔ 09-25 封版合入 ➔ 10-16 状态) 📋 Launch-sprint PR board (09-14 baseline ➔ 09-25 release cut ➔ 10-16 status)

⚠ 「是否阻塞上线」是这张表最关键的一列,切勿省略 ⚠ "Blocks launch?" is the most important column here — do not drop it
下表跟踪主模型与下一代超大 MoE 上生产的核心 PR 状态。9 月下旬核心架构与通信修复已集中合入主干并切出 official/release-0.0.1 发布分支;10 月起工作重心转向生产长稳、精度对齐与分支收敛。 This table tracks the core PRs gating production launch. By late September the key mesh, KV-transfer, and MoE kernel PRs landed on main and cut into official/release-0.0.1; October focus shifted to fleet stability, numerical determinism, and branch convergence.
工作项Item 优先级Pri PR / 模块PR / Module 状态(截至 2026-10-16)Status (as of 2026-10-16) 阻塞上线?Blocks launch?
PCP SEQ_ON_LANE 与在线 FP8 / GDN 量化合入PCP SEQ_ON_LANE & online FP8 / GDN quantization P0 #608 / #750 / #913 ✅ 已合入merged 是yes
FusedMoE v2 NVFP4 + W4A8 与自适应 2+1 / 3+2 双缓冲FusedMoE v2 NVFP4 + W4A8 & adaptive 2+1 / 3+2 buffer P0 #912 · #1108 · #1133 ✅ 均已合入(含 Decode MoE 性能优化)all merged (incl. Decode MoE perf) 是yes
marconi-prefix-cache 使能(混合模型前缀缓存)marconi-prefix-cache enablement (hybrid-model prefix cache) P1 #765 ✅ 完成done 否no
MTP 性能优化与 PCP+MTP / KVCM 冲突修复MTP perf & PCP+MTP / KVCM conflict fix P0 #1008 · #1106 ✅ 已合入merged 是yes
torchtpu OOM 回归修复与 mesh / runtime partition 修复torchtpu OOM regression & mesh / runtime partition fix P0 #952 · #1182 (PCP+TP) ✅ 端到端验证通过,已于 09-25 合入E2E verified & merged by 09-25 是yes
跨拓扑 KV Transfer(PCP8 ➔ DP4·TP2 及 PCP+TP ➔ DCP+TP)Cross-topology KV Transfer (PCP8 ➔ DP4·TP2 & PCP+TP ➔ DCP+TP) P0 #1183 · #1121 (PCP+HND DMA) ✅ 已合入(单机有效 TP 2→1/2/4 均验证通过,推进多机)merged (effective TP 2→1/2/4 verified single-host; scaling multi-host) 是(上线部署形态)yes (deployed topology)
Raiden 双网卡 400G 与 KV Cache HBM ↔ DRAM 全量映射修复Raiden dual-NIC 400G & KV-cache HBM ↔ DRAM mapping fix P0 #839 / #923 · #1031 ✅ 双网卡实测达 367 Gbps;全量映射修复已提交dual-NIC verified at 367 Gbps; full-mapping fix landed 是yes
10 月生产分支收敛与关键正确性补丁回移October release-branch convergence & critical correctness backports P0 #1745 / #2024 · #1941 · #1983 · #2022 🔄 Logprob FP32 (#1745)、跨设备时钟 (#1941)、DP+MTP 步序对齐 (#1983) 已合入;连接池化与超时优化审阅合入 official/release-0.0.1 中Logprob FP32 (#1745), cross-device clock (#1941), and DP+MTP step alignment (#1983) landed; connection pooling & timeout PRs under review for official/release-0.0.1 是(以合入 official 分支为准)yes (gated on landing in official branch)

🛡️ 容灾与长稳:09-18 ➔ 10-30 演进对账 🛡️ Resilience and long-run stability: 09-18 ➔ 10-30 status

⚠ 请勿把这一组笼统表述为「容灾体系已建成、零内存泄漏」 ⚠ Do not summarize this group as "resilience complete, zero memory leaks"
9 月下旬修复了 tpu-sync 线程泄漏与多项稳定性缺陷(#1112 / #1115 / #1116),但在批量节点试投产与真实长稳压测中,又进一步收敛出硬件级 CoreHalt、MTP 接收率偶发跌零及边缘竞态等深水区问题: Late-September PRs resolved the tpu-sync thread leak and core stability issues (#1112 / #1115 / #1116), while subsequent fleet burn-in surfaced deeper hardware CoreHalt events, intermittent MTP acceptance drops to 0%, and edge-case race conditions:
场景Scenario 真实状态(截至 10-30)Actual status (as of 10-30) 工程依据与处理进展Engineering evidence & progress
Decode 阶段 client abort 的 KV / 请求泄漏KV / request leak on client abort during decode ✅ 已修复fixed #1039 (Raiden Stage-3 抢占路径)(Raiden Stage-3 preemption path)
长稳压测 tpu-sync 线程泄漏与基础稳定性修复Prolonged stress tpu-sync thread leak & stability fixes ✅ 已修复合入fixed & merged tpu-sync frame_rpc / RawBufferTransport 线程泄漏已修复,伴随 #1112、#1115、#1116 合入tpu-sync frame_rpc / RawBufferTransport thread leaks fixed; #1112, #1115, #1116 merged
Raiden controller / vLLM worker 崩溃恢复与就绪超时Raiden controller / vLLM worker crash recovery & readiness timeout ✅ 已通过(多 Rank 启动超时阈值通过 #2022 调优)passed (multi-rank readiness timeout tuned in #2022) 稳定性测试 4.1 / 4.2;针对多 Rank 启动快慢不均导致的超时做参数加固stability tests 4.1 / 4.2; readiness timeout adjusted for multi-rank startup skew
MTP 与 KVCM 交互竞态 / 异常防护(Block Table 竞争、NaN 传染、Decode 重算)MTP & KVCM race / fault protection (block-table race, NaN poisoning, decode recompute) 🔄 补丁已在生产分支验证,向官方发布分支合入中patches validated on production branch; upstreaming to official release branch ① 修复 MTP 异步上传 Block Table 时 CPU 源缓冲区被覆盖的竞态(a503c18);② Hidden states 出现 NaN 时通过 wait_for_save(has_nan) 阻断污染缓存槽位;③ PD 分离下彻底禁用 Decode 重算(1) Fix CPU buffer overwrite race during async MTP block-table H2D upload (a503c18); (2) block KVCM caching on NaN via wait_for_save(has_nan); (3) strictly disable decode recomputation in PD disagg
在线接流过程中 MTP 接收率偶发跌 0%Intermittent MTP acceptance rate drop to 0% under live traffic 🔄 传染路径已在 Kernel 层堵住,现象已很少出现;NaN 源头仍在排查(疑似 Prefill 侧)propagation blocked at kernel level and now rare; NaN origin still under investigation (suspected Prefill side) 根因是 0.0 × NaN = NaN:QK 阶段的 mask 只把 softmax 权重置 0,挡不住 PV 对整个 tile 的累加。三道防线均已合入:① GMM Kernel 修复(#1661);② batched_rpa Decode(fuse_accum)模式下 l_prev/o_prev 跨序列边界显式重置(#1877,Prefill 性能 0% 变化);③ mask_invalid_v 在 PV 前清零无效 V 条目(#2454,SEQ_ALONG_LANE 默认开启,开销 +0.30% geomean)。NVFP4 397B 1P1D(PCP8 ➔ DP4·TP2)54h / 56h 两轮长稳测试 tpu_model_nan_rounds_total 均为 0;下一步在 Prefill / Decode / KV Transfer 三段加日志定位 NaN 源头。更正:此前「FA D2H 竞态」的推测未获证实。Root cause is 0.0 × NaN = NaN: masking QK only zeroes softmax weights and cannot stop PV from reducing across the whole tile. Three defenses merged: (1) GMM kernel fix (#1661); (2) explicit l_prev/o_prev reset at sequence boundaries in batched_rpa decode (fuse_accum) mode (#1877, 0% prefill impact); (3) mask_invalid_v zeroes invalid V entries before PV (#2454, on by default for SEQ_ALONG_LANE, +0.30% geomean). Two NVFP4 397B 1P1D (PCP8 ➔ DP4·TP2) soak runs of 54h / 56h recorded zero tpu_model_nan_rounds_total; next step is stage-level logging across Prefill / Decode / KV transfer. Correction: the earlier "FA D2H race" hypothesis was not confirmed.
大规模集群低频硬件 E200 CoreHaltLow-frequency fleet-level hardware E200 CoreHalt 🛠️ 已定位并修复两类触发路径,持续完善 Halt 堆栈诊断two root-cause paths identified & patched; enhancing crash stack diagnostics ① SparseCore ragged_gather_reduce 边界条件修复;② DP>1 + EP + MTP 跨 Rank 步序错位死锁修复(#1983);③ Decode Pallas Kernel 越界访问 [96, 128) GiB 未映射区理论复现(详见下方专题剖析)(1) SparseCore ragged_gather_reduce boundary fix; (2) DP>1 + EP + MTP cross-rank step-order deadlock fixed in #1983; (3) Pallas DMA out-of-bounds into unmapped [96, 128) GiB reproduced in unit test (see deep dive below)

🔭 下一代超大 MoE 架构演进与跨机扩展状态(截至 2026-10-16) 🔭 Next-Gen Ultra-Large MoE Scaling & Multi-Host Status (as of 2026-10-16)

🧭
多机部署形态:PCP16·TP2 ➔ TP32·DCP4/8 与 PP4·PCP8Multi-Host Topologies: PCP16·TP2 ➔ TP32·DCP4/8 & PP4·PCP8
4-Host (16 物理芯片 / 32 Device) 跨机扩展与 P/D 独立拉起 4-Host (16 physical chips / 32 devices) scaling & disaggregated bring-up
架构层Architecture
• P/D 独立拉起与端到端连通:PCP16·TP2 与 TP32·DCP4/8 跨拓扑 KV Transfer(#1183)及 PCP+TP 正交 Mesh(#1182)均已合入,真实权重加载与端到端问答已验证通过。
• 算子与布局完备性:Fused MoE v2 自适应 3+2 / 2+1 双缓冲(#1108)、Fused MoE + TP2、DCP + HND、GDN Q/K 头复制(#1082)及 1M 超长上下文 RPA 修复(3f57750 / b5a29fe)均已就绪。
• 混合并行探索:除 PP4·PCP8 + MTP 跨机流水线外,新一代架构实验已跑通 DP + TP + EP 混合模式,主干用例同步从 PCP4→DP4 / PCP8→DP8 升级对齐至生产标准 PCP4→TP2·DP2 / PCP8→TP2·DP4。
• Disaggregated Bring-Up Verified: Cross-topology KV transfer (#1183) and orthogonal PCP+TP mesh support (#1182) have landed; independent P/D startup and real-weight E2E inference are verified.
• Kernel & Layout Readiness: Adaptive 3+2 / 2+1 buffering in Fused MoE v2 (#1108), Fused MoE + TP2, DCP + HND, GDN Q/K head replication (#1082), and 1M-context RPA fixes (3f57750 / b5a29fe) are all in place.
• Hybrid Parallelism: Alongside PP4·PCP8 + MTP, experimental next-gen variants have validated mixed DP + TP + EP execution, and CI test cases upgraded from PCP8→DP8 to production-aligned PCP8→TP2·DP4.
当前实测(10-16)Measured so far (10-16)
4-Host PP4·PCP8 + MTP 真实权重 P8K·D1 吞吐达 55k–58k tok/s(由 44k 提升,逼近 60k 预期) 4-Host PP4·PCP8 + MTP real-weight P8K·D1 reaches 55k–58k tok/s (up from 44k, approaching 60k target)
🔧
已收敛阻塞 vs. 剩余待攻克项Resolved Blockers vs. Remaining Frontier Items
9 月底阻塞项复盘与 10 月新阶段聚焦 Late-Sep blocker resolution & October engineering focus
演进跟踪Tracking
• [已收敛] 跨机 Token All-Gather 退化:原 4-Host 下全局 All-Gather 因非全连接 Z 轴带宽受限损失约 30% 性能,现已正式废弃纯全局 All-Gather 路线,切换为 PP4·PCP8 跨机流水线。
• [已合入] Fused MoE v2 自适应双缓冲与 TP2 协同:#1108 与 #1182 已合入主干。
• [待收尾] 长序列与分块调度精调:① DCP 长上下文 batched_rpa 的 Scoped VMEM 精确估算仍在收尾(重构前置 #1105 已合入,移植 #1081 跟进中);② PP4·PCP8 下 8K 请求按 block_size 切分为 3K+5K 引发的非对称微批次 Padding 抖动待合入对称调度修复;③ 单机 Host DRAM(已由 440 GiB 降至 230 GiB)继续下压以扩大 KV Store 缓存池。
• [Resolved] Cross-Host Token All-Gather Degradation: The pure 4-host All-Gather path (~30% E2E loss on non-fully-connected Z links) has been formally retired in favor of PP4·PCP8 pipeline hybrid parallelism.
• [Merged] Fused MoE v2 Adaptive Buffering & TP2 Mesh: Landed via #1108 and #1182.
• [Remaining] Long-Context VMEM & Scheduler Polish: (1) Exact VMEM accounting for DCP long-context batched_rpa (refactor #1105 merged, port from #1081 in progress); (2) symmetric micro-batch scheduling for PP4·PCP8 to eliminate 3K+5K chunk-padding variance; (3) further Host DRAM reduction beyond 440 ➔ 230 GiB to expand local KV store capacity.
口径校准提示Calibration note
区分单机 PP8 实测 (80k) 与 4-Host PP4·PCP8 实测 (55k–58k);「Marconi −60% TTFT」「dispatch −25%」「120% 过载 2h」无实测出处 Distinguish 1-host PP8 (80k) from 4-host PP4·PCP8 measured (55k–58k); "Marconi −60% TTFT", "−25% dispatch", "2h at 120% overload" remain unsourced
🔬 2026-09 ➔ 2026-10 生产深水区工程沉淀:硬件异常诊断、位级确定性、Host 内存解剖与发布治理 Sep ➔ Oct Production Engineering Deep Dives: Hardware Fault Forensics, Bit-Exact Determinism, Host DRAM Anatomy & Release Governance
🌐 EN / 中文

当大规模 MoE + Hybrid Attention 模型从基准测试迈向百台级 TPU v7x (Ironwood) 生产集群长稳运行时,工程重心从单一峰值吞吐转向低频硬件异常(CoreHalt)根因归因、跨 Rank 位级数值确定性(Bit-Exact Determinism)、Host DRAM 内存池解剖、控制面长尾延迟治理以及生产分支 CI/CD 交付规范。以下为脱敏后的五项核心系统工程沉淀: As ultra-large MoE + Hybrid Attention serving transitions from benchmarking to multi-hundred-node TPU v7x (Ironwood) production fleets, engineering focus shifts from peak throughput to low-frequency hardware fault forensics (CoreHalt), cross-rank bit-exact numerical determinism, Host DRAM footprint anatomy, control-plane tail-latency elimination, and release-branch CI/CD governance. Below are five sanitized systems-engineering deep dives:

🚨
1. TPU v7x E200 CoreHalt 假阳性日志表象与三类物理根因1. TPU v7x E200 CoreHalt Misleading Signatures & Three Root Causes
为什么报错总指向 SC3 @ 0x3 与首算子 tt_jit_layernorm_*? Why do fatal halts repeatedly point to idle SC3 @ 0x3 and tt_jit_layernorm_*?
硬件诊断Forensics
• 日志表象陷阱:生产集群偶发硬件致命错误 cmn_local_fatal_errors_transform.set raw=0x1 时,底层驱动往往回退打印当前处于闲置状态的 SparseCore 3 @ 0x3 与计算图首个子图 tt_jit_layernorm_*,缺乏直接调用栈,极易误导排查方向。
• 根因路径 A(SparseCore 边界):ragged_gather_reduce 在极端 Ragged 分桶下的边界条件越界,已通过算子边界补丁修复。
• 根因路径 B(Pallas Kernel DMA 越界落入未映射物理区):Decode 阶段 Pallas 内核(如 gdn/v3、batched_rpa)显式开启 disable_bounds_checks=True 时,一旦读取到越界 Block / Page ID 且计算出的物理地址落入 TPU v7x 单 Die [96, 128) GiB HBM 未映射地址空洞(单 Die 物理 HBM 为 96 GiB),硬件立即触发 Fatal Halt 并回退打出上述假阳性签名(已通过单元测试精准复现同一错误日志)。
• 根因路径 C(跨 DP Rank 步序错位死锁 + SIGTERM 级联 Halt):在 DP>1 + EP + MTP 复合并行下,当单一 DP Rank 请求数跨过阈值被拆分为 2 个 Chunks 时,真实请求执行的 propose()(Step-Major 步序)与其余空闲 Rank 执行的 run_dp_dummy_draft()(Chunk-Major 步序)发生步序与 Token Shape 错位,导致跨 DP 的 MoE reduce-scatter 集合通信死锁;300s sample_tokens 超时触发 SIGTERM 杀进程后,同颗物理芯片上的对端 Core 因 Peer VF 先行关闭而连带报出 SC3 @ 0x3 Halt。修复方案(#1983):移除 SMEM request chunk 拆分上限(62f98fb / 5575f78)并强制对齐 Dummy Draft 与真实 Chunk 的步序(ebe95fd / d5d5114)。
• Misleading Log Signature: When hardware raises cmn_local_fatal_errors_transform.set raw=0x1, the runtime fallback dumps the idle SparseCore 3 @ 0x3 and the very first subgraph tt_jit_layernorm_* without a Python/HLO stack trace.
• Path A (SparseCore Boundary): Edge-case index bounds in ragged_gather_reduce, resolved via kernel guard fixes.
• Path B (Pallas DMA Out-of-Bounds into Unmapped Silicon Address Space): When decode Pallas kernels (gdn/v3, batched_rpa) run with disable_bounds_checks=True, an corrupted block/page ID whose physical address lands in the unmapped [96, 128) GiB address hole on a 96 GiB TPU v7x die immediately triggers the fatal error and prints the exact SC3 @ 0x3 signature (reproduced deterministically in unit tests).
• Path C (Cross-DP Step-Order Deadlock + SIGTERM Cascade Halt): Under DP>1 + EP + MTP, when a single DP rank splits its batch into 2 chunks, active propose() (Step-Major) and peer ranks' run_dp_dummy_draft() (Chunk-Major) desynchronize on step order and token shapes, deadlocking cross-DP MoE reduce-scatter. After the 300s sample_tokens timeout sends SIGTERM, the surviving die on the same physical chip halts when its peer VF tears down first. Fix (#1983): remove the SMEM request-chunk split cap (62f98fb / 5575f78) and align dummy-draft step ordering with active chunks (ebe95fd / d5d5114).
🎯
2. 跨 Rank 数值确定性:reduce_scatter 非结合律偏差与零开销 Bit-Exact 修复2. Cross-Rank Numerical Determinism: reduce_scatter Non-Associativity & Zero-Cost Bit-Exact Fix
消除 8-Core 环网旋转累加导致的 ~1 ULP 跨 Rank 分裂 + Logits/Logprob FP32 对齐 Eliminating ~1 ULP ring-rotation divergence across 8 cores + FP32 Logits/Logprob
精度与确定性Determinism
• 物理根因(环网旋转累加序):在单机 8-Device(2×2×1)MoE EP Combine 阶段,XLA 默认选用单跳环形设备序 [0,1,4,5,6,7,2,3] 执行平坦 8-Core 双向环 reduce_scatter。此时第 k 个 Shard 的累加顺序按 k 旋转: shard 0 = ((((((v1+v2)+v3)+v4)+v5)+v6)+v7)+v0,而 shard 1 = ((((((v2+v3)+v4)+v5)+v6)+v7)+v0)+v1。由于 BF16 浮点加法满足交换律但不满足结合律,每步舍入差异使相同输入在不同 Rank 上产生约 1 ULP(Cross-Rank Diff ~1.56e-2,分裂为多个哈希簇),在长序列与采样下引发隐蔽精度漂移。
• 零开销 Bit-Exact 修复(重排设备可见序触发分层两两规约):无须将通信张量升精度至 FP32(避免带宽翻倍开销),只需按 Rank 将环境变量 TPU_VISIBLE_DEVICES 设为默认映射的逆置换 _INV = [0, 1, 6, 7, 2, 3, 4, 5],向 XLA 暴露自然设备序 0,1,2,...,7。XLA 编译器随即自动切换为分层成对规约策略(Hierarchical Pairwise,4 × core_count=2),令所有 Shard 共享完全一致的二叉树括号分组,实测 Cross-Rank Max Diff 归零(0.0000e+00),8 个 Rank SHA-256 哈希 100% 收敛为单一簇(BIT-EXACT)。
• 采样与 Logprob FP32 精度对齐:修复 Logits 与 Logprob 在 BF16 下计算导致的采样温度(temperature)失效与对数概率失真,全面对齐上游 vLLM 0.29.0 标准升级为 FP32 计算(0875ee7、#1745 已合入,#2024 跟进)。
• Root Cause (Ring Rotation Accumulation Order): On an 8-device single host (2×2×1), XLA's default single-hop ring order [0,1,4,5,6,7,2,3] schedules reduce_scatter as a flat 8-core bidirectional ring where shard k accumulates operands rotated by k: shard 0 = ((((((v1+v2)+v3)+v4)+v5)+v6)+v7)+v0 vs. shard 1 = ((((((v2+v3)+v4)+v5)+v6)+v7)+v0)+v1. Because BF16 addition is commutative but not associative, step-by-step rounding causes identical addends to diverge by ~1 ULP across ranks.
• Zero-Overhead Bit-Exact Fix (Inverse Device Map ➔ Hierarchical Pairwise Reduction): Instead of upcasting payloads to FP32, setting per-rank TPU_VISIBLE_DEVICES to the inverse permutation _INV = [0, 1, 6, 7, 2, 3, 4, 5] presents XLA with natural device order 0..7. XLA then selects its order-invariant hierarchical pairwise reduction (4 × core_count=2), giving every shard identical binary-tree grouping and achieving 0.0000e+00 cross-rank diff (SHA-256 BIT-EXACT across all 8 ranks) at native BF16 speed.
• FP32 Logits & Logprob Alignment: Upgraded logits and logprob computation from BF16 to FP32 (0875ee7, #1745 merged, #2024), preventing temperature scaling distortion and aligning with vLLM 0.29.0.
🧮
3. Host DRAM 内存解剖:单 Worker 生命周期 VmRSS 账本与 PjRt Heap 归因3. Host DRAM Anatomy: Per-Worker VmRSS Lifecycle & PjRt Heap Attribution
为什么增加编译档位(Compile Sizes)与 DP Worker 数会急剧吞噬上百 GB 主机内存? Why do extra compile shapes and DP workers consume over 100 GB of Host DRAM?
内存剖析Memory Profile
• 单 Worker VmRSS 生命周期实测(Decode DP8 基线): 0. 权重加载后基线 3.31 GB ➔ 1. KV Cache 编译初始化 +2.11 GB (5.47 GB) ➔ 2. Raiden KV Staging 预分配 +6.53 GB (12.16 GB) ➔ 3. 四个编译档位 [32, 64, 128, 256] 依次增加 +192 MB / +372 MB / +923 MB / +698 MB(共 +2.18 GB) ➔ 4. 编译后 Trim 释放 −650 MB,最终稳态 13.85 GB / Worker。
• 编译档位数对比(2 档 [32,64] vs. 4 档 [32,64,128,256]):单 Worker 稳态 VmRSS 由 11.98 GB 增至 13.85 GB(净增 +1.87 GB / Worker);在单机 8-Worker(DP8)下放大 8 倍即推导增加 14.96 GB,与容器级监控实测 Anon 内存增量(109.43 GB − 95.47 GB = 13.96 GB)吻合度达 93.3%。
• go tool pprof Heap 差分归因:对编译前后进程堆空间(净增 3,503 MB)做 Inuse-Space 剖析,libtpu.so 独占 +2,529.8 MB (72.21%)(对应 PjRt XLA LoadedExecutable 常驻编译产物),libpython3.12.so 占 +808.5 MB (23.08%)(Python 字节码与构图对象),二者合计占净增堆内存的 95.29%。
• 三大收敛方向与生产实效:① 清理非预期编译副本引用与连接器 leases=1 估算偏差后,超大 MoE 单机容器 DRAM 已从 440 GiB 压降至 230 GiB;② 推动跨 Worker 共享 PjRtLoadedExecutable(消除每 Worker ~2.53 GB 私有重复加载);③ 将 Raiden Staging Buffer(占基线增量 ~50.3%)改为进程间共享内存池(Shared Memory Pool),可再省下约 40 GB 主机内存供 KV Store 提升前缀命中率。
• Single-Worker VmRSS Progression (Decode DP8 Baseline): 0. Post-weights 3.31 GB ➔ 1. KV-cache init +2.11 GB (5.47 GB) ➔ 2. Raiden KV staging allocation +6.53 GB (12.16 GB) ➔ 3. Four compile sizes [32, 64, 128, 256] add +192 / +372 / +923 / +698 MB (+2.18 GB total) ➔ 4. Post-compile trim −650 MB, settling at 13.85 GB / worker.
• 2 vs. 4 Compile Sizes: Single-worker settled VmRSS rises from 11.98 GB to 13.85 GB (+1.87 GB / worker); across 8 DP workers this predicts +14.96 GB, matching pod-level Anon delta (109.43 − 95.47 = 13.96 GB) within 93.3%.
• Heap Profiling Attribution (pprof): Out of 3,503 MB net heap growth, libtpu.so accounts for +2,529.8 MB (72.21%) (resident PjRt XLA LoadedExecutable binaries) and libpython3.12.so accounts for +808.5 MB (23.08%) — together 95.29% of heap growth.
• Reduction Roadmap & Impact: (1) Fixing leaked compilation references and leases=1 estimation already cut ultra-large MoE single-host container DRAM from 440 GiB to 230 GiB; (2) sharing PjRtLoadedExecutable across workers avoids duplicating ~2.53 GB per worker; (3) pooling Raiden staging buffers in shared memory saves up to ~40 GB of Host DRAM for KV caching.
⚡
4. P/D 分离控制面演进:GrpcControlPlaneBackend vs. 裸 TCP 硬件实测4. Disaggregated Control Plane: GrpcControlPlaneBackend vs. Raw POSIX TCP
HTTP/2 长连接信道池化消除每次 D2D Read 的双临时 Socket 与三次握手抖动 Persistent HTTP/2 channel pooling eliminates 2 ephemeral sockets & 3.0 RTT per D2D Read
传输控制面Control Plane
• 架构痛点(Legacy TCP):在 KVCacheManagerWithTransfer 中,每次 D2D Read 需先后执行 SendPullRequest(启动拉流)与 SendAck(释放源端 Staging 缓冲)两次控制面 RPC。旧版 TcpControlPlaneBackend 每次 RPC 均新建并销毁临时 Socket(单次传输共 2 sockets / 3.0 RTT),在高并发下与 D2H/H2D 线程争抢内核 accept()/poll() 队列,引发显著尾延迟抖动与端口耗尽风险。
• gRPC 长连接多路复用优势:新版 GrpcControlPlaneBackend(TPU_RAIDEN_CONTROL_PLANE_BACKEND=grpc)按远端 Endpoint 缓存持久化 HTTP/2 Stub(GetOrCreateStub,热通道 0 RTT 建连开销),内置原生 RPC Deadline 传播与 10s HTTP/2 PING 保活,并与数据面并行 Socket(BlockTransport::SyncPush,P=8 下维持 24.3–24.9 GB/s)完全解耦无干扰:
• 真机三层基准实测结果:
- 小负载(2 MB,8层×4块×64KB,延迟敏感):p50 持平(2.94 ms vs. TCP 2.89 ms),但 p99 尾延迟大幅削减 35.8%(从 4.93 ms 降至 3.16 ms),平均有效吞吐提升 +7.9%(0.72 vs. 0.66 GB/s);
- 中大负载(128 MB / 512 MB,带宽主导):控制面握手耗时占比 < 0.8%(~0.3 ms / 128 ms),两端到端吞吐完全一致(512 MB P=8 下 gRPC 4.22 GB/s / 127.21 ms vs. TCP 4.19 GB/s / 128.07 ms);
- 连接数治理配套:针对 Worker 暴露连接数过多问题,新增 pooled 与 per_transfer 两种传输连接模式。
• Legacy TCP Bottleneck: Every D2D Read in KVCacheManagerWithTransfer issues two control RPCs (SendPullRequest + SendAck). TcpControlPlaneBackend opened and closed two ephemeral sockets per read (3.0 RTT setup), causing kernel accept()/poll() wakeup jitter and TIME_WAIT port churn when contending with D2H/H2D worker threads.
• Persistent HTTP/2 Multiplexing (GrpcControlPlaneBackend): Enabled via TPU_RAIDEN_CONTROL_PLANE_BACKEND=grpc, caching persistent HTTP/2 stubs per endpoint (0 RTT on warm channels) with native deadline propagation and 10s HTTP/2 PING keepalives, while leaving raw data-plane sockets (BlockTransport::SyncPush at 24.3–24.9 GB/s, P=8) completely untouched.
• Hardware Benchmark Tiers:
- Small Payloads (2 MB, Latency-Sensitive): Matches TCP at p50 (2.94 vs. 2.89 ms) while slashing p99 tail latency by 35.8% (3.16 ms vs. 4.93 ms) and lifting mean throughput by +7.9%;
- Large Payloads (128 MB / 512 MB, Bandwidth-Dominated): Handshake overhead is < 0.8% of transfer time; both achieve ~4.2 GB/s end-to-end D2D throughput at 512 MB (127.21 ms gRPC vs. 128.07 ms TCP);
- Connection Count Governance: Added pooled and per_transfer modes to cap exposed worker socket counts.
🚀 5. 生产启动加速(GCS XLA 编译缓存 & MTP 权重裁剪)、XLA 延迟调度 Hint 与双轨 Release 分支治理
• 启动时间优化(目标 ≤15 min):① MTP Weight Loader 读取范围裁剪:消除主模型与 Draft 模块的冗余权重扫描,在超大 MoE 模型上直接减少 ~10 分钟 重复读取耗时;② GCS 共享 XLA 编译缓存(VLLM_XLA_CACHE_PATH=gs://...):采用「单 Producer Job 预编译写入 GCS ➔ 多 Host LeaderWorkerSet 消费只读缓存」的解耦范式(配合 Workload Identity 绑定 roles/storage.objectUser),避免多 Pod 并发写冲突并彻底绕过冷启动重复编译。
• XLA 编译器集合通信调度微调:针对手动调整 all_gather 顺序的局限,引入 OpenXLA latency_hiding_scheduler 元数据注解 xla_metadata(scheduler_hint="force_delay_async", scheduler_delay_priority="1"),显式引导编译器推迟非关键异步通信、优先保障计算核心流水。
• 生产双轨分支收敛规范(official/release-0.0.1 vs. partner/release-0.0.1):为隔绝主干(origin/main)持续迭代引入的回归风险,在 vllm-torchtpu 与 tpu-sync 双仓建立带自动化 CI/CD 门禁的 official/release-0.0.1 官方发布分支(交付准入红线:PCP4→TP2·DP2 E2E、PCP8→TP2·DP4 E2E 及超大模型 1P1D 压测全通)。生产侧分支可在尾部先行挂载紧急热修,所有问题修复以正式合入 official/release-0.0.1 并通过 Nightly CI 为准,生产分支定期 Rebase 官方分支并剔除重复补丁,最终收敛为单一稳定基线;后续版本演进将对齐 tpu-inference 定期切分新 Release 分支的迭代模式。
• Startup Acceleration (Target ≤15 min): (1) MTP Weight Loader Scope Pruning eliminates redundant weight scans on ultra-large MoE models, saving ~10 minutes of cold-start I/O; (2) Shared XLA Compilation Cache on GCS (VLLM_XLA_CACHE_PATH=gs://...) decouples cache generation (a single Producer Job compiles and populates GCS) from consumption (multi-host LeaderWorkerSet replicas download pre-compiled artifacts and bypass XLA compilation).
• XLA Latency-Hiding Scheduler Hints: Uses xla_metadata(scheduler_hint="force_delay_async", scheduler_delay_priority="1") to declaratively control asynchronous all_gather scheduling order in XLA's latency-hiding scheduler.
• Dual-Track Release Branch Governance (official/release-0.0.1 & partner/release-0.0.1): To insulate production deployments from origin/main churn, both vllm-torchtpu and tpu-sync maintain CI/CD-gated official/release-0.0.1 branches (gated on PCP4→TP2·DP2 E2E, PCP8→TP2·DP4 E2E, and 1P1D stress tests). Production hotfixes may stage at the tip of the partner branch first, but formal delivery is defined strictly by merging into official/release-0.0.1 with passing CI, after which the partner branch rebases and drops duplicate patches to converge onto a single release baseline.
🏛️ 6. CoreML 混合 Mamba/GDN 精确页对齐数学(397B.md)、双层孪生微基准(2-Layer Twin)与 7 层编译器栈 × 6 大史诗级 PR 全景映射CoreML Exact Hybrid Mamba/GDN Page Alignment Math (397B.md), 2-Layer Twin Microbenchmark & 7-Layer Stack Mapping of the 6 Epic PRs
4 / 6 Epic PRs = 0% Pallas 2 / 6 = Surgical Pallas Kernels Step 0 Roofline: 6.73 ms Floor · 22.72 ms vs 24.10 ms (<6% Error)
📐 A. CoreML 混合 Mamba/GDN 页对齐精确内存算术(为什么 block_size: 256 ➔ 1,152 tokens?)与单机 2-Layer Twin 微基准 A. CoreML Exact Hybrid Mamba/GDN Page Alignment & Memory Math (Why block_size: 256 ➔ 1,152 tokens?) & Single-Host 2-Layer Twin Microbenchmark
• 为什么混合池化下 block_size 从 256 跃升至 1,152 tokens(tpu_block_size_utils.py:_derive_tpu_block_slot_config,mamba-cache-mode=align / TPU_VLLM_ENABLE_UNIFIED_BLOCK_POOL=1)?
  ① 单层 Mamba/GDN 原始状态字节数(mamba_raw_state_bytes = 542,720 B):在 TP=8, spec=3 下,每层每 Rank 包含 conv_state = (3 + 3) × 1,536 × 2 B (BF16) = 18,432 B(在 TP=2, spec=0 下为 3 × 6,144 × 2 B = 36,864 B,spec=1 为 4 × 6,144 × 2 B = 49,152 B)与 recurrent_state = 8 heads × 128 × 128 × 4 B (FP32) = 524,288 B(PR #1458 转为 BF16 后在 TP=2 即 16 heads/rank 下为 16 × 128 × 128 × 2 B = 524,288 B,在 TP=8 即 8 heads/rank 下为 262,144 B),基准对齐槽大小固定为 542,720 B/layer/slot。
  ② 单层 Full Attention (GQA) 每 Token 字节数(512 B/token/layer):当 TP ≥ 2 时,H_kv = 2 被切分为每 Rank attn_kv_heads_per_rank = 1 个 KV Head(配合 PR #608 SEQ_ON_LANE 消除半槽 Padding),每 Token 每层恰好占用 1 KV head × 2 (K,V) × 256 head_dim × 1 B (FP8) = 512 B/token/layer。
  ③ 向上取整至 256-Token 倍数对齐:为了让 1 个 Attention KV Page 在字节容量上完整容纳 1 个 GDN 状态槽(从而在 15 个 3×GDN + 1×Attn 混合 Block 上实现零碎片统一物理页池),最小 Token 数为 ⌈542,720 / 512⌉ = 1,060 tokens;按 TPU Page 粒度向上对齐到 256 的整数倍即得 block_size = 1,152 tokens/page(1,152 × 512 B = 589,824 B ≥ 542,720 B)!
• 2-Layer Twin Microbenchmark(1 linear_attention + 1 full_attention + MTP)如何在单台 2x2x1(4 Chips = 8 Devices)上 1:1 复刻 397B 单 Rank 行为?
  通过 hf_overrides: {"text_config": {"num_hidden_layers": 2, "layer_types": ["linear_attention", "full_attention"]}} 将 60 层(45 GDN + 15 Full Attn)缩减为 1 层 GDN + 1 层 Full Attention + 1 层 MTP,同时严格锁定单 Rank 张量维度与生产 DP4×TP2 + EP8 完全一致:q_heads_projected = 16、attn_kv_heads_per_rank = 1、gdn_k_heads_per_rank = 8、gdn_v_heads_per_rank = 32、experts_per_rank = 64(512 / EP8)、cache_block_tokens = 1,152、pages_per_rank = 4,007、kv_pool_tokens_per_rank = 4,617,186,并通过 HBM Ballast 垫片将每 Device 显存占用锁定在 76.49 GiB / 96 GiB,单步迭代仅需数毫秒即可在单台 2x2x1 上完成算子与编译标志的精确 A/B 验证。
• Why does block_size scale from 256 ➔ 1,152 tokens under Unified Hybrid Pooling (tpu_block_size_utils.py:_derive_tpu_block_slot_config, mamba-cache-mode=align / TPU_VLLM_ENABLE_UNIFIED_BLOCK_POOL=1)?
  ① Raw Mamba/GDN State per Layer (mamba_raw_state_bytes = 542,720 B): At TP=8, spec=3, each layer per rank holds conv_state = (3 + 3) × 1,536 × 2 B (BF16) = 18,432 B (or 49,152 B at TP=2, spec=1) plus recurrent_state = 524,288 B (FP32 at TP=8, or BF16 at TP=2 with 16 V-heads/rank via PR #1458; 262,144 B BF16 at TP=8), yielding a canonical alignment slot of 542,720 B/layer/slot.
  ② Full Attention (GQA) Bytes per Token per Layer (512 B/token/layer): For TP ≥ 2, each rank owns attn_kv_heads_per_rank = 1 KV head (with PR #608 SEQ_ON_LANE eliminating half-lane padding), consuming 1 KV head × 2 (K,V) × 256 head_dim × 1 B (FP8) = 512 B/token/layer.
  ③ Rounding Up to a 256-Token Multiple: For one Attention KV page to house one GDN state slot without fragmentation across all 15 hybrid (3×GDN + 1×Attn) blocks, the minimum page size is ⌈542,720 / 512⌉ = 1,060 tokens; rounding up to the next multiple of 256 yields block_size = 1,152 tokens/page (589,824 B ≥ 542,720 B)!
• How the 2-Layer Twin Microbenchmark (1 linear_attention + 1 full_attention + MTP) reproduces per-rank 397B behavior on a single 2x2x1 host (4 chips = 8 devices):
  Using hf_overrides: {"text_config": {"num_hidden_layers": 2, "layer_types": ["linear_attention", "full_attention"]}}, the twin instantiates exactly 1 GDN layer + 1 Full Attention layer + MTP with identical per-rank shapes to production DP4×TP2 + EP8: q_heads_projected=16, attn_kv_heads_per_rank=1, gdn_k_heads_per_rank=8, gdn_v_heads_per_rank=32, experts_per_rank=64, cache_block_tokens=1,152, pages_per_rank=4,007, kv_pool_tokens_per_rank=4,617,186, and HBM ballast pinned to 76.49 GiB/device.
🧬 B. 7 层 TPU 软件与编译器栈 × 6 大史诗级 vllm-torchtpu PR 架构对位(为什么 4 大突破是 0% Pallas,而 2 大突破必须手写 Pallas?) B. The 7-Layer Stack Mapping of the 6 Epic vllm-torchtpu PR Breakthroughs (Why 4 Require 0% Pallas While 2 Are Surgical Pallas Kernels)
编译器/运行时栈层级Stack Layer Pallas 占比与属性Pallas Share & Mode 核心 PR 与模块定位Key PRs & Modules 物理机制与实测端到端收益Physical Mechanism & Measured E2E Impact
Layer 1
PyTorch & Distributed
0% Pallas
Sharding / State / D2D
DP4×TP2 + EP8
PR #1458 · PR #106
PR #1183, #1558, #1677
(tpu-connector / Raiden)
同机 8 Devices (2x2x1) 解耦 Attention DP4×TP2(卡住 $H_{kv}=2$ 零复制点,芯片内 600 GB/s D2D AllReduce)与 MoE EP8(64 专家/Rank);PR #1458 将 GDN 循环状态从 FP32 降为 BF16(KV 池 +12.2%、Decode +9.1%);PR #106 统一 KV/GDN 页池;TPUConnectorHMA / Raiden 一次性跨机直传完整 Tail KV + 45 层 GDN 状态(~349 MB),实现零 Tail 重算(Zero Tail Recompute)、异构 TP 重分片与双 200G 网卡 NUMA 绑核满跑 367 Gbps。 Decouples Attention DP4×TP2 ($H_{kv}=2$ zero-duplication point, intra-chip 600 GB/s D2D AllReduce) from MoE EP8 (64 experts/rank) on the same 8 TPU7x devices; PR #1458 converts GDN state FP32➔BF16 (+12.2% KV pool, +9.1% decode); PR #106 unifies KV/GDN page pools; tpu-connector / TPUConnectorHMA / Raiden ships full tail KV + 45-layer GDN state (~349 MB) with zero tail recompute, heterogeneous resharding, and dual-200G NIC NUMA pinning at 367 Gbps.
Layer 2
Graph Capture
0% Pallas
Dynamo / AOT Export
√2 Precompile Buckets
PR #2277 (AOT Export)
按 $\sqrt{2}$ 几何级数离散化 Shape Buckets 并结合 PR #2277 Ahead-of-Time (AOT) 导出与 GCS 共享编译缓存,彻底消灭在线 Serving 期间的 JIT 重编译毛刺。 Geometric $\sqrt{2}$ precompile shape buckets paired with PR #2277 AOT Export and shared GCS compilation cache eliminate runtime JIT recompilation stalls during live serving.
Layer 3
StableHLO Views
0% Pallas
Zero-Copy IR Bitcast
PR #766
(4D Zero-Copy bitcast)
针对混合缓存池的跨视角切片,用 4D 零拷贝 bitcast 视图替代物理拷贝,将 XLA 生成的 DMA 搬运指令从 366 条骤降至 87 条,GDN 算子耗时从 4.12 ms ➔ 0.774 ms(5.3× 加速)。 Replaces physical tensor copies with a 4D zero-copy bitcast view in StableHLO, slashing DMA instructions from 366 ➔ 87 and accelerating GDN by 5.3× (4.12 ➔ 0.774 ms).
Layer 4
XLA [8, 128] Tile Layout
Layout Co-Design
32-bit Lane Packing
PR #608
(SEQ_ON_LANE)
将 TPU [8, 128] 物理 Tile 的 32-bit 槽位轴从 head 旋转至 sequence,使 4 个连续 FP8 Token 紧密打包进同 1 个 32-bit 物理槽,消灭 TP≥2 单 KV Head 下的 50% 0-Padding,单页步进从 1,024 B 减半至 512 B/token/layer(直接支撑 1,152 页对齐)。 Rotates the 32-bit slot axis of the TPU [8, 128] tile from head to sequence so 4 FP8 tokens pack into 1×32-bit slot, halving KV page stride from 1,024 ➔ 512 B/token/layer at TP≥2.
Layer 5 (Left)
Kernels: XLA Fusions
0% Pallas
Native MXU Matmul
PR #371 / #509
(F.one_hot + MXU)
在小 Batch Decode 下主动绕开 Pallas GMM,仅用 3 行纯 PyTorch F.one_hot + 矩阵乘交给 XLA 融合直推 256×256 MXU,在 c64 并发下反超手写 Kernel +14.5% 端到端吞吐。 Bypasses Pallas on small decode batches with 3 lines of pure PyTorch F.one_hot + matmul lowered by XLA directly onto the 256×256 MXU, gaining +14.5% E2E throughput at c64.
Layer 5 (Right)
Kernels: Surgical Pallas
Surgical Pallas
Async DMA + VMEM Alias
PR #543 + #891 (Fused MoE)
PR #1008 (RPA v3 MTP)
PR #2276–#2361 + #2227
① PR #543 + #891 全层 SPMD Fused MoE:利用双缓冲异步 DMA 彻底掩盖 1,190 µs(占全步 22%)的裸露 ICI Reduce-Scatter,单层加速 4.8×、E2E 吞吐翻倍(10,614 ➔ 21,297 tok/s);② PR #1008 RPA v3:MTP 树验证耗时 10.2 ➔ 5.6 ms;③ PR #2276–#2361 声明式 Pallas Megakernel + pool_alias (#2227):跨算子分时复用 VMEM scratchpad。 ① PR #543 + #891 Whole-Layer SPMD Fused MoE: Double-buffered async DMA hides 1,190 µs (22% exposed ICI Reduce-Scatter), yielding 4.8× layer speedup and 2× E2E throughput (10,614 ➔ 21,297 tok/s); ② PR #1008 RPA v3: MTP verify 10.2 ➔ 5.6 ms; ③ PR #2276–#2361 Declarative Pallas Megakernel + pool_alias (#2227) VMEM time-sharing.
Layer 6
ICI Collectives
SPMD Collective
Sequence ↔ Head Switch
PR #65 / #129
(PCP=8 All-to-All)
Prefill Context Parallelism (PCP=8) 通过 ICI All-to-All 在序列切分与 Head 切分间动态切换,将 128K 超长上下文 TTFT 从 32s 骤降至 4.62s(6.9× 加速)。 PCP=8 dynamic sequence-to-head switch via ICI All-to-All slashes 128K long-context TTFT from 32s ➔ 4.62s (6.9× speedup).
Layer 7
PJRT Runtime & Host OS
0% Pallas
Runtime / THP / DRAM
PR #1810 (Peer Replay)
PR #2141 / #2068
(Host-RAM & THP Release)
PR #1810 跨 Rank 编译产物对等重放(Cross-Rank Peer Replay)将 DP=8 启动时间从 741s 砍至 270s;PR #2141 / #2068 权重加载期临时内存释放与 Linux THP 治理将单机峰值 Host DRAM 从 440 GiB 压降至 230 GiB。 PR #1810 Cross-Rank Peer Replay cuts DP=8 startup from 741s ➔ 270s; PR #2141 / #2068 startup Host-RAM & Linux THP release cuts peak Host DRAM from 440 ➔ 230 GiB.
🎯 C. Step 0 第一性原理 Roofline 闭环验证:6.73 ms HBM 物理极限下界 → 22.72 ms 解析预测 vs. 24.10 ms 9,216 芯片生产实测(误差 <6%) C. Step 0 First-Principles Roofline Closure: 6.73 ms HBM Weight-Read Floor → 22.72 ms Analytic Prediction vs. 24.10 ms 9,216-Chip Production Canary (<6% Error)
• 绝对 HBM 权重读取下界(6.73 ms):在单台 2x2x1(4 颗 TPU7x 芯片 = 8 Devices,聚合有效 HBM 带宽 8 × 7,180 GB/s = 57.44 TB/s)上,当 MoE 激活专家趋近全激活(512 专家全读 387.17 GB FP8 权重,单步总活跃权重 48.62 GB/dev)时,仅从 HBM 搬运权重的物理极限时间即为 6.73 ms。
• DP4×TP2 + EP8 生产工作点闭环(误差仅 5.7%):计入每序列 166.8 MB 状态流式读取、芯片内 600 GB/s D2D TP=2 AllReduce 与 EP=8 Dispatch/Combine 集合通信后,Step 0 解析 Roofline 预测单步 Decode 耗时为 22.72 ms;而 9,216 颗芯片(1,152 台主机)生产集群灰度实测中位数 TPOT 为 24.10 ms(1,083.9 tok/s),理论模型与超大规模生产实测偏差 < 6%(5.7%),彻底闭环了「先用 Step 0 数学锁死物理下界与最优切分,再按 7 层栈精准下刀」的工程方法论!
• Absolute HBM Weight-Read Floor (6.73 ms): On a single 2x2x1 host (4 TPU7x chips = 8 devices, aggregate effective HBM bandwidth 8 × 7,180 GB/s = 57.44 TB/s), reading the active FP8 weights (48.62 GB/dev at full expert activation) sets a hard physical weight-read floor of 6.73 ms.
• DP4×TP2 + EP8 Production Canary Closure (5.7% Error): Adding per-sequence KV+GDN state streaming (166.8 MB/seq), intra-chip 600 GB/s D2D TP=2 AllReduce, and EP=8 All-to-All collectives yields a Step 0 analytic prediction of 22.72 ms per decode step — matching the 24.10 ms median TPOT measured on the 9,216-chip production canary within < 6% (5.7%)!
💡 Qwen 3.5 config.json 参数矩阵与「人生哲学」隐喻指南 Qwen 3.5 config.json parameter matrix, read as a set of life metaphors 对齐本文档主口径Aligned to the main configuration
超参数不只是冰冷的数字,它们其实描述了一种心智模型与处世态度。
下面按四大领域(基础骨干、注意力与记忆、细粒度 MoE、动态采样解码)逐项列出:每个参数的技术物理实质放在下半张卡,人生隐喻放在上半张卡,两者一一对应。
Hyperparameters are not just cold numbers — together they describe a way of thinking and of meeting the world.
Four domains follow (backbone, attention and memory, fine-grained MoE, sampling and decoding). On each card the engineering reality sits at the bottom and the life metaphor at the top, one mapped onto the other.
⚠ 口径说明⚠ Which configuration these numbers describe
⚠ 已按联合优化文档修正架构口径:实际部署的 Qwen 3.5 是混合架构 —— 约 15 层 full-attention (GQA) + 45 层 GDN 线性注意力,共 60 层(文档 prefill 分层实测:full 492 µs × 15 / linear 774 µs × 45 / MoE 1.41 ms × 60),GDN 并非某条实验分支上的可选优化,而是模型本身的组成部分。 其余架构参数为 96 Q 头 / 32 KV 头 (3:1) · head_dim 128 · hidden_size 4096 · vocab 248,320 · 512 专家 Top-8。 KV 只由 FA 层产生:15 × 32 × 128 × 2 × 131072 = 15 GiB/请求 (FP8),摊到 8 卡即单卡 1.875 GiB(实测记作 1.92 GB)。若误按 60 层甚至 80 层全部计 KV,会高估 4 ~ 5.3 倍。 采样段(temperature / top_p / top_k / presence_penalty / context_window / speculative_tokens)取自 generation_config.json 默认值,可拖动滑块实时体会。 ⚠ Architecture corrected against the joint-optimization record: the Qwen 3.5 actually deployed is a hybrid — roughly 15 full-attention (GQA) layers plus 45 GDN linear-attention layers, 60 in total (the record's measured prefill breakdown: full 492 µs × 15, linear 774 µs × 45, MoE 1.41 ms × 60). GDN is not an optional optimization on some experimental branch; it is part of the model. The remaining architecture figures are 96 Q heads / 32 KV heads (3:1) · head_dim 128 · hidden_size 4096 · vocab 248,320 · 512 experts, Top-8. Only the FA layers produce KV: 15 × 32 × 128 × 2 × 131072 = 15 GiB per request in FP8, or 1.875 GiB per chip across 8 chips (recorded as 1.92 GB). Counting all 60 — let alone all 80 — layers overstates KV by 4× to 5.3×. The sampling section (temperature / top_p / top_k / presence_penalty / context_window / speculative_tokens) comes from generation_config.json defaults; drag the sliders to feel each one.
🏗️ 第一领域:模型骨干与容量底座 🏗️ Domain 1 — Backbone and capacity —— 先天的身躯骨架、认知通道宽度与思维反刍深度 — the frame you were born with: how wide your channels are, how many times you turn a thought over
hidden_size (d_model)
残差流主干道向量维度Width of the residual-stream highway
4096
🧘 人生隐喻:心胸宽广与思维气量🧘 Metaphor: how much you can hold at once
主干道的宽度就是你的气量。道越宽(4096),能并行容纳的思维分支越多——理性、感性、道德与直觉并存不悖;气量窄(比如 1024),再宏大的思考塞进来也只会撞成一场思想上的交通堵塞。 The width of the highway is how much you can carry at once. Wide (4096) means many lines of thought run side by side — reason, feeling, conscience and instinct all fit. Narrow (say 1024) and however grand the thought, it arrives as a traffic jam.
⚙️ 底层原理:Reality: 决定 embedding 与每层输入输出张量的尺寸 [B, T, 4096]。4096 = 32 × 128,是 MXU 通道宽度的整数倍,GEMM 收缩维无 padding 损耗。 Sets the embedding and per-layer tensor shape [B, T, 4096]. 4096 = 32 × 128 is an exact multiple of the MXU lane width, so the GEMM contraction dimension wastes no padding.
num_hidden_layers
Transformer Block 堆叠总层数 · 15 层 FA + 45 层 GDNStacked transformer blocks · 15 FA + 45 GDN
60
📜 人生隐喻:阅历的沉淀与反刍📜 Metaphor: turning experience over
60 层意味着一句话要经历 60 轮抽象与再抽象。只有几层的网络像涉世未深的孩子,只看得见字面;反复萃取几十轮的大脑,才能穿过表象直击本质。当然,每一次深思都要花时间——这正是深度与延迟之间那笔永恒的交易。 Sixty layers means every sentence is abstracted and re-abstracted sixty times. A shallow network is a child who reads only the literal words; dozens of rounds of distillation is what it takes to see past the surface. Each round costs time, of course — that is the standing trade between depth and latency.
⚙️ 底层原理:Reality: 实际部署的是 60 层混合架构:15 层 full-attention (GQA) + 45 层 GDN 线性注意力,每层各自执行 RMSNorm、QK-Norm、注意力、残差与 MoE 路由。只有 FA 层产生 KV Cache:15 × 32 KV 头 × 128 × 2(K,V) × 131072 ≈ 15 GiB / 请求 (FP8)。若误把 60 层全部计入则为 60 GiB,按 80 层全 GQA 计则为 80 GiB —— 都是高估。 What is deployed is a 60-layer hybrid: 15 full-attention (GQA) layers plus 45 GDN linear-attention layers, each running RMSNorm, QK-Norm, attention, the residual add and MoE routing. Only the FA layers produce KV cache: 15 × 32 KV heads × 128 × 2 (K,V) × 131072 ≈ 15 GiB per request in FP8. Counting all 60 layers would give 60 GiB, and all 80 as GQA would give 80 GiB — both overstate it.
vocab_size
多语言与代码词表总规模Multilingual + code vocabulary
248,320
🌍 人生隐喻:表达与涉猎的广度🌍 Metaphor: the breadth of what you can name
词表是你一生中能辨识并说出口的概念总和。24.8 万个 token,是数百种语言、各类代码、数学符号与艺术隐喻的底气。见过星辰大海的人,面对再复杂的问题也总能找到那个恰好的词。 The vocabulary is the sum of everything you can recognize and put into words. 248,320 tokens covers hundreds of languages, code, mathematical notation and figurative language. Someone who has seen that much can usually find the exact word for a hard thing.
⚙️ 底层原理:Reality: embedding 矩阵与 LM Head 的分类维度,形状 [248320, 4096],BF16 下单个矩阵即 1.89 GiB。LM Head 的 fusion.988 就是在这个维度上做最终投影。 The classification dimension of both the embedding matrix and the LM head, shaped [248320, 4096] — 1.89 GiB per matrix in BF16. The LM head's fusion.988 projects onto exactly this axis.
tie_word_embeddings
输入嵌入与输出投影是否共享权重Whether input embedding and output projection share weights
false
🌱 人生隐喻:不忘初心 vs 游刃有余🌱 Metaphor: staying true vs. growing into range
绑定(true)意味着「我出发时理解的词义,和我最终说出口的判断,必须锁在同一个模具里」——质朴、省事。解耦(false)则允许「听得懂」和「说得出」各自长成不同的本事:既有聆听的敏锐,也有表达的练达。 Tying (true) says "what a word meant to me on the way in must be the same thing I mean on the way out" — simple and frugal. Untying (false) lets listening and speaking grow into separate skills: one sharp at taking things in, the other fluent at giving them back.
⚙️ 底层原理:Reality: 控制是否复用 embed_tokens.weight 作为 lm_head.weight。解耦多占 1.89 GiB 权重,换来更高的表征自由度与生成准确率。 Controls whether embed_tokens.weight is reused as lm_head.weight. Untying costs an extra 1.89 GiB of weights and buys representational freedom and generation accuracy.
rms_norm_eps
均方根归一化的数值平滑量Numerical floor inside RMSNorm
1e-06
⚖️ 人生隐喻:为人处世的容错缓冲⚖️ Metaphor: the slack you keep for yourself
防止除零崩溃的那个极小常数 ε,就像做人必须留的那 0.000001 的弹性底线。若毫无容错(ε = 0),一旦遇上归零式的打击,心态会瞬间炸成无穷大或 NaN。 The tiny constant that keeps a division from blowing up is the 0.000001 of slack a person has to keep. With none of it (ε = 0), the first blow that takes you to zero sends everything to infinity or NaN at once.
⚙️ 底层原理:Reality: y = x / sqrt(mean(x²) + ε) ⊙ γ。该常数保证激活在整条浮点链路上不会除以零,杜绝 NaN 传播。Qwen 还在 Q/K 上各加了一次 QK-Norm,用同一个 ε。 y = x / sqrt(mean(x²) + ε) ⊙ γ. The constant keeps activations from dividing by zero anywhere along the float path, so no NaN propagates. Qwen additionally applies QK-Norm to Q and K, using the same ε.
⚡ 第二领域:注意力与长距记忆 ⚡ Domain 2 — Attention and long-range memory —— 如何分配精力:哪些是过眼云烟,哪些值得刻在记忆里 — how attention gets rationed: what passes through, and what gets kept
num_attention_heads
Query 观察头数Query heads
96
👁️ 人生隐喻:看待问题的多维视角👁️ Metaphor: how many angles you see from
96 个 Query 头,意味着同时从 96 个切面审视眼前这件事:逻辑的拆解、共情的体悟、道德的审视、利弊的权衡。视角单一的人容易钻牛角尖,多头并进才看得清全貌。 Ninety-six query heads means looking at the same moment from ninety-six angles at once: the logical reading, the empathetic one, the ethical one, the practical one. One angle is how you end up stuck in a corner; many is how the whole shape becomes visible.
⚙️ 底层原理:Reality: Q 投影把 4096 维升到 96 × 128 = 12288 列;本段 trace(TP=8 变体)每卡拿到 12 个 Q 头(slice.7534 取 [0:1536]);上线的 PCP=8 形态下注意力权重是复制的,每卡仍持 96 头、只切序列;Decode TP=2 时每卡 48 个头。 The Q projection lifts 4096 up to 96 × 128 = 12288 columns. In this trace (the TP=8 variant) each chip holds 12 Q heads (slice.7534 takes [0:1536]); in the deployed PCP=8 form attention weights are replicated, so every chip keeps all 96 heads and only the sequence is split. At decode TP=2 each chip holds 48.
num_key_value_heads
GQA 分组键值头数(3:1)GQA key/value heads (3:1)
32
🗂️ 人生隐喻:抓住主干的记忆提炼术🗂️ Metaphor: keeping only the index, not the archive
审视的视角有 96 个,真正刻进记忆的钥匙只有 32 把——每 3 个 Query 头共用一组 Key/Value。人不需要给每一个念头都单独立档备份;懂得提纲挈领,记忆的负担立刻轻下来一大半。 Ninety-six angles of looking, but only thirty-two keys actually filed away — every three query heads share one key/value group. Not every passing thought needs its own archive; keeping just the index is what makes a long memory affordable.
⚙️ 底层原理:Reality: Grouped-Query Attention 3:1 分组。相比 96 头 MHA,KV Cache 直接降到 1/3;128K 请求下 15 层 FA 的 FP8 KV 仍需 约 15 GiB(单卡 1.875 GiB × 8),这也是必须做 PD 分离与 KV 卸载的根本原因。 Grouped-query attention at 3:1. Against full 96-head MHA the KV cache drops to one third — yet a 128K request still needs about 15 GiB in FP8 across the 15 FA layers (1.875 GiB per chip × 8), which is precisely why PD disaggregation and KV offload exist at all.
head_dim
单头特征向量深度Per-head feature depth
128
🎯 人生隐喻:每个视角能看多深🎯 Metaphor: how deep each angle goes
头数决定看问题的角度多少,head_dim 决定每个角度能看多深。128 维是个务实的折中:够承载一个语义子空间,又不至于让注意力矩阵的开销失控。广度与深度之间,总得有人先让一步。 Head count decides how many angles you have; head_dim decides how far each one sees. 128 is the pragmatic middle — enough to carry a semantic subspace, not so much that the attention matrices become unaffordable. Breadth and depth always negotiate.
⚙️ 底层原理:Reality: 注意力核心算 Softmax(Q·Kᵀ / √128)·V。注意这里有真实损耗:head_dim = 128 只填满 TPU v7 那 256 宽 MXU 的一半,Q·Kᵀ 这一步的形状对齐并不完美——它是本文档里少数几处硬件利用率没跑满的地方。 The attention core computes Softmax(Q·Kᵀ / √128)·V. There is a genuine cost here: head_dim = 128 fills only half of the v7's 256-wide MXU, so the Q·Kᵀ step is not perfectly shape-aligned — one of the few places in this document where the hardware is not fully fed.
GDN 线性循环层占比GDN linear-recurrent layer share
门控 Delta 网络层数分布 · 即为实际部署架构 Gated delta net distribution · this is the deployed architecture
45 / 60 (75%)
🍃 人生隐喻:断舍离的智慧🍃 Metaphor: the wisdom of letting go
混合架构里最有想法的一条哲学:75% 的层都是线性循环层。琐事如过眼云烟,状态随时间恒定更新、零写回、不堆积。学会及时放下不重要的枝节,哪怕读到第 128K 个字,脚步依然轻快。 The most opinionated idea in the hybrid build: three quarters of the layers are linear-recurrent. Small things pass through; the state updates in constant space, never written back, never piling up. Let the unimportant go in time and you are still walking lightly at token 128,000.
⚙️ 底层原理:Reality: 45 层 Gated Delta Net 维护 O(1) 恒定状态,长文本不产生膨胀的 KV Cache。FP8 把 128×128 状态压到 16 KB 后完全锁在 VMEM 内,实测该算子 4,120 µs → 840 µs(4.9×)。GDN 是模型本身的组成部分——实际部署的 60 层中有 45 层是 GDN,只有 15 层 full-attention 产生 KV Cache。 Forty-five gated-delta-net layers hold an O(1) state, so long inputs never inflate a KV cache. FP8 shrinks the 128×128 state to 16 KB, small enough to pin in VMEM; measured, the operator goes 4,120 µs → 840 µs (4.9×). GDN is part of the model itself — 45 of the 60 deployed layers are GDN, and only the 15 full-attention layers produce KV cache.
rope_theta
旋转位置编码基频Rotary position encoding base frequency
1,000,000
🌌 人生隐喻:看透沧海桑田的岁月感🌌 Metaphor: a sense of time long enough to hold the whole story
基频高达一百万,如同拥有看穿漫长周期的眼力。哪怕是几万字之前一句极轻的伏笔,经过百万周期的相位沉淀,此刻依然能被精准地共鸣到。 A base of a million is the kind of long sight that takes in an entire arc. A faint line dropped tens of thousands of tokens ago still resonates precisely now, because a million cycles is slow enough that its phase has not blurred.
⚙️ 底层原理:Reality: θ = 10⁶ 使高频与低频分量在 128K~256K 序列下都不发生相位混叠,长文外推能力显著优于 θ = 10⁴。RoPE 在 Q-Norm / K-Norm 之后、注意力之前逐头施加。 θ = 10⁶ keeps high- and low-frequency components from aliasing across 128K–256K sequences, extrapolating far better than θ = 10⁴. RoPE is applied per head after Q-Norm / K-Norm and before attention.
🔀 第三领域:细粒度 MoE 与社会分工 🔀 Domain 3 — Fine-grained MoE as a division of labour —— 没人能成为全才,但群体智慧可以被动态协同放大 — nobody knows everything, but a crowd routed well can
num_experts (n_routed)
细粒度路由专家总数Total fine-grained routed experts
512
🏛️ 人生隐喻:百花齐放的专业社会🏛️ Metaphor: a society of specialists
模型内部住着 512 位各有所长的专家:有专修 bug 的工匠,有吟诗作对的文人,有钻研数学的精算师,有洞察人性的旁观者。分工越细,每个人越能在自己的那一小块地里做到极致。 Inside the model live 512 specialists: one fixes bugs, one writes verse, one does the arithmetic, one reads people. The finer the division, the further each can go in their own small patch of ground.
⚙️ 底层原理:Reality: 512 个前馈网络并发待命,总参数 397B。EP=8 下每卡常驻 64 个专家,路由通过 all-to-all.18/19 派发与回收。 512 feed-forward networks stand by, 397B parameters in total. At EP=8 each chip hosts 64 experts, and routing dispatches and collects through all-to-all.18/19.
top_k_experts
单 Token 激活的专家数Experts activated per token
8
🤝 人生隐喻:知人善任与精简决策🤝 Metaphor: calling the right few, not everyone
遇到具体问题时,明白人不会把 512 个人都叫来开会,只精准请来最在行的 8 位。会开得小,事办得快,力气也没白费。 Faced with an actual problem, the sensible move is not to summon all 512 — just the eight who know the subject. Small meeting, quick decision, no effort wasted.
⚙️ 底层原理:Reality: 路由网络 fusion.815 对 512 个专家打分并取 Top-8,再经 All-to-All 派发。专家输出回到宿主卡后由 owner_output 分支原地加权聚合,不再申请临时显存。 The router fusion.815 scores all 512 experts, takes the top 8, and dispatches via All-to-All. Results return to the owner chip, where the owner_output branch combines them in place with no staging allocation.
n_shared_experts
所有 Token 必经的共享专家Shared expert every token passes through
1
🧱 人生隐喻:共同的底色与常识🧱 Metaphor: the common ground underneath
无论一个 token 被派到多偏门的专家那里,它都得先走一遍这位共享专家。这就像一个群体共同持有的常识与默契:没有这层托底,社会会因为过度碎片化而失去彼此理解的能力。 However specialized the expert a token gets routed to, it passes through the shared one first. That is the common sense a group holds jointly — without it, specialization fragments into people who can no longer understand each other.
⚙️ 底层原理:Reality: 所有 token 必经的基础 FFN,d_ff = 5632(是单个路由专家 1408 的 4 倍),负责吸收最高频的通用语言与语法逻辑,避免动态门控出现常识盲区。 A base FFN on every token's path with d_ff = 5632 — four times a routed expert's 1408 — absorbing the highest-frequency grammar and general language so the dynamic gate never has a blind spot on the obvious.
moe_intermediate_size
单个路由专家的中间维度Hidden width of one routed expert
1408
🛠️ 人生隐喻:一技之长的工匠精神🛠️ Metaphor: one metre wide, a thousand metres deep
专家不需要臃肿,1408 维刚好。与其样样通样样松,不如在属于自己那一米宽的地方,挖到一千米深。 A specialist does not need to be large; 1408 is enough. Better to dig a thousand metres down in your own metre of ground than to skim everywhere.
⚙️ 底层原理:Reality: SwiGLU 结构下每个专家有 gate / up / down 三个矩阵:[4096, 1408] ×2 与 [1408, 4096]。维度小意味着 All-to-All 搬运的张量体积小,EP8 通信可压在毫秒级以内。 Under SwiGLU each expert carries gate / up / down matrices: [4096, 1408] ×2 and [1408, 4096]. Keeping the width small keeps the All-to-All payload small, which is what holds EP8 communication under a millisecond.
激活稀疏度Activation sparsity
单次前向真正动用的比例Fraction of the model a single token wakes
8/512 = 1.56%
🧘‍♂️ 人生隐喻:大智若愚与省力之道🧘‍♂️ Metaphor: quiet on the surface, exact when it moves
每个 token 只唤醒 1.56% 的专家。 从外面看,大半个脑子仿佛都在打盹;但这恰恰是高手的状态——平时深藏不露,出手时一击中的,一分力气都不浪费。 Each token wakes just 1.56% of the experts. From outside, most of the mind looks asleep — which is exactly the state of someone good at this: still by default, and when it moves, it moves once and precisely.
⚙️ 底层原理:Reality: 路由层面 8 / 512 = 1.56%。但注意力、共享专家与 LM Head 是稠密的,所以按参数算是 17B 激活 / 397B 总量 ≈ 4.3%。两个数字分别回答「专家稀疏度」与「每 token 实际算力」,不要混用。 At the routing layer, 8 / 512 = 1.56%. Attention, the shared expert and the LM head are dense, though, so by parameter count it is 17B active out of 397B ≈ 4.3%. The two numbers answer different questions — expert sparsity versus actual work per token — so don't substitute one for the other.
🎲 第四领域:动态生成与处世态度 🎲 Domain 4 — Sampling, or how you decide —— 面对不确定的世界,一个人如何决断、取舍与表达(可拖动) — facing an uncertain world: how you choose, what you drop, how you say it (draggable)
temperature
采样随机性与发散倾向Sampling randomness
创造力开关Creativity dial
当前设定值:Current value: 0.70
🔥 人生隐喻:热忱的体温🔥 Metaphor: your running temperature
温度是你对待世界的发散性与童心。温度适中时你风趣而有主见;过低如同一块冰冷的钟表,分毫不差却也毫无生气;过高则像发着高烧,妙语连珠的背后其实是神志不清。 Temperature is how loosely you hold the world. In the middle you are witty and still your own person; too low and you are an accurate clock with nothing alive in it; too high and the brilliant phrases are really just fever.
0.0 — 谨小慎微的老会计,每句话都绝对确定,毫无意外。 a careful old accountant: every sentence certain, nothing unexpected.
2.0 — 喝了三瓶威士忌的诗人,满纸荒唐言。 a poet three bottles in: every page inspired, none of it usable.
⚙️ 底层原理:Reality: 作用在 logits 的除数上:P(wᵢ) = exp(zᵢ/T) / Σ exp(zⱼ/T)。T → 0 坍缩为贪婪解码(argmax),T → ∞ 退化为均匀随机。 Divides the logits: P(wᵢ) = exp(zᵢ/T) / Σ exp(zⱼ/T). As T → 0 it collapses to greedy decoding (argmax); as T → ∞ it degenerates to a uniform draw.
top_p (nucleus)
累积置信度截断Cumulative-probability cutoff
格局与视野Breadth of ear
当前设定值:Current value: 0.90
🔭 人生隐喻:愿意听多少种声音🔭 Metaphor: how many voices you let in
Top-P 是你做决定时愿意听进多少比例的声音。听排名前 90% 的主流意见,既博采众长,又自动滤掉了街角疯子的胡话;太窄(0.2)是一意孤行;完全不设防(1.0)则会被噪音淹没。 Top-p is what share of the room you are willing to hear before deciding. The top 90% takes in the sensible range and quietly filters out the person shouting on the corner; too narrow (0.2) is just your own mind talking back; wide open (1.0) is drowning in noise.
0.1 — 只听最主流的一两句,拒绝一切非主流见解。 only the one or two loudest opinions get through.
1.0 — 词表里概率亿分之一的偏门词也获得入选资格。 even a token with one-in-a-hundred-million odds is eligible.
⚙️ 底层原理:Reality: 按概率降序排列,取累积和刚好达到 p 的最小候选集(核采样),尾部长尾分布物理归零后再重新归一化。 Sorts tokens by probability, keeps the smallest set whose cumulative mass reaches p (nucleus sampling), zeroes the tail outright and renormalizes.
top_k
固定候选数上限Hard cap on candidates
智囊团席位Seats at the table
当前设定值:Current value: 40
🤝 人生隐喻:核心智囊团的人数🤝 Metaphor: how many advisers you keep
Top-K 是你做重大决定前最多咨询几位顾问。K=1 是孤家寡人,只听第一心腹;K=40 是一个精干的董事会;K=248320 则是把整个菜市场都请来做民意调查,场面通常收不住。 Top-k is how many advisers you consult before a real decision. K=1 is one confidant and no one else; K=40 is a working board; K=248,320 is polling the entire marketplace, which rarely ends in a decision at all.
1 — 永远走最保险的第一条路,单调刻板。 always the safest first path; predictable to a fault.
100+ — 候选极多,语言多姿多彩,跑题风险也随之上升。 many candidates, colourful language, and a rising risk of wandering off.
⚙️ 底层原理:Reality: 只保留概率最大的 K 个 logits,其余截断。通常与 Top-P 配合:K 设上限防止长尾幻觉,P 负责按分布动态收紧。 Keeps only the K highest logits and truncates the rest. Usually paired with top-p: K sets a hard ceiling against long-tail hallucination, p tightens dynamically with the distribution.
presence_penalty
重复用词的惩罚强度Penalty on words already used
喜新厌旧度Appetite for the new
当前设定值:Current value: 0.00
🧭 人生隐喻:走出舒适区的魄力🧭 Metaphor: the nerve to change the subject
Presence penalty 是你开辟新话题的魄力。正值高时,你像个拒绝把同一句话说第二遍的人,逼着自己往新领域走;负值则是沉在旧事里的固执,翻来覆去讲同一段往事。 Presence penalty is your nerve for opening a new subject. High and positive, you are someone who refuses to say the same sentence twice and keeps pushing into unfamiliar ground; negative, you are settled in old stories, telling the same one again.
-1.5 — 疯狂复读同一句话,陷入死循环。 repeats the same line until it becomes a loop.
+1.5 — 极度抗拒任何重复,为换话题不惜生造生僻概念。 so allergic to repetition it invents concepts to avoid it.
⚙️ 底层原理:Reality: 只要某 token 在已生成文本中出现过至少一次,就给它的 logit 减去固定惩罚:logit − c × p(与出现次数无关,这点区别于 frequency_penalty)。 If a token has appeared even once in the generated text, a fixed penalty is subtracted from its logit: logit − c × p. Unlike frequency_penalty, the count does not matter — only presence.
context_window
上下文跨度与 KV 驻留Context span and resident KV
记忆与胸襟Memory
当前设定值:Current value: 131072 (128K)
📜 人生隐喻:长情记忆与底蕴📜 Metaphor: how far back you still remember
上下文窗口是你的记忆深度。只记得住 4K 的人聊两句就忘了初衷;能装下 128K 的人,读完整本《战争与和平》还记得第一章那个路人留下的伏笔,并能从漫长的篇幅里把线索一根根抽出来。 The context window is how deep your memory runs. With 4K you lose the thread a few exchanges in; with 128K you finish War and Peace still holding the detail a minor character left in chapter one, and can pull the thread back out of all that length.
4K — 只能就事论事,前文稍复杂便顾此失彼。 lives entirely in the present turn; any complexity earlier is already gone.
128K — 吞下整座图书馆的底蕴,代价是约 15 GiB (FP8) 的 KV 驻留。 holds a library's worth of context, at a cost of about 15 GiB of resident KV in FP8.
⚙️ 底层原理:Reality: 决定注意力矩阵 O(N²) 的上限。128K 经 PCP=8 切成 8 个 16K chunk 分布在 8 张卡上;单卡 KV 实测 1.92 GB,整请求 8 卡合计 约 15 GiB (FP8)——两者是同一笔账的单卡视角与全局视角。 Sets the ceiling on the O(N²) attention matrix. At PCP=8 a 128K sequence is cut into eight 16K chunks across eight chips; measured per-chip KV is 1.92 GB and the whole request comes to about 15 GiB in FP8 across the 8 chips — the same figure viewed per-chip and globally.
speculative_tokens (MTP)
多 Token 投机步长Multi-token speculation depth
前瞻远见Foresight
当前设定值:Current value: 2 Tokens
♟️ 人生隐喻:下棋能看几步♟️ Metaphor: how many moves ahead you see
MTP 步长就是你行事时往前预判几步。走一步看一步的人慢(传统自回归 22.45 ms/词);能稳稳看透后两步的人,一手落子两处,效率直接翻倍(11.20 ms/词)。但若非要盲目看透十步,反复纠偏验证的代价反而会把收益吃掉。 The MTP depth is how far ahead you commit. Move-by-move is slow (22.45 ms per token in plain autoregression); seeing the next two reliably places two stones at once and halves the cost (11.20 ms per token). Insisting on ten, though, spends more on checking and correcting than the foresight is worth.
1 — 绝不抢跑,极其稳健,但每一步都要付全额延迟。 never gets ahead of itself — and pays the full latency on every step.
2~3 — 实测甜点位:单步命中 2 词,接受率与加速比同时最优。 the measured sweet spot: 2 tokens accepted per step, best acceptance rate and speedup together.
⚙️ 底层原理:Reality: 轻量投机头一次前瞻 k 个候选,主模型单次并行验证(fusion.990 / RPA verify 特化)。命中即一次权重读出产出多词,实测吞吐 5,650 tok/s。 A lightweight draft head proposes k candidates at once and the main model verifies them in a single parallel pass (fusion.990, the specialized RPA verify). On a hit, one weight read yields several tokens — measured at 5,650 tok/s.
🌟 调参的终极哲学:中庸之道 🌟 The point of all this tuning: there is no universally right setting
从来不存在一组永远正确的「死参数」。《中庸》讲:「喜怒哀乐之未发,谓之中;发而皆中节,谓之和。」
需要严谨推导代码与定理时,就收敛心性——低 temperature、精准的 Top-K 门控、纯粹的 GQA 检索;需要写诗探索未知时,就放开一点——高 temperature、适度惩罚、更广的采样空间。把所有超参数读懂之后,读到的其实是同一件事:在确定与开放之间,永远要重新找一次平衡。
No fixed set of values is right for everything. The Doctrine of the Mean puts it this way: before feeling stirs, that is centre; when it stirs and stays in measure, that is harmony.
Deriving a proof or debugging code, pull inward — low temperature, tight top-k gating, straightforward GQA retrieval. Writing a poem or exploring something unfamiliar, let it out — higher temperature, a little penalty, a wider sampling space. Read all the hyperparameters and they turn out to say one thing: the balance between certainty and openness has to be found again for every task.
⚖️ 权衡与妥协对账(按「三个旋钮 / 三个房间」框架) Trade-off and compromise ledger (framed by "three knobs / three rooms")
本页用一套外部课程讲义的分析框架,回头审这份部署记录:一个 query 只做三步,所以只有三个旋钮—— ① 每个 token 存多少(MQA→GQA→MLA)、② 每个 query 看多少(SWA/NSA/DSA 稀疏)、③ 换一套数学(DeltaNet→GDN 线性)。 而注意力的变体史是在三种资源之间反复搬家:显存 → 算力 → 访存规整度,顺序不是随机的——先搬能拿尺子量的,最后才搬量不出来的。
结论先放这里:这个部署在旋钮① 上保守、在③ 上激进、在② 上主动弃权; 它几乎全部工程预算都投在第三个房间(访存规整度),拿到了 8.3× 这种「一个字节 KV 都没少」的收益; 而它交的学费是 PD 带宽只兑现一半、可用性欠账,以及一张已经不在注意力上的账单。
This page re-reads the deployment record through an external lecture's framework: a query only does three things, so there are only three knobs — ① how much is stored per token (MQA→GQA→MLA), ② how much each query looks at (SWA/NSA/DSA sparsity), and ③ swapping the mathematics (DeltaNet→GDN linear attention). The history of attention variants is a series of moves between three resources: memory → compute → memory-access regularity, and the order is not accidental — what can be measured with a ruler moves first, what cannot moves last.
The conclusion up front: this deployment is conservative on knob ①, aggressive on ③, and abstains on ②. Nearly all of its engineering budget went into the third room (access regularity), which is where a 8.3× win came from without saving a single byte of KV. The price paid: half the PD bandwidth actually delivered, an availability debt, and a bill that no longer sits on attention at all.
框架来源:Framework source: 专题三 · 注意力演进 (L300) §十「落到硬件」 (卡 A–D、G) (cards A–D, G) ;卡 E–F 另用 ; cards E–F additionally use 专题二 · TPU 与 GPU (L300) 的口径规矩,并对照一份 GB200 上同族 MoE 的扩展性基准。 for its caliber discipline, and cross-check against a GB200 scaling benchmark on a same-family MoE.
卡 A–D、G 里被对账的实测数字全部出自本文档既有的联合优化记录;卡 E 是纯推导(只用已发布常数),卡 F 引用的是外部实测并已逐条标注口径。 Every measured number audited in cards A–D and G comes from this document's existing joint-optimization record. Card E is pure derivation from published constants only; card F cites external measurements, each annotated with its caliber.
🎛️ A · 三个旋钮上的位置 —— 以及那个没有拧的旋钮 A · Where this deployment sits on the three knobs — and the one it did not turn
旋钮Knob 改什么What it changes Qwen 3.5 的选择What Qwen 3.5 chose 代价 / 理由Cost / rationale
① 每个 token 存多少bytes per token 减少 KV 份数或维度Fewer or smaller KV copies 保守Conservative
GQA 96Q / 32KV(3:1),未上 MLAGQA 96Q / 32KV (3:1), no MLA
MLA 的前提是「算力充裕、显存紧张」,且压缩在训练前向里不生效;3:1 的 GQA 换来的是完全规整的读法。MLA assumes "compute abundant, memory tight" and its compression does not apply in the training forward pass; a 3:1 GQA buys fully regular reads instead.
② 每个 query 看多少how much each query reads 限制范围或运行时动态挑选Restrict the span, or pick at runtime 主动弃权Abstained
整份记录里没有 DSA / NSA / 任何 runtime top-kNo DSA, no NSA, no runtime top-k anywhere in the record
这是本案例最大的一次权衡。稀疏注意力的隐含硬件假设是「gather 不太贵」——而这正是 TPU 为换取规整访存主动放弃的能力。放弃旋钮② = 放弃一条公认有效的省算力路线,换「不必在 TPU 上打那场还没打赢的仗」。The single biggest trade here. Sparse attention silently assumes gathers are cheap — exactly the capability TPU gives up in exchange for regular access. Abstaining from knob ② means giving up a proven compute-saving route to avoid a battle that has not yet been won on this hardware.
③ 换一套数学swap the mathematics 用固定大小状态替代变长 KVA fixed-size state instead of a growing KV cache 激进Aggressive
45 / 60 层是 GDN 线性注意力45 of 60 layers are GDN linear attention
线性注意力的前提是「片上内存放得下 chunk 的中间量」。128×128 状态 FP8 压到 16 KB,v7 有 64 MiB VMEM —— 假设成立,兑现 4.9×。这是本案例里少数赌对了的地方。Linear attention assumes the chunk's intermediate state fits on chip. The 128×128 state compresses to 16 KB in FP8 against v7's 64 MiB VMEM — the assumption holds and the 4.9× is realised. One of the few bets here that paid.
①+③ 混合Hybrid 不同层用不同方案Different layers, different schemes 15 层 FA + 45 层 GDN = 60 层15 FA + 45 GDN = 60 layers 只有 15 层产生 KV → 128K 全请求 15 GiB 而非 60 / 80 GiB。Only 15 layers produce KV → 15 GiB for a full 128K request rather than 60 or 80 GiB.
⭐ 讲义列了现代注意力的三个「动态性来源」:ragged 批(各请求长度不同)、分页 KV(一条序列散在不连续的页上)、运行时 top-k(开工了才知道读哪 2048 条)。 这个部署硬扛下了前两个(interleaved paging + RPAd + HND 布局),而第三个、也是最难的那个,是靠选模型绕开的,不是靠写 kernel 解决的。 ⭐ The lecture names three sources of dynamism in modern attention: ragged batches (requests of differing length), paged KV (one sequence scattered across non-contiguous pages), and runtime top-k (which 2,048 entries to read is only known once work begins). This deployment absorbs the first two head-on (interleaved paging, RPAd, HND layout), while the third and hardest one is side-stepped by the choice of model, not solved by writing kernels.
🚚 B · 搬家史的落点 —— 收益最大的几项,一个字节 KV 都没少 B · Where the moving went — the biggest wins saved no KV bytes at all
讲义的断言是:显存和算力能拿尺子量,所以先被搬完;访存规整度量不出来,它只出现在 kernel 里,所以留到最后,也最难。这份部署的战果清单是这句话的实证。 The lecture's claim: memory and compute can be measured with a ruler, so they were moved first; access regularity cannot be measured — it only shows up inside kernels — so it was left for last, and it is the hardest. This deployment's scoreboard bears that out.
优化项Optimization 搬的是哪个房间Which room it moves 省了显存?Saves memory? 省了 FLOPs?Saves FLOPs? 实测结果Measured result
SEQ_ON_LANE / HND KV 布局KV layout 纯访存规整度Pure access regularity 否No 否No decode 700 → 5,790 tok/s (8.3×)
PCP ring 取代 all-gatherreplacing all-gather 通信规整度(连号 P2P vs 广播散取)Comm regularity (contiguous P2P vs scattered broadcast) 否No 否No 950 µs 传输 100% 藏进计算阴影(bubble = 0)950 µs of transfer fully hidden under compute (bubble = 0)
owner_output blockwise combine 写回规整度Write-back regularity 否No 否No decode 5,500 → 5,650 tok/s
GDN 融合 FP8 状态锁 VMEMFused FP8 GDN state pinned in VMEM 消灭 HBM 写回(访存量)Eliminates HBM write-back (traffic) 是Yes 否No 4,120 µs → 840 µs (4.9×)
PCP=8 序列并行sequence parallelism 显存 + 算力(把 128K 拆成 8 × 16K)Memory + compute (128K cut into 8 × 16K) 是Yes 重分布Redistributed 128K TTFT 32 s → 4.62 s (6.9×,同 8 芯片)128K TTFT 32 s → 4.62 s (6.9×, same 8 chips)
⭐ 8.3× 只靠改同一批字节的摆放方式——这是「访存规整度是最难搬也最值钱的那一样」最干净的一个证据。也解释了为什么这一支的前沿工作看起来越来越像写 kernel 而不是改模型。 ⭐ 8.3× purely from re-arranging the same bytes — the cleanest evidence available that access regularity is both the hardest resource to move and the most valuable. It is also why frontier work in this area increasingly looks like writing kernels rather than changing models.
🧾 C · 四个取舍,逐条对账 C · The four trade-offs, itemised against this deployment
①
省显存 ≠ 省计算Saving memory is not saving compute
本案例对得最疼的一条 The one that bites hardest here
已兑现Realised
实测分层:full 492 µs × 15 vs linear 774 µs × 45。
• FA 15 层合计 7.4 ms
• GDN 45 层合计 34.8 ms
那个负责省 KV 的机制,是全模型里更贵的那一个——单层比 full-attention 还慢 57%,总耗时是它的 4.7 倍。旋钮③ 把显存账做没了(O(1) 状态、长文本零膨胀),代价原封不动记到了算力账上。
Measured per layer: full 492 µs × 15 against linear 774 µs × 45.
• 15 FA layers total 7.4 ms
• 45 GDN layers total 34.8 ms
The mechanism that saves the KV cache is the more expensive one in the model — 57% slower per layer than full attention, and 4.7× its total cost. Knob ③ erases the memory bill and posts the whole amount to the compute bill.
②
训练时省 ≠ 推理时省Saving at training is not saving at inference
本案例基本免疫 Largely immune here
低风险Low risk
GDN 是模型本身的结构,不是服务期技巧,所以训练与推理两边都省 —— 这是混合架构相对「推理期压缩」方案的结构性优势。
但要分清:FP8 KV、NVFP4、W4A8 属于纯推理期,训练不享受;拿它们的收益去论证模型效率是偷换口径。
GDN is part of the model's structure, not a serving-time trick, so it saves on both sides — a structural advantage over inference-only compression schemes.
But keep them separate: FP8 KV, NVFP4 and W4A8 are inference-only; training sees none of it. Citing their gains as model efficiency is a change of accounting under cover.
③
不规则访存 / 通信的代价常被低估Irregular access and transfer cost more than the slide says
纸面 2×,落地 1× 2× on paper, 1× delivered
已踩Hit
• PD 交接:KV 要从 prefill 布局重排为 decode 的 (16 头, 全序列) 并跨节点搬运——头与序列两个维度同时重分布,是 PD 分离固有的主要开销。
• 单网卡 P→D 峰值 > 12.5 GB/s,线速 25 GB/s,达标线就是线速的 50%;双网卡 400 Gbps 至今未交付(b/558565285,PR #839 open)。
• 3.8-Max 侧:fused MoE token all-gather 在 4 host 下直接吃掉 −30% E2E,记录标注「暂时无解」。
• PD handoff: KV must be resharded from the prefill layout into decode's (16 heads, full sequence) and moved across nodes — redistributed along the head and sequence axes at once, the dominant intrinsic cost of PD disaggregation.
• Single-NIC P→D peaks at > 12.5 GB/s against a 25 GB/s line rate, and the bar was set at 50% of line rate. Dual-NIC 400 Gbps has still not been delivered (b/558565285, PR #839 open).
• On the 3.8-Max side, the fused-MoE token all-gather costs −30% end-to-end at 4 hosts, marked "no solution for now" in the record.
④
收益有天花板 —— 注意力只是账单的一部分Gains have a ceiling — attention is only part of the bill
最该带走的一条 The one to take away
决定性Decisive
在这个案例里,这条可以精确到小数点: Here this can be settled to the decimal point:
full   492 µs × 15 =   7.4 ms  →  5.8%
linear 774 µs × 45 =  34.8 ms  → 27.5%
MoE   1.41 ms × 60 =  84.6 ms  → 66.7%
─────────────────────────────
合计total               126.8 ms  (文档记录的总时延 158 msrecorded total 158 ms)
FA
GDN 27.5%
MoE 66.7%
MoE 占三分之二,所有注意力加起来只占三分之一,full-attention 自己只占 5.8%。 (这是 prefill 的口径 —— 三个数都来自 prefill 实测。decode 另有一本账,见下面 E 卡:那里 MoE 依然是大头,但机制换成了权重流式读取,块浪费退到第二顺位。) 把 15 层 full-attention 全部删光,prefill 也只快 5.8%。
这正解释了为什么这份记录里真正的大头优化是 fused EP MoE v2、DP↔EP all-to-all 重排、owner_output——而不是注意力。
MoE takes two thirds; all attention combined takes one third; full attention alone takes 5.8%. Deleting all 15 full-attention layers outright would speed prefill up by 5.8%. (This is the prefill caliber — all three numbers are prefill measurements. Decode keeps a separate ledger; see card E below: MoE still dominates there, but the mechanism becomes weight streaming and tile waste drops to second place.)
Which is exactly why the real headline optimizations in this record are fused EP MoE v2, the DP↔EP all-to-all reshuffle and owner_output — not attention.
⭐ 账本交棒:旋钮③ 的真实回报形式⭐ The ledger changes hands: what knob ③ actually buys
讲义给的参照是 DeepSeek V3:128K 下平方项占 81.8%。这个模型在 128K 下平方项只剩 5.8% —— 恰恰因为拧了旋钮③,60 层里 45 层的平方项被抹掉了。所以 GDN 的回报形式不是「注意力变快」,而是「注意力不再是问题,于是问题变成了 MoE」。 The lecture's reference point is DeepSeek V3: at 128K the quadratic term is 81.8% of the forward pass. In this model at 128K it is 5.8% — precisely because knob ③ removed the quadratic term from 45 of 60 layers. So GDN's payoff is not "attention got faster"; it is "attention stopped being the problem, and the problem became MoE".
🔍 D · 防骗判据 —— 拿来审这份材料自己 D · The anti-bluff criterion, turned on this document itself
讲义两句最狠的话:「任何一个倍数,必须带上『在多长的上下文下』」,以及 「还得说清它是哪一样的倍数」 (同一个 MLA:显存 56.9× vs 单用户 decode 时间 7.08×,差整整八倍 —— 报哪一个,取决于你想让听的人以为你省了多少)。
这份材料犯过、并已逐条修掉的,基本就是这两条:
The lecture's two sharpest rules: "every multiple must carry the context length it was measured at", and "it must also say which resource the multiple is in" (the same MLA: 56.9× on memory against 7.08× on single-user decode time — a factor of eight apart; which one you quote depends on how much you want your audience to think you saved).
These are, itemised, the errors this document made and has since corrected:
曾经的说法What it used to say 缺了哪个限定Missing qualifier 现在的处理How it reads now
TTFT 32 s → 4.62 s(6.9×) (6.9×) 「128K 单请求」+「同样 8 颗芯片」。不写就会被读成扩容收益。"128K, single request" and "the same 8 chips". Without them it reads as a scaling win. ✅ 全文均标注「同 8 芯片」Annotated "same 8 chips" throughout
700 → 5,790 与and 5,500 → 5,650 tok/s 两个不同基线,不能相乘、不能相加 —— 与「512 倍与 846 倍是两把尺子」同型。Two different baselines; they cannot be multiplied or added — the same error as chaining a 512× and an 846× figure. ✅ 各自注明分支与口径Each scoped to its own branch
492 µs = 每跳 × 8 跳 → 15 层 61.8 ms492 µs per hop × 8 → 61.8 ms for 15 layers 与文档自身预算矛盾:按每跳算三项合计 178.5 ms > 记录的 158 ms 总时延;按每层算合计 126.8 ms,落在预算内。Contradicts the document's own budget: per-hop makes the three terms sum to 178.5 ms against a recorded 158 ms total; per-layer sums to 126.8 ms, inside it. ✅ 改判为每层,15 层 ≈ 7.4 msRe-read as per layer; 15 layers ≈ 7.4 ms
4,120 µs / 层4,120 µs per layer 另一把尺子,来源不明(可能是另一配置或含 MoE 的整层)。A different ruler of unknown origin (possibly another configuration, or a whole layer including MoE). ⚠ 保留但标注「不要与 492 µs 混用」Kept, flagged "do not mix with the 492 µs figure"
阶段带 1.85 s vs 算子表 126.8 ms1.85 s on the stage ribbon vs 126.8 ms in the operator table 差约 15× —— 典型的「报哪一个」问题。Roughly 15× apart — a textbook "which one do you quote" problem. ⚠ 标注为未闭合,禁止同表相加Marked unreconciled; must not be summed in one table
QKV 输出 2560 列 → 推断芯片数QKV output of 2,560 columns → inferring the chip count 与上线的 PCP8+EP8(注意力权重复制)对不上。Inconsistent with the deployed PCP8+EP8 form, where attention weights are replicated. ⚠ 挂起:不得据 trace 形状推断芯片数Open: do not infer chip count from trace shapes
🧱 E · decode 的第二本账 —— TPOT 有一个 6.7 ms 的权重地板 E · Decode keeps a second ledger — TPOT has a 6.7 ms weight floor
上面 C④ 那张 5.8 / 27.5 / 66.7 的饼是 prefill 的账 —— 它是从 492 µs × 15 + 774 µs × 45 + 1.41 ms × 60 拆出来的,那组数就是 prefill 实测。 decode 是另一本账:那里既不是注意力、也不是 MXU 说了算,而是专家权重每步都要被完整读一遍。 这一条不需要任何实测就能定下来,因为它只用到权重体积和 HBM 带宽两个数。 The 5.8 / 27.5 / 66.7 split in C④ above is the prefill ledger — it comes out of 492 µs × 15 + 774 µs × 45 + 1.41 ms × 60, and those are prefill measurements. Decode keeps a different book: there neither attention nor the MXU sets the pace — the expert weights have to be read end to end on every single step. This one needs no measurement at all; it uses only two numbers, weight volume and HBM bandwidth.
步骤Step 算式Arithmetic 结果Result
每芯片权重Weights per chip 397 GB (fp8) ÷ 8 chips 49.63 GB = 46.22 GiB
每层Per layer 49.63 GB ÷ 60 0.827 GB
读一遍的时间Time to read it once 0.827 GB ÷ 7.37 TB/s 112 µs / 层layer
decode 单步地板Floor for one decode step 112 µs × 60 6.73 ms
量化到 4 bit 权重后With 4-bit weights W4A8 / NVFP4 3.37 ms
地板要读到 batch 这一层才准。权重字节正比于本步被点亮的专家数,而不是行数。top-8 打 512 个专家,按泊松近似命中率是 1 − e^(−B/64),于是:
TPOT 地板(B) = 6.73 ms × (1 − e^(−B/64))
B ≳ 256 时命中率 98%,地板才真正「与 batch 无关」;小 batch 下地板会一起缩。 它不受块浪费影响 —— 补白浪费的是 MXU 周期,不是从 HBM 搬过来的字节。
三条可以直接拿去用的推论:
① 大 batch 下,任何低于 ~6.7 ms 的 TPOT 目标在 8 卡 fp8 上不可达。不是调优问题,是带宽除法。要么摊到更多卡上做 EP,要么把权重变成 4 bit。
② 这给 NVFP4 / W4A8 那条线一个明确的量化理由:它把地板直接砍一半,而且是唯一一个能砍地板的手段。
③ 反过来用,记录里的 5,650 tok/s 能把 decode 的 MBU 反解出来 —— 这正是和外部平台唯一同口径可比的那个量(见下面 F 卡,GB200 在 EP8 上是 55.6%):
The floor only reads correctly once batch is in it. Weight bytes are proportional to the number of experts lit up on this step, not to the number of rows. With top-8 over 512 experts the Poisson approximation gives a hit rate of 1 − e^(−B/64), so:
TPOT floor(B) = 6.73 ms × (1 − e^(−B/64))
At B ≳ 256 the hit rate is 98% and only then is the floor genuinely batch-independent; at small batch the floor shrinks with it. It is immune to tile waste either way — padding burns MXU cycles, not bytes moved from HBM.
Three consequences you can act on:
① At large batch, any TPOT target below ~6.7 ms is unreachable on 8 chips at fp8. Not a tuning problem, a division. Either spread EP over more chips or make the weights 4-bit.
② This gives the NVFP4 / W4A8 workstream a concrete quantitative justification: it halves the floor, and it is the only lever that moves the floor at all.
③ Used backwards, the recorded 5,650 tok/s pins down the decode MBU — the one quantity that is like-for-like comparable against an external platform (see card F: GB200 reaches 55.6% at EP8):
假设 batchAssumed batch 命中专家比例Experts lit TPOT 地板TPOT floor 实际 TPOT(由 5,650 tok/s 反推)Actual TPOT (from 5,650 tok/s) 权重 MBUWeight-only MBU
3239.3%2.65 ms5.66 ms46.8%
6463.2%4.26 ms11.3 ms37.6%
12886.5%5.82 ms22.7 ms25.7%
25698.2%6.61 ms45.3 ms14.6%
⭐ 于是「decode 实际跑在哪个 batch」不再是一个背景参数,而是决定下一步做什么的分叉点:
• 若在 batch 32 一带 → 46.8%,已经是 GB200 EP8 档(55.6%)的水平,实现层面没剩多少,能动地板的只有 4 bit 权重;
• 若在 batch 256 一带 → 14.6%,还有约 3 倍的实现红利没拿,这时先做 NVFP4 就是投错方向。
口径:只算专家权重字节,未计 KV 读取与激活,所以这是 MBU 的下界;长上下文下 KV 流量会把真实 MBU 抬上去。decode batch 目前没有记录 —— 这是补一个数就能立刻收敛的分叉。
⭐ So "what batch does decode actually run at" stops being background and becomes the fork in the road:
• Around batch 32 → 46.8%, already in GB200-EP8 territory (55.6%): little left in the implementation, and only 4-bit weights move the floor;
• Around batch 256 → 14.6%, roughly 3× of implementation headroom still unclaimed, and doing NVFP4 first would be aiming at the wrong thing.
Caliber: expert-weight bytes only — KV reads and activations excluded, so this is a lower bound on MBU; at long context KV traffic pushes the real figure up. The decode batch is not recorded anywhere — one measurement collapses this fork.
块浪费在这本账里排第几?用 megablox 的公开上界算,而不是拍脑袋 —— TPU 不补零,它重复执行跨界块,所以代价是「多跑 < 段数 − 1 个块」:
tiles_m ≤ 实际执行块数 ≤ tiles_m + 段数 − 1(每芯片 64 个本地专家 = 64 段)
• decode · 256 并发:每芯片 256 行 → tiles_m = 2,上界 2 + 63 = 65 块 → 32×。对应约 123 µs(bf16 口径),和 112 µs 的权重地板基本持平 —— 第二顺位,但已经贴着。
• prefill · 一个 16K chunk:每芯片 16,384 行 → tiles_m = 128,上界 128 + 63 = 191 块 → 仅 +49%。
所以 MoE 在 prefill 占 66.7%,主因不是块塌陷 —— 块税封顶只有 +49%,解释不了这个比例。真正的原因朴素得多:60 层 × top-8 × 三矩阵,MoE 本来就拿着绝大部分 FLOPs,+49% 是加在上面的一笔附加税。块塌陷真正致命的地方在 decode,而那里它又恰好被带宽挡在第二位。
Where does tile waste rank in this ledger? Use megablox's published bound rather than a guess — TPU does not zero-pad, it re-executes boundary tiles, so the cost is "fewer than segments − 1 extra tiles":
tiles_m ≤ tiles actually executed ≤ tiles_m + segments − 1 (64 local experts per chip = 64 segments)
• decode, 256 concurrent: 256 rows per chip → tiles_m = 2, bound 2 + 63 = 65 tiles → 32×. That is ~123 µs in bf16 terms, level with the 112 µs weight floor — second in line, but right up against it.
• prefill, one 16K chunk: 16,384 rows per chip → tiles_m = 128, bound 128 + 63 = 191 tiles → only +49%.
So tile collapse is not the reason MoE holds 66.7% of prefill — a tax capped at +49% cannot produce that share. The plain reason is that 60 layers × top-8 × three matrices means MoE genuinely owns most of the FLOPs, with the +49% levied on top. Where tile collapse really bites is decode — and there bandwidth happens to keep it in second place.
📐 口径:这张表是推导,不是实测📐 Caliber: this table is derived, not measured
输入只有两个已发布常数(397B 参数 fp8、7.37 TB/s)和一个部署事实(EP=8 摊在 8 芯片上)。它是下界:KV 读取、激活、all-to-all、以及达不到 100% 的 MBU 全都叠在它上面。按 60–70% 的 MBU 反推,现实中的单步在 9.6–11.2 ms 一带。还没有实测过 —— 这正是值得补的一个数。 The inputs are two published constants (397B params at fp8, 7.37 TB/s) and one deployment fact (EP=8 over 8 chips). It is a lower bound: KV reads, activations, all-to-all and any MBU short of 100% all stack on top. At a realistic 60–70% MBU the real step lands around 9.6–11.2 ms. It has not been measured — which is exactly why it is worth measuring.
🔬 F · 横向对照 —— GB200 上同族 MoE 的实测,以及不能从它得出的那个数 F · Cross-check — a same-family MoE measured on GB200, and the number that cannot be drawn from it
有一份 GB200 上的 MoE 扩展性基准(GDN + 512 专家、60 层,EP 8 / 16 / 32 卡),结构和我们这个模型同族。 按专题二 §6 的四档阶梯, 它和我们这份记录只能挂在第 ① 档:模型不同、精度不同(fp8 vs fp8×fp4)、他们没写上下文长度、我们没写 decode batch、而且只有一侧调过。 所以这张卡里没有、也不会有「TPU v7 比 GB200 快 N 倍」这句话。 能带走的不是倍数,是机制 —— 而机制是可以跨平台搬的。 There is a MoE scaling benchmark on GB200 (GDN + 512 experts, 60 layers, EP over 8 / 16 / 32 GPUs) whose architecture is the same family as this model. On the four-rung ladder in topic-02 §6, pairing it with this record lands on rung ① and nowhere higher: different model, different precision (fp8 vs fp8×fp4), no context length stated on their side, no decode batch stated on ours, and only one side tuned. So this card does not contain — and will not contain — a sentence of the form "TPU v7 is N× a GB200". What travels across platforms is not the multiple, it is the mechanism.
F-1 · 报告的两条结论都不是模型属性,是 tile 属性 F-1 · Neither headline finding is a property of the model — both are properties of the tile
先说一件报告自己没说、但从它的 config 直接推得出来的事:三档卡数下每个 rank 的 FLOPs 完全一样(每 rank 路由行数 = tokens/rank × top-k = 2,560,与 EP 无关)。所以「8→32 卡延迟下降」根本不是强扩展,每卡的活没变少;变的只有每专家行数(涨)和 all-to-all 对端数(涨)。拿 tile_m = 128 的块占用模型算一遍: Start with something the report does not say but its own config implies: per-rank FLOPs are identical across all three GPU counts (routed rows per rank = tokens/rank × top-k = 2,560, independent of EP). So "latency drops from 8 to 32 GPUs" is not strong scaling at all — no rank does less work. Only two things change: rows per expert (up) and all-to-all peers (up). Run the tile_m = 128 occupancy model over it:
配置Config 每专家行数 8 / 16 / 32Rows per expert, 8 / 16 / 32 块浪费Tile waste 实测加速 8→16 / 8→32Observed 8→16 / 8→32 模型预测Model predicts
Qwen 256 tok, k=10 40 / 80 / 160 3.2× → 1.6× → 1.6× 1.44× / 1.31× 2.0 / 2.0
Qwen 512 tok, k=10 80 / 160 / 320 1.6× → 1.6× → 1.2× 1.15× / 0.88× 1.0 / 1.33
DeepSeek 256 tok 32 / 64 / 128 4.0× → 2.0× → 1.00× 1.46× / 1.73× 2.0 / 4.0
DeepSeek 512 tok 64 / 128 / 256 2.0× → 1.00× → 1.00× 1.32× / 1.36× 2.0 / 2.0
模型预测的是「收益在哪里停」,四组全中:
• DeepSeek 256 在 32 卡正好撞上 128 行 = 一整块,所以它是唯一一个一路降到 32 卡的;
• Qwen 256 的占用率从 16 卡起就卡在 62.5% 不动,所以它「16 卡最好」;
• 报告自己的数据里就有反例:同一个模型换成 512 token/rank,8→16 只剩 1.15×、32 卡直接倒退到 0.88×。
⭐ 于是那条能搬走的结论:最佳 EP 数不是模型的属性,是 tokens/rank × top-k ÷ 专家数 与 GEMM 块的关系。换个 batch,最佳点就搬家。
诚实标注:模型预测的幅度普遍偏大(实测只有预测的 0.6–0.75),因为每卡权重字节也在减半,两个机制叠着。Qwen 512 的 8→32 那格(预测 1.33×、实测 0.88×)是模型明确失效的一格 —— 那里通信已经压过一切。
The model predicts where the gains stop, and it is 4 for 4:
• DeepSeek 256 at 32 GPUs lands exactly on 128 rows = one whole tile, which is why it is the only config that keeps improving all the way to 32;
• Qwen 256 pins at 62.5% occupancy from 16 GPUs onward, which is why "16 is best" for it;
• the counterexample is inside the report's own data: the same model at 512 tokens/rank gets only 1.15× from 8→16 and regresses to 0.88× at 32.
⭐ Hence the transferable conclusion: the optimal EP count is not a property of the model. It is the relationship between tokens/rank × top-k ÷ experts and the GEMM tile. Change the batch and the optimum moves.
Stated honestly: the model over-predicts magnitude (observed runs 0.6–0.75 of predicted) because per-GPU weight bytes are halving at the same time — two mechanisms stacked. The Qwen 512 8→32 cell (predicted 1.33×, observed 0.88×) is where the model plainly fails: communication has taken over there.
F-2 · 报告写「HBM utilization 没达到最好,以后再深挖」—— 深挖的结果是 MBU 三档砍掉三分之二 F-2 · The report says "HBM utilization doesn't achieve best, could deep dive later" — the deep dive says MBU loses two thirds over three steps
Qwen 256 k=10    8 → 55.6%   16 → 42.1%   32 → 20.6%
Qwen 512 k=10    8 → 51.2%   16 → 32.1%   32 → 14.1%
DeepSeek 256     8 → 60.8%   16 → 45.9%   32 → 28.7%
这不是 kernel 变差了,是每卡权重字节减半、而 all-to-all 的对端数翻倍。专题二 §4.4 那句「四个原语里三个能拆成环、只有 all-to-all 不能,5 个点从 5 条边跳到 10 条边」在这张表上第一次有了数字。讲义 §4.5 自己测的 DeepEP 在同一个 NVL72 域内从 8 卡到 32 卡就掉了 Dispatch −9% / Combine −19%;这份报告是跨 2→8 个节点,掉得更狠,量级对得上。 This is not the kernel degrading. Per-GPU weight bytes halve while the number of all-to-all peers doubles. Topic-02 §4.4 — "three of the four primitives decompose into rings, all-to-all does not; five nodes go from five edges to ten" — finally has numbers attached to it here. The lecture's own DeepEP measurement inside a single NVL72 domain loses 9% on Dispatch and 19% on Combine going from 8 to 32; this report spans 2→8 nodes and loses more, which is the right direction and the right order of magnitude.
F-3 · Top-K 10→6 的收益曲线,证明瓶颈不只是通信 F-3 · The Top-K 10→6 gain curve shows the bottleneck is not only communication
报告把 32 卡那 30% 的收益全部归给「通信量减少 40%」。但同一个改动在 8 卡只值 8% —— 通信量在 8 卡也少了 40%,为什么不灵?块模型给的答案很干脆: The report credits the 30% win at 32 GPUs entirely to "40% less communication". But the same change is worth only 8% at 8 GPUs — communication drops 40% there too, so why doesn't it pay? The tile model answers bluntly:
8 卡8 GPUs 32 卡32 GPUs
top-k=10 每专家行数 → 块数top-k=10 rows/expert → tiles 40 → 1 160 → 2
top-k=6 每专家行数 → 块数top-k=6 rows/expert → tiles 24 → 1 96 → 1
块数变化Change in tile count 没变none 减半halved
实测收益Observed gain 8% 30%
8 卡上砍掉 40% 的数学只买到 8%,因为块数一个没少 —— 那 40% 的行本来就落在补白里。这不否定报告的通信解释(两者共同作用),但它是一个更便宜的检验:真是通信主导的话,8 卡也该有两位数收益。 Removing 40% of the arithmetic buys 8% at 8 GPUs because not one tile disappeared — those 40% of rows were already sitting in the padding. This does not refute the report's communication explanation (both act at once), but it is a cheaper test: if communication really dominated, 8 GPUs should have shown a double-digit gain too.
F-4 · 对我们自己的一句:我们在这张表的最左端之外 F-4 · One line about ourselves: we sit off the left edge of that table
GB200 那份里最差的一档是每专家 40 行。我们 decode 一步 256 token、top-8、512 专家,每专家 4 行 —— 差 10 倍,而 megablox 的 bf16_bf16_tile_m = 128 要 8,192 个 token 才填满一块。TPU 的 megablox 是重复执行边界块而不是补零,形状上不同、账是一样的。所以 EP 的第一价值不是省通信,是把段拼长;而这也解释了为什么 E 卡里那 123 µs 的浪费已经和权重地板同量级。GB200 在 EP8 上做到 55.6% MBU —— 我们 decode 侧的 MBU 是多少,目前没有能算出它的数(缺 batch 和上下文长度),但这是两边口径可比的量,值得测一个出来。 The worst rung in the GB200 set is 40 rows per expert. One decode step here is 256 tokens, top-8, 512 experts — 4 rows per expert, 10× worse, against a megablox bf16_bf16_tile_m = 128 that needs 8,192 tokens to fill a single tile. TPU megablox re-executes boundary tiles rather than zero-padding them; different shape, same bill. So the first value of EP is not saving communication, it is making the segments longer — and it is why the 123 µs of waste in card E is already the same order as the weight floor. GB200 reaches 55.6% MBU at EP8; what our decode MBU is, we currently cannot compute (batch and context length are both missing) — but it is a like-for-like quantity across the two platforms and worth measuring.
🧾 三条对这份报告的口径批注(按专题二 §8「自曝台账」的规矩摆出来)🧾 Three caliber notes on that report (posted the way topic-02 §8's self-audit ledger requires)
① E2E 投影表的单位错了。表头写 GDN (ns) 320、Per-layer (ns) 688,但 688 ns × 60 层 = 41.3 µs,而 Total 行写的是 41 ms。四组全部验过(41.27 / 67.52 / 42.94 / 74.54 对表里的 41 / 67 / 43 / 75)—— 那一列应该是 µs。差 1000 倍,而它长得完全正常。
② 投影把 15 层全注意力整个漏掉了。它假设 60 层全是 GDN + MoE;真实结构是 45 GDN + 15 FA,而 FA 层在 decode 还要读 KV。41 ms 是地板,不是估计。
③ 全文没有一处写序列长度。hybrid 模型的 decode benchmark 不写上下文长度 —— GDN 的开销与上下文无关(这正是线性注意力的卖点),FA 层的开销与上下文成正比,而 FA 层不在表里。所以这个 TPOT 既不能外推,也不能和我们的数比。
再用 §6 三问打它一遍:① 分母是 legacy DeepEP,没说调没调过(被替换方 vs 主推方,典型不对等);② 没说两边各调了多久;③ 报的既不是吞吐也不是 MFU,是孤立 kernel 延迟。这三条同样适用于任何人来打我们这份文件。
① The E2E projection table has the wrong unit. The headers read GDN (ns) 320 and Per-layer (ns) 688, but 688 ns × 60 layers = 41.3 µs, while the Total row prints 41 ms. All four configs check out (41.27 / 67.52 / 42.94 / 74.54 against the printed 41 / 67 / 43 / 75) — that column must be µs. A factor of 1000, and it looks entirely normal.
② The projection drops all 15 full-attention layers. It models 60 layers of GDN + MoE; the real structure is 45 GDN + 15 FA, and the FA layers also read KV at decode. 41 ms is a floor, not an estimate.
③ No sequence length appears anywhere in the report. A decode benchmark of a hybrid model without a context length is uninterpretable — GDN cost is context-independent (that is the whole selling point of linear attention), FA cost is proportional to context, and the FA layers are not in the table. So this TPOT can neither be extrapolated nor compared against ours.
Run §6's three questions over it as well: ① the denominator is legacy DeepEP and the report never says whether it was tuned (the replaced side against the promoted side — the classic asymmetry); ② no statement of how long either side was tuned; ③ what is reported is neither throughput nor MFU but isolated kernel latency. The same three questions apply to anyone auditing this document.
📉 G · 明写在账上的妥协与欠账 G · Compromises and debts, written on the ledger
讲义说三招的共同形状是「把一个动态换成一批静态」。这个部署照做了(chunked prefill 固定 16K 块、SEQ_ON_LANE 选定序列住哪一维、多套预编译拓扑 PCP8→DP8 / DP16 / TP2DP4EP8 / TP2DP8EP16 / P4D4),但换来的静态性是有代价的: The lecture's three techniques share one shape: replace one dynamic thing with a handful of static ones. This deployment does exactly that (chunked prefill fixed at 16K blocks, SEQ_ON_LANE choosing which axis the sequence lives on, and a set of pre-compiled topologies: PCP8→DP8 / DP16 / TP2DP4EP8 / TP2DP8EP16 / P4D4). But that static-ness was bought:
换到了什么What was gained 付出了什么What it cost 状态Status
规整访存(不必打稀疏 gather 那场仗)Regular access — no sparse-gather battle to fight 放弃旋钮②,也放弃 MLA;两条公认有效的路线留在桌上Knob ② abandoned, and MLA with it — two proven routes left on the table 设计取舍By design
吞吐(5,650 ➔ 6,500–7,600 tok/s decode · 44k ➔ 55k–58k tok/s 4-Host prefill)Throughput (5,650 ➔ 6,500–7,600 tok/s decode; 44k ➔ 55k–58k tok/s 4-Host prefill) tpu-sync 线程泄漏已修(#1112/#1115/#1116),但大规模试投产仍面临低频 E200 CoreHalt 与 MTP 接收率偶发跌 0%tpu-sync leaks fixed (#1112/#1115/#1116), while fleet burn-in still tracks low-frequency E200 CoreHalt and intermittent MTP 0% drops 持续收敛Converging
PD 分离(prefill 与 decode 各自打满)PD disaggregation (each phase saturated independently) 单网卡初期仅兑现 50% 线速;后经 Raiden NUMA 双网卡(367 Gbps / 91.8% 线速)+ gRPC 控制面(2MB p99 −35.8%)破局,代价是单 Worker 需常驻 ~6.53 GB Staging 缓冲Single-NIC initially hit 50% line rate; resolved via Raiden NUMA dual-NIC (367 Gbps / 91.8% line rate) + gRPC control plane (2MB p99 −35.8%), at the cost of ~6.53 GB resident staging buffer per worker 已破局兑现Delivered
下一档扩容(PCP16·TP2 ➔ TP32·DCP4/8 及 PP4·PCP8)The next scale-up (PCP16·TP2 ➔ TP32·DCP4/8 & PP4·PCP8) 废弃纯 4-Host All-Gather(−30% E2E)改走 PP4·PCP8 实测达 55k–58k tok/s,#1108/#1182/#1183 已合入;剩余 3K+5K 分块 Padding 抖动与 DCP VMEM 估算收尾 —— 核心矛盾始终在「第三个房间」(通信与调度编排)Retired pure 4-Host All-Gather (−30% E2E) in favor of PP4·PCP8 (55k–58k tok/s measured) with #1108/#1182/#1183 merged; remaining items are 3K+5K chunk-padding variance and DCP VMEM estimation — every bottleneck still lives in the third room (collectives & scheduling) 收尾调优Final tuning
⭐⭐ 一句话收全页:这个案例是那条主线的一个完整样本 —— 它在旋钮① 上保守、在③ 上激进、在② 上主动弃权,用「不要 gather」换「要规整」; 然后把几乎全部工程预算投进第三个房间,拿到了 8.3× 这种只靠摆放方式的收益; 而它交的学费是 PD 带宽只兑现一半、可用性欠账,以及一张已经不在注意力上的账单 —— prefill 里 MoE 占了 66.7%。 而 decode 那一本账更硬:6.7 ms 的权重地板连调优的余地都不给,只有把权重变成 4 bit 才动得了它。 ⭐⭐ The page in one sentence: this case is a complete specimen of that storyline — conservative on knob ①, aggressive on ③, abstaining on ②, trading away gathers to keep access regular; then pouring nearly the whole engineering budget into the third room, where a 8.3× came from arrangement alone. The tuition paid: half the PD bandwidth, an availability debt, and a bill that no longer sits on attention — at prefill, MoE holds 66.7% of it. The decode ledger is harder still: a 6.7 ms weight floor that leaves no room for tuning at all — only 4-bit weights move it.
📐 底层物理铁律对账:DeepMind Austin Roofline 极限与 Ironwood 双 Die 架构真相 Silicon Physics Ledger: DeepMind Austin Roofline Limit & Ironwood Dual-Die Reality
🌐 EN / 中文

结合 Google DeepMind Jacob Austin《A Whirlwind Tour of LLM Serving》(go/llm-inference-slides)与 SemiAnalysis 针对 TPU v7 (Ironwood) 的芯片级解构,本项目各项关键拓扑选型背后存在着无可辩驳的物理必然性: Synthesizing Google DeepMind Jacob Austin's foundational framework ("A Whirlwind Tour of LLM Serving", go/llm-inference-slides) with SemiAnalysis's silicon deconstruction of TPU v7 (Ironwood), the key topology choices in this project are governed by strict physical laws:

1. 为什么 Decode 死守 TP=2?模型并行物理极限公式 1. Why Decode Caps at TP=2? The Model Parallelism Limit
根据 DeepMind Austin 模型并行上限判据: Y ≤ F / 2550(其中 F 为 FFW 中间维,2550 为 TPU 脉动算力与 ICI 网络带宽的物理特征常数)。
• Dense 模型 (Llama-3 70B): F = 28672 → Y ≤ 11.2,可轻松扩展至 TP=8。
• 细粒度 MoE (Qwen 3.5 397B): 单专家 F_expert ≤ 4096 → Y ≤ 1.60! 一旦强开 TP=4,All-Gather 通信延迟将直接吞噬算力收益(坠入 Comms-bound)。因此阿里选择 TP=2 · DP=4 · EP=8 是守住通信物理红线的唯一数学正解。
Per DeepMind Austin's model parallelism threshold: Y ≤ F / 2550 (where F is FFW intermediate dim, and 2550 is the TPU FLOPs/ICI-bandwidth ratio).
• Dense (Llama-3 70B): F = 28672 → Y ≤ 11.2, easily scaling to TP=8.
• Fine-grained MoE (Qwen 3.5): Single expert F_expert ≤ 4096 → Y ≤ 1.60! Scaling to TP=4 collapses into a comms-bound stall where All-Gather completely dwarfs systolic compute. Thus TP=2 · DP=4 · EP=8 is the mathematically enforced frontier.
2. TPU v7 芯片真相:双裸晶架构与 EP=8 对账 2. Silicon Reality: Dual-Die Chiplet & EP=8 Reconciliation
• 双 Die 封装 (SemiAnalysis 证实):Ironwood 单颗物理 Chip 封装了 2 个独立计算裸晶(Compute Dies),JAX/TorchTPU 驱动层直接识别为 2 个独立逻辑 Device,片上配备极速 Die-to-Die 互联。
• 阿里 EP=8 真实物理指代:阿里 2×4 拓扑 中的单机 8device 语境指代整机 8 个物理 Chip 插槽(底层映射 16 个逻辑 Device),512 个专家被均衡分发到 8 颗物理芯片上,每芯承载 64 个专家并由片内 SparseCore 硬件路由。
• Dual-Die Packaging (Per SemiAnalysis): Each physical Ironwood package contains 2 compute dies exposed as 2 distinct logical devices in JAX/TorchTPU, linked by on-package high-speed Die-to-Die traces.
• Alibaba EP=8 Truth: The 2×4 topology refers to 8 physical chip sockets per host (mapping to 16 logical devices). The 512 experts are mapped across the 8 physical chips (64 experts/chip) accelerated by on-chip SparseCores.
3. FP4 真相:硬件无原生计算,以 W4A8 曲线破局 3. FP4 Reality: No Native Hardware MAC, Solved via W4A8
• 无原生 FP4 阵列:TPU v7 硬件 MXU 仅支持 INT8 / FP8 / BF16,原生 FP4 浮点乘加要到 TPU v8i 才会搭载。
• 工程务实之道 (PR #912):阿里之所以落地 NVFP4 + W4A8,是因为在 Decode 阶段瓶颈受困于 HBM 读取带宽;将权重以 NVFP4 紧凑格式装载入片上 SRAM,再通过向量指令在线解压至 8-bit 送入 MXU,以 0 精度惩罚换取了 2× 显存带宽等效提升!
• No Native FP4 Engine: TPU v7's MXU only accelerates INT8/FP8/BF16 in silicon; native hardware FP4 computation is reserved for TPU v8i.
• Engineering Pragmatism (PR #912): Alibaba adopted NVFP4 + W4A8 because Decode is HBM-bandwidth bound; storing weights in NVFP4 cuts memory traffic in half, decompressing on-chip into 8-bit for MXU execution with zero accuracy penalty!
🎬 3Blue1Brown 风格核心技术全景动画放映厅 (4 大篇章) 3Blue1Brown Style Technical Animation Theater (4 Core Chapters)
Manim Community · 1080p @ 60fps
📐 篇章 1 物理极限推导:从 $T_{\text{math}}$ 与 $T_{\text{comms}}$ 微架构公式出发,动态代数变形推导出 $Y \le F/2550$ 判据,直观展示 Dense (Llama-3 70B, TP=8) 与细粒度 MoE (Qwen 3.5, TP=2) 的物理安全红线。
🤖 前沿功能:Agentic TPU 优化工具链 (torchtpu-agents) Frontier Capability: The Agentic TPU Optimization Toolchain (torchtpu-agents)
前面几页记录的是一场手工驱动的攻坚战役。这一页记录的是把这场战役工业化的 Agentic 工具链—— 一个 31 个 Agent 的软件包,把 Kernel 调优、切分搜索与迁移验证转化为可复现、可审计的流水线。
口径声明:以下全部结论均来自对已安装包 torchtpu-agents@0.1.0 的逐行阅读, 而非厂商文档;第 5 卡的性能数字来自本文档配套集群上的真实物理派发,非估算。
The preceding pages record a hand-driven optimization campaign. This page records the agentic toolchain that industrializes it — a 31-agent package turning kernel tuning, sharding search and migration verification into reproducible, auditable pipelines.
Caliber note: every finding below is read line by line from the installed package torchtpu-agents@0.1.0, not from vendor documentation. The numbers in card 5 come from real physical dispatch onto this document's companion cluster, not from estimation.
🌳 1 · 委派拓扑:31 个 Agent(1 路由 + 4 大功能族)、4 层深度(3 跳委派) 1 · Delegation topology: 31 agents (1 router + 4 families), 4 levels (3 delegation hops)

这个包不是 31 个平级工具,而是一棵 4 层深(最长路径 tpu-coworker → maxkernel-loop → worker → implement-kernel 共 3 跳委派)的严格分层树。分层本身就是让优化成本随规模保持平坦的机制。 The package is not 31 peer tools. It is a strictly layered 4-level tree (deepest chain: tpu-coworker → maxkernel-loop → worker → implement-kernel, 3 delegation hops). Layering is the mechanism that keeps cost flat as an optimization effort grows.

tpu-coworker  (L0 唯一入口 — 纯路由,自身不持有 shell)
│
├─ [族 1 · 诊断与溯源 (3)]
│    ├─ hello-tpu ................. 硬件可达性冒烟测试与 CPU 交叉校验
│    ├─ provenance ................ TPU 技术结论的可引用公开出处校验
│    └─ xprof-analyst ............. 全包唯一 Profile 入口(各层级共享)
│
├─ [族 2 · Kernel 与跨轮研究 (13)]
│    ├─ maxkernel-loop ............ 单个命名 Pallas Kernel 迭代编排(固定 5 轮)
│    │    └─ worker ............... 单轮完整迭代(7 相位执行器)
│    │         └─ 10 个 Kernel 专家(plan / implement / fix-compile / autotune / summarize 等)
│    └─ maxauto-round ............. 跨 Session 长期课题账本(每轮推进单个 Arm)
│
├─ [族 3 · 分布式切分与配置搜索 (5)]
│    └─ maxshard-orchestrator ..... 已知 Workload 的配置爬山编排
│         ├─ maxshard-optimizer ... "大脑":形成假设与下一套配置
│         ├─ maxshard-executor .... "肌肉":提交作业 / 监控 / 解析指标
│         ├─ maxshard-researcher .. 启发式用尽后的广度搜索
│         └─ maxshard-memory-contributor ... 实证经验去重与入库
│
└─ [族 4 · HF 迁移与常驻服务化 (9)]
     └─ torch-tpu-migration-orchestrator ... HF 模型迁移 TPU 并执行基准测试
          ├─ smart-hf-search / preflight-check / torch-coder / remote-tpu-runner
          └─ torch-tpu-inference ... 常驻 HTTP 推理服务生命周期编排
               └─ engine-compatibility / torch-inference-coder / numerical-parity
tpu-coworker  (L0 sole entrypoint — pure router, holds no shell)
│
├─ [Family 1 · Diagnostics & Provenance (3)]
│    ├─ hello-tpu ................. hardware reachability smoke test & CPU cross-check
│    ├─ provenance ................ citable public source verifier for TPU claims
│    └─ xprof-analyst ............. the ONLY profiling entrypoint (shared across tiers)
│
├─ [Family 2 · Kernel Optimization & Study Ledger (13)]
│    ├─ maxkernel-loop ............ single named Pallas kernel loop (fixed 5 rounds)
│    │    └─ worker ............... one full iteration (7-phase executor)
│    │         └─ 10 kernel specialists (plan / implement / fix-compile / autotune / summarize)
│    └─ maxauto-round ............. cross-session study ledger (advances one arm per round)
│
├─ [Family 3 · Sharding & Config Hill-Climbing (5)]
│    └─ maxshard-orchestrator ..... config hill-climbing on a known workload
│         ├─ maxshard-optimizer ... the "brain": hypothesis & next configuration
│         ├─ maxshard-executor .... the "muscle": job submission / monitoring / parsing
│         ├─ maxshard-researcher .. broad search when greedy heuristics saturate
│         └─ maxshard-memory-contributor ... empirical takeaway deduplication & ingestion
│
└─ [Family 4 · HF Migration & Persistent Serving (9)]
     └─ torch-tpu-migration-orchestrator ... HF-to-TPU migration & benchmark pipeline
          ├─ smart-hf-search / preflight-check / torch-coder / remote-tpu-runner
          └─ torch-tpu-inference ... persistent HTTP serving lifecycle orchestrator
               └─ engine-compatibility / torch-inference-coder / numerical-parity
分区 / 结构性不变量Partition / Structural invariant 数量Count 工程构成与约束后果Composition & engineering consequence
31 个 Agent 完整分族核算Complete 31-agent family accounting 1 + 3 + 13 + 5 + 9 = 31 1 个顶层路由(tpu-coworker)+ 3 个诊断/溯源叶节点 + 13 个 Kernel/Study 节点 + 5 个 MaxShard 切分节点 + 9 个 HF 迁移与服务化节点,零遗漏。 1 router (tpu-coworker) + 3 diagnostic/provenance leaves + 13 kernel/study agents + 5 MaxShard agents + 9 HF migration & serving agents = 31 total.
操作者可直接调用的 AgentAgents callable directly by an operator 7 / 31 入口白名单是一条禁令:"Do not invoke any other package agent directly."
绕过它意味着请求会在触及任何硬件之前被目标 Agent 的契约拒绝。
The allowlist is a prohibition: "Do not invoke any other package agent directly."
Bypassing it sends a request to an agent whose contract rejects it before hardware is touched.
持有委派权的 AgentAgents holding delegation rights 9 / 31 其余 22 个是叶子。叶子无法扇出,因此一个错误计划的爆炸半径被这 9 个封死。 The remaining 22 are leaves. A leaf cannot fan out, so the blast radius of a bad plan is bounded by the 9.
无 shell 访问权的 AgentAgents with no shell access 7 / 31 编排者与归纳者被刻意设计成"没有手"。tpu-coworker 与 maxkernel-loop 连一条命令都跑不了,只能路由和读状态。 Orchestrators and summarizers are deliberately handless. tpu-coworker and maxkernel-loop cannot run a command at all — only route and read state.
🚪 2 · 七个面向操作者的入口 2 · The seven operator-facing entrypoints
Agent 它回答的唯一问题The one question it answers 终态产物Terminal artifact
hello-tpu PyTorch 到底能不能摸到真硅片?Can PyTorch actually reach real silicon? 设备标签 + CPU 交叉校验判决Device label + CPU cross-check verdict
provenance 这条 TPU 事实的可引用公开出处是什么?What is the citable public source for this TPU fact? 缓存条目 + 唯一可打脚注的 URLCached entry + the single URL to footnote
xprof-analyst 时间花在哪了,以及工具看不见什么?Where did time go, and what could the tools not see? 实测与推断严格分离的 Profile 报告Report separating measured from inferred
maxkernel-loop 这一个指名的 Pallas Kernel 还能更快吗?Can this one named Pallas kernel go faster? iter5/optimized.py
maxauto-round 该攻哪一层,且这个收益经得起推敲吗?Which layer to attack, and does the win survive scrutiny? 排序过的实测瓶颈分解 + 持久课题台账Ranked measured decomposition + durable ledger
maxshard-orchestrator 哪套切分 / 编译配置最快?Which sharding / compiler configuration is fastest? 最优配方 + 每次尝试的完整轨迹Best recipe + full attempt trajectory
torch-tpu-migration-orchestrator 这个 HF 模型能上 TPU 吗,比 CPU 快多少?Does this HF model run on TPU, and how much faster than CPU? results.json + benchmark_report.md

两条契约约束着上述全部 Agent:一个 completed 结果必须携带至少一个 Artifact 描述符——"一段长到足以顶替产物的摘要"正是这条规则要阻止的失败;以及,最终响应若不是符合 Schema 的 JSON 信封,会在操作者看到之前就被运行时校验拒绝。 Two contracts govern all of them: a completed result must carry at least one artifact descriptor — "a summary long enough to stand in for the artifact" is the exact failure the rule exists to stop; and a final response that is not a schema-conforming JSON envelope is rejected by runtime validation before the operator ever sees it.

⚙️ 3 · 三个优化引擎,三类不同的问题 3 · Three optimization engines, three different problems

最常见的误用是抄错引擎。判别标准不是"问题有多难",而是"你已经知道什么"。 The commonest misuse is reaching for the wrong engine. The discriminator is not "how hard is the problem" but "what do you already know."

维度Dimension maxkernel-loop maxshard-orchestrator maxauto-round
前置条件Precondition 已知是哪个 Pallas KernelYou know which Pallas kernel 已知 Workload 与目标指标You know workload and target metric 尚不知道瓶颈在哪一层You do not yet know the bottleneck layer
它改什么What it changes Kernel 源码Kernel source 配置 / 编译器 FlagConfig / compiler flags 暂不改——它产出诊断Nothing yet — it produces the diagnosis
状态位置State location Run 目录内部Inside the run directory Run 目录内部Inside the run directory 在 runs 之外,与之并列Beside runs, not inside one
能否扛过 Session 结束Survives a session ending ✗ ✗ ✓ 刻意设计by design
统计严谨性Statistical rigour 仅比较 optimized_timeCompares optimized_time only 与当前最优比较Compares against current best 预注册协议:噪声地板、重复次数、准入门禁;运行时会拒绝写入与协议矛盾的记录Pre-registered protocol: noise floor, replicate count, admissibility gates; the runtime refuses to write a record contradicting it
停止条件Stop condition iteration >= 5(硬编码) (hard-coded) 3 次贪婪 → 3 次广搜 → 终止3 greedy → 3 broad-search → halt max_rounds / plateau_rounds / target_improvement_pct
⚠ 一个反直觉却承重的事实 ⚠ A counter-intuitive but load-bearing fact
maxauto-round 并不包裹 Kernel 循环,也不派发任何执行器。它唯一的委派目标是 xprof-analyst。 包内明确写出了理由——worker 是 MaxKernel 的 Kernel 迭代 Agent,其契约要求 Kernel 产物且禁止其他, 因此一个切分或调度类的 Arm 派过去会在运行前就校验失败。所以课题层今天交付的是一台测量与台账仪器; 通用执行器被推迟为独立变更,未来将原样读取同一份 Arm 记录。

成本设计同样刻意:课题状态中每个列表段落都设了上限,使得第 60 轮的成本等于第 2 轮的成本。 包内把 MaxKernel 那个无界增长的迭代历史点名为 "precisely the thing not to copy"—— 课题层正是针对下一层已知的扩展性缺陷而设计的。
maxauto-round does not wrap the kernel loop and dispatches no executor. Its only delegation target is xprof-analyst. The package states why — worker is MaxKernel's kernel-iteration agent, its contract requires a kernel artifact and forbids anything else, so a sharding or scheduling arm sent there fails validation before it runs. The study layer therefore ships as a measurement and ledger instrument; a generic executor is deferred to its own change, reading the same arm record unmodified.

The cost design is equally deliberate: every list section in the study state is capped so that round 60 costs what round 2 cost. The package names MaxKernel's unbounded iteration history as "precisely the thing not to copy" — the study layer was designed in reaction to a known scaling flaw one tier below it.
🔬 4 · 数值对齐门禁:为什么只看 Logits 不够 4 · The numerical parity gate: why logits alone are not enough

当 CUDA 锁定的代码被替换为原生 PyTorch,迁移只有在数字活下来时才算真的成立。 numerical-parity 以 Teacher Forcing 方式运行模型两次——在固定 Token 序列上做一次前向,绝不自由生成—— 并比较三样东西:最终 Logits、每个被改写模块的 Forward Hook 激活值、以及逐位置 Top-1 一致性。 When CUDA-locked code is replaced with native PyTorch, the migration is only real if the numbers survive. numerical-parity runs the model twice under teacher forcing — one forward pass over a frozen token sequence, never free generation — comparing three things: final logits, per-module forward-hook activations for every rewritten module, and per-position top-1 agreement.

🧪 证明"逐模块"必要性的那次实验 🧪 The experiment that proves per-module rows are necessary
在单层刻意注入 10% 误差,结果每个 Token 都仍然匹配,且最终 Logits 反而更接近参考。 只有逐模块那一行暴露了故障——坏掉的那层读数 10.85% 相对误差,而邻层是 0.70%。 A deliberate 10% error injected into a single layer left every token still matching and the final logits slightly closer to the reference. Only the per-module row exposed the fault — the broken layer read 10.85% relative error against 0.70% for its neighbour.
阈值Threshold 取值Value 实测依据Empirical basis
相对容差Relative tolerance 5% 在 mamba-130m 上标定:CPU bf16 与 TPU v6e bf16 运行完全相同的代码,layer 23 读数 2.92%、Logits 读数 3.09%——这是纯设备噪声。而注入的 10% 故障读数为 10.85%。 Calibrated on mamba-130m: CPU bf16 vs TPU v6e bf16 running identical code measured 2.92% at layer 23 and 3.09% on logits — pure device noise. The injected 10% fault read 10.85%.
绝对容差Absolute tolerance 默认关闭Off by default 同一模型各层张量尺度跨越 8.1 至 341.9,单一绝对界没有意义。 Tensor scales ranged 8.1 to 341.9 across layers of one model; a single absolute bound is meaningless.
平局阈值Tie threshold max(2.0 × RMS, 1e-3) 用 RMS 而非 max-abs,这样单个坏元素无法撑大本该抓住它的那个阈值。 RMS rather than max-abs, so one bad element cannot inflate the very threshold meant to catch it.
文档化噪声地板Documented noise floor ~3% 错得比这还少的改写无法分辨。降低阈值只会制造误报——正确做法是扩大 Probe 集合。 A rewrite wrong by less than this is indistinguishable. Lowering the threshold only manufactures false alarms — the correct response is to widen the probe set.

判决按固定顺序求值,而顺序本身是有意义的:若参考运行不独立(两侧执行了同一份改写后代码),结果在任何失败检查之前就被判定为 UNVERIFIED,因此一个非独立的参考永远不可能被报成通过。包内要求 UNVERIFIED 必须原词上报,绝不允许包装成成功。
代际标定声明:上述 5% 相对容差与 ~3% 噪声地板是在 TPU v6e(mamba-130m 对照 CPU bf16)上标定的先验常数,尚未在 TPU v7x (Ironwood) 上重新标定。
The verdict is evaluated in a fixed order, and the ordering matters: if the reference run was not independent (both sides ran the same rewritten code), the result is UNVERIFIED before any failure check runs, so a non-independent reference can never be reported as a pass. The package requires UNVERIFIED be reported in those words and never dressed up as success.
Generation calibration caveat: the 5% relative tolerance and ~3% noise floor were calibrated on TPU v6e (mamba-130m vs CPU bf16) and have not yet been re-derived on TPU v7x (Ironwood).

✅ 5 · TPU v7x (Ironwood) 物理验证、P0 审计撤稿与 Pallas Grouped GEMM 真实 A/B 实测 5 · Physical validation on TPU v7x (Ironwood), P0 Audit Retraction & Real Pallas Grouped GEMM A/B

本节记录在真实 GKE TPU v7x 集群(2x2x1,8 芯片,jax 0.10.2)上的端到端物理验证。内部代码审计发现初版报告的 0.821 ms / 10.46 TFLOP/s 存在 P0 归因与形状缺陷;以下完整披露审计撤稿根因、Pallas Kernel 的 4 项编译器修复,以及修复后在真机硅片上横跨 3 个问题规模 × 3 种瓦片尺寸(128 / 256 / 512)的真实 A/B 实测。 This card records end-to-end physical validation on the live GKE TPU v7x cluster (2x2x1, 8 chips, jax 0.10.2). An internal code audit found that the initially reported 0.821 ms / 10.46 TFLOP/s suffered from P0 attribution and shape defects; below we disclose the full retraction ledger, the 4 compiler fixes required to make the Pallas kernel compile on Mosaic, and the real 3-scale × 3-tile-size (128 / 256 / 512) A/B benchmark on silicon.

缺陷等级 / 类别Defect class 初版问题与编译器报错证据Observed defect & compiler diagnostic 工程修复(已落地 pallas_grouped_gemm.py 与 pallas_ab_benchmark.py)Engineering fix applied
P0-1 归因错误 初版 tpu_grouped_gemm_benchmark.py 仅导入 torch 并调用 torch.matmul,从未加载 pallas_grouped_gemm.py,且末尾硬编码打印 "256x256 Native Alignment Confirmed"。初版 0.821 ms / 10.46 TFLOP/s 实测的是 PyTorch/XLA 算子,在此正式撤稿。 The original tpu_grouped_gemm_benchmark.py imported only torch and called torch.matmul, never loading pallas_grouped_gemm.py, and printed a hardcoded literal "256x256 Native Alignment Confirmed". The 0.821 ms / 10.46 TFLOP/s figure measured XLA matmul and is formally retracted. 新基准脚本通过 SHA-256 指纹(c498fe09…39fb)原样内联并调用真实 Pallas Kernel,每次迭代前通过密集 fp32 参考矩阵做门禁校验(cosine = 1.000000)。 New harness inlines the verbatim Pallas kernel with SHA-256 provenance (c498fe09…39fb) and gates every arm against a dense fp32 reference (cosine = 1.000000).
P0-2 形状自相矛盾 初版形状 1024 tokens ÷ 8 experts = 128 tokens/expert,放入 256 行瓦片后恰好半空(50% 补零),与"消除补零开销"结论完全相反;且循环内仅同步 res[0,0].cpu(),存在 7/8 专家被 XLA 死代码消除(DCE)风险。 Original shape 1024 tokens ÷ 8 experts = 128 tokens/expert left every 256-row tile 50% empty (worsening padding), and synced only res[0,0].cpu(), leaving 7 of 8 experts exposed to XLA dead-code elimination (DCE). 将每专家 Token 数提升至 512 与 1024(同时整除 128 / 256 / 512,零补零),并对完整输出张量执行 jax.block_until_ready(out) 彻底阻断 DCE。 Raised tokens/expert to 512 and 1024 (exact multiples of 128 / 256 / 512, zero padding) and called jax.block_until_ready(out) on the full output tensor.
Kernel 修复 × 4 原版 pallas_grouped_gemm.py 上机后暴露出 4 处编译阻塞:① 第 46 行对 expert_offsets_ref 追踪值使用 Python if 触发 TracerBoolConversionError;② in_specs 定义后未传入 pallas_call,且标量索引未放入 SMEM;③ 动态切片偏移未声明对齐,触发 Mosaic E2003 CompileTimeMosaicUnprovenMemoryAccessAlignment;④ 对 bf16 传入 Precision.HIGHEST 触发 Mosaic Bad lhs type。 Running the published pallas_grouped_gemm.py on silicon exposed 4 compiler blockers: ① Python if on traced expert_offsets_ref raised TracerBoolConversionError; ② in_specs were dead code never passed to pallas_call and scalar offsets lacked pltpu.SMEM; ③ dynamic slice offsets failed Mosaic E2003 (unproven 8-row alignment); ④ Precision.HIGHEST over bf16 failed Mosaic with Bad lhs type. 改为 @pl.when 谓词、pl.multiple_of(row, 8) / pl.multiple_of(col, 128) 静态对齐证明、移除 Precision.HIGHEST(保留 preferred_element_type=jnp.float32),并提供 VMEM 动态偏移(repaired)与 4D BlockSpec HBM 流式分块(blockspec)双实现。 Replaced with @pl.when, pl.multiple_of(row, 8) / pl.multiple_of(col, 128) static alignment proofs, removed Precision.HIGHEST (retaining preferred_element_type=jnp.float32), and shipped both VMEM dynamic-offset (repaired) and 4D BlockSpec HBM-streaming (blockspec) implementations.
规模配置(Shape)Config (Shape) Kernel 变体Variant 瓦片 bsTile bs 中位延迟 (9 次)Median (n=9) 有效吞吐Throughput 单设备 MFU / 整芯片 MFUMFU/dev / MFU/chip 相对 128 加速比(显著性)Speedup vs 128 (σ)
small
4 × 512 tok
1024 × 1024
(4.29 GFLOPs)
as_published 128 / 256 / 512 FAILED · TracerBoolConversionError (line 46 Python if on traced ref)
repaired
(VMEM + SMEM)
128×128 0.2049 ms (±0.011) 20.96 TFLOP/s 1.82% / 0.91% 1.00× (baseline)
256×256 0.1593 ms (±0.005) 26.96 TFLOP/s 2.34% / 1.17% 1.286× (3.7σ)
512×512 0.1893 ms (±0.008) 22.69 TFLOP/s 1.97% / 0.98% 1.082×
blockspec
(4D HBM stream)
128×128 0.4094 ms (±0.006) 10.49 TFLOP/s 0.91% / 0.45% 1.00× (baseline)
256×256 0.1944 ms (±0.008) 22.09 TFLOP/s 1.92% / 0.96% 2.106× (21.3σ)
512×512 0.1456 ms (±0.005) 29.50 TFLOP/s 2.56% / 1.28% 2.812× vs 128 · 1.335× vs 256
medium
8 × 512 tok
2048 × 2048
(34.36 GFLOPs, 白皮书同维)
repaired 128 / 256 / 512 VMEM OOM (88 MiB 全张量驻留超出 VMEM 容量,证明实产规模必须采用 4D BlockSpec 流式分块)
blockspec
(4D HBM stream)
128×128 4.2929 ms (±0.010) 8.00 TFLOP/s 0.69% / 0.35% 1.00× (baseline)
256×256 0.8164 ms (±0.006) 42.09 TFLOP/s 3.65% / 1.82% 5.258× (296.0σ)
512×512 0.2697 ms (±0.004) 127.40 TFLOP/s 11.04% / 5.52% 15.92× vs 128 · 3.03× vs 256
large
8 × 1024 tok
4096 × 4096
(274.88 GFLOPs)
repaired 128 / 256 / 512 VMEM OOM (384 MiB 全张量驻留超出 VMEM 容量)
blockspec
(4D HBM stream)
128×128 34.6004 ms (±0.255) 7.94 TFLOP/s 0.69% / 0.34% 1.00× (baseline)
256×256 5.3650 ms (±0.007) 51.24 TFLOP/s 4.44% / 2.22% 6.449× (115.2σ)
512×512 1.1428 ms (±0.006) 240.53 TFLOP/s 20.85% / 10.43% 30.28× vs 128 · 4.69× vs 256

核心发现与知识库交叉印证(引用 knowledgebase/wiki/hardware/tpu-7x.md 与 concepts/mxu.md):
① 为什么 512×512 比 256×256 还要快 3.03×(medium)至 4.69×(large)? concepts/mxu.md 明确指出:256×256 脉动阵列目前仅在 TPU v6e 上有正式公开信源支持(延伸至更新代际在知识库中标注为 "human-reviewed assertion — not yet stated by an ingested source",且 index.md 写作 128×128 (256×256 v6e));更重要的是,同一页给出了有效瓦片尺寸定律——有效瓦片维度必须超过 array-size × #MXUs。当存在 2 个 MXU 时,门槛是 512 而非 256。实测完美印证了该定律:在 large 规模下,256×256 相比 128×128 带来 6.45× 提速(51.24 TFLOP/s),而跨过 512×512 门槛后再翻 4.69×,单 Chiplet 达到 240.53 TFLOP/s(相对 fp32 参考余弦相似度 1.000000,最大绝对误差 0.0)。
② 单设备(Chiplet)vs 整芯片(Chip)峰值口径: 根据 hardware/tpu-7x.md:20,22,TPU7x 整芯片 bf16 峰值为 2307 TFLOP/s,由 2 个拥有独立地址空间、无 Megacore 的 Chiplet(Device)组成,因此单个 JAX Device(单 Chiplet)bf16 峰值为 1153.5 TFLOP/s。上表全部 Kernel 均在单 Device 上运行,240.53 TFLOP/s 对应单 Chiplet 20.85% MFU(整芯片 10.43% MFU)。此外,hello-tpu 在同集群通过真机门禁(device_type=native-tpu,max_abs_diff=0.111501 < 0.5);早期 6.48e-05 误差数据则属于宿主机 CPU 上的 NumPy 算术仿真,在此与真机数据严格区分。
Key findings & knowledge-base cross-verification (citing knowledgebase/wiki/hardware/tpu-7x.md & concepts/mxu.md):
① Why does 512×512 beat 256×256 by 3.03× (medium) to 4.69× (large)? concepts/mxu.md notes that a 256×256 systolic array is sourced only for TPU v6e (extension to later generations is marked "human-reviewed assertion — not yet stated by an ingested source", and index.md lists 128×128 (256×256 v6e)). More importantly, the same page states the effective-tiling rule: effective tiling dimensions must exceed array-size × #MXUs. With 2 MXUs, that threshold is 512, not 256. Silicon measurements confirm this rule: at the large scale, moving from 128×128 to 256×256 yields 6.45× (51.24 TFLOP/s), while crossing to 512×512 yields an additional 4.69× jump (30.28× over 128), reaching 240.53 TFLOP/s (cosine 1.000000, max_abs_diff 0.0 vs dense fp32 reference) on a single chiplet.
② Single-device (chiplet) vs full-chip peak caliber: Per hardware/tpu-7x.md:20,22, a TPU7x chip delivers 2307 TFLOP/s bf16 across two chiplets with separate memory spaces and no megacore, making each addressable JAX device (chiplet) 1153.5 TFLOP/s bf16. All kernels above ran on 1 device, so 240.53 TFLOP/s is 20.85% MFU per device (10.43% per chip). Separately, hello-tpu passed on silicon (device_type=native-tpu, max_abs_diff=0.111501 < 0.5), whereas the earlier 6.48e-05 check was a host-CPU NumPy tiling simulation and is kept strictly separate from device measurements.

发现的基础设施与运行时故障(共 3 项)Infrastructure & runtime faults found (3 total) 根因Root cause 修复Fix
Job 已创建但永久挂起Job created but permanently suspended 当容器只声明 resources.limits 而不声明 resources.requests 时,Kueue Webhook 无法解析配额请求,不会创建 Workload CRD,Job 因此无限等待。 When a container declares resources.limits without resources.requests, the Kueue webhook cannot resolve the quota request, no Workload CRD is created, and the Job waits forever. 为 google.com/tpu 同时下发 requests 与 limits。Emit both requests and limits for google.com/tpu.
Job 在准入阶段被拒Job rejected at admission 无条件注入了 kueue.x-k8s.io/priority-class: medium 标签,而集群中并不存在同名的 WorkloadPriorityClass。 An unconditional kueue.x-k8s.io/priority-class: medium label was injected, but no WorkloadPriorityClass of that name exists in the cluster. 仅在显式配置且可解析的优先级存在时才附加该标签。Attach the label only when an explicit, resolvable priority is configured.
首次派发在容器内异常退出First dispatch aborted in-container 脚本调用了不存在的 torch_tpu.core.sync()(报错 AttributeError: module 'torch_tpu' has no attribute 'core')。 Script called non-existent torch_tpu.core.sync() (AttributeError: module 'torch_tpu' has no attribute 'core'). PyTorch 侧通过 tensor.cpu() 往返同步;JAX/Pallas 侧对完整输出调用 jax.block_until_ready(out)。Synchronize via host round-trip (tensor.cpu()) in PyTorch or jax.block_until_ready(out) on the full output in JAX/Pallas.

前两处故障位于派发驱动(gke_job.py)而非负载本身,且对用户呈现完全相同的静默挂起现象(注:该修复目前保存在本地已安装包中,重装包会丢失,需向上游提交补丁)。 The first two faults sat in the dispatch driver (gke_job.py) rather than the workload and presented identically as a silent hang (note: these fixes are local to the installed package and will be lost on reinstall until upstreamed).

📋 6 · 已安装包中识别出的工程债 6 · Engineering debt identified in the installed package

通过将 Agent 定义与其自身 JSON Schema 交叉比对,发现 7 处内部矛盾。此处原样记录,不作裁决。 Seven internal contradictions were found by reading the agent definitions against their own JSON schemas. Recorded as observed, not adjudicated.

# 矛盾Contradiction 运行影响Operational impact
1 两份请求 Schema 的 required 中均无 run_id,但两个 Agent 契约都写着 "Require the request's run_id"。run_id is absent from required in both request schemas, yet both agent contracts state "Require the request's run_id". 约束存在于散文,而非校验。Enforcement lives in prose, not validation.
2 worker-request.json 要求 iteration 与 base_kernel_artifact;散文却说唯一给出的参数是 run_id。worker-request.json requires iteration and base_kernel_artifact; the prose says the only parameter given is run_id. Schema 与散文对"真相来源"意见不一。Schema and prose disagree on the source of truth.
3 worker 被指示返回 invalid_input,但其结果 Schema 只允许 success / failed / completed。worker is told to return invalid_input, but its result schema permits only success / failed / completed. 遵照指示会产出被自己 Schema 拒绝的响应。Following the instruction produces a response its own schema rejects.
4 max_iterations 是状态字段,但停止检查在散文中硬编码为 5。max_iterations is a state field, but the stop check hard-codes 5 in prose. 修改该字段会静默造成循环失同步。Editing the field silently desynchronizes the loop.
5 preflight-check 文档称 --allow_gpu_imports 默认开启;脚本定义的默认值是 False。preflight-check docs say --allow_gpu_imports is on by default; the script defines it as False. 省略该 Flag 的调用方会得到比文档更严格的行为。Callers omitting the flag get stricter behaviour than documented.
6 numerical-parity 仅在服务化路径可达;基准测试编排者无权委派给它。numerical-parity is reachable only from the serving path; the benchmark orchestrator cannot delegate to it. 经迁移路径做基准的模型得不到任何逐模块数值验证——而第 4 卡恰好证明这是唯一能抓住单层故障的检查。Models benchmarked through the migration path receive no per-module numerical verification — precisely the check card 4 shows is the only one that catches a single-layer fault.
7 迁移编排者文档中,本该放搜索工具构建指令的位置是一个空代码块。The migration orchestrator contains an empty code block where search-tool build instructions belong. 一个已文档化的步骤没有命令。A documented step has no command.

第 6 项是唯一具备生产后果的:基准路径可能为一个改写层已悄然出错的模型,报告出一份干净的加速比。 Item 6 is the one with production consequence: the benchmark path can report a clean speedup on a model whose rewritten layer is quietly wrong.

🚀 7 · tpu-agent-suite-experimental@0.7.0 升级实证:Qwen3.5-397B-A17B-FP8 与 GLM-5.2-FP8 算子级 Roofline、140 方案切分漏斗与 2-Layer Twin 微基准 7 · tpu-agent-suite-experimental@0.7.0 Live Verification: Qwen3.5-397B-A17B-FP8 & GLM-5.2-FP8 Operator Roofline, 140-Way Sharding Funnel & 2-Layer Twin Microbenchmark

基于升级至 tpu-agent-suite-experimental@0.7.0 的三大确定性分析引擎(roofline-analysis、sharding-strategy、workload-microbenchmark,已通过全部 5 项黄金基准自检),以下展示针对开源旗舰 Qwen/Qwen3.5-397B-A17B-FP8(单机 2×2×1 = 4 Chips / 8 Devices)与超大规模 MLA+DSA 架构 GLM-5.2-FP8(双机 2×2×2 = 8 Chips / 16 Devices)的完整逐 Device 硬件建模与切分策略推演。所有硬件指标严格遵循 TPU v7x (Ironwood) 1 Chip = 2 Devices (Dies/Cores,每 Die 96 GiB HBM) 的物理对齐基准(TPU v7x 官方文档)。 Powered by the three deterministic analysis engines in tpu-agent-suite-experimental@0.7.0 (roofline-analysis, sharding-strategy, and workload-microbenchmark, passing all 5 golden selftests), below is the complete per-device roofline and sharding derivation for Qwen/Qwen3.5-397B-A17B-FP8 (single-host 2×2×1 = 4 chips / 8 devices) and GLM-5.2-FP8 (dual-host 2×2×2 = 8 chips / 16 devices). All figures strictly align by device count (1 TPU v7x chip = 2 devices/dies, 96 GiB HBM per device; TPU v7x docs).

7.1 Qwen/Qwen3.5-397B-A17B-FP8 单机 2×2×1(4 Chips / 8 Devices)四路切分推导与三大反直觉硬件规律(基于真实 AgentX 任务 j-9fb52480) 7.1 Qwen/Qwen3.5-397B-A17B-FP8 Single-Host 2×2×1 (4 Chips / 8 Devices) 4-Way Sharding Derivation & Three Hardware Laws (AgentX Job j-9fb52480)
逐 Device (Per-Rank) 核心指标(整机 4 Chips = 8 Devices)Per-Device (Per-Rank) Metric (4 Chips = 8 Devices Total) tp8-dp1-ep
(原始任务配置)(Baseline Job)
tp4-dp2-ep tp2-dp4-ep
(单 Chip 内 TP2)(Intra-Chip TP2)
tp1-dp8-ep
(Golden 黄金布局)(Golden Layout)
每 Engine 占用 Device 数 (TP) × 独立引擎数 (DP)Devices per Engine (TP) × Independent Engines (DP) 8 dev (4 chips) × 1 4 dev (2 chips) × 2 2 dev (1 chip) × 4 1 dev (0.5 chip) × 8
每 Device 权重常驻内存 (Weights / Device)Resident Weights per Device 47.41 GiB 48.78 GiB 51.51 GiB 57.01 GiB
每 Device 路由专家数(512 专家 Top-10,EP=8)Routed Experts per Device (512 Top-10, EP=8) 64 (0.805 GiB/layer) 64 (0.805 GiB/layer) 64 (0.805 GiB/layer) 64 (0.805 GiB/layer)
每 Device GQA Q 头 / KV 头(开源权重 $H_q=32, H_{kv}=2, D_h=256$)GQA Q / KV Heads per Device ($H_q=32, H_{kv}=2, D_h=256$) 4 Q / 1 KV 4× KV 冗余 8 Q / 1 KV 2× KV 冗余 16 Q / 1 KV 零冗余 32 Q / 2 KV 零冗余
每 Token 每 Device 15 层 GQA KV 占用(FP8)15-Layer GQA KV Footprint per Token per Device (FP8) 7.5 KiB (7,680 B) 7.5 KiB (7,680 B) 7.5 KiB (7,680 B) 15.0 KiB (15,360 B)
每条序列每 Device 45 层 GDN 固定状态(fp32 Recurrent + bf16 Conv)45-Layer GDN Fixed State per Sequence per Device (fp32 + bf16) 24.42 MB / seq 48.84 MB / seq 97.69 MB / seq 195.38 MB / seq (整模型)
2-Layer Twin(1 GDN + 1 GQA)压测显存映射(Weights → Twin)2-Layer Twin (1 GDN + 1 GQA) Memory Mapping 47.41 → 2.66 GiB
(Ballast 至 72.37 GiB)
48.78 → 3.33 GiB
(Ballast 至 74.43 GiB)
51.51 → 4.36 GiB
(Ballast 至 76.49 GiB)
57.01 → 6.42 GiB
(需下调 KV 池预算以装入 96 GiB)
💡 从源码与 Roofline 推导出的 3 条关键架构定律(Qwen/Qwen3.5-397B-A17B-FP8) 💡 Three Architectural Laws Derived from Source & Roofline (Qwen/Qwen3.5-397B-A17B-FP8)
① Fused Output Gate 令 15 层 GQA 的 qkv_proj 膨胀 1.89×:在 Qwen3NextAttention 中,因 attn_output_gate = true($A_{\text{gate}}=2$),qkv_proj 形状为 [4096, 17408](Q: 8192 | Gate: 8192 | K: 512 | V: 512)而非标准 GQA 的 [4096, 9216],仅 Output Gate 分支即消耗该 GEMM 47.1% 的算力与带宽。
② 短序列并发上限由 192.06 MB/seq 的 $O(1)$ GDN 状态决定,而非 KV Cache:45 层 GDN 的 Recurrent State [B, 64, 128, 128] 必须以 float32 保存(188.74 MB/seq),外加 bfloat16 Conv1D 状态(3.32 MB/seq)。在上下文长度 $S < 12,500$ tokens 时,每条序列固定的 192 MB GDN 状态大于 15 层 GQA 的 FP8 KV Cache(15.0 KiB/token)。此外,因 $H_{kv}=2$,任何 $\text{TP} > 2$ 的配置都会触发 KV Head 跨 Rank 冗余复制(tp8-dp1 下冗余 4 倍)。
③ 512 专家 MoE 在 Decode 阶段处于深度 HBM 带宽受限区(Crossover = 1,698 Local Tokens):在 EP=8 下,算术强度公式简化为 $\text{AI} = \frac{2 N_{\text{local}} K \cdot \text{EP}}{E} = 0.3125 \times N_{\text{local}}\text{ FLOP/B}$(与 $D=4096, F=1024$ 无关)。对照 TPU v7x 单 Device P3 derated 平衡点(FP8 1,785.6 TFLOPS ÷ HBM 3,364.9 GB/s = 530.7 FLOP/B),单 Device 必须同时处理 1,698 个 local tokens 才能摆脱 HBM 带宽受限(每层固定权重读取地板 0.239 ms)。在 AgentX 真实任务(1,437 请求,MTP×3 实测平均接受长度 accept_len = 3.389 tokens/step)中,微基准直接提取出 3 个 Prefill 单元(P1/P2/P3: 16,376 chunk @ 38K/82K/138K ctx)与 3 个 Decode 单元(D1/D2/D3: 1~3 rows × 4 tokens @ 63K~185K ctx),通过 2-Layer Twin 在单机 8 Devices 上秒级复现单层耗时。
① Fused Output Gate inflates 15-layer GQA qkv_proj by 1.89×: Because attn_output_gate = true ($A_{\text{gate}}=2$) in Qwen3NextAttention, qkv_proj has shape [4096, 17408] (Q: 8192 | Gate: 8192 | K: 512 | V: 512) rather than [4096, 9216]; the gate alone accounts for 47.1% of this GEMM.
② Short-context concurrency is bounded by the 192.06 MB/seq $O(1)$ GDN state, not KV cache: The 45 GDN layers hold their recurrent state [B, 64, 128, 128] in float32 (188.74 MB/seq) plus bfloat16 conv state (3.32 MB/seq). Below $S \approx 12,500$ tokens, this fixed 192 MB/seq state exceeds the 15-layer FP8 GQA KV cache (15.0 KiB/token). Moreover, because $H_{kv}=2$, any $\text{TP} > 2$ replicates KV heads across TP ranks (4× replication at tp8-dp1).
③ 512-expert MoE in decode is deeply HBM-bandwidth-bound (Crossover = 1,698 local tokens): At EP=8, arithmetic intensity simplifies to $\text{AI} = \frac{2 N_{\text{local}} K \cdot \text{EP}}{E} = 0.3125 \times N_{\text{local}}\text{ FLOP/B}$. Against the TPU v7x per-device P3 derated balance point (530.7 FLOP/B), a device needs 1,698 local tokens before exiting the memory-bound regime (fixed HBM floor of 0.239 ms/layer for 805 MB of local expert weights). From the live AgentX trace (1,437 requests, measured MTP×3 accept_len = 3.389 tokens/step), workload-microbenchmark distills 3 prefill cells (P1/P2/P3) and 3 decode cells (D1/D2/D3) to drive a 2-layer Twin on 8 devices.
7.2 GLM-5.2-FP8(692.33 GiB,78 层 MLA + 21 层 DSA Indexer + 256 专家)双机 2×2×2(8 Chips / 16 Devices)140 方案切分漏斗与吞吐对决(ISL=8192 / OSL=1024, C=256) 7.2 GLM-5.2-FP8 (692.33 GiB, 78-Layer MLA + 21-Layer DSA Indexer + 256 Experts) Dual-Host 2×2×2 (8 Chips / 16 Devices) 140-Way Sharding Funnel & Benchmark (ISL=8192 / OSL=1024, C=256)

漏斗筛选:140 种全枚举 ➔ 124 种数学自洽(剔除 16 种非法 DCP 整除)➔ 91 种装入单 Device 88.3 GiB HBM 预算(剔除 33 种 OOM,如不开 EP 的 tp1-dp16 需 692.3 GiB/dev)➔ 22 种当前直接可跑(Shippable) + 69 种前瞻投射(Projections)。 Funnel: 140 enumerated ➔ 124 mathematically coherent ➔ 91 fit within the 88.3 GiB usable per-device HBM budget (33 OOM excluded) ➔ 22 shippable today + 69 forward-looking projections.

# 切分方案(16 Devices / 8 Chips)Sharding (16 Devices / 8 Chips) 类别 / 状态Class / Status 整机输出吞吐 (out tok/s)Fleet Out (tok/s) 整机总吞吐 (in+out tok/s)Fleet Total (tok/s) 每 Device / 每 Chip 总吞吐Per-Device / Per-Chip Total TTFT (ms) TPOT (ms) 每 Device 权重 / KV 池 (可容请求)Weights / KV Pool per Device (Cap) 主导瓶颈Bound
1 tp1-dp16-ep DP · Shippable (冠军) 5,643 (1.00×) 50,789 3,174 /dev · 6,349 /chip 1,496.0 21.99 59.51 GiB / 24.81 GiB (60.6 req × 16 = 969) HBM (ICI 11%)
2 tp1-dp8-pp2-ep DP×PP · Projection 4,528 (0.80×) 40,748 2,547 /dev · 5,094 /chip 724.2 36.35 49.33 GiB / 34.99 GiB (171 req × 8) HBM
3 tp1-dp8-pcp2-ep DP×PCP · Projection 4,181 (0.74×) 37,629 2,352 /dev · 4,704 /chip 784.1 36.73 59.51 GiB / 24.81 GiB (121 req × 8) HBM
4 tp2-dp8-ep TP×DP · Shippable (亚军) 3,913 (0.69×) 35,217 2,201 /dev · 4,402 /chip 1,360.5 22.91 51.63 GiB / 32.69 GiB (80.0 req × 8 = 640) HBM (ICI 21%)
8 tp2-dp8-dcp2-ep TP×DP · Shippable 3,010 (0.53×) 27,091 1,693 /dev · 3,386 /chip 1,893.3 25.88 51.63 GiB / 32.69 GiB (159.9 req × 8) HBM (ICI 32%)
13 tp4-dp4-ep TP×DP · Shippable 2,429 (0.43×) 21,862 1,366 /dev · 2,733 /chip 1,259.6 26.66 47.69 GiB / 36.63 GiB (89.6 req × 4) HBM (ICI 30%)
46 tp16-dp1 Pure TP · Shippable 1,577 (0.28×) 14,190 887 /dev · 1,774 /chip 103,748.9 22.05 44.65 GiB / 39.67 GiB (仅 96.9 req 整机!) HBM (KV 排队 103s)
81 tp1-dp1-pp16 Pure PP · Shippable 920 (0.16×) 8,284 518 /dev · 1,036 /chip 424.7 239.80 45.97 GiB / 38.35 GiB (1,421 req) HBM (16 级流水延迟)

核心物理洞察:① 为什么 tp1-dp16-ep 比 tp2-dp8-ep 快 1.44×、比纯 TP tp16-dp1 快 3.58×? 因为 MLA 的 KV Latent(c_kv + k_rope = 576 B/层,加上 21 层 DSA Indexer 132 B/层,全模型 46.58 KiB/token)在所有 Query Head 间共享,没有 Head 轴可供 TP 切分。在 tp16-dp1 下,16 个 Device 100% 重复存储完全相同的 KV Cache,整机最多只装得下 96.9 个请求(其余 159 个请求在队列外苦等 103.3 秒);而 tp1-dp16-ep 让 16 个 Device 组成 16 个独立 DP 引擎(通过 EP=16 跨全切片均摊 675 GiB 路由专家权重至 42.19 GiB/dev),整机并发容量瞬间扩大 16 倍至 969 个请求,实现零排队!② 当上下文拉长至 256K/1K 时:21 层 DSA Indexer: QK 跃升为 Prefill 第三大耗时算子(占 12.01%),且因 [T=1024, S=262144] 分数矩阵超出 TPU v7x 单核 8 MiB VMEM 限制触发 HBM Spill(额外消耗 871 ms HBM 带宽时间),这正是 Pallas STRAT-MLA-* 与 STRAT-SC-* 内核优化的主战场。 Core physical insight: ① Why does tp1-dp16-ep beat tp2-dp8-ep by 1.44× and pure TP tp16-dp1 by 3.58×? Because MLA's KV latent (c_kv + k_rope = 576 B/layer plus 21-layer DSA indexer 132 B/layer = 46.58 KiB/token) is shared across all query heads and has no head axis for TP to split. Under tp16-dp1, all 16 devices replicate identical KV caches, capping the entire 16-device slice at 96.9 requests (forcing 159 requests to queue for 103.3 s); whereas tp1-dp16-ep runs 16 independent DP engines while pooling the 675 GiB of routed experts across EP=16 (42.19 GiB/device), expanding fleet KV capacity to 969 requests with zero queueing. ② At 256K/1K ultra-long context: the 21-layer DSA Indexer: QK becomes the #3 prefill bottleneck (12.01% of prefill time) and spills its [T=1024, S=262144] score matrix out of the 8 MiB VMEM onto HBM (871 ms extra HBM traffic), pinpointing the exact target for Pallas STRAT-MLA-* and STRAT-SC-* kernels.

7.3 Qwen3.5-397B-A17B-FP8 跨切片 Roofline 切分扫描(2×2×1 [8 Devices] vs 2×2×2 [16 Devices])与 2026-10-23 千机灰度实测闭环 7.3 Qwen3.5-397B-A17B-FP8 Cross-Slice Roofline Sharding Sweep (2×2×1 [8 Devices] vs 2×2×2 [16 Devices]) & 2026-10-23 1,000-Host Grayscale Validation
切片 (Chips / Devices)Slice (Chips / Devs) 切分方案Sharding 每 Device 权重Weights / Dev 每序列每 Device 状态
(GDN + GQA KV @ 9K)
State / Seq / Dev
(GDN + GQA KV @ 9K)
整切片最大并发序列Max Fleet Seqs Prefill TTFT
(8K, ms)
Prefill TTFT
(8K, ms)
Decode Step T_decode
(45 GDN + 15 GQA + 60 MoE)
Decode Step T_decode
(45 GDN + 15 GQA + 60 MoE)
整切片输出吞吐
(out tok/s)
Fleet Out
(tok/s)
每 Device 输出吞吐
(out tok/s/dev)
Per-Device Out
(tok/s/dev)
2×2×1 (4C / 8D) TP1×DP8 (EP8) 56.16 GiB 192.06 + 141.56 MB 82 × 8 = 656 200.8 23.57 ms (5.24+1.74+16.59) 10,771.2 1,346.4 /dev
2×2×1 (4C / 8D) TP2×DP4 (EP8) 生产 Decode 最优 50.70 GiB 96.03 + 70.78 MB 200 × 4 = 800 149.9 22.72 ms (4.57+1.55+16.61) 11,194.0 1,399.3 /dev
2×2×1 (4C / 8D) TP4×DP2 (EP8) 47.98 GiB 48.02 + 70.78 MB 2×KV 306 × 2 = 612 157.1 24.59 ms (4.84+2.93+16.82) 10,346.1 1,293.3 /dev
2×2×1 (4C / 8D) TP8×DP1 (EP8) 46.62 GiB 24.01 + 70.78 MB 4×KV 399 × 1 = 399 153.5 28.50 ms (5.49+5.70+17.31) 8,934.8 1,116.8 /dev
2×2×2 (8C / 16D) TP1×DP16 (EP16) 33.66 GiB 192.06 + 141.56 MB 155 × 16 = 2,480 209.7 15.59 ms (3.41+1.10+11.08) 16,203.9 1,012.7 /dev
2×2×2 (8C / 16D) TP2×DP8 (EP16) 16D 冠军 28.20 GiB 96.03 + 70.78 MB 345 × 8 = 2,760 135.1 14.66 ms (2.72+0.90+11.04) 17,308.3 1,081.8 /dev
2×2×2 (8C / 16D) TP4×DP4 (EP16) 25.48 GiB 48.02 + 70.78 MB 2×KV 509 × 4 = 2,036 137.2 15.53 ms (2.82+1.59+11.12) 16,345.0 1,021.6 /dev
🔥 2026-10-23 生产千机灰度实测与 Roofline 理论的 4 大硬核印证 🔥 Four Production Validations from the 2026-10-23 1,000-Host Grayscale Rollout vs. Roofline Theory
1. TP2×DP4 (EP8) 成为生产 Decode 黄金标准(Roofline 吻合度 94.3%):Roofline 扫描显示,在单机 8 Devices 上,TP2×DP4 (EP8) 恰好卡在 $H_{kv}=2$ 的零冗余分界线上,将每序列状态(GDN+KV)减半至 166.8 MB、整机并发容量推高至 800 序列,且因 TP=2 走同颗 TPU v7x Chip 内部 600 GB/s D2D 互联,理论 T_decode = 22.72 ms。2026-10-23 生产与主干 CI(PR #2006)正式将 1P1D 标准配置从 PCP8 -> DP8 切换为 PCP8 -> DP4×TP2,5 分钟压测实测 TPOT 中位数 24.10 ms、吞吐 1,083.9 tok/s,与 Roofline 理论预测误差仅 5.7%!
2. 为什么双机独立 2×DP8 (Proxy) 比跨机 DP16 (Ray) 快 +35.3%?Roofline 表揭示:当从单机 2×2×1 (EP8) 扩至跨机 2×2×2 (EP16) 时,虽然总吞吐上升,但受跨机 ICI Z 轴 All-Gather/Reduce-Scatter 拖累,每 Device 输出吞吐从 1,399.3 tok/s/dev 降至 1,081.8 tok/s/dev(-22.7%)。真机 1P2D 实测(32K/64K C=4)完全印证:采用 Dual-Decoder Proxy 编排两个独立单机 DP8+EP8(PCP8 -> 2×DP8),比跨双机单 Ray 集群(PCP8 -> DP16)TTFT 降低 40.4%–42.5%、TPOT 提速 +32.8%–35.2%(28.44 ms vs 37.67 ms)、总输出吞吐高出 +35.3%!
3. 千机灰度 CoreHalt 发生率暴降 15×–30×:合入 DP>1 + EP + MTP 跨 Rank 步序对齐与移除 SMEM chunk 拆分上限(PR #1983 / 9f0891a)后,生产环境已进入灰度流量,CoreHalt 频率从 200 台物理机每天 1–2 次 骤降至 1,000 台物理机平均 3 天 1 次。
4. Host DRAM 三大隐形吞噬源水落石出(单机立省 30~48 GiB):① jemalloc + Linux THP 陷阱(+30 GiB/Host):jemalloc 默认 retain:true 保留虚拟地址,触发内核 khugepaged 将已释放的 2 MiB 区间重新合并回物理大页(Prefill/Decode Anon 暴涨至 129.8 / 128.2 GiB vs tcmalloc 的 98.7 / 98.7 GiB);保持默认 tcmalloc 或设置 MALLOC_CONF="thp:never,retain:false,dirty_decay_ms:1000,background_thread:true" 即可消除 ~30 GiB 虚耗且吞吐零损(1,083.9 tok/s);② vLLM v0.30 多模态 Processor 泄漏(vllm#62880):即使开启 --language-model-only,Decode Engine Core 仍加载 MM processor 多耗 ~3.4 GB/Host;③ Raiden Staging Pool 跨 Worker 池化:针对 8-Worker Decode 节点 12 GiB × 8 = 96 GiB Staging 内存,推进跨 DP Group 共享 48 GiB 内存池与单次 PD KVT 多批传输。
1. TP2×DP4 (EP8) validated as the production Decode sweet spot (94.3% Roofline accuracy): On a single 8-device host, TP2×DP4 (EP8) sits right at the $H_{kv}=2$ zero-replication boundary, halving per-sequence state to 166.8 MB, expanding fleet capacity to 800 sequences, and keeping TP=2 inside each chip's 600 GB/s D2D link for a theoretical T_decode = 22.72 ms. On 2026-10-23, both production and main CI (PR #2006) aligned 1P1D to PCP8 -> DP4×TP2, measuring 24.10 ms median TPOT and 1,083.9 tok/s throughput.
2. Why dual independent 2×DP8 (Proxy) beats cross-host DP16 (Ray) by +35.3%: Our Roofline table shows that scaling from single-host 2×2×1 (EP8) to dual-host 2×2×2 (EP16) reduces per-device throughput from 1,399.3 to 1,081.8 tok/s/dev (-22.7%) due to cross-host ICI Z-link collectives. Live 1P2D benchmarks at 32K/64K confirm that orchestrating two independent single-host DP8+EP8 nodes via a proxy (PCP8 -> 2×DP8) delivers 40.4%–42.5% lower TTFT and +35.3% higher decode throughput than a single cross-host DP16 Ray cluster.
3. CoreHalt frequency reduced by 15×–30× across 1,000 grayscale hosts: After landing PR #1983 (9f0891a, fixing DP>1 + EP + MTP step-order mismatch), production entered grayscale traffic with CoreHalt dropping from 1–2/day on 200 hosts to 1 per 3 days across 1,000 hosts.
4. Three Host DRAM culprits resolved (saving 30–48 GiB/host): ① jemalloc + Linux THP re-merging (+30 GiB/host): jemalloc default retain:true interacts with khugepaged to re-materialize freed 2 MiB ranges into physical huge pages (129.8 / 128.2 GiB vs tcmalloc's 98.7 / 98.7 GiB); using default tcmalloc or setting MALLOC_CONF="thp:never,retain:false,dirty_decay_ms:1000,background_thread:true" eliminates the 30 GiB bloat with identical throughput (1,083.9 tok/s); ② vLLM v0.30 MM processor leak (vllm#62880, +3.4 GB/host) even under --language-model-only; ③ Raiden Staging Pool sharing to cut 12 GiB × 8 = 96 GiB in half.
🧭 7.5 2026-10-22 ➔ 10-30 增量沉淀:冷启动内存、1M 上下文、NaN 隔离、PP 流水与万卡级上线教训 7.5 2026-10-22 ➔ 10-30 Increments: Cold-Start DRAM, 1M Context, NaN Isolation, PP Pipelining & Large-Fleet Launch Lessons
A. 10-30 周会:6 项新的工程结论(数字均标注每 Device / 每 Rank / 每 Host 口径) A. 10-30 sync: six new engineering results (figures labeled per device / per rank / per host)
1. 冷启动比热启动多占 ~18 GiB/host Host DRAM 的三个根因:397B Decode 单机(4 Chips = 8 Devices,DP4·TP2·EP8·MTP3)冷启动 168.25 GiB/host vs 热启动 ~150 GiB/host;分阶段探针显示差距 100% 出现在 compile_or_warm_up_model(),权重加载与 KV 初始化贡献 <0.2 GiB。 A:CompileAndLoad 产出的 executable 保留编译期专用元数据(CompilerMetadata、未去重的 BackendConfigWrapper、MosaicMlirCacheEntry、NameUniquer),而 LoadSerializedExecutable 加载的没有; B:c10d group_name 是 rank 本地字符串,使 graph fingerprint 跨 rank 不一致,follower rank 无法复用 peer 产物,8 个 worker 各自做完整 FX→MLIR→Pallas lowering; C:gc.freeze() 把前端缓存与 PiecewiseBackend.graph 冻结在 Python 堆中。 修复:编译后序列化→卸载→LoadSerializedExecutable 重载;规范化 group_name 并开启 TPU_PRECOMPILE_PEER_REPLAY=1(96 次编译中 84 次直接 replay);freeze_gc_heap() 前清缓存。三项合计消除 90.5%–93.9% 的冷热差距。
2. Raiden 窗口化子传输:staging pool 从按最坏情况预留改为按预算分配。线上平均上下文约 170K,但 staging pool 按 1M 最坏情况常驻,而容器 RSS 预算 <200 GiB/host 还要留给 KV offload。新增 TPU_RAIDEN_POOL_STAGING_BUDGET_MB / TPU_RAIDEN_POOL_STAGING_WINDOW_BLOCKS:Decode 端只开 W 个 Full-Attention 页的窗口,长请求拆成按页对齐的子传输链,GDN 状态块放在最后一段。 Qwen3.5-397B(1M,PCP8·TP1 ➔ DP4·TP2):每 Rank staging 7.63 → 1.81 GiB,1M token 分 10 段 780 ms 传完(20.9 GB/s);一个 92 层、1M 上下文的更大模型上:93.2 → 2.71 GiB/host(61×),每多一段子传输仅增加 ~2.3–3.1 ms。
3. 1M 上下文 SMEM 墙正式修复(vllm-torchtpu#2614,10-30 合入 main):原先 page_indices 全量驻留 SMEM,192 seqs × (1M / 256 页) × 4 B = 3 MiB 远超 1 MiB SMEM,只能把 max-num-seqs 压到 <64。现在页表查找移到 scheduler:以双缓冲、lane 对齐的窗口把 page_indices 流式搬进 SMEM,schedule 直接携带物理页号,主 kernel 不再读页表。Qwen3.5-397B-FP8(TP8 = 8 Devices,MTP×3)实测吞吐 +0.8% ~ +2.7%、TTFT -3.3% ~ +0.6%,无回退。
4. NaN 跨请求传染的 Kernel 级根因:0.0 × NaN = NaN。无效 V 条目有两个来源:三缓冲 VMEM 槽复用残留,以及 SEQ_ALONG_LANE 下相邻请求共享 128-lane 新 KV 页。三道防线(#1661 / #1877 / #2454 mask_invalid_v)合入后,「MTP 接受率跌 0」现象已很少出现;排查重心转向 NaN 源头(疑似 Prefill 侧)。
5. PP 缓冲式 ring hand-off(TPU_PP_HANDOFF=ring,#2527 草稿):默认 lockstep hand-off 是所有 stage 同时发起的 collective-permute,每轮等最慢的 forward,trace 显示每个 stage 每次 forward 后空等 10–14 ms;改为 isend/irecv 后算完即发给下一 stage。2x2x4(16 Chips = 32 Devices,每 Host 一个 stage = 8 Devices)、PP4·PCP8·EP、Qwen3.5-35B-FP8 上:低并发(1–4 streams)输入吞吐 +14% ~ +20%,TTFT p99 最多 1158 → 648 ms;64 streams +3%;多轮 agent 场景(prefix 命中 53%)8 会话 +44.8%、TPOT 109 → 74 ms;叠加 VLLM_PP_LAYER_PARTITION=11,9,11,9 再 +2.5% ~ +12.5%。需 libtpu ≥ 0.0.50 + torch_tpu ≥ 0.2.0。
6. GDN 内部 prefill checkpoint(#2169):Mamba align 模式下 GDN kernel 只在 chunk 末尾写 recurrent state,scheduler 只能在可缓存块边界切开 prefill,例如 PP4·PCP8(逻辑块 3072)下 8K prompt 开 MTP 被拆成 3072 + 5120、不开为 6144 + 2048。每个 GDN 层多声明 1 个 checkpoint block 后 prefill 一次跑到底,中途写边界状态;checkpoint 落在 PCP tile 内时用影子 tile 保存 / 恢复 carry。
1. Three root causes of a ~18 GiB/host cold-vs-warm start Host DRAM gap: on a single 397B decode host (4 chips = 8 devices, DP4·TP2·EP8·MTP3) cold start uses 168.25 GiB/host vs ~150 GiB/host warm; stage probes place 100% of the gap in compile_or_warm_up_model() (weights + KV init contribute <0.2 GiB). A: executables from CompileAndLoad retain compile-only metadata (CompilerMetadata, non-deduplicated BackendConfigWrapper, MosaicMlirCacheEntry, NameUniquer) that LoadSerializedExecutable does not; B: rank-local c10d group_name strings break cross-rank graph fingerprints, so every follower rank re-runs the full FX→MLIR→Pallas lowering; C: gc.freeze() pins frontend caches and PiecewiseBackend.graph in the frozen Python heap. Fixes: serialize→unload→reload via LoadSerializedExecutable; canonicalize group_name and enable TPU_PRECOMPILE_PEER_REPLAY=1 (84 of 96 compilations replayed); clear caches before freeze_gc_heap(). Together they remove 90.5%–93.9% of the gap.
2. Raiden windowed sub-transfers: budgeted staging instead of worst-case reservation. Online traffic averages ~170K tokens while staging was sized for 1M, under a <200 GiB/host RSS budget that must also host KV offload. New TPU_RAIDEN_POOL_STAGING_BUDGET_MB / TPU_RAIDEN_POOL_STAGING_WINDOW_BLOCKS bound the decode arena to a window of W full-attention pages and pull long requests as a chain of page-aligned sub-transfers (GDN state in the last one). Qwen3.5-397B (1M, PCP8·TP1 ➔ DP4·TP2): staging 7.63 → 1.81 GiB/rank, 1M tokens in 10 sub-transfers / 780 ms (20.9 GB/s); on a larger 92-layer 1M-context model: 93.2 → 2.71 GiB/host (61×) at ~2.3–3.1 ms per extra sub-transfer.
3. 1M-context SMEM wall fixed (vllm-torchtpu#2614, merged 10-30): fully resident page_indices needed 192 × (1M / 256) × 4 B = 3 MiB vs a 1 MiB SMEM, forcing max-num-seqs < 64. The scheduler now streams page_indices through double-buffered lane-aligned SMEM windows and emits physical page ids, so the main kernel never loads the page table. Qwen3.5-397B-FP8 (TP8 = 8 devices, MTP×3): throughput +0.8% ~ +2.7%, TTFT -3.3% ~ +0.6%.
4. Kernel-level cause of cross-request NaN poisoning: 0.0 × NaN = NaN. Invalid V entries come from reused triple-buffered VMEM slots and from shared 128-lane new-KV pages under SEQ_ALONG_LANE. With three defenses merged (#1661 / #1877 / #2454 mask_invalid_v), MTP drop-to-0 events are now rare; focus has moved to the NaN origin (suspected Prefill).
5. Buffered PP ring hand-off (TPU_PP_HANDOFF=ring, draft #2527): the default lockstep hand-off is one collective-permute launched by all stages together, so every round waits for the slowest forward (traces show 10–14 ms idle per stage per forward); isend/irecv forwards each output as soon as it is ready. On 2x2x4 (16 chips = 32 devices, one 8-device stage per host), PP4·PCP8·EP, Qwen3.5-35B-FP8: +14% ~ +20% input throughput at 1–4 streams (TTFT p99 up to 1158 → 648 ms), +3% at 64 streams, and +44.8% on 8 concurrent multi-turn agent sessions (53% prefix hits, TPOT 109 → 74 ms); VLLM_PP_LAYER_PARTITION=11,9,11,9 adds +2.5% ~ +12.5%. Requires libtpu ≥ 0.0.50 + torch_tpu ≥ 0.2.0.
6. GDN internal prefill checkpoints (#2169): because the GDN kernel writes recurrent state only at chunk end, Mamba-align prefix caching split prompts at cacheable block boundaries (e.g. 8K under PP4·PCP8, logical block 3072: 3072 + 5120 with MTP, 6144 + 2048 without). One extra checkpoint block per GDN layer lets prefill run to the end while writing the boundary state mid-chunk; a shadow tile saves/restores the carry when the checkpoint falls inside a PCP tile.
B. 万卡级 TPU7x 集群上线复盘(10-22):可复用的 6 条工程教训 B. Large-fleet TPU7x launch review (10-22): six reusable lessons
1. Raiden 握手元数据回归:32 个 Receiver、162 个 cache pool 的 TP2·PCP16 ➔ TP4·DCP8 拓扑下,每次请求握手都把完整 Receiver 地址表(~10 MB)序列化一遍,哪怕 prompt 只有 87 token;单线程 start_transfer() 耗时 239 ms,KV 传输启动率被卡在 4.2 req/s。去掉冗余转发后 Prefill 分发 212 → 6.1 ms(35×),协调吞吐 4.2 → 22–30 req/s。
2. ICI「慢链路」假象:在一个 @jax.jit 里串行放入 32 对以上 ppermute 测 2x2x4 链路,会耗尽 36 个自定义 sync flag,后面的链路静默回退为 GLOBAL barrier,测出 93 → ~50 GB/s 的假降速。改用边着色的多对并行轮次(或统一全局 barrier 后计时),可在 <10 s 内无误报地扫完一个 4x4x4 Cube(64 Chips = 128 Devices)。
3. 「假健康」降频节点:某台主机因功率传感器故障降频,单 Device 算力 34.4 TFLOP/s(约正常的 1/30)、HBM 带宽 365 GB/s(约 1/9),却仍被报告为 HEALTHY,把一个数千芯片的同步作业拖慢 12.5×(10.7 → 212 s/step)。被动健康信号不够,需要准入 / 周期性的短时 GEMM + HBM 带宽探针。
4. 224 vCPU × 2,000+ 节点会击穿 K8s 默认值:tpu-device-plugin 默认 80Mi 内存上限导致约 78% 节点反复 OOMKilled(调到 160–512Mi);随节点数扩容的 konnectivity-agent 打满系统节点 ARP 表(gc_thresh3=1024),引发 webhook / DNS 大面积超时;kube-state-metrics 需从 500Mi 调到 ~10Gi。
5. Pallas disable_bounds_checks=True 越界 = 硬件致命中断:越界的 page / block 索引会触发 SparseCore addr_out_of_range;主进程崩溃约 4 秒后,卡在 reduce-scatter 的其余 worker 会刷出次生错误日志,排障要锁定毫秒级最早的那条。
6. 运维铁律:多机切片未拆分前不要对单台 VM 直接 stop/start(会触发固件错误锁死、健康硬件被送修);容器设 ulimit -c 0(数百 GB 进程的 core dump 会写满小启动盘,触发 0 秒宽限期驱逐);分层收紧拓扑发现超时(默认值可达 4,000 s,单点故障会让上千台主机空等)。
1. Raiden handshake metadata regression: with 32 receivers / 162 cache pools (TP2·PCP16 ➔ TP4·DCP8) every request serialized the full receiver address table (~10 MB) even for an 87-token prompt; single-threaded start_transfer() took 239 ms and capped KV transfer launches at 4.2 req/s. Dropping the redundant forwarding cut prefill dispatch 212 → 6.1 ms (35×) and raised coordination throughput to 22–30 req/s.
2. False "slow ICI link": packing 32+ single-pair ppermutes into one @jax.jit to test 2x2x4 links exhausts the 36 custom sync flags; remaining links silently fall back to GLOBAL barriers and read 93 → ~50 GB/s. Edge-colored multi-pair rounds (or forced global barriers with post-barrier timing) scan a full 4x4x4 cube (64 chips = 128 devices) in <10 s with no false alarms.
3. "Zombie healthy" throttled hosts: a faulty power sensor throttled one host to 34.4 TFLOP/s per device (~1/30) and 365 GB/s HBM (~1/9) while it still reported HEALTHY, slowing a multi-thousand-chip synchronous job 12.5× (10.7 → 212 s/step). Passive health signals are not enough; use short GEMM + HBM probes at admission and periodically.
4. 224 vCPU × 2,000+ nodes break K8s defaults: tpu-device-plugin's 80Mi limit OOM-killed ~78% of nodes (raise to 160–512Mi); node-count-scaled konnectivity-agent overflowed the system nodes' ARP table (gc_thresh3=1024), timing out webhooks and DNS; kube-state-metrics needs ~10Gi instead of 500Mi.
5. Out-of-range DMA under Pallas disable_bounds_checks=True is a hardware fatal: a bad page/block index raises SparseCore addr_out_of_range; ~4 s later, peers blocked in reduce-scatter log secondary errors, so triage must key on the earliest millisecond timestamp.
6. Ops rules: never stop/start a single VM of a multi-host slice before deforming the slice (firmware-error latch, healthy hardware sent to repair); set ulimit -c 0 (core dumps of multi-hundred-GB processes fill the small boot disk and trigger 0-second-grace eviction); tier topology-discovery timeouts (defaults up to 4,000 s leave thousands of hosts idle on one failure).
🏛️ 8. CoreML 混合 Mamba/GDN 页对齐数学(1,152 tokens/page)、2-Layer Twin 微基准与 7 层编译器栈 × 6 大 PR 架构收官 8. CoreML Hybrid Mamba/GDN Page Alignment Math (1,152 tokens/page), 2-Layer Twin Microbenchmark & 7-Layer Stack Synthesis
• CoreML 混合 Mamba/GDN 页对齐精确推导(397B.md & tpu_block_size_utils.py:_derive_tpu_block_slot_config):在 mamba-cache-mode=align / TPU_VLLM_ENABLE_UNIFIED_BLOCK_POOL=1 下,单层 GDN 原始状态槽大小为 mamba_raw_state_bytes = 542,720 B(conv_state = 18,432 B @ TP=8 spec=3 或 49,152 B @ TP=2 spec=1,加上 recurrent_state = 524,288 B FP32 / TP=2 BF16 或 262,144 B TP=8 BF16)。由于在 TP ≥ 2 下每 Rank 仅持有 1 个 GQA KV Head(配合 PR #608 SEQ_ON_LANE),单层 Attention KV 恰好消耗 1 × 2 (K,V) × 256 × 1 B (FP8) = 512 B/token/layer;因此容纳 1 个 GDN 状态槽需 ⌈542,720 / 512⌉ = 1,060 tokens,向上取整至 256 的整数倍即得 block_size = 1,152 tokens/page(589,824 B ≥ 542,720 B)。
• 单机 2-Layer Twin 微基准(1 linear_attention + 1 full_attention + MTP):通过 hf_overrides: {"text_config": {"num_hidden_layers": 2, "layer_types": ["linear_attention", "full_attention"]}} 在单台 2x2x1(8 Devices)上复刻 397B 的精确单 Rank 形状(q_heads_projected=16、attn_kv_heads_per_rank=1、gdn_k_heads_per_rank=8、gdn_v_heads_per_rank=32、experts_per_rank=64、cache_block_tokens=1,152、pages_per_rank=4,007、kv_pool_tokens_per_rank=4,617,186,HBM Ballast 锁定 76.49 GiB/device)。
• 7 层栈 × 6 大 PR 对位(4 个 0% Pallas + 2 个外科手术级 Pallas):Layer 1 (PyTorch & Distributed, 0% Pallas) DP4×TP2 + EP8、PR #1458 BF16 GDN State(+12.2% KV 池、+9.1% Decode)、PR #106 统一页池、tpu-connector / TPUConnectorHMA / Raiden (PR #1183, #1558, #1677) 直传完整 Tail + 45 层 GDN 状态 ~349 MB(零 Tail 重算 + 双 200G NIC NUMA 绑核 367 Gbps);Layer 2 (Graph Capture, 0% Pallas) √2 几何桶 + PR #2277 AOT Export;Layer 3 (StableHLO Views, 0% Pallas) PR #766 4D 零拷贝 bitcast(DMA 指令 366 ➔ 87,GDN 5.3× 加速 4.12 ➔ 0.774 ms);Layer 4 (XLA [8, 128] Tile Layout) PR #608 SEQ_ON_LANE(4 个 FP8 Token 打包进 1 个 32-bit 槽,1,024 ➔ 512 B/page);Layer 5 左 (XLA Fusions, 0% Pallas) PR #371 / #509 小 Batch 3 行 F.one_hot + MXU matmul(c64 +14.5% 吞吐) vs. Layer 5 右 (Surgical Pallas) PR #543 + #891 全层 SPMD Fused MoE(双缓冲异步 DMA 掩盖 1,190 µs / 22% ICI Reduce-Scatter,单层 4.8×、E2E 2× 吞吐 10,614 ➔ 21,297 tok/s)、PR #1008 RPA v3 MTP 验证(10.2 ➔ 5.6 ms)、PR #2276–#2361 声明式 Megakernel + pool_alias (#2227);Layer 6 (ICI Collectives) PR #65 / #129 PCP=8 All-to-All(128K TTFT 32s ➔ 4.62s);Layer 7 (PJRT Runtime & Host OS, 0% Pallas) PR #1810 Peer Replay(DP=8 启动 741s ➔ 270s)与 PR #2141 / #2068 Host DRAM & Linux THP 释放(440 ➔ 230 GiB)。
• Step 0 Roofline 终极闭环:单步 HBM 权重读取物理下界 6.73 ms,DP4×TP2 + EP8 解析预测 22.72 ms 对决 9,216 芯片生产灰度实测 24.10 ms,误差 < 6%(5.7%)。
• CoreML Exact Hybrid Mamba/GDN Page Alignment Math (397B.md & tpu_block_size_utils.py:_derive_tpu_block_slot_config): In mamba-cache-mode=align / TPU_VLLM_ENABLE_UNIFIED_BLOCK_POOL=1, mamba_raw_state_bytes = 542,720 B per layer (conv 18,432 B at TP=8 spec=3 or 49,152 B at TP=2 spec=1 + recurrent 524,288 B FP32 / TP=2 BF16 or 262,144 B TP=8 BF16). Because 1 GQA layer per rank at TP ≥ 2 uses 1 KV head × 2(K,V) × 256 head_dim × 1 B (FP8) = 512 B/token/layer (with PR #608 SEQ_ON_LANE), fitting one GDN slot requires ⌈542,720 / 512⌉ = 1,060 tokens, rounded up to a 256-token multiple = 1,152 tokens/block (589,824 B ≥ 542,720 B).
• Single-Host 2-Layer Twin Microbenchmark (1 linear_attention + 1 full_attention + MTP): Uses hf_overrides: {"text_config": {"num_hidden_layers": 2, "layer_types": ["linear_attention", "full_attention"]}} on a single 2x2x1 host (8 devices) with identical per-rank shapes (q_heads_projected=16, attn_kv_heads_per_rank=1, gdn_k_heads_per_rank=8, gdn_v_heads_per_rank=32, experts_per_rank=64, cache_block_tokens=1,152, pages_per_rank=4,007, kv_pool_tokens_per_rank=4,617,186, HBM ballast to 76.49 GiB/device).
• 7-Layer Stack Mapping of the 6 Epic PRs (4 with 0% Pallas + 2 Surgical Pallas Kernels): Layer 1 (PyTorch & Distributed — 0% Pallas) DP4×TP2 + EP8, PR #1458 BF16 GDN state (+12.2% KV pool, +9.1% decode), PR #106 Unified Pool, and tpu-connector / TPUConnectorHMA / Raiden (PR #1183, #1558, #1677) shipping full tail + 45-layer GDN state ~349 MB with zero tail recompute and dual-200G NIC NUMA pinning at 367 Gbps; Layer 2 (Graph Capture — 0% Pallas) √2 Precompile Buckets + PR #2277 AOT Export; Layer 3 (StableHLO Views — 0% Pallas) PR #766 4D zero-copy bitcast cutting DMAs 366 ➔ 87 and speeding up GDN 5.3× (4.12 ➔ 0.774 ms); Layer 4 (XLA [8, 128] Tile Layout) PR #608 SEQ_ON_LANE packing 4 FP8 tokens into 1×32-bit slot (1,024 ➔ 512 B/page); Layer 5 Left (XLA Fusions — 0% Pallas) PR #371 / #509 3-line PyTorch F.one_hot + matmul on the 256×256 MXU (+14.5% at c64) vs. Layer 5 Right (Surgical Pallas) PR #543 + #891 Whole-Layer SPMD Fused MoE hiding 1,190 µs (22%) exposed ICI Reduce-Scatter (4.8× layer speedup, 2× E2E 10,614 ➔ 21,297 tok/s), PR #1008 RPA v3 MTP verification (10.2 ➔ 5.6 ms), and PR #2276–#2361 Declarative Pallas Megakernel with pool_alias (#2227); Layer 6 (ICI Collectives) PR #65 / #129 PCP=8 All-to-All cutting 128K TTFT 32s ➔ 4.62s; Layer 7 (PJRT Runtime & Host OS — 0% Pallas) PR #1810 Cross-Rank Peer Replay (741s ➔ 270s) & PR #2141 / #2068 Startup Host-RAM & Linux THP release (440 ➔ 230 GiB).
• Step 0 Roofline Closure: 6.73 ms HBM weight-read floor, 22.72 ms analytic prediction for DP4×TP2 + EP8 vs. 24.10 ms 9,216-chip production canary (< 6% / 5.7% error).