展示 Qwen 3.5 在单层 Transformer Block 内的完整张量流动生命周期，包含 GQA (3:1)、QK-Norm (Qwen 原创特色)、★★★ KV Cache 显存物理驻留点 ★★★ 以及 MoE (共享专家 + 512 细粒度路由专家)。精确映射 Xprof Trace 中的 fusion.243 (QKV GEMM) 算子。
⚠ 部署形态：Prefill 节点 PCP=8 + EP=8（8 芯片），Decode 节点 TP=2 × DP=4 + EP=8（8 芯片），一对 PD 共 16 芯片。PCP / TP / EP 是同一组 8 芯片上的不同切分轴，不相乘。两阶段的张量形状与显存数字仍需分开读。
The full tensor lifecycle inside one Qwen 3.5 Transformer block: GQA (3:1), QK-Norm (a Qwen signature), the ★★★ physical KV-cache residency point ★★★, and MoE (shared expert + 512 fine-grained routed experts) — mapped onto the fusion.243 (QKV GEMM) operator from an Xprof trace.
⚠ Deployed topology: prefill node PCP=8 + EP=8 (8 chips), decode node TP=2 × DP=4 + EP=8 (8 chips), 16 chips per PD pair. PCP / TP / EP are different partitioning axes over the same 8 chips — they do not multiply. Shapes and memory figures still have to be read per-phase.
bodaborg-tpu7x-nap 上 8 芯片 prefill 节点（PCP=8 + EP=8）· 60 层混合架构（45 GDN + 15 GQA） 那次压测，端到端 4.62 s。
这就是本文档的主口径 —— 联合优化文档记录的上线部署形态即为「1 Prefill node (8 chips, PCP8+EP8) → 1 Decode node (8 chips)，一对共 16 芯片」，
PCP / TP / EP 是同一组 8 芯片上的不同切分轴，不相乘，不存在 64 芯片实例。
4.62 s 的对照组是同样 8 芯片上的 DP8（128K 单请求 TTFT 32 s），因此 6.9× 是「同硬件下 DP → PCP 的收益」，不是扩容到更多芯片的超线性加速。
This timeline replays one stress run on bodaborg-tpu7x-nap: an 8-chip prefill node (PCP=8 + EP=8) running a 60-layer hybrid build (45 GDN + 15 GQA), 4.62 s end to end.
This is the document's main configuration — the joint-optimization record describes the deployed form as "1 prefill node (8 chips, PCP8+EP8) → 1 decode node (8 chips), 16 chips per pair".
PCP / TP / EP are different partitioning axes over the same 8 chips and do not multiply; there is no 64-chip instance.
The 4.62 s baseline is DP8 on the very same 8 chips (128K single-request TTFT 32 s), so the 6.9× is "DP → PCP on identical hardware", not superlinear scaling to more chips.
① 每卡 Q 头 12 → 48，KV 头 4 → 16（TP=2）；
② fusion.243 的输出列 2560 → 10240，o_proj 为 6144 → 4096；
③ Attn 后的 All-Reduce 域从 8 卡缩为 2 卡；4 个 DP 副本各处理不同请求，互不通信；
④ PCP 消失：decode 时 T=1，序列维无可切分，因此没有 ring/all-gather，每卡持有完整序列的 KV；
⑤ 注意力是 DP=4 而 MoE 是 EP=8，进出 MoE 需要一次 DP↔EP 的 All-to-All 重排。
⑥ PD 交接：KV 要从 prefill 8 卡的布局重排为 decode 8 卡上的 (16 头, 全序列) —— 跨节点搬运且需重分布，是 PD 分离的主要固有开销（#554 / #582）。本图未画。
① Q heads per chip 12 → 48, KV heads 4 → 16 (TP=2);
② fusion.243 output columns 2560 → 10240, o_proj becomes 6144 → 4096;
③ the post-attention All-Reduce domain shrinks from 8 chips to 2; the 4 DP replicas serve different requests and never talk;
④ PCP disappears: at decode T=1, so there is nothing to split along the sequence — no ring/all-gather, and each chip holds the full-length KV;
⑤ attention is DP=4 while MoE is EP=8, so entering and leaving the MoE needs a DP↔EP All-to-All reshuffle.
⑥ PD handoff: KV is reshaped from the prefill layout on 8 chips into (16 heads, full S) on the 8 decode chips — a cross-node move plus a redistribution, and the dominant intrinsic cost of PD disaggregation (#554 / #582). Not drawn here.
bodaborg-tpu7x-nap · 2×4 上 8 芯片 prefill 节点 · 45 GDN + 15 GQA 的压测。
引用前请先看数字溯源：
✓ 有联合优化文档背书 —— 128K 单请求 TTFT DP8 32 s → PCP8 4.62 s、decode 5,650 tok/s（TP2DP4 + owner_output 分支）、单网卡 P→D 峰值 > 12.5 GB/s（线速 25 GB/s，达标线为其 50%）、prefill 分层耗时 full 492 µs × 15 层 / linear 774 µs × 45 层 / MoE 1.41 ms × 60 层。
△ 自洽但属推算 —— 1.92 GB 可由 [131072, 15, 4, 128] FP8 × 2(K,V) 反算得到，是单卡份额（精确值 1.875 GiB；8 卡合计 15 GiB = 整请求 15 层 FA 的 KV），账对得上；但 450 ms → 138 ms（3.2×）是 1.92 GB 分别除以 4.2 / 13.6 GB/s 的算术结果，而文档只背书「峰值超过 12.5 GB/s」。按 12.5 GB/s 应为 ~154 ms。
✗ 与实测冲突 —— 22.45 → 11.20 ms/tok（2.0×）：实测 TPOT 为 25.4 ~ 28.4 ms，MTP 在 coding 场景的典型加速比是 1.34× 而非 2.0×。
Absolute latencies come from a stress run on bodaborg-tpu7x-nap · 2×4: an 8-chip prefill node running 45 GDN + 15 GQA layers.
Check provenance before quoting:
✓ Corroborated by the joint-optimization record — 128K single-request TTFT 32 s under DP8 → 4.62 s under PCP8, decode 5,650 tok/s (TP2DP4 on the owner_output branch), single-NIC P→D peak > 12.5 GB/s (line rate 25 GB/s; the target was 50% of it), per-layer prefill cost full 492 µs × 15 / linear 774 µs × 45 / MoE 1.41 ms × 60.
△ Self-consistent but derived — 1.92 GB falls out of [131072, 15, 4, 128] FP8 × 2 (K,V) and is the per-chip share (exactly 1.875 GiB; 8 chips = 15 GiB, i.e. the whole request's KV over 15 FA layers), so the arithmetic checks out. But 450 ms → 138 ms (3.2×) is just 1.92 GB divided by 4.2 and 13.6 GB/s, whereas the record only backs "peak above 12.5 GB/s" — which would give ~154 ms.
✗ Contradicted by measurement — 22.45 → 11.20 ms/tok (2.0×): measured TPOT is 25.4–28.4 ms, and MTP's typical coding-workload speedup is 1.34×, not 2.0×.
传统模式是「先算完再通信」或「先等数据到齐再开算」，计算单元会大量闲置等待，形成通信气泡（bubble）。 在 PCP8 Ring Attention 里，TPU 的 MXU 计算引擎与 ICI 片间互联 DMA 引擎在硬件层面完全解耦，配合双缓冲，计算与通信 100% 异步并发： The naive pattern is compute, then communicate — or wait for the data, then start computing — which leaves the compute units idle and opens a communication bubble. Under PCP8 ring attention the TPU's MXU compute engine and its ICI interconnect DMA engine are fully decoupled in hardware; combined with double buffering, compute and communication run 100% concurrently:
k 块 KV 时，独立 DMA 引擎已经通过 ICI 把邻居的第 k+1 块推进片上 SRAM 备用。
While the compute unit chews on KV chunk k, an independent DMA engine has already pulled the neighbour's chunk k+1 over ICI into on-chip SRAM.
这段窗口完整切下了 Qwen 3.5 397B 进入 Attention 模块时的 QKV 融合投影流水线。三个算子合计约 71.06 µs，加上指令发射与对齐填满 100 µs。 The window captures one complete fused QKV projection pipeline as Qwen 3.5 397B enters its attention block. The three ops total ≈71.06 µs; issue and alignment overhead fill the remaining 100 µs.
• 输入：bf16[4096, 4096]（来自 Pre-RMSNorm 的激活）
• 输出：f8e4m3fn[4096, 4096]
• 逻辑：扫描激活绝对值极大值求 Scale，再把 BF16 压成 FP8 送入 MXU。占整个投影 ~25%。
• Input: bf16[4096, 4096] (activations out of Pre-RMSNorm)
• Output: f8e4m3fn[4096, 4096]
• Logic: scan for the absolute max to derive a scale, then squeeze BF16 into FP8 for the MXU. About 25% of the whole projection.
• 算子：convolution.708（MXU 脉动阵列）
• 计算：[4096, 4096] × [4096, 2560] → f32[4096, 2560]
• 流水线反量化：片上连续两次广播乘（mul.14009 / mul.14007）乘以输入与权重 Scale，再 convert 回 bf16。
• Op: convolution.708 (MXU systolic array)
• Compute: [4096, 4096] × [4096, 2560] → f32[4096, 2560]
• Pipelined dequant: two back-to-back on-chip broadcast multiplies (mul.14009 / mul.14007) apply the input and weight scales, then convert back to bf16.
从单卡 bf16[4096, 2560] 中切分：
• Query 分支（slice.7534）：前 [0:1536] 列 = 12 个 Q 头（12 × 128），送往 Q-RMSNorm；
• Key & Value 分支（copy-start.766）：后 [1536:2560] 共 1024 列 = 4 个 K 头 + 4 个 V 头，送往 K-RMSNorm 并写入 ★★★ KV Cache ★★★。
Splitting the per-chip bf16[4096, 2560]:
• Query branch (slice.7534): columns [0:1536] = 12 Q heads (12 × 128), on to Q-RMSNorm;
• Key & Value branch (copy-start.766): columns [1536:2560], 1024 wide = 4 K heads + 4 V heads, on to K-RMSNorm and then written to the ★★★ KV cache ★★★.
| 算子 | 工作量 | 实测 | 达成率 | 结论 |
|---|---|---|---|---|
| Operator | Work | Measured | Attainment | Verdict |
fusion.243 GEMM |
2×4096×4096×2560 = 85.9 GFLOP 算术强度arithmetic intensity 1781 FLOP/byte |
50.00 µs → 1.72 PFLOP/s | 37.2% of 4614 TFLOPS (fp8) | 理想 18.6 µs，慢 2.7× — 确认为真实低效 ideal 18.6 µs, 2.7× slower — a genuine inefficiency |
clamp_convert_fusion.117 |
两遍：absmax 读 33.5 + convert 读 33.5 + 写 16.8 = 83.9 MB two passes: absmax read 33.5 + convert read 33.5 + write 16.8 = 83.9 MB | 17.81 µs → 4.71 TB/s | 64% of 7.37 TB/s | 访存受限算子，达成率健康（按单遍 50.3 MB 算只有 38%，与实际不符） memory-bound and healthy (a single-pass model would give only 38%, which does not match) |
① GEMM 的 37% 不是 roofline 造成的，两条辩解都已排除：
• 算术强度 1781 FLOP/byte ≫ v7 拐点 312.9（2306 TFLOPS bf16 ÷ 7.37 TB/s），纯 compute-bound；
• v7 的 MXU 为 256×256，本 GEMM 切成 M=16 × K=16 × N=10 个 tile 全部整除，无边角浪费。
→ 嫌疑集中在 epilogue：尾部两次反量化广播乘与 convert 是否被计入这 50 µs。下一步应把 fusion 内部逐指令展开。
② 量化那 17.81 µs 应融进上游 Pre-RMSNorm 的 epilogue：激活 [4096,4096] bf16 = 32 MiB，v7 VMEM 为 64 MiB/core，整块可驻留。在 RMSNorm 输出还在 VMEM 时完成 absmax 与 convert，可省掉 两遍 HBM 读（约 67 MB）。
③ 另需单独检查注意力核心的 Q·Kᵀ：head_dim = 128 只填满 256 宽 MXU 的一半，那里才是形状对齐的真实受害者（本 GEMM 收缩维是 4096，不受影响）。
① The GEMM's 37% is not a roofline effect — both excuses are ruled out:
• arithmetic intensity 1781 FLOP/byte ≫ the v7 ridge point of 312.9 (2306 TFLOPS bf16 ÷ 7.37 TB/s), so it is purely compute-bound;
• the v7 MXU is 256×256, and this GEMM tiles into M=16 × K=16 × N=10 — all exact, no ragged edges.
→ Suspicion lands on the epilogue: whether the two trailing dequant broadcast multiplies and the convert are counted inside these 50 µs. Next step is an instruction-level expansion of the fusion.
② Fold that 17.81 µs quantization into the Pre-RMSNorm epilogue: the [4096,4096] bf16 activation is 32 MiB against 64 MiB/core of v7 VMEM, so the whole tile stays resident. Doing absmax and convert while the RMSNorm output is still in VMEM saves two HBM read passes (~67 MB).
③ Separately audit Q·Kᵀ in the attention core: head_dim = 128 fills only half of a 256-wide MXU — that is where shape misalignment actually bites (this GEMM contracts over 4096 and is unaffected).
本段 trace 同时满足两个特征：QKV 输出 2560 列 = 20480/8（权重按头 8 路切 → TP=8），以及 M = 4096 = 32768/8（token 维 8 等分 → PCP=8 或 chunked prefill 的单块大小）。
⚠ 存疑未决：上线形态是 8 芯片 PCP8 + EP8，其中注意力权重是复制而非按头切分，不应出现 2560 列。因此本段 trace 更可能来自另一个 TP=8 的 prefill 变体（联合优化文档确实并列记录过 DP / TP / PCP 三种 prefill 配置），或 M=4096 只是 chunked prefill 的块大小而非 PCP 切分。在拿到原始 run 配置前，不要据此推断芯片数。
Decode (TP=2) 下同一算子输出应为 10240 列 且 M 不再被 8 整除，不要用本页的 µs 数字推 decode 延迟。
This trace satisfies two signatures at once: QKV output of 2560 columns = 20480/8 (weights split 8-way by head → TP=8), and M = 4096 = 32768/8 (the token axis in eight parts → PCP=8, or one chunked-prefill block).
⚠ Unresolved: the deployed form is 8 chips with PCP8 + EP8, where attention weights are replicated rather than head-sharded, so 2560 columns should not appear. This trace more likely comes from a different TP=8 prefill variant (the joint-optimization record does list DP / TP / PCP prefill configurations side by side), or M=4096 is simply the chunked-prefill block size rather than a PCP shard. Do not infer a chip count from this until the original run config is available.
At decode (TP=2) the same op should emit 10240 columns with M no longer divided by 8 — do not use the µs figures on this page to infer decode latency.
| 计算阶段 / 算子 | 单卡形状 (Prefill, 取自 TP=8 变体 trace ⚠) | 全局形状 | 精度 | 物理驻留 | 功能与架构含义 |
|---|---|---|---|---|---|
| 输入残差流 | [B, T/8, 4096] |
[B, T, 4096] |
bf16 | HBM | 主干隐藏激活；PCP 沿序列维切 8 份 |
| Pre-Attn RMSNorm | [B, T/8, 4096] |
[B, T, 4096] |
bf16 | VPU | 逐 token 归一化，与切分方式无关 |
| FP8 动态量化 (clamp_convert) | [4096, 4096] |
— | f8e4m3fn | VMEM (32 MiB / 64 MiB) | 两遍 absmax + convert，17.81 µs，达峰值带宽 64% |
| ★ QKV GEMM (fusion.243) | [4096, 2560] |
[T, 20480] |
FP8 → bf16 | MXU 256×256 | M=4096 是 PCP 本地 token；2560 是 TP 列分片。50.00 µs，峰值 37.2% |
| QKV Slice (slice.7534) | Q: [B, T/8, 12, 128]KV: [B, T/8, 8, 128] |
Q: [B, T, 96, 128]KV: [B, T, 64, 128] |
bf16 | VPU | 从 2560 列解构出 12 个 Q 头与 4+4 个 KV 头 |
| Q-RMSNorm & K-RMSNorm | [B, T/8, 12, 128][B, T/8, 4, 128] |
[B, T, 96, 128][B, T, 32, 128] |
bf16 | VPU | Qwen 核心设计：限制 Q/K 模长，杜绝超长文本 Softmax 溢出。V 不参与 |
| ★★★ KV Cache (Prefill) ★★★ | [B, S/8, 4, 128] × 2 |
[B, S, 32, 128] × 2 |
fp8 / bf16 | HBM | 单卡 0.47 GiB @32K fp8（15 层 FA）。切分轴存疑：trace 形状读作 4 头 × 全序列，而记录中的部署形态 PCP=8 应为 32 头 × S/8 —— 两者体积完全相同，详见 Xprof 页存疑说明 |
| ★★★ KV Cache (Decode) ★★★ | [B, S, 16, 128] × 2 |
[B, S, 32, 128] × 2 |
fp8 / bf16 | HBM | 只切头不切序列，且 DP=4 不切 KV → 单卡 1.88 GiB @32K fp8（15 层 FA），是 prefill 单卡的 4 倍，容量瓶颈以此行为准 |
| PD 交接 KV Reshard | Prefill 8 卡 (单卡 0.47 GiB @32K) → Decode 8 卡 (16 头, 全序列, 1.88 GiB) |
— | fp8 / bf16 | 跨实例网络 | 头与序列两个维度同时重分布，并跨节点搬运，是 PD 分离的主要固有开销（对应 vllm-torchtpu#554 / #582） |
| PCP Ring / All-Gather | [B, S, 4, 128] × 2 (补齐后) |
— | bf16 | ICI | 仅 Prefill：每 rank 只有 S/8 个 token，注意力需沿 CP 维环形传递 KV |
| GQA FlashAttention | [B, T/8, 12, 128] |
[B, T, 96, 128] |
bf16 | MXU + VPU | 12 个 Q 头共享 4 组 KV (3:1)。head_dim=128 只填满 256 宽 MXU 一半 |
| Out Proj & All-Reduce | [B, T/8, 4096] |
[B, T, 4096] |
bf16 | ICI | 行并行汇聚：Prefill 8 卡，Decode 仅 2 卡（DP 副本之间不通信） |
| Qwen 共享专家 (Shared) | [B, T/8, 4096] |
[B, T, 4096] |
bf16/fp8 | MXU | 常驻激活，中间维 5632，承载通用常识 |
| 路由专家 (512 选 Top-8) | [B, T/8, 4096] |
[B, T, 4096] |
bf16/fp8 | ICI All-to-All (EP=8) | 中间维 1408；EP=8 下每卡常驻 64 个专家，两阶段一致。Decode 时注意力 DP=4 与此处 EP=8 不一致，需额外一次重排 |
| Stage / operator | Per-chip shape (prefill, from the TP=8-variant trace ⚠) | Global shape | Precision | Resides in | What it means |
|---|---|---|---|---|---|
| Input residual stream | [B, T/8, 4096] |
[B, T, 4096] |
bf16 | HBM | Backbone hidden activations; PCP splits the sequence 8 ways |
| Pre-Attn RMSNorm | [B, T/8, 4096] |
[B, T, 4096] |
bf16 | VPU | Per-token normalization; indifferent to how tokens are sharded |
| FP8 dynamic quant (clamp_convert) | [4096, 4096] |
— | f8e4m3fn | VMEM (32 of 64 MiB) | Two passes, absmax + convert; 17.81 µs at 64% of peak bandwidth |
| ★ QKV GEMM (fusion.243) | [4096, 2560] |
[T, 20480] |
FP8 → bf16 | MXU 256×256 | M=4096 is the PCP-local token count; 2560 is the TP column shard. 50.00 µs, 37.2% of peak |
| QKV slice (slice.7534) | Q: [B, T/8, 12, 128]KV: [B, T/8, 8, 128] |
Q: [B, T, 96, 128]KV: [B, T, 64, 128] |
bf16 | VPU | Recovers 12 Q heads and 4+4 KV heads out of the 2560 columns |
| Q-RMSNorm & K-RMSNorm | [B, T/8, 12, 128][B, T/8, 4, 128] |
[B, T, 96, 128][B, T, 32, 128] |
bf16 | VPU | A Qwen signature: bounds the Q/K norms so long-context softmax cannot blow up. V is not normalized |
| ★★★ KV cache (prefill) ★★★ | [B, S/8, 4, 128] × 2 |
[B, S, 32, 128] × 2 |
fp8 / bf16 | HBM | 0.47 GiB per chip at 32K fp8 (15 FA layers). The axis is unresolved: the traced shape reads as 4 heads × full S, while the documented PCP=8 form implies 32 heads × S/8 — identical volume either way; see the open question on the Xprof tab |
| ★★★ KV cache (decode) ★★★ | [B, S, 16, 128] × 2 |
[B, S, 32, 128] × 2 |
fp8 / bf16 | HBM | Heads split, sequence not, and DP=4 does not shard KV → 1.88 GiB per chip at 32K fp8 (15 FA layers), 4× the prefill figure; this row sets the capacity limit |
| PD handoff KV reshard | 8 prefill chips (0.47 GiB each @32K) → 8 decode chips (16 heads, full S, 1.88 GiB) |
— | fp8 / bf16 | Cross-instance network | Redistributes along both the head and sequence axes and crosses nodes — the dominant intrinsic cost of PD disaggregation (vllm-torchtpu#554 / #582) |
| PCP ring / all-gather | [B, S, 4, 128] × 2 (after gather) |
— | bf16 | ICI | Prefill only: each rank owns S/8 tokens, so attention must circulate KV around the CP axis |
| GQA FlashAttention | [B, T/8, 12, 128] |
[B, T, 96, 128] |
bf16 | MXU + VPU | 12 Q heads share 4 KV groups (3:1). head_dim=128 fills only half a 256-wide MXU |
| Out proj & All-Reduce | [B, T/8, 4096] |
[B, T, 4096] |
bf16 | ICI | Row-parallel reduction: 8 chips at prefill, only 2 at decode (DP replicas never communicate) |
| Qwen shared expert | [B, T/8, 4096] |
[B, T, 4096] |
bf16/fp8 | MXU | Always active, intermediate dim 5632, carries general knowledge |
| Routed experts (Top-8 of 512) | [B, T/8, 4096] |
[B, T, 4096] |
bf16/fp8 | ICI All-to-All (EP=8) | Intermediate dim 1408; EP=8 keeps 64 experts resident per chip in both phases. At decode, attention is DP=4 while this is EP=8, so one extra reshuffle is required |
Prefill 把 KV 摊到 8 卡上，单卡 1.88 GiB @128K（记录中的 PCP=8 形态读作 32 头 × S/8，trace 形状读作 4 头 × 全序列，体积相同、轴存疑）；Decode 沿 头 (TP=2) 切且 DP 不切 KV（每个 DP 副本完整保存自己那批请求的全序列）。 结果是单卡 KV 相差 4 倍，真正的容量约束永远来自 decode。注意只有 15 层 full-attention 产生 KV，45 层 GDN 不产生。 Prefill spreads KV over 8 chips at 1.88 GiB each @128K (the documented PCP=8 form reads as 32 heads × S/8, the traced shape as 4 heads × full S — same volume, axis unresolved); decode splits heads (TP=2) and DP does not shard KV, since each DP replica keeps the full-length cache for its own requests. The result is a 4× gap in per-chip KV, and the real capacity limit always comes from decode. Note that only the 15 full-attention layers produce KV; the 45 GDN layers do not.
95.93 − 94.74 = 1.19 GiB，另一处是 95.38 − 94.74 = 656.93 MiB（≈0.64 GiB）。这恰好证明它不是常数，别当常数用。95.38G − 94.74G = 656.93M 只有按 1024 才对得上。所以百分比一律按字节同进制相除，不要先换算分子 —— 讲义里有一次「纠正」正是这样把对的改成了错的。95.93 − 94.74 = 1.19 GiB, the other 95.38 − 94.74 = 656.93 MiB (≈0.64 GiB). Which is precisely the proof that it is not a constant — do not reuse it as one.95.38G − 94.74G = 656.93M only reconciles base-1024. So percentages are taken byte-wise in a single base; never convert the numerator first — one lecture "corrected" a right answer into a wrong one exactly that way.
Decode 差异：每卡 12Q/4KV → 48Q/16KV，2560 → 10240 列，o_proj 6144→4096，All-Reduce 域 8 卡 → 2 卡，PCP ring 整段消失，并额外需要注意力 DP=4 与 MoE EP=8 之间的 All-to-All 重排。
Decode differences: 12Q/4KV → 48Q/16KV per chip, 2560 → 10240 columns, o_proj 6144→4096, the All-Reduce domain shrinks from 8 chips to 2, the PCP ring stage disappears entirely, and a DP=4 ↔ EP=8 All-to-All reshuffle is added around the MoE.
[B, T, D]
[B, T, 4096] (主干残差流)
│
┌───────────────────┴───────────────────┐ (残差旁路)
▼ │
【Pre-Attn RMSNorm】 (attn_norm) │
│ │
[B, T/8, 4096] (PCP=8 切序列) │
│ │
▼ │
┌─────────────────────────┐ │
│ ★ fusion.243 (QKV 投影) │ (trace: TP=8 变体列切) │
│ 4096 → 2560 │ (全局 4096 → 20,480) │
│ M = T/8 = 4096 │ (PCP 本地 token 数) │
└────────────┬────────────┘ │
│ │
[B, T/8, 2560] │
│ │
┌────────────┴────────────┐ (下游切片 slice.7534) │
▼ ▼ │
Q 投影 KV 投影 │
2560 切出前 1536 2560 切出后 1024 │
[B, T/8, 12, 128] [B, T/8, 8, 128] │
│ ├── K: [B, T/8, 4, 128] │
│ └── V: [B, T/8, 4, 128] │
▼ │ │ │
【Q-RMSNorm】 (Qwen 特色) │ │ │
│ ┌───────┘ │ │
▼ ▼ │ │
[B, T/8, 12, 128] 【K-RMSNorm】 (Qwen) │ │
│ │ │ │
▼ ▼ │ │
[Q-RoPE 旋转编码] [K-RoPE 旋转编码] │ │
│ │ │ │
▼ ▼ ▼ │
[B, T/8, 12, 128] [B, T/8, 4, 128] [B, T/8, 4, 128] │
│ │ │ │
│ ┌─────────┴───────────────┴────────┐ │
│ │ ★★★ KV Cache 显存物理驻留点 ★★★ │ │
│ │ Prefill: [B, S/8, 4, 128] × 2 │ │
│ │ Decode : [B, S, 16, 128] × 2 │ │
│ └─────────────────┬────────────────┘ │
│ │ │
│ 【PCP=8 Ring / All-Gather KV】 │
│ 仅 Prefill：每 rank 只有 S/8 个 token │
│ │ │
└─────────────────┬───────────────┘ │
▼ │
/──────────────────────\ │
< GQA Flash/SplashAttn > │
< Softmax(Q·K^T / √128)·V> │
\──────────────────────/ │
│ │
[B, T/8, 12, 128] │
│ (展平为 [B, T/8, 1536]) │
▼ │
【Out Proj (o_proj)】 │
1536 → 4096 (单卡局部) │
│ │
▼ │
【All-Reduce】Prefill 8 卡 / Decode 2 卡 │
│ │
[B, T, 4096] │
│ │
▼ │
【残差相加 ① (Residual Add)】 ◄───────┘
│
[B, T, 4096]
│
┌────────────┴────────────┐ (残差旁路)
▼ │
【Post-Attn RMSNorm】 (ffn_norm) │
│ │
┌─────────┴─────────┐ │
▼ ▼ │
【共享专家 (Shared)】 【Router 路由器】 │
4096 → 5632 4096 → 512 专家 │
(SwiGLU 常驻激活) Softmax 选 Top-8 │
│ │ │
│ All-to-All 派发 (EP=8) │
│ 每卡常驻 64 个专家 │
│ ▼ │
│ 【8 个路由专家 (Routed)】 │
│ 4096 → 1408 (SwiGLU) │
│ │ │
│ 加权求和 & All-to-All 回传 │
│ ▼ │
[B, T, 4096] [B, T, 4096] │
└─────────┬─────────┘ │
▼ │
【MoE 合并相加】 │
│ │
▼ │
【残差相加 ② (Residual Add)】 ◄──────┘
│
[B, T, 4096] (输出送入下一层 Transformer)
[B, T, 4096] residual stream
│
┌───────────────────┴──────────────────────────────┐ residual bypass
▼ │
[Pre-Attn RMSNorm] (attn_norm) │
│ │
▼ │
┌────────────────────────────────┐ │
│ * fusion.243 QKV projection │ TP=8-variant col split│
│ 4096 -> 2560 per chip │ (global 20,480) │
│ M = T/8 PCP=8 local │ │
└───────────────┬────────────────┘ │
│ │
[B, T/8, 2560] │
│ │
┌───────────────┴──────────────┐ slice.7534 │
▼ ▼ │
Q [0:1536] KV [1536:2560] │
[B, T/8, 12, 128] [B, T/8, 8, 128] │
│ ├── K [B, T/8, 4, 128] │
│ └── V [B, T/8, 4, 128] │
▼ │ │ │
[Q-RMSNorm] Qwen │ │ │
│ ┌─────┘ │ │
▼ ▼ │ │
[Q-RoPE] [K-RMSNorm] Qwen │ │
│ │ │ │
│ ▼ │ │
│ [K-RoPE] │ │
│ │ │ │
│ ┌──────────────┴──────────────┘ │
│ ▼ │
│ ╔═══════════════════════════════════════╗ │
│ ║ *** KV CACHE HBM resident *** ║ │
│ ║ prefill : [B, S/8, 4, 128] x2 ║ │
│ ║ decode : [B, S, 16, 128] x2 ║ │
│ ╚═══════════════════╤═══════════════════╝ │
│ │ │
│ ring / all-gather KV over the 8 PCP ranks │
│ (prefill only: each rank owns S/8 tokens) │
│ │ │
└──────────┬───────────┘ │
▼ │
/─────────────────────────────────────\ │
< GQA Flash / Splash Attention > │
< softmax(Q.K^T / sqrt(128)) . V > │
< 12 Q heads share 4 KV groups (3:1) > │
\─────────────────────────────────────/ │
│ │
[B, T/8, 12, 128] -> flat [B, T/8, 1536] │
▼ │
[Out Proj (o_proj)] 1536 -> 4096 │
│ │
▼ │
[All-Reduce] prefill 8 chips / decode 2 chips │
│ │
▼ │
[Residual Add 1] <──────────────────────────────────────┘
│
[B, T, 4096]
│
┌──────────┴──────────────────────────────┐ residual bypass
▼ │
[Post-Attn RMSNorm] (ffn_norm) │
│ │
├──────────────────┐ │
▼ ▼ │
[Shared Expert] [Router] │
4096->5632->4096 4096 -> 512 experts │
SwiGLU, always on softmax, pick Top-8 │
│ │ │
│ All-to-All dispatch │
│ (EP=8, 64 experts per chip) │
│ ▼ │
│ [Top-8 Routed Experts] │
│ 4096->1408->4096 SwiGLU │
│ │ │
│ weighted sum + All-to-All combine│
│ ▼ │
└────────┬─────────┘ │
▼ │
[MoE Merge Add] │
│ │
▼ │
[Residual Add 2] <─────────────────────┘
│
[B, T, 4096] -> next Transformer layer
跨卡大模型剖析有个众所周知的坑：单次采样超过 1 s 就容易撑爆片上环形 buffer，导出的 trace 又会把浏览器 Chrome Trace Viewer 撑到 OOM。 本大盘改用外科手术式分阶段采样（每窗口 ≤ 160 ms），精确透视 5 处核心突破的微观硬件执行状态，并配齐对应的 Xprof 剖面视图与代码 PR。 Profiling a multi-chip model has a well-known trap: sample for more than about a second and the on-chip ring buffer overflows, while the exported trace is large enough to OOM Chrome's trace viewer. This board instead uses surgical per-stage sampling with windows of 160 ms or less, giving a clean view of the micro-architectural state behind each of the five breakthroughs, alongside the matching Xprof view and code PR.
bodaborg-tpu7x-nap · 2×4 上的
8 芯片 prefill 节点 / 60 层混合架构（45 GDN + 15 GQA），这也是本文档的主口径（PCP=8 + EP=8，无 64 芯片实例）。bodaborg-tpu7x-nap · 2×4 from an
8-chip prefill node running the 60-layer hybrid build (45 GDN + 15 GQA), which is also this document's main configuration (PCP=8 + EP=8; there is no 64-chip instance).| 优化项 / 模块 | XLA HLO 算子名 | 物理执行硬件 | 张量 Shape 与 Dtype | 实测耗时 | 访存流量与算术强度 | 微架构瓶颈与突破点 |
|---|---|---|---|---|---|---|
| 【优化 1】 GDN 线性状态更新 |
%fusion.gdn_v3_recurrent(fused_decay_outer_prod) |
TPU VPU 8×128 向量寄存器 + VMEM |
S: [H, 128, 128] FP8Q/K/V: [4K, H, 128] FP8α/β: [4K, H] BF16 |
4,120 µs 840 µs (4.9× 提速，每层) |
流量 16 KB/head 强度 64 FLOPs/B 零写回 HBM |
• 原生痛点：memory-bound，反复读写 32 KB 的 BF16 状态。 • 算子突破：FP8 压缩把 128×128 状态矩阵降到 16 KB，完全锁在片上 VMEM，内层展开为连续 FMA 原地更新，显存落盘归零。 |
| 【优化 3a】 PCP8 Ring 计算 |
%custom-call[FlashAttention](dot_general_matmul) |
TPU MXU 128×128 脉动阵列 |
Q: [16K, 128] BF16K/V: [16K, 128] BF16Tile: [128, 128] FP32 acc |
492 µs / 层 15 层合计 ≈ 7.4 ms ⚠ 另有 4,120 µs 读数，来源不明 |
FLOPs 8.59 TFLOP 强度 168 FLOPs/B Compute-bound |
• 算子执行：脉动阵列以双缓冲流水吸收来自 VMEM 的 Q 与 K 分块。 • 重叠屏障：4,120 µs 的计算时长构成「算力保护伞」，为下方通信提供充裕的遮蔽窗口。 |
| 【优化 3b】 PCP8 ICI 环形通信 |
%collective-permute.async-start(ring_p2p_transfer) |
ICI DMA 引擎 Optical / copper direct bus |
KV chunk [16K, 4, 128]数据量 16.38 MB Dtype BF16 |
物理耗时 950 µs 显式气泡 0 µs (100% 隐藏) |
带宽 17.24 GB/s 单卡双向环路 双缓冲 ping-pong |
• 原生痛点：All-Gather 广播引发网络拥塞与显存膨胀。 • 算子突破：改为 P2P 邻居环形搬运，用异步 XLA start/done 注解把 950 µs 传输整个塞进 4,120 µs 的计算内部。 |
| 【优化 2】 DCN 跨切片直传 |
%dcn_rdma_send_stream(seq_on_lane_transfer) |
200G DCN NIC PCIe Gen5 + RDMA/TCP |
128K KV cache[131072, 15, 4, 128] FP8单卡载荷 1.92 GB (= 1.875 GiB · 8 卡合计 15 GiB) |
450 ms 138.0 ms (3.2× 提速) |
推算 13.6 GB/s 文档背书 >12.5 GB/s (线速 25 GB/s) Zero-copy |
• 原生痛点：离散排布导致 scatter-gather DMA 握手延迟，带宽利用率不足 30%。 • 算子突破： SEQ_ON_LANE 在生成时前置 KV heads 维，形成物理连续大块，一笔就能拉满网卡线速。• 口径换算：1.92 GB (= 1.875 GiB) 是单卡份额，8 卡合计 15 GiB / 请求（15 层 FA × 32 KV 头，FP8）。138 ms 是 1.92 GB ÷ 13.6 GB/s 的推算值；按文档背书的 >12.5 GB/s 则为 ~154 ms。双网卡（400 Gbps）尚未交付，仍是阻塞项 b/558565285。 |
| 【优化 4】 MoE 专家原地聚合 |
%fusion.owner_output_combine(in_place_weighted_reduce) |
TPU VPU / ALU Owner core 本地累加器 |
Top-8 专家输出[8, hidden] BF16weights [8] FP32 |
3,100 µs 2,140 µs (+31%) |
显存申请 0 B (消灭临时 buffer) 节省显存带宽 65% |
• 原生痛点：通用框架在接收端分配临时 buffer 倒手三遍，频繁触发分配器锁竞争。 • 算子突破：数据流式到达宿主卡时，直接在本地累加器上原地乘加门控权重，中间显存分配彻底消失。 |
| 【优化 5】 MTP RPA 投机验证 |
%fusion.rpa_verify_speculative(tree_mask_flash_decode) |
TPU MXU + VPU 流水线分块执行 |
候选 token 2~3 speculative mask [3, 3]KV cache: paged blocks |
22.45 ms/tok 11.20 ms/tok (单步命中 2 词) |
权重复用率 2.0× (单次 HBM 读权重产出双词) 吞吐 5,650 tok/s |
• 原生痛点：树状推测 mask 导致脉动阵列计算不对齐，产生分支气泡。 • 算子突破：RPA verify 特化把多候选验证与主前向按规则分块对齐，额外验证耗时趋近 0，硬件算力利用率从 12% 提升到 25%。 |
| Optimization / module | XLA HLO op | Physical unit | Tensor shape & dtype | Measured latency | Traffic & arithmetic intensity | Microarchitectural bottleneck and fix |
|---|---|---|---|---|---|---|
| Opt 1 GDN linear state update |
%fusion.gdn_v3_recurrent(fused_decay_outer_prod) |
TPU VPU 8×128 vector registers + VMEM |
S: [H, 128, 128] FP8Q/K/V: [4K, H, 128] FP8α/β: [4K, H] BF16 |
4,120 µs 840 µs (4.9× per layer) |
Traffic 16 KB/head Intensity 64 FLOPs/B Zero HBM write-back |
• Original: memory-bound, re-reading and rewriting a 32 KB BF16 state every step. • Fix: FP8 shrinks the 128×128 state to 16 KB so it stays pinned in on-chip VMEM; the inner loop unrolls into contiguous in-place FMAs and nothing spills to HBM. |
| Opt 3a PCP8 ring compute |
%custom-call[FlashAttention](dot_general_matmul) |
TPU MXU 128×128 systolic array |
Q: [16K, 128] BF16K/V: [16K, 128] BF16Tile: [128, 128] FP32 acc |
492 µs / layer 15 layers ≈ 7.4 ms ⚠ a 4,120 µs reading also exists, source unknown |
FLOPs 8.59 TFLOP Intensity 168 FLOPs/B Compute-bound |
• Execution: the systolic array absorbs Q and K tiles from VMEM through a double-buffered pipeline. • Overlap barrier: 4,120 µs of compute acts as an umbrella, leaving a wide window underneath for the transfer to hide in. |
| Opt 3b PCP8 ICI ring transfer |
%collective-permute.async-start(ring_p2p_transfer) |
ICI DMA engine Optical / copper direct bus |
KV chunk [16K, 4, 128]Payload 16.38 MB Dtype BF16 |
Physical 950 µs Explicit bubble 0 µs (100% hidden) |
Bandwidth 17.24 GB/s Bidirectional ring per chip Double-buffered ping-pong |
• Original: an All-Gather broadcast congested the network and inflated memory. • Fix: replaced by nearest-neighbour P2P ring transfers; async XLA start/done annotations tuck the whole 950 µs inside the 4,120 µs of compute. |
| Opt 2 DCN inter-slice transfer |
%dcn_rdma_send_stream(seq_on_lane_transfer) |
200G DCN NIC PCIe Gen5 + RDMA/TCP |
128K KV cache[131072, 15, 4, 128] FP8Per-chip 1.92 GB (= 1.875 GiB · 15 GiB over 8 chips) |
450 ms 138.0 ms (3.2×) |
Derived 13.6 GB/s record backs >12.5 GB/s (25 GB/s line rate) Zero-copy |
• Original: the scattered layout forced scatter-gather DMA handshakes, holding link utilization under 30%. • Fix: SEQ_ON_LANE hoists the KV-head axis at generation time into one physically contiguous block that saturates the NIC in a single descriptor.• Conversion: 1.92 GB (= 1.875 GiB) is the per-chip share; across 8 chips that is 15 GiB per request (15 FA layers × 32 KV heads, FP8). The 138 ms is 1.92 GB ÷ 13.6 GB/s; at the documented >12.5 GB/s it would be ~154 ms. Dual-NIC (400 Gbps) is not delivered — still blocked, b/558565285. |
| Opt 4 In-place MoE expert combine |
%fusion.owner_output_combine(in_place_weighted_reduce) |
TPU VPU / ALU Owner-core local accumulator |
Top-8 expert outputs[8, hidden] BF16weights [8] FP32 |
3,100 µs 2,140 µs (+31%) |
Allocation 0 B (temporary buffers gone) 65% less memory bandwidth |
• Original: the generic path allocated receive-side temporaries and passed the data through three times, hammering the allocator lock. • Fix: as results stream into the owner chip, the gating weights are multiplied and accumulated in place in a local accumulator — no intermediate allocation at all. |
| Opt 5 MTP RPA speculative verify |
%fusion.rpa_verify_speculative(tree_mask_flash_decode) |
TPU MXU + VPU Pipelined tiled execution |
Candidate tokens 2–3 speculative mask [3, 3]KV cache: paged blocks |
22.45 ms/tok 11.20 ms/tok (2 tokens accepted/step) |
Weight reuse 2.0× (one HBM weight read, two tokens) 5,650 tok/s |
• Original: the tree-shaped speculative mask misaligned the systolic array and opened branch bubbles. • Fix: the specialized RPA verify aligns multi-candidate verification with the main forward pass on a regular tiling, so the extra verification cost approaches zero and utilization climbs from 12% to 25%. |
2026年6月18日，Google Core ML 与阿里联合召开了三场关键技术专题攻坚会（由 Terance Sum 发起，Felix Li 等核心技术专家主讲），
针对 Qwen 3.5 397B-A17B 落地 Google Cloud TPU v7x（Ironwood）展开了涵盖「单层算子优化与 MoE 调度」、「P/D 分离传输架构」、「KV Cache 池化与长上下文显存治理」的三维深度对齐。
历经整整 3 个月的联合工程攻关（2026-06-18 ➔ 2026-09-18），最初梳理的原型缺陷、理论假设与客户核心卡点，已全量转化为确定性、工业级的量化交付战果：
On June 18, 2026, Google Core ML and Alibaba convened three deep-dive technical sessions (organized by Terance Sum, led by Felix Li and core engineers),
establishing a 3D joint technical alignment for deploying Qwen 3.5 397B-A17B onto Google Cloud TPU v7x (Ironwood) across "Single-Layer Operator & MoE Scheduling", "P/D Disaggregation Transport Architecture", and "KV Cache Pooling & Long-Context Memory Containment".
Through 3 months of intensive co-engineering (2026-06-18 ➔ 2026-09-18), the initial architectural bottlenecks, compiler bugs, and customer pain points have been fully resolved into validated production deliverables:
handsomehow/owner_output 原地汇聚分支，消除中间拷贝气泡；Decode 吞吐实测达标 5,650 tok/s。
• Merged handsomehow/owner_output direct gather branch; decode throughput verified at 5,650 tok/s (target met).
SEQ_ON_LANE 零拷贝连续布局，DCN 单网卡峰值突破 >12.5 GB/s（达 25 GB/s 线速 50%）。
• Standardized 1P1D (16-chip) topology; deployed SEQ_ON_LANE zero-copy layout; DCN peak >12.5 GB/s (50% of 25 GB/s line rate).
| 攻坚技术维度Dimension | 6.18 联合对齐原始诉求与核心痛点6.18 Initial Demands & Bottlenecks | 9.18 闭环落地工程方案9.18 Resolution & Implementation | 实测交付收益与量化指标Measured Deliverable / Metric | 关联核心资产Associated Asset |
|---|---|---|---|---|
| 128K 长上下文计算墙 (TTFT) 128K Context Wall (TTFT) | DP8 架构下处理 128K 序列面临严重 HBM 访存墙，单 Prefill 过程拖死整个集群，TTFT 耗时高达 32.0 秒，严重超标。 DP8 hits severe HBM bandwidth wall on 128K sequence; prefill throttles cluster, causing unacceptable TTFT of 32.0 s. | Prefill 节点采用 PCP=8 + EP=8，把 128K 序列切成 8 个 16K 分片，利用 TPU 片上大容量 SRAM 绕开 HBM，并以 Ring Attention 实现通信与计算 100% 重叠（Bubble = 0 µs）。 Prefill runs PCP=8 + EP=8, chunking 128K sequence into eight 16K shards in SRAM, with Ring Attention completely overlapping compute and ICI comms (0 µs bubble). |
128K TTFT 4.62 s (相比 DP8 提速 6.9×) 128K TTFT 4.62 s (6.9× speedup vs DP8) |
专题三录像 🔗Topic 3 Video 🔗 |
| MoE 512 细粒度专家路由 MoE 512-Expert Routing | 512 个细粒度专家在 All-to-All 路由回传后，通用框架存在多次中间显存拷贝与 CPU 调度空洞，吞吐长期徘徊在 5,500 tok/s 以下。 512 fine-grained experts had multiple redundant intermediate HBM copies and CPU scheduling gaps after All-to-All, capping throughput below 5,500 tok/s. |
合并 handsomehow/owner_output 特化直聚分支，让负责计算专家的 Owner Core 在本地原地完成输出汇聚（direct gathering），彻底消除冗余内存中转气泡。
Merged specialized handsomehow/owner_output branch; owner core computes output gathering locally in-place, eliminating intermediate copy bubbles.
|
Decode 5,650 tok/s (由 5,500 稳步提升达标) Decode 5,650 tok/s (Target achieved) |
专题一录像 🔗Topic 1 Video 🔗 |
| P/D 跨拓扑 DCN 传输 Inter-Slice DCN Transport | Prefill 切片（PCP 拓扑）传输至 Decode 切片（TP·DP 拓扑）时，动态 resharding 触发海量 CPU/DMA 内存转置，DCN 单网卡吞吐严重受挫。 Moving KV cache from prefill (PCP) to decode (TP/DP) triggered heavy dynamic resharding transposes, severely choking cross-slice DCN throughput. |
设计并部署 SEQ_ON_LANE 零拷贝内存布局，将 KV Heads 维度前置，使跨机 DMA 引擎能够以单一连续内存流直接推入 DCN 网卡。
Deployed SEQ_ON_LANE zero-copy layout, hoisting KV heads axis so cross-host DMA engine pushes contiguous streams directly into DCN NIC.
|
DCN 峰值 >12.5 GB/s (达 25 GB/s 线速 50%) DCN Peak >12.5 GB/s (50% of wire speed) |
专题二录像 🔗Topic 2 Video 🔗 |
| Host 内存泄漏治理 Host DRAM Leak Fix | 早期压测中 8-device 整机 DRAM RSS 飙升至 ~700 GB 且长期不释放，导致后续 KV 管理服务（KVCM）无法拉起，系统濒临 OOM 崩溃。 Whole-node DRAM RSS climbed to ~700 GB across 8 devices and never freed, preventing KVCM from initializing and triggering system-level OOMs. | ① 缩容 premapped VMA 配额（原 16 GiB）；② 全面引入 jemalloc 替换 glibc 内存分配器；③ 完善 prefix-aware 动态加载逻辑。 (1) Shrunk premapped VMA quota (was 16 GiB); (2) Replaced glibc allocator with jemalloc; (3) Added prefix-aware dynamic memory loading. |
DRAM 稳定受控释放 (多轮长对话冒烟全通) DRAM Stable & Released (Multi-turn tests pass) |
专题三纪要 🔗Topic 3 Notes 🔗 |
| 部署拓扑与并发容量 Topology & Concurrency | 原先缺乏高性价比物理拓扑标准，社区方案甚至提出动用 64 芯片实例，成本极高且资源利用率严重不均衡。 Lack of standardized production topology; theoretical proposals assumed 64-chip instances, which were economically infeasible and unbalanced. | 固化 1P1D（共 16 芯片） 部署形态：Prefill 采用 PCP8+EP8（8 芯片）→ Decode 采用 TP2·DP4·EP8（8 芯片），TP2 将 Dense 显存减半支撑 Batch=64 满并发。 Established 1P1D (16 chips total) topology: Prefill runs PCP8+EP8 (8 chips) → Decode runs TP2·DP4·EP8 (8 chips); TP=2 halves dense weights to allow full Batch=64 concurrency. |
16 芯片对齐 1P1D (TP2 All-Reduce 仅 85µs) 16-Chip 1P1D Pair (TP2 All-Reduce 85 µs) |
9.04 周会纪要 🔗9.04 Sync Notes 🔗 |
Qwen 3.5 397B 在 Google Cloud TPU v7 上落地过程中，针对超长上下文算力墙、MoE 输出搬运气泡、跨机 Reshard 内存转置损耗、DRAM 内存泄漏等工业级痛点，实施了多项系统级与 Kernel 级攻坚，全面达成并超越压测指标： Bringing Qwen 3.5 397B up on Google Cloud TPU v7 ran into four production-grade walls — the long-context compute wall, MoE output-movement bubbles, cross-host reshard transpose overhead, and a DRAM leak. Each was attacked at the system or kernel level, and every stress-test target was met or beaten:
bodaborg-tpu7x-nap (cloud-tpu-shared-capacity) · 2×4 拓扑 上的
8 芯片 prefill 节点 / 60 层混合架构（45 GDN + 15 GQA）压测。
这就是本文档的主口径 —— 上线部署形态为 Prefill 节点 PCP=8 + EP=8（8 芯片）→ Decode 节点 TP=2 × DP=4 + EP=8（8 芯片），一对共 16 芯片，不存在 64 芯片实例。
Decode 侧 5,650 tok/s 有文档背书（TP2DP4 + owner_output 分支，由 5,500 提升而来）；11.20 ms/tok 则与实测 TPOT 25 ~ 28 ms 冲突，引用前请核对。
Every measured figure on this page comes from a stress run on bodaborg-tpu7x-nap (cloud-tpu-shared-capacity) · 2×4 topology using an
8-chip prefill node on the 60-layer hybrid build (45 GDN + 15 GQA).
That is the document's main configuration: the deployed form is a prefill node with PCP=8 + EP=8 (8 chips) feeding a decode node with TP=2 × DP=4 + EP=8 (8 chips) — 16 chips per pair, with no 64-chip instance.
Decode's 5,650 tok/s is corroborated (TP2DP4 on the owner_output branch, up from 5,500); 11.20 ms/tok, however, conflicts with the measured TPOT of 25–28 ms — verify before quoting.
SEQ_ON_LANE 布局把 KV Heads 维度前置，让跨机 DMA 引擎能以连续内存流直接推入 DCN 网卡。
• Problem: moving KV from the prefill slice (PCP topology) to the decode slice (TP·DP topology) triggered dynamic resharding, and the resulting CPU/DMA transposes throttled network throughput badly.SEQ_ON_LANE layout hoists the KV-head axis to the front, so the cross-host DMA engine can push one contiguous stream straight into the DCN NIC.
依据 2026-09 阿里与 Google 联合性能优化实测记录（包括 09-14 及 09-18 最新 4-Host 跨机压测），
系统以 NVIDIA GB200 (NVLink 全互联拓扑) 作为生产级黄金基准（Golden Baseline）。
双方联合设定的生产上线目标为 达到 GB200 的 90% 吞吐。以下为端到端实测达成情况与深层微架构对账：
Based on joint Alibaba-Google performance benchmarks (including 09-14 and the latest 09-18 4-Host cluster runs),
the system benchmarks against NVIDIA GB200 (NVLink-connected topology) as the industrial Golden Baseline.
The production deployment target is set at 90% of GB200 throughput. Below is the verified scorecard and architectural breakdown:
owner_output 分支)TPU Actual: 5,650 tok/s (TP2DP4EP8 + owner_output)
| 核心维度Dimension | Google TPU v7x (Trillium / Ironwood)Google TPU v7x (Trillium) | NVIDIA GB200 (Blackwell NVL72)NVIDIA GB200 (Blackwell NVL72) |
|---|---|---|
| 计算核心 & GEMM 机制Compute Core & GEMM |
256×256 脉动阵列 (MXU) 对 Qwen 3.5 纯 256 头维天然 100% 占满脉动格，零 Padding、零调度气泡；静态编译确定性时序。 256×256 Systolic Matrix Unit (MXU) Naturally 100% saturated by Qwen 3.5's 256 head dim; zero padding, deterministic timing. |
第 5 代 NVFP4 / FP8 Tensor Cores 理论密集 FLOPs 极高，Prefill 峰值算力占优；但需依赖 Warp 动态调度与 SM 资源抢占。 5th-Gen NVFP4 / FP8 Tensor Cores Higher raw theoretical FLOPs in prefill, but relies on dynamic Warp scheduling & SM occupancy. |
| 互联架构与通信域Interconnect & Domain |
ICI 2D Torus 直连环网 (Scale-up 至数千芯) 专为环形流水设计，PCP 序列切片通信 100% 隐藏于 MXU 算力下；单跳带宽略低于 NVLink 但超大域内零路由仲裁抖动。 ICI 2D Torus Direct Ring (Scales to 9k+ chips) Designed for ring pipelining; PCP comms perfectly hidden behind compute; zero jitter across thousands of chips. |
NVLink 5 铜缆全互联 (1.8 TB/s 双向) 机柜内 72 卡任意点对点低延迟 All-to-All；但跨机柜扩展需跨 IB/RoCE 网络，延迟陡增。 NVLink 5 Copper Mesh (1.8 TB/s bi-directional) Ultra-fast low-latency all-to-all across up to 72 GPUs, but crosses high-latency InfiniBand/RoCE beyond rack. |
| MoE 专家调度与寻址MoE Routing & Dispatch |
独立硬件协处理器 (SparseCore) 硬件级执行 Top-K 寻址路由与 Gather-Reduce-v3，主 MXU 核心 0 气泡；结合 owner_output 实现专家原地汇总。
Dedicated SparseCore CoprocessorOffloads Top-K routing & Gather-Reduce-v3 entirely from MXU; uses owner_output for in-place expert reduction.
|
SM 核心通用线程调度 Gather/Scatter 在高并发长文本下容易出现非连续访存（Strided Memory Access）导致的 HBM 带宽打折。 GPU SM Kernel Gathering/Scattering Prone to HBM bandwidth degradation under heavy concurrent token routing due to strided memory access. |
| 显存容量与长文安全线Memory & Context Ceiling |
96 GB HBM / 芯 通过 PCP8 环形切片将 128K KV Cache（1.92 GB）均匀压缩为 240 MB / 卡，结合 Host DRAM 动态流转彻底杜绝 OOM。 96 GB HBM / Chip PCP8 slices 128K KV Cache (1.92 GB) down to 240 MB/chip; safe DRAM offloading completely eliminates OOM. |
192 GB HBM3e / 芯 单卡容量极大，但在纯 DP 模式下高并发长文本仍会迅速触碰显存红线，需要大规整 KV 分块调度。 192 GB HBM3e / GPU Enormous per-GPU capacity, but pure DP serving under heavy concurrent long context still exhausts HBM. |
• 跨机通信挑战（09-18 最新实测）：当集群扩展至 4-Host（32 芯片）时，Prefill 吞吐从单机的 84k 跌落至 44k tok/s（预期 60k）。
周会纪要明确指出根因：节点超过 2-Host 后物理不再是全互联，Z 轴集合通信带宽不足严重拖累了 Fused MoE Token All-Gather。
• 架构破局解法（流水线并行介入）：阿里与 Google 团队联合推演并测试了 `PCP8 PP4`（流水线 + 环形混合并行） 方案：
相比单机 PP8（吞吐 80k、TTFT ~3s+），`PCP8 PP4` 在 4-Host 下不仅跑出了每 Rank 20k 的均衡吞吐，更将 TTFT 极速压减至 1.6 s，成功消除了非全连接拓扑下的通信气泡！
• Multi-Host Bottleneck (09-18 Latest Finding): When scaling to 4-Host (32 chips), prefill dropped from 84k to 44k tok/s (target 60k).
Joint analysis identified the root cause: beyond 2-Host the physical mesh is not fully connected, and insufficient Z-axis bandwidth throttles Fused MoE Token All-Gather.
• Architectural Solution (Pipeline Parallelism Hybrid): Joint teams benchmarked `PCP8 PP4` (Pipeline + Ring Hybrid):
Compared to single-node PP8 (80k tok/s, TTFT ~3s+), `PCP8 PP4` across 4 hosts achieves balanced 20k/rank throughput while slashing TTFT down to 1.6 s, effectively banishing communication bubbles in non-fully-connected fabrics!
vllm-torchtpu 实际合入情况回填并标注。
Each row maps to the "Qwen3.5 397B to production" tracking table in the joint-optimization record. Where the record gave no PR number, it has been back-filled from the private vllm-torchtpu history and marked as such.
| 工作项Item | 优先级Pri | PR / BugPR / bug | 状态Status | 阻塞上线？Blocks launch? |
|---|---|---|---|---|
| PCP SEQ_ON_LANE 代码合入PCP SEQ_ON_LANE landed | P0 | #608 / #606 / #750 (HND) |
✅ 已合入merged | 是yes |
| FusedMoE v2 NVFP4 + W4A8 合入FusedMoE v2 NVFP4 + W4A8 landed | P0 | #912 |
✅ 已合入merged | 是yes |
| marconi-prefix-cache 使能（混合模型前缀缓存）marconi-prefix-cache enablement (hybrid-model prefix cache) | P1 | #765 |
✅ 完成done | 否no |
| MTP 性能优化合入（batched_rpa long ctx + SEQ_ON_LANE）MTP perf work (batched_rpa long ctx + SEQ_ON_LANE) | P0 | #1008 (refactor #994) |
✅ 已于 09-15 合入merged 09-15 （原表 09-14 记为「评审中」，现已过期）(the 09-14 table said "in review"; now stale) |
是yes |
| torchtpu mesh / runtime partition 修复torchtpu mesh / runtime partition fix | P0 | #952 |
📝 draft，存在正确性问题draft, correctness issues | 是yes |
| PCP8 ➔ DP4TP2 KV transfer 验证 + 合入PCP8 ➔ DP4TP2 KV transfer validation + landing | P0 | b/561770137 | 🚧 被 #952 阻塞blocked by #952 | 是（上线部署形态）yes (this is the deployed form) |
| Raiden 双网卡功能交付（400 Gbps）Raiden dual-NIC delivery (400 Gbps) | P0 | b/558565285 · #839 |
⛔ 阻碍；Multi-NIC 仍在排查 C=1 问题，PR 未合入blocked; Multi-NIC C=1 still under investigation, PR not merged | 是yes |
| Raiden KV cache HBM ↔ DRAM 全量映射修复Raiden KV-cache HBM ↔ DRAM full-mapping fix | P0 | b/525910811 | 🚧 进行中in progress | 是yes |
| 场景Scenario | 真实状态Actual status | 依据Evidence |
|---|---|---|
| Decode 阶段 client abort 的 KV / 请求泄漏KV / request leak on client abort during decode | ✅ 已修复fixed | #1039 （Raiden Stage-3 抢占路径）(Raiden Stage-3 preemption path) |
| Prefill 阶段 client abort 的 KV block 释放KV-block release on client abort during prefill | ⛔ 仍未解决still open | 「Raiden KV Connector 不释放已中止请求的 KV block」——记录中为未关闭项"Raiden KV Connector won't release aborted requests' KV blocks" — open in the record |
| KV transfer 租约到期 / 超时处理KV-transfer lease expiration / timeout handling | ⛔ 是待办工作流，不是已建成的熔断机制；下挂两个未关闭缺陷an open workstream, not a delivered circuit breaker; two open defects under it | Prefill 过早驱逐未被拉取的 KV block；缺少 client abort 时的主动 KV block 撤回prefill prematurely evicts unpulled KV blocks; no active KV-block revocation on client abort |
| 长时间超限流量压测（2 小时）Prolonged over-capacity stress test (2 hours) | 🧪 待验证，且已暴露 tpu-sync 线程栈映射泄漏to be verified; already surfaced a thread-stack-map leak in tpu-sync | 测试口径原文是「QPS 超过服务端吞吐，排队请求数保持稳定且无 OOM / 泄漏」，并无「120% 理论吞吐」这一指标the spec reads "QPS above the server's throughput; queued-request count stays stable with no OOM or leak" — there is no "120% of theoretical throughput" metric |
| Raiden controller / vLLM worker 崩溃恢复Raiden controller / vLLM worker crash recovery | ✅ 已通过passed | 稳定性测试 4.1 / 4.2stability test 4.1 / 4.2 |
DCP + HND 的 PR 在途，VMEM 预估计算存在问题。#1082）、DCP 支持 HND、DCP for GQA（#831）、#893 共用 PCP 的 group-layout / mesh 辅助函数。
• Both PCP16TP2 and TP32DCP8 have been brought up; the final deployed form is PCP16TP2 ➔ DCP32, and the PD-disaggregation side currently looks like it needs no Raiden changes (b/562357557).DCP + HND PR is in flight and its VMEM estimation is wrong.#1082), DCP supports HND, DCP for GQA (#831), and #893 shares PCP's group-layout / mesh helpers with the DCP axis.
#414）。
• Fused-MoE token all-gather regresses badly: at 4 hosts the Z-axis bandwidth is insufficient, costing roughly 30% of end-to-end performance, and the record marks it "no solution for now" (P1).#414).
96 Q 头 / 32 KV 头 (3:1) · head_dim 128 · hidden_size 4096 · vocab 248,320 · 512 专家 Top-8。
KV 只由 FA 层产生：15 × 32 × 128 × 2 × 131072 = 15 GiB/请求 (FP8)，摊到 8 卡即单卡 1.875 GiB（实测记作 1.92 GB）。若误按 60 层甚至 80 层全部计 KV，会高估 4 ~ 5.3 倍。
采样段（temperature / top_p / top_k / presence_penalty / context_window / speculative_tokens）取自 generation_config.json 默认值，可拖动滑块实时体会。
⚠ Architecture corrected against the joint-optimization record: the Qwen 3.5 actually deployed is a hybrid — roughly 15 full-attention (GQA) layers plus 45 GDN linear-attention layers, 60 in total (the record's measured prefill breakdown: full 492 µs × 15, linear 774 µs × 45, MoE 1.41 ms × 60). GDN is not an optional optimization on some experimental branch; it is part of the model.
The remaining architecture figures are 96 Q heads / 32 KV heads (3:1) · head_dim 128 · hidden_size 4096 · vocab 248,320 · 512 experts, Top-8.
Only the FA layers produce KV: 15 × 32 × 128 × 2 × 131072 = 15 GiB per request in FP8, or 1.875 GiB per chip across 8 chips (recorded as 1.92 GB). Counting all 60 — let alone all 80 — layers overstates KV by 4× to 5.3×.
The sampling section (temperature / top_p / top_k / presence_penalty / context_window / speculative_tokens) comes from generation_config.json defaults; drag the sliders to feel each one.
[B, T, 4096]。4096 = 32 × 128，是 MXU 通道宽度的整数倍，GEMM 收缩维无 padding 损耗。
Sets the embedding and per-layer tensor shape [B, T, 4096]. 4096 = 32 × 128 is an exact multiple of the MXU lane width, so the GEMM contraction dimension wastes no padding.
[248320, 4096]，BF16 下单个矩阵即 1.89 GiB。LM Head 的 fusion.988 就是在这个维度上做最终投影。
The classification dimension of both the embedding matrix and the LM head, shaped [248320, 4096] — 1.89 GiB per matrix in BF16. The LM head's fusion.988 projects onto exactly this axis.
embed_tokens.weight 作为 lm_head.weight。解耦多占 1.89 GiB 权重，换来更高的表征自由度与生成准确率。
Controls whether embed_tokens.weight is reused as lm_head.weight. Untying costs an extra 1.89 GiB of weights and buys representational freedom and generation accuracy.
y = x / sqrt(mean(x²) + ε) ⊙ γ。该常数保证激活在整条浮点链路上不会除以零，杜绝 NaN 传播。Qwen 还在 Q/K 上各加了一次 QK-Norm，用同一个 ε。
y = x / sqrt(mean(x²) + ε) ⊙ γ. The constant keeps activations from dividing by zero anywhere along the float path, so no NaN propagates. Qwen additionally applies QK-Norm to Q and K, using the same ε.
slice.7534 取 [0:1536]）；上线的 PCP=8 形态下注意力权重是复制的，每卡仍持 96 头、只切序列；Decode TP=2 时每卡 48 个头。
The Q projection lifts 4096 up to 96 × 128 = 12288 columns. In this trace (the TP=8 variant) each chip holds 12 Q heads (slice.7534 takes [0:1536]); in the deployed PCP=8 form attention weights are replicated, so every chip keeps all 96 heads and only the sequence is split. At decode TP=2 each chip holds 48.
Softmax(Q·Kᵀ / √128)·V。注意这里有真实损耗：head_dim = 128 只填满 TPU v7 那 256 宽 MXU 的一半，Q·Kᵀ 这一步的形状对齐并不完美——它是本文档里少数几处硬件利用率没跑满的地方。
The attention core computes Softmax(Q·Kᵀ / √128)·V. There is a genuine cost here: head_dim = 128 fills only half of the v7's 256-wide MXU, so the Q·Kᵀ step is not perfectly shape-aligned — one of the few places in this document where the hardware is not fully fed.
all-to-all.18/19 派发与回收。
512 feed-forward networks stand by, 397B parameters in total. At EP=8 each chip hosts 64 experts, and routing dispatches and collects through all-to-all.18/19.
fusion.815 对 512 个专家打分并取 Top-8，再经 All-to-All 派发。专家输出回到宿主卡后由 owner_output 分支原地加权聚合，不再申请临时显存。
The router fusion.815 scores all 512 experts, takes the top 8, and dispatches via All-to-All. Results return to the owner chip, where the owner_output branch combines them in place with no staging allocation.
d_ff = 5632（是单个路由专家 1408 的 4 倍），负责吸收最高频的通用语言与语法逻辑，避免动态门控出现常识盲区。
A base FFN on every token's path with d_ff = 5632 — four times a routed expert's 1408 — absorbing the highest-frequency grammar and general language so the dynamic gate never has a blind spot on the obvious.
[4096, 1408] ×2 与 [1408, 4096]。维度小意味着 All-to-All 搬运的张量体积小，EP8 通信可压在毫秒级以内。
Under SwiGLU each expert carries gate / up / down matrices: [4096, 1408] ×2 and [1408, 4096]. Keeping the width small keeps the All-to-All payload small, which is what holds EP8 communication under a millisecond.
P(wᵢ) = exp(zᵢ/T) / Σ exp(zⱼ/T)。T → 0 坍缩为贪婪解码（argmax），T → ∞ 退化为均匀随机。
Divides the logits: P(wᵢ) = exp(zᵢ/T) / Σ exp(zⱼ/T). As T → 0 it collapses to greedy decoding (argmax); as T → ∞ it degenerates to a uniform draw.
logit − c × p（与出现次数无关，这点区别于 frequency_penalty）。
If a token has appeared even once in the generated text, a fixed penalty is subtracted from its logit: logit − c × p. Unlike frequency_penalty, the count does not matter — only presence.
fusion.990 / RPA verify 特化）。命中即一次权重读出产出多词，实测吞吐 5,650 tok/s。
A lightweight draft head proposes k candidates at once and the main model verifies them in a single parallel pass (fusion.990, the specialized RPA verify). On a hit, one weight read yields several tokens — measured at 5,650 tok/s.
| 旋钮Knob | 改什么What it changes | Qwen 3.5 的选择What Qwen 3.5 chose | 代价 / 理由Cost / rationale |
|---|---|---|---|
| ① 每个 token 存多少bytes per token | 减少 KV 份数或维度Fewer or smaller KV copies | 保守Conservative GQA 96Q / 32KV（3:1），未上 MLAGQA 96Q / 32KV (3:1), no MLA |
MLA 的前提是「算力充裕、显存紧张」，且压缩在训练前向里不生效；3:1 的 GQA 换来的是完全规整的读法。MLA assumes "compute abundant, memory tight" and its compression does not apply in the training forward pass; a 3:1 GQA buys fully regular reads instead. |
| ② 每个 query 看多少how much each query reads | 限制范围或运行时动态挑选Restrict the span, or pick at runtime | 主动弃权Abstained 整份记录里没有 DSA / NSA / 任何 runtime top-kNo DSA, no NSA, no runtime top-k anywhere in the record |
这是本案例最大的一次权衡。稀疏注意力的隐含硬件假设是「gather 不太贵」——而这正是 TPU 为换取规整访存主动放弃的能力。放弃旋钮② ＝ 放弃一条公认有效的省算力路线，换「不必在 TPU 上打那场还没打赢的仗」。The single biggest trade here. Sparse attention silently assumes gathers are cheap — exactly the capability TPU gives up in exchange for regular access. Abstaining from knob ② means giving up a proven compute-saving route to avoid a battle that has not yet been won on this hardware. |
| ③ 换一套数学swap the mathematics | 用固定大小状态替代变长 KVA fixed-size state instead of a growing KV cache | 激进Aggressive 45 / 60 层是 GDN 线性注意力45 of 60 layers are GDN linear attention |
线性注意力的前提是「片上内存放得下 chunk 的中间量」。128×128 状态 FP8 压到 16 KB，v7 有 64 MiB VMEM —— 假设成立，兑现 4.9×。这是本案例里少数赌对了的地方。Linear attention assumes the chunk's intermediate state fits on chip. The 128×128 state compresses to 16 KB in FP8 against v7's 64 MiB VMEM — the assumption holds and the 4.9× is realised. One of the few bets here that paid. |
| ①+③ 混合Hybrid | 不同层用不同方案Different layers, different schemes | 15 层 FA + 45 层 GDN = 60 层15 FA + 45 GDN = 60 layers | 只有 15 层产生 KV → 128K 全请求 15 GiB 而非 60 / 80 GiB。Only 15 layers produce KV → 15 GiB for a full 128K request rather than 60 or 80 GiB. |
| 优化项Optimization | 搬的是哪个房间Which room it moves | 省了显存？Saves memory? | 省了 FLOPs？Saves FLOPs? | 实测结果Measured result |
|---|---|---|---|---|
SEQ_ON_LANE / HND KV 布局KV layout |
纯访存规整度Pure access regularity | 否No | 否No | decode 700 → 5,790 tok/s (8.3×) |
| PCP ring 取代 all-gatherreplacing all-gather | 通信规整度（连号 P2P vs 广播散取）Comm regularity (contiguous P2P vs scattered broadcast) | 否No | 否No | 950 µs 传输 100% 藏进计算阴影（bubble = 0）950 µs of transfer fully hidden under compute (bubble = 0) |
owner_output blockwise combine |
写回规整度Write-back regularity | 否No | 否No | decode 5,500 → 5,650 tok/s |
| GDN 融合 FP8 状态锁 VMEMFused FP8 GDN state pinned in VMEM | 消灭 HBM 写回（访存量）Eliminates HBM write-back (traffic) | 是Yes | 否No | 4,120 µs → 840 µs (4.9×) |
| PCP=8 序列并行sequence parallelism | 显存 + 算力（把 128K 拆成 8 × 16K）Memory + compute (128K cut into 8 × 16K) | 是Yes | 重分布Redistributed | 128K TTFT 32 s → 4.62 s (6.9×，同 8 芯片)128K TTFT 32 s → 4.62 s (6.9×, same 8 chips) |
full 492 µs × 15 vs linear 774 µs × 45。full 492 µs × 15 against linear 774 µs × 45.(16 头, 全序列) 并跨节点搬运——头与序列两个维度同时重分布，是 PD 分离固有的主要开销。#839 open）。(16 heads, full sequence) and moved across nodes — redistributed along the head and sequence axes at once, the dominant intrinsic cost of PD disaggregation.#839 open).owner_output——而不是注意力。
MoE takes two thirds; all attention combined takes one third; full attention alone takes 5.8%.
Deleting all 15 full-attention layers outright would speed prefill up by 5.8%.
(This is the prefill caliber — all three numbers are prefill measurements. Decode keeps a separate ledger; see card E below: MoE still dominates there, but the mechanism becomes weight streaming and tile waste drops to second place.)owner_output — not attention.
| 曾经的说法What it used to say | 缺了哪个限定Missing qualifier | 现在的处理How it reads now |
|---|---|---|
| TTFT 32 s → 4.62 s（6.9×） (6.9×) | 「128K 单请求」+「同样 8 颗芯片」。不写就会被读成扩容收益。"128K, single request" and "the same 8 chips". Without them it reads as a scaling win. | ✅ 全文均标注「同 8 芯片」Annotated "same 8 chips" throughout |
| 700 → 5,790 与and 5,500 → 5,650 tok/s | 两个不同基线，不能相乘、不能相加 —— 与「512 倍与 846 倍是两把尺子」同型。Two different baselines; they cannot be multiplied or added — the same error as chaining a 512× and an 846× figure. | ✅ 各自注明分支与口径Each scoped to its own branch |
| 492 µs = 每跳 × 8 跳 → 15 层 61.8 ms492 µs per hop × 8 → 61.8 ms for 15 layers | 与文档自身预算矛盾：按每跳算三项合计 178.5 ms > 记录的 158 ms 总时延；按每层算合计 126.8 ms，落在预算内。Contradicts the document's own budget: per-hop makes the three terms sum to 178.5 ms against a recorded 158 ms total; per-layer sums to 126.8 ms, inside it. | ✅ 改判为每层，15 层 ≈ 7.4 msRe-read as per layer; 15 layers ≈ 7.4 ms |
| 4,120 µs / 层4,120 µs per layer | 另一把尺子，来源不明（可能是另一配置或含 MoE 的整层）。A different ruler of unknown origin (possibly another configuration, or a whole layer including MoE). | ⚠ 保留但标注「不要与 492 µs 混用」Kept, flagged "do not mix with the 492 µs figure" |
| 阶段带 1.85 s vs 算子表 126.8 ms1.85 s on the stage ribbon vs 126.8 ms in the operator table | 差约 15× —— 典型的「报哪一个」问题。Roughly 15× apart — a textbook "which one do you quote" problem. | ⚠ 标注为未闭合，禁止同表相加Marked unreconciled; must not be summed in one table |
| QKV 输出 2560 列 → 推断芯片数QKV output of 2,560 columns → inferring the chip count | 与上线的 PCP8+EP8（注意力权重复制）对不上。Inconsistent with the deployed PCP8+EP8 form, where attention weights are replicated. | ⚠ 挂起：不得据 trace 形状推断芯片数Open: do not infer chip count from trace shapes |
492 µs × 15 + 774 µs × 45 + 1.41 ms × 60 拆出来的，那组数就是 prefill 实测。
decode 是另一本账：那里既不是注意力、也不是 MXU 说了算，而是专家权重每步都要被完整读一遍。
这一条不需要任何实测就能定下来，因为它只用到权重体积和 HBM 带宽两个数。
The 5.8 / 27.5 / 66.7 split in C④ above is the prefill ledger — it comes out of
492 µs × 15 + 774 µs × 45 + 1.41 ms × 60, and those are prefill measurements.
Decode keeps a different book: there neither attention nor the MXU sets the pace —
the expert weights have to be read end to end on every single step.
This one needs no measurement at all; it uses only two numbers, weight volume and HBM bandwidth.
| 步骤Step | 算式Arithmetic | 结果Result |
|---|---|---|
| 每芯片权重Weights per chip | 397 GB (fp8) ÷ 8 chips |
49.63 GB = 46.22 GiB |
| 每层Per layer | 49.63 GB ÷ 60 |
0.827 GB |
| 读一遍的时间Time to read it once | 0.827 GB ÷ 7.37 TB/s |
112 µs / 层layer |
| decode 单步地板Floor for one decode step | 112 µs × 60 |
6.73 ms |
| 量化到 4 bit 权重后With 4-bit weights | W4A8 / NVFP4 |
3.37 ms |
1 − e^(−B/64)，于是：TPOT 地板(B) = 6.73 ms × (1 − e^(−B/64))1 − e^(−B/64), so:TPOT floor(B) = 6.73 ms × (1 − e^(−B/64))| 假设 batchAssumed batch | 命中专家比例Experts lit | TPOT 地板TPOT floor | 实际 TPOT（由 5,650 tok/s 反推）Actual TPOT (from 5,650 tok/s) | 权重 MBUWeight-only MBU |
|---|---|---|---|---|
| 32 | 39.3% | 2.65 ms | 5.66 ms | 46.8% |
| 64 | 63.2% | 4.26 ms | 11.3 ms | 37.6% |
| 128 | 86.5% | 5.82 ms | 22.7 ms | 25.7% |
| 256 | 98.2% | 6.61 ms | 45.3 ms | 14.6% |
tiles_m ≤ 实际执行块数 ≤ tiles_m + 段数 − 1（每芯片 64 个本地专家 = 64 段）tiles_m = 2，上界 2 + 63 = 65 块 → 32×。对应约 123 µs（bf16 口径），和 112 µs 的权重地板基本持平 —— 第二顺位，但已经贴着。tiles_m = 128，上界 128 + 63 = 191 块 → 仅 +49%。tiles_m ≤ tiles actually executed ≤ tiles_m + segments − 1 (64 local experts per chip = 64 segments)tiles_m = 2, bound 2 + 63 = 65 tiles → 32×. That is ~123 µs in bf16 terms, level with the 112 µs weight floor — second in line, but right up against it.tiles_m = 128, bound 128 + 63 = 191 tiles → only +49%.tile_m = 128 的块占用模型算一遍：
Start with something the report does not say but its own config implies: per-rank FLOPs are identical across all three GPU counts (routed rows per rank = tokens/rank × top-k = 2,560, independent of EP). So "latency drops from 8 to 32 GPUs" is not strong scaling at all — no rank does less work. Only two things change: rows per expert (up) and all-to-all peers (up). Run the tile_m = 128 occupancy model over it:
| 配置Config | 每专家行数 8 / 16 / 32Rows per expert, 8 / 16 / 32 | 块浪费Tile waste | 实测加速 8→16 / 8→32Observed 8→16 / 8→32 | 模型预测Model predicts |
|---|---|---|---|---|
| Qwen 256 tok, k=10 | 40 / 80 / 160 |
3.2× → 1.6× → 1.6× | 1.44× / 1.31× | 2.0 / 2.0 |
| Qwen 512 tok, k=10 | 80 / 160 / 320 |
1.6× → 1.6× → 1.2× | 1.15× / 0.88× | 1.0 / 1.33 |
| DeepSeek 256 tok | 32 / 64 / 128 |
4.0× → 2.0× → 1.00× | 1.46× / 1.73× | 2.0 / 4.0 |
| DeepSeek 512 tok | 64 / 128 / 256 |
2.0× → 1.00× → 1.00× | 1.32× / 1.36× | 2.0 / 2.0 |
tokens/rank × top-k ÷ 专家数 与 GEMM 块的关系。换个 batch，最佳点就搬家。tokens/rank × top-k ÷ experts and the GEMM tile. Change the batch and the optimum moves.| 8 卡8 GPUs | 32 卡32 GPUs | |
|---|---|---|
| top-k=10 每专家行数 → 块数top-k=10 rows/expert → tiles | 40 → 1 | 160 → 2 |
| top-k=6 每专家行数 → 块数top-k=6 rows/expert → tiles | 24 → 1 | 96 → 1 |
| 块数变化Change in tile count | 没变none | 减半halved |
| 实测收益Observed gain | 8% | 30% |
bf16_bf16_tile_m = 128 要 8,192 个 token 才填满一块。TPU 的 megablox 是重复执行边界块而不是补零，形状上不同、账是一样的。所以 EP 的第一价值不是省通信，是把段拼长；而这也解释了为什么 E 卡里那 123 µs 的浪费已经和权重地板同量级。GB200 在 EP8 上做到 55.6% MBU —— 我们 decode 侧的 MBU 是多少，目前没有能算出它的数（缺 batch 和上下文长度），但这是两边口径可比的量，值得测一个出来。
The worst rung in the GB200 set is 40 rows per expert. One decode step here is 256 tokens, top-8, 512 experts — 4 rows per expert, 10× worse, against a megablox bf16_bf16_tile_m = 128 that needs 8,192 tokens to fill a single tile. TPU megablox re-executes boundary tiles rather than zero-padding them; different shape, same bill. So the first value of EP is not saving communication, it is making the segments longer — and it is why the 123 µs of waste in card E is already the same order as the weight floor. GB200 reaches 55.6% MBU at EP8; what our decode MBU is, we currently cannot compute (batch and context length are both missing) — but it is a like-for-like quantity across the two platforms and worth measuring.
GDN (ns) 320、Per-layer (ns) 688，但 688 ns × 60 层 = 41.3 µs，而 Total 行写的是 41 ms。四组全部验过（41.27 / 67.52 / 42.94 / 74.54 对表里的 41 / 67 / 43 / 75）—— 那一列应该是 µs。差 1000 倍，而它长得完全正常。GDN (ns) 320 and Per-layer (ns) 688, but 688 ns × 60 layers = 41.3 µs, while the Total row prints 41 ms. All four configs check out (41.27 / 67.52 / 42.94 / 74.54 against the printed 41 / 67 / 43 / 75) — that column must be µs. A factor of 1000, and it looks entirely normal.SEQ_ON_LANE 选定序列住哪一维、多套预编译拓扑 PCP8→DP8 / DP16 / TP2DP4EP8 / TP2DP8EP16 / P4D4），但换来的静态性是有代价的：
The lecture's three techniques share one shape: replace one dynamic thing with a handful of static ones. This deployment does exactly that (chunked prefill fixed at 16K blocks, SEQ_ON_LANE choosing which axis the sequence lives on, and a set of pre-compiled topologies: PCP8→DP8 / DP16 / TP2DP4EP8 / TP2DP8EP16 / P4D4). But that static-ness was bought:
| 换到了什么What was gained | 付出了什么What it cost | 状态Status |
|---|---|---|
| 规整访存（不必打稀疏 gather 那场仗）Regular access — no sparse-gather battle to fight | 放弃旋钮②，也放弃 MLA；两条公认有效的路线留在桌上Knob ② abandoned, and MLA with it — two proven routes left on the table | 设计取舍By design |
| 吞吐（5,650 tok/s decode · 44k tok/s prefill）Throughput (5,650 tok/s decode, 44k tok/s prefill) | prefill abort 仍未实现；lease 机制仍是 open workstream；2 小时压测未验证且 tpu-sync 有泄漏Prefill abort still unimplemented; the lease mechanism is still an open workstream; the 2-hour stress run is unverified and tpu-sync leaks | 可用性欠账Availability debt |
| PD 分离（prefill 与 decode 各自打满）PD disaggregation (each phase saturated independently) | 跨节点 KV 重分布；带宽只兑现线速的 50%，双网卡未交付Cross-node KV redistribution; only 50% of line rate delivered, dual-NIC still missing | 部分兑现Partly delivered |
| 下一档扩容（PCP16TP2 → DCP32）The next scale-up (PCP16TP2 → DCP32) | all-gather −30% E2E、44k vs 60k tok/s 目标、VMEM 估算 bug、PCP16TP2 正确性 —— 阻塞项清一色是第三个房间的东西，没有一个是显存或算力All-gather at −30% E2E, 44k against a 60k tok/s target, a VMEM estimation bug, PCP16TP2 correctness — every blocker lives in the third room; not one is memory or compute | 阻塞中Blocked |