展示 Qwen 3.5 在单层 Transformer Block 内的完整张量流动生命周期,包含 GQA (3:1)、QK-Norm (Qwen 原创特色)、★★★ KV Cache 显存物理驻留点 ★★★ 以及 MoE (共享专家 + 512 细粒度路由专家)。精确映射 Xprof Trace 中的 fusion.243 (QKV GEMM) 算子。
⚠ 部署形态:Prefill 节点 PCP=8 + EP=8(8 芯片),Decode 节点 TP=2 × DP=4 + EP=8(8 芯片),一对 PD 共 16 芯片。PCP / TP / EP 是同一组 8 芯片上的不同切分轴,不相乘。两阶段的张量形状与显存数字仍需分开读。
The full tensor lifecycle inside one Qwen 3.5 Transformer block: GQA (3:1), QK-Norm (a Qwen signature), the ★★★ physical KV-cache residency point ★★★, and MoE (shared expert + 512 fine-grained routed experts) — mapped onto the fusion.243 (QKV GEMM) operator from an Xprof trace.
⚠ Deployed topology: prefill node PCP=8 + EP=8 (8 chips), decode node TP=2 × DP=4 + EP=8 (8 chips), 16 chips per PD pair. PCP / TP / EP are different partitioning axes over the same 8 chips — they do not multiply. Shapes and memory figures still have to be read per-phase.
bodaborg-tpu7x-nap 上 8 芯片 prefill 节点(PCP=8 + EP=8)· 60 层混合架构(45 GDN + 15 GQA) 那次压测,端到端 4.62 s。
这就是本文档的主口径 —— 联合优化文档记录的上线部署形态即为「1 Prefill node (8 chips, PCP8+EP8) → 1 Decode node (8 chips),一对共 16 芯片」,
PCP / TP / EP 是同一组 8 芯片上的不同切分轴,不相乘,不存在 64 芯片实例。
4.62 s 的对照组是同样 8 芯片上的 DP8(128K 单请求 TTFT 32 s),因此 6.9× 是「同硬件下 DP → PCP 的收益」,不是扩容到更多芯片的超线性加速。
This timeline replays one stress run on bodaborg-tpu7x-nap: an 8-chip prefill node (PCP=8 + EP=8) running a 60-layer hybrid build (45 GDN + 15 GQA), 4.62 s end to end.
This is the document's main configuration — the joint-optimization record describes the deployed form as "1 prefill node (8 chips, PCP8+EP8) → 1 decode node (8 chips), 16 chips per pair".
PCP / TP / EP are different partitioning axes over the same 8 chips and do not multiply; there is no 64-chip instance.
The 4.62 s baseline is DP8 on the very same 8 chips (128K single-request TTFT 32 s), so the 6.9× is "DP → PCP on identical hardware", not superlinear scaling to more chips.
① 每卡 Q 头 12 → 48,KV 头 4 → 16(TP=2);
② fusion.243 的输出列 2560 → 10240,o_proj 为 6144 → 4096;
③ Attn 后的 All-Reduce 域从 8 卡缩为 2 卡;4 个 DP 副本各处理不同请求,互不通信;
④ PCP 消失:decode 时 T=1,序列维无可切分,因此没有 ring/all-gather,每卡持有完整序列的 KV;
⑤ 注意力是 DP=4 而 MoE 是 EP=8,进出 MoE 需要一次 DP↔EP 的 All-to-All 重排。
⑥ PD 交接与物理实例边界:单物理机(4 芯片 / 8 Core)受 758 GiB HBM 限制,只能独立承担 1 个完整的 Prefill 节点或 1 个 Decode 节点。一套端到端 PD 分离集群最小物理开销至少为 2 台物理机(8 芯片)。KV 要从 prefill 8 核的布局重排为 decode 8 核上的 (16 头, 全序列) —— 跨节点搬运且需重分布,是 PD 分离的主要固有开销(#554 / #582)。本图未画。
① Q heads per chip 12 → 48, KV heads 4 → 16 (TP=2);
② fusion.243 output columns 2560 → 10240, o_proj becomes 6144 → 4096;
③ the post-attention All-Reduce domain shrinks from 8 chips to 2; the 4 DP replicas serve different requests and never talk;
④ PCP disappears: at decode T=1, so there is nothing to split along the sequence — no ring/all-gather, and each chip holds the full-length KV;
⑤ attention is DP=4 while MoE is EP=8, so entering and leaving the MoE needs a DP↔EP All-to-All reshuffle.
⑥ PD Handoff & Physical Host Boundary: A single physical host (4 chips / 8 cores, 758 GiB HBM) can only act as an isolated worker for either 1 Prefill node OR 1 Decode node; a full end-to-end PD cluster requires at least 2 hosts (8 chips). KV is reshaped from the prefill layout on 8 cores into (16 heads, full S) on the 8 decode cores — a cross-node move plus a redistribution, and the dominant intrinsic cost of PD disaggregation (#554 / #582). Not drawn here.
bodaborg-tpu7x-nap · 2×4 上 8 芯片 prefill 节点 · 45 GDN + 15 GQA 的压测。
引用前请先看数字溯源:
✓ 有联合优化文档背书 —— 128K 单请求 TTFT DP8 32 s → PCP8 4.62 s、decode 5,650 tok/s(TP2DP4 + owner_output 分支)、单网卡 P→D 峰值 > 12.5 GB/s(线速 25 GB/s,达标线为其 50%)、prefill 分层耗时 full 492 µs × 15 层 / linear 774 µs × 45 层 / MoE 1.41 ms × 60 层。
△ 自洽但属推算 —— 1.92 GB 可由 [131072, 15, 4, 128] FP8 × 2(K,V) 反算得到,是单卡份额(精确值 1.875 GiB;8 卡合计 15 GiB = 整请求 15 层 FA 的 KV),账对得上;但 450 ms → 138 ms(3.2×)是 1.92 GB 分别除以 4.2 / 13.6 GB/s 的算术结果,而文档只背书「峰值超过 12.5 GB/s」。按 12.5 GB/s 应为 ~154 ms。
✗ 与实测冲突 —— 22.45 → 11.20 ms/tok(2.0×):实测 TPOT 为 25.4 ~ 28.4 ms,MTP 在 coding 场景的典型加速比是 1.34× 而非 2.0×。
Absolute latencies come from a stress run on bodaborg-tpu7x-nap · 2×4: an 8-chip prefill node running 45 GDN + 15 GQA layers.
Check provenance before quoting:
✓ Corroborated by the joint-optimization record — 128K single-request TTFT 32 s under DP8 → 4.62 s under PCP8, decode 5,650 tok/s (TP2DP4 on the owner_output branch), single-NIC P→D peak > 12.5 GB/s (line rate 25 GB/s; the target was 50% of it), per-layer prefill cost full 492 µs × 15 / linear 774 µs × 45 / MoE 1.41 ms × 60.
△ Self-consistent but derived — 1.92 GB falls out of [131072, 15, 4, 128] FP8 × 2 (K,V) and is the per-chip share (exactly 1.875 GiB; 8 chips = 15 GiB, i.e. the whole request's KV over 15 FA layers), so the arithmetic checks out. But 450 ms → 138 ms (3.2×) is just 1.92 GB divided by 4.2 and 13.6 GB/s, whereas the record only backs "peak above 12.5 GB/s" — which would give ~154 ms.
✗ Contradicted by measurement — 22.45 → 11.20 ms/tok (2.0×): measured TPOT is 25.4–28.4 ms, and MTP's typical coding-workload speedup is 1.34×, not 2.0×.
传统模式是「先算完再通信」或「先等数据到齐再开算」,计算单元会大量闲置等待,形成通信气泡(bubble)。 在 PCP8 Ring Attention 里,TPU 的 MXU 计算引擎与 ICI 片间互联 DMA 引擎在硬件层面完全解耦,配合双缓冲,计算与通信 100% 异步并发: The naive pattern is compute, then communicate — or wait for the data, then start computing — which leaves the compute units idle and opens a communication bubble. Under PCP8 ring attention the TPU's MXU compute engine and its ICI interconnect DMA engine are fully decoupled in hardware; combined with double buffering, compute and communication run 100% concurrently:
k 块 KV 时,独立 DMA 引擎已经通过 ICI 把邻居的第 k+1 块推进片上 SRAM 备用。
While the compute unit chews on KV chunk k, an independent DMA engine has already pulled the neighbour's chunk k+1 over ICI into on-chip SRAM.
这段窗口完整切下了 Qwen 3.5 397B 进入 Attention 模块时的 QKV 融合投影流水线。三个算子合计约 71.06 µs,加上指令发射与对齐填满 100 µs。 The window captures one complete fused QKV projection pipeline as Qwen 3.5 397B enters its attention block. The three ops total ≈71.06 µs; issue and alignment overhead fill the remaining 100 µs.
• 输入:bf16[4096, 4096](来自 Pre-RMSNorm 的激活)
• 输出:f8e4m3fn[4096, 4096]
• 逻辑:扫描激活绝对值极大值求 Scale,再把 BF16 压成 FP8 送入 MXU。占整个投影 ~25%。
• Input: bf16[4096, 4096] (activations out of Pre-RMSNorm)
• Output: f8e4m3fn[4096, 4096]
• Logic: scan for the absolute max to derive a scale, then squeeze BF16 into FP8 for the MXU. About 25% of the whole projection.
• 算子:convolution.708(MXU 脉动阵列)
• 计算:[4096, 4096] × [4096, 2560] → f32[4096, 2560]
• 流水线反量化:片上连续两次广播乘(mul.14009 / mul.14007)乘以输入与权重 Scale,再 convert 回 bf16。
• Op: convolution.708 (MXU systolic array)
• Compute: [4096, 4096] × [4096, 2560] → f32[4096, 2560]
• Pipelined dequant: two back-to-back on-chip broadcast multiplies (mul.14009 / mul.14007) apply the input and weight scales, then convert back to bf16.
从单卡 bf16[4096, 2560] 中切分:
• Query 分支(slice.7534):前 [0:1536] 列 = 12 个 Q 头(12 × 128),送往 Q-RMSNorm;
• Key & Value 分支(copy-start.766):后 [1536:2560] 共 1024 列 = 4 个 K 头 + 4 个 V 头,送往 K-RMSNorm 并写入 ★★★ KV Cache ★★★。
Splitting the per-chip bf16[4096, 2560]:
• Query branch (slice.7534): columns [0:1536] = 12 Q heads (12 × 128), on to Q-RMSNorm;
• Key & Value branch (copy-start.766): columns [1536:2560], 1024 wide = 4 K heads + 4 V heads, on to K-RMSNorm and then written to the ★★★ KV cache ★★★.
| 算子 | 工作量 | 实测 | 达成率 | 结论 |
|---|---|---|---|---|
| Operator | Work | Measured | Attainment | Verdict |
fusion.243 GEMM |
2×4096×4096×2560 = 85.9 GFLOP 算术强度arithmetic intensity 1781 FLOP/byte |
50.00 µs → 1.72 PFLOP/s | 37.2% of 4614 TFLOPS (fp8) | 理想 18.6 µs,慢 2.7× — 确认为真实低效 ideal 18.6 µs, 2.7× slower — a genuine inefficiency |
clamp_convert_fusion.117 |
两遍:absmax 读 33.5 + convert 读 33.5 + 写 16.8 = 83.9 MB two passes: absmax read 33.5 + convert read 33.5 + write 16.8 = 83.9 MB | 17.81 µs → 4.71 TB/s | 64% of 7.37 TB/s | 访存受限算子,达成率健康(按单遍 50.3 MB 算只有 38%,与实际不符) memory-bound and healthy (a single-pass model would give only 38%, which does not match) |
① GEMM 的 37% 不是 roofline 造成的,两条辩解都已排除:
• 算术强度 1781 FLOP/byte ≫ v7 拐点 312.9(2306 TFLOPS bf16 ÷ 7.37 TB/s),纯 compute-bound;
• v7 的 MXU 为 256×256,本 GEMM 切成 M=16 × K=16 × N=10 个 tile 全部整除,无边角浪费。
→ 嫌疑集中在 epilogue:尾部两次反量化广播乘与 convert 是否被计入这 50 µs。下一步应把 fusion 内部逐指令展开。
② 量化那 17.81 µs 应融进上游 Pre-RMSNorm 的 epilogue:激活 [4096,4096] bf16 = 32 MiB,v7 VMEM 为 64 MiB/core,整块可驻留。在 RMSNorm 输出还在 VMEM 时完成 absmax 与 convert,可省掉 两遍 HBM 读(约 67 MB)。
③ 另需单独检查注意力核心的 Q·Kᵀ:head_dim = 128 只填满 256 宽 MXU 的一半,那里才是形状对齐的真实受害者(本 GEMM 收缩维是 4096,不受影响)。
① The GEMM's 37% is not a roofline effect — both excuses are ruled out:
• arithmetic intensity 1781 FLOP/byte ≫ the v7 ridge point of 312.9 (2306 TFLOPS bf16 ÷ 7.37 TB/s), so it is purely compute-bound;
• the v7 MXU is 256×256, and this GEMM tiles into M=16 × K=16 × N=10 — all exact, no ragged edges.
→ Suspicion lands on the epilogue: whether the two trailing dequant broadcast multiplies and the convert are counted inside these 50 µs. Next step is an instruction-level expansion of the fusion.
② Fold that 17.81 µs quantization into the Pre-RMSNorm epilogue: the [4096,4096] bf16 activation is 32 MiB against 64 MiB/core of v7 VMEM, so the whole tile stays resident. Doing absmax and convert while the RMSNorm output is still in VMEM saves two HBM read passes (~67 MB).
③ Separately audit Q·Kᵀ in the attention core: head_dim = 128 fills only half of a 256-wide MXU — that is where shape misalignment actually bites (this GEMM contracts over 4096 and is unaffected).
本段 trace 同时满足两个特征:QKV 输出 2560 列 = 20480/8(权重按头 8 路切 → TP=8),以及 M = 4096 = 32768/8(token 维 8 等分 → PCP=8 或 chunked prefill 的单块大小)。
⚠ 存疑未决:上线形态是 8 芯片 PCP8 + EP8,其中注意力权重是复制而非按头切分,不应出现 2560 列。因此本段 trace 更可能来自另一个 TP=8 的 prefill 变体(联合优化文档确实并列记录过 DP / TP / PCP 三种 prefill 配置),或 M=4096 只是 chunked prefill 的块大小而非 PCP 切分。在拿到原始 run 配置前,不要据此推断芯片数。
Decode (TP=2) 下同一算子输出应为 10240 列 且 M 不再被 8 整除,不要用本页的 µs 数字推 decode 延迟。
This trace satisfies two signatures at once: QKV output of 2560 columns = 20480/8 (weights split 8-way by head → TP=8), and M = 4096 = 32768/8 (the token axis in eight parts → PCP=8, or one chunked-prefill block).
⚠ Unresolved: the deployed form is 8 chips with PCP8 + EP8, where attention weights are replicated rather than head-sharded, so 2560 columns should not appear. This trace more likely comes from a different TP=8 prefill variant (the joint-optimization record does list DP / TP / PCP prefill configurations side by side), or M=4096 is simply the chunked-prefill block size rather than a PCP shard. Do not infer a chip count from this until the original run config is available.
At decode (TP=2) the same op should emit 10240 columns with M no longer divided by 8 — do not use the µs figures on this page to infer decode latency.
| 计算阶段 / 算子 | 单卡形状 (Prefill, 取自 TP=8 变体 trace ⚠) | 全局形状 | 精度 | 物理驻留 | 功能与架构含义 |
|---|---|---|---|---|---|
| 输入残差流 | [B, T/8, 4096] |
[B, T, 4096] |
bf16 | HBM | 主干隐藏激活;PCP 沿序列维切 8 份 |
| Pre-Attn RMSNorm | [B, T/8, 4096] |
[B, T, 4096] |
bf16 | VPU | 逐 token 归一化,与切分方式无关 |
| FP8 动态量化 (clamp_convert) | [4096, 4096] |
— | f8e4m3fn | VMEM (32 MiB / 64 MiB) | 两遍 absmax + convert,17.81 µs,达峰值带宽 64% |
| ★ QKV GEMM (fusion.243) | [4096, 2560] |
[T, 20480] |
FP8 → bf16 | MXU 256×256 | M=4096 是 PCP 本地 token;2560 是 TP 列分片。50.00 µs,峰值 37.2% |
| QKV Slice (slice.7534) | Q: [B, T/8, 12, 128]KV: [B, T/8, 8, 128] |
Q: [B, T, 96, 128]KV: [B, T, 64, 128] |
bf16 | VPU | 从 2560 列解构出 12 个 Q 头与 4+4 个 KV 头 |
| Q-RMSNorm & K-RMSNorm | [B, T/8, 12, 128][B, T/8, 4, 128] |
[B, T, 96, 128][B, T, 32, 128] |
bf16 | VPU | Qwen 核心设计:限制 Q/K 模长,杜绝超长文本 Softmax 溢出。V 不参与 |
| ★★★ KV Cache (Prefill) ★★★ | [B, S/8, 4, 128] × 2 |
[B, S, 32, 128] × 2 |
fp8 / bf16 | HBM | 单卡 0.47 GiB @32K fp8(15 层 FA)。切分轴存疑:trace 形状读作 4 头 × 全序列,而记录中的部署形态 PCP=8 应为 32 头 × S/8 —— 两者体积完全相同,详见 Xprof 页存疑说明 |
| ★★★ KV Cache (Decode) ★★★ | [B, S, 16, 128] × 2 |
[B, S, 32, 128] × 2 |
fp8 / bf16 | HBM | 只切头不切序列,且 DP=4 不切 KV → 单卡 1.88 GiB @32K fp8(15 层 FA),是 prefill 单卡的 4 倍,容量瓶颈以此行为准 |
| PD 交接 KV Reshard | Prefill 8 卡 (单卡 0.47 GiB @32K) → Decode 8 卡 (16 头, 全序列, 1.88 GiB) |
— | fp8 / bf16 | 跨实例网络 | 头与序列两个维度同时重分布,并跨节点搬运,是 PD 分离的主要固有开销(对应 vllm-torchtpu#554 / #582) |
| PCP Ring / All-Gather | [B, S, 4, 128] × 2 (补齐后) |
— | bf16 | ICI | 仅 Prefill:每 rank 只有 S/8 个 token,注意力需沿 CP 维环形传递 KV |
| GQA FlashAttention | [B, T/8, 12, 128] |
[B, T, 96, 128] |
bf16 | MXU + VPU | 12 个 Q 头共享 4 组 KV (3:1)。head_dim=128 只填满 256 宽 MXU 一半 |
| Out Proj & All-Reduce | [B, T/8, 4096] |
[B, T, 4096] |
bf16 | ICI | 行并行汇聚:Prefill 8 卡,Decode 仅 2 卡(DP 副本之间不通信) |
| Qwen 共享专家 (Shared) | [B, T/8, 4096] |
[B, T, 4096] |
bf16/fp8 | MXU | 常驻激活,中间维 5632,承载通用常识 |
| 路由专家 (512 选 Top-8) | [B, T/8, 4096] |
[B, T, 4096] |
bf16/fp8 | ICI All-to-All (EP=8) | 中间维 1408;EP=8 下每卡常驻 64 个专家,两阶段一致。Decode 时注意力 DP=4 与此处 EP=8 不一致,需额外一次重排 |
| Stage / operator | Per-chip shape (prefill, from the TP=8-variant trace ⚠) | Global shape | Precision | Resides in | What it means |
|---|---|---|---|---|---|
| Input residual stream | [B, T/8, 4096] |
[B, T, 4096] |
bf16 | HBM | Backbone hidden activations; PCP splits the sequence 8 ways |
| Pre-Attn RMSNorm | [B, T/8, 4096] |
[B, T, 4096] |
bf16 | VPU | Per-token normalization; indifferent to how tokens are sharded |
| FP8 dynamic quant (clamp_convert) | [4096, 4096] |
— | f8e4m3fn | VMEM (32 of 64 MiB) | Two passes, absmax + convert; 17.81 µs at 64% of peak bandwidth |
| ★ QKV GEMM (fusion.243) | [4096, 2560] |
[T, 20480] |
FP8 → bf16 | MXU 256×256 | M=4096 is the PCP-local token count; 2560 is the TP column shard. 50.00 µs, 37.2% of peak |
| QKV slice (slice.7534) | Q: [B, T/8, 12, 128]KV: [B, T/8, 8, 128] |
Q: [B, T, 96, 128]KV: [B, T, 64, 128] |
bf16 | VPU | Recovers 12 Q heads and 4+4 KV heads out of the 2560 columns |
| Q-RMSNorm & K-RMSNorm | [B, T/8, 12, 128][B, T/8, 4, 128] |
[B, T, 96, 128][B, T, 32, 128] |
bf16 | VPU | A Qwen signature: bounds the Q/K norms so long-context softmax cannot blow up. V is not normalized |
| ★★★ KV cache (prefill) ★★★ | [B, S/8, 4, 128] × 2 |
[B, S, 32, 128] × 2 |
fp8 / bf16 | HBM | 0.47 GiB per chip at 32K fp8 (15 FA layers). The axis is unresolved: the traced shape reads as 4 heads × full S, while the documented PCP=8 form implies 32 heads × S/8 — identical volume either way; see the open question on the Xprof tab |
| ★★★ KV cache (decode) ★★★ | [B, S, 16, 128] × 2 |
[B, S, 32, 128] × 2 |
fp8 / bf16 | HBM | Heads split, sequence not, and DP=4 does not shard KV → 1.88 GiB per chip at 32K fp8 (15 FA layers), 4× the prefill figure; this row sets the capacity limit |
| PD handoff KV reshard | 8 prefill chips (0.47 GiB each @32K) → 8 decode chips (16 heads, full S, 1.88 GiB) |
— | fp8 / bf16 | Cross-instance network | Redistributes along both the head and sequence axes and crosses nodes — the dominant intrinsic cost of PD disaggregation (vllm-torchtpu#554 / #582) |
| PCP ring / all-gather | [B, S, 4, 128] × 2 (after gather) |
— | bf16 | ICI | Prefill only: each rank owns S/8 tokens, so attention must circulate KV around the CP axis |
| GQA FlashAttention | [B, T/8, 12, 128] |
[B, T, 96, 128] |
bf16 | MXU + VPU | 12 Q heads share 4 KV groups (3:1). head_dim=128 fills only half a 256-wide MXU |
| Out proj & All-Reduce | [B, T/8, 4096] |
[B, T, 4096] |
bf16 | ICI | Row-parallel reduction: 8 chips at prefill, only 2 at decode (DP replicas never communicate) |
| Qwen shared expert | [B, T/8, 4096] |
[B, T, 4096] |
bf16/fp8 | MXU | Always active, intermediate dim 5632, carries general knowledge |
| Routed experts (Top-8 of 512) | [B, T/8, 4096] |
[B, T, 4096] |
bf16/fp8 | ICI All-to-All (EP=8) | Intermediate dim 1408; EP=8 keeps 64 experts resident per chip in both phases. At decode, attention is DP=4 while this is EP=8, so one extra reshuffle is required |
Prefill 把 KV 摊到 8 卡上,单卡 1.88 GiB @128K(记录中的 PCP=8 形态读作 32 头 × S/8,trace 形状读作 4 头 × 全序列,体积相同、轴存疑);Decode 沿 头 (TP=2) 切且 DP 不切 KV(每个 DP 副本完整保存自己那批请求的全序列)。 结果是单卡 KV 相差 4 倍,真正的容量约束永远来自 decode。注意只有 15 层 full-attention 产生 KV,45 层 GDN 不产生。 Prefill spreads KV over 8 chips at 1.88 GiB each @128K (the documented PCP=8 form reads as 32 heads × S/8, the traced shape as 4 heads × full S — same volume, axis unresolved); decode splits heads (TP=2) and DP does not shard KV, since each DP replica keeps the full-length cache for its own requests. The result is a 4× gap in per-chip KV, and the real capacity limit always comes from decode. Note that only the 15 full-attention layers produce KV; the 45 GDN layers do not.
95.93 − 94.74 = 1.19 GiB,另一处是 95.38 − 94.74 = 656.93 MiB(≈0.64 GiB)。这恰好证明它不是常数,别当常数用。95.38G − 94.74G = 656.93M 只有按 1024 才对得上。所以百分比一律按字节同进制相除,不要先换算分子 —— 讲义里有一次「纠正」正是这样把对的改成了错的。95.93 − 94.74 = 1.19 GiB, the other 95.38 − 94.74 = 656.93 MiB (≈0.64 GiB). Which is precisely the proof that it is not a constant — do not reuse it as one.95.38G − 94.74G = 656.93M only reconciles base-1024. So percentages are taken byte-wise in a single base; never convert the numerator first — one lecture "corrected" a right answer into a wrong one exactly that way.
Decode 差异:每卡 12Q/4KV → 48Q/16KV,2560 → 10240 列,o_proj 6144→4096,All-Reduce 域 8 卡 → 2 卡,PCP ring 整段消失,并额外需要注意力 DP=4 与 MoE EP=8 之间的 All-to-All 重排。
Decode differences: 12Q/4KV → 48Q/16KV per chip, 2560 → 10240 columns, o_proj 6144→4096, the All-Reduce domain shrinks from 8 chips to 2, the PCP ring stage disappears entirely, and a DP=4 ↔ EP=8 All-to-All reshuffle is added around the MoE.
[B, T, D]
[B, T, 4096] (主干残差流)
│
┌───────────────────┴───────────────────┐ (残差旁路)
▼ │
【Pre-Attn RMSNorm】 (attn_norm) │
│ │
[B, T/8, 4096] (PCP=8 切序列) │
│ │
▼ │
┌─────────────────────────┐ │
│ ★ fusion.243 (QKV 投影) │ (trace: TP=8 变体列切) │
│ 4096 → 2560 │ (全局 4096 → 20,480) │
│ M = T/8 = 4096 │ (PCP 本地 token 数) │
└────────────┬────────────┘ │
│ │
[B, T/8, 2560] │
│ │
┌────────────┴────────────┐ (下游切片 slice.7534) │
▼ ▼ │
Q 投影 KV 投影 │
2560 切出前 1536 2560 切出后 1024 │
[B, T/8, 12, 128] [B, T/8, 8, 128] │
│ ├── K: [B, T/8, 4, 128] │
│ └── V: [B, T/8, 4, 128] │
▼ │ │ │
【Q-RMSNorm】 (Qwen 特色) │ │ │
│ ┌───────┘ │ │
▼ ▼ │ │
[B, T/8, 12, 128] 【K-RMSNorm】 (Qwen) │ │
│ │ │ │
▼ ▼ │ │
[Q-RoPE 旋转编码] [K-RoPE 旋转编码] │ │
│ │ │ │
▼ ▼ ▼ │
[B, T/8, 12, 128] [B, T/8, 4, 128] [B, T/8, 4, 128] │
│ │ │ │
│ ┌─────────┴───────────────┴────────┐ │
│ │ ★★★ KV Cache 显存物理驻留点 ★★★ │ │
│ │ Prefill: [B, S/8, 4, 128] × 2 │ │
│ │ Decode : [B, S, 16, 128] × 2 │ │
│ └─────────────────┬────────────────┘ │
│ │ │
│ 【PCP=8 Ring / All-Gather KV】 │
│ 仅 Prefill:每 rank 只有 S/8 个 token │
│ │ │
└─────────────────┬───────────────┘ │
▼ │
/──────────────────────\ │
< GQA Flash/SplashAttn > │
< Softmax(Q·K^T / √128)·V> │
\──────────────────────/ │
│ │
[B, T/8, 12, 128] │
│ (展平为 [B, T/8, 1536]) │
▼ │
【Out Proj (o_proj)】 │
1536 → 4096 (单卡局部) │
│ │
▼ │
【All-Reduce】Prefill 8 卡 / Decode 2 卡 │
│ │
[B, T, 4096] │
│ │
▼ │
【残差相加 ① (Residual Add)】 ◄───────┘
│
[B, T, 4096]
│
┌────────────┴────────────┐ (残差旁路)
▼ │
【Post-Attn RMSNorm】 (ffn_norm) │
│ │
┌─────────┴─────────┐ │
▼ ▼ │
【共享专家 (Shared)】 【Router 路由器】 │
4096 → 5632 4096 → 512 专家 │
(SwiGLU 常驻激活) Softmax 选 Top-8 │
│ │ │
│ All-to-All 派发 (EP=8) │
│ 每卡常驻 64 个专家 │
│ ▼ │
│ 【8 个路由专家 (Routed)】 │
│ 4096 → 1408 (SwiGLU) │
│ │ │
│ 加权求和 & All-to-All 回传 │
│ ▼ │
[B, T, 4096] [B, T, 4096] │
└─────────┬─────────┘ │
▼ │
【MoE 合并相加】 │
│ │
▼ │
【残差相加 ② (Residual Add)】 ◄──────┘
│
[B, T, 4096] (输出送入下一层 Transformer)
[B, T, 4096] residual stream
│
┌───────────────────┴──────────────────────────────┐ residual bypass
▼ │
[Pre-Attn RMSNorm] (attn_norm) │
│ │
▼ │
┌────────────────────────────────┐ │
│ * fusion.243 QKV projection │ TP=8-variant col split│
│ 4096 -> 2560 per chip │ (global 20,480) │
│ M = T/8 PCP=8 local │ │
└───────────────┬────────────────┘ │
│ │
[B, T/8, 2560] │
│ │
┌───────────────┴──────────────┐ slice.7534 │
▼ ▼ │
Q [0:1536] KV [1536:2560] │
[B, T/8, 12, 128] [B, T/8, 8, 128] │
│ ├── K [B, T/8, 4, 128] │
│ └── V [B, T/8, 4, 128] │
▼ │ │ │
[Q-RMSNorm] Qwen │ │ │
│ ┌─────┘ │ │
▼ ▼ │ │
[Q-RoPE] [K-RMSNorm] Qwen │ │
│ │ │ │
│ ▼ │ │
│ [K-RoPE] │ │
│ │ │ │
│ ┌──────────────┴──────────────┘ │
│ ▼ │
│ ╔═══════════════════════════════════════╗ │
│ ║ *** KV CACHE HBM resident *** ║ │
│ ║ prefill : [B, S/8, 4, 128] x2 ║ │
│ ║ decode : [B, S, 16, 128] x2 ║ │
│ ╚═══════════════════╤═══════════════════╝ │
│ │ │
│ ring / all-gather KV over the 8 PCP ranks │
│ (prefill only: each rank owns S/8 tokens) │
│ │ │
└──────────┬───────────┘ │
▼ │
/─────────────────────────────────────\ │
< GQA Flash / Splash Attention > │
< softmax(Q.K^T / sqrt(128)) . V > │
< 12 Q heads share 4 KV groups (3:1) > │
\─────────────────────────────────────/ │
│ │
[B, T/8, 12, 128] -> flat [B, T/8, 1536] │
▼ │
[Out Proj (o_proj)] 1536 -> 4096 │
│ │
▼ │
[All-Reduce] prefill 8 chips / decode 2 chips │
│ │
▼ │
[Residual Add 1] <──────────────────────────────────────┘
│
[B, T, 4096]
│
┌──────────┴──────────────────────────────┐ residual bypass
▼ │
[Post-Attn RMSNorm] (ffn_norm) │
│ │
├──────────────────┐ │
▼ ▼ │
[Shared Expert] [Router] │
4096->5632->4096 4096 -> 512 experts │
SwiGLU, always on softmax, pick Top-8 │
│ │ │
│ All-to-All dispatch │
│ (EP=8, 64 experts per chip) │
│ ▼ │
│ [Top-8 Routed Experts] │
│ 4096->1408->4096 SwiGLU │
│ │ │
│ weighted sum + All-to-All combine│
│ ▼ │
└────────┬─────────┘ │
▼ │
[MoE Merge Add] │
│ │
▼ │
[Residual Add 2] <─────────────────────┘
│
[B, T, 4096] -> next Transformer layer
跨卡大模型剖析有个众所周知的坑:单次采样超过 1 s 就容易撑爆片上环形 buffer,导出的 trace 又会把浏览器 Chrome Trace Viewer 撑到 OOM。 本大盘改用外科手术式分阶段采样(每窗口 ≤ 160 ms),精确透视 5 处核心突破的微观硬件执行状态,并配齐对应的 Xprof 剖面视图与代码 PR。 Profiling a multi-chip model has a well-known trap: sample for more than about a second and the on-chip ring buffer overflows, while the exported trace is large enough to OOM Chrome's trace viewer. This board instead uses surgical per-stage sampling with windows of 160 ms or less, giving a clean view of the micro-architectural state behind each of the five breakthroughs, alongside the matching Xprof view and code PR.
bodaborg-tpu7x-nap · 2×4 上的
8 芯片 prefill 节点 / 60 层混合架构(45 GDN + 15 GQA),这也是本文档的主口径(PCP=8 + EP=8,无 64 芯片实例)。bodaborg-tpu7x-nap · 2×4 from an
8-chip prefill node running the 60-layer hybrid build (45 GDN + 15 GQA), which is also this document's main configuration (PCP=8 + EP=8; there is no 64-chip instance).
在真实的 Google Cloud TPU v7x 物理集群(bodaborg-tpu7x-nap,8 TensorCores + 16 SparseCore Subcores)上实测 49,152 Prefix Cache + 1,024 并发长文本场景。攻克传统动态稀疏注意力在 TPU 上的两大死穴:① 页面粗筛语义稀释(Naive 64-Page 导致 Needle 召回跌至 6.97%)与 ② TensorCore 停机空转等待 9.59ms Gather 搬运。通过方案 3+4(32-Tile + Attention Sink + 局部滑动窗口)与 Native SparseCore 32 子核心异步 DMA 双缓冲流水线(Pallas Kernel),核心 Sink 与局域窗口语义强制锁定,Gather 暴露开销降至 ~0.00ms(完全与 TC 计算重叠)!
Benchmarked on physical TPU v7x hardware (bodaborg-tpu7x-nap, 8 TensorCores + 16 SparseCore Subcores) with 49,152 Prefix Cache + 1,024 concurrency. Overcame two major hurdles of dynamic sparse attention on TPUs: 1) Coarse page dilution (Naive 64-page drops Needle recall to 6.97%) and 2) TensorCore idling for 9.59ms during gather. With Scheme 3+4 (32-Tile + Attention Sink + Sliding Window) and Native SparseCore 32-subcore async DMA double buffering (Pallas Kernel), key semantic contexts are locked, and exposed gather latency is eliminated (~0.00ms)!
| 优化项 / 模块 | XLA HLO 算子名 | 物理执行硬件 | 张量 Shape 与 Dtype | 实测耗时 | 访存流量与算术强度 | 微架构瓶颈与突破点 |
|---|---|---|---|---|---|---|
| 【优化 1】 GDN 线性状态更新 |
%fusion.gdn_v3_recurrent(fused_decay_outer_prod) |
TPU VPU 8×128 向量寄存器 + VMEM |
S: [H, 128, 128] FP8Q/K/V: [4K, H, 128] FP8α/β: [4K, H] BF16 |
4,120 µs 840 µs (4.9× 提速,每层) |
流量 16 KB/head 强度 64 FLOPs/B 零写回 HBM |
• 原生痛点:memory-bound,反复读写 32 KB 的 BF16 状态。 • 算子突破:FP8 压缩把 128×128 状态矩阵降到 16 KB,完全锁在片上 VMEM,内层展开为连续 FMA 原地更新,显存落盘归零。 |
| 【优化 3a】 PCP8 Ring 计算 |
%custom-call[FlashAttention](dot_general_matmul) |
TPU MXU 128×128 脉动阵列 |
Q: [16K, 128] BF16K/V: [16K, 128] BF16Tile: [128, 128] FP32 acc |
492 µs / 层 15 层合计 ≈ 7.4 ms ⚠ 另有 4,120 µs 读数,来源不明 |
FLOPs 8.59 TFLOP 强度 168 FLOPs/B Compute-bound |
• 算子执行:脉动阵列以双缓冲流水吸收来自 VMEM 的 Q 与 K 分块。 • 重叠屏障:4,120 µs 的计算时长构成「算力保护伞」,为下方通信提供充裕的遮蔽窗口。 |
| 【优化 3b】 PCP8 ICI 环形通信 |
%collective-permute.async-start(ring_p2p_transfer) |
ICI DMA 引擎 Optical / copper direct bus |
KV chunk [16K, 4, 128]数据量 16.38 MB Dtype BF16 |
物理耗时 950 µs 显式气泡 0 µs (100% 隐藏) |
带宽 17.24 GB/s 单卡双向环路 双缓冲 ping-pong |
• 原生痛点:All-Gather 广播引发网络拥塞与显存膨胀。 • 算子突破:改为 P2P 邻居环形搬运,用异步 XLA start/done 注解把 950 µs 传输整个塞进 4,120 µs 的计算内部。 |
| 【优化 2】 DCN 跨切片直传 |
%dcn_rdma_send_stream(seq_on_lane_transfer) |
200G DCN NIC PCIe Gen5 + RDMA/TCP |
128K KV cache[131072, 15, 4, 128] FP8单卡载荷 1.92 GB (= 1.875 GiB · 8 卡合计 15 GiB) |
450 ms 138.0 ms (3.2× 提速) |
推算 13.6 GB/s 文档背书 >12.5 GB/s (线速 25 GB/s) Zero-copy |
• 原生痛点:离散排布导致 scatter-gather DMA 握手延迟,带宽利用率不足 30%。 • 算子突破: SEQ_ON_LANE 在生成时前置 KV heads 维,形成物理连续大块,一笔就能拉满网卡线速。• 口径换算:1.92 GB (= 1.875 GiB) 是单卡份额,8 卡合计 15 GiB / 请求(15 层 FA × 32 KV 头,FP8)。138 ms 是 1.92 GB ÷ 13.6 GB/s 的推算值;按文档背书的 >12.5 GB/s 则为 ~154 ms。双网卡(400 Gbps)尚未交付,仍是阻塞项 b/558565285。 |
| 【优化 4】 MoE 专家原地聚合 |
%fusion.owner_output_combine(in_place_weighted_reduce) |
TPU VPU / ALU Owner core 本地累加器 |
Top-8 专家输出[8, hidden] BF16weights [8] FP32 |
3,100 µs 2,140 µs (+31%) |
显存申请 0 B (消灭临时 buffer) 节省显存带宽 65% |
• 原生痛点:通用框架在接收端分配临时 buffer 倒手三遍,频繁触发分配器锁竞争。 • 算子突破:数据流式到达宿主卡时,直接在本地累加器上原地乘加门控权重,中间显存分配彻底消失。 |
| 【优化 5】 MTP RPA 投机验证 |
%fusion.rpa_verify_speculative(tree_mask_flash_decode) |
TPU MXU + VPU 流水线分块执行 |
候选 token 2~3 speculative mask [3, 3]KV cache: paged blocks |
22.45 ms/tok 11.20 ms/tok (单步命中 2 词) |
权重复用率 2.0× (单次 HBM 读权重产出双词) 吞吐 5,650 tok/s |
• 原生痛点:树状推测 mask 导致脉动阵列计算不对齐,产生分支气泡。 • 算子突破:RPA verify 特化把多候选验证与主前向按规则分块对齐,额外验证耗时趋近 0,硬件算力利用率从 12% 提升到 25%。 |
| Optimization / module | XLA HLO op | Physical unit | Tensor shape & dtype | Measured latency | Traffic & arithmetic intensity | Microarchitectural bottleneck and fix |
|---|---|---|---|---|---|---|
| Opt 1 GDN linear state update |
%fusion.gdn_v3_recurrent(fused_decay_outer_prod) |
TPU VPU 8×128 vector registers + VMEM |
S: [H, 128, 128] FP8Q/K/V: [4K, H, 128] FP8α/β: [4K, H] BF16 |
4,120 µs 840 µs (4.9× per layer) |
Traffic 16 KB/head Intensity 64 FLOPs/B Zero HBM write-back |
• Original: memory-bound, re-reading and rewriting a 32 KB BF16 state every step. • Fix: FP8 shrinks the 128×128 state to 16 KB so it stays pinned in on-chip VMEM; the inner loop unrolls into contiguous in-place FMAs and nothing spills to HBM. |
| Opt 3a PCP8 ring compute |
%custom-call[FlashAttention](dot_general_matmul) |
TPU MXU 128×128 systolic array |
Q: [16K, 128] BF16K/V: [16K, 128] BF16Tile: [128, 128] FP32 acc |
492 µs / layer 15 layers ≈ 7.4 ms ⚠ a 4,120 µs reading also exists, source unknown |
FLOPs 8.59 TFLOP Intensity 168 FLOPs/B Compute-bound |
• Execution: the systolic array absorbs Q and K tiles from VMEM through a double-buffered pipeline. • Overlap barrier: 4,120 µs of compute acts as an umbrella, leaving a wide window underneath for the transfer to hide in. |
| Opt 3b PCP8 ICI ring transfer |
%collective-permute.async-start(ring_p2p_transfer) |
ICI DMA engine Optical / copper direct bus |
KV chunk [16K, 4, 128]Payload 16.38 MB Dtype BF16 |
Physical 950 µs Explicit bubble 0 µs (100% hidden) |
Bandwidth 17.24 GB/s Bidirectional ring per chip Double-buffered ping-pong |
• Original: an All-Gather broadcast congested the network and inflated memory. • Fix: replaced by nearest-neighbour P2P ring transfers; async XLA start/done annotations tuck the whole 950 µs inside the 4,120 µs of compute. |
| Opt 2 DCN inter-slice transfer |
%dcn_rdma_send_stream(seq_on_lane_transfer) |
200G DCN NIC PCIe Gen5 + RDMA/TCP |
128K KV cache[131072, 15, 4, 128] FP8Per-chip 1.92 GB (= 1.875 GiB · 15 GiB over 8 chips) |
450 ms 138.0 ms (3.2×) |
Derived 13.6 GB/s record backs >12.5 GB/s (25 GB/s line rate) Zero-copy |
• Original: the scattered layout forced scatter-gather DMA handshakes, holding link utilization under 30%. • Fix: SEQ_ON_LANE hoists the KV-head axis at generation time into one physically contiguous block that saturates the NIC in a single descriptor.• Conversion: 1.92 GB (= 1.875 GiB) is the per-chip share; across 8 chips that is 15 GiB per request (15 FA layers × 32 KV heads, FP8). The 138 ms is 1.92 GB ÷ 13.6 GB/s; at the documented >12.5 GB/s it would be ~154 ms. Dual-NIC (400 Gbps) is not delivered — still blocked, b/558565285. |
| Opt 4 In-place MoE expert combine |
%fusion.owner_output_combine(in_place_weighted_reduce) |
TPU VPU / ALU Owner-core local accumulator |
Top-8 expert outputs[8, hidden] BF16weights [8] FP32 |
3,100 µs 2,140 µs (+31%) |
Allocation 0 B (temporary buffers gone) 65% less memory bandwidth |
• Original: the generic path allocated receive-side temporaries and passed the data through three times, hammering the allocator lock. • Fix: as results stream into the owner chip, the gating weights are multiplied and accumulated in place in a local accumulator — no intermediate allocation at all. |
| Opt 5 MTP RPA speculative verify |
%fusion.rpa_verify_speculative(tree_mask_flash_decode) |
TPU MXU + VPU Pipelined tiled execution |
Candidate tokens 2–3 speculative mask [3, 3]KV cache: paged blocks |
22.45 ms/tok 11.20 ms/tok (2 tokens accepted/step) |
Weight reuse 2.0× (one HBM weight read, two tokens) 5,650 tok/s |
• Original: the tree-shaped speculative mask misaligned the systolic array and opened branch bubbles. • Fix: the specialized RPA verify aligns multi-candidate verification with the main forward pass on a regular tiling, so the extra verification cost approaches zero and utilization climbs from 12% to 25%. |
2026年6月18日,Google Core ML 与阿里联合召开了三场关键技术专题攻坚会(由 Terance Sum 发起,Felix Li 等核心技术专家主讲),
针对 Qwen 3.5 397B-A17B 落地 Google Cloud TPU v7x(Ironwood)展开了涵盖「单层算子优化与 MoE 调度」、「P/D 分离传输架构」、「KV Cache 池化与长上下文显存治理」的三维深度对齐。
历经整整 3 个月的联合工程攻关(2026-06-18 ➔ 2026-09-18),最初梳理的原型缺陷、理论假设与客户核心卡点,已全量转化为确定性、工业级的量化交付战果:
On June 18, 2026, Google Core ML and Alibaba convened three deep-dive technical sessions (organized by Terance Sum, led by Felix Li and core engineers),
establishing a 3D joint technical alignment for deploying Qwen 3.5 397B-A17B onto Google Cloud TPU v7x (Ironwood) across "Single-Layer Operator & MoE Scheduling", "P/D Disaggregation Transport Architecture", and "KV Cache Pooling & Long-Context Memory Containment".
Through 3 months of intensive co-engineering (2026-06-18 ➔ 2026-09-18), the initial architectural bottlenecks, compiler bugs, and customer pain points have been fully resolved into validated production deliverables:
handsomehow/owner_output 原地汇聚分支,消除中间拷贝气泡;Decode 吞吐实测达标 5,650 tok/s。
• Merged handsomehow/owner_output direct gather branch; decode throughput verified at 5,650 tok/s (target met).
SEQ_ON_LANE 零拷贝连续布局,DCN 单网卡峰值突破 >12.5 GB/s(达 25 GB/s 线速 50%)。
• Standardized 1P1D (16-chip) topology; deployed SEQ_ON_LANE zero-copy layout; DCN peak >12.5 GB/s (50% of 25 GB/s line rate).
| 攻坚技术维度Dimension | 6.18 联合对齐原始诉求与核心痛点6.18 Initial Demands & Bottlenecks | 9.18 闭环落地工程方案9.18 Resolution & Implementation | 实测交付收益与量化指标Measured Deliverable / Metric | 关联核心资产Associated Asset |
|---|---|---|---|---|
| 128K 长上下文计算墙 (TTFT) 128K Context Wall (TTFT) | DP8 架构下处理 128K 序列面临严重 HBM 访存墙,单 Prefill 过程拖死整个集群,TTFT 耗时高达 32.0 秒,严重超标。 DP8 hits severe HBM bandwidth wall on 128K sequence; prefill throttles cluster, causing unacceptable TTFT of 32.0 s. | Prefill 节点采用 PCP=8 + EP=8,把 128K 序列切成 8 个 16K 分片,利用 TPU 片上大容量 SRAM 绕开 HBM,并以 Ring Attention 实现通信与计算 100% 重叠(Bubble = 0 µs)。 Prefill runs PCP=8 + EP=8, chunking 128K sequence into eight 16K shards in SRAM, with Ring Attention completely overlapping compute and ICI comms (0 µs bubble). |
128K TTFT 4.62 s (相比 DP8 提速 6.9×) 128K TTFT 4.62 s (6.9× speedup vs DP8) |
专题三录像 🔗Topic 3 Video 🔗 |
| MoE 512 细粒度专家路由 MoE 512-Expert Routing | 512 个细粒度专家在 All-to-All 路由回传后,通用框架存在多次中间显存拷贝与 CPU 调度空洞,吞吐长期徘徊在 5,500 tok/s 以下。 512 fine-grained experts had multiple redundant intermediate HBM copies and CPU scheduling gaps after All-to-All, capping throughput below 5,500 tok/s. |
合并 handsomehow/owner_output 特化直聚分支,让负责计算专家的 Owner Core 在本地原地完成输出汇聚(direct gathering),彻底消除冗余内存中转气泡。
Merged specialized handsomehow/owner_output branch; owner core computes output gathering locally in-place, eliminating intermediate copy bubbles.
|
Decode 5,650 tok/s (由 5,500 稳步提升达标) Decode 5,650 tok/s (Target achieved) |
专题一录像 🔗Topic 1 Video 🔗 |
| P/D 跨拓扑 DCN 传输 Inter-Slice DCN Transport | Prefill 切片(PCP 拓扑)传输至 Decode 切片(TP·DP 拓扑)时,动态 resharding 触发海量 CPU/DMA 内存转置,DCN 单网卡吞吐严重受挫。 Moving KV cache from prefill (PCP) to decode (TP/DP) triggered heavy dynamic resharding transposes, severely choking cross-slice DCN throughput. |
设计并部署 SEQ_ON_LANE 零拷贝内存布局,将 KV Heads 维度前置,使跨机 DMA 引擎能够以单一连续内存流直接推入 DCN 网卡。
Deployed SEQ_ON_LANE zero-copy layout, hoisting KV heads axis so cross-host DMA engine pushes contiguous streams directly into DCN NIC.
|
DCN 峰值 >12.5 GB/s (达 25 GB/s 线速 50%) DCN Peak >12.5 GB/s (50% of wire speed) |
专题二录像 🔗Topic 2 Video 🔗 |
| Host 内存泄漏治理 Host DRAM Leak Fix | 早期压测中 8-device 整机 DRAM RSS 飙升至 ~700 GB 且长期不释放,导致后续 KV 管理服务(KVCM)无法拉起,系统濒临 OOM 崩溃。 Whole-node DRAM RSS climbed to ~700 GB across 8 devices and never freed, preventing KVCM from initializing and triggering system-level OOMs. | ① 缩容 premapped VMA 配额(原 16 GiB);② 全面引入 jemalloc 替换 glibc 内存分配器;③ 完善 prefix-aware 动态加载逻辑。 (1) Shrunk premapped VMA quota (was 16 GiB); (2) Replaced glibc allocator with jemalloc; (3) Added prefix-aware dynamic memory loading. |
DRAM 稳定受控释放 (多轮长对话冒烟全通) DRAM Stable & Released (Multi-turn tests pass) |
专题三纪要 🔗Topic 3 Notes 🔗 |
| 部署拓扑与并发容量 Topology & Concurrency | 原先缺乏高性价比物理拓扑标准,社区方案甚至提出动用 64 芯片实例,成本极高且资源利用率严重不均衡。 Lack of standardized production topology; theoretical proposals assumed 64-chip instances, which were economically infeasible and unbalanced. | 固化 1P1D(共 16 芯片) 部署形态:Prefill 采用 PCP8+EP8(8 芯片)→ Decode 采用 TP2·DP4·EP8(8 芯片),TP2 将 Dense 显存减半支撑 Batch=64 满并发。 Established 1P1D (16 chips total) topology: Prefill runs PCP8+EP8 (8 chips) → Decode runs TP2·DP4·EP8 (8 chips); TP=2 halves dense weights to allow full Batch=64 concurrency. |
16 芯片对齐 1P1D (TP2 All-Reduce 仅 85µs) 16-Chip 1P1D Pair (TP2 All-Reduce 85 µs) |
9.04 周会纪要 🔗9.04 Sync Notes 🔗 |
Qwen 3.5 397B 在 Google Cloud TPU v7 上落地过程中,针对超长上下文算力墙、MoE 输出搬运气泡、跨机 Reshard 内存转置损耗、DRAM 内存泄漏等工业级痛点,实施了多项系统级与 Kernel 级攻坚,全面达成并超越压测指标: Bringing Qwen 3.5 397B up on Google Cloud TPU v7 ran into four production-grade walls — the long-context compute wall, MoE output-movement bubbles, cross-host reshard transpose overhead, and a DRAM leak. Each was attacked at the system or kernel level, and every stress-test target was met or beaten:
bodaborg-tpu7x-nap (cloud-tpu-shared-capacity) · 2×4 拓扑 上的
8 芯片 prefill 节点 / 60 层混合架构(45 GDN + 15 GQA)压测。
这就是本文档的主口径 —— 上线部署形态为 Prefill 节点 PCP=8 + EP=8(8 芯片)→ Decode 节点 TP=2 × DP=4 + EP=8(8 芯片),一对共 16 芯片,不存在 64 芯片实例。
Decode 侧 5,650 tok/s 有文档背书(TP2DP4 + owner_output 分支,由 5,500 提升而来);11.20 ms/tok 则与实测 TPOT 25 ~ 28 ms 冲突,引用前请核对。
Every measured figure on this page comes from a stress run on bodaborg-tpu7x-nap (cloud-tpu-shared-capacity) · 2×4 topology using an
8-chip prefill node on the 60-layer hybrid build (45 GDN + 15 GQA).
That is the document's main configuration: the deployed form is a prefill node with PCP=8 + EP=8 (8 chips) feeding a decode node with TP=2 × DP=4 + EP=8 (8 chips) — 16 chips per pair, with no 64-chip instance.
Decode's 5,650 tok/s is corroborated (TP2DP4 on the owner_output branch, up from 5,500); 11.20 ms/tok, however, conflicts with the measured TPOT of 25–28 ms — verify before quoting.
SEQ_ON_LANE 布局把 KV Heads 维度前置,让跨机 DMA 引擎能以连续内存流直接推入 DCN 网卡。
• Problem: moving KV from the prefill slice (PCP topology) to the decode slice (TP·DP topology) triggered dynamic resharding, and the resulting CPU/DMA transposes throttled network throughput badly.SEQ_ON_LANE layout hoists the KV-head axis to the front, so the cross-host DMA engine can push one contiguous stream straight into the DCN NIC.
leases=1 DRAM 预估偏差。
• Problem: early in the stress campaign, whole-node DRAM RSS across 8 devices climbed to about 700 GB and never came back down, which left KVCM unable to start afterwards.leases=1 DRAM estimation.
依据 2026-09 ➔ 2026-10 联合性能优化实测记录(含 4-Host 跨机压测与 9 月下旬生产级冲刺), 系统以 NVIDIA GB200 (NVLink 全互联拓扑) 作为生产级黄金基准(Golden Baseline)。 双方联合设定的生产上线目标为 达到 GB200 的 90% 吞吐。以下为端到端实测达成情况与深层微架构对账: Based on 2026-09 ➔ 2026-10 performance benchmarks (including 4-Host multi-node runs and the late-September production sprint), the system benchmarks against NVIDIA GB200 (NVLink-connected topology) as the industrial Golden Baseline. The production deployment target is set at 90% of GB200 throughput. Below is the verified scorecard and architectural breakdown:
owner_output)Mid-Sep Baseline: 5,650 tok/s (TP2·DP4·EP8 + owner_output)| 核心维度Dimension | Google TPU v7 (Ironwood)Google TPU v7 (Ironwood) | NVIDIA GB200 (Blackwell NVL72)NVIDIA GB200 (Blackwell NVL72) |
|---|---|---|
| 计算核心、精度与 FP4 / W4A8 真相Compute Cores & Precision / W4A8 |
256×256 脉动阵列 (MXU) · 官方白盒参数与 W4A8 瞬时反量化 • 硬件真相(OpenXLA 官方白盒实锤):根据 OpenXLA 2026 最新开源基准( xla/benchmarks/core/platform_info.py),TPU v7 (Ironwood) 运行在 P-State 3(默认 1.9 GHz,可动态突发至 P7 2.2 GHz)。其硬件阵列原生仅支持 INT8 / FP8 / BF16(发射周期 8 cycles,延迟 204~211 cycles),完全无原生 FP4 浮点乘加硬件(原生 FP4 延迟至 TPU v8i 引入)。• 阿里软件工程突破 (W4A8 & Subchannel 量化):阿里团队通过 NVFP4 + W4A8(PR #912)将静态权重以 4-bit(INT4/FP4)脱水存储,在片上向量单元(VPU)以 Subchannel 微缩放粒度就地瞬时反量化(De-quantize)为 8-bit 送入 256×256 脉动阵列,以微小计算代价换取 HBM 访存带宽直接翻倍!
256×256 Systolic Array (MXU) · OpenXLA Ground Truth & W4A8 Dequantization• Silicon Reality (OpenXLA Benchmarks): Per OpenXLA 2026 specs ( xla/benchmarks/core/platform_info.py), TPU v7 (Ironwood) runs at P-State 3 (default 1.9 GHz, burstable to P7 2.2 GHz). It natively supports INT8/FP8/BF16 (cadence 8 cycles, latency 204~211 cycles) and completely lacks native FP4 MAC silicon (native FP4 is deferred to TPU v8i).• Alibaba Software Breakthrough (W4A8 & Subchannel Quant): The NVFP4 + W4A8 pipeline (PR #912) compresses weights to 4-bit in HBM with subchannel micro-scaling, de-quantizing on-chip before feeding the 8-bit MXU, doubling effective memory bandwidth.
|
第 5 代原生 NVFP4 / FP8 Tensor Cores • 具备硬件原生 4-bit 浮点微缩放乘加器(Micro-scaling FP4),在密集 Prefill GEMM 上理论 FLOPs 占优;但原生 FP4 存在一定数值溢出与精度损失风险,需严苛评估 MMLU。 5th-Gen Native NVFP4 / FP8 Tensor Cores • Features true silicon-native micro-scaling FP4 MAC units, giving theoretical FLOPs advantage in dense prefill; however, native FP4 requires rigorous eval to prevent accuracy drop. |
| 芯片封装、物理主机 vs. Device 对账 & 拓扑Packaging: Host vs. Chip vs. Device & Fabric |
物理事实:单 Host 4 芯片(8 Core)· 8 芯片切片为双机(2 Hosts) • 微架构与 Die-to-Die:TPU v7 (Ironwood) 单颗物理 Chip 封装 2 个独立计算裸晶(Die),在驱动层暴露为 2 个逻辑 Device(Core),两 Die 间由片上高速超宽 Die-to-Die 互联直连。 • 单机与双机拓扑基准: - 物理单机(1 Host, tpu7x-standard-4t):仅含 4 颗物理芯片(8 个 Core),拓扑为 3D Torus 2×2×1。其 758 GiB 可用 HBM 只能作为单一角色的最小物理部署单元(独占跑 PCP8 Prefill 或独占跑 TP2·DP4 Decode),物理上无法在一台单机内混跑 P/D 双实例;- 双机切片(2 Hosts,8 颗物理芯片 / 16 Core):物理拓扑为 2×2×2 Torus,在 XLA 与通信算法上展平为 2×4 逻辑网格,用于构建 8-hop 的单向哈密顿大环。• EP=8 真实映射口径:单机部署时按 8 个 Core 切分(每 Core 64 专家);双机部署时按 8 个 Chip 插槽分片。
Hardware Ground Truth: 1 Host = 4 Chips (8 Cores) · 8-Chip Slice = 2 Hosts• Microarchitecture & Die-to-Die: Each physical TPU v7 chip integrates 2 distinct compute dies exposed as 2 logical devices (cores), linked by an ultra-fast on-package Die-to-Die bus. • Single-Host vs. Dual-Host Topologies: - Single Physical Host ( tpu7x-standard-4t): Contains exactly 4 physical chips (8 cores) in a 2×2×1 3D Torus. Its 758 GiB HBM capacity can only act as an isolated worker for either PCP8 Prefill OR TP2·DP4 Decode; it physically cannot host a mixed P/D serving cluster on a single machine;- Dual-Host Slice (2 Hosts, 8 Chips / 16 Cores): Forms a physical 2×2×2 Torus, logically flattened into a 2×4 mesh in XLA to build an 8-hop Hamiltonian ring for Ring Attention.• EP=8 Alignment: Sharded across 8 cores in single-host mode (64 experts/core); sharded across 8 chip sockets in dual-host mode.
|
单裸晶/双芯 NVLink 全互联 (1.8 TB/s 双向) • 单卡即单个独立 CUDA 设备;在 NVL72 机柜内 72 卡任意点对点低延迟 All-to-All;但跨机柜扩展需跨 IB/RoCE 网络,延迟陡增。 Monolithic / Dual-Die NVLink Mesh (1.8 TB/s bi-directional) • 1 GPU maps directly to 1 CUDA device; 72 GPUs fully connected via NVLink 5 copper backplane; cross-rack scaling relies on higher-latency InfiniBand/RoCE. |
| MoE 专家调度与分级缓冲MoE Dispatch & Hierarchical Buffering |
4× 独立硬件协处理器 (SparseCore v3) + 片上 VMEM 双缓冲 • 每颗物理 Chip 搭载 4 个第 3 代 SparseCore,硬件级卸载 Top-K 寻址与 Gather-Reduce-v3,主 MXU 核心 0 调度气泡。 • 双缓冲 (Double Buffering) 物理机制:阿里攻坚的 Fused MoE v2 双 buffer (W1×2 + W2×1),底层正是利用超高速 Die-to-Die 链路在片内预先聚合,并在超导 VMEM 内部建立双缓冲蓄水池(Buffer A 算当前专家 / Buffer B 异步预取解压下个专家),完美隐藏跨 Chip 较慢的 ICI 环网流量与 692ns HBM 搬运延迟!
4× Dedicated SparseCore v3 + On-Chip VMEM Double Buffering• 4 third-gen SparseCores per chip offload Top-K routing and Gather-Reduce-v3, leaving MXU completely bubble-free. • Double Buffering Silicon Rationale: Alibaba's Fused MoE v2 dual-buffer (W1×2 + W2×1) leverages fast Die-to-Die links and establishes double buffering inside high-speed VMEM (Buffer A computes / Buffer B prefetches & dequantizes), hiding inter-chip ICI latency and the 692ns HBM penalty.
|
SM 核心通用线程调度 Gather/Scatter 在高并发长文本下容易出现非连续访存(Strided Memory Access)导致的 HBM 带宽打折,且 SM 兼职处理路由调度会导致计算核心气泡。 GPU SM Kernel Gathering/Scattering Prone to HBM bandwidth degradation under heavy concurrent token routing due to strided memory access; SM thread-divergence induces pipeline bubbles. |
| 片上超导缓存 (VMEM) 与 HBM 访存墙On-Chip VMEM vs. HBM Wall |
32 MiB Scoped VMEM · 3,207 GB/s 带宽 · 软件可控超导水槽 • 官方物理常数(OpenXLA PlatformInfo):单片 HBM $\leftrightarrow$ VMEM 实测带宽打满 3,207 GB/s,读取延迟 692 ns,写回延迟 594 ns。 • Pallas Tiling 优化:显式开放 --xla_tpu_scoped_vmem_limit_kib=32768,使 Pallas 内核将 128K Attention 与 MoE 的 Window Tile 切割开满,运算全程在 VMEM 内部超导回旋,将昂贵的片外 HBM 穿透率降低 30% 以上!
32 MiB Scoped VMEM · 3,207 GB/s Bandwidth · Software-Controlled Scratchpad• Silicon Ground Truth (OpenXLA PlatformInfo): Delivers 3,207 GB/s HBM-to-VMEM bandwidth with 692 ns read and 594 ns write latency. • Pallas Tiling Gains: Enabling --xla_tpu_scoped_vmem_limit_kib=32768 expands Pallas window tiles, keeping 128K attention and MoE working sets circulating within VMEM, slashing off-chip HBM roundtrips by >30%.
|
硬件自动管理 L2 Cache / Shared Memory (SRAM) • GB200 单卡配备超大 L2 Cache,但由硬件缓存控制器自动置换(LRU);多线程并发冲突时易发生 Bank Conflict 与缓存驱逐(Cache Thrashing),需极复杂的 TMA 异步拷贝指令微调。 Hardware-Managed L2 Cache / Shared Memory (SRAM) • Large L2 cache on B200 is managed implicitly via hardware LRU; heavy concurrent access risks bank conflicts and cache thrashing, requiring intricate TMA async copy tuning. |
| 显存规格与长文安全线Memory & Context Ceiling |
192 GiB HBM3e / 物理 Chip(96 GiB / Core) • 单机 4 芯片总计 768 GiB 高速 HBM3e(2 NUMA 节点池化); • 长文解耦:通过 PCP8 环形切片将 128K KV Cache(15 GiB 全量 FA)均匀切分为 1.87 GiB / 卡,结合 Host DRAM 动态流转彻底杜绝 OOM。 192 GiB HBM3e / Physical Chip (96 GiB / Core) • 4-chip single node totals 768 GiB HBM3e across 2 NUMA nodes; • Long Context Scaling: PCP8 slices 128K FA KV Cache (15 GiB) down to 1.87 GiB/chip; safe DRAM offloading completely eliminates OOM. |
192 GB HBM3e / GPU • 单卡容量物理相当,但在纯 DP 模式下高并发长文本仍会迅速触碰显存红线,需要大规整 PagedAttention 分块调度;单卡显存碎片率较高。 192 GB HBM3e / GPU • Physical capacity is identical, but pure DP serving under heavy concurrent long context still exhausts HBM; higher memory fragmentation risk. |
| P/D 越界投递与网络亲和性Disagg KV Handoff & NIC |
Raiden NUMA-Aware 双网卡 (2×200G 聚合 400G) · 100% 本地 DMA • 硬件亲和绑定:Rank 0..3 绑 NUMA 0 ( ens6),Rank 4..7 绑 NUMA 1 (enp192s4),接收端达成 100% NUMA-local 投递,彻底根除跨 CPU Socket UPI 内存总线搬迁;• 实测性能:ProcFS 实测打满 367.0 Gbps(91.8% 线速),长文 64K Reshard 耗时压减至 336.6 ms(提速 2.10×)。 Raiden NUMA-Aware Dual-NIC (2×200G Aggregate 400G) · 100% Local DMA • Hardware Affinity: Binds Ranks 0..3 to NUMA 0 ( ens6) and Ranks 4..7 to NUMA 1 (enp192s4), completely eliminating cross-socket UPI bus traffic;• Empirical Benchmark: Delivers 367.0 Gbps (91.8% wire speed), slashing 64K reshard latency to 336.6 ms (2.10× faster). |
机柜内 NVLink 极强 / 跨机柜依赖 RoCE/IB • 机柜内(NVL72)72 卡点对点 1.8 TB/s 具备无可比拟优势;但进行跨机柜或跨 Pod 的 P/D 分离投递时,依赖外部以太网/InfiniBand,若缺少 NUMA-网卡-GPU 的细粒度亲和调度,容易引发 host 内存通道与软中断不均。 Intra-Rack NVLink Dominance / Cross-Rack RoCE/IB Bottlenecks • NVL72 provides unmatched 1.8 TB/s intra-rack peer-to-peer; however, cross-rack/disaggregated serving relies on RoCE/IB fabrics where lack of NUMA-NIC-GPU affinity introduces host memory bottlenecks. |
PP4·PCP8 混合并行:理论推算 vs. 真实权重实测校准
⚡ 4-Host Inter-Node Bottleneck & PP4·PCP8 Hybrid Parallelism: Analytical Projection vs. Real-Weight Calibration
• 跨机通信挑战(09-18 实测):当集群扩展至 4-Host(16 物理芯片 / 32 Device)时,Prefill 吞吐从单机 PCP8 的 84k 跌落至 44k tok/s(预期 60k)。
根因在于节点超过 2-Host 后物理不再是全互联,跨机 Z 轴集合通信带宽不足严重拖累了 Fused MoE Token All-Gather(端到端约 −30%)。
• 架构破局推算 vs. 10 月真实权重实测校准:团队放棄跨机全局 All-Gather 优化,转向 PP4·PCP8(流水线 + 环形混合并行):
09-18 依据单机 PP8(80k tok/s,TTFT ~3s+)与单机 PCP8(84k tok/s)理论推算 4-Host 可达每 Rank ~20k、TTFT ~1.6 s;
10 月中旬在 PP4·PCP8 + MTP 真实权重(P8K·D1)压测中,实测吞吐为 55k–58k tok/s(其中 MTP 带来约 1–2% 可接受损耗),但吞吐存在波动——根因是 Scheduler 按 block_size 将 8K 请求切分为 3K + 5K,仅当单步恰好凑齐均衡组合(如 4×3K + 4×5K)时才能打满,若调出非对称组合(如 5×3K + 3×5K)即产生跨 Stage Padding 空泡(可借鉴上游 Kimi 的对称分桶调度修复)。
• Multi-Host Bottleneck (09-18 Measurement): When scaling to 4-Host (16 physical chips / 32 devices), prefill throughput dropped from 84k (single-host PCP8) to 44k tok/s (target 60k).
Root cause: beyond 2 hosts the physical torus is no longer all-to-all connected, and limited Z-axis bandwidth throttles Fused MoE Token All-Gather (~30% E2E penalty).
• Analytical Projection vs. October Real-Weight Calibration: Dropping cross-host All-Gather tuning in favor of PP4·PCP8 (Pipeline + Ring Hybrid):
on 09-18, extrapolating from single-host PP8 (80k tok/s, TTFT ~3s+) and single-host PCP8 (84k tok/s) analytically projected ~20k/rank and ~1.6 s TTFT across 4 hosts;
in mid-October real-weight benchmarks (PP4·PCP8 + MTP, P8K·D1), measured throughput reached 55k–58k tok/s (with an acceptable 1–2% MTP overhead), exhibiting variance caused by block-size-aligned scheduler chunking: an 8K prompt is split into 3K + 5K, so peak throughput requires balanced micro-batches (e.g., 4×3K + 4×5K), whereas skewed draws (e.g., 5×3K + 3×5K) incur padding bubbles (addressable via upstream Kimi-style symmetric bucket scheduling).
xla/benchmarks),现代大规模 MoE 推理的本质是一场严密的“高压张量流体力学”协同:NVFP4 + W4A8 Subchannel 瞬时反量化破除物理限制的创举;--xla_tpu_scoped_vmem_limit_kib=32768,结合 VMEM 双缓冲(Double Buffering),将 128K 矩阵与 MoE 专家权重死锁在片上微型水槽内,彻底隐藏片外 HBM 延迟!
Per SemiAnalysis 2026.09 Architecture Report (Computation and Data Movement for Inference) and OpenXLA Benchmarks (xla/benchmarks), frontier MoE inference is fundamentally an exact science of tensor fluid dynamics:NVFP4 + W4A8 subchannel dequantization strategy;--xla_tpu_scoped_vmem_limit_kib=32768 paired with VMEM double buffering locks 128K tiles and MoE working sets entirely on-chip, completely masking off-chip latency.
这场具有里程碑意义的工业级攻坚,本质是一场“Google 死磕单芯片与指令级微观公差,阿里主导全局集群网络与长序列宏观拓扑”的四手联弹协同: This milestone industrial deployment is a textbook full-stack co-design: Google engineered microarchitectural precision and Pallas kernels, while Alibaba pioneered multi-host topologies, long-context parallelisms, and cluster networking.
| 攻坚维度Domain | Google TPU 官方团队 (微观算子与编译器底座)Google TPU Team (Kernel & Silicon Stack) | 阿里 Qwen 团队 (宏观拓扑、长序列与量化落地)Alibaba Qwen Team (Architecture & Cluster Serving) |
|---|---|---|
| 注意力与长序列核心Attention & Long-Context |
GDN 线性注意力代数重排与算子级融合 • [PR #2498] 代数结合律解耦:将 GDN 输出展开为 $O=(\text{decayed } S)q + \Delta(k^\top q)$,让 MXU 提前乘旧状态,与 VPU 修正更新 100% 并发重叠,彻底消除 MXU 挂起气泡(吞吐提升 +4.48%); • [PR #3016] GDN v3 终极融合:将 Conv1D 与 Gated DeltaNet 融合成单 Kernel,减少 HBM 换入换出,算子提速 1.41× ~ 2.14×; • [PR #2741 / PR #2650]:切片 Q/K 杜绝向量寄存器溢出,状态 DMA 双缓冲使吞吐提升 11.3%。 GDN Algebraic Rearrangement & Kernel Fusions • [PR #2498] Associative Decoupling: Rewrites GDN as $O=(\text{decayed } S)q + \Delta(k^\top q)$, enabling MXU to compute on decayed state in parallel with VPU updates (slashes bubbles, +4.48% throughput); • [PR #3016] GDN v3 Fusion: Fuses Conv1D and GDN into a single kernel, slashing HBM roundtrips for 1.41× ~ 2.14× speedup; • [PR #2741 / #2650]: Eliminates vector register spills; async DMA state double-buffering yields +11.3% throughput. |
PCP8 环形流水线 (Ring Attention) 编排 • 解决二次方显存爆炸:设计并落地 `PCP8`(Prefill Context Parallelism = 8),在 8 个 Core 上构建单向哈密顿闭环,分块并行流动计算; • 极致掩盖:在计算当前 16K 块的同时预取下一块,计算与通信 100% 异步重叠,128K 超长文 TTFT 达到 4.62s,硬核反超 NVIDIA GB200(~5.0s)。 PCP8 Ring Attention Pipeline Orchestration • Taming $O(N^2)$ Complexity: Designed and deployed `PCP8` (Prefill Context Parallelism = 8) across an 8-core Hamiltonian ring with segmented streaming attention; • 100% Latency Masking: Overlaps chunk computation with next-chunk ICI transfer, clocking a 128K TTFT of 4.62s, outpacing NVIDIA GB200 (~5.0s). |
| MoE 稀疏路由与通信MoE Routing & Collectives |
通信合并与 SparseCore 协处理器卸载 • [PR #2174] 单次 All-Gather 合并:将 Expert IDs 与路由权重合并为一次通信,每层节省 80µs,全模型 58 层直接净省 4.64 ms! • [PR #2888 / #2137] 协处理器接管碎活:将 Reduce-Scatter 与 MoE Ragged Token 重排卸载到 SparseCore 专用协处理器,利用片内 Die-to-Die 预先求和,主 MXU 0 调度气泡。 Collective Merging & SparseCore Silicon Offloading • [PR #2174] Single All-Gather Fusion: Merged expert IDs and routing weights into one collective, saving 80µs/layer and 4.64 ms across 58 layers per forward pass; • [PR #2888 / #2137] SparseCore Offloading: Offloaded Reduce-Scatter and ragged token sorting to dedicated SparseCore co-processors, pre-reducing on fast Die-to-Die links. |
`owner_output` 原地沉淀与 Fused MoE 双缓冲 • 消减 50% 冗余通信:推行 owner_output,让 Token 在目标专家 Core 上计算完成后原地就地累加沉淀,彻底砍掉倒流回源 Core 的回程流量;• 片上 VMEM 双缓冲 (PR #891): Fused MoE v2 (W1×2 + W2×1) 在超导 VMEM 内部建立双池交替,隐藏跨 Chip ICI 流量,Decode 冲上 5,650 tok/s。
`owner_output` In-Place Accumulation & MoE Buffering• 50% Traffic Elimination: Implemented owner_output to accumulate expert outputs in-place, eliminating reverse token transmission back to source ranks;• On-Chip VMEM Double Buffering (PR #891): Fused MoE v2 (W1×2 + W2×1) alternates compute and prefetch inside VMEM, driving decode to 5,650 tok/s.
|
| 集群扩展与多机瓶颈Multi-Host & Cluster Fabric |
拓扑感知与下一代 Boardfly 架构 • 3D Torus 缠绕环优化:通过 Twisted Torus 莫比乌斯环拓扑将机架内最坏跳数从 N 压缩为 N/2; • TPU v8i 规划:针对超大规模 MoE 放弃 3D Torus,全面演进为 Boardfly 高基交换拓扑(19.2 Tb/s ICI),网络直径削减 50% 以上。 Topology Awareness & Future Boardfly Interconnect • 3D Torus Wrapping: Twisted Torus shaves worst-case hop count from N to N/2 within 64-chip racks; • TPU v8i Roadmap: Transitions from Torus to Boardfly high-radix fabric (19.2 Tb/s ICI), slashing network diameter by >50% for wide MoE scaling. |
`PCP8 PP4` 跨机破局与 Raiden 双网卡 400G • 4-Host 跨机破局:针对 4 机 Z 轴带宽瓶颈,设计 `PP4·PCP8` 混合切分,跨机仅传递单 Token 隐层向量(理论推算 ~20k/rank、TTFT ~1.6 s;真实权重 P8K·D1 + MTP 实测从 44k 提升至 55k–58k tok/s); • Raiden 100% 本地 DMA:首创绑定双 200G 网卡亲和调度(Rank 0..3 绑 NUMA 0 / Rank 4..7 绑 NUMA 1),实测打满 367.0 Gbps (91.8% 线速),根除跨 Socket UPI 拥塞。 `PCP8 PP4` Multi-Host Solution & Raiden Dual-NIC 400G • 4-Host Bottleneck Mitigation: Overcame Z-axis bandwidth saturation via `PP4·PCP8` (passing only micro-activations across hosts; analytical target ~20k/rank & ~1.6 s TTFT, real-weight P8K·D1 + MTP measured at 55k–58k tok/s vs. 44k baseline); • Raiden NUMA-Local DMA: Bound dual 200G NICs to local NUMA nodes (0..3 to NUMA 0 / 4..7 to NUMA 1), hitting 367.0 Gbps (91.8% wire speed) with zero UPI bus cross-traffic. |
| 量化与生态基础设施Quantization & Ecosystem |
Pallas Subchannel 算子与原生 TorchTPU • Subchannel Matmul:在 xla/benchmarks 中沉淀标准化 s4 微缩放算子内核;• 原生 TorchTPU 生态外化:联合 vLLM 推进原生后端,使 PyTorch 无缝挂接底层 Pallas 高性能内核,逐步摆脱中间转译层。 Pallas Subchannel Kernels & Native TorchTPU • Subchannel Matmul: Standardized s4 micro-scaled matrix multiply primitives in OpenXLA benchmarks;• TorchTPU Externalization: Collaborating with vLLM on a first-class backend allowing PyTorch to natively invoke underlying Pallas kernels without translation overhead. |
`NVFP4 + W4A8` 软件反量化破局 (PR #912) • 打破无原生 FP4 困境:针对 TPU v7 无原生 4-bit 计算硬件的限制,设计 HBM 存 4-bit 权重 + 片上 VPU 实时反量化为 8-bit 的软件管线,零精度损失下访存吞吐翻倍,直追 Blackwell。 `NVFP4 + W4A8` Software Dequantization (PR #912) • Bypassing Silicon FP4 Absence: Compensated for lack of native FP4 silicon by storing 4-bit weights in HBM and dequantizing on-chip into 8-bit, effectively doubling memory bandwidth while matching FP8 accuracy. |
针对 Qwen 3.5 397B MoE 稀疏路由中专家动态切片(Dynamic Slicing)开销大、访存未合并(Un-coalesced HBM Gathers)以及 TPU7x(Ironwood)新一代硬件物理特性的瓶颈,基于 Google DeepMind 官方开源优化标准与 torchtpu-agents 框架,开展物理集群实证调优:
Addressing the bottlenecks in Qwen 3.5 397B MoE routing—dynamic slicing penalties, un-coalesced HBM gathers, and physical nuances of TPU7x (Ironwood)—we conducted autonomous hardware-level optimization via the torchtpu-agents study protocol:
.T 会强制调用跨通道单元(Cross-Lane Unit, XLU)重排向量通道,引入 9.4% 的物理停顿。改用 jax.lax.dot_general 维度契约,由 MXU 脉动阵列在累加阶段直接吸收转置布局,彻底消除 XLU 物化惩罚。
Explicit transpositions in VMEM trigger Cross-Lane Unit (XLU) lane swizzling, imposing a 9.4% hardware stall. Replacing with transverse contraction via dot_general contracts directly across dimensions, letting MXU systolic arrays absorb transposed layouts at zero cycle cost.
pl.Buffered(buffer_count=2) 双缓冲管线,在前一个专家瓦片进行 MXU 脉动计算的同时,DMA 异步预取下一个专家权重切片,使高带宽访存搬运完全隐匿在计算延迟背后。
Configured pl.Buffered(buffer_count=2) to overlap asynchronous DMA HBM-to-VMEM transfers with active MXU systolic computation, achieving near-perfect compute-communication overlap.
| 优化维度 (Dimension) | 基线设计 (Baseline) | TPU7x 优化实现 (Optimized) | 实测收益与物理对账 (Verified Metrics) |
|---|---|---|---|
| 脉动阵列切块 (Tiling) | 128×128 (v4/v5p 遗留) | 256×256 (Ironwood 原生) | 消灭 50% Zero-Padding;单步延迟 0.821 ms,吞吐达 10.46 TFLOPs |
| 累加精度与类型铸造 | 隐式 VPU Downcast (35% 延迟膨胀) | 32-bit VMEM 原生累加器 | 消除 17%~35% VPU 降准开销;Logits 余弦相似度 1.000000 |
| 权重转置通道开销 | 片上 VMEM .T 物化 (XLU 竞争) |
横向收缩 (Transverse Contraction) | 消除 9.4% XLU Swizzle 硬件停顿;纯 MXU 原生契约乘加 |
| 数值对齐门禁 (Gate) | Max Diff < 0.001 | 实测 Max Diff = 0.000065 | PASS(Top-1 一致率 99.80%,Top-5 100.0%) |
在 Qwen 3.5 397B 异构推理系统(Prefill PCP8+EP8 → Decode TP2·DP4·EP8)中,集合通信算子并非随意调用,而是根据物理拓扑层级(片内 Die-to-Die、机箱内 ICI 铜线 Torus、跨机光纤)和模型并行维度(PCP / TP / DP / EP)进行了严格的数学正交编排。下表全景梳理系统内每一个集合通信算子的物理通路、触发阶段、数据负载与防阻塞设计: In the Qwen 3.5 397B serving architecture (Prefill PCP8+EP8 → Decode TP2·DP4·EP8), collective communication primitives are mapped with rigorous orthogonality to both physical topology tiers (Die-to-Die, in-chassis ICI copper torus, cross-host optical links) and parallelism axes (PCP / TP / DP / EP). The ledger below dissects every collective primitive across stages, payload shapes, and latency masking mechanisms:
| 并行维度 / 算子类型Parallel Axis / Primitive | 作用阶段Stage | 物理传输介质 & 硬件接口通道Physical Medium & Hardware Interface | XLA 原语 / 通信模式XLA Primitive / Pattern | 传输内容 & 张量形状Payload & Tensor Shape | 典型时延 & 掩盖机制Latency & Overlap Rationale |
|---|---|---|---|---|---|
PCP=8环形流水上下文并行 Ring Context Parallel |
Prefill |
硅基 Die-to-Die + 板级铜箔 (PCB Traces) • 单机 8 Core(4 芯片 / 1 Host): - 同芯片内:片上超高密度硅中介层微凸块(On-Package Die-to-Die,比单条 ICI 快 6×); - 跨芯片间:主板级 2×2×1 ICI 铜箔差分走线(每链路双向 200 GB/s)。
|
collective-permute8 跳单向 P2P 环网传递 |
KV Cache Chunk[16384, 4, 128] FP8每跳单向约 8 MB |
~950 μs / 跳 100% 异步双缓冲掩盖!在本地计算 16K QK GEMM(~4,120 μs)时后台 DMA 流动,气泡 = 0 μs。 |
TP=2片内张量并行 Intra-Chip Tensor Parallel |
Decode |
基板微凸块硅互联(Silicon Interposer / D2D Micro-bumps) • 绝对不出芯片封装(Zero-Off-Package): - 纯依靠封装内部 Core 0 ↔ Core 1 超短距高密度 Die-to-Die 金属微互联,带宽达到 ICI 单链路的 6 倍以上; - 不占用任何板级 ICI 铜线或外部网络带宽。 |
all-reduce (Sum)双核对称规约 |
Attention 投影分量[Batch, Hidden_Dim=4096]分担 Dense GEMV 输出 |
~85 μs / 次 不跨物理芯片,走硅片微米级互联,时延极低;将单核 Dense 显存砍半,使 Decode 能开满 Batch=64。 |
EP=8 (Dispatch)MoE 专家路由分发 MoE Token Dispatch |
Prefill & Decode |
板级直连铜线 + 机箱背板(Backplane Copper Traces) • 机内 3D 环面铜网(In-Chassis Torus): - 8 Core 专家域全展开;跨芯片部分完全通过主板/背板差分铜线以 3D Torus 拓扑点对点直连(总出口双向 1,200 GB/s); - 单机模式不出机箱,双机模式走机柜级短铜缆(DAC, Direct Attach Copper)。 |
all-to-all (Scatter)多对多非连续 Token 重分发 |
激活值 Token 向量[Routed_Tokens, 4096]Top-8 路由(每 Token 派发 8 个副本) |
~1.1 ms (Prefill) / ~420 μs (Decode) 由 SparseCore v3 硬件协处理器卸载 Top-K 索引与分拣;通过 Fused MoE v2 双缓冲重叠跨芯片搬运。 |
EP=8 (Combine)MoE 专家聚合回传 MoE Token Combine |
Prefill & Decode |
片上 D2D 总线 + 背板铜线(Hybrid Silicon/Copper Fabric) • 分层介质聚合: - 本地 Core 间通过超高速封装微凸块(D2D)做首级局部聚合( owner_output 分支);- 跨芯片部分走板级差分铜线回传归属 Host,降低出芯流量。 |
all-to-all (Gather-Reduce)专家前向加权累加后回传原籍 |
专家 FFN 输出向量[Tokens, 4096] FP8/BF16带 Softmax 门控权重的加权和 |
~1.3 ms (Prefill) / ~450 μs (Decode) 阿里 PR #912 采用 owner_output 分支优化,在片内 D2D 预先聚合局部专家输出,减少 ICI 出口拥塞。
|
DP=4推理数据并行 Inference Data Parallel |
Decode |
零物理介质(No Physical Transmission / Air Gap) • 物理介质完全解耦: - 4 个 DP 副本各占 1 颗独立物理芯片(2 Cores); - 前向推理无需梯度同步,Attention 阶段物理介质传输开销恒等于 0(完全不消耗铜线、微凸块或光纤)。 |
无需通信(Zero-Comm) 无反向梯度传播,请求完全解耦 |
N/A (无数据交换) 各组独立处理不同用户 Batch=64 |
0 μs(彻底免除 All-Reduce)! 规避了 DCP(解码上下文并行)高频跨芯片 Softmax 规约的毁灭性延迟,将网络净空留给 MoE All-to-All。 |
P/D Disagg TransferKV 缓存越界解耦投递 P/D KV Handoff |
P → D 跨节点 |
PCIe Gen5 总线 + 数据中心双网卡光纤(DCN 2×200G / 400G 聚合 RoCE/以太网) • 跨物理主机(Host-to-Host): - 端到端链路:Prefill TPU HBM → PCIe 5.0 主机通道(~119.2 GiB/s)→ Host DRAM 池化缓冲 → 双 200G 网卡(聚合 400G)数据中心光纤上行 → Decode Host DRAM → Decode TPU HBM。 |
Async P2P / RDMA Send-Recv自建 TariMempool / Raiden 同步通道 |
15 层 FA 全量 KV15 GiB / 128K 单请求45 层 GDN 线性状态(16 KB)近乎零开销 |
~180 Gbps 线速推流 利用 Host DRAM 异步池化中转;Decode 首字吐出前提前在后台触发投递,解耦两端调度压力。 |
PCP8 PP4 (跨机扩展)流水线 + 环形混合并行 Pipeline + Ring Hybrid |
4-Host (16 Chip / 32 Dev) |
光回路交换机光纤链路(OCS Optical Circuit Switched Fibers) • 机柜级跨 Host 拓扑: - Cube 内部走直连铜缆,超过单 Cube 后 Z 轴全靠 OCS 动态光路交换与多模光纤(Optical Fibers)传输; - PP4 将原本带宽受抑的跨机 Z 轴限制为仅传送 Stage 激活值。 |
P2P Pipeline Activation Handoff跨 Stage 阶段切片(PP4: 15层/Stage) |
Stage 边界激活值[Tokens, Hidden_Dim=4096]仅在 15/30/45 层交界处传输 |
破除 4-Host 跨机带宽塌陷 将原本受限的全局 Z 轴 All-Gather 限制在单机内,跨机仅传输轻量激活值;理论推算 ~20k/rank(TTFT ~1.6 s),真实权重 P8K·D1 实测达 55k–58k tok/s。 |
all-reduce)只走片内硅微凸块(~85 μs),绝不出芯片,零板级网络开销;collective-permute)与 MoE 专家路由(EP8 all-to-all)完全约束在主板与机箱差分铜缆内(双向 1,200 GB/s),配合双缓冲与 SparseCore 彻底隐藏时延;在 Qwen 3.5 397B 长文分离推理中,单个 128K 请求跨机投递的 15 层 FA KV Cache 达到 15 GiB。如果仅走单张 200 Gbps 网卡,线路在并发 C≥4 时就会迅速被打满在 192 Gbps 上限;更致命的是跨 CPU Socket 的 UPI 内存总线瓶颈——发往远端 Socket 芯片的数据必须二次翻越 UPI,造成严重的内存总线争抢与软中断抖动。Google 与阿里联合研发的 Raiden NUMA-Aware Dual-NIC 方案通过物理亲和性编排彻底破局:
tpu_utils.cc 扫描 /sys/class/net/* 排除控制面(MTU<1500)与虚拟网卡,自动识别物理数据 NIC;local_rank 自动关联物理芯片 PCIe BDF;Rank 0..3 绑定 NUMA 0 (ens6),Rank 4..7 绑定 NUMA 1 (enp192s4);TPU_RAIDEN_ENABLE_SOURCE_IP_BIND=0,让 Linux 内核路由表根据目标 IP 自动匹配本地出口;sar -n DEV 监控)!
official/release-0.0.1 发布分支;10 月起工作重心转向生产长稳、精度对齐与分支收敛。
This table tracks the core PRs gating production launch. By late September the key mesh, KV-transfer, and MoE kernel PRs landed on main and cut into official/release-0.0.1; October focus shifted to fleet stability, numerical determinism, and branch convergence.
| 工作项Item | 优先级Pri | PR / 模块PR / Module | 状态(截至 2026-10-16)Status (as of 2026-10-16) | 阻塞上线?Blocks launch? |
|---|---|---|---|---|
| PCP SEQ_ON_LANE 与在线 FP8 / GDN 量化合入PCP SEQ_ON_LANE & online FP8 / GDN quantization | P0 | #608 / #750 / #913 |
✅ 已合入merged | 是yes |
| FusedMoE v2 NVFP4 + W4A8 与自适应 2+1 / 3+2 双缓冲FusedMoE v2 NVFP4 + W4A8 & adaptive 2+1 / 3+2 buffer | P0 | #912 · #1108 · #1133 |
✅ 均已合入(含 Decode MoE 性能优化)all merged (incl. Decode MoE perf) | 是yes |
| marconi-prefix-cache 使能(混合模型前缀缓存)marconi-prefix-cache enablement (hybrid-model prefix cache) | P1 | #765 |
✅ 完成done | 否no |
| MTP 性能优化与 PCP+MTP / KVCM 冲突修复MTP perf & PCP+MTP / KVCM conflict fix | P0 | #1008 · #1106 |
✅ 已合入merged | 是yes |
| torchtpu OOM 回归修复与 mesh / runtime partition 修复torchtpu OOM regression & mesh / runtime partition fix | P0 | #952 · #1182 (PCP+TP) |
✅ 端到端验证通过,已于 09-25 合入E2E verified & merged by 09-25 | 是yes |
| 跨拓扑 KV Transfer(PCP8 ➔ DP4·TP2 及 PCP+TP ➔ DCP+TP)Cross-topology KV Transfer (PCP8 ➔ DP4·TP2 & PCP+TP ➔ DCP+TP) | P0 | #1183 · #1121 (PCP+HND DMA) |
✅ 已合入(单机有效 TP 2→1/2/4 均验证通过,推进多机)merged (effective TP 2→1/2/4 verified single-host; scaling multi-host) |
是(上线部署形态)yes (deployed topology) |
| Raiden 双网卡 400G 与 KV Cache HBM ↔ DRAM 全量映射修复Raiden dual-NIC 400G & KV-cache HBM ↔ DRAM mapping fix | P0 | #839 / #923 · #1031 |
✅ 双网卡实测达 367 Gbps;全量映射修复已提交dual-NIC verified at 367 Gbps; full-mapping fix landed | 是yes |
| 10 月生产分支收敛与关键正确性补丁回移October release-branch convergence & critical correctness backports | P0 | #1745 / #2024 · #1941 · #1983 · #2022 |
🔄 Logprob FP32 (#1745)、跨设备时钟 (#1941)、DP+MTP 步序对齐 (#1983) 已合入;连接池化与超时优化审阅合入 official/release-0.0.1 中Logprob FP32 (#1745), cross-device clock (#1941), and DP+MTP step alignment (#1983) landed; connection pooling & timeout PRs under review for official/release-0.0.1 |
是(以合入 official 分支为准)yes (gated on landing in official branch) |
tpu-sync 线程泄漏与多项稳定性缺陷(#1112 / #1115 / #1116),但在批量节点试投产与真实长稳压测中,又进一步收敛出硬件级 CoreHalt、MTP 接收率偶发跌零及边缘竞态等深水区问题:
Late-September PRs resolved the tpu-sync thread leak and core stability issues (#1112 / #1115 / #1116), while subsequent fleet burn-in surfaced deeper hardware CoreHalt events, intermittent MTP acceptance drops to 0%, and edge-case race conditions:
| 场景Scenario | 真实状态(截至 10-30)Actual status (as of 10-30) | 工程依据与处理进展Engineering evidence & progress |
|---|---|---|
| Decode 阶段 client abort 的 KV / 请求泄漏KV / request leak on client abort during decode | ✅ 已修复fixed | #1039 (Raiden Stage-3 抢占路径)(Raiden Stage-3 preemption path) |
长稳压测 tpu-sync 线程泄漏与基础稳定性修复Prolonged stress tpu-sync thread leak & stability fixes |
✅ 已修复合入fixed & merged | tpu-sync frame_rpc / RawBufferTransport 线程泄漏已修复,伴随 #1112、#1115、#1116 合入tpu-sync frame_rpc / RawBufferTransport thread leaks fixed; #1112, #1115, #1116 merged |
| Raiden controller / vLLM worker 崩溃恢复与就绪超时Raiden controller / vLLM worker crash recovery & readiness timeout | ✅ 已通过(多 Rank 启动超时阈值通过 #2022 调优)passed (multi-rank readiness timeout tuned in #2022) |
稳定性测试 4.1 / 4.2;针对多 Rank 启动快慢不均导致的超时做参数加固stability tests 4.1 / 4.2; readiness timeout adjusted for multi-rank startup skew |
| MTP 与 KVCM 交互竞态 / 异常防护(Block Table 竞争、NaN 传染、Decode 重算)MTP & KVCM race / fault protection (block-table race, NaN poisoning, decode recompute) | 🔄 补丁已在生产分支验证,向官方发布分支合入中patches validated on production branch; upstreaming to official release branch | ① 修复 MTP 异步上传 Block Table 时 CPU 源缓冲区被覆盖的竞态(a503c18);② Hidden states 出现 NaN 时通过 wait_for_save(has_nan) 阻断污染缓存槽位;③ PD 分离下彻底禁用 Decode 重算(1) Fix CPU buffer overwrite race during async MTP block-table H2D upload (a503c18); (2) block KVCM caching on NaN via wait_for_save(has_nan); (3) strictly disable decode recomputation in PD disagg |
| 在线接流过程中 MTP 接收率偶发跌 0%Intermittent MTP acceptance rate drop to 0% under live traffic | 🔄 传染路径已在 Kernel 层堵住,现象已很少出现;NaN 源头仍在排查(疑似 Prefill 侧)propagation blocked at kernel level and now rare; NaN origin still under investigation (suspected Prefill side) | 根因是 0.0 × NaN = NaN:QK 阶段的 mask 只把 softmax 权重置 0,挡不住 PV 对整个 tile 的累加。三道防线均已合入:① GMM Kernel 修复(#1661);② batched_rpa Decode(fuse_accum)模式下 l_prev/o_prev 跨序列边界显式重置(#1877,Prefill 性能 0% 变化);③ mask_invalid_v 在 PV 前清零无效 V 条目(#2454,SEQ_ALONG_LANE 默认开启,开销 +0.30% geomean)。NVFP4 397B 1P1D(PCP8 ➔ DP4·TP2)54h / 56h 两轮长稳测试 tpu_model_nan_rounds_total 均为 0;下一步在 Prefill / Decode / KV Transfer 三段加日志定位 NaN 源头。更正:此前「FA D2H 竞态」的推测未获证实。Root cause is 0.0 × NaN = NaN: masking QK only zeroes softmax weights and cannot stop PV from reducing across the whole tile. Three defenses merged: (1) GMM kernel fix (#1661); (2) explicit l_prev/o_prev reset at sequence boundaries in batched_rpa decode (fuse_accum) mode (#1877, 0% prefill impact); (3) mask_invalid_v zeroes invalid V entries before PV (#2454, on by default for SEQ_ALONG_LANE, +0.30% geomean). Two NVFP4 397B 1P1D (PCP8 ➔ DP4·TP2) soak runs of 54h / 56h recorded zero tpu_model_nan_rounds_total; next step is stage-level logging across Prefill / Decode / KV transfer. Correction: the earlier "FA D2H race" hypothesis was not confirmed. |
大规模集群低频硬件 E200 CoreHaltLow-frequency fleet-level hardware E200 CoreHalt |
🛠️ 已定位并修复两类触发路径,持续完善 Halt 堆栈诊断two root-cause paths identified & patched; enhancing crash stack diagnostics | ① SparseCore ragged_gather_reduce 边界条件修复;② DP>1 + EP + MTP 跨 Rank 步序错位死锁修复(#1983);③ Decode Pallas Kernel 越界访问 [96, 128) GiB 未映射区理论复现(详见下方专题剖析)(1) SparseCore ragged_gather_reduce boundary fix; (2) DP>1 + EP + MTP cross-rank step-order deadlock fixed in #1983; (3) Pallas DMA out-of-bounds into unmapped [96, 128) GiB reproduced in unit test (see deep dive below) |
PCP16·TP2 ➔ TP32·DCP4/8 与 PP4·PCP8Multi-Host Topologies: PCP16·TP2 ➔ TP32·DCP4/8 & PP4·PCP8PCP16·TP2 与 TP32·DCP4/8 跨拓扑 KV Transfer(#1183)及 PCP+TP 正交 Mesh(#1182)均已合入,真实权重加载与端到端问答已验证通过。Fused MoE v2 自适应 3+2 / 2+1 双缓冲(#1108)、Fused MoE + TP2、DCP + HND、GDN Q/K 头复制(#1082)及 1M 超长上下文 RPA 修复(3f57750 / b5a29fe)均已就绪。PP4·PCP8 + MTP 跨机流水线外,新一代架构实验已跑通 DP + TP + EP 混合模式,主干用例同步从 PCP4→DP4 / PCP8→DP8 升级对齐至生产标准 PCP4→TP2·DP2 / PCP8→TP2·DP4。
• Disaggregated Bring-Up Verified: Cross-topology KV transfer (#1183) and orthogonal PCP+TP mesh support (#1182) have landed; independent P/D startup and real-weight E2E inference are verified.3+2 / 2+1 buffering in Fused MoE v2 (#1108), Fused MoE + TP2, DCP + HND, GDN Q/K head replication (#1082), and 1M-context RPA fixes (3f57750 / b5a29fe) are all in place.PP4·PCP8 + MTP, experimental next-gen variants have validated mixed DP + TP + EP execution, and CI test cases upgraded from PCP8→DP8 to production-aligned PCP8→TP2·DP4.
PP4·PCP8 + MTP 真实权重 P8K·D1 吞吐达 55k–58k tok/s(由 44k 提升,逼近 60k 预期)
4-Host PP4·PCP8 + MTP real-weight P8K·D1 reaches 55k–58k tok/s (up from 44k, approaching 60k target)
PP4·PCP8 跨机流水线。#1108 与 #1182 已合入主干。DCP 长上下文 batched_rpa 的 Scoped VMEM 精确估算仍在收尾(重构前置 #1105 已合入,移植 #1081 跟进中);② PP4·PCP8 下 8K 请求按 block_size 切分为 3K+5K 引发的非对称微批次 Padding 抖动待合入对称调度修复;③ 单机 Host DRAM(已由 440 GiB 降至 230 GiB)继续下压以扩大 KV Store 缓存池。
• [Resolved] Cross-Host Token All-Gather Degradation: The pure 4-host All-Gather path (~30% E2E loss on non-fully-connected Z links) has been formally retired in favor of PP4·PCP8 pipeline hybrid parallelism.#1108 and #1182.DCP long-context batched_rpa (refactor #1105 merged, port from #1081 in progress); (2) symmetric micro-batch scheduling for PP4·PCP8 to eliminate 3K+5K chunk-padding variance; (3) further Host DRAM reduction beyond 440 ➔ 230 GiB to expand local KV store capacity.
当大规模 MoE + Hybrid Attention 模型从基准测试迈向百台级 TPU v7x (Ironwood) 生产集群长稳运行时,工程重心从单一峰值吞吐转向低频硬件异常(CoreHalt)根因归因、跨 Rank 位级数值确定性(Bit-Exact Determinism)、Host DRAM 内存池解剖、控制面长尾延迟治理以及生产分支 CI/CD 交付规范。以下为脱敏后的五项核心系统工程沉淀: As ultra-large MoE + Hybrid Attention serving transitions from benchmarking to multi-hundred-node TPU v7x (Ironwood) production fleets, engineering focus shifts from peak throughput to low-frequency hardware fault forensics (CoreHalt), cross-rank bit-exact numerical determinism, Host DRAM footprint anatomy, control-plane tail-latency elimination, and release-branch CI/CD governance. Below are five sanitized systems-engineering deep dives:
E200 CoreHalt 假阳性日志表象与三类物理根因1. TPU v7x E200 CoreHalt Misleading Signatures & Three Root CausesSC3 @ 0x3 与首算子 tt_jit_layernorm_*?
Why do fatal halts repeatedly point to idle SC3 @ 0x3 and tt_jit_layernorm_*?
cmn_local_fatal_errors_transform.set raw=0x1 时,底层驱动往往回退打印当前处于闲置状态的 SparseCore 3 @ 0x3 与计算图首个子图 tt_jit_layernorm_*,缺乏直接调用栈,极易误导排查方向。ragged_gather_reduce 在极端 Ragged 分桶下的边界条件越界,已通过算子边界补丁修复。gdn/v3、batched_rpa)显式开启 disable_bounds_checks=True 时,一旦读取到越界 Block / Page ID 且计算出的物理地址落入 TPU v7x 单 Die [96, 128) GiB HBM 未映射地址空洞(单 Die 物理 HBM 为 96 GiB),硬件立即触发 Fatal Halt 并回退打出上述假阳性签名(已通过单元测试精准复现同一错误日志)。DP>1 + EP + MTP 复合并行下,当单一 DP Rank 请求数跨过阈值被拆分为 2 个 Chunks 时,真实请求执行的 propose()(Step-Major 步序)与其余空闲 Rank 执行的 run_dp_dummy_draft()(Chunk-Major 步序)发生步序与 Token Shape 错位,导致跨 DP 的 MoE reduce-scatter 集合通信死锁;300s sample_tokens 超时触发 SIGTERM 杀进程后,同颗物理芯片上的对端 Core 因 Peer VF 先行关闭而连带报出 SC3 @ 0x3 Halt。修复方案(#1983):移除 SMEM request chunk 拆分上限(62f98fb / 5575f78)并强制对齐 Dummy Draft 与真实 Chunk 的步序(ebe95fd / d5d5114)。
• Misleading Log Signature: When hardware raises cmn_local_fatal_errors_transform.set raw=0x1, the runtime fallback dumps the idle SparseCore 3 @ 0x3 and the very first subgraph tt_jit_layernorm_* without a Python/HLO stack trace.ragged_gather_reduce, resolved via kernel guard fixes.gdn/v3, batched_rpa) run with disable_bounds_checks=True, an corrupted block/page ID whose physical address lands in the unmapped [96, 128) GiB address hole on a 96 GiB TPU v7x die immediately triggers the fatal error and prints the exact SC3 @ 0x3 signature (reproduced deterministically in unit tests).DP>1 + EP + MTP, when a single DP rank splits its batch into 2 chunks, active propose() (Step-Major) and peer ranks' run_dp_dummy_draft() (Chunk-Major) desynchronize on step order and token shapes, deadlocking cross-DP MoE reduce-scatter. After the 300s sample_tokens timeout sends SIGTERM, the surviving die on the same physical chip halts when its peer VF tears down first. Fix (#1983): remove the SMEM request-chunk split cap (62f98fb / 5575f78) and align dummy-draft step ordering with active chunks (ebe95fd / d5d5114).
reduce_scatter 非结合律偏差与零开销 Bit-Exact 修复2. Cross-Rank Numerical Determinism: reduce_scatter Non-Associativity & Zero-Cost Bit-Exact Fix2×2×1)MoE EP Combine 阶段,XLA 默认选用单跳环形设备序 [0,1,4,5,6,7,2,3] 执行平坦 8-Core 双向环 reduce_scatter。此时第 k 个 Shard 的累加顺序按 k 旋转:
shard 0 = ((((((v1+v2)+v3)+v4)+v5)+v6)+v7)+v0,而 shard 1 = ((((((v2+v3)+v4)+v5)+v6)+v7)+v0)+v1。由于 BF16 浮点加法满足交换律但不满足结合律,每步舍入差异使相同输入在不同 Rank 上产生约 1 ULP(Cross-Rank Diff ~1.56e-2,分裂为多个哈希簇),在长序列与采样下引发隐蔽精度漂移。TPU_VISIBLE_DEVICES 设为默认映射的逆置换 _INV = [0, 1, 6, 7, 2, 3, 4, 5],向 XLA 暴露自然设备序 0,1,2,...,7。XLA 编译器随即自动切换为分层成对规约策略(Hierarchical Pairwise,4 × core_count=2),令所有 Shard 共享完全一致的二叉树括号分组,实测 Cross-Rank Max Diff 归零(0.0000e+00),8 个 Rank SHA-256 哈希 100% 收敛为单一簇(BIT-EXACT)。temperature)失效与对数概率失真,全面对齐上游 vLLM 0.29.0 标准升级为 FP32 计算(0875ee7、#1745 已合入,#2024 跟进)。
• Root Cause (Ring Rotation Accumulation Order): On an 8-device single host (2×2×1), XLA's default single-hop ring order [0,1,4,5,6,7,2,3] schedules reduce_scatter as a flat 8-core bidirectional ring where shard k accumulates operands rotated by k:
shard 0 = ((((((v1+v2)+v3)+v4)+v5)+v6)+v7)+v0 vs. shard 1 = ((((((v2+v3)+v4)+v5)+v6)+v7)+v0)+v1. Because BF16 addition is commutative but not associative, step-by-step rounding causes identical addends to diverge by ~1 ULP across ranks.TPU_VISIBLE_DEVICES to the inverse permutation _INV = [0, 1, 6, 7, 2, 3, 4, 5] presents XLA with natural device order 0..7. XLA then selects its order-invariant hierarchical pairwise reduction (4 × core_count=2), giving every shard identical binary-tree grouping and achieving 0.0000e+00 cross-rank diff (SHA-256 BIT-EXACT across all 8 ranks) at native BF16 speed.0875ee7, #1745 merged, #2024), preventing temperature scaling distortion and aligning with vLLM 0.29.0.
0. 权重加载后基线 3.31 GB ➔ 1. KV Cache 编译初始化 +2.11 GB (5.47 GB) ➔ 2. Raiden KV Staging 预分配 +6.53 GB (12.16 GB) ➔ 3. 四个编译档位 [32, 64, 128, 256] 依次增加 +192 MB / +372 MB / +923 MB / +698 MB(共 +2.18 GB) ➔ 4. 编译后 Trim 释放 −650 MB,最终稳态 13.85 GB / Worker。[32,64] vs. 4 档 [32,64,128,256]):单 Worker 稳态 VmRSS 由 11.98 GB 增至 13.85 GB(净增 +1.87 GB / Worker);在单机 8-Worker(DP8)下放大 8 倍即推导增加 14.96 GB,与容器级监控实测 Anon 内存增量(109.43 GB − 95.47 GB = 13.96 GB)吻合度达 93.3%。go tool pprof Heap 差分归因:对编译前后进程堆空间(净增 3,503 MB)做 Inuse-Space 剖析,libtpu.so 独占 +2,529.8 MB (72.21%)(对应 PjRt XLA LoadedExecutable 常驻编译产物),libpython3.12.so 占 +808.5 MB (23.08%)(Python 字节码与构图对象),二者合计占净增堆内存的 95.29%。leases=1 估算偏差后,超大 MoE 单机容器 DRAM 已从 440 GiB 压降至 230 GiB;② 推动跨 Worker 共享 PjRtLoadedExecutable(消除每 Worker ~2.53 GB 私有重复加载);③ 将 Raiden Staging Buffer(占基线增量 ~50.3%)改为进程间共享内存池(Shared Memory Pool),可再省下约 40 GB 主机内存供 KV Store 提升前缀命中率。
• Single-Worker VmRSS Progression (Decode DP8 Baseline):
0. Post-weights 3.31 GB ➔ 1. KV-cache init +2.11 GB (5.47 GB) ➔ 2. Raiden KV staging allocation +6.53 GB (12.16 GB) ➔ 3. Four compile sizes [32, 64, 128, 256] add +192 / +372 / +923 / +698 MB (+2.18 GB total) ➔ 4. Post-compile trim −650 MB, settling at 13.85 GB / worker.11.98 GB to 13.85 GB (+1.87 GB / worker); across 8 DP workers this predicts +14.96 GB, matching pod-level Anon delta (109.43 − 95.47 = 13.96 GB) within 93.3%.pprof): Out of 3,503 MB net heap growth, libtpu.so accounts for +2,529.8 MB (72.21%) (resident PjRt XLA LoadedExecutable binaries) and libpython3.12.so accounts for +808.5 MB (23.08%) — together 95.29% of heap growth.leases=1 estimation already cut ultra-large MoE single-host container DRAM from 440 GiB to 230 GiB; (2) sharing PjRtLoadedExecutable across workers avoids duplicating ~2.53 GB per worker; (3) pooling Raiden staging buffers in shared memory saves up to ~40 GB of Host DRAM for KV caching.
GrpcControlPlaneBackend vs. 裸 TCP 硬件实测4. Disaggregated Control Plane: GrpcControlPlaneBackend vs. Raw POSIX TCPKVCacheManagerWithTransfer 中,每次 D2D Read 需先后执行 SendPullRequest(启动拉流)与 SendAck(释放源端 Staging 缓冲)两次控制面 RPC。旧版 TcpControlPlaneBackend 每次 RPC 均新建并销毁临时 Socket(单次传输共 2 sockets / 3.0 RTT),在高并发下与 D2H/H2D 线程争抢内核 accept()/poll() 队列,引发显著尾延迟抖动与端口耗尽风险。GrpcControlPlaneBackend(TPU_RAIDEN_CONTROL_PLANE_BACKEND=grpc)按远端 Endpoint 缓存持久化 HTTP/2 Stub(GetOrCreateStub,热通道 0 RTT 建连开销),内置原生 RPC Deadline 传播与 10s HTTP/2 PING 保活,并与数据面并行 Socket(BlockTransport::SyncPush,P=8 下维持 24.3–24.9 GB/s)完全解耦无干扰:2.94 ms vs. TCP 2.89 ms),但 p99 尾延迟大幅削减 35.8%(从 4.93 ms 降至 3.16 ms),平均有效吞吐提升 +7.9%(0.72 vs. 0.66 GB/s);< 0.8%(~0.3 ms / 128 ms),两端到端吞吐完全一致(512 MB P=8 下 gRPC 4.22 GB/s / 127.21 ms vs. TCP 4.19 GB/s / 128.07 ms);pooled 与 per_transfer 两种传输连接模式。
• Legacy TCP Bottleneck: Every D2D Read in KVCacheManagerWithTransfer issues two control RPCs (SendPullRequest + SendAck). TcpControlPlaneBackend opened and closed two ephemeral sockets per read (3.0 RTT setup), causing kernel accept()/poll() wakeup jitter and TIME_WAIT port churn when contending with D2H/H2D worker threads.GrpcControlPlaneBackend): Enabled via TPU_RAIDEN_CONTROL_PLANE_BACKEND=grpc, caching persistent HTTP/2 stubs per endpoint (0 RTT on warm channels) with native deadline propagation and 10s HTTP/2 PING keepalives, while leaving raw data-plane sockets (BlockTransport::SyncPush at 24.3–24.9 GB/s, P=8) completely untouched.2.94 vs. 2.89 ms) while slashing p99 tail latency by 35.8% (3.16 ms vs. 4.93 ms) and lifting mean throughput by +7.9%;< 0.8% of transfer time; both achieve ~4.2 GB/s end-to-end D2D throughput at 512 MB (127.21 ms gRPC vs. 128.07 ms TCP);pooled and per_transfer modes to cap exposed worker socket counts.
VLLM_XLA_CACHE_PATH=gs://...):采用「单 Producer Job 预编译写入 GCS ➔ 多 Host LeaderWorkerSet 消费只读缓存」的解耦范式(配合 Workload Identity 绑定 roles/storage.objectUser),避免多 Pod 并发写冲突并彻底绕过冷启动重复编译。all_gather 顺序的局限,引入 OpenXLA latency_hiding_scheduler 元数据注解 xla_metadata(scheduler_hint="force_delay_async", scheduler_delay_priority="1"),显式引导编译器推迟非关键异步通信、优先保障计算核心流水。official/release-0.0.1 vs. partner/release-0.0.1):为隔绝主干(origin/main)持续迭代引入的回归风险,在 vllm-torchtpu 与 tpu-sync 双仓建立带自动化 CI/CD 门禁的 official/release-0.0.1 官方发布分支(交付准入红线:PCP4→TP2·DP2 E2E、PCP8→TP2·DP4 E2E 及超大模型 1P1D 压测全通)。生产侧分支可在尾部先行挂载紧急热修,所有问题修复以正式合入 official/release-0.0.1 并通过 Nightly CI 为准,生产分支定期 Rebase 官方分支并剔除重复补丁,最终收敛为单一稳定基线;后续版本演进将对齐 tpu-inference 定期切分新 Release 分支的迭代模式。
• Startup Acceleration (Target ≤15 min): (1) MTP Weight Loader Scope Pruning eliminates redundant weight scans on ultra-large MoE models, saving ~10 minutes of cold-start I/O; (2) Shared XLA Compilation Cache on GCS (VLLM_XLA_CACHE_PATH=gs://...) decouples cache generation (a single Producer Job compiles and populates GCS) from consumption (multi-host LeaderWorkerSet replicas download pre-compiled artifacts and bypass XLA compilation).xla_metadata(scheduler_hint="force_delay_async", scheduler_delay_priority="1") to declaratively control asynchronous all_gather scheduling order in XLA's latency-hiding scheduler.official/release-0.0.1 & partner/release-0.0.1): To insulate production deployments from origin/main churn, both vllm-torchtpu and tpu-sync maintain CI/CD-gated official/release-0.0.1 branches (gated on PCP4→TP2·DP2 E2E, PCP8→TP2·DP4 E2E, and 1P1D stress tests). Production hotfixes may stage at the tip of the partner branch first, but formal delivery is defined strictly by merging into official/release-0.0.1 with passing CI, after which the partner branch rebases and drops duplicate patches to converge onto a single release baseline.
397B.md)、双层孪生微基准(2-Layer Twin)与 7 层编译器栈 × 6 大史诗级 PR 全景映射CoreML Exact Hybrid Mamba/GDN Page Alignment Math (397B.md), 2-Layer Twin Microbenchmark & 7-Layer Stack Mapping of the 6 Epic PRs
block_size: 256 ➔ 1,152 tokens?)与单机 2-Layer Twin 微基准
A. CoreML Exact Hybrid Mamba/GDN Page Alignment & Memory Math (Why block_size: 256 ➔ 1,152 tokens?) & Single-Host 2-Layer Twin Microbenchmark
block_size 从 256 跃升至 1,152 tokens(tpu_block_size_utils.py:_derive_tpu_block_slot_config,mamba-cache-mode=align / TPU_VLLM_ENABLE_UNIFIED_BLOCK_POOL=1)?mamba_raw_state_bytes = 542,720 B):在 TP=8, spec=3 下,每层每 Rank 包含 conv_state = (3 + 3) × 1,536 × 2 B (BF16) = 18,432 B(在 TP=2, spec=0 下为 3 × 6,144 × 2 B = 36,864 B,spec=1 为 4 × 6,144 × 2 B = 49,152 B)与 recurrent_state = 8 heads × 128 × 128 × 4 B (FP32) = 524,288 B(PR #1458 转为 BF16 后在 TP=2 即 16 heads/rank 下为 16 × 128 × 128 × 2 B = 524,288 B,在 TP=8 即 8 heads/rank 下为 262,144 B),基准对齐槽大小固定为 542,720 B/layer/slot。512 B/token/layer):当 TP ≥ 2 时,H_kv = 2 被切分为每 Rank attn_kv_heads_per_rank = 1 个 KV Head(配合 PR #608 SEQ_ON_LANE 消除半槽 Padding),每 Token 每层恰好占用 1 KV head × 2 (K,V) × 256 head_dim × 1 B (FP8) = 512 B/token/layer。3×GDN + 1×Attn 混合 Block 上实现零碎片统一物理页池),最小 Token 数为 ⌈542,720 / 512⌉ = 1,060 tokens;按 TPU Page 粒度向上对齐到 256 的整数倍即得 block_size = 1,152 tokens/page(1,152 × 512 B = 589,824 B ≥ 542,720 B)!2-Layer Twin Microbenchmark(1 linear_attention + 1 full_attention + MTP)如何在单台 2x2x1(4 Chips = 8 Devices)上 1:1 复刻 397B 单 Rank 行为?hf_overrides: {"text_config": {"num_hidden_layers": 2, "layer_types": ["linear_attention", "full_attention"]}} 将 60 层(45 GDN + 15 Full Attn)缩减为 1 层 GDN + 1 层 Full Attention + 1 层 MTP,同时严格锁定单 Rank 张量维度与生产 DP4×TP2 + EP8 完全一致:q_heads_projected = 16、attn_kv_heads_per_rank = 1、gdn_k_heads_per_rank = 8、gdn_v_heads_per_rank = 32、experts_per_rank = 64(512 / EP8)、cache_block_tokens = 1,152、pages_per_rank = 4,007、kv_pool_tokens_per_rank = 4,617,186,并通过 HBM Ballast 垫片将每 Device 显存占用锁定在 76.49 GiB / 96 GiB,单步迭代仅需数毫秒即可在单台 2x2x1 上完成算子与编译标志的精确 A/B 验证。
• Why does block_size scale from 256 ➔ 1,152 tokens under Unified Hybrid Pooling (tpu_block_size_utils.py:_derive_tpu_block_slot_config, mamba-cache-mode=align / TPU_VLLM_ENABLE_UNIFIED_BLOCK_POOL=1)?mamba_raw_state_bytes = 542,720 B): At TP=8, spec=3, each layer per rank holds conv_state = (3 + 3) × 1,536 × 2 B (BF16) = 18,432 B (or 49,152 B at TP=2, spec=1) plus recurrent_state = 524,288 B (FP32 at TP=8, or BF16 at TP=2 with 16 V-heads/rank via PR #1458; 262,144 B BF16 at TP=8), yielding a canonical alignment slot of 542,720 B/layer/slot.512 B/token/layer): For TP ≥ 2, each rank owns attn_kv_heads_per_rank = 1 KV head (with PR #608 SEQ_ON_LANE eliminating half-lane padding), consuming 1 KV head × 2 (K,V) × 256 head_dim × 1 B (FP8) = 512 B/token/layer.3×GDN + 1×Attn) blocks, the minimum page size is ⌈542,720 / 512⌉ = 1,060 tokens; rounding up to the next multiple of 256 yields block_size = 1,152 tokens/page (589,824 B ≥ 542,720 B)!2-Layer Twin Microbenchmark (1 linear_attention + 1 full_attention + MTP) reproduces per-rank 397B behavior on a single 2x2x1 host (4 chips = 8 devices):hf_overrides: {"text_config": {"num_hidden_layers": 2, "layer_types": ["linear_attention", "full_attention"]}}, the twin instantiates exactly 1 GDN layer + 1 Full Attention layer + MTP with identical per-rank shapes to production DP4×TP2 + EP8: q_heads_projected=16, attn_kv_heads_per_rank=1, gdn_k_heads_per_rank=8, gdn_v_heads_per_rank=32, experts_per_rank=64, cache_block_tokens=1,152, pages_per_rank=4,007, kv_pool_tokens_per_rank=4,617,186, and HBM ballast pinned to 76.49 GiB/device.
vllm-torchtpu PR 架构对位(为什么 4 大突破是 0% Pallas,而 2 大突破必须手写 Pallas?)
B. The 7-Layer Stack Mapping of the 6 Epic vllm-torchtpu PR Breakthroughs (Why 4 Require 0% Pallas While 2 Are Surgical Pallas Kernels)
| 编译器/运行时栈层级Stack Layer | Pallas 占比与属性Pallas Share & Mode | 核心 PR 与模块定位Key PRs & Modules | 物理机制与实测端到端收益Physical Mechanism & Measured E2E Impact |
|---|---|---|---|
Layer 1PyTorch & Distributed |
0% Pallas Sharding / State / D2D |
DP4×TP2 + EP8PR #1458 · PR #106PR #1183, #1558, #1677( tpu-connector / Raiden) |
同机 8 Devices (2x2x1) 解耦 Attention DP4×TP2(卡住 $H_{kv}=2$ 零复制点,芯片内 600 GB/s D2D AllReduce)与 MoE EP8(64 专家/Rank);PR #1458 将 GDN 循环状态从 FP32 降为 BF16(KV 池 +12.2%、Decode +9.1%);PR #106 统一 KV/GDN 页池;TPUConnectorHMA / Raiden 一次性跨机直传完整 Tail KV + 45 层 GDN 状态(~349 MB),实现零 Tail 重算(Zero Tail Recompute)、异构 TP 重分片与双 200G 网卡 NUMA 绑核满跑 367 Gbps。
Decouples Attention DP4×TP2 ($H_{kv}=2$ zero-duplication point, intra-chip 600 GB/s D2D AllReduce) from MoE EP8 (64 experts/rank) on the same 8 TPU7x devices; PR #1458 converts GDN state FP32➔BF16 (+12.2% KV pool, +9.1% decode); PR #106 unifies KV/GDN page pools; tpu-connector / TPUConnectorHMA / Raiden ships full tail KV + 45-layer GDN state (~349 MB) with zero tail recompute, heterogeneous resharding, and dual-200G NIC NUMA pinning at 367 Gbps.
|
Layer 2Graph Capture |
0% Pallas Dynamo / AOT Export |
√2 Precompile BucketsPR #2277 (AOT Export) |
按 $\sqrt{2}$ 几何级数离散化 Shape Buckets 并结合 PR #2277 Ahead-of-Time (AOT) 导出与 GCS 共享编译缓存,彻底消灭在线 Serving 期间的 JIT 重编译毛刺。
Geometric $\sqrt{2}$ precompile shape buckets paired with PR #2277 AOT Export and shared GCS compilation cache eliminate runtime JIT recompilation stalls during live serving.
|
Layer 3StableHLO Views |
0% Pallas Zero-Copy IR Bitcast |
PR #766(4D Zero-Copy bitcast) |
针对混合缓存池的跨视角切片,用 4D 零拷贝 bitcast 视图替代物理拷贝,将 XLA 生成的 DMA 搬运指令从 366 条骤降至 87 条,GDN 算子耗时从 4.12 ms ➔ 0.774 ms(5.3× 加速)。
Replaces physical tensor copies with a 4D zero-copy bitcast view in StableHLO, slashing DMA instructions from 366 ➔ 87 and accelerating GDN by 5.3× (4.12 ➔ 0.774 ms).
|
Layer 4XLA [8, 128] Tile Layout |
Layout Co-Design 32-bit Lane Packing |
PR #608( SEQ_ON_LANE) |
将 TPU [8, 128] 物理 Tile 的 32-bit 槽位轴从 head 旋转至 sequence,使 4 个连续 FP8 Token 紧密打包进同 1 个 32-bit 物理槽,消灭 TP≥2 单 KV Head 下的 50% 0-Padding,单页步进从 1,024 B 减半至 512 B/token/layer(直接支撑 1,152 页对齐)。
Rotates the 32-bit slot axis of the TPU [8, 128] tile from head to sequence so 4 FP8 tokens pack into 1×32-bit slot, halving KV page stride from 1,024 ➔ 512 B/token/layer at TP≥2.
|
Layer 5 (Left)Kernels: XLA Fusions |
0% Pallas Native MXU Matmul |
PR #371 / #509( F.one_hot + MXU) |
在小 Batch Decode 下主动绕开 Pallas GMM,仅用 3 行纯 PyTorch F.one_hot + 矩阵乘交给 XLA 融合直推 256×256 MXU,在 c64 并发下反超手写 Kernel +14.5% 端到端吞吐。
Bypasses Pallas on small decode batches with 3 lines of pure PyTorch F.one_hot + matmul lowered by XLA directly onto the 256×256 MXU, gaining +14.5% E2E throughput at c64.
|
Layer 5 (Right)Kernels: Surgical Pallas |
Surgical Pallas Async DMA + VMEM Alias |
PR #543 + #891 (Fused MoE)PR #1008 (RPA v3 MTP)PR #2276–#2361 + #2227 |
① PR #543 + #891 全层 SPMD Fused MoE:利用双缓冲异步 DMA 彻底掩盖 1,190 µs(占全步 22%)的裸露 ICI Reduce-Scatter,单层加速 4.8×、E2E 吞吐翻倍(10,614 ➔ 21,297 tok/s);② PR #1008 RPA v3:MTP 树验证耗时 10.2 ➔ 5.6 ms;③ PR #2276–#2361 声明式 Pallas Megakernel + pool_alias (#2227):跨算子分时复用 VMEM scratchpad。
① PR #543 + #891 Whole-Layer SPMD Fused MoE: Double-buffered async DMA hides 1,190 µs (22% exposed ICI Reduce-Scatter), yielding 4.8× layer speedup and 2× E2E throughput (10,614 ➔ 21,297 tok/s); ② PR #1008 RPA v3: MTP verify 10.2 ➔ 5.6 ms; ③ PR #2276–#2361 Declarative Pallas Megakernel + pool_alias (#2227) VMEM time-sharing.
|
Layer 6ICI Collectives |
SPMD Collective Sequence ↔ Head Switch |
PR #65 / #129( PCP=8 All-to-All) |
Prefill Context Parallelism (PCP=8) 通过 ICI All-to-All 在序列切分与 Head 切分间动态切换,将 128K 超长上下文 TTFT 从 32s 骤降至 4.62s(6.9× 加速)。
PCP=8 dynamic sequence-to-head switch via ICI All-to-All slashes 128K long-context TTFT from 32s ➔ 4.62s (6.9× speedup).
|
Layer 7PJRT Runtime & Host OS |
0% Pallas Runtime / THP / DRAM |
PR #1810 (Peer Replay)PR #2141 / #2068(Host-RAM & THP Release) |
PR #1810 跨 Rank 编译产物对等重放(Cross-Rank Peer Replay)将 DP=8 启动时间从 741s 砍至 270s;PR #2141 / #2068 权重加载期临时内存释放与 Linux THP 治理将单机峰值 Host DRAM 从 440 GiB 压降至 230 GiB。
PR #1810 Cross-Rank Peer Replay cuts DP=8 startup from 741s ➔ 270s; PR #2141 / #2068 startup Host-RAM & Linux THP release cuts peak Host DRAM from 440 ➔ 230 GiB.
|
6.73 ms HBM 物理极限下界 → 22.72 ms 解析预测 vs. 24.10 ms 9,216 芯片生产实测(误差 <6%)
C. Step 0 First-Principles Roofline Closure: 6.73 ms HBM Weight-Read Floor → 22.72 ms Analytic Prediction vs. 24.10 ms 9,216-Chip Production Canary (<6% Error)
6.73 ms):在单台 2x2x1(4 颗 TPU7x 芯片 = 8 Devices,聚合有效 HBM 带宽 8 × 7,180 GB/s = 57.44 TB/s)上,当 MoE 激活专家趋近全激活(512 专家全读 387.17 GB FP8 权重,单步总活跃权重 48.62 GB/dev)时,仅从 HBM 搬运权重的物理极限时间即为 6.73 ms。DP4×TP2 + EP8 生产工作点闭环(误差仅 5.7%):计入每序列 166.8 MB 状态流式读取、芯片内 600 GB/s D2D TP=2 AllReduce 与 EP=8 Dispatch/Combine 集合通信后,Step 0 解析 Roofline 预测单步 Decode 耗时为 22.72 ms;而 9,216 颗芯片(1,152 台主机)生产集群灰度实测中位数 TPOT 为 24.10 ms(1,083.9 tok/s),理论模型与超大规模生产实测偏差 < 6%(5.7%),彻底闭环了「先用 Step 0 数学锁死物理下界与最优切分,再按 7 层栈精准下刀」的工程方法论!
• Absolute HBM Weight-Read Floor (6.73 ms): On a single 2x2x1 host (4 TPU7x chips = 8 devices, aggregate effective HBM bandwidth 8 × 7,180 GB/s = 57.44 TB/s), reading the active FP8 weights (48.62 GB/dev at full expert activation) sets a hard physical weight-read floor of 6.73 ms.DP4×TP2 + EP8 Production Canary Closure (5.7% Error): Adding per-sequence KV+GDN state streaming (166.8 MB/seq), intra-chip 600 GB/s D2D TP=2 AllReduce, and EP=8 All-to-All collectives yields a Step 0 analytic prediction of 22.72 ms per decode step — matching the 24.10 ms median TPOT measured on the 9,216-chip production canary within < 6% (5.7%)!
96 Q 头 / 32 KV 头 (3:1) · head_dim 128 · hidden_size 4096 · vocab 248,320 · 512 专家 Top-8。
KV 只由 FA 层产生:15 × 32 × 128 × 2 × 131072 = 15 GiB/请求 (FP8),摊到 8 卡即单卡 1.875 GiB(实测记作 1.92 GB)。若误按 60 层甚至 80 层全部计 KV,会高估 4 ~ 5.3 倍。
采样段(temperature / top_p / top_k / presence_penalty / context_window / speculative_tokens)取自 generation_config.json 默认值,可拖动滑块实时体会。
⚠ Architecture corrected against the joint-optimization record: the Qwen 3.5 actually deployed is a hybrid — roughly 15 full-attention (GQA) layers plus 45 GDN linear-attention layers, 60 in total (the record's measured prefill breakdown: full 492 µs × 15, linear 774 µs × 45, MoE 1.41 ms × 60). GDN is not an optional optimization on some experimental branch; it is part of the model.
The remaining architecture figures are 96 Q heads / 32 KV heads (3:1) · head_dim 128 · hidden_size 4096 · vocab 248,320 · 512 experts, Top-8.
Only the FA layers produce KV: 15 × 32 × 128 × 2 × 131072 = 15 GiB per request in FP8, or 1.875 GiB per chip across 8 chips (recorded as 1.92 GB). Counting all 60 — let alone all 80 — layers overstates KV by 4× to 5.3×.
The sampling section (temperature / top_p / top_k / presence_penalty / context_window / speculative_tokens) comes from generation_config.json defaults; drag the sliders to feel each one.
[B, T, 4096]。4096 = 32 × 128,是 MXU 通道宽度的整数倍,GEMM 收缩维无 padding 损耗。
Sets the embedding and per-layer tensor shape [B, T, 4096]. 4096 = 32 × 128 is an exact multiple of the MXU lane width, so the GEMM contraction dimension wastes no padding.
[248320, 4096],BF16 下单个矩阵即 1.89 GiB。LM Head 的 fusion.988 就是在这个维度上做最终投影。
The classification dimension of both the embedding matrix and the LM head, shaped [248320, 4096] — 1.89 GiB per matrix in BF16. The LM head's fusion.988 projects onto exactly this axis.
embed_tokens.weight 作为 lm_head.weight。解耦多占 1.89 GiB 权重,换来更高的表征自由度与生成准确率。
Controls whether embed_tokens.weight is reused as lm_head.weight. Untying costs an extra 1.89 GiB of weights and buys representational freedom and generation accuracy.
y = x / sqrt(mean(x²) + ε) ⊙ γ。该常数保证激活在整条浮点链路上不会除以零,杜绝 NaN 传播。Qwen 还在 Q/K 上各加了一次 QK-Norm,用同一个 ε。
y = x / sqrt(mean(x²) + ε) ⊙ γ. The constant keeps activations from dividing by zero anywhere along the float path, so no NaN propagates. Qwen additionally applies QK-Norm to Q and K, using the same ε.
slice.7534 取 [0:1536]);上线的 PCP=8 形态下注意力权重是复制的,每卡仍持 96 头、只切序列;Decode TP=2 时每卡 48 个头。
The Q projection lifts 4096 up to 96 × 128 = 12288 columns. In this trace (the TP=8 variant) each chip holds 12 Q heads (slice.7534 takes [0:1536]); in the deployed PCP=8 form attention weights are replicated, so every chip keeps all 96 heads and only the sequence is split. At decode TP=2 each chip holds 48.
Softmax(Q·Kᵀ / √128)·V。注意这里有真实损耗:head_dim = 128 只填满 TPU v7 那 256 宽 MXU 的一半,Q·Kᵀ 这一步的形状对齐并不完美——它是本文档里少数几处硬件利用率没跑满的地方。
The attention core computes Softmax(Q·Kᵀ / √128)·V. There is a genuine cost here: head_dim = 128 fills only half of the v7's 256-wide MXU, so the Q·Kᵀ step is not perfectly shape-aligned — one of the few places in this document where the hardware is not fully fed.
all-to-all.18/19 派发与回收。
512 feed-forward networks stand by, 397B parameters in total. At EP=8 each chip hosts 64 experts, and routing dispatches and collects through all-to-all.18/19.
fusion.815 对 512 个专家打分并取 Top-8,再经 All-to-All 派发。专家输出回到宿主卡后由 owner_output 分支原地加权聚合,不再申请临时显存。
The router fusion.815 scores all 512 experts, takes the top 8, and dispatches via All-to-All. Results return to the owner chip, where the owner_output branch combines them in place with no staging allocation.
d_ff = 5632(是单个路由专家 1408 的 4 倍),负责吸收最高频的通用语言与语法逻辑,避免动态门控出现常识盲区。
A base FFN on every token's path with d_ff = 5632 — four times a routed expert's 1408 — absorbing the highest-frequency grammar and general language so the dynamic gate never has a blind spot on the obvious.
[4096, 1408] ×2 与 [1408, 4096]。维度小意味着 All-to-All 搬运的张量体积小,EP8 通信可压在毫秒级以内。
Under SwiGLU each expert carries gate / up / down matrices: [4096, 1408] ×2 and [1408, 4096]. Keeping the width small keeps the All-to-All payload small, which is what holds EP8 communication under a millisecond.
P(wᵢ) = exp(zᵢ/T) / Σ exp(zⱼ/T)。T → 0 坍缩为贪婪解码(argmax),T → ∞ 退化为均匀随机。
Divides the logits: P(wᵢ) = exp(zᵢ/T) / Σ exp(zⱼ/T). As T → 0 it collapses to greedy decoding (argmax); as T → ∞ it degenerates to a uniform draw.
logit − c × p(与出现次数无关,这点区别于 frequency_penalty)。
If a token has appeared even once in the generated text, a fixed penalty is subtracted from its logit: logit − c × p. Unlike frequency_penalty, the count does not matter — only presence.
fusion.990 / RPA verify 特化)。命中即一次权重读出产出多词,实测吞吐 5,650 tok/s。
A lightweight draft head proposes k candidates at once and the main model verifies them in a single parallel pass (fusion.990, the specialized RPA verify). On a hit, one weight read yields several tokens — measured at 5,650 tok/s.
| 旋钮Knob | 改什么What it changes | Qwen 3.5 的选择What Qwen 3.5 chose | 代价 / 理由Cost / rationale |
|---|---|---|---|
| ① 每个 token 存多少bytes per token | 减少 KV 份数或维度Fewer or smaller KV copies | 保守Conservative GQA 96Q / 32KV(3:1),未上 MLAGQA 96Q / 32KV (3:1), no MLA |
MLA 的前提是「算力充裕、显存紧张」,且压缩在训练前向里不生效;3:1 的 GQA 换来的是完全规整的读法。MLA assumes "compute abundant, memory tight" and its compression does not apply in the training forward pass; a 3:1 GQA buys fully regular reads instead. |
| ② 每个 query 看多少how much each query reads | 限制范围或运行时动态挑选Restrict the span, or pick at runtime | 主动弃权Abstained 整份记录里没有 DSA / NSA / 任何 runtime top-kNo DSA, no NSA, no runtime top-k anywhere in the record |
这是本案例最大的一次权衡。稀疏注意力的隐含硬件假设是「gather 不太贵」——而这正是 TPU 为换取规整访存主动放弃的能力。放弃旋钮② = 放弃一条公认有效的省算力路线,换「不必在 TPU 上打那场还没打赢的仗」。The single biggest trade here. Sparse attention silently assumes gathers are cheap — exactly the capability TPU gives up in exchange for regular access. Abstaining from knob ② means giving up a proven compute-saving route to avoid a battle that has not yet been won on this hardware. |
| ③ 换一套数学swap the mathematics | 用固定大小状态替代变长 KVA fixed-size state instead of a growing KV cache | 激进Aggressive 45 / 60 层是 GDN 线性注意力45 of 60 layers are GDN linear attention |
线性注意力的前提是「片上内存放得下 chunk 的中间量」。128×128 状态 FP8 压到 16 KB,v7 有 64 MiB VMEM —— 假设成立,兑现 4.9×。这是本案例里少数赌对了的地方。Linear attention assumes the chunk's intermediate state fits on chip. The 128×128 state compresses to 16 KB in FP8 against v7's 64 MiB VMEM — the assumption holds and the 4.9× is realised. One of the few bets here that paid. |
| ①+③ 混合Hybrid | 不同层用不同方案Different layers, different schemes | 15 层 FA + 45 层 GDN = 60 层15 FA + 45 GDN = 60 layers | 只有 15 层产生 KV → 128K 全请求 15 GiB 而非 60 / 80 GiB。Only 15 layers produce KV → 15 GiB for a full 128K request rather than 60 or 80 GiB. |
| 优化项Optimization | 搬的是哪个房间Which room it moves | 省了显存?Saves memory? | 省了 FLOPs?Saves FLOPs? | 实测结果Measured result |
|---|---|---|---|---|
SEQ_ON_LANE / HND KV 布局KV layout |
纯访存规整度Pure access regularity | 否No | 否No | decode 700 → 5,790 tok/s (8.3×) |
| PCP ring 取代 all-gatherreplacing all-gather | 通信规整度(连号 P2P vs 广播散取)Comm regularity (contiguous P2P vs scattered broadcast) | 否No | 否No | 950 µs 传输 100% 藏进计算阴影(bubble = 0)950 µs of transfer fully hidden under compute (bubble = 0) |
owner_output blockwise combine |
写回规整度Write-back regularity | 否No | 否No | decode 5,500 → 5,650 tok/s |
| GDN 融合 FP8 状态锁 VMEMFused FP8 GDN state pinned in VMEM | 消灭 HBM 写回(访存量)Eliminates HBM write-back (traffic) | 是Yes | 否No | 4,120 µs → 840 µs (4.9×) |
| PCP=8 序列并行sequence parallelism | 显存 + 算力(把 128K 拆成 8 × 16K)Memory + compute (128K cut into 8 × 16K) | 是Yes | 重分布Redistributed | 128K TTFT 32 s → 4.62 s (6.9×,同 8 芯片)128K TTFT 32 s → 4.62 s (6.9×, same 8 chips) |
full 492 µs × 15 vs linear 774 µs × 45。full 492 µs × 15 against linear 774 µs × 45.(16 头, 全序列) 并跨节点搬运——头与序列两个维度同时重分布,是 PD 分离固有的主要开销。#839 open)。(16 heads, full sequence) and moved across nodes — redistributed along the head and sequence axes at once, the dominant intrinsic cost of PD disaggregation.#839 open).owner_output——而不是注意力。
MoE takes two thirds; all attention combined takes one third; full attention alone takes 5.8%.
Deleting all 15 full-attention layers outright would speed prefill up by 5.8%.
(This is the prefill caliber — all three numbers are prefill measurements. Decode keeps a separate ledger; see card E below: MoE still dominates there, but the mechanism becomes weight streaming and tile waste drops to second place.)owner_output — not attention.
| 曾经的说法What it used to say | 缺了哪个限定Missing qualifier | 现在的处理How it reads now |
|---|---|---|
| TTFT 32 s → 4.62 s(6.9×) (6.9×) | 「128K 单请求」+「同样 8 颗芯片」。不写就会被读成扩容收益。"128K, single request" and "the same 8 chips". Without them it reads as a scaling win. | ✅ 全文均标注「同 8 芯片」Annotated "same 8 chips" throughout |
| 700 → 5,790 与and 5,500 → 5,650 tok/s | 两个不同基线,不能相乘、不能相加 —— 与「512 倍与 846 倍是两把尺子」同型。Two different baselines; they cannot be multiplied or added — the same error as chaining a 512× and an 846× figure. | ✅ 各自注明分支与口径Each scoped to its own branch |
| 492 µs = 每跳 × 8 跳 → 15 层 61.8 ms492 µs per hop × 8 → 61.8 ms for 15 layers | 与文档自身预算矛盾:按每跳算三项合计 178.5 ms > 记录的 158 ms 总时延;按每层算合计 126.8 ms,落在预算内。Contradicts the document's own budget: per-hop makes the three terms sum to 178.5 ms against a recorded 158 ms total; per-layer sums to 126.8 ms, inside it. | ✅ 改判为每层,15 层 ≈ 7.4 msRe-read as per layer; 15 layers ≈ 7.4 ms |
| 4,120 µs / 层4,120 µs per layer | 另一把尺子,来源不明(可能是另一配置或含 MoE 的整层)。A different ruler of unknown origin (possibly another configuration, or a whole layer including MoE). | ⚠ 保留但标注「不要与 492 µs 混用」Kept, flagged "do not mix with the 492 µs figure" |
| 阶段带 1.85 s vs 算子表 126.8 ms1.85 s on the stage ribbon vs 126.8 ms in the operator table | 差约 15× —— 典型的「报哪一个」问题。Roughly 15× apart — a textbook "which one do you quote" problem. | ⚠ 标注为未闭合,禁止同表相加Marked unreconciled; must not be summed in one table |
| QKV 输出 2560 列 → 推断芯片数QKV output of 2,560 columns → inferring the chip count | 与上线的 PCP8+EP8(注意力权重复制)对不上。Inconsistent with the deployed PCP8+EP8 form, where attention weights are replicated. | ⚠ 挂起:不得据 trace 形状推断芯片数Open: do not infer chip count from trace shapes |
492 µs × 15 + 774 µs × 45 + 1.41 ms × 60 拆出来的,那组数就是 prefill 实测。
decode 是另一本账:那里既不是注意力、也不是 MXU 说了算,而是专家权重每步都要被完整读一遍。
这一条不需要任何实测就能定下来,因为它只用到权重体积和 HBM 带宽两个数。
The 5.8 / 27.5 / 66.7 split in C④ above is the prefill ledger — it comes out of
492 µs × 15 + 774 µs × 45 + 1.41 ms × 60, and those are prefill measurements.
Decode keeps a different book: there neither attention nor the MXU sets the pace —
the expert weights have to be read end to end on every single step.
This one needs no measurement at all; it uses only two numbers, weight volume and HBM bandwidth.
| 步骤Step | 算式Arithmetic | 结果Result |
|---|---|---|
| 每芯片权重Weights per chip | 397 GB (fp8) ÷ 8 chips |
49.63 GB = 46.22 GiB |
| 每层Per layer | 49.63 GB ÷ 60 |
0.827 GB |
| 读一遍的时间Time to read it once | 0.827 GB ÷ 7.37 TB/s |
112 µs / 层layer |
| decode 单步地板Floor for one decode step | 112 µs × 60 |
6.73 ms |
| 量化到 4 bit 权重后With 4-bit weights | W4A8 / NVFP4 |
3.37 ms |
1 − e^(−B/64),于是:TPOT 地板(B) = 6.73 ms × (1 − e^(−B/64))1 − e^(−B/64), so:TPOT floor(B) = 6.73 ms × (1 − e^(−B/64))| 假设 batchAssumed batch | 命中专家比例Experts lit | TPOT 地板TPOT floor | 实际 TPOT(由 5,650 tok/s 反推)Actual TPOT (from 5,650 tok/s) | 权重 MBUWeight-only MBU |
|---|---|---|---|---|
| 32 | 39.3% | 2.65 ms | 5.66 ms | 46.8% |
| 64 | 63.2% | 4.26 ms | 11.3 ms | 37.6% |
| 128 | 86.5% | 5.82 ms | 22.7 ms | 25.7% |
| 256 | 98.2% | 6.61 ms | 45.3 ms | 14.6% |
tiles_m ≤ 实际执行块数 ≤ tiles_m + 段数 − 1(每芯片 64 个本地专家 = 64 段)tiles_m = 2,上界 2 + 63 = 65 块 → 32×。对应约 123 µs(bf16 口径),和 112 µs 的权重地板基本持平 —— 第二顺位,但已经贴着。tiles_m = 128,上界 128 + 63 = 191 块 → 仅 +49%。tiles_m ≤ tiles actually executed ≤ tiles_m + segments − 1 (64 local experts per chip = 64 segments)tiles_m = 2, bound 2 + 63 = 65 tiles → 32×. That is ~123 µs in bf16 terms, level with the 112 µs weight floor — second in line, but right up against it.tiles_m = 128, bound 128 + 63 = 191 tiles → only +49%.tile_m = 128 的块占用模型算一遍:
Start with something the report does not say but its own config implies: per-rank FLOPs are identical across all three GPU counts (routed rows per rank = tokens/rank × top-k = 2,560, independent of EP). So "latency drops from 8 to 32 GPUs" is not strong scaling at all — no rank does less work. Only two things change: rows per expert (up) and all-to-all peers (up). Run the tile_m = 128 occupancy model over it:
| 配置Config | 每专家行数 8 / 16 / 32Rows per expert, 8 / 16 / 32 | 块浪费Tile waste | 实测加速 8→16 / 8→32Observed 8→16 / 8→32 | 模型预测Model predicts |
|---|---|---|---|---|
| Qwen 256 tok, k=10 | 40 / 80 / 160 |
3.2× → 1.6× → 1.6× | 1.44× / 1.31× | 2.0 / 2.0 |
| Qwen 512 tok, k=10 | 80 / 160 / 320 |
1.6× → 1.6× → 1.2× | 1.15× / 0.88× | 1.0 / 1.33 |
| DeepSeek 256 tok | 32 / 64 / 128 |
4.0× → 2.0× → 1.00× | 1.46× / 1.73× | 2.0 / 4.0 |
| DeepSeek 512 tok | 64 / 128 / 256 |
2.0× → 1.00× → 1.00× | 1.32× / 1.36× | 2.0 / 2.0 |
tokens/rank × top-k ÷ 专家数 与 GEMM 块的关系。换个 batch,最佳点就搬家。tokens/rank × top-k ÷ experts and the GEMM tile. Change the batch and the optimum moves.| 8 卡8 GPUs | 32 卡32 GPUs | |
|---|---|---|
| top-k=10 每专家行数 → 块数top-k=10 rows/expert → tiles | 40 → 1 | 160 → 2 |
| top-k=6 每专家行数 → 块数top-k=6 rows/expert → tiles | 24 → 1 | 96 → 1 |
| 块数变化Change in tile count | 没变none | 减半halved |
| 实测收益Observed gain | 8% | 30% |
bf16_bf16_tile_m = 128 要 8,192 个 token 才填满一块。TPU 的 megablox 是重复执行边界块而不是补零,形状上不同、账是一样的。所以 EP 的第一价值不是省通信,是把段拼长;而这也解释了为什么 E 卡里那 123 µs 的浪费已经和权重地板同量级。GB200 在 EP8 上做到 55.6% MBU —— 我们 decode 侧的 MBU 是多少,目前没有能算出它的数(缺 batch 和上下文长度),但这是两边口径可比的量,值得测一个出来。
The worst rung in the GB200 set is 40 rows per expert. One decode step here is 256 tokens, top-8, 512 experts — 4 rows per expert, 10× worse, against a megablox bf16_bf16_tile_m = 128 that needs 8,192 tokens to fill a single tile. TPU megablox re-executes boundary tiles rather than zero-padding them; different shape, same bill. So the first value of EP is not saving communication, it is making the segments longer — and it is why the 123 µs of waste in card E is already the same order as the weight floor. GB200 reaches 55.6% MBU at EP8; what our decode MBU is, we currently cannot compute (batch and context length are both missing) — but it is a like-for-like quantity across the two platforms and worth measuring.
GDN (ns) 320、Per-layer (ns) 688,但 688 ns × 60 层 = 41.3 µs,而 Total 行写的是 41 ms。四组全部验过(41.27 / 67.52 / 42.94 / 74.54 对表里的 41 / 67 / 43 / 75)—— 那一列应该是 µs。差 1000 倍,而它长得完全正常。GDN (ns) 320 and Per-layer (ns) 688, but 688 ns × 60 layers = 41.3 µs, while the Total row prints 41 ms. All four configs check out (41.27 / 67.52 / 42.94 / 74.54 against the printed 41 / 67 / 43 / 75) — that column must be µs. A factor of 1000, and it looks entirely normal.SEQ_ON_LANE 选定序列住哪一维、多套预编译拓扑 PCP8→DP8 / DP16 / TP2DP4EP8 / TP2DP8EP16 / P4D4),但换来的静态性是有代价的:
The lecture's three techniques share one shape: replace one dynamic thing with a handful of static ones. This deployment does exactly that (chunked prefill fixed at 16K blocks, SEQ_ON_LANE choosing which axis the sequence lives on, and a set of pre-compiled topologies: PCP8→DP8 / DP16 / TP2DP4EP8 / TP2DP8EP16 / P4D4). But that static-ness was bought:
| 换到了什么What was gained | 付出了什么What it cost | 状态Status |
|---|---|---|
| 规整访存(不必打稀疏 gather 那场仗)Regular access — no sparse-gather battle to fight | 放弃旋钮②,也放弃 MLA;两条公认有效的路线留在桌上Knob ② abandoned, and MLA with it — two proven routes left on the table | 设计取舍By design |
| 吞吐(5,650 ➔ 6,500–7,600 tok/s decode · 44k ➔ 55k–58k tok/s 4-Host prefill)Throughput (5,650 ➔ 6,500–7,600 tok/s decode; 44k ➔ 55k–58k tok/s 4-Host prefill) | tpu-sync 线程泄漏已修(#1112/#1115/#1116),但大规模试投产仍面临低频 E200 CoreHalt 与 MTP 接收率偶发跌 0%tpu-sync leaks fixed (#1112/#1115/#1116), while fleet burn-in still tracks low-frequency E200 CoreHalt and intermittent MTP 0% drops |
持续收敛Converging |
| PD 分离(prefill 与 decode 各自打满)PD disaggregation (each phase saturated independently) | 单网卡初期仅兑现 50% 线速;后经 Raiden NUMA 双网卡(367 Gbps / 91.8% 线速)+ gRPC 控制面(2MB p99 −35.8%)破局,代价是单 Worker 需常驻 ~6.53 GB Staging 缓冲Single-NIC initially hit 50% line rate; resolved via Raiden NUMA dual-NIC (367 Gbps / 91.8% line rate) + gRPC control plane (2MB p99 −35.8%), at the cost of ~6.53 GB resident staging buffer per worker | 已破局兑现Delivered |
下一档扩容(PCP16·TP2 ➔ TP32·DCP4/8 及 PP4·PCP8)The next scale-up (PCP16·TP2 ➔ TP32·DCP4/8 & PP4·PCP8) |
废弃纯 4-Host All-Gather(−30% E2E)改走 PP4·PCP8 实测达 55k–58k tok/s,#1108/#1182/#1183 已合入;剩余 3K+5K 分块 Padding 抖动与 DCP VMEM 估算收尾 —— 核心矛盾始终在「第三个房间」(通信与调度编排)Retired pure 4-Host All-Gather (−30% E2E) in favor of PP4·PCP8 (55k–58k tok/s measured) with #1108/#1182/#1183 merged; remaining items are 3K+5K chunk-padding variance and DCP VMEM estimation — every bottleneck still lives in the third room (collectives & scheduling) |
收尾调优Final tuning |
结合 Google DeepMind Jacob Austin《A Whirlwind Tour of LLM Serving》(go/llm-inference-slides)与 SemiAnalysis 针对 TPU v7 (Ironwood) 的芯片级解构,本项目各项关键拓扑选型背后存在着无可辩驳的物理必然性: Synthesizing Google DeepMind Jacob Austin's foundational framework ("A Whirlwind Tour of LLM Serving", go/llm-inference-slides) with SemiAnalysis's silicon deconstruction of TPU v7 (Ironwood), the key topology choices in this project are governed by strict physical laws:
Y ≤ F / 2550(其中 F 为 FFW 中间维,2550 为 TPU 脉动算力与 ICI 网络带宽的物理特征常数)。TP=8。TP=2 · DP=4 · EP=8 是守住通信物理红线的唯一数学正解。
Per DeepMind Austin's model parallelism threshold:
Y ≤ F / 2550 (where F is FFW intermediate dim, and 2550 is the TPU FLOPs/ICI-bandwidth ratio).TP=8.TP=2 · DP=4 · EP=8 is the mathematically enforced frontier.
EP=8 真实物理指代:阿里 2×4 拓扑 中的单机 8device 语境指代整机 8 个物理 Chip 插槽(底层映射 16 个逻辑 Device),512 个专家被均衡分发到 8 颗物理芯片上,每芯承载 64 个专家并由片内 SparseCore 硬件路由。
• Dual-Die Packaging (Per SemiAnalysis): Each physical Ironwood package contains 2 compute dies exposed as 2 distinct logical devices in JAX/TorchTPU, linked by on-package high-speed Die-to-Die traces.EP=8 Truth: The 2×4 topology refers to 8 physical chip sockets per host (mapping to 16 logical devices). The 512 experts are mapped across the 8 physical chips (64 experts/chip) accelerated by on-chip SparseCores.
NVFP4 + W4A8,是因为在 Decode 阶段瓶颈受困于 HBM 读取带宽;将权重以 NVFP4 紧凑格式装载入片上 SRAM,再通过向量指令在线解压至 8-bit 送入 MXU,以 0 精度惩罚换取了 2× 显存带宽等效提升!
• No Native FP4 Engine: TPU v7's MXU only accelerates INT8/FP8/BF16 in silicon; native hardware FP4 computation is reserved for TPU v8i.NVFP4 + W4A8 because Decode is HBM-bandwidth bound; storing weights in NVFP4 cuts memory traffic in half, decompressing on-chip into 8-bit for MXU execution with zero accuracy penalty!
torchtpu-agents@0.1.0 的逐行阅读,
而非厂商文档;第 5 卡的性能数字来自本文档配套集群上的真实物理派发,非估算。
The preceding pages record a hand-driven optimization campaign. This page records the agentic toolchain that
industrializes it — a 31-agent package turning kernel tuning, sharding search and migration verification into
reproducible, auditable pipelines.torchtpu-agents@0.1.0, not from vendor documentation. The numbers in card 5 come from real physical
dispatch onto this document's companion cluster, not from estimation.
这个包不是 31 个平级工具,而是一棵 4 层深(最长路径 tpu-coworker → maxkernel-loop → worker → implement-kernel 共 3 跳委派)的严格分层树。分层本身就是让优化成本随规模保持平坦的机制。
The package is not 31 peer tools. It is a strictly layered 4-level tree (deepest chain: tpu-coworker → maxkernel-loop → worker → implement-kernel, 3 delegation hops). Layering is the mechanism that keeps cost flat as an optimization effort grows.
tpu-coworker (L0 唯一入口 — 纯路由,自身不持有 shell)
│
├─ [族 1 · 诊断与溯源 (3)]
│ ├─ hello-tpu ................. 硬件可达性冒烟测试与 CPU 交叉校验
│ ├─ provenance ................ TPU 技术结论的可引用公开出处校验
│ └─ xprof-analyst ............. 全包唯一 Profile 入口(各层级共享)
│
├─ [族 2 · Kernel 与跨轮研究 (13)]
│ ├─ maxkernel-loop ............ 单个命名 Pallas Kernel 迭代编排(固定 5 轮)
│ │ └─ worker ............... 单轮完整迭代(7 相位执行器)
│ │ └─ 10 个 Kernel 专家(plan / implement / fix-compile / autotune / summarize 等)
│ └─ maxauto-round ............. 跨 Session 长期课题账本(每轮推进单个 Arm)
│
├─ [族 3 · 分布式切分与配置搜索 (5)]
│ └─ maxshard-orchestrator ..... 已知 Workload 的配置爬山编排
│ ├─ maxshard-optimizer ... "大脑":形成假设与下一套配置
│ ├─ maxshard-executor .... "肌肉":提交作业 / 监控 / 解析指标
│ ├─ maxshard-researcher .. 启发式用尽后的广度搜索
│ └─ maxshard-memory-contributor ... 实证经验去重与入库
│
└─ [族 4 · HF 迁移与常驻服务化 (9)]
└─ torch-tpu-migration-orchestrator ... HF 模型迁移 TPU 并执行基准测试
├─ smart-hf-search / preflight-check / torch-coder / remote-tpu-runner
└─ torch-tpu-inference ... 常驻 HTTP 推理服务生命周期编排
└─ engine-compatibility / torch-inference-coder / numerical-parity
tpu-coworker (L0 sole entrypoint — pure router, holds no shell)
│
├─ [Family 1 · Diagnostics & Provenance (3)]
│ ├─ hello-tpu ................. hardware reachability smoke test & CPU cross-check
│ ├─ provenance ................ citable public source verifier for TPU claims
│ └─ xprof-analyst ............. the ONLY profiling entrypoint (shared across tiers)
│
├─ [Family 2 · Kernel Optimization & Study Ledger (13)]
│ ├─ maxkernel-loop ............ single named Pallas kernel loop (fixed 5 rounds)
│ │ └─ worker ............... one full iteration (7-phase executor)
│ │ └─ 10 kernel specialists (plan / implement / fix-compile / autotune / summarize)
│ └─ maxauto-round ............. cross-session study ledger (advances one arm per round)
│
├─ [Family 3 · Sharding & Config Hill-Climbing (5)]
│ └─ maxshard-orchestrator ..... config hill-climbing on a known workload
│ ├─ maxshard-optimizer ... the "brain": hypothesis & next configuration
│ ├─ maxshard-executor .... the "muscle": job submission / monitoring / parsing
│ ├─ maxshard-researcher .. broad search when greedy heuristics saturate
│ └─ maxshard-memory-contributor ... empirical takeaway deduplication & ingestion
│
└─ [Family 4 · HF Migration & Persistent Serving (9)]
└─ torch-tpu-migration-orchestrator ... HF-to-TPU migration & benchmark pipeline
├─ smart-hf-search / preflight-check / torch-coder / remote-tpu-runner
└─ torch-tpu-inference ... persistent HTTP serving lifecycle orchestrator
└─ engine-compatibility / torch-inference-coder / numerical-parity
| 分区 / 结构性不变量Partition / Structural invariant | 数量Count | 工程构成与约束后果Composition & engineering consequence |
|---|---|---|
| 31 个 Agent 完整分族核算Complete 31-agent family accounting | 1 + 3 + 13 + 5 + 9 = 31 |
1 个顶层路由(tpu-coworker)+ 3 个诊断/溯源叶节点 + 13 个 Kernel/Study 节点 + 5 个 MaxShard 切分节点 + 9 个 HF 迁移与服务化节点,零遗漏。
1 router (tpu-coworker) + 3 diagnostic/provenance leaves + 13 kernel/study agents + 5 MaxShard agents + 9 HF migration & serving agents = 31 total.
|
| 操作者可直接调用的 AgentAgents callable directly by an operator | 7 / 31 |
入口白名单是一条禁令:"Do not invoke any other package agent directly." 绕过它意味着请求会在触及任何硬件之前被目标 Agent 的契约拒绝。 The allowlist is a prohibition: "Do not invoke any other package agent directly." Bypassing it sends a request to an agent whose contract rejects it before hardware is touched. |
| 持有委派权的 AgentAgents holding delegation rights | 9 / 31 | 其余 22 个是叶子。叶子无法扇出,因此一个错误计划的爆炸半径被这 9 个封死。 The remaining 22 are leaves. A leaf cannot fan out, so the blast radius of a bad plan is bounded by the 9. |
| 无 shell 访问权的 AgentAgents with no shell access | 7 / 31 |
编排者与归纳者被刻意设计成"没有手"。tpu-coworker 与 maxkernel-loop 连一条命令都跑不了,只能路由和读状态。
Orchestrators and summarizers are deliberately handless. tpu-coworker and maxkernel-loop cannot run a command at all — only route and read state.
|
| Agent | 它回答的唯一问题The one question it answers | 终态产物Terminal artifact |
|---|---|---|
hello-tpu |
PyTorch 到底能不能摸到真硅片?Can PyTorch actually reach real silicon? | 设备标签 + CPU 交叉校验判决Device label + CPU cross-check verdict |
provenance |
这条 TPU 事实的可引用公开出处是什么?What is the citable public source for this TPU fact? | 缓存条目 + 唯一可打脚注的 URLCached entry + the single URL to footnote |
xprof-analyst |
时间花在哪了,以及工具看不见什么?Where did time go, and what could the tools not see? | 实测与推断严格分离的 Profile 报告Report separating measured from inferred |
maxkernel-loop |
这一个指名的 Pallas Kernel 还能更快吗?Can this one named Pallas kernel go faster? | iter5/optimized.py |
maxauto-round |
该攻哪一层,且这个收益经得起推敲吗?Which layer to attack, and does the win survive scrutiny? | 排序过的实测瓶颈分解 + 持久课题台账Ranked measured decomposition + durable ledger |
maxshard-orchestrator |
哪套切分 / 编译配置最快?Which sharding / compiler configuration is fastest? | 最优配方 + 每次尝试的完整轨迹Best recipe + full attempt trajectory |
torch-tpu-migration-orchestrator |
这个 HF 模型能上 TPU 吗,比 CPU 快多少?Does this HF model run on TPU, and how much faster than CPU? | results.json + benchmark_report.md |
两条契约约束着上述全部 Agent:一个 completed 结果必须携带至少一个 Artifact 描述符——"一段长到足以顶替产物的摘要"正是这条规则要阻止的失败;以及,最终响应若不是符合 Schema 的 JSON 信封,会在操作者看到之前就被运行时校验拒绝。
Two contracts govern all of them: a completed result must carry at least one artifact descriptor — "a summary long enough to stand in for the artifact" is the exact failure the rule exists to stop; and a final response that is not a schema-conforming JSON envelope is rejected by runtime validation before the operator ever sees it.
最常见的误用是抄错引擎。判别标准不是"问题有多难",而是"你已经知道什么"。 The commonest misuse is reaching for the wrong engine. The discriminator is not "how hard is the problem" but "what do you already know."
| 维度Dimension | maxkernel-loop |
maxshard-orchestrator |
maxauto-round |
|---|---|---|---|
| 前置条件Precondition | 已知是哪个 Pallas KernelYou know which Pallas kernel | 已知 Workload 与目标指标You know workload and target metric | 尚不知道瓶颈在哪一层You do not yet know the bottleneck layer |
| 它改什么What it changes | Kernel 源码Kernel source | 配置 / 编译器 FlagConfig / compiler flags | 暂不改——它产出诊断Nothing yet — it produces the diagnosis |
| 状态位置State location | Run 目录内部Inside the run directory | Run 目录内部Inside the run directory | 在 runs 之外,与之并列Beside runs, not inside one |
| 能否扛过 Session 结束Survives a session ending | ✗ | ✗ | ✓ 刻意设计by design |
| 统计严谨性Statistical rigour | 仅比较 optimized_timeCompares optimized_time only |
与当前最优比较Compares against current best | 预注册协议:噪声地板、重复次数、准入门禁;运行时会拒绝写入与协议矛盾的记录Pre-registered protocol: noise floor, replicate count, admissibility gates; the runtime refuses to write a record contradicting it |
| 停止条件Stop condition | iteration >= 5(硬编码) (hard-coded) |
3 次贪婪 → 3 次广搜 → 终止3 greedy → 3 broad-search → halt | max_rounds / plateau_rounds / target_improvement_pct |
maxauto-round 并不包裹 Kernel 循环,也不派发任何执行器。它唯一的委派目标是 xprof-analyst。
包内明确写出了理由——worker 是 MaxKernel 的 Kernel 迭代 Agent,其契约要求 Kernel 产物且禁止其他,
因此一个切分或调度类的 Arm 派过去会在运行前就校验失败。所以课题层今天交付的是一台测量与台账仪器;
通用执行器被推迟为独立变更,未来将原样读取同一份 Arm 记录。maxauto-round does not wrap the kernel loop and dispatches no executor. Its only delegation
target is xprof-analyst. The package states why — worker is MaxKernel's kernel-iteration agent, its contract
requires a kernel artifact and forbids anything else, so a sharding or scheduling arm sent there fails validation before
it runs. The study layer therefore ships as a measurement and ledger instrument; a generic executor is deferred to its
own change, reading the same arm record unmodified.
当 CUDA 锁定的代码被替换为原生 PyTorch,迁移只有在数字活下来时才算真的成立。
numerical-parity 以 Teacher Forcing 方式运行模型两次——在固定 Token 序列上做一次前向,绝不自由生成——
并比较三样东西:最终 Logits、每个被改写模块的 Forward Hook 激活值、以及逐位置 Top-1 一致性。
When CUDA-locked code is replaced with native PyTorch, the migration is only real if the numbers survive.
numerical-parity runs the model twice under teacher forcing — one forward pass over a frozen token sequence,
never free generation — comparing three things: final logits, per-module forward-hook activations for every
rewritten module, and per-position top-1 agreement.
| 阈值Threshold | 取值Value | 实测依据Empirical basis |
|---|---|---|
| 相对容差Relative tolerance | 5% | 在 mamba-130m 上标定:CPU bf16 与 TPU v6e bf16 运行完全相同的代码,layer 23 读数 2.92%、Logits 读数 3.09%——这是纯设备噪声。而注入的 10% 故障读数为 10.85%。 Calibrated on mamba-130m: CPU bf16 vs TPU v6e bf16 running identical code measured 2.92% at layer 23 and 3.09% on logits — pure device noise. The injected 10% fault read 10.85%. |
| 绝对容差Absolute tolerance | 默认关闭Off by default | 同一模型各层张量尺度跨越 8.1 至 341.9,单一绝对界没有意义。 Tensor scales ranged 8.1 to 341.9 across layers of one model; a single absolute bound is meaningless. |
| 平局阈值Tie threshold | max(2.0 × RMS, 1e-3) |
用 RMS 而非 max-abs,这样单个坏元素无法撑大本该抓住它的那个阈值。 RMS rather than max-abs, so one bad element cannot inflate the very threshold meant to catch it. |
| 文档化噪声地板Documented noise floor | ~3% | 错得比这还少的改写无法分辨。降低阈值只会制造误报——正确做法是扩大 Probe 集合。 A rewrite wrong by less than this is indistinguishable. Lowering the threshold only manufactures false alarms — the correct response is to widen the probe set. |
判决按固定顺序求值,而顺序本身是有意义的:若参考运行不独立(两侧执行了同一份改写后代码),结果在任何失败检查之前就被判定为 UNVERIFIED,因此一个非独立的参考永远不可能被报成通过。包内要求 UNVERIFIED 必须原词上报,绝不允许包装成成功。
代际标定声明:上述 5% 相对容差与 ~3% 噪声地板是在 TPU v6e(mamba-130m 对照 CPU bf16)上标定的先验常数,尚未在 TPU v7x (Ironwood) 上重新标定。
The verdict is evaluated in a fixed order, and the ordering matters: if the reference run was not independent (both sides ran the same rewritten code), the result is UNVERIFIED before any failure check runs, so a non-independent reference can never be reported as a pass. The package requires UNVERIFIED be reported in those words and never dressed up as success.
Generation calibration caveat: the 5% relative tolerance and ~3% noise floor were calibrated on TPU v6e (mamba-130m vs CPU bf16) and have not yet been re-derived on TPU v7x (Ironwood).
本节记录在真实 GKE TPU v7x 集群(2x2x1,8 芯片,jax 0.10.2)上的端到端物理验证。内部代码审计发现初版报告的 0.821 ms / 10.46 TFLOP/s 存在 P0 归因与形状缺陷;以下完整披露审计撤稿根因、Pallas Kernel 的 4 项编译器修复,以及修复后在真机硅片上横跨 3 个问题规模 × 3 种瓦片尺寸(128 / 256 / 512)的真实 A/B 实测。
This card records end-to-end physical validation on the live GKE TPU v7x cluster (2x2x1, 8 chips, jax 0.10.2). An internal code audit found that the initially reported 0.821 ms / 10.46 TFLOP/s suffered from P0 attribution and shape defects; below we disclose the full retraction ledger, the 4 compiler fixes required to make the Pallas kernel compile on Mosaic, and the real 3-scale × 3-tile-size (128 / 256 / 512) A/B benchmark on silicon.
| 缺陷等级 / 类别Defect class | 初版问题与编译器报错证据Observed defect & compiler diagnostic | 工程修复(已落地 pallas_grouped_gemm.py 与 pallas_ab_benchmark.py)Engineering fix applied |
|---|---|---|
| P0-1 归因错误 |
初版 tpu_grouped_gemm_benchmark.py 仅导入 torch 并调用 torch.matmul,从未加载 pallas_grouped_gemm.py,且末尾硬编码打印 "256x256 Native Alignment Confirmed"。初版 0.821 ms / 10.46 TFLOP/s 实测的是 PyTorch/XLA 算子,在此正式撤稿。
The original tpu_grouped_gemm_benchmark.py imported only torch and called torch.matmul, never loading pallas_grouped_gemm.py, and printed a hardcoded literal "256x256 Native Alignment Confirmed". The 0.821 ms / 10.46 TFLOP/s figure measured XLA matmul and is formally retracted.
|
新基准脚本通过 SHA-256 指纹(c498fe09…39fb)原样内联并调用真实 Pallas Kernel,每次迭代前通过密集 fp32 参考矩阵做门禁校验(cosine = 1.000000)。
New harness inlines the verbatim Pallas kernel with SHA-256 provenance (c498fe09…39fb) and gates every arm against a dense fp32 reference (cosine = 1.000000).
|
| P0-2 形状自相矛盾 |
初版形状 1024 tokens ÷ 8 experts = 128 tokens/expert,放入 256 行瓦片后恰好半空(50% 补零),与"消除补零开销"结论完全相反;且循环内仅同步 res[0,0].cpu(),存在 7/8 专家被 XLA 死代码消除(DCE)风险。
Original shape 1024 tokens ÷ 8 experts = 128 tokens/expert left every 256-row tile 50% empty (worsening padding), and synced only res[0,0].cpu(), leaving 7 of 8 experts exposed to XLA dead-code elimination (DCE).
|
将每专家 Token 数提升至 512 与 1024(同时整除 128 / 256 / 512,零补零),并对完整输出张量执行 jax.block_until_ready(out) 彻底阻断 DCE。
Raised tokens/expert to 512 and 1024 (exact multiples of 128 / 256 / 512, zero padding) and called jax.block_until_ready(out) on the full output tensor.
|
| Kernel 修复 × 4 |
原版 pallas_grouped_gemm.py 上机后暴露出 4 处编译阻塞:① 第 46 行对 expert_offsets_ref 追踪值使用 Python if 触发 TracerBoolConversionError;② in_specs 定义后未传入 pallas_call,且标量索引未放入 SMEM;③ 动态切片偏移未声明对齐,触发 Mosaic E2003 CompileTimeMosaicUnprovenMemoryAccessAlignment;④ 对 bf16 传入 Precision.HIGHEST 触发 Mosaic Bad lhs type。
Running the published pallas_grouped_gemm.py on silicon exposed 4 compiler blockers: ① Python if on traced expert_offsets_ref raised TracerBoolConversionError; ② in_specs were dead code never passed to pallas_call and scalar offsets lacked pltpu.SMEM; ③ dynamic slice offsets failed Mosaic E2003 (unproven 8-row alignment); ④ Precision.HIGHEST over bf16 failed Mosaic with Bad lhs type.
|
改为 @pl.when 谓词、pl.multiple_of(row, 8) / pl.multiple_of(col, 128) 静态对齐证明、移除 Precision.HIGHEST(保留 preferred_element_type=jnp.float32),并提供 VMEM 动态偏移(repaired)与 4D BlockSpec HBM 流式分块(blockspec)双实现。
Replaced with @pl.when, pl.multiple_of(row, 8) / pl.multiple_of(col, 128) static alignment proofs, removed Precision.HIGHEST (retaining preferred_element_type=jnp.float32), and shipped both VMEM dynamic-offset (repaired) and 4D BlockSpec HBM-streaming (blockspec) implementations.
|
| 规模配置(Shape)Config (Shape) | Kernel 变体Variant | 瓦片 bsTile bs |
中位延迟 (9 次)Median (n=9) | 有效吞吐Throughput | 单设备 MFU / 整芯片 MFUMFU/dev / MFU/chip | 相对 128 加速比(显著性)Speedup vs 128 (σ) |
|---|---|---|---|---|---|---|
small4 × 512 tok1024 × 1024(4.29 GFLOPs) |
as_published |
128 / 256 / 512 | FAILED · TracerBoolConversionError (line 46 Python if on traced ref) | |||
repaired(VMEM + SMEM) |
128×128 |
0.2049 ms (±0.011) | 20.96 TFLOP/s | 1.82% / 0.91% | 1.00× (baseline) | |
256×256 |
0.1593 ms (±0.005) | 26.96 TFLOP/s | 2.34% / 1.17% | 1.286× (3.7σ) | ||
512×512 |
0.1893 ms (±0.008) | 22.69 TFLOP/s | 1.97% / 0.98% | 1.082× | ||
blockspec(4D HBM stream) |
128×128 |
0.4094 ms (±0.006) | 10.49 TFLOP/s | 0.91% / 0.45% | 1.00× (baseline) | |
256×256 |
0.1944 ms (±0.008) | 22.09 TFLOP/s | 1.92% / 0.96% | 2.106× (21.3σ) | ||
512×512 |
0.1456 ms (±0.005) | 29.50 TFLOP/s | 2.56% / 1.28% | 2.812× vs 128 · 1.335× vs 256 | ||
medium8 × 512 tok2048 × 2048(34.36 GFLOPs, 白皮书同维) |
repaired |
128 / 256 / 512 | VMEM OOM (88 MiB 全张量驻留超出 VMEM 容量,证明实产规模必须采用 4D BlockSpec 流式分块) | |||
blockspec(4D HBM stream) |
128×128 |
4.2929 ms (±0.010) | 8.00 TFLOP/s | 0.69% / 0.35% | 1.00× (baseline) | |
256×256 |
0.8164 ms (±0.006) | 42.09 TFLOP/s | 3.65% / 1.82% | 5.258× (296.0σ) | ||
512×512 |
0.2697 ms (±0.004) | 127.40 TFLOP/s | 11.04% / 5.52% | 15.92× vs 128 · 3.03× vs 256 | ||
large8 × 1024 tok4096 × 4096(274.88 GFLOPs) |
repaired |
128 / 256 / 512 | VMEM OOM (384 MiB 全张量驻留超出 VMEM 容量) | |||
blockspec(4D HBM stream) |
128×128 |
34.6004 ms (±0.255) | 7.94 TFLOP/s | 0.69% / 0.34% | 1.00× (baseline) | |
256×256 |
5.3650 ms (±0.007) | 51.24 TFLOP/s | 4.44% / 2.22% | 6.449× (115.2σ) | ||
512×512 |
1.1428 ms (±0.006) | 240.53 TFLOP/s | 20.85% / 10.43% | 30.28× vs 128 · 4.69× vs 256 | ||
核心发现与知识库交叉印证(引用 knowledgebase/wiki/hardware/tpu-7x.md 与 concepts/mxu.md):
① 为什么 512×512 比 256×256 还要快 3.03×(medium)至 4.69×(large)?
concepts/mxu.md 明确指出:256×256 脉动阵列目前仅在 TPU v6e 上有正式公开信源支持(延伸至更新代际在知识库中标注为 "human-reviewed assertion — not yet stated by an ingested source",且 index.md 写作 128×128 (256×256 v6e));更重要的是,同一页给出了有效瓦片尺寸定律——有效瓦片维度必须超过 array-size × #MXUs。当存在 2 个 MXU 时,门槛是 512 而非 256。实测完美印证了该定律:在 large 规模下,256×256 相比 128×128 带来 6.45× 提速(51.24 TFLOP/s),而跨过 512×512 门槛后再翻 4.69×,单 Chiplet 达到 240.53 TFLOP/s(相对 fp32 参考余弦相似度 1.000000,最大绝对误差 0.0)。
② 单设备(Chiplet)vs 整芯片(Chip)峰值口径:
根据 hardware/tpu-7x.md:20,22,TPU7x 整芯片 bf16 峰值为 2307 TFLOP/s,由 2 个拥有独立地址空间、无 Megacore 的 Chiplet(Device)组成,因此单个 JAX Device(单 Chiplet)bf16 峰值为 1153.5 TFLOP/s。上表全部 Kernel 均在单 Device 上运行,240.53 TFLOP/s 对应单 Chiplet 20.85% MFU(整芯片 10.43% MFU)。此外,hello-tpu 在同集群通过真机门禁(device_type=native-tpu,max_abs_diff=0.111501 < 0.5);早期 6.48e-05 误差数据则属于宿主机 CPU 上的 NumPy 算术仿真,在此与真机数据严格区分。
Key findings & knowledge-base cross-verification (citing knowledgebase/wiki/hardware/tpu-7x.md & concepts/mxu.md):
① Why does 512×512 beat 256×256 by 3.03× (medium) to 4.69× (large)?
concepts/mxu.md notes that a 256×256 systolic array is sourced only for TPU v6e (extension to later generations is marked "human-reviewed assertion — not yet stated by an ingested source", and index.md lists 128×128 (256×256 v6e)). More importantly, the same page states the effective-tiling rule: effective tiling dimensions must exceed array-size × #MXUs. With 2 MXUs, that threshold is 512, not 256. Silicon measurements confirm this rule: at the large scale, moving from 128×128 to 256×256 yields 6.45× (51.24 TFLOP/s), while crossing to 512×512 yields an additional 4.69× jump (30.28× over 128), reaching 240.53 TFLOP/s (cosine 1.000000, max_abs_diff 0.0 vs dense fp32 reference) on a single chiplet.
② Single-device (chiplet) vs full-chip peak caliber:
Per hardware/tpu-7x.md:20,22, a TPU7x chip delivers 2307 TFLOP/s bf16 across two chiplets with separate memory spaces and no megacore, making each addressable JAX device (chiplet) 1153.5 TFLOP/s bf16. All kernels above ran on 1 device, so 240.53 TFLOP/s is 20.85% MFU per device (10.43% per chip). Separately, hello-tpu passed on silicon (device_type=native-tpu, max_abs_diff=0.111501 < 0.5), whereas the earlier 6.48e-05 check was a host-CPU NumPy tiling simulation and is kept strictly separate from device measurements.
| 发现的基础设施与运行时故障(共 3 项)Infrastructure & runtime faults found (3 total) | 根因Root cause | 修复Fix |
|---|---|---|
| Job 已创建但永久挂起Job created but permanently suspended |
当容器只声明 resources.limits 而不声明 resources.requests 时,Kueue Webhook 无法解析配额请求,不会创建 Workload CRD,Job 因此无限等待。
When a container declares resources.limits without resources.requests, the Kueue webhook cannot resolve the quota request, no Workload CRD is created, and the Job waits forever.
|
为 google.com/tpu 同时下发 requests 与 limits。Emit both requests and limits for google.com/tpu. |
| Job 在准入阶段被拒Job rejected at admission |
无条件注入了 kueue.x-k8s.io/priority-class: medium 标签,而集群中并不存在同名的 WorkloadPriorityClass。
An unconditional kueue.x-k8s.io/priority-class: medium label was injected, but no WorkloadPriorityClass of that name exists in the cluster.
|
仅在显式配置且可解析的优先级存在时才附加该标签。Attach the label only when an explicit, resolvable priority is configured. |
| 首次派发在容器内异常退出First dispatch aborted in-container |
脚本调用了不存在的 torch_tpu.core.sync()(报错 AttributeError: module 'torch_tpu' has no attribute 'core')。
Script called non-existent torch_tpu.core.sync() (AttributeError: module 'torch_tpu' has no attribute 'core').
|
PyTorch 侧通过 tensor.cpu() 往返同步;JAX/Pallas 侧对完整输出调用 jax.block_until_ready(out)。Synchronize via host round-trip (tensor.cpu()) in PyTorch or jax.block_until_ready(out) on the full output in JAX/Pallas. |
前两处故障位于派发驱动(gke_job.py)而非负载本身,且对用户呈现完全相同的静默挂起现象(注:该修复目前保存在本地已安装包中,重装包会丢失,需向上游提交补丁)。
The first two faults sat in the dispatch driver (gke_job.py) rather than the workload and presented identically as a silent hang (note: these fixes are local to the installed package and will be lost on reinstall until upstreamed).
通过将 Agent 定义与其自身 JSON Schema 交叉比对,发现 7 处内部矛盾。此处原样记录,不作裁决。 Seven internal contradictions were found by reading the agent definitions against their own JSON schemas. Recorded as observed, not adjudicated.
| # | 矛盾Contradiction | 运行影响Operational impact |
|---|---|---|
| 1 | 两份请求 Schema 的 required 中均无 run_id,但两个 Agent 契约都写着 "Require the request's run_id"。run_id is absent from required in both request schemas, yet both agent contracts state "Require the request's run_id". |
约束存在于散文,而非校验。Enforcement lives in prose, not validation. |
| 2 | worker-request.json 要求 iteration 与 base_kernel_artifact;散文却说唯一给出的参数是 run_id。worker-request.json requires iteration and base_kernel_artifact; the prose says the only parameter given is run_id. |
Schema 与散文对"真相来源"意见不一。Schema and prose disagree on the source of truth. |
| 3 | worker 被指示返回 invalid_input,但其结果 Schema 只允许 success / failed / completed。worker is told to return invalid_input, but its result schema permits only success / failed / completed. |
遵照指示会产出被自己 Schema 拒绝的响应。Following the instruction produces a response its own schema rejects. |
| 4 | max_iterations 是状态字段,但停止检查在散文中硬编码为 5。max_iterations is a state field, but the stop check hard-codes 5 in prose. |
修改该字段会静默造成循环失同步。Editing the field silently desynchronizes the loop. |
| 5 | preflight-check 文档称 --allow_gpu_imports 默认开启;脚本定义的默认值是 False。preflight-check docs say --allow_gpu_imports is on by default; the script defines it as False. |
省略该 Flag 的调用方会得到比文档更严格的行为。Callers omitting the flag get stricter behaviour than documented. |
| 6 | numerical-parity 仅在服务化路径可达;基准测试编排者无权委派给它。numerical-parity is reachable only from the serving path; the benchmark orchestrator cannot delegate to it. |
经迁移路径做基准的模型得不到任何逐模块数值验证——而第 4 卡恰好证明这是唯一能抓住单层故障的检查。Models benchmarked through the migration path receive no per-module numerical verification — precisely the check card 4 shows is the only one that catches a single-layer fault. |
| 7 | 迁移编排者文档中,本该放搜索工具构建指令的位置是一个空代码块。The migration orchestrator contains an empty code block where search-tool build instructions belong. | 一个已文档化的步骤没有命令。A documented step has no command. |
第 6 项是唯一具备生产后果的:基准路径可能为一个改写层已悄然出错的模型,报告出一份干净的加速比。 Item 6 is the one with production consequence: the benchmark path can report a clean speedup on a model whose rewritten layer is quietly wrong.
tpu-agent-suite-experimental@0.7.0 升级实证:Qwen3.5-397B-A17B-FP8 与 GLM-5.2-FP8 算子级 Roofline、140 方案切分漏斗与 2-Layer Twin 微基准
7 · tpu-agent-suite-experimental@0.7.0 Live Verification: Qwen3.5-397B-A17B-FP8 & GLM-5.2-FP8 Operator Roofline, 140-Way Sharding Funnel & 2-Layer Twin Microbenchmark
基于升级至 tpu-agent-suite-experimental@0.7.0 的三大确定性分析引擎(roofline-analysis、sharding-strategy、workload-microbenchmark,已通过全部 5 项黄金基准自检),以下展示针对开源旗舰 Qwen/Qwen3.5-397B-A17B-FP8(单机 2×2×1 = 4 Chips / 8 Devices)与超大规模 MLA+DSA 架构 GLM-5.2-FP8(双机 2×2×2 = 8 Chips / 16 Devices)的完整逐 Device 硬件建模与切分策略推演。所有硬件指标严格遵循 TPU v7x (Ironwood) 1 Chip = 2 Devices (Dies/Cores,每 Die 96 GiB HBM) 的物理对齐基准(TPU v7x 官方文档)。
Powered by the three deterministic analysis engines in tpu-agent-suite-experimental@0.7.0 (roofline-analysis, sharding-strategy, and workload-microbenchmark, passing all 5 golden selftests), below is the complete per-device roofline and sharding derivation for Qwen/Qwen3.5-397B-A17B-FP8 (single-host 2×2×1 = 4 chips / 8 devices) and GLM-5.2-FP8 (dual-host 2×2×2 = 8 chips / 16 devices). All figures strictly align by device count (1 TPU v7x chip = 2 devices/dies, 96 GiB HBM per device; TPU v7x docs).
Qwen/Qwen3.5-397B-A17B-FP8 单机 2×2×1(4 Chips / 8 Devices)四路切分推导与三大反直觉硬件规律(基于真实 AgentX 任务 j-9fb52480)
7.1 Qwen/Qwen3.5-397B-A17B-FP8 Single-Host 2×2×1 (4 Chips / 8 Devices) 4-Way Sharding Derivation & Three Hardware Laws (AgentX Job j-9fb52480)
| 逐 Device (Per-Rank) 核心指标(整机 4 Chips = 8 Devices)Per-Device (Per-Rank) Metric (4 Chips = 8 Devices Total) | tp8-dp1-ep(原始任务配置)(Baseline Job) |
tp4-dp2-ep |
tp2-dp4-ep(单 Chip 内 TP2)(Intra-Chip TP2) |
tp1-dp8-ep(Golden 黄金布局)(Golden Layout) |
|---|---|---|---|---|
每 Engine 占用 Device 数 (TP) × 独立引擎数 (DP)Devices per Engine (TP) × Independent Engines (DP) |
8 dev (4 chips) × 1 | 4 dev (2 chips) × 2 | 2 dev (1 chip) × 4 | 1 dev (0.5 chip) × 8 |
| 每 Device 权重常驻内存 (Weights / Device)Resident Weights per Device | 47.41 GiB | 48.78 GiB | 51.51 GiB | 57.01 GiB |
每 Device 路由专家数(512 专家 Top-10,EP=8)Routed Experts per Device (512 Top-10, EP=8) |
64 (0.805 GiB/layer) | 64 (0.805 GiB/layer) | 64 (0.805 GiB/layer) | 64 (0.805 GiB/layer) |
| 每 Device GQA Q 头 / KV 头(开源权重 $H_q=32, H_{kv}=2, D_h=256$)GQA Q / KV Heads per Device ($H_q=32, H_{kv}=2, D_h=256$) | 4 Q / 1 KV 4× KV 冗余 | 8 Q / 1 KV 2× KV 冗余 | 16 Q / 1 KV 零冗余 | 32 Q / 2 KV 零冗余 |
| 每 Token 每 Device 15 层 GQA KV 占用(FP8)15-Layer GQA KV Footprint per Token per Device (FP8) | 7.5 KiB (7,680 B) | 7.5 KiB (7,680 B) | 7.5 KiB (7,680 B) | 15.0 KiB (15,360 B) |
每条序列每 Device 45 层 GDN 固定状态(fp32 Recurrent + bf16 Conv)45-Layer GDN Fixed State per Sequence per Device (fp32 + bf16) |
24.42 MB / seq | 48.84 MB / seq | 97.69 MB / seq | 195.38 MB / seq (整模型) |
2-Layer Twin(1 GDN + 1 GQA)压测显存映射(Weights → Twin)2-Layer Twin (1 GDN + 1 GQA) Memory Mapping |
47.41 → 2.66 GiB (Ballast 至 72.37 GiB) |
48.78 → 3.33 GiB (Ballast 至 74.43 GiB) |
51.51 → 4.36 GiB (Ballast 至 76.49 GiB) |
57.01 → 6.42 GiB (需下调 KV 池预算以装入 96 GiB) |
Qwen/Qwen3.5-397B-A17B-FP8)
💡 Three Architectural Laws Derived from Source & Roofline (Qwen/Qwen3.5-397B-A17B-FP8)
qkv_proj 膨胀 1.89×:在 Qwen3NextAttention 中,因 attn_output_gate = true($A_{\text{gate}}=2$),qkv_proj 形状为 [4096, 17408](Q: 8192 | Gate: 8192 | K: 512 | V: 512)而非标准 GQA 的 [4096, 9216],仅 Output Gate 分支即消耗该 GEMM 47.1% 的算力与带宽。192.06 MB/seq 的 $O(1)$ GDN 状态决定,而非 KV Cache:45 层 GDN 的 Recurrent State [B, 64, 128, 128] 必须以 float32 保存(188.74 MB/seq),外加 bfloat16 Conv1D 状态(3.32 MB/seq)。在上下文长度 $S < 12,500$ tokens 时,每条序列固定的 192 MB GDN 状态大于 15 层 GQA 的 FP8 KV Cache(15.0 KiB/token)。此外,因 $H_{kv}=2$,任何 $\text{TP} > 2$ 的配置都会触发 KV Head 跨 Rank 冗余复制(tp8-dp1 下冗余 4 倍)。EP=8 下,算术强度公式简化为 $\text{AI} = \frac{2 N_{\text{local}} K \cdot \text{EP}}{E} = 0.3125 \times N_{\text{local}}\text{ FLOP/B}$(与 $D=4096, F=1024$ 无关)。对照 TPU v7x 单 Device P3 derated 平衡点(FP8 1,785.6 TFLOPS ÷ HBM 3,364.9 GB/s = 530.7 FLOP/B),单 Device 必须同时处理 1,698 个 local tokens 才能摆脱 HBM 带宽受限(每层固定权重读取地板 0.239 ms)。在 AgentX 真实任务(1,437 请求,MTP×3 实测平均接受长度 accept_len = 3.389 tokens/step)中,微基准直接提取出 3 个 Prefill 单元(P1/P2/P3: 16,376 chunk @ 38K/82K/138K ctx)与 3 个 Decode 单元(D1/D2/D3: 1~3 rows × 4 tokens @ 63K~185K ctx),通过 2-Layer Twin 在单机 8 Devices 上秒级复现单层耗时。
① Fused Output Gate inflates 15-layer GQA qkv_proj by 1.89×: Because attn_output_gate = true ($A_{\text{gate}}=2$) in Qwen3NextAttention, qkv_proj has shape [4096, 17408] (Q: 8192 | Gate: 8192 | K: 512 | V: 512) rather than [4096, 9216]; the gate alone accounts for 47.1% of this GEMM.192.06 MB/seq $O(1)$ GDN state, not KV cache: The 45 GDN layers hold their recurrent state [B, 64, 128, 128] in float32 (188.74 MB/seq) plus bfloat16 conv state (3.32 MB/seq). Below $S \approx 12,500$ tokens, this fixed 192 MB/seq state exceeds the 15-layer FP8 GQA KV cache (15.0 KiB/token). Moreover, because $H_{kv}=2$, any $\text{TP} > 2$ replicates KV heads across TP ranks (4× replication at tp8-dp1).EP=8, arithmetic intensity simplifies to $\text{AI} = \frac{2 N_{\text{local}} K \cdot \text{EP}}{E} = 0.3125 \times N_{\text{local}}\text{ FLOP/B}$. Against the TPU v7x per-device P3 derated balance point (530.7 FLOP/B), a device needs 1,698 local tokens before exiting the memory-bound regime (fixed HBM floor of 0.239 ms/layer for 805 MB of local expert weights). From the live AgentX trace (1,437 requests, measured MTP×3 accept_len = 3.389 tokens/step), workload-microbenchmark distills 3 prefill cells (P1/P2/P3) and 3 decode cells (D1/D2/D3) to drive a 2-layer Twin on 8 devices.
GLM-5.2-FP8(692.33 GiB,78 层 MLA + 21 层 DSA Indexer + 256 专家)双机 2×2×2(8 Chips / 16 Devices)140 方案切分漏斗与吞吐对决(ISL=8192 / OSL=1024, C=256)
7.2 GLM-5.2-FP8 (692.33 GiB, 78-Layer MLA + 21-Layer DSA Indexer + 256 Experts) Dual-Host 2×2×2 (8 Chips / 16 Devices) 140-Way Sharding Funnel & Benchmark (ISL=8192 / OSL=1024, C=256)
漏斗筛选:140 种全枚举 ➔ 124 种数学自洽(剔除 16 种非法 DCP 整除)➔ 91 种装入单 Device 88.3 GiB HBM 预算(剔除 33 种 OOM,如不开 EP 的 tp1-dp16 需 692.3 GiB/dev)➔ 22 种当前直接可跑(Shippable) + 69 种前瞻投射(Projections)。
Funnel: 140 enumerated ➔ 124 mathematically coherent ➔ 91 fit within the 88.3 GiB usable per-device HBM budget (33 OOM excluded) ➔ 22 shippable today + 69 forward-looking projections.
| # | 切分方案(16 Devices / 8 Chips)Sharding (16 Devices / 8 Chips) | 类别 / 状态Class / Status | 整机输出吞吐 (out tok/s)Fleet Out (tok/s) |
整机总吞吐 (in+out tok/s)Fleet Total (tok/s) |
每 Device / 每 Chip 总吞吐Per-Device / Per-Chip Total | TTFT (ms) |
TPOT (ms) |
每 Device 权重 / KV 池 (可容请求)Weights / KV Pool per Device (Cap) | 主导瓶颈Bound |
|---|---|---|---|---|---|---|---|---|---|
| 1 | tp1-dp16-ep |
DP · Shippable (冠军) | 5,643 (1.00×) | 50,789 | 3,174 /dev · 6,349 /chip | 1,496.0 | 21.99 | 59.51 GiB / 24.81 GiB (60.6 req × 16 = 969) | HBM (ICI 11%) |
| 2 | tp1-dp8-pp2-ep |
DP×PP · Projection | 4,528 (0.80×) | 40,748 | 2,547 /dev · 5,094 /chip | 724.2 | 36.35 | 49.33 GiB / 34.99 GiB (171 req × 8) | HBM |
| 3 | tp1-dp8-pcp2-ep |
DP×PCP · Projection | 4,181 (0.74×) | 37,629 | 2,352 /dev · 4,704 /chip | 784.1 | 36.73 | 59.51 GiB / 24.81 GiB (121 req × 8) | HBM |
| 4 | tp2-dp8-ep |
TP×DP · Shippable (亚军) | 3,913 (0.69×) | 35,217 | 2,201 /dev · 4,402 /chip | 1,360.5 | 22.91 | 51.63 GiB / 32.69 GiB (80.0 req × 8 = 640) | HBM (ICI 21%) |
| 8 | tp2-dp8-dcp2-ep |
TP×DP · Shippable | 3,010 (0.53×) | 27,091 | 1,693 /dev · 3,386 /chip | 1,893.3 | 25.88 | 51.63 GiB / 32.69 GiB (159.9 req × 8) | HBM (ICI 32%) |
| 13 | tp4-dp4-ep |
TP×DP · Shippable | 2,429 (0.43×) | 21,862 | 1,366 /dev · 2,733 /chip | 1,259.6 | 26.66 | 47.69 GiB / 36.63 GiB (89.6 req × 4) | HBM (ICI 30%) |
| 46 | tp16-dp1 |
Pure TP · Shippable | 1,577 (0.28×) | 14,190 | 887 /dev · 1,774 /chip | 103,748.9 | 22.05 | 44.65 GiB / 39.67 GiB (仅 96.9 req 整机!) | HBM (KV 排队 103s) |
| 81 | tp1-dp1-pp16 |
Pure PP · Shippable | 920 (0.16×) | 8,284 | 518 /dev · 1,036 /chip | 424.7 | 239.80 | 45.97 GiB / 38.35 GiB (1,421 req) | HBM (16 级流水延迟) |
核心物理洞察:① 为什么 tp1-dp16-ep 比 tp2-dp8-ep 快 1.44×、比纯 TP tp16-dp1 快 3.58×? 因为 MLA 的 KV Latent(c_kv + k_rope = 576 B/层,加上 21 层 DSA Indexer 132 B/层,全模型 46.58 KiB/token)在所有 Query Head 间共享,没有 Head 轴可供 TP 切分。在 tp16-dp1 下,16 个 Device 100% 重复存储完全相同的 KV Cache,整机最多只装得下 96.9 个请求(其余 159 个请求在队列外苦等 103.3 秒);而 tp1-dp16-ep 让 16 个 Device 组成 16 个独立 DP 引擎(通过 EP=16 跨全切片均摊 675 GiB 路由专家权重至 42.19 GiB/dev),整机并发容量瞬间扩大 16 倍至 969 个请求,实现零排队!② 当上下文拉长至 256K/1K 时:21 层 DSA Indexer: QK 跃升为 Prefill 第三大耗时算子(占 12.01%),且因 [T=1024, S=262144] 分数矩阵超出 TPU v7x 单核 8 MiB VMEM 限制触发 HBM Spill(额外消耗 871 ms HBM 带宽时间),这正是 Pallas STRAT-MLA-* 与 STRAT-SC-* 内核优化的主战场。
Core physical insight: ① Why does tp1-dp16-ep beat tp2-dp8-ep by 1.44× and pure TP tp16-dp1 by 3.58×? Because MLA's KV latent (c_kv + k_rope = 576 B/layer plus 21-layer DSA indexer 132 B/layer = 46.58 KiB/token) is shared across all query heads and has no head axis for TP to split. Under tp16-dp1, all 16 devices replicate identical KV caches, capping the entire 16-device slice at 96.9 requests (forcing 159 requests to queue for 103.3 s); whereas tp1-dp16-ep runs 16 independent DP engines while pooling the 675 GiB of routed experts across EP=16 (42.19 GiB/device), expanding fleet KV capacity to 969 requests with zero queueing. ② At 256K/1K ultra-long context: the 21-layer DSA Indexer: QK becomes the #3 prefill bottleneck (12.01% of prefill time) and spills its [T=1024, S=262144] score matrix out of the 8 MiB VMEM onto HBM (871 ms extra HBM traffic), pinpointing the exact target for Pallas STRAT-MLA-* and STRAT-SC-* kernels.
Qwen3.5-397B-A17B-FP8 跨切片 Roofline 切分扫描(2×2×1 [8 Devices] vs 2×2×2 [16 Devices])与 2026-10-23 千机灰度实测闭环
7.3 Qwen3.5-397B-A17B-FP8 Cross-Slice Roofline Sharding Sweep (2×2×1 [8 Devices] vs 2×2×2 [16 Devices]) & 2026-10-23 1,000-Host Grayscale Validation
| 切片 (Chips / Devices)Slice (Chips / Devs) | 切分方案Sharding | 每 Device 权重Weights / Dev | 每序列每 Device 状态 (GDN + GQA KV @ 9K)State / Seq / Dev (GDN + GQA KV @ 9K) |
整切片最大并发序列Max Fleet Seqs | Prefill TTFT (8K, ms)Prefill TTFT(8K, ms) |
Decode Step T_decode( 45 GDN + 15 GQA + 60 MoE)Decode Step T_decode( 45 GDN + 15 GQA + 60 MoE) |
整切片输出吞吐 ( out tok/s)Fleet Out( tok/s) |
每 Device 输出吞吐 ( out tok/s/dev)Per-Device Out( tok/s/dev) |
|---|---|---|---|---|---|---|---|---|
2×2×1 (4C / 8D) |
TP1×DP8 (EP8) |
56.16 GiB | 192.06 + 141.56 MB | 82 × 8 = 656 | 200.8 | 23.57 ms (5.24+1.74+16.59) | 10,771.2 | 1,346.4 /dev |
2×2×1 (4C / 8D) |
TP2×DP4 (EP8) 生产 Decode 最优 |
50.70 GiB | 96.03 + 70.78 MB | 200 × 4 = 800 | 149.9 | 22.72 ms (4.57+1.55+16.61) | 11,194.0 | 1,399.3 /dev |
2×2×1 (4C / 8D) |
TP4×DP2 (EP8) |
47.98 GiB | 48.02 + 70.78 MB 2×KV | 306 × 2 = 612 | 157.1 | 24.59 ms (4.84+2.93+16.82) | 10,346.1 | 1,293.3 /dev |
2×2×1 (4C / 8D) |
TP8×DP1 (EP8) |
46.62 GiB | 24.01 + 70.78 MB 4×KV | 399 × 1 = 399 | 153.5 | 28.50 ms (5.49+5.70+17.31) | 8,934.8 | 1,116.8 /dev |
2×2×2 (8C / 16D) |
TP1×DP16 (EP16) |
33.66 GiB | 192.06 + 141.56 MB | 155 × 16 = 2,480 | 209.7 | 15.59 ms (3.41+1.10+11.08) | 16,203.9 | 1,012.7 /dev |
2×2×2 (8C / 16D) |
TP2×DP8 (EP16) 16D 冠军 |
28.20 GiB | 96.03 + 70.78 MB | 345 × 8 = 2,760 | 135.1 | 14.66 ms (2.72+0.90+11.04) | 17,308.3 | 1,081.8 /dev |
2×2×2 (8C / 16D) |
TP4×DP4 (EP16) |
25.48 GiB | 48.02 + 70.78 MB 2×KV | 509 × 4 = 2,036 | 137.2 | 15.53 ms (2.82+1.59+11.12) | 16,345.0 | 1,021.6 /dev |
2026-10-23 生产千机灰度实测与 Roofline 理论的 4 大硬核印证
🔥 Four Production Validations from the 2026-10-23 1,000-Host Grayscale Rollout vs. Roofline Theory
TP2×DP4 (EP8) 成为生产 Decode 黄金标准(Roofline 吻合度 94.3%):Roofline 扫描显示,在单机 8 Devices 上,TP2×DP4 (EP8) 恰好卡在 $H_{kv}=2$ 的零冗余分界线上,将每序列状态(GDN+KV)减半至 166.8 MB、整机并发容量推高至 800 序列,且因 TP=2 走同颗 TPU v7x Chip 内部 600 GB/s D2D 互联,理论 T_decode = 22.72 ms。2026-10-23 生产与主干 CI(PR #2006)正式将 1P1D 标准配置从 PCP8 -> DP8 切换为 PCP8 -> DP4×TP2,5 分钟压测实测 TPOT 中位数 24.10 ms、吞吐 1,083.9 tok/s,与 Roofline 理论预测误差仅 5.7%!2×DP8 (Proxy) 比跨机 DP16 (Ray) 快 +35.3%?Roofline 表揭示:当从单机 2×2×1 (EP8) 扩至跨机 2×2×2 (EP16) 时,虽然总吞吐上升,但受跨机 ICI Z 轴 All-Gather/Reduce-Scatter 拖累,每 Device 输出吞吐从 1,399.3 tok/s/dev 降至 1,081.8 tok/s/dev(-22.7%)。真机 1P2D 实测(32K/64K C=4)完全印证:采用 Dual-Decoder Proxy 编排两个独立单机 DP8+EP8(PCP8 -> 2×DP8),比跨双机单 Ray 集群(PCP8 -> DP16)TTFT 降低 40.4%–42.5%、TPOT 提速 +32.8%–35.2%(28.44 ms vs 37.67 ms)、总输出吞吐高出 +35.3%!CoreHalt 发生率暴降 15×–30×:合入 DP>1 + EP + MTP 跨 Rank 步序对齐与移除 SMEM chunk 拆分上限(PR #1983 / 9f0891a)后,生产环境已进入灰度流量,CoreHalt 频率从 200 台物理机每天 1–2 次 骤降至 1,000 台物理机平均 3 天 1 次。30~48 GiB):① jemalloc + Linux THP 陷阱(+30 GiB/Host):jemalloc 默认 retain:true 保留虚拟地址,触发内核 khugepaged 将已释放的 2 MiB 区间重新合并回物理大页(Prefill/Decode Anon 暴涨至 129.8 / 128.2 GiB vs tcmalloc 的 98.7 / 98.7 GiB);保持默认 tcmalloc 或设置 MALLOC_CONF="thp:never,retain:false,dirty_decay_ms:1000,background_thread:true" 即可消除 ~30 GiB 虚耗且吞吐零损(1,083.9 tok/s);② vLLM v0.30 多模态 Processor 泄漏(vllm#62880):即使开启 --language-model-only,Decode Engine Core 仍加载 MM processor 多耗 ~3.4 GB/Host;③ Raiden Staging Pool 跨 Worker 池化:针对 8-Worker Decode 节点 12 GiB × 8 = 96 GiB Staging 内存,推进跨 DP Group 共享 48 GiB 内存池与单次 PD KVT 多批传输。
1. TP2×DP4 (EP8) validated as the production Decode sweet spot (94.3% Roofline accuracy): On a single 8-device host, TP2×DP4 (EP8) sits right at the $H_{kv}=2$ zero-replication boundary, halving per-sequence state to 166.8 MB, expanding fleet capacity to 800 sequences, and keeping TP=2 inside each chip's 600 GB/s D2D link for a theoretical T_decode = 22.72 ms. On 2026-10-23, both production and main CI (PR #2006) aligned 1P1D to PCP8 -> DP4×TP2, measuring 24.10 ms median TPOT and 1,083.9 tok/s throughput.2×DP8 (Proxy) beats cross-host DP16 (Ray) by +35.3%: Our Roofline table shows that scaling from single-host 2×2×1 (EP8) to dual-host 2×2×2 (EP16) reduces per-device throughput from 1,399.3 to 1,081.8 tok/s/dev (-22.7%) due to cross-host ICI Z-link collectives. Live 1P2D benchmarks at 32K/64K confirm that orchestrating two independent single-host DP8+EP8 nodes via a proxy (PCP8 -> 2×DP8) delivers 40.4%–42.5% lower TTFT and +35.3% higher decode throughput than a single cross-host DP16 Ray cluster.CoreHalt frequency reduced by 15×–30× across 1,000 grayscale hosts: After landing PR #1983 (9f0891a, fixing DP>1 + EP + MTP step-order mismatch), production entered grayscale traffic with CoreHalt dropping from 1–2/day on 200 hosts to 1 per 3 days across 1,000 hosts.30–48 GiB/host): ① jemalloc + Linux THP re-merging (+30 GiB/host): jemalloc default retain:true interacts with khugepaged to re-materialize freed 2 MiB ranges into physical huge pages (129.8 / 128.2 GiB vs tcmalloc's 98.7 / 98.7 GiB); using default tcmalloc or setting MALLOC_CONF="thp:never,retain:false,dirty_decay_ms:1000,background_thread:true" eliminates the 30 GiB bloat with identical throughput (1,083.9 tok/s); ② vLLM v0.30 MM processor leak (vllm#62880, +3.4 GB/host) even under --language-model-only; ③ Raiden Staging Pool sharing to cut 12 GiB × 8 = 96 GiB in half.
2026-10-22 ➔ 10-30 增量沉淀:冷启动内存、1M 上下文、NaN 隔离、PP 流水与万卡级上线教训
7.5 2026-10-22 ➔ 10-30 Increments: Cold-Start DRAM, 1M Context, NaN Isolation, PP Pipelining & Large-Fleet Launch Lessons
~18 GiB/host Host DRAM 的三个根因:397B Decode 单机(4 Chips = 8 Devices,DP4·TP2·EP8·MTP3)冷启动 168.25 GiB/host vs 热启动 ~150 GiB/host;分阶段探针显示差距 100% 出现在 compile_or_warm_up_model(),权重加载与 KV 初始化贡献 <0.2 GiB。
A:CompileAndLoad 产出的 executable 保留编译期专用元数据(CompilerMetadata、未去重的 BackendConfigWrapper、MosaicMlirCacheEntry、NameUniquer),而 LoadSerializedExecutable 加载的没有;
B:c10d group_name 是 rank 本地字符串,使 graph fingerprint 跨 rank 不一致,follower rank 无法复用 peer 产物,8 个 worker 各自做完整 FX→MLIR→Pallas lowering;
C:gc.freeze() 把前端缓存与 PiecewiseBackend.graph 冻结在 Python 堆中。
修复:编译后序列化→卸载→LoadSerializedExecutable 重载;规范化 group_name 并开启 TPU_PRECOMPILE_PEER_REPLAY=1(96 次编译中 84 次直接 replay);freeze_gc_heap() 前清缓存。三项合计消除 90.5%–93.9% 的冷热差距。170K,但 staging pool 按 1M 最坏情况常驻,而容器 RSS 预算 <200 GiB/host 还要留给 KV offload。新增 TPU_RAIDEN_POOL_STAGING_BUDGET_MB / TPU_RAIDEN_POOL_STAGING_WINDOW_BLOCKS:Decode 端只开 W 个 Full-Attention 页的窗口,长请求拆成按页对齐的子传输链,GDN 状态块放在最后一段。
Qwen3.5-397B(1M,PCP8·TP1 ➔ DP4·TP2):每 Rank staging 7.63 → 1.81 GiB,1M token 分 10 段 780 ms 传完(20.9 GB/s);一个 92 层、1M 上下文的更大模型上:93.2 → 2.71 GiB/host(61×),每多一段子传输仅增加 ~2.3–3.1 ms。vllm-torchtpu#2614,10-30 合入 main):原先 page_indices 全量驻留 SMEM,192 seqs × (1M / 256 页) × 4 B = 3 MiB 远超 1 MiB SMEM,只能把 max-num-seqs 压到 <64。现在页表查找移到 scheduler:以双缓冲、lane 对齐的窗口把 page_indices 流式搬进 SMEM,schedule 直接携带物理页号,主 kernel 不再读页表。Qwen3.5-397B-FP8(TP8 = 8 Devices,MTP×3)实测吞吐 +0.8% ~ +2.7%、TTFT -3.3% ~ +0.6%,无回退。0.0 × NaN = NaN。无效 V 条目有两个来源:三缓冲 VMEM 槽复用残留,以及 SEQ_ALONG_LANE 下相邻请求共享 128-lane 新 KV 页。三道防线(#1661 / #1877 / #2454 mask_invalid_v)合入后,「MTP 接受率跌 0」现象已很少出现;排查重心转向 NaN 源头(疑似 Prefill 侧)。TPU_PP_HANDOFF=ring,#2527 草稿):默认 lockstep hand-off 是所有 stage 同时发起的 collective-permute,每轮等最慢的 forward,trace 显示每个 stage 每次 forward 后空等 10–14 ms;改为 isend/irecv 后算完即发给下一 stage。2x2x4(16 Chips = 32 Devices,每 Host 一个 stage = 8 Devices)、PP4·PCP8·EP、Qwen3.5-35B-FP8 上:低并发(1–4 streams)输入吞吐 +14% ~ +20%,TTFT p99 最多 1158 → 648 ms;64 streams +3%;多轮 agent 场景(prefix 命中 53%)8 会话 +44.8%、TPOT 109 → 74 ms;叠加 VLLM_PP_LAYER_PARTITION=11,9,11,9 再 +2.5% ~ +12.5%。需 libtpu ≥ 0.0.50 + torch_tpu ≥ 0.2.0。#2169):Mamba align 模式下 GDN kernel 只在 chunk 末尾写 recurrent state,scheduler 只能在可缓存块边界切开 prefill,例如 PP4·PCP8(逻辑块 3072)下 8K prompt 开 MTP 被拆成 3072 + 5120、不开为 6144 + 2048。每个 GDN 层多声明 1 个 checkpoint block 后 prefill 一次跑到底,中途写边界状态;checkpoint 落在 PCP tile 内时用影子 tile 保存 / 恢复 carry。
1. Three root causes of a ~18 GiB/host cold-vs-warm start Host DRAM gap: on a single 397B decode host (4 chips = 8 devices, DP4·TP2·EP8·MTP3) cold start uses 168.25 GiB/host vs ~150 GiB/host warm; stage probes place 100% of the gap in compile_or_warm_up_model() (weights + KV init contribute <0.2 GiB).
A: executables from CompileAndLoad retain compile-only metadata (CompilerMetadata, non-deduplicated BackendConfigWrapper, MosaicMlirCacheEntry, NameUniquer) that LoadSerializedExecutable does not;
B: rank-local c10d group_name strings break cross-rank graph fingerprints, so every follower rank re-runs the full FX→MLIR→Pallas lowering;
C: gc.freeze() pins frontend caches and PiecewiseBackend.graph in the frozen Python heap.
Fixes: serialize→unload→reload via LoadSerializedExecutable; canonicalize group_name and enable TPU_PRECOMPILE_PEER_REPLAY=1 (84 of 96 compilations replayed); clear caches before freeze_gc_heap(). Together they remove 90.5%–93.9% of the gap.~170K tokens while staging was sized for 1M, under a <200 GiB/host RSS budget that must also host KV offload. New TPU_RAIDEN_POOL_STAGING_BUDGET_MB / TPU_RAIDEN_POOL_STAGING_WINDOW_BLOCKS bound the decode arena to a window of W full-attention pages and pull long requests as a chain of page-aligned sub-transfers (GDN state in the last one).
Qwen3.5-397B (1M, PCP8·TP1 ➔ DP4·TP2): staging 7.63 → 1.81 GiB/rank, 1M tokens in 10 sub-transfers / 780 ms (20.9 GB/s); on a larger 92-layer 1M-context model: 93.2 → 2.71 GiB/host (61×) at ~2.3–3.1 ms per extra sub-transfer.vllm-torchtpu#2614, merged 10-30): fully resident page_indices needed 192 × (1M / 256) × 4 B = 3 MiB vs a 1 MiB SMEM, forcing max-num-seqs < 64. The scheduler now streams page_indices through double-buffered lane-aligned SMEM windows and emits physical page ids, so the main kernel never loads the page table. Qwen3.5-397B-FP8 (TP8 = 8 devices, MTP×3): throughput +0.8% ~ +2.7%, TTFT -3.3% ~ +0.6%.0.0 × NaN = NaN. Invalid V entries come from reused triple-buffered VMEM slots and from shared 128-lane new-KV pages under SEQ_ALONG_LANE. With three defenses merged (#1661 / #1877 / #2454 mask_invalid_v), MTP drop-to-0 events are now rare; focus has moved to the NaN origin (suspected Prefill).TPU_PP_HANDOFF=ring, draft #2527): the default lockstep hand-off is one collective-permute launched by all stages together, so every round waits for the slowest forward (traces show 10–14 ms idle per stage per forward); isend/irecv forwards each output as soon as it is ready. On 2x2x4 (16 chips = 32 devices, one 8-device stage per host), PP4·PCP8·EP, Qwen3.5-35B-FP8: +14% ~ +20% input throughput at 1–4 streams (TTFT p99 up to 1158 → 648 ms), +3% at 64 streams, and +44.8% on 8 concurrent multi-turn agent sessions (53% prefix hits, TPOT 109 → 74 ms); VLLM_PP_LAYER_PARTITION=11,9,11,9 adds +2.5% ~ +12.5%. Requires libtpu ≥ 0.0.50 + torch_tpu ≥ 0.2.0.#2169): because the GDN kernel writes recurrent state only at chunk end, Mamba-align prefix caching split prompts at cacheable block boundaries (e.g. 8K under PP4·PCP8, logical block 3072: 3072 + 5120 with MTP, 6144 + 2048 without). One extra checkpoint block per GDN layer lets prefill run to the end while writing the boundary state mid-chunk; a shadow tile saves/restores the carry when the checkpoint falls inside a PCP tile.
TP2·PCP16 ➔ TP4·DCP8 拓扑下,每次请求握手都把完整 Receiver 地址表(~10 MB)序列化一遍,哪怕 prompt 只有 87 token;单线程 start_transfer() 耗时 239 ms,KV 传输启动率被卡在 4.2 req/s。去掉冗余转发后 Prefill 分发 212 → 6.1 ms(35×),协调吞吐 4.2 → 22–30 req/s。@jax.jit 里串行放入 32 对以上 ppermute 测 2x2x4 链路,会耗尽 36 个自定义 sync flag,后面的链路静默回退为 GLOBAL barrier,测出 93 → ~50 GB/s 的假降速。改用边着色的多对并行轮次(或统一全局 barrier 后计时),可在 <10 s 内无误报地扫完一个 4x4x4 Cube(64 Chips = 128 Devices)。34.4 TFLOP/s(约正常的 1/30)、HBM 带宽 365 GB/s(约 1/9),却仍被报告为 HEALTHY,把一个数千芯片的同步作业拖慢 12.5×(10.7 → 212 s/step)。被动健康信号不够,需要准入 / 周期性的短时 GEMM + HBM 带宽探针。tpu-device-plugin 默认 80Mi 内存上限导致约 78% 节点反复 OOMKilled(调到 160–512Mi);随节点数扩容的 konnectivity-agent 打满系统节点 ARP 表(gc_thresh3=1024),引发 webhook / DNS 大面积超时;kube-state-metrics 需从 500Mi 调到 ~10Gi。disable_bounds_checks=True 越界 = 硬件致命中断:越界的 page / block 索引会触发 SparseCore addr_out_of_range;主进程崩溃约 4 秒后,卡在 reduce-scatter 的其余 worker 会刷出次生错误日志,排障要锁定毫秒级最早的那条。stop/start(会触发固件错误锁死、健康硬件被送修);容器设 ulimit -c 0(数百 GB 进程的 core dump 会写满小启动盘,触发 0 秒宽限期驱逐);分层收紧拓扑发现超时(默认值可达 4,000 s,单点故障会让上千台主机空等)。
1. Raiden handshake metadata regression: with 32 receivers / 162 cache pools (TP2·PCP16 ➔ TP4·DCP8) every request serialized the full receiver address table (~10 MB) even for an 87-token prompt; single-threaded start_transfer() took 239 ms and capped KV transfer launches at 4.2 req/s. Dropping the redundant forwarding cut prefill dispatch 212 → 6.1 ms (35×) and raised coordination throughput to 22–30 req/s.ppermutes into one @jax.jit to test 2x2x4 links exhausts the 36 custom sync flags; remaining links silently fall back to GLOBAL barriers and read 93 → ~50 GB/s. Edge-colored multi-pair rounds (or forced global barriers with post-barrier timing) scan a full 4x4x4 cube (64 chips = 128 devices) in <10 s with no false alarms.34.4 TFLOP/s per device (~1/30) and 365 GB/s HBM (~1/9) while it still reported HEALTHY, slowing a multi-thousand-chip synchronous job 12.5× (10.7 → 212 s/step). Passive health signals are not enough; use short GEMM + HBM probes at admission and periodically.tpu-device-plugin's 80Mi limit OOM-killed ~78% of nodes (raise to 160–512Mi); node-count-scaled konnectivity-agent overflowed the system nodes' ARP table (gc_thresh3=1024), timing out webhooks and DNS; kube-state-metrics needs ~10Gi instead of 500Mi.disable_bounds_checks=True is a hardware fatal: a bad page/block index raises SparseCore addr_out_of_range; ~4 s later, peers blocked in reduce-scatter log secondary errors, so triage must key on the earliest millisecond timestamp.stop/start a single VM of a multi-host slice before deforming the slice (firmware-error latch, healthy hardware sent to repair); set ulimit -c 0 (core dumps of multi-hundred-GB processes fill the small boot disk and trigger 0-second-grace eviction); tier topology-discovery timeouts (defaults up to 4,000 s leave thousands of hosts idle on one failure).
1,152 tokens/page)、2-Layer Twin 微基准与 7 层编译器栈 × 6 大 PR 架构收官
8. CoreML Hybrid Mamba/GDN Page Alignment Math (1,152 tokens/page), 2-Layer Twin Microbenchmark & 7-Layer Stack Synthesis
397B.md & tpu_block_size_utils.py:_derive_tpu_block_slot_config):在 mamba-cache-mode=align / TPU_VLLM_ENABLE_UNIFIED_BLOCK_POOL=1 下,单层 GDN 原始状态槽大小为 mamba_raw_state_bytes = 542,720 B(conv_state = 18,432 B @ TP=8 spec=3 或 49,152 B @ TP=2 spec=1,加上 recurrent_state = 524,288 B FP32 / TP=2 BF16 或 262,144 B TP=8 BF16)。由于在 TP ≥ 2 下每 Rank 仅持有 1 个 GQA KV Head(配合 PR #608 SEQ_ON_LANE),单层 Attention KV 恰好消耗 1 × 2 (K,V) × 256 × 1 B (FP8) = 512 B/token/layer;因此容纳 1 个 GDN 状态槽需 ⌈542,720 / 512⌉ = 1,060 tokens,向上取整至 256 的整数倍即得 block_size = 1,152 tokens/page(589,824 B ≥ 542,720 B)。2-Layer Twin 微基准(1 linear_attention + 1 full_attention + MTP):通过 hf_overrides: {"text_config": {"num_hidden_layers": 2, "layer_types": ["linear_attention", "full_attention"]}} 在单台 2x2x1(8 Devices)上复刻 397B 的精确单 Rank 形状(q_heads_projected=16、attn_kv_heads_per_rank=1、gdn_k_heads_per_rank=8、gdn_v_heads_per_rank=32、experts_per_rank=64、cache_block_tokens=1,152、pages_per_rank=4,007、kv_pool_tokens_per_rank=4,617,186,HBM Ballast 锁定 76.49 GiB/device)。DP4×TP2 + EP8、PR #1458 BF16 GDN State(+12.2% KV 池、+9.1% Decode)、PR #106 统一页池、tpu-connector / TPUConnectorHMA / Raiden (PR #1183, #1558, #1677) 直传完整 Tail + 45 层 GDN 状态 ~349 MB(零 Tail 重算 + 双 200G NIC NUMA 绑核 367 Gbps);Layer 2 (Graph Capture, 0% Pallas) √2 几何桶 + PR #2277 AOT Export;Layer 3 (StableHLO Views, 0% Pallas) PR #766 4D 零拷贝 bitcast(DMA 指令 366 ➔ 87,GDN 5.3× 加速 4.12 ➔ 0.774 ms);Layer 4 (XLA [8, 128] Tile Layout) PR #608 SEQ_ON_LANE(4 个 FP8 Token 打包进 1 个 32-bit 槽,1,024 ➔ 512 B/page);Layer 5 左 (XLA Fusions, 0% Pallas) PR #371 / #509 小 Batch 3 行 F.one_hot + MXU matmul(c64 +14.5% 吞吐) vs. Layer 5 右 (Surgical Pallas) PR #543 + #891 全层 SPMD Fused MoE(双缓冲异步 DMA 掩盖 1,190 µs / 22% ICI Reduce-Scatter,单层 4.8×、E2E 2× 吞吐 10,614 ➔ 21,297 tok/s)、PR #1008 RPA v3 MTP 验证(10.2 ➔ 5.6 ms)、PR #2276–#2361 声明式 Megakernel + pool_alias (#2227);Layer 6 (ICI Collectives) PR #65 / #129 PCP=8 All-to-All(128K TTFT 32s ➔ 4.62s);Layer 7 (PJRT Runtime & Host OS, 0% Pallas) PR #1810 Peer Replay(DP=8 启动 741s ➔ 270s)与 PR #2141 / #2068 Host DRAM & Linux THP 释放(440 ➔ 230 GiB)。6.73 ms,DP4×TP2 + EP8 解析预测 22.72 ms 对决 9,216 芯片生产灰度实测 24.10 ms,误差 < 6%(5.7%)。
• CoreML Exact Hybrid Mamba/GDN Page Alignment Math (397B.md & tpu_block_size_utils.py:_derive_tpu_block_slot_config): In mamba-cache-mode=align / TPU_VLLM_ENABLE_UNIFIED_BLOCK_POOL=1, mamba_raw_state_bytes = 542,720 B per layer (conv 18,432 B at TP=8 spec=3 or 49,152 B at TP=2 spec=1 + recurrent 524,288 B FP32 / TP=2 BF16 or 262,144 B TP=8 BF16). Because 1 GQA layer per rank at TP ≥ 2 uses 1 KV head × 2(K,V) × 256 head_dim × 1 B (FP8) = 512 B/token/layer (with PR #608 SEQ_ON_LANE), fitting one GDN slot requires ⌈542,720 / 512⌉ = 1,060 tokens, rounded up to a 256-token multiple = 1,152 tokens/block (589,824 B ≥ 542,720 B).2-Layer Twin Microbenchmark (1 linear_attention + 1 full_attention + MTP): Uses hf_overrides: {"text_config": {"num_hidden_layers": 2, "layer_types": ["linear_attention", "full_attention"]}} on a single 2x2x1 host (8 devices) with identical per-rank shapes (q_heads_projected=16, attn_kv_heads_per_rank=1, gdn_k_heads_per_rank=8, gdn_v_heads_per_rank=32, experts_per_rank=64, cache_block_tokens=1,152, pages_per_rank=4,007, kv_pool_tokens_per_rank=4,617,186, HBM ballast to 76.49 GiB/device).DP4×TP2 + EP8, PR #1458 BF16 GDN state (+12.2% KV pool, +9.1% decode), PR #106 Unified Pool, and tpu-connector / TPUConnectorHMA / Raiden (PR #1183, #1558, #1677) shipping full tail + 45-layer GDN state ~349 MB with zero tail recompute and dual-200G NIC NUMA pinning at 367 Gbps; Layer 2 (Graph Capture — 0% Pallas) √2 Precompile Buckets + PR #2277 AOT Export; Layer 3 (StableHLO Views — 0% Pallas) PR #766 4D zero-copy bitcast cutting DMAs 366 ➔ 87 and speeding up GDN 5.3× (4.12 ➔ 0.774 ms); Layer 4 (XLA [8, 128] Tile Layout) PR #608 SEQ_ON_LANE packing 4 FP8 tokens into 1×32-bit slot (1,024 ➔ 512 B/page); Layer 5 Left (XLA Fusions — 0% Pallas) PR #371 / #509 3-line PyTorch F.one_hot + matmul on the 256×256 MXU (+14.5% at c64) vs. Layer 5 Right (Surgical Pallas) PR #543 + #891 Whole-Layer SPMD Fused MoE hiding 1,190 µs (22%) exposed ICI Reduce-Scatter (4.8× layer speedup, 2× E2E 10,614 ➔ 21,297 tok/s), PR #1008 RPA v3 MTP verification (10.2 ➔ 5.6 ms), and PR #2276–#2361 Declarative Pallas Megakernel with pool_alias (#2227); Layer 6 (ICI Collectives) PR #65 / #129 PCP=8 All-to-All cutting 128K TTFT 32s ➔ 4.62s; Layer 7 (PJRT Runtime & Host OS — 0% Pallas) PR #1810 Cross-Rank Peer Replay (741s ➔ 270s) & PR #2141 / #2068 Startup Host-RAM & Linux THP release (440 ➔ 230 GiB).6.73 ms HBM weight-read floor, 22.72 ms analytic prediction for DP4×TP2 + EP8 vs. 24.10 ms 9,216-chip production canary (< 6% / 5.7% error).