🌐 中文版:MPK 单卡性能根因分析 —— 为什么在 PPU-ZW810E 上比 SGLang 慢 3.4×

Previously: Porting NVIDIA’s CUDA Megakernel (MPK) to China’s T-Head PPU-ZW810E. The last post got MPK running, correct, and beating native on a domestic card. This one answers a different question: why is single-card MPK still 3.4× slower than SGLang, and how do we close the gap?

Goal: optimize MPK (Mirage Persistent Kernel) single-card inference to beat the SGLang single-card baseline.
Hardware: Alibaba T-Head Zhenwu ZW810E (8×/node, 96GB HBM2e, 64 SMs, CUDA-compatible SIMT, __CUDA_ARCH__=800 Ampere path).
Model: Qwen3-8B, bf16, decode (single-token generation).
Date: 2026-07-01.


1. Conclusion (TL;DR)

MPK single-card decode = 30.8 ms/token, SGLang single-card = 8.99 ms/token — 3.43× slower.

The single root cause of the gap has been locked down with an isolation experiment:

A kernel that contains tensor-core (wmma/mma) instructions loses occupancy (the number of blocks that can co-reside per SM) sharply as its shared-memory usage grows. MPK’s megakernel both uses wmma for GEMM (93% of the time) AND launches with 160KB of dynamic smem (inflated by the attention task), so only 20 blocks reside at a time = only 20 of the 64 SMs are used = the 3.4× gap.

Only 20 of 64 SMs resident

  • It’s not registers (VReg/SReg), not cooperative launch, not launch_bounds, not code size, not cp.async, not scheduler logic — every one of these was falsified in turn.
  • The fix is clear and fully preserves the megakernel model: drop launch smem to ≤64KB → occupancy 20→64 (3.2× more SMs) → expected ~9.4 ms/token ≈ matching/beating SGLang.
  • The only sticking point: the 160KB is inflated by attention’s flash stage-2 buffer, and attention is only 3.8% of the time — shrinking it to buy 3× parallelism on the 93% path is a good trade.

2. The Bottleneck: Memory Bandwidth, Not Compute

Decode (batch≈1) is memory-bandwidth-bound: each weight is read exactly once, arithmetic intensity ~1 FLOP/byte.

Measured ZW810E HBM peak bandwidth (high-MLP, unrolled float4 stream micro-benchmark):

blocks×threads achieved bandwidth
20×128 600 GB/s (ceiling)
20×256 1055 GB/s
20×512 1524 GB/s
256×512 1906 GB/s (still climbing, peak ≈2.3 TB/s)
  • SGLang ≈ 1.78 TB/s (fully using 64 SMs + CUDA graph + hand-tuned kernels)
  • MPK ≈ 0.53 TB/s (20 SMs)
  • MPK’s linear_kernel already hits ~88% of that config’s theoretical ceiling at the 20×128 setting (530/600 GB/s) → optimizing the linear_kernel inner loop is pointless; the bottleneck is only using 20/64 SMs.

Profiling (single card, no-cutlass, W=19/S=4): TASK_LINEAR + TASK_LINEAR_WITH_RESIDUAL = 93% (the same linear_kernel), attention 3.8%, everything else <3%.

Bandwidth: MPK vs SGLang vs HBM peak


3. Systematic Investigation: Falsifying Each Hypothesis in Turn

The key methodology shift: from trial-and-error (“change one parameter → recompile the demo (3min) → look at the result”) to **measuring directly in isolated micro-benchmarks + single-variable cause elimination (10-30s per run)**. Core tools:

  1. In-kernel census: add a device flag g_probe_exit to persistent_kernel so that on entry it does atomicAdd(&live) / atomicMax(&maxlive) / spin / atomicSub and immediately returns → use an ordinary <<<64>>> launch to directly measure “how many blocks co-reside at most.”
  2. Synthetic dummy: an empty kernel whose resource attributes (cudaFuncGetAttributes: numRegs / static smem / local / maxThreads / dynamic smem) are identical to the real megakernel, running the same census / cooperative sweep.

The Key Anomaly

kernel occupancy (co-resident out of 64)
real persistent_kernel 20
attribute-identical dummy 64

Their cudaFuncGetAttributes are byte-for-byte identical, launched at the same moment on the same stream — yet 20 vs 64.

Falsification, One by One (all synthetic kernels reach 64)

hypothesis test result
VReg registers force numRegs 32→254 all 64
SReg scalar registers force 56→110 (>persistent’s 90) all 64
static smem 0→2048 all 64
local memory 0→372 all 64
dynamic smem 160KB 64 (plain kernel)
code size 0→20K instructions all 64
launch_bounds remove / change min-blocks no effect
load-latency staggering spin duration ×10 (300M cycles) still 20 (not a staggering artifact)

Traps ruled out along the way:

  • cudaOccupancyMaxActiveBlocksPerMultiprocessor is unreliable on the PPU: it reports occ=1/SM→cap 64 for persistent_kernel, but the measured value is 20.
  • An early “cooperative cap=20” grid sweep was briefly contaminated by persistent_checker‘s assert(gridDim.x == num_workers + num_scheds/4) (when the grid doesn’t match the config, the failed assertion was misread as a launch failure) — only after reading the full launch-path code did this surface; moving the probe’s early return ahead of the assertion fixed it.
  • PPU compiler-flag facts: -Xptxas --maxrregcount=N can squeeze VReg but is silently overridden by __launch_bounds__; bare --maxrregcount and -Xcicc -ppu-max-vreg-count are both ignored; there is no flag to squeeze SReg.

4. Root Cause (The Decisive Isolation Experiment)

After ruling out all the resources above, the only structural difference not yet reproduced was: the real task kernel uses cp.async and wmma (tensor core), while the synthetic kernel does not.

Stuffing those instructions into a census kernel (compiled in, guarded by if(*go==999) so they never execute — mimicking persistent_kernel’s early return) and sweeping smem size:

kernel contains \ smem 8KB 49KB 64KB 73KB 82KB 98KB 160KB
plain load 64 64 64 — — 64 64
cp.async 64 64 64 — — 64 64
wmma (tensor core) 64 64 64 60 60 40 20

The smoking gun:

  • cp.async is innocent, plain load is innocent — 64 throughout.
  • Only wmma + large smem: occupancy stays at 64 below ~65KB, then 73-82KB→60, 98KB→40, 160KB→20.

Occupancy vs smem: wmma vs plain load

Mechanism (inferred)

On the ZW810E, the tensor-core unit occupies extra per-SM state, which cuts the effective smem budget of a wmma-containing kernel down to ~64-72KB/block (1 block/SM covering all 64 SMs). Above that, large-smem wmma blocks can only land on fewer and fewer SMs: 98KB→40 SMs used, 24 idle; 160KB→20 SMs used, 44 idle.

This perfectly explains everything:

  • The MPK megakernel uses wmma (linear_kernel GEMM = 93%) and launches at 160KB (MAX_DYNAMIC_SHARED_MEMORY_SIZE, inflated by attention) → stuck at 20 blocks = 20/64 SMs = the measured 3.4× gap.
  • The dummy contains no wmma → 64.
  • Tuning registers/SReg/code size did nothing (because none of them were the cause).

5. The Fix (Keep the Megakernel Model)

Drop the megakernel’s launch smem from 160KB to ≤64KB → occupancy 20→64 → ~3× more SMs → ~3× bandwidth → expected ~9.4 ms/token ≈ SGLang.

This is purely a shared-memory footprint adjustment — it doesn’t touch any of the three Mirage megakernel benefits (task fusion / eliminating per-operator launch overhead / decentralized persistent scheduling). The earlier worry — “loading config fields from memory on demand would lose the megakernel benefits” — was the wrong direction; the SReg hypothesis is already falsified, so there’s no need to touch the config at all.

The Sticking Point, Quantified

The megakernel is a single launch with fixed smem = the max requirement across all tasks. The 160KB is inflated by attention’s flash stage-2 output buffer:

multitoken_paged_attention_32_64.cuh:191

1
2
S_O_BUFFER_SIZE = sizeof(float) * GLOBAL_ITERS_M * NUM_THREADS * 64
≈ 4 * GLOBAL_ITERS_M * 128 * 64 ≈ GLOBAL_ITERS_M * 32KB

At GLOBAL_ITERS_M ≈ 5 that’s already ≈160KB. Meanwhile linear_kernel (93%) uses far less smem (a GEMM tile is typically 32-48KB, should fit in 64KB — needs confirming).

Attention is only 3.8% of the time → shrinking its tile (or decoupling attention’s smem budget from linear’s) to buy 3× occupancy on the 93% linear path is a very good trade.

Paths to Ship (pick one, ordered by risk/reward)

  1. First validate the 3× gain end-to-end (low risk, do this first): temporarily force MAX_DYNAMIC_SHARED_MEMORY_SIZE down to 64KB, shrink attention’s MAX_TOKENS/GLOBAL_ITERS_M to fit (don’t chase exact numerics yet), and measure whether occupancy and per-token latency really improve ~3×. Fix attention correctness properly once the gain is confirmed.
  2. Properly optimize attention smem: restructure the flash stage-2 buffer layout (S_O_BUFFER) to genuinely get attention to ≤64KB while staying correct.
  3. First measure each task’s actual smem: confirm the 64KB target is feasible and exactly which tasks to shrink, then act.

6. Expected Payoff

target megakernel smem expected occupancy expected ms/token vs SGLang (8.99)
160KB (current) 20 30.8 3.4× slower
98KB 40 ~15.4 1.7× slower
≤64KB 64 ~9.6 ≈ parity
≤50KB + more threads/block 64 + higher BW/block <9 beats it

(Latency extrapolated linearly assuming bandwidth ∝ resident block count; needs end-to-end validation in practice.)

Expected ms/token per smem target

Stacking further: on top of 20→64 occupancy, if linear_kernel uses more threads/block (128→256), per-block bandwidth rises another ~1.76× (micro-benchmark), which should clearly beat SGLang.


7. Engineering Notes

  • Build reality: the megakernel is a single giant compilation unit, and the PPU’s nvcc (a release-13.0 fork) has no -split-compile; --threads only parallelizes across multiple architectures → a single-file compile is inherently single-threaded (~2-3min), and 256 cores go unused. For diagnostic iteration, use -O1 (the -O3 LLVM passes are the main slowdown).
  • Diagnostic tricks: device printf does not flush when an illegal access/abort tears down the kernel → use a mapped host-memory ring + __threadfence_system() to read back. hggc-memcheck segfaults on itself at megakernel scale and is unreliable. In-kernel census (device flag + atomicMax) is the reliable way to measure real occupancy.
  • SSH: the box’s connection is flaky — always set ServerAliveInterval=10.
  • Probe files (scratchpad): bw_hi.cu (bandwidth), coop_cap_probe.cu / smem_cap.cu (cooperative cap), coop_regsweep.cu (VReg elimination), occ_census.cu (code-size elimination), sreg_confirm.cu (SReg elimination), async_census.cu (found wmma), async_census2.cu (isolated wmma vs cp.async + the smem threshold curve).
  • Box state: /work/mirage/include/.../persistent_kernel.cuh has env-gated diagnostic injections (mpk_coop_probe_dummy, the g_probe_exit census, attr-diff, and the MPK_NO_LB/MPK_PTXAS_REG/MPK_SMEM_OVERRIDE/MPK_COOP_MIN_BLOCKS build switches) — uncommitted, box-local only; clean up before any real change.

8. One Sentence for the Decision-Maker

MPK is 3.4× slower not because the kernel is badly written, but because the PPU’s tensor-core conflicts with large shared memory and locks it into 20 of the 64 SMs. Drop the megakernel’s smem from 160KB to ≤64KB and you unlock all 64 SMs, reclaim ~3× bandwidth, and match or beat SGLang — without touching any core mechanism of the Mirage megakernel.