MPK Single-Card Performance Root-Cause: Why It's 3.4× Slower Than SGLang on the PPU-ZW810E
Previously: Porting NVIDIA’s CUDA Megakernel (MPK) to China’s T-Head PPU-ZW810E. The last post got MPK running, correct, and beating native on a domestic card. This one answers a different question: why is single-card MPK still 3.4× slower than SGLang, and how do we close the gap?
Goal: optimize MPK (Mirage Persistent Kernel) single-card inference to beat the SGLang single-card baseline.
Hardware: Alibaba T-Head Zhenwu ZW810E (8×/node, 96GB HBM2e, 64 SMs, CUDA-compatible SIMT,__CUDA_ARCH__=800Ampere path).
Model: Qwen3-8B, bf16, decode (single-token generation).
Date: 2026-07-01.
1. Conclusion (TL;DR)
MPK single-card decode = 30.8 ms/token, SGLang single-card = 8.99 ms/token — 3.43× slower.
The single root cause of the gap has been locked down with an isolation experiment:
A kernel that contains tensor-core (
wmma/mma) instructions loses occupancy (the number of blocks that can co-reside per SM) sharply as its shared-memory usage grows. MPK’s megakernel both uses wmma for GEMM (93% of the time) AND launches with 160KB of dynamic smem (inflated by the attention task), so only 20 blocks reside at a time = only 20 of the 64 SMs are used = the 3.4× gap.

- It’s not registers (VReg/SReg), not cooperative launch, not launch_bounds, not code size, not cp.async, not scheduler logic — every one of these was falsified in turn.
- The fix is clear and fully preserves the megakernel model: drop launch smem to ≤64KB → occupancy 20→64 (3.2× more SMs) → expected ~9.4 ms/token ≈ matching/beating SGLang.
- The only sticking point: the 160KB is inflated by attention’s flash stage-2 buffer, and attention is only 3.8% of the time — shrinking it to buy 3× parallelism on the 93% path is a good trade.
2. The Bottleneck: Memory Bandwidth, Not Compute
Decode (batch≈1) is memory-bandwidth-bound: each weight is read exactly once, arithmetic intensity ~1 FLOP/byte.
Measured ZW810E HBM peak bandwidth (high-MLP, unrolled float4 stream micro-benchmark):
| blocks×threads | achieved bandwidth |
|---|---|
| 20×128 | 600 GB/s (ceiling) |
| 20×256 | 1055 GB/s |
| 20×512 | 1524 GB/s |
| 256×512 | 1906 GB/s (still climbing, peak ≈2.3 TB/s) |
- SGLang ≈ 1.78 TB/s (fully using 64 SMs + CUDA graph + hand-tuned kernels)
- MPK ≈ 0.53 TB/s (20 SMs)
- MPK’s
linear_kernelalready hits ~88% of that config’s theoretical ceiling at the 20×128 setting (530/600 GB/s) → optimizing the linear_kernel inner loop is pointless; the bottleneck is only using 20/64 SMs.
Profiling (single card, no-cutlass, W=19/S=4): TASK_LINEAR + TASK_LINEAR_WITH_RESIDUAL = 93% (the same linear_kernel), attention 3.8%, everything else <3%.

3. Systematic Investigation: Falsifying Each Hypothesis in Turn
The key methodology shift: from trial-and-error (“change one parameter → recompile the demo (3min) → look at the result”) to **measuring directly in isolated micro-benchmarks + single-variable cause elimination (10-30s per run)**. Core tools:
- In-kernel census: add a device flag
g_probe_exittopersistent_kernelso that on entry it doesatomicAdd(&live)/atomicMax(&maxlive)/ spin /atomicSuband immediately returns → use an ordinary<<<64>>>launch to directly measure “how many blocks co-reside at most.” - Synthetic dummy: an empty kernel whose resource attributes (
cudaFuncGetAttributes: numRegs / static smem / local / maxThreads / dynamic smem) are identical to the real megakernel, running the same census / cooperative sweep.
The Key Anomaly
| kernel | occupancy (co-resident out of 64) |
|---|---|
real persistent_kernel |
20 |
| attribute-identical dummy | 64 |
Their cudaFuncGetAttributes are byte-for-byte identical, launched at the same moment on the same stream — yet 20 vs 64.
Falsification, One by One (all synthetic kernels reach 64)
| hypothesis | test | result |
|---|---|---|
| VReg registers | force numRegs 32→254 | all 64 |
| SReg scalar registers | force 56→110 (>persistent’s 90) | all 64 |
| static smem | 0→2048 | all 64 |
| local memory | 0→372 | all 64 |
| dynamic smem | 160KB | 64 (plain kernel) |
| code size | 0→20K instructions | all 64 |
| launch_bounds | remove / change min-blocks | no effect |
| load-latency staggering | spin duration ×10 (300M cycles) | still 20 (not a staggering artifact) |
Traps ruled out along the way:
cudaOccupancyMaxActiveBlocksPerMultiprocessoris unreliable on the PPU: it reports occ=1/SM→cap 64 for persistent_kernel, but the measured value is 20.- An early “cooperative cap=20” grid sweep was briefly contaminated by
persistent_checker‘sassert(gridDim.x == num_workers + num_scheds/4)(when the grid doesn’t match the config, the failed assertion was misread as a launch failure) — only after reading the full launch-path code did this surface; moving the probe’s early return ahead of the assertion fixed it. - PPU compiler-flag facts:
-Xptxas --maxrregcount=Ncan squeeze VReg but is silently overridden by__launch_bounds__; bare--maxrregcountand-Xcicc -ppu-max-vreg-countare both ignored; there is no flag to squeeze SReg.
4. Root Cause (The Decisive Isolation Experiment)
After ruling out all the resources above, the only structural difference not yet reproduced was: the real task kernel uses cp.async and wmma (tensor core), while the synthetic kernel does not.
Stuffing those instructions into a census kernel (compiled in, guarded by if(*go==999) so they never execute — mimicking persistent_kernel’s early return) and sweeping smem size:
| kernel contains \ smem | 8KB | 49KB | 64KB | 73KB | 82KB | 98KB | 160KB |
|---|---|---|---|---|---|---|---|
| plain load | 64 | 64 | 64 | — | — | 64 | 64 |
| cp.async | 64 | 64 | 64 | — | — | 64 | 64 |
| wmma (tensor core) | 64 | 64 | 64 | 60 | 60 | 40 | 20 |
The smoking gun:
- cp.async is innocent, plain load is innocent — 64 throughout.
- Only
wmma+ large smem: occupancy stays at 64 below ~65KB, then 73-82KB→60, 98KB→40, 160KB→20.

Mechanism (inferred)
On the ZW810E, the tensor-core unit occupies extra per-SM state, which cuts the effective smem budget of a wmma-containing kernel down to ~64-72KB/block (1 block/SM covering all 64 SMs). Above that, large-smem wmma blocks can only land on fewer and fewer SMs: 98KB→40 SMs used, 24 idle; 160KB→20 SMs used, 44 idle.
This perfectly explains everything:
- The MPK megakernel uses wmma (
linear_kernelGEMM = 93%) and launches at 160KB (MAX_DYNAMIC_SHARED_MEMORY_SIZE, inflated by attention) → stuck at 20 blocks = 20/64 SMs = the measured 3.4× gap. - The dummy contains no wmma → 64.
- Tuning registers/SReg/code size did nothing (because none of them were the cause).
5. The Fix (Keep the Megakernel Model)
Drop the megakernel’s launch smem from 160KB to ≤64KB → occupancy 20→64 → ~3× more SMs → ~3× bandwidth → expected ~9.4 ms/token ≈ SGLang.
This is purely a shared-memory footprint adjustment — it doesn’t touch any of the three Mirage megakernel benefits (task fusion / eliminating per-operator launch overhead / decentralized persistent scheduling). The earlier worry — “loading config fields from memory on demand would lose the megakernel benefits” — was the wrong direction; the SReg hypothesis is already falsified, so there’s no need to touch the config at all.
The Sticking Point, Quantified
The megakernel is a single launch with fixed smem = the max requirement across all tasks. The 160KB is inflated by attention’s flash stage-2 output buffer:
multitoken_paged_attention_32_64.cuh:191
1 | S_O_BUFFER_SIZE = sizeof(float) * GLOBAL_ITERS_M * NUM_THREADS * 64 |
At GLOBAL_ITERS_M ≈ 5 that’s already ≈160KB. Meanwhile linear_kernel (93%) uses far less smem (a GEMM tile is typically 32-48KB, should fit in 64KB — needs confirming).
Attention is only 3.8% of the time → shrinking its tile (or decoupling attention’s smem budget from linear’s) to buy 3× occupancy on the 93% linear path is a very good trade.
Paths to Ship (pick one, ordered by risk/reward)
- First validate the 3× gain end-to-end (low risk, do this first): temporarily force
MAX_DYNAMIC_SHARED_MEMORY_SIZEdown to 64KB, shrink attention’sMAX_TOKENS/GLOBAL_ITERS_Mto fit (don’t chase exact numerics yet), and measure whether occupancy and per-token latency really improve ~3×. Fix attention correctness properly once the gain is confirmed. - Properly optimize attention smem: restructure the flash stage-2 buffer layout (
S_O_BUFFER) to genuinely get attention to ≤64KB while staying correct. - First measure each task’s actual smem: confirm the 64KB target is feasible and exactly which tasks to shrink, then act.
6. Expected Payoff
| target megakernel smem | expected occupancy | expected ms/token | vs SGLang (8.99) |
|---|---|---|---|
| 160KB (current) | 20 | 30.8 | 3.4× slower |
| 98KB | 40 | ~15.4 | 1.7× slower |
| ≤64KB | 64 | ~9.6 | ≈ parity |
| ≤50KB + more threads/block | 64 + higher BW/block | <9 | beats it |
(Latency extrapolated linearly assuming bandwidth ∝ resident block count; needs end-to-end validation in practice.)

Stacking further: on top of 20→64 occupancy, if linear_kernel uses more threads/block (128→256), per-block bandwidth rises another ~1.76× (micro-benchmark), which should clearly beat SGLang.
7. Engineering Notes
- Build reality: the megakernel is a single giant compilation unit, and the PPU’s nvcc (a release-13.0 fork) has no
-split-compile;--threadsonly parallelizes across multiple architectures → a single-file compile is inherently single-threaded (~2-3min), and 256 cores go unused. For diagnostic iteration, use-O1(the-O3LLVM passes are the main slowdown). - Diagnostic tricks: device
printfdoes not flush when an illegal access/abort tears down the kernel → use a mapped host-memory ring +__threadfence_system()to read back.hggc-memchecksegfaults on itself at megakernel scale and is unreliable. In-kernel census (device flag + atomicMax) is the reliable way to measure real occupancy. - SSH: the box’s connection is flaky — always set
ServerAliveInterval=10. - Probe files (scratchpad):
bw_hi.cu(bandwidth),coop_cap_probe.cu/smem_cap.cu(cooperative cap),coop_regsweep.cu(VReg elimination),occ_census.cu(code-size elimination),sreg_confirm.cu(SReg elimination),async_census.cu(found wmma),async_census2.cu(isolated wmma vs cp.async + the smem threshold curve). - Box state:
/work/mirage/include/.../persistent_kernel.cuhhas env-gated diagnostic injections (mpk_coop_probe_dummy, theg_probe_exitcensus, attr-diff, and theMPK_NO_LB/MPK_PTXAS_REG/MPK_SMEM_OVERRIDE/MPK_COOP_MIN_BLOCKSbuild switches) — uncommitted, box-local only; clean up before any real change.
8. One Sentence for the Decision-Maker
MPK is 3.4× slower not because the kernel is badly written, but because the PPU’s tensor-core conflicts with large shared memory and locks it into 20 of the 64 SMs. Drop the megakernel’s smem from 160KB to ≤64KB and you unlock all 64 SMs, reclaim ~3× bandwidth, and match or beat SGLang — without touching any core mechanism of the Mirage megakernel.
