🌐 中文版:把 NVIDIA 的 CUDA 巨核 (MPK) 移植到平头哥 PPU-ZW810E

A “theoretically impossible” experiment: take the Mirage Persistent Kernel (MPK) megakernel — hand-written for NVIDIA Hopper/Blackwell — and run it on Alibaba T-Head’s Zhenwu 810E (PPU-ZW810E, a non-NVIDIA architecture). The result: it not only ran, but single-card Qwen3-8B inference beat PPU’s own optimized stack.

This post records the full process of hitting walls, cracking the case, and tuning. All numbers come from real hardware.


TL;DR

  • ✅ MPK built from source on the PPU (CUDA-compatible, __CUDA_ARCH__=800).
  • ✅ The Qwen3-8B megakernel ran and produced correct output.
  • ✅ After fixing one deadlock, 16-worker single-kernel path = 35.5 ms/token, beating native’s 39.4 ms (1.11×).
  • 🔬 Using PPU’s full debugging toolchain (ppu-gdb / racecheck / synccheck / occupancy probes / instrumentation), I diagnosed a high-worker-count deadlock layer by layer as a scheduler-scale livelock.

16× PPU-ZW810E overview


1. The Cast: A CUDA Megakernel, and a Card That Isn’t NVIDIA

MPK (Mirage Persistent Kernel) takes a radical approach: fuse every operator and all communication of the entire LLM inference pass into a single “giant persistent kernel” launched once, with an in-kernel scheduler dispatching tasks to resident workers. It was hand-written for NVIDIA — full of NVIDIA-exclusive instructions like cp.async, wgmma, TMA, and nvshmem.

PPU-ZW810E (Zhenwu 810E) is Alibaba T-Head’s in-house AI chip. It is not an NVIDIA architecture — it uses T-Head’s own PPU parallel architecture plus an ICN inter-chip interconnect.

The first instinct, moving the former onto the latter, is “the architectures aren’t compatible.” But the truth flipped that on its head.


2. The First Surprise: the PPU Actually “Speaks CUDA”

The PPU’s SDK is CUDA source-compatible: ppu-clang (clang/llvm) can compile CUDA C/C++, ships with an nvcc wrapper, and even supports Triton. The single most important line, measured directly:

1
__CUDA_ARCH__ == 800

The PPU reports itself as sm_80 (Ampere) compatible! That means every #if __CUDA_ARCH__ >= 800 sm80 code path in MPK is activated as-is — it doesn’t have the Hopper/Blackwell-exclusive TMA/wgmma, but the Ampere path doesn’t need them.

PPU reporting __CUDA_ARCH__=800


3. Build Hell: Domestic Mirror Sources All the Way

The PPU lives in an Alibaba Cloud data center, and almost nothing outside the firewall is reachable. Building MPK meant a chain of pitfalls, each requiring a switch to a domestic mirror:

  • pip: the container defaults to T-Head’s internal source, which lacks public packages → add -i https://mirrors.aliyun.com/pypi/simple/.
  • Rust: when setup.py was missing cargo, running curl https://sh.rustup.rs | sh hung at info: downloading installer → switch to the aliyun rustup mirror + the aliyun crates mirror.
  • cutlass/json submodules: shipping just the includes wasn’t enough; cmake needs CUDA.cmake/CMakeLists.txt.
  • transformers: the container had 5.3, but the demo was written against the pinned 4.57.1 and threw pad_token_id → downgrade the version.

Once through all that, import mirage succeeded and the PersistentKernel API was in place.

MPK build success + import verification


4. First Run: the Native Baseline

Before bringing in the megakernel, I ran Qwen3-8B via the demo’s built-in torch path (which goes through PPU’s own optimized libraries):

1
Prompt length 39, generate length 256, per-token latency 39.444 ms

The output was a coherent, correct “introduction to large language models.” On a domestic card, Qwen3-8B inference is correct at 39.4 ms/token. Baseline established.

native inference output + latency


5. Bringing in the Megakernel: Compiles, and Produces Correct Results

With --use-mirage on, MPK generated and compiled the entire 14,000+ task megakernel on the PPU:

1
2
3
4
[MPK INIT] Total tasks: 14140, Total events: 2246
Compiling megakernel ... # nvcc compiles for ~2 minutes
Finished megakernel compilation...
<coherent generated text>

⚠️ A big trap here: after [MPK INIT] comes ~2 minutes of nvcc compilation, during which the GPU is also occupied. I initially thought it had “hung,” killed it early, and misjudged it as a deadlock — when it was just compiling.

The megakernel produced correct output. But the first version (limited to 8 workers) was 68.2 ms/token, slower than native — because it used only 8 of the 64 SMs, and it used MPK’s generic Ampere kernel rather than the PPU’s optimized libraries.

megakernel compilation + MPK INIT


6. The Deadlock Mystery: It Hangs Once There Are Too Many Workers

When I tried to use more workers for speed, something strange happened: 8 workers ran fine, but ≥24 workers always deadlocked — the PPU spun at 100% utilization doing nothing, and finally threw unspecified launch failure.

deadlock scene — one card spinning at 100%


7. Detective Work: Cracking the Case with PPU Debug Tools

This is the most interesting part. The PPU has a full debugging toolchain that mirrors NVIDIA’s, and I brought them out one by one:

1. ppu-gdb attached to the deadlocked process — found that every device warp was a worker, all stuck in execute_worker spin-waiting for tasks, with not a single scheduler frame present.

ppu-gdb backtrace — all workers stuck, no scheduler in sight

2. racecheck (the counterpart to compute-sanitizer) — 0 hazards, ruling out data races.

3. synccheck (checking for __syncthreads/barrier deadlocks, ran for 13 minutes) — 0 errors, ruling out barrier misuse.

racecheck / synccheck passing cleanly

4. cooperative occupancy probe — unexpectedly discovered the PPU actually has 256KB of shared memory/SM (MPK assumed 163KB under sm_80), and that all 64 megakernel blocks could be co-resident simultaneously, ruling out “co-residency exceeded.”

occupancy probe — 256KB/SM, all 64 blocks resident


8. The Truth and the Fix: From “Two Kernels” to “One Kernel”

The key to cracking the case was in the launch code: MPK defaults to split_worker_scheduler=true, splitting workers and the scheduler into two independent kernels on two streams, counting on the PPU to execute the two persistent kernels concurrently.

But the PPU’s concurrent multi-kernel scheduling is more conservative than NVIDIA’s: once the worker count is high (≥~24), it stops scheduling the second scheduler kernel → the scheduler never starts → the workers wait forever. That’s exactly the source of “only workers, no scheduler.”

A one-line fix:

1
2
3
// include/mirage/persistent_kernel/persistent_kernel.cuh:1617
- global_runtime_config.split_worker_scheduler = true;
+ global_runtime_config.split_worker_scheduler = false;

Switching to the single-kernel path: workers and scheduler merge into one kernel, with roles assigned by blockIdx, naturally co-resident in the same kernel, no longer dependent on “two concurrent kernels.”

The result: single-kernel, 16 workers = 35.5 ms/token, beating native’s 39.4 ms!

after the fix, the megakernel beats native


9. The Unsolved Mystery: the “Chronic Deadlock” at 32 Workers

The fix raised the ceiling from 8w to 16w and beat native, but 32w still hangs — and in a different way. After ruling everything out with the full toolchain: it’s not occupancy, not a race, not barrier misuse, not a single-point spin, but a distributed livelock where workers “grab sporadic tasks and then wait, never converging as a whole.” This kind of high-level progress deadlock is invisible to standard sanitizers.

Breaking through it requires source-level changes to how MPK’s scheduler dispatches tasks at scale — a good candidate for an upstream issue, backed by this body of ruling-out evidence. For now, 16w (beating native) is the production-usable ceiling.

performance comparison bar chart


Closing

Path per-token latency
Native (torch + PPU optimized libs) 39.4 ms
Megakernel, two kernels, 8w 68.2 ms
Megakernel, single kernel, 16w (after fix) 35.5 ms ✅

A CUDA megakernel hand-written for NVIDIA not only ran and computed correctly on a domestic PPU, but beat the vendor’s own inference stack — thanks to the PPU’s solid CUDA compatibility (down to __CUDA_ARCH__=800, cooperative launch, and the nvshmem-equivalent sailSHMEM), plus a one-line fix switching the scheduling mode.

The remaining 32w livelock is a deeper problem at the intersection of a domestic card’s conservative concurrent scheduling and MPK’s at-scale scheduler — saved for the next post.