🌐 中文版:把 CMP 170HX 的 GPU-to-GPU P2P 解出来 —— 九个环节,和中途搞死的一台机器

The CMP 170HX is NVIDIA’s mining variant of the GA100: same silicon, same HBM2e, but the firmware reports 8GB of memory, pins PCIe at Gen1, and fuses BAR1 to 64MB. The community already has solid work unlocking memory geometry and Gen2. This post is about the last piece — GPU-to-GPU P2P.

The conclusion first: it works. All 14 intra-socket directed pairs pass, and latency drops from 15–37 µs to 2.0 µs — the same order of magnitude as a GPU touching its own framebuffer (2.04–2.59 µs).

Getting there took longer than I expected. This is not “apply a patch.” It is a chain of nine links in series: miss any one and P2P is refused at runtime, and each one fails with a different symptom. Along the way I got the machine into a state where only ping answered, and it needed a physical power cycle to come back.

Hardware: Supermicro SYS-4028GR-TR, dual Xeon E5 v4, 6× CMP 170HX (10de:20c2), Ubuntu 24.04 with nvidia-open 610.43.02.


1. What we’re actually unlocking

GA100 PCIe P2P normally runs over the write-mailbox protocol. On this card, that path goes like this:

The driver configures it completely — the connection type is selected, both GSP RPCs return NV_OK, the mailbox region inside BAR1 is allocated. And then it moves zero bytes. No error. No Xid. Nothing.

Everything configured. Zero bytes moved.

Everything configured successfully, data plane at zero. This is the nastiest class of failure, because every checkpoint tells you it’s fine.

The way around it is BAR1 P2P: rewrite the peer PTEs from GMMU_APERTURE_PEER to SYS_COH, pointing at the peer’s BAR1 bus address. Peer memory is then accessed as though it were system memory, and the dead peer aperture never participates.

The two paths are mutually exclusive — the driver only allows one PCIe P2P protocol at a time. So this isn’t “have both.” It’s “you must give up the mailbox.”


2. BAR1 P2P wants a 64GB BAR1; the card ships with 64MB

BAR1 P2P requires a 64GB BAR1 on every card (the whole framebuffer has to map into it). The CMP 170HX ships with BAR1 at 64MB.

The good news is the hardware supports it. Once the driver’s CYA unlock has run, lspci reports:

1
2
BAR 1: current size: 64MB, supported: 64MB 128MB ... 32GB 64GB
↑ what got assigned ↑ what the card says it can do

The bad news is that an entire PCIe resource-allocation problem sits between those two numbers.

I got this wrong once

I saw a 96MB bridge window and a 224MB root-port window against a requirement of 96GB/128GB, and blurted out “that’s three orders of magnitude short.”

That was wrong. 96MB is the window BIOS already assigned — not the address space available. The actual space was 512 GiB per socket:

1
2
20000000000-27fffffffff : PCI Bus 0000:00     ← 512 GiB
28000000000-2ffffffffff : PCI Bus 0000:80 ← 512 GiB

Less dramatic, but it changes the diagnosis entirely: the space wasn’t missing, the windows were locked to the wrong size.


3. The address-space arithmetic, and a timing problem

The unlock runs after enumeration — too late

Space first. Each card’s BAR1 is 64GB and must be 64GB-aligned; adding the 32MB BAR3, a PLX downstream port needs a 96GB window that has to start on a 64GB boundary — so it really occupies 128GB.

1
2
3
per downstream port   96 GB window, 64 GB aligned
per root port 192 GB (two cards behind it)
a 4-GPU socket 512 GB — before alignment padding

Socket 1 on this box carries 4 cards, so the requirement lands exactly on 512GB, and the bottom of that window is already occupied by other bridges. It cannot fit. Changing MMIO High Granularity Size to 1024G in BIOS gives each socket 1 TiB, and the problem goes away.

One common misconception worth correcting: the setting is not Above 4G Decoding. That was already enabled on this machine — the proof being that GPU BARs sat at 0x27ff8000000, around 2.75 TB, which is unreachable without it.

The genuinely awkward part is timing

Space was sufficient and BAR1 was still 64MB. Because:

The card ships with BAR1 fused to 64MB, and the unlock happens at driver probe time — by which point PCIe enumeration is long finished, every window in the bridge chain has been sized for 64MB, and pci_resize_resource() has nowhere to put the larger BAR. It steps back down.

That is the reason the two kernel patches under kernel-patches/ exist:

  • 0002 moves the ReBAR programming into pci_fixup_early, which runs before pci_read_bases(), so enumeration sees a 64GB BAR from the start
  • 0001 fixes one line in pbus_size_mem() — size += max(r_size, align) becomes ALIGN(r_size, align). The original doesn’t budget for the gap a child leaves when its size isn’t a multiple of its alignment, so parent windows come out undersized and some GPUs silently get no BAR at all

Upstream ships these against the 7.0 kernel. This box runs 6.8, and the way I rebased them was: make the equivalent edit in the real 6.8 source, then diff -u it back out — rather than hand-adjusting line numbers. A patch generated that way is guaranteed to apply, because it came from the tree it targets.


4. [The pitfall] One reboot took the whole machine offline

With the driver installed, I typed sudo reboot.

The machine came back. nvidia-smi hung. Then ssh stopped accepting connections. Then the only thing still answering was ping.

One warm reboot, one unreachable machine

dmesg said:

1
2
3
4
NVRM: GPU0 kflcnWaitForHalt_TU102: Timeout waiting for Falcon to halt
NVRM: GPU0 s_executeBooterUcode_TU102: failed to execute Booter: status 0x65
NVRM: GPU0 kgspInitRm_IMPL: Max GSP-RM boot attempts exceeded: 4/4
NVRM: GPU 0000:04:00.0: RmInitAdapter failed! (0x62:0x65:2119)

My first root-cause call was wrong

Before I had the log, I asserted: “I made P2P default-on and wrote RMForceStaticBar1=1 unconditionally, and this machine only has a 64MiB BAR1 — forcing a 64GB static mapping into a 64MiB BAR1 hangs driver init.”

It sounded reasonable. The log matched none of it. The failure is in GSP/Booter bring-up, which belongs to the SEC2 unlock path and runs before any BAR1 or P2P code. RMForceStaticBar1 had nothing to do with it.

“I just changed X, so X broke it” is the easiest attribution error to make — especially when X really does look suspicious. Suspicious is not guilty.

The actual cause

This unlock uses the Booter as a write primitive for privileged registers. Many registers on a CMP simply cannot be written by the host at Level 0, so the code builds a SEC2 payload and calls kgspExecuteBooterLoad to have the Booter perform the write — one register at a time, dozens to hundreds of times.

And WPR2 and the PLM state live inside the GPU; a warm reboot does not clear them. Boot with WPR2 still armed from the previous session and the Booter can no longer run, which kills the write primitive — so every call burns a 4-second timeout.

Then the cascade: GSP boot fails 4/4 → RmInitAdapter failed → nvidia-smi and the nv_open_q / nv_queue kernel threads wedge in uninterruptible sleep, unkillable → every subsequent process touching the driver piles up behind them → the process count climbs until fork() fails → update-initramfs reports Cannot fork → sshd exits.

The cascade after GSP boot fails: D-state threads, fork exhaustion, sshd exit

A driver failing to initialize should not be able to take a machine down to the point where ssh no longer works. That is a genuine robustness defect, and part of the responsibility sits with the unlock patch itself. Upstream turned NVIDIA’s hard refusal:

1
2
NV_PRINTF("unexpected WPR2 already up, cannot proceed with booting GSP");
return NV_ERR_INVALID_STATE; // ← clean exit

into a warning and carried on. What should have been a clean failure in seconds, with ssh still usable, became four rounds of timeouts that dragged the driver down with them.

The lesson

This unlock requires a cold boot; reboot does not count. Power actually has to leave the card: shutdown -h then power on, or chassis power cycle over IPMI. chassis power reset is not enough either — it doesn’t drop the 12V rails.

At the time, that machine had no working out-of-band management. Someone had to walk into the machine room.

So the correct order is: establish your recovery path first, then do the risky thing. I later brought up the BMC (it can be configured from inside the OS over KCS, without being able to reach it on the network first), and every cold boot after that was one command.


5. Kernel patches in, and only 4 of 6 cards came up

With the custom kernel booted, the pci_fixup_early quirk fired on all six cards — the log is unambiguous:

1
2
pci 0000:04:00.0: CMP 170HX: BAR1 REBAR programmed to 64GB before enumeration
pci 0000:04:00.0: BAR 1 [mem 0x2f000000000-0x2ffffffffff 64bit pref]

64GB at enumeration time. And yet: four cards got a 64GB BAR1, and the two on socket 0 got nothing — not even the 64MB they used to have.

1
2
pci 0000:04:00.0: BAR 1 [mem size 0x1000000000 64bit pref]: can't assign; no space
pci 0000:00:02.0: bridge window [mem 0x2fff0000000-0x2fffdffffff 64bit pref]

That root-port window is still the 224MB value from firmware. The kernel never tried to reassign it — while socket 0 had a full, empty terabyte and needed 192GB.

The reason is Linux’s realloc heuristic: it only engages when it happens to notice a device with unassignable BARs. The SR-IOV device that tripped it sat on socket 1, so socket 1 got rescued and socket 0 was skipped.

The fix is a kernel parameter:

1
GRUB_CMDLINE_LINUX_DEFAULT="... pci=realloc"

which forces reallocation on every root bus instead of relying on the heuristic. After that:

All six cards at 64G BAR1, plus the root-port and PLX downstream windows

6/6, all 64GB.


6. Everything in place, P2P still refused — and “no output” was the answer

BAR1 at 64GB on all six. Kernel patches in. Module parameters RMForceStaticBar1=1;RMPcieP2PType=1 live. Three of the four driver patches applied.

Ran the test. All 30 directed pairs: NO PEER MAPPING.

The driver-side gate function kbusIsPcieBar1P2PMappingSupported_GH100 has four checks, and I didn’t know which one was rejecting. This time I stopped guessing and ported a print-only, behaviour-free debug patch — an unconditional NV_PRINTF at the top of that function, dumping all four gate inputs.

Rebuilt, cold-booted, ran the test. And:

Not a single line of output.

That was the answer. The print is unconditional; it fires on entry. Zero output means the function was never called — the rejection happened further upstream, and none of the four gates was ever reached.

Following it up: GSP reports P2P as unsupported on a CMP, _gpuInitPcieP2PCapability stores that verdict, and p2pCapsGetStatus short-circuits on it — so control never reaches the BAR1 branch at all.

The fourth patch, p2p-force-caps, forces the advertised capability bits to OK. The upstream notes describe it as optional, something that “only changes what the driver claims.”

On this code path that is wrong — it is a hard prerequisite. Without it, the other three patches never get a chance to execute.

With it applied:

1
2
3
4
CMPUNLOCK: PCIe P2P caps forced to OK
CMPUNLOCK_BAR1P2P: gpu0->gpu1 pcieP2PType=1 loopback=0 staticBar1_0=1
staticBar1_1=1 peerMask0=0x0 peerMask1=0x0
CMPUNLOCK_BAR1P2P: ACCEPTED - BAR1 P2P supported

Through.

There’s an inverse trap worth noting here: the p2p-mailbox-skip patch’s log line never appears, even when the patch is applied — release builds strip NV_PRINTF format strings in that translation unit. A missing log line does not mean missing code. Verify with nm / objdump on the object file, not by grepping dmesg.


7. The nine required pieces

Nine links. Miss one and it's refused.

Miss any one and it’s refused at runtime, each with a different symptom:

# Piece Layer
1 BIOS MMIO High Granularity Size = 1024G firmware
2 Kernel patch: size bridge windows for alignment padding kernel
3 Kernel patch: program ReBAR before enumeration kernel
4 pci=realloc kernel cmdline
5 p2p-bar1: BAR1 P2P HALs + peer PTE rewrite driver
6 p2p-mailbox-skip: skip mailbox peer pre-registration driver
7 p2p-readcap-override: restore the misjudged read cap driver
8 p2p-force-caps: force advertised caps (not optional) driver
9 RMForceStaticBar1=1;RMPcieP2PType=1 module params

Number 6 deserves a note: it actively disables the mailbox peer pre-registration. _kbusInitP2P_GM107 unconditionally registers a mailbox peer id for every GPU pair at init, and the fourth BAR1 P2P gate explicitly refuses to coexist with a mailbox peer. Leave it in place and peerNumberMask stays non-zero, so BAR1 P2P is never selected.

So “the mailbox is still locked” isn’t a loose end — it’s a precondition of this approach.

One more trap in the module parameters: they must go on a single NVreg_RegistryDwords= line. modprobe does not merge duplicate assignments of the same parameter across files — a second options nvidia NVreg_RegistryDwords=... anywhere in /etc/modprobe.d silently replaces the whole set.


8. What it buys

Down to local-memory latency

Measured with NVIDIA’s own p2pBandwidthLatencyTest, the latency matrix is the most convincing number:

1
2
3
4
5
P2P=Disabled Latency (us)          P2P=Enabled Latency (us)
GPU 0 1 2 GPU 0 1 2
0 2.06 37.12 20.60 0 2.06 1.84 21.36
1 15.05 2.26 16.71 1 1.85 2.10 14.76
2 17.76 17.49 2.58 2 17.26 16.86 2.60

Intra-socket pairs go from 15–37 µs to ~2.0 µs. The diagonal — a GPU touching its own framebuffer — is 2.04–2.59 µs. P2P latency is now at local-memory scale. Only a genuinely direct path does that; a staging protocol cannot.

Bandwidth had a bonus in it. The check tool flagged ACS redirection enabled on all six downstream ports (ReqRedir+ CmpltRedir+), which forces peer traffic within a switch up to the root complex and back. The symptom was obvious in hindsight — same-switch pairs were measuring slower than cross-switch ones:

Bidirectional ACS redirect on ACS redirect off
Same switch (0↔1, 2↔3, 4↔5) 9.89 GB/s 12.98 GB/s
Cross switch, same socket 10.99 10.99
Cross socket 8.53 8.53

Only those three same-switch pairs moved; everything else stayed put — exactly the distribution you’d expect when redirection stops pushing traffic out of the switch. +31%.

Making it permanent has a GRUB trap: pci=disable_acs_redir= is semicolon-separated, and ; is a statement separator in GRUB’s own script language, so the parameter gets truncated at the first one. Escape them:

1
pci=disable_acs_redir=0000:03:08.0\;0000:03:10.0\;...

The cost is that GPUs under one switch merge into a single IOMMU group. On this box (iommu=pt, no VMs) that’s free — but don’t do it if you need to pass individual GPUs through to guests.


9. The ceiling

14 of 30 directed pairs pass. All 16 failures are cross-socket.

That’s not a configuration problem, it’s topology. GPU 0/1 are on socket 0 and GPU 2–5 on socket 1 (SYS in nvidia-smi topo -m); peer traffic would have to cross QPI, and the driver refuses.

Put differently: of the 14 pairs that are physically capable of it, 14 work. None missing.

The latency matrix side by side, and the alias-proof test's 14/30 result

Worth noting that the advertised flag stays honest — 0 for cross-socket pairs, 1 for intra-socket. Even with forced capability bits, the pair-level check still correctly rejects cross-socket, so nothing claims to work and then times out.

On verification, one thing has to be said plainly: do not trust cudaDeviceCanAccessPeer(), nvidia-smi topo -p2p, or cudaMemcpyPeer(). The first two report what the driver claims, and this build forces that claim to OK. The third silently stages through host memory when P2P is unavailable, so it succeeds either way.

The test I wrote dereferences the peer pointer from inside a kernel — that can only succeed through a real peer mapping — and seeds both buffers with distinct per-GPU, per-slot patterns, so a mapping that quietly aliases back to local memory gets caught instead of reading back as a pass.


10. The reproducible checklist

The full walkthrough, including the pitfalls, is in COOKBOOK.md in the repo. Shortest path:

The order matters — build your recovery path before doing anything risky:

  1. Get the BMC working (zero risk, pure upside). It can be configured from inside the OS over KCS, without reaching it on the network first:

    1
    2
    3
    4
    sudo ipmitool lan set 1 ipsrc static
    sudo ipmitool lan set 1 ipaddr <ip>
    sudo ipmitool user set password 2 '<pw>'
    sudo ipmitool mc reset cold # resets only the BMC; the host keeps running

    After that chassis power cycle is a real cold boot, and sol activate gives you a serial console.

  2. BIOS: Advanced → PCIe/PCI/PnP Configuration → MMIO High Granularity Size = 1024G. Don’t just pick the maximum — the requirement is known and 1024G is already 2× headroom, while MMIO High Base set too high is the setting that stops the machine POSTing.

  3. Build the kernel with the two patches from kernel-patches/. Boot it one-shot so a failure falls back on its own:

    1
    sudo grub-reboot "Advanced options for Ubuntu>Ubuntu, with Linux <version>-cmp"

    The entry name must match grub.cfg exactly — a wrong name silently boots the default. And grub-reboot is consumed by one boot; set it again before each attempt.

  4. Add pci=realloc to the kernel command line.

  5. Install the driver: sudo ./install.sh --p2p (it checks the BAR1 prerequisite first and refuses if unmet).

  6. Cold boot, then verify:

    1
    sudo ./tools/p2p-check.sh

Closing

Looking back, the time didn’t go into writing code. It went into two bad calls:

Once, asserting a root cause before I had the log (blaming RMForceStaticBar1). Once, taking the upstream “optional” label at face value (p2p-force-caps is a hard prerequisite).

Both are the same pattern: I chose to infer when I could have verified.

The second time taught me something. When P2P was being refused and I didn’t know which of four gates was failing, I stopped enumerating possibilities and added one print statement. Three minutes to a definite answer, replacing several rounds of guesswork.

When you catch yourself saying “it could be A or it could be B,” ask first: is there an observation that separates them? Adding a log line or running a read-only command is almost always cheaper than another round of trial and error.

And the piece of evidence that settled it was nothing at all — not a single line printed.