P2P over 2 nodes (pcie over fiber)

[chardog]

NVIDIA Open GPU Kernel Modules Version

Na

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu

Kernel Release

Na

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

4090

Describe the bug

If I connect two nodes with two mellanox cards (pcie over fiber) and I have 4090 gpus in both servers, will the p2p communication work correctly over NCCL and RDMA?

To Reproduce

Q

Bug Incidence

Once

nvidia-bug-report.log.gz

Q

More Info

If I connect two nodes with two mellanox cards (pcie over fiber) and I have 4090 gpus in both servers, will the p2p communication work correctly over NCCL and RDMA?

[chardog]

If I connect two nodes with two mellanox cards (pcie over fiber) and I have 4090 gpus in both servers, will the p2p communication work correctly over NCCL and RDMA?

[zvorinji]

No. 1) NCCL will block it. But you can get around this by modifying it. 2) Once you've modified it, you will run into the issue where the NIC can't access the memory. Local protection error/bad address. I think the address needs to be translated into a BAR1 address (somehow/somewhere) for this to work. It's possible you also need to modify page sizes and modify the peermem module to incorporate a BAR1 workaround (given lack of GPUDirect RDMA support).

[nwpumabian]

No. 1) NCCL will block it. But you can get around this by modifying it. 2) Once you've modified it, you will run into the issue where the NIC can't access the memory. Local protection error/bad address. I think the address needs to be translated into a BAR1 address (somehow/somewhere) for this to work. It's possible you also need to modify page sizes and modify the peermem module to incorporate a BAR1 workaround (given lack of GPUDirect RDMA support).

Is it possible to perform RDMA reads on a single 4090 card? Can gdrcopy run using RDMA on a single 4090?

[zvorinji]

No. 1) NCCL will block it. But you can get around this by modifying it. 2) Once you've modified it, you will run into the issue where the NIC can't access the memory. Local protection error/bad address. I think the address needs to be translated into a BAR1 address (somehow/somewhere) for this to work. It's possible you also need to modify page sizes and modify the peermem module to incorporate a BAR1 workaround (given lack of GPUDirect RDMA support).

Is it possible to perform RDMA reads on a single 4090 card? Can gdrcopy run using RDMA on a single 4090?

No.

[bendchen]

I suspect this might be the reason:
Peer-to-Peer (P2P) is limited to GPUs within the same host.
NVIDIA GPU P2P allows direct access to video memory between GPUs within the same physical server via PCIe or NVLink.
P2P is not available for GPUs across servers because there is no direct physical PCIe/NVLink connection.