xref: /linux/Documentation/gpu/amdgpu/ualink.rst (revision 4f9ab6eb259437701991c2af3fbd666f2fbc4e19)
1==============
2UALink Support
3==============
4
5Overview
6========
7
8Connected GPUs in a pod can directly access the remove memory on another GPU
9over UALink.  Unlike RMDA, there is no copy involved; it is direct loads/stores
10over the fabric.  Shared memory can only be accessed by a remote GPU if the
11memory was exported and the importer has been authorized. For the memory to be
12shared, it must be part of a unified physical address space shared between
13nodes.  This address space is called NPA (Nework Physical Address) space.  This
14address space is partitioned between the GPUs so that each GPU has its own
15segment of the address space in which to export its memory.  Each GPU maintains
16a dedicated set of page tables for their NPA space similar to GPUVM.  Note that
17this mechanism only allows for GPU access to remote memory.  The remote memory
18is not CPU accessible.
19
20Exported memory is pinned.  This is similar to how dma-bufs in VRAM are pinned
21for P2P access.  Remote TLB shootdowns from the exporter are used when the
22exported memory is freed in order to remove access by remote GPUs.
23
24To access remote memory, the driver can map NPA addresses into its per process
25GPUVM page tables just like local memory.  Applications use opaque handles to
26represent remote memory.  GPUs in a pod communicate with eachother directly to
27exchange NPA addresses between importers and exporters.  If a node goes offline
28or is reset, their peers will clean up any remaining refrences that are lost
29when that happens.  NPA addresses are exchanged directly between the kernel
30drivers using the scale up fabric.  NPA addresses are never exposed to user
31space.
32
33On the importer, the NPA space is like another physical address space. NPA
34addresses can be used as physical addresses for GPUVM to provide GPU virtual
35addresses to the memory for processes using the GPU.
36
37On the exporter, the NPA space provides a way to expose discontiguous local
38memory as a contiguous address range for remote GPUs.  This allows the exporter
39to locally manage the pages mapped into the NPA space.
40
41Remote NPAs are managed like another device specific TTM pool similar to
42doorbells or VRAM, however they cannot be CPU mapped.
43
44
45User Interface
46==============
47Two IOCTLs are provided to export and import remote memory.
48
49Export Memory
50-------------
51To export memory, a UALINK handle must be created for an allocation that can be
52shared with another node in the pod.  To do this the exporter calls the GEM
53UALink IOCTL with the GEM handle to the buffer it wants to export.  The IOCTL
54returns a unique 128 bit handle which can be shared with the remote host.
55Calling export on the same GEM handle always returns the same UALink handle.
56The UALink handle is destroyed when the GEM object reference count reaches 0.
57
58Import Memory
59-------------
60To import remote memory, the UALink handle from the remote node must be
61converted from a UALink handle to a local GEM object which represents the local
62reference to the NPA space on the importer.  If the memory has already been
63imported, it just returns a new reference to the existing GEM object.  If not,
64the importer queries the exporter to get the NPA address.  Once it has that, the
65importer can create the GEM to represent the NPA space used by the allocation.
66The GEM object is then exported to the caller as a dma-buf. The dma-buf is
67leveraged for dynamic attachment which provides the ability to revoke access
68when necessary.
69
70
71Device to Device Communications
72===============================
73
74Devices communicate via a protocol implemented in firmware.  Mesages sent to a
75remote node generate an interrupt on that node for servicing.
76