1============== 2UALink Support 3============== 4 5Overview 6======== 7 8Connected GPUs in a pod can directly access the remove memory on another GPU 9over UALink. Unlike RMDA, there is no copy involved; it is direct loads/stores 10over the fabric. Shared memory can only be accessed by a remote GPU if the 11memory was exported and the importer has been authorized. For the memory to be 12shared, it must be part of a unified physical address space shared between 13nodes. This address space is called NPA (Nework Physical Address) space. This 14address space is partitioned between the GPUs so that each GPU has its own 15segment of the address space in which to export its memory. Each GPU maintains 16a dedicated set of page tables for their NPA space similar to GPUVM. Note that 17this mechanism only allows for GPU access to remote memory. The remote memory 18is not CPU accessible. 19 20Exported memory is pinned. This is similar to how dma-bufs in VRAM are pinned 21for P2P access. Remote TLB shootdowns from the exporter are used when the 22exported memory is freed in order to remove access by remote GPUs. 23 24To access remote memory, the driver can map NPA addresses into its per process 25GPUVM page tables just like local memory. Applications use opaque handles to 26represent remote memory. GPUs in a pod communicate with eachother directly to 27exchange NPA addresses between importers and exporters. If a node goes offline 28or is reset, their peers will clean up any remaining refrences that are lost 29when that happens. NPA addresses are exchanged directly between the kernel 30drivers using the scale up fabric. NPA addresses are never exposed to user 31space. 32 33On the importer, the NPA space is like another physical address space. NPA 34addresses can be used as physical addresses for GPUVM to provide GPU virtual 35addresses to the memory for processes using the GPU. 36 37On the exporter, the NPA space provides a way to expose discontiguous local 38memory as a contiguous address range for remote GPUs. This allows the exporter 39to locally manage the pages mapped into the NPA space. 40 41Remote NPAs are managed like another device specific TTM pool similar to 42doorbells or VRAM, however they cannot be CPU mapped. 43 44 45User Interface 46============== 47Two IOCTLs are provided to export and import remote memory. 48 49Export Memory 50------------- 51To export memory, a UALINK handle must be created for an allocation that can be 52shared with another node in the pod. To do this the exporter calls the GEM 53UALink IOCTL with the GEM handle to the buffer it wants to export. The IOCTL 54returns a unique 128 bit handle which can be shared with the remote host. 55Calling export on the same GEM handle always returns the same UALink handle. 56The UALink handle is destroyed when the GEM object reference count reaches 0. 57 58Import Memory 59------------- 60To import remote memory, the UALink handle from the remote node must be 61converted from a UALink handle to a local GEM object which represents the local 62reference to the NPA space on the importer. If the memory has already been 63imported, it just returns a new reference to the existing GEM object. If not, 64the importer queries the exporter to get the NPA address. Once it has that, the 65importer can create the GEM to represent the NPA space used by the allocation. 66The GEM object is then exported to the caller as a dma-buf. The dma-buf is 67leveraged for dynamic attachment which provides the ability to revoke access 68when necessary. 69 70 71Device to Device Communications 72=============================== 73 74Devices communicate via a protocol implemented in firmware. Mesages sent to a 75remote node generate an interrupt on that node for servicing. 76