1Buffer Sharing and Synchronization 2================================== 3 4The dma-buf subsystem provides the framework for sharing buffers for 5hardware (DMA) access across multiple device drivers and subsystems, and 6for synchronizing asynchronous hardware access. 7 8This is used, for example, by drm "prime" multi-GPU support, but is of 9course not limited to GPU use cases. 10 11The three main components of this are: (1) dma-buf, representing a 12sg_table and exposed to userspace as a file descriptor to allow passing 13between devices, (2) fence, which provides a mechanism to signal when 14one device has finished access, and (3) reservation, which manages the 15shared or exclusive fence(s) associated with the buffer. 16 17Shared DMA Buffers 18------------------ 19 20This document serves as a guide to device-driver writers on what is the dma-buf 21buffer sharing API, how to use it for exporting and using shared buffers. 22 23Any device driver which wishes to be a part of DMA buffer sharing, can do so as 24either the 'exporter' of buffers, or the 'user' or 'importer' of buffers. 25 26Say a driver A wants to use buffers created by driver B, then we call B as the 27exporter, and A as buffer-user/importer. 28 29The exporter 30 31 - implements and manages operations in :c:type:`struct dma_buf_ops 32 <dma_buf_ops>` for the buffer, 33 - allows other users to share the buffer by using dma_buf sharing APIs, 34 - manages the details of buffer allocation, wrapped in a :c:type:`struct 35 dma_buf <dma_buf>`, 36 - decides about the actual backing storage where this allocation happens, 37 - and takes care of any migration of scatterlist - for all (shared) users of 38 this buffer. 39 40The buffer-user 41 42 - is one of (many) sharing users of the buffer. 43 - doesn't need to worry about how the buffer is allocated, or where. 44 - and needs a mechanism to get access to the scatterlist that makes up this 45 buffer in memory, mapped into its own address space, so it can access the 46 same area of memory. This interface is provided by :c:type:`struct 47 dma_buf_attachment <dma_buf_attachment>`. 48 49Any exporters or users of the dma-buf buffer sharing framework must have a 50'select DMA_SHARED_BUFFER' in their respective Kconfigs. 51 52Userspace Interface Notes 53~~~~~~~~~~~~~~~~~~~~~~~~~ 54 55Mostly a DMA buffer file descriptor is simply an opaque object for userspace, 56and hence the generic interface exposed is very minimal. There's a few things to 57consider though: 58 59- Since kernel 3.12 the dma-buf FD supports the llseek system call, but only 60 with offset=0 and whence=SEEK_END|SEEK_SET. SEEK_SET is supported to allow 61 the usual size discover pattern size = SEEK_END(0); SEEK_SET(0). Every other 62 llseek operation will report -EINVAL. 63 64 If llseek on dma-buf FDs isn't support the kernel will report -ESPIPE for all 65 cases. Userspace can use this to detect support for discovering the dma-buf 66 size using llseek. 67 68- In order to avoid fd leaks on exec, the FD_CLOEXEC flag must be set 69 on the file descriptor. This is not just a resource leak, but a 70 potential security hole. It could give the newly exec'd application 71 access to buffers, via the leaked fd, to which it should otherwise 72 not be permitted access. 73 74 The problem with doing this via a separate fcntl() call, versus doing it 75 atomically when the fd is created, is that this is inherently racy in a 76 multi-threaded app[3]. The issue is made worse when it is library code 77 opening/creating the file descriptor, as the application may not even be 78 aware of the fd's. 79 80 To avoid this problem, userspace must have a way to request O_CLOEXEC 81 flag be set when the dma-buf fd is created. So any API provided by 82 the exporting driver to create a dmabuf fd must provide a way to let 83 userspace control setting of O_CLOEXEC flag passed in to dma_buf_fd(). 84 85- Memory mapping the contents of the DMA buffer is also supported. See the 86 discussion below on `CPU Access to DMA Buffer Objects`_ for the full details. 87 88- The DMA buffer FD is also pollable, see `Implicit Fence Poll Support`_ below for 89 details. 90 91Basic Operation and Device DMA Access 92~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 93 94.. kernel-doc:: drivers/dma-buf/dma-buf.c 95 :doc: dma buf device access 96 97CPU Access to DMA Buffer Objects 98~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 99 100.. kernel-doc:: drivers/dma-buf/dma-buf.c 101 :doc: cpu access 102 103Implicit Fence Poll Support 104~~~~~~~~~~~~~~~~~~~~~~~~~~~ 105 106.. kernel-doc:: drivers/dma-buf/dma-buf.c 107 :doc: implicit fence polling 108 109Kernel Functions and Structures Reference 110~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 111 112.. kernel-doc:: drivers/dma-buf/dma-buf.c 113 :export: 114 115.. kernel-doc:: include/linux/dma-buf.h 116 :internal: 117 118Buffer Mapping Helpers 119~~~~~~~~~~~~~~~~~~~~~~ 120 121.. kernel-doc:: include/linux/dma-buf-map.h 122 :doc: overview 123 124.. kernel-doc:: include/linux/dma-buf-map.h 125 :internal: 126 127Reservation Objects 128------------------- 129 130.. kernel-doc:: drivers/dma-buf/dma-resv.c 131 :doc: Reservation Object Overview 132 133.. kernel-doc:: drivers/dma-buf/dma-resv.c 134 :export: 135 136.. kernel-doc:: include/linux/dma-resv.h 137 :internal: 138 139DMA Fences 140---------- 141 142.. kernel-doc:: drivers/dma-buf/dma-fence.c 143 :doc: DMA fences overview 144 145DMA Fence Cross-Driver Contract 146~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 147 148.. kernel-doc:: drivers/dma-buf/dma-fence.c 149 :doc: fence cross-driver contract 150 151DMA Fence Signalling Annotations 152~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 153 154.. kernel-doc:: drivers/dma-buf/dma-fence.c 155 :doc: fence signalling annotation 156 157DMA Fences Functions Reference 158~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 159 160.. kernel-doc:: drivers/dma-buf/dma-fence.c 161 :export: 162 163.. kernel-doc:: include/linux/dma-fence.h 164 :internal: 165 166Seqno Hardware Fences 167~~~~~~~~~~~~~~~~~~~~~ 168 169.. kernel-doc:: include/linux/seqno-fence.h 170 :internal: 171 172DMA Fence Array 173~~~~~~~~~~~~~~~ 174 175.. kernel-doc:: drivers/dma-buf/dma-fence-array.c 176 :export: 177 178.. kernel-doc:: include/linux/dma-fence-array.h 179 :internal: 180 181DMA Fence uABI/Sync File 182~~~~~~~~~~~~~~~~~~~~~~~~ 183 184.. kernel-doc:: drivers/dma-buf/sync_file.c 185 :export: 186 187.. kernel-doc:: include/linux/sync_file.h 188 :internal: 189 190Indefinite DMA Fences 191~~~~~~~~~~~~~~~~~~~~~ 192 193At various times struct dma_fence with an indefinite time until dma_fence_wait() 194finishes have been proposed. Examples include: 195 196* Future fences, used in HWC1 to signal when a buffer isn't used by the display 197 any longer, and created with the screen update that makes the buffer visible. 198 The time this fence completes is entirely under userspace's control. 199 200* Proxy fences, proposed to handle &drm_syncobj for which the fence has not yet 201 been set. Used to asynchronously delay command submission. 202 203* Userspace fences or gpu futexes, fine-grained locking within a command buffer 204 that userspace uses for synchronization across engines or with the CPU, which 205 are then imported as a DMA fence for integration into existing winsys 206 protocols. 207 208* Long-running compute command buffers, while still using traditional end of 209 batch DMA fences for memory management instead of context preemption DMA 210 fences which get reattached when the compute job is rescheduled. 211 212Common to all these schemes is that userspace controls the dependencies of these 213fences and controls when they fire. Mixing indefinite fences with normal 214in-kernel DMA fences does not work, even when a fallback timeout is included to 215protect against malicious userspace: 216 217* Only the kernel knows about all DMA fence dependencies, userspace is not aware 218 of dependencies injected due to memory management or scheduler decisions. 219 220* Only userspace knows about all dependencies in indefinite fences and when 221 exactly they will complete, the kernel has no visibility. 222 223Furthermore the kernel has to be able to hold up userspace command submission 224for memory management needs, which means we must support indefinite fences being 225dependent upon DMA fences. If the kernel also support indefinite fences in the 226kernel like a DMA fence, like any of the above proposal would, there is the 227potential for deadlocks. 228 229.. kernel-render:: DOT 230 :alt: Indefinite Fencing Dependency Cycle 231 :caption: Indefinite Fencing Dependency Cycle 232 233 digraph "Fencing Cycle" { 234 node [shape=box bgcolor=grey style=filled] 235 kernel [label="Kernel DMA Fences"] 236 userspace [label="userspace controlled fences"] 237 kernel -> userspace [label="memory management"] 238 userspace -> kernel [label="Future fence, fence proxy, ..."] 239 240 { rank=same; kernel userspace } 241 } 242 243This means that the kernel might accidentally create deadlocks 244through memory management dependencies which userspace is unaware of, which 245randomly hangs workloads until the timeout kicks in. Workloads, which from 246userspace's perspective, do not contain a deadlock. In such a mixed fencing 247architecture there is no single entity with knowledge of all dependencies. 248Thefore preventing such deadlocks from within the kernel is not possible. 249 250The only solution to avoid dependencies loops is by not allowing indefinite 251fences in the kernel. This means: 252 253* No future fences, proxy fences or userspace fences imported as DMA fences, 254 with or without a timeout. 255 256* No DMA fences that signal end of batchbuffer for command submission where 257 userspace is allowed to use userspace fencing or long running compute 258 workloads. This also means no implicit fencing for shared buffers in these 259 cases. 260