1.. SPDX-License-Identifier: GPL-2.0 2 3=========================================== 4Userspace block device driver (ublk driver) 5=========================================== 6 7Overview 8======== 9 10ublk is a generic framework for implementing block device logic from userspace. 11The motivation behind it is that moving virtual block drivers into userspace, 12such as loop, nbd and similar can be very helpful. It can help to implement 13new virtual block device such as ublk-qcow2 (there are several attempts of 14implementing qcow2 driver in kernel). 15 16Userspace block devices are attractive because: 17 18- They can be written many programming languages. 19- They can use libraries that are not available in the kernel. 20- They can be debugged with tools familiar to application developers. 21- Crashes do not kernel panic the machine. 22- Bugs are likely to have a lower security impact than bugs in kernel 23 code. 24- They can be installed and updated independently of the kernel. 25- They can be used to simulate block device easily with user specified 26 parameters/setting for test/debug purpose 27 28ublk block device (``/dev/ublkb*``) is added by ublk driver. Any IO request 29on the device will be forwarded to ublk userspace program. For convenience, 30in this document, ``ublk server`` refers to generic ublk userspace 31program. ``ublksrv`` [#userspace]_ is one of such implementation. It 32provides ``libublksrv`` [#userspace_lib]_ library for developing specific 33user block device conveniently, while also generic type block device is 34included, such as loop and null. Richard W.M. Jones wrote userspace nbd device 35``nbdublk`` [#userspace_nbdublk]_ based on ``libublksrv`` [#userspace_lib]_. 36 37After the IO is handled by userspace, the result is committed back to the 38driver, thus completing the request cycle. This way, any specific IO handling 39logic is totally done by userspace, such as loop's IO handling, NBD's IO 40communication, or qcow2's IO mapping. 41 42``/dev/ublkb*`` is driven by blk-mq request-based driver. Each request is 43assigned by one queue wide unique tag. ublk server assigns unique tag to each 44IO too, which is 1:1 mapped with IO of ``/dev/ublkb*``. 45 46Both the IO request forward and IO handling result committing are done via 47``io_uring`` passthrough command; that is why ublk is also one io_uring based 48block driver. It has been observed that using io_uring passthrough command can 49give better IOPS than block IO; which is why ublk is one of high performance 50implementation of userspace block device: not only IO request communication is 51done by io_uring, but also the preferred IO handling in ublk server is io_uring 52based approach too. 53 54ublk provides control interface to set/get ublk block device parameters. 55The interface is extendable and kabi compatible: basically any ublk request 56queue's parameter or ublk generic feature parameters can be set/get via the 57interface. Thus, ublk is generic userspace block device framework. 58For example, it is easy to setup a ublk device with specified block 59parameters from userspace. 60 61Using ublk 62========== 63 64ublk requires userspace ublk server to handle real block device logic. 65 66Below is example of using ``ublksrv`` to provide ublk-based loop device. 67 68- add a device:: 69 70 ublk add -t loop -f ublk-loop.img 71 72- format with xfs, then use it:: 73 74 mkfs.xfs /dev/ublkb0 75 mount /dev/ublkb0 /mnt 76 # do anything. all IOs are handled by io_uring 77 ... 78 umount /mnt 79 80- list the devices with their info:: 81 82 ublk list 83 84- delete the device:: 85 86 ublk del -a 87 ublk del -n $ublk_dev_id 88 89See usage details in README of ``ublksrv`` [#userspace_readme]_. 90 91Design 92====== 93 94Control plane 95------------- 96 97ublk driver provides global misc device node (``/dev/ublk-control``) for 98managing and controlling ublk devices with help of several control commands: 99 100- ``UBLK_CMD_ADD_DEV`` 101 102 Add a ublk char device (``/dev/ublkc*``) which is talked with ublk server 103 WRT IO command communication. Basic device info is sent together with this 104 command. It sets UAPI structure of ``ublksrv_ctrl_dev_info``, 105 such as ``nr_hw_queues``, ``queue_depth``, and max IO request buffer size, 106 for which the info is negotiated with the driver and sent back to the server. 107 When this command is completed, the basic device info is immutable. 108 109- ``UBLK_CMD_SET_PARAMS`` / ``UBLK_CMD_GET_PARAMS`` 110 111 Set or get parameters of the device, which can be either generic feature 112 related, or request queue limit related, but can't be IO logic specific, 113 because the driver does not handle any IO logic. This command has to be 114 sent before sending ``UBLK_CMD_START_DEV``. 115 116- ``UBLK_CMD_START_DEV`` 117 118 After the server prepares userspace resources (such as creating I/O handler 119 threads & io_uring for handling ublk IO), this command is sent to the 120 driver for allocating & exposing ``/dev/ublkb*``. Parameters set via 121 ``UBLK_CMD_SET_PARAMS`` are applied for creating the device. 122 123- ``UBLK_CMD_STOP_DEV`` 124 125 Halt IO on ``/dev/ublkb*`` and remove the device. When this command returns, 126 ublk server will release resources (such as destroying I/O handler threads & 127 io_uring). 128 129- ``UBLK_CMD_DEL_DEV`` 130 131 Remove ``/dev/ublkc*``. When this command returns, the allocated ublk device 132 number can be reused. 133 134- ``UBLK_CMD_GET_QUEUE_AFFINITY`` 135 136 When ``/dev/ublkc`` is added, the driver creates block layer tagset, so 137 that each queue's affinity info is available. The server sends 138 ``UBLK_CMD_GET_QUEUE_AFFINITY`` to retrieve queue affinity info. It can 139 set up the per-queue context efficiently, such as bind affine CPUs with IO 140 pthread and try to allocate buffers in IO thread context. 141 142- ``UBLK_CMD_GET_DEV_INFO`` 143 144 For retrieving device info via ``ublksrv_ctrl_dev_info``. It is the server's 145 responsibility to save IO target specific info in userspace. 146 147- ``UBLK_CMD_GET_DEV_INFO2`` 148 Same purpose with ``UBLK_CMD_GET_DEV_INFO``, but ublk server has to 149 provide path of the char device of ``/dev/ublkc*`` for kernel to run 150 permission check, and this command is added for supporting unprivileged 151 ublk device, and introduced with ``UBLK_F_UNPRIVILEGED_DEV`` together. 152 Only the user owning the requested device can retrieve the device info. 153 154 How to deal with userspace/kernel compatibility: 155 156 1) if kernel is capable of handling ``UBLK_F_UNPRIVILEGED_DEV`` 157 158 If ublk server supports ``UBLK_F_UNPRIVILEGED_DEV``: 159 160 ublk server should send ``UBLK_CMD_GET_DEV_INFO2``, given anytime 161 unprivileged application needs to query devices the current user owns, 162 when the application has no idea if ``UBLK_F_UNPRIVILEGED_DEV`` is set 163 given the capability info is stateless, and application should always 164 retrieve it via ``UBLK_CMD_GET_DEV_INFO2`` 165 166 If ublk server doesn't support ``UBLK_F_UNPRIVILEGED_DEV``: 167 168 ``UBLK_CMD_GET_DEV_INFO`` is always sent to kernel, and the feature of 169 UBLK_F_UNPRIVILEGED_DEV isn't available for user 170 171 2) if kernel isn't capable of handling ``UBLK_F_UNPRIVILEGED_DEV`` 172 173 If ublk server supports ``UBLK_F_UNPRIVILEGED_DEV``: 174 175 ``UBLK_CMD_GET_DEV_INFO2`` is tried first, and will be failed, then 176 ``UBLK_CMD_GET_DEV_INFO`` needs to be retried given 177 ``UBLK_F_UNPRIVILEGED_DEV`` can't be set 178 179 If ublk server doesn't support ``UBLK_F_UNPRIVILEGED_DEV``: 180 181 ``UBLK_CMD_GET_DEV_INFO`` is always sent to kernel, and the feature of 182 ``UBLK_F_UNPRIVILEGED_DEV`` isn't available for user 183 184- ``UBLK_CMD_START_USER_RECOVERY`` 185 186 This command is valid if ``UBLK_F_USER_RECOVERY`` feature is enabled. This 187 command is accepted after the old process has exited, ublk device is quiesced 188 and ``/dev/ublkc*`` is released. User should send this command before he starts 189 a new process which re-opens ``/dev/ublkc*``. When this command returns, the 190 ublk device is ready for the new process. 191 192- ``UBLK_CMD_END_USER_RECOVERY`` 193 194 This command is valid if ``UBLK_F_USER_RECOVERY`` feature is enabled. This 195 command is accepted after ublk device is quiesced and a new process has 196 opened ``/dev/ublkc*`` and get all ublk queues be ready. When this command 197 returns, ublk device is unquiesced and new I/O requests are passed to the 198 new process. 199 200- user recovery feature description 201 202 Three new features are added for user recovery: ``UBLK_F_USER_RECOVERY``, 203 ``UBLK_F_USER_RECOVERY_REISSUE``, and ``UBLK_F_USER_RECOVERY_FAIL_IO``. To 204 enable recovery of ublk devices after the ublk server exits, the ublk server 205 should specify the ``UBLK_F_USER_RECOVERY`` flag when creating the device. The 206 ublk server may additionally specify at most one of 207 ``UBLK_F_USER_RECOVERY_REISSUE`` and ``UBLK_F_USER_RECOVERY_FAIL_IO`` to 208 modify how I/O is handled while the ublk server is dying/dead (this is called 209 the ``nosrv`` case in the driver code). 210 211 With just ``UBLK_F_USER_RECOVERY`` set, after the ublk server exits, 212 ublk does not delete ``/dev/ublkb*`` during the whole 213 recovery stage and ublk device ID is kept. It is ublk server's 214 responsibility to recover the device context by its own knowledge. 215 Requests which have not been issued to userspace are requeued. Requests 216 which have been issued to userspace are aborted. 217 218 With ``UBLK_F_USER_RECOVERY_REISSUE`` additionally set, after the ublk server 219 exits, contrary to ``UBLK_F_USER_RECOVERY``, 220 requests which have been issued to userspace are requeued and will be 221 re-issued to the new process after handling ``UBLK_CMD_END_USER_RECOVERY``. 222 ``UBLK_F_USER_RECOVERY_REISSUE`` is designed for backends who tolerate 223 double-write since the driver may issue the same I/O request twice. It 224 might be useful to a read-only FS or a VM backend. 225 226 With ``UBLK_F_USER_RECOVERY_FAIL_IO`` additionally set, after the ublk server 227 exits, requests which have issued to userspace are failed, as are any 228 subsequently issued requests. Applications continuously issuing I/O against 229 devices with this flag set will see a stream of I/O errors until a new ublk 230 server recovers the device. 231 232Unprivileged ublk device is supported by passing ``UBLK_F_UNPRIVILEGED_DEV``. 233Once the flag is set, all control commands can be sent by unprivileged 234user. Except for command of ``UBLK_CMD_ADD_DEV``, permission check on 235the specified char device(``/dev/ublkc*``) is done for all other control 236commands by ublk driver, for doing that, path of the char device has to 237be provided in these commands' payload from ublk server. With this way, 238ublk device becomes container-ware, and device created in one container 239can be controlled/accessed just inside this container. 240 241Data plane 242---------- 243 244The ublk server should create dedicated threads for handling I/O. Each 245thread should have its own io_uring through which it is notified of new 246I/O, and through which it can complete I/O. These dedicated threads 247should focus on IO handling and shouldn't handle any control & 248management tasks. 249 250The's IO is assigned by a unique tag, which is 1:1 mapping with IO 251request of ``/dev/ublkb*``. 252 253UAPI structure of ``ublksrv_io_desc`` is defined for describing each IO from 254the driver. A fixed mmapped area (array) on ``/dev/ublkc*`` is provided for 255exporting IO info to the server; such as IO offset, length, OP/flags and 256buffer address. Each ``ublksrv_io_desc`` instance can be indexed via queue id 257and IO tag directly. 258 259The following IO commands are communicated via io_uring passthrough command, 260and each command is only for forwarding the IO and committing the result 261with specified IO tag in the command data: 262 263Traditional Per-I/O Commands 264~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 265 266- ``UBLK_U_IO_FETCH_REQ`` 267 268 Sent from the server I/O pthread for fetching future incoming I/O requests 269 destined to ``/dev/ublkb*``. This command is sent only once from the server 270 IO pthread for ublk driver to setup IO forward environment. 271 272 Once a thread issues this command against a given (qid,tag) pair, the thread 273 registers itself as that I/O's daemon. In the future, only that I/O's daemon 274 is allowed to issue commands against the I/O. If any other thread attempts 275 to issue a command against a (qid,tag) pair for which the thread is not the 276 daemon, the command will fail. Daemons can be reset only be going through 277 recovery. 278 279 The ability for every (qid,tag) pair to have its own independent daemon task 280 is indicated by the ``UBLK_F_PER_IO_DAEMON`` feature. If this feature is not 281 supported by the driver, daemons must be per-queue instead - i.e. all I/Os 282 associated to a single qid must be handled by the same task. 283 284- ``UBLK_U_IO_COMMIT_AND_FETCH_REQ`` 285 286 When an IO request is destined to ``/dev/ublkb*``, the driver stores 287 the IO's ``ublksrv_io_desc`` to the specified mapped area; then the 288 previous received IO command of this IO tag (either ``UBLK_IO_FETCH_REQ`` 289 or ``UBLK_IO_COMMIT_AND_FETCH_REQ)`` is completed, so the server gets 290 the IO notification via io_uring. 291 292 After the server handles the IO, its result is committed back to the 293 driver by sending ``UBLK_IO_COMMIT_AND_FETCH_REQ`` back. Once ublkdrv 294 received this command, it parses the result and complete the request to 295 ``/dev/ublkb*``. In the meantime setup environment for fetching future 296 requests with the same IO tag. That is, ``UBLK_IO_COMMIT_AND_FETCH_REQ`` 297 is reused for both fetching request and committing back IO result. 298 299- ``UBLK_U_IO_NEED_GET_DATA`` 300 301 With ``UBLK_F_NEED_GET_DATA`` enabled, the WRITE request will be firstly 302 issued to ublk server without data copy. Then, IO backend of ublk server 303 receives the request and it can allocate data buffer and embed its addr 304 inside this new io command. After the kernel driver gets the command, 305 data copy is done from request pages to this backend's buffer. Finally, 306 backend receives the request again with data to be written and it can 307 truly handle the request. 308 309 ``UBLK_IO_NEED_GET_DATA`` adds one additional round-trip and one 310 io_uring_enter() syscall. Any user thinks that it may lower performance 311 should not enable UBLK_F_NEED_GET_DATA. ublk server pre-allocates IO 312 buffer for each IO by default. Any new project should try to use this 313 buffer to communicate with ublk driver. However, existing project may 314 break or not able to consume the new buffer interface; that's why this 315 command is added for backwards compatibility so that existing projects 316 can still consume existing buffers. 317 318- data copy between ublk server IO buffer and ublk block IO request 319 320 The driver needs to copy the block IO request pages into the server buffer 321 (pages) first for WRITE before notifying the server of the coming IO, so 322 that the server can handle WRITE request. 323 324 When the server handles READ request and sends 325 ``UBLK_IO_COMMIT_AND_FETCH_REQ`` to the server, ublkdrv needs to copy 326 the server buffer (pages) read to the IO request pages. 327 328Batch I/O Commands (UBLK_F_BATCH_IO) 329~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 330 331The ``UBLK_F_BATCH_IO`` feature provides an alternative high-performance 332I/O handling model that replaces the traditional per-I/O commands with 333per-queue batch commands. This significantly reduces communication overhead 334and enables better load balancing across multiple server tasks. 335 336Key differences from traditional mode: 337 338- **Per-queue vs Per-I/O**: Commands operate on queues rather than individual I/Os 339- **Batch processing**: Multiple I/Os are handled in single operations 340- **Multishot commands**: Use io_uring multishot for reduced submission overhead 341- **Flexible task assignment**: Any task can handle any I/O (no per-I/O daemons) 342- **Better load balancing**: Tasks can adjust their workload dynamically 343 344Batch I/O Commands: 345 346- ``UBLK_U_IO_PREP_IO_CMDS`` 347 348 Prepares multiple I/O commands in batch. The server provides a buffer 349 containing multiple I/O descriptors that will be processed together. 350 This reduces the number of individual command submissions required. 351 352- ``UBLK_U_IO_COMMIT_IO_CMDS`` 353 354 Commits results for multiple I/O operations in batch, and prepares the 355 I/O descriptors to accept new requests. The server provides a buffer 356 containing the results of multiple completed I/Os, allowing efficient 357 bulk completion of requests. 358 359- ``UBLK_U_IO_FETCH_IO_CMDS`` 360 361 **Multishot command** for fetching I/O commands in batch. This is the key 362 command that enables high-performance batch processing: 363 364 * Uses io_uring multishot capability for reduced submission overhead 365 * Single command can fetch multiple I/O requests over time 366 * Buffer size determines maximum batch size per operation 367 * Multiple fetch commands can be submitted for load balancing 368 * Only one fetch command is active at any time per queue 369 * Supports dynamic load balancing across multiple server tasks 370 371 It is one typical multishot io_uring request with provided buffer, and it 372 won't be completed until any failure is triggered. 373 374 Each task can submit ``UBLK_U_IO_FETCH_IO_CMDS`` with different buffer 375 sizes to control how much work it handles. This enables sophisticated 376 load balancing strategies in multi-threaded servers. 377 378Migration: Applications using traditional commands (``UBLK_U_IO_FETCH_REQ``, 379``UBLK_U_IO_COMMIT_AND_FETCH_REQ``) cannot use batch mode simultaneously. 380 381Zero copy 382--------- 383 384ublk zero copy relies on io_uring's fixed kernel buffer, which provides 385two APIs: `io_buffer_register_bvec()` and `io_buffer_unregister_bvec`. 386 387ublk adds IO command of `UBLK_IO_REGISTER_IO_BUF` to call 388`io_buffer_register_bvec()` for ublk server to register client request 389buffer into io_uring buffer table, then ublk server can submit io_uring 390IOs with the registered buffer index. IO command of `UBLK_IO_UNREGISTER_IO_BUF` 391calls `io_buffer_unregister_bvec()` to unregister the buffer, which is 392guaranteed to be live between calling `io_buffer_register_bvec()` and 393`io_buffer_unregister_bvec()`. Any io_uring operation which supports this 394kind of kernel buffer will grab one reference of the buffer until the 395operation is completed. 396 397ublk server implementing zero copy or user copy has to be CAP_SYS_ADMIN and 398be trusted, because it is ublk server's responsibility to make sure IO buffer 399filled with data for handling read command, and ublk server has to return 400correct result to ublk driver when handling READ command, and the result 401has to match with how many bytes filled to the IO buffer. Otherwise, 402uninitialized kernel IO buffer will be exposed to client application. 403 404ublk server needs to align the parameter of `struct ublk_param_dma_align` 405with backend for zero copy to work correctly. 406 407For reaching best IO performance, ublk server should align its segment 408parameter of `struct ublk_param_segment` with backend for avoiding 409unnecessary IO split, which usually hurts io_uring performance. 410 411Auto Buffer Registration 412------------------------ 413 414The ``UBLK_F_AUTO_BUF_REG`` feature automatically handles buffer registration 415and unregistration for I/O requests, which simplifies the buffer management 416process and reduces overhead in the ublk server implementation. 417 418This is another feature flag for using zero copy, and it is compatible with 419``UBLK_F_SUPPORT_ZERO_COPY``. 420 421Feature Overview 422~~~~~~~~~~~~~~~~ 423 424This feature automatically registers request buffers to the io_uring context 425before delivering I/O commands to the ublk server and unregisters them when 426completing I/O commands. This eliminates the need for manual buffer 427registration/unregistration via ``UBLK_IO_REGISTER_IO_BUF`` and 428``UBLK_IO_UNREGISTER_IO_BUF`` commands, then IO handling in ublk server 429can avoid dependency on the two uring_cmd operations. 430 431IOs can't be issued concurrently to io_uring if there is any dependency 432among these IOs. So this way not only simplifies ublk server implementation, 433but also makes concurrent IO handling becomes possible by removing the 434dependency on buffer registration & unregistration commands. 435 436Usage Requirements 437~~~~~~~~~~~~~~~~~~ 438 4391. The ublk server must create a sparse buffer table on the same ``io_ring_ctx`` 440 used for ``UBLK_IO_FETCH_REQ`` and ``UBLK_IO_COMMIT_AND_FETCH_REQ``. If 441 uring_cmd is issued on a different ``io_ring_ctx``, manual buffer 442 unregistration is required. 443 4442. Buffer registration data must be passed via uring_cmd's ``sqe->addr`` with the 445 following structure:: 446 447 struct ublk_auto_buf_reg { 448 __u16 index; /* Buffer index for registration */ 449 __u8 flags; /* Registration flags */ 450 __u8 reserved0; /* Reserved for future use */ 451 __u32 reserved1; /* Reserved for future use */ 452 }; 453 454 ublk_auto_buf_reg_to_sqe_addr() is for converting the above structure into 455 ``sqe->addr``. 456 4573. All reserved fields in ``ublk_auto_buf_reg`` must be zeroed. 458 4594. Optional flags can be passed via ``ublk_auto_buf_reg.flags``. 460 461Fallback Behavior 462~~~~~~~~~~~~~~~~~ 463 464If auto buffer registration fails: 465 4661. When ``UBLK_AUTO_BUF_REG_FALLBACK`` is enabled: 467 468 - The uring_cmd is completed 469 - ``UBLK_IO_F_NEED_REG_BUF`` is set in ``ublksrv_io_desc.op_flags`` 470 - The ublk server must manually deal with the failure, such as, register 471 the buffer manually, or using user copy feature for retrieving the data 472 for handling ublk IO 473 4742. If fallback is not enabled: 475 476 - The ublk I/O request fails silently 477 - The uring_cmd won't be completed 478 479Limitations 480~~~~~~~~~~~ 481 482- Requires same ``io_ring_ctx`` for all operations 483- May require manual buffer management in fallback cases 484- io_ring_ctx buffer table has a max size of 16K, which may not be enough 485 in case that too many ublk devices are handled by this single io_ring_ctx 486 and each one has very large queue depth 487 488References 489========== 490 491.. [#userspace] https://github.com/ming1/ubdsrv 492 493.. [#userspace_lib] https://github.com/ming1/ubdsrv/tree/master/lib 494 495.. [#userspace_nbdublk] https://gitlab.com/rwmjones/libnbd/-/tree/nbdublk 496 497.. [#userspace_readme] https://github.com/ming1/ubdsrv/blob/master/README 498