1.. SPDX-License-Identifier: GPL-2.0 2 3.. _stateless_decoder: 4 5************************************************** 6Memory-to-memory Stateless Video Decoder Interface 7************************************************** 8 9A stateless decoder is a decoder that works without retaining any kind of state 10between processed frames. This means that each frame is decoded independently 11of any previous and future frames, and that the client is responsible for 12maintaining the decoding state and providing it to the decoder with each 13decoding request. This is in contrast to the stateful video decoder interface, 14where the hardware and driver maintain the decoding state and all the client 15has to do is to provide the raw encoded stream and dequeue decoded frames in 16display order. 17 18This section describes how user-space ("the client") is expected to communicate 19with stateless decoders in order to successfully decode an encoded stream. 20Compared to stateful codecs, the decoder/client sequence is simpler, but the 21cost of this simplicity is extra complexity in the client which is responsible 22for maintaining a consistent decoding state. 23 24Stateless decoders make use of the :ref:`media-request-api`. A stateless 25decoder must expose the ``V4L2_BUF_CAP_SUPPORTS_REQUESTS`` capability on its 26``OUTPUT`` queue when :c:func:`VIDIOC_REQBUFS` or :c:func:`VIDIOC_CREATE_BUFS` 27are invoked. 28 29Depending on the encoded formats supported by the decoder, a single decoded 30frame may be the result of several decode requests (for instance, H.264 streams 31with multiple slices per frame). Decoders that support such formats must also 32expose the ``V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`` capability on their 33``OUTPUT`` queue. 34 35Querying capabilities 36===================== 37 381. To enumerate the set of coded formats supported by the decoder, the client 39 calls :c:func:`VIDIOC_ENUM_FMT` on the ``OUTPUT`` queue. 40 41 * The driver must always return the full set of supported ``OUTPUT`` formats, 42 irrespective of the format currently set on the ``CAPTURE`` queue. 43 44 * Simultaneously, the driver must restrain the set of values returned by 45 codec-specific capability controls (such as H.264 profiles) to the set 46 actually supported by the hardware. 47 482. To enumerate the set of supported raw formats, the client calls 49 :c:func:`VIDIOC_ENUM_FMT` on the ``CAPTURE`` queue. 50 51 * The driver must return only the formats supported for the format currently 52 active on the ``OUTPUT`` queue. 53 54 * Depending on the currently set ``OUTPUT`` format, the set of supported raw 55 formats may depend on the value of some codec-dependent controls. 56 The client is responsible for making sure that these controls are set 57 before querying the ``CAPTURE`` queue. Failure to do so will result in the 58 default values for these controls being used, and a returned set of formats 59 that may not be usable for the media the client is trying to decode. 60 613. The client may use :c:func:`VIDIOC_ENUM_FRAMESIZES` to detect supported 62 resolutions for a given format, passing desired pixel format in 63 :c:type:`v4l2_frmsizeenum`'s ``pixel_format``. 64 654. Supported profiles and levels for the current ``OUTPUT`` format, if 66 applicable, may be queried using their respective controls via 67 :c:func:`VIDIOC_QUERYCTRL`. 68 69Initialization 70============== 71 721. Set the coded format on the ``OUTPUT`` queue via :c:func:`VIDIOC_S_FMT`. 73 74 * **Required fields:** 75 76 ``type`` 77 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. 78 79 ``pixelformat`` 80 a coded pixel format. 81 82 ``width``, ``height`` 83 coded width and height parsed from the stream. 84 85 other fields 86 follow standard semantics. 87 88 .. note:: 89 90 Changing the ``OUTPUT`` format may change the currently set ``CAPTURE`` 91 format. The driver will derive a new ``CAPTURE`` format from the 92 ``OUTPUT`` format being set, including resolution, colorimetry 93 parameters, etc. If the client needs a specific ``CAPTURE`` format, 94 it must adjust it afterwards. 95 962. Call :c:func:`VIDIOC_S_EXT_CTRLS` to set all the controls (parsed headers, 97 etc.) required by the ``OUTPUT`` format to enumerate the ``CAPTURE`` formats. 98 993. Call :c:func:`VIDIOC_G_FMT` for ``CAPTURE`` queue to get the format for the 100 destination buffers parsed/decoded from the bytestream. 101 102 * **Required fields:** 103 104 ``type`` 105 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 106 107 * **Returned fields:** 108 109 ``width``, ``height`` 110 frame buffer resolution for the decoded frames. 111 112 ``pixelformat`` 113 pixel format for decoded frames. 114 115 ``num_planes`` (for _MPLANE ``type`` only) 116 number of planes for pixelformat. 117 118 ``sizeimage``, ``bytesperline`` 119 as per standard semantics; matching frame buffer format. 120 121 .. note:: 122 123 The value of ``pixelformat`` may be any pixel format supported for the 124 ``OUTPUT`` format, based on the hardware capabilities. It is suggested 125 that the driver chooses the preferred/optimal format for the current 126 configuration. For example, a YUV format may be preferred over an RGB 127 format, if an additional conversion step would be required for RGB. 128 1294. *[optional]* Enumerate ``CAPTURE`` formats via :c:func:`VIDIOC_ENUM_FMT` on 130 the ``CAPTURE`` queue. The client may use this ioctl to discover which 131 alternative raw formats are supported for the current ``OUTPUT`` format and 132 select one of them via :c:func:`VIDIOC_S_FMT`. 133 134 .. note:: 135 136 The driver will return only formats supported for the currently selected 137 ``OUTPUT`` format and currently set controls, even if more formats may be 138 supported by the decoder in general. 139 140 For example, a decoder may support YUV and RGB formats for 141 resolutions 1920x1088 and lower, but only YUV for higher resolutions (due 142 to hardware limitations). After setting a resolution of 1920x1088 or lower 143 as the ``OUTPUT`` format, :c:func:`VIDIOC_ENUM_FMT` may return a set of 144 YUV and RGB pixel formats, but after setting a resolution higher than 145 1920x1088, the driver will not return RGB pixel formats, since they are 146 unsupported for this resolution. 147 1485. *[optional]* Choose a different ``CAPTURE`` format than suggested via 149 :c:func:`VIDIOC_S_FMT` on ``CAPTURE`` queue. It is possible for the client to 150 choose a different format than selected/suggested by the driver in 151 :c:func:`VIDIOC_G_FMT`. 152 153 * **Required fields:** 154 155 ``type`` 156 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 157 158 ``pixelformat`` 159 a raw pixel format. 160 161 ``width``, ``height`` 162 frame buffer resolution of the decoded stream; typically unchanged from 163 what was returned with :c:func:`VIDIOC_G_FMT`, but it may be different 164 if the hardware supports composition and/or scaling. 165 166 After performing this step, the client must perform step 3 again in order 167 to obtain up-to-date information about the buffers size and layout. 168 1696. Allocate source (bytestream) buffers via :c:func:`VIDIOC_REQBUFS` on 170 ``OUTPUT`` queue. 171 172 * **Required fields:** 173 174 ``count`` 175 requested number of buffers to allocate; greater than zero. 176 177 ``type`` 178 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. 179 180 ``memory`` 181 follows standard semantics. 182 183 * **Returned fields:** 184 185 ``count`` 186 actual number of buffers allocated. 187 188 * If required, the driver will adjust ``count`` to be equal or bigger to the 189 minimum of required number of ``OUTPUT`` buffers for the given format and 190 requested count. The client must check this value after the ioctl returns 191 to get the actual number of buffers allocated. 192 1937. Allocate destination (raw format) buffers via :c:func:`VIDIOC_REQBUFS` on the 194 ``CAPTURE`` queue. 195 196 * **Required fields:** 197 198 ``count`` 199 requested number of buffers to allocate; greater than zero. The client 200 is responsible for deducing the minimum number of buffers required 201 for the stream to be properly decoded (taking e.g. reference frames 202 into account) and pass an equal or bigger number. 203 204 ``type`` 205 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 206 207 ``memory`` 208 follows standard semantics. ``V4L2_MEMORY_USERPTR`` is not supported 209 for ``CAPTURE`` buffers. 210 211 * **Returned fields:** 212 213 ``count`` 214 adjusted to allocated number of buffers, in case the codec requires 215 more buffers than requested. 216 217 * The driver must adjust count to the minimum of required number of 218 ``CAPTURE`` buffers for the current format, stream configuration and 219 requested count. The client must check this value after the ioctl 220 returns to get the number of buffers allocated. 221 2228. Allocate requests (likely one per ``OUTPUT`` buffer) via 223 :c:func:`MEDIA_IOC_REQUEST_ALLOC` on the media device. 224 2259. Start streaming on both ``OUTPUT`` and ``CAPTURE`` queues via 226 :c:func:`VIDIOC_STREAMON`. 227 228Decoding 229======== 230 231For each frame, the client is responsible for submitting at least one request to 232which the following is attached: 233 234* The amount of encoded data expected by the codec for its current 235 configuration, as a buffer submitted to the ``OUTPUT`` queue. Typically, this 236 corresponds to one frame worth of encoded data, but some formats may allow (or 237 require) different amounts per unit. 238* All the metadata needed to decode the submitted encoded data, in the form of 239 controls relevant to the format being decoded. 240 241The amount of data and contents of the source ``OUTPUT`` buffer, as well as the 242controls that must be set on the request, depend on the active coded pixel 243format and might be affected by codec-specific extended controls, as stated in 244documentation of each format. 245 246If there is a possibility that the decoded frame will require one or more 247decode requests after the current one in order to be produced, then the client 248must set the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag on the ``OUTPUT`` 249buffer. This will result in the (potentially partially) decoded ``CAPTURE`` 250buffer not being made available for dequeueing, and reused for the next decode 251request if the timestamp of the next ``OUTPUT`` buffer has not changed. 252 253A typical frame would thus be decoded using the following sequence: 254 2551. Queue an ``OUTPUT`` buffer containing one unit of encoded bytestream data for 256 the decoding request, using :c:func:`VIDIOC_QBUF`. 257 258 * **Required fields:** 259 260 ``index`` 261 index of the buffer being queued. 262 263 ``type`` 264 type of the buffer. 265 266 ``bytesused`` 267 number of bytes taken by the encoded data frame in the buffer. 268 269 ``flags`` 270 the ``V4L2_BUF_FLAG_REQUEST_FD`` flag must be set. Additionally, if 271 we are not sure that the current decode request is the last one needed 272 to produce a fully decoded frame, then 273 ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` must also be set. 274 275 ``request_fd`` 276 must be set to the file descriptor of the decoding request. 277 278 ``timestamp`` 279 must be set to a unique value per frame. This value will be propagated 280 into the decoded frame's buffer and can also be used to use this frame 281 as the reference of another. If using multiple decode requests per 282 frame, then the timestamps of all the ``OUTPUT`` buffers for a given 283 frame must be identical. If the timestamp changes, then the currently 284 held ``CAPTURE`` buffer will be made available for dequeuing and the 285 current request will work on a new ``CAPTURE`` buffer. 286 2872. Set the codec-specific controls for the decoding request, using 288 :c:func:`VIDIOC_S_EXT_CTRLS`. 289 290 * **Required fields:** 291 292 ``which`` 293 must be ``V4L2_CTRL_WHICH_REQUEST_VAL``. 294 295 ``request_fd`` 296 must be set to the file descriptor of the decoding request. 297 298 other fields 299 other fields are set as usual when setting controls. The ``controls`` 300 array must contain all the codec-specific controls required to decode 301 a frame. 302 303 .. note:: 304 305 It is possible to specify the controls in different invocations of 306 :c:func:`VIDIOC_S_EXT_CTRLS`, or to overwrite a previously set control, as 307 long as ``request_fd`` and ``which`` are properly set. The controls state 308 at the moment of request submission is the one that will be considered. 309 310 .. note:: 311 312 The order in which steps 1 and 2 take place is interchangeable. 313 3143. Submit the request by invoking :c:func:`MEDIA_REQUEST_IOC_QUEUE` on the 315 request FD. 316 317 If the request is submitted without an ``OUTPUT`` buffer, or if some of the 318 required controls are missing from the request, then 319 :c:func:`MEDIA_REQUEST_IOC_QUEUE` will return ``-ENOENT``. If more than one 320 ``OUTPUT`` buffer is queued, then it will return ``-EINVAL``. 321 :c:func:`MEDIA_REQUEST_IOC_QUEUE` returning non-zero means that no 322 ``CAPTURE`` buffer will be produced for this request. 323 324``CAPTURE`` buffers must not be part of the request, and are queued 325independently. They are returned in decode order (i.e. the same order as coded 326frames were submitted to the ``OUTPUT`` queue). 327 328Runtime decoding errors are signaled by the dequeued ``CAPTURE`` buffers 329carrying the ``V4L2_BUF_FLAG_ERROR`` flag. If a decoded reference frame has an 330error, then all following decoded frames that refer to it also have the 331``V4L2_BUF_FLAG_ERROR`` flag set, although the decoder will still try to 332produce (likely corrupted) frames. 333 334Buffer management while decoding 335================================ 336Contrary to stateful decoders, a stateless decoder does not perform any kind of 337buffer management: it only guarantees that dequeued ``CAPTURE`` buffers can be 338used by the client for as long as they are not queued again. "Used" here 339encompasses using the buffer for compositing or display. 340 341A dequeued capture buffer can also be used as the reference frame of another 342buffer. 343 344A frame is specified as reference by converting its timestamp into nanoseconds, 345and storing it into the relevant member of a codec-dependent control structure. 346The :c:func:`v4l2_timeval_to_ns` function must be used to perform that 347conversion. The timestamp of a frame can be used to reference it as soon as all 348its units of encoded data are successfully submitted to the ``OUTPUT`` queue. 349 350A decoded buffer containing a reference frame must not be reused as a decoding 351target until all the frames referencing it have been decoded. The safest way to 352achieve this is to refrain from queueing a reference buffer until all the 353decoded frames referencing it have been dequeued. However, if the driver can 354guarantee that buffers queued to the ``CAPTURE`` queue are processed in queued 355order, then user-space can take advantage of this guarantee and queue a 356reference buffer when the following conditions are met: 357 3581. All the requests for frames affected by the reference frame have been 359 queued, and 360 3612. A sufficient number of ``CAPTURE`` buffers to cover all the decoded 362 referencing frames have been queued. 363 364When queuing a decoding request, the driver will increase the reference count of 365all the resources associated with reference frames. This means that the client 366can e.g. close the DMABUF file descriptors of reference frame buffers if it 367won't need them afterwards. 368 369Seeking 370======= 371In order to seek, the client just needs to submit requests using input buffers 372corresponding to the new stream position. It must however be aware that 373resolution may have changed and follow the dynamic resolution change sequence in 374that case. Also depending on the codec used, picture parameters (e.g. SPS/PPS 375for H.264) may have changed and the client is responsible for making sure that a 376valid state is sent to the decoder. 377 378The client is then free to ignore any returned ``CAPTURE`` buffer that comes 379from the pre-seek position. 380 381Pausing 382======= 383 384In order to pause, the client can just cease queuing buffers onto the ``OUTPUT`` 385queue. Without source bytestream data, there is no data to process and the codec 386will remain idle. 387 388Dynamic resolution change 389========================= 390 391If the client detects a resolution change in the stream, it will need to perform 392the initialization sequence again with the new resolution: 393 3941. If the last submitted request resulted in a ``CAPTURE`` buffer being 395 held by the use of the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag, then the 396 last frame is not available on the ``CAPTURE`` queue. In this case, a 397 ``V4L2_DEC_CMD_FLUSH`` command shall be sent. This will make the driver 398 dequeue the held ``CAPTURE`` buffer. 399 4002. Wait until all submitted requests have completed and dequeue the 401 corresponding output buffers. 402 4033. Call :c:func:`VIDIOC_STREAMOFF` on both the ``OUTPUT`` and ``CAPTURE`` 404 queues. 405 4064. Free all ``CAPTURE`` buffers by calling :c:func:`VIDIOC_REQBUFS` on the 407 ``CAPTURE`` queue with a buffer count of zero. 408 4095. Perform the initialization sequence again (minus the allocation of 410 ``OUTPUT`` buffers), with the new resolution set on the ``OUTPUT`` queue. 411 Note that due to resolution constraints, a different format may need to be 412 picked on the ``CAPTURE`` queue. 413 414Drain 415===== 416 417If the last submitted request resulted in a ``CAPTURE`` buffer being 418held by the use of the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag, then the 419last frame is not available on the ``CAPTURE`` queue. In this case, a 420``V4L2_DEC_CMD_FLUSH`` command shall be sent. This will make the driver 421dequeue the held ``CAPTURE`` buffer. 422 423After that, in order to drain the stream on a stateless decoder, the client 424just needs to wait until all the submitted requests are completed. 425