1.. SPDX-License-Identifier: GPL-2.0 2 3Media Controller devices 4------------------------ 5 6Media Controller 7~~~~~~~~~~~~~~~~ 8 9The media controller userspace API is documented in 10:ref:`the Media Controller uAPI book <media_controller>`. This document focus 11on the kernel-side implementation of the media framework. 12 13Abstract media device model 14^^^^^^^^^^^^^^^^^^^^^^^^^^^ 15 16Discovering a device internal topology, and configuring it at runtime, is one 17of the goals of the media framework. To achieve this, hardware devices are 18modelled as an oriented graph of building blocks called entities connected 19through pads. 20 21An entity is a basic media hardware building block. It can correspond to 22a large variety of logical blocks such as physical hardware devices 23(CMOS sensor for instance), logical hardware devices (a building block 24in a System-on-Chip image processing pipeline), DMA channels or physical 25connectors. 26 27A pad is a connection endpoint through which an entity can interact with 28other entities. Data (not restricted to video) produced by an entity 29flows from the entity's output to one or more entity inputs. Pads should 30not be confused with physical pins at chip boundaries. 31 32A link is a point-to-point oriented connection between two pads, either 33on the same entity or on different entities. Data flows from a source 34pad to a sink pad. 35 36Media device 37^^^^^^^^^^^^ 38 39A media device is represented by a struct media_device 40instance, defined in ``include/media/media-device.h``. 41Allocation of the structure is handled by the media device driver, usually by 42embedding the :c:type:`media_device` instance in a larger driver-specific 43structure. 44 45Drivers initialise media device instances by calling 46:c:func:`media_device_init()`. After initialising a media device instance, it is 47registered by calling :c:func:`__media_device_register()` via the macro 48``media_device_register()`` and unregistered by calling 49:c:func:`media_device_unregister()`. An initialised media device must be 50eventually cleaned up by calling :c:func:`media_device_cleanup()`. 51 52Note that it is not allowed to unregister a media device instance that was not 53previously registered, or clean up a media device instance that was not 54previously initialised. 55 56Entities 57^^^^^^^^ 58 59Entities are represented by a struct media_entity 60instance, defined in ``include/media/media-entity.h``. The structure is usually 61embedded into a higher-level structure, such as 62:c:type:`v4l2_subdev` or :c:type:`video_device` 63instances, although drivers can allocate entities directly. 64 65Drivers initialize entity pads by calling 66:c:func:`media_entity_pads_init()`. 67 68Drivers register entities with a media device by calling 69:c:func:`media_device_register_entity()` 70and unregistered by calling 71:c:func:`media_device_unregister_entity()`. 72 73Interfaces 74^^^^^^^^^^ 75 76Interfaces are represented by a 77struct media_interface instance, defined in 78``include/media/media-entity.h``. Currently, only one type of interface is 79defined: a device node. Such interfaces are represented by a 80struct media_intf_devnode. 81 82Drivers initialize and create device node interfaces by calling 83:c:func:`media_devnode_create()` 84and remove them by calling: 85:c:func:`media_devnode_remove()`. 86 87Pads 88^^^^ 89Pads are represented by a struct media_pad instance, 90defined in ``include/media/media-entity.h``. Each entity stores its pads in 91a pads array managed by the entity driver. Drivers usually embed the array in 92a driver-specific structure. 93 94Pads are identified by their entity and their 0-based index in the pads 95array. 96 97Both information are stored in the struct media_pad, 98making the struct media_pad pointer the canonical way 99to store and pass link references. 100 101Pads have flags that describe the pad capabilities and state. 102 103``MEDIA_PAD_FL_SINK`` indicates that the pad supports sinking data. 104``MEDIA_PAD_FL_SOURCE`` indicates that the pad supports sourcing data. 105 106.. note:: 107 108 One and only one of ``MEDIA_PAD_FL_SINK`` or ``MEDIA_PAD_FL_SOURCE`` must 109 be set for each pad. 110 111Links 112^^^^^ 113 114Links are represented by a struct media_link instance, 115defined in ``include/media/media-entity.h``. There are two types of links: 116 117**1. pad to pad links**: 118 119Associate two entities via their PADs. Each entity has a list that points 120to all links originating at or targeting any of its pads. 121A given link is thus stored twice, once in the source entity and once in 122the target entity. 123 124Drivers create pad to pad links by calling: 125:c:func:`media_create_pad_link()` and remove with 126:c:func:`media_entity_remove_links()`. 127 128**2. interface to entity links**: 129 130Associate one interface to a Link. 131 132Drivers create interface to entity links by calling: 133:c:func:`media_create_intf_link()` and remove with 134:c:func:`media_remove_intf_links()`. 135 136.. note:: 137 138 Links can only be created after having both ends already created. 139 140Links have flags that describe the link capabilities and state. The 141valid values are described at :c:func:`media_create_pad_link()` and 142:c:func:`media_create_intf_link()`. 143 144Graph traversal 145^^^^^^^^^^^^^^^ 146 147The media framework provides APIs to traverse media graphs, locating connected 148entities and links. 149 150To iterate over all entities belonging to a media device, drivers can use 151the media_device_for_each_entity macro, defined in 152``include/media/media-device.h``. 153 154.. code-block:: c 155 156 struct media_entity *entity; 157 158 media_device_for_each_entity(entity, mdev) { 159 // entity will point to each entity in turn 160 ... 161 } 162 163Helper functions can be used to find a link between two given pads, or a pad 164connected to another pad through an enabled link 165(:c:func:`media_entity_find_link()`, :c:func:`media_pad_remote_pad_first()`, 166:c:func:`media_entity_remote_source_pad_unique()` and 167:c:func:`media_pad_remote_pad_unique()`). 168 169Use count and power handling 170^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 171 172Due to the wide differences between drivers regarding power management 173needs, the media controller does not implement power management. However, 174the struct media_entity includes a ``use_count`` 175field that media drivers 176can use to track the number of users of every entity for power management 177needs. 178 179The :c:type:`media_entity<media_entity>`.\ ``use_count`` field is owned by 180media drivers and must not be 181touched by entity drivers. Access to the field must be protected by the 182:c:type:`media_device`.\ ``graph_mutex`` lock. 183 184Links setup 185^^^^^^^^^^^ 186 187Link properties can be modified at runtime by calling 188:c:func:`media_entity_setup_link()`. 189 190Pipelines and media streams 191^^^^^^^^^^^^^^^^^^^^^^^^^^^ 192 193A media stream is a stream of pixels or metadata originating from one or more 194source devices (such as a sensors) and flowing through media entity pads 195towards the final sinks. The stream can be modified on the route by the 196devices (e.g. scaling or pixel format conversions), or it can be split into 197multiple branches, or multiple branches can be merged. 198 199A media pipeline is a set of media streams which are interdependent. This 200interdependency can be caused by the hardware (e.g. configuration of a second 201stream cannot be changed if the first stream has been enabled) or by the driver 202due to the software design. Most commonly a media pipeline consists of a single 203stream which does not branch. 204 205When starting streaming, drivers must notify all entities in the pipeline to 206prevent link states from being modified during streaming by calling 207:c:func:`media_pipeline_start()`. 208 209The function will mark all the pads which are part of the pipeline as streaming. 210 211The struct media_pipeline instance pointed to by the pipe argument will be 212stored in every pad in the pipeline. Drivers should embed the struct 213media_pipeline in higher-level pipeline structures and can then access the 214pipeline through the struct media_pad pipe field. 215 216Calls to :c:func:`media_pipeline_start()` can be nested. 217The pipeline pointer must be identical for all nested calls to the function. 218 219:c:func:`media_pipeline_start()` may return an error. In that case, 220it will clean up any of the changes it did by itself. 221 222When stopping the stream, drivers must notify the entities with 223:c:func:`media_pipeline_stop()`. 224 225If multiple calls to :c:func:`media_pipeline_start()` have been 226made the same number of :c:func:`media_pipeline_stop()` calls 227are required to stop streaming. 228The :c:type:`media_entity`.\ ``pipe`` field is reset to ``NULL`` on the last 229nested stop call. 230 231Link configuration will fail with ``-EBUSY`` by default if either end of the 232link is a streaming entity. Links that can be modified while streaming must 233be marked with the ``MEDIA_LNK_FL_DYNAMIC`` flag. 234 235If other operations need to be disallowed on streaming entities (such as 236changing entities configuration parameters) drivers can explicitly check the 237media_entity stream_count field to find out if an entity is streaming. This 238operation must be done with the media_device graph_mutex held. 239 240Link validation 241^^^^^^^^^^^^^^^ 242 243Link validation is performed by :c:func:`media_pipeline_start()` 244for any entity which has sink pads in the pipeline. The 245:c:type:`media_entity`.\ ``link_validate()`` callback is used for that 246purpose. In ``link_validate()`` callback, entity driver should check 247that the properties of the source pad of the connected entity and its own 248sink pad match. It is up to the type of the entity (and in the end, the 249properties of the hardware) what matching actually means. 250 251Subsystems should facilitate link validation by providing subsystem specific 252helper functions to provide easy access for commonly needed information, and 253in the end provide a way to use driver-specific callbacks. 254 255Pipeline traversal 256^^^^^^^^^^^^^^^^^^ 257 258Once a pipeline has been constructed with :c:func:`media_pipeline_start()`, 259drivers can iterate over entities or pads in the pipeline with the 260:c:macro:´media_pipeline_for_each_entity` and 261:c:macro:´media_pipeline_for_each_pad` macros. Iterating over pads is 262straightforward: 263 264.. code-block:: c 265 266 media_pipeline_pad_iter iter; 267 struct media_pad *pad; 268 269 media_pipeline_for_each_pad(pipe, &iter, pad) { 270 /* 'pad' will point to each pad in turn */ 271 ... 272 } 273 274To iterate over entities, the iterator needs to be initialized and cleaned up 275as an additional steps: 276 277.. code-block:: c 278 279 media_pipeline_entity_iter iter; 280 struct media_entity *entity; 281 int ret; 282 283 ret = media_pipeline_entity_iter_init(pipe, &iter); 284 if (ret) 285 ...; 286 287 media_pipeline_for_each_entity(pipe, &iter, entity) { 288 /* 'entity' will point to each entity in turn */ 289 ... 290 } 291 292 media_pipeline_entity_iter_cleanup(&iter); 293 294Media Controller Device Allocator API 295^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 296 297When the media device belongs to more than one driver, the shared media 298device is allocated with the shared struct device as the key for look ups. 299 300The shared media device should stay in registered state until the last 301driver unregisters it. In addition, the media device should be released when 302all the references are released. Each driver gets a reference to the media 303device during probe, when it allocates the media device. If media device is 304already allocated, the allocate API bumps up the refcount and returns the 305existing media device. The driver puts the reference back in its disconnect 306routine when it calls :c:func:`media_device_delete()`. 307 308The media device is unregistered and cleaned up from the kref put handler to 309ensure that the media device stays in registered state until the last driver 310unregisters the media device. 311 312**Driver Usage** 313 314Drivers should use the appropriate media-core routines to manage the shared 315media device life-time handling the two states: 3161. allocate -> register -> delete 3172. get reference to already registered device -> delete 318 319call :c:func:`media_device_delete()` routine to make sure the shared media 320device delete is handled correctly. 321 322**driver probe:** 323Call :c:func:`media_device_usb_allocate()` to allocate or get a reference 324Call :c:func:`media_device_register()`, if media devnode isn't registered 325 326**driver disconnect:** 327Call :c:func:`media_device_delete()` to free the media_device. Freeing is 328handled by the kref put handler. 329 330API Definitions 331^^^^^^^^^^^^^^^ 332 333.. kernel-doc:: include/media/media-device.h 334 335.. kernel-doc:: include/media/media-devnode.h 336 337.. kernel-doc:: include/media/media-entity.h 338 339.. kernel-doc:: include/media/media-request.h 340 341.. kernel-doc:: include/media/media-dev-allocator.h 342