1.. SPDX-License-Identifier: GPL-2.0 2 3Media Controller devices 4------------------------ 5 6Media Controller 7~~~~~~~~~~~~~~~~ 8 9The media controller userspace API is documented in 10:ref:`the Media Controller uAPI book <media_controller>`. This document focus 11on the kernel-side implementation of the media framework. 12 13Abstract media device model 14^^^^^^^^^^^^^^^^^^^^^^^^^^^ 15 16Discovering a device internal topology, and configuring it at runtime, is one 17of the goals of the media framework. To achieve this, hardware devices are 18modelled as an oriented graph of building blocks called entities connected 19through pads. 20 21An entity is a basic media hardware building block. It can correspond to 22a large variety of logical blocks such as physical hardware devices 23(CMOS sensor for instance), logical hardware devices (a building block 24in a System-on-Chip image processing pipeline), DMA channels or physical 25connectors. 26 27A pad is a connection endpoint through which an entity can interact with 28other entities. Data (not restricted to video) produced by an entity 29flows from the entity's output to one or more entity inputs. Pads should 30not be confused with physical pins at chip boundaries. 31 32A link is a point-to-point oriented connection between two pads, either 33on the same entity or on different entities. Data flows from a source 34pad to a sink pad. 35 36Media device 37^^^^^^^^^^^^ 38 39A media device is represented by a struct media_device 40instance, defined in ``include/media/media-device.h``. 41Allocation of the structure is handled by the media device driver, usually by 42embedding the :c:type:`media_device` instance in a larger driver-specific 43structure. 44 45Drivers initialise media device instances by calling 46:c:func:`media_device_init()`. After initialising a media device instance, it is 47registered by calling :c:func:`__media_device_register()` via the macro 48``media_device_register()`` and unregistered by calling 49:c:func:`media_device_unregister()`. An initialised media device must be 50eventually cleaned up by calling :c:func:`media_device_cleanup()`. 51 52Note that it is not allowed to unregister a media device instance that was not 53previously registered, or clean up a media device instance that was not 54previously initialised. 55 56Entities 57^^^^^^^^ 58 59Entities are represented by a struct media_entity 60instance, defined in ``include/media/media-entity.h``. The structure is usually 61embedded into a higher-level structure, such as 62:c:type:`v4l2_subdev` or :c:type:`video_device` 63instances, although drivers can allocate entities directly. 64 65Drivers initialize entity pads by calling 66:c:func:`media_entity_pads_init()`. 67 68Drivers register entities with a media device by calling 69:c:func:`media_device_register_entity()` 70and unregistered by calling 71:c:func:`media_device_unregister_entity()`. 72 73Interfaces 74^^^^^^^^^^ 75 76Interfaces are represented by a 77struct media_interface instance, defined in 78``include/media/media-entity.h``. Currently, only one type of interface is 79defined: a device node. Such interfaces are represented by a 80struct media_intf_devnode. 81 82Drivers initialize and create device node interfaces by calling 83:c:func:`media_devnode_create()` 84and remove them by calling: 85:c:func:`media_devnode_remove()`. 86 87Pads 88^^^^ 89Pads are represented by a struct media_pad instance, 90defined in ``include/media/media-entity.h``. Each entity stores its pads in 91a pads array managed by the entity driver. Drivers usually embed the array in 92a driver-specific structure. 93 94Pads are identified by their entity and their 0-based index in the pads 95array. 96 97Both information are stored in the struct media_pad, 98making the struct media_pad pointer the canonical way 99to store and pass link references. 100 101Pads have flags that describe the pad capabilities and state. 102 103``MEDIA_PAD_FL_SINK`` indicates that the pad supports sinking data. 104``MEDIA_PAD_FL_SOURCE`` indicates that the pad supports sourcing data. 105 106.. note:: 107 108 One and only one of ``MEDIA_PAD_FL_SINK`` or ``MEDIA_PAD_FL_SOURCE`` must 109 be set for each pad. 110 111Links 112^^^^^ 113 114Links are represented by a struct media_link instance, 115defined in ``include/media/media-entity.h``. There are two types of links: 116 117**1. pad to pad links**: 118 119Associate two entities via their PADs. Each entity has a list that points 120to all links originating at or targeting any of its pads. 121A given link is thus stored twice, once in the source entity and once in 122the target entity. 123 124Drivers create pad to pad links by calling: 125:c:func:`media_create_pad_link()` and remove with 126:c:func:`media_entity_remove_links()`. 127 128**2. interface to entity links**: 129 130Associate one interface to a Link. 131 132Drivers create interface to entity links by calling: 133:c:func:`media_create_intf_link()` and remove with 134:c:func:`media_remove_intf_links()`. 135 136.. note:: 137 138 Links can only be created after having both ends already created. 139 140Links have flags that describe the link capabilities and state. The 141valid values are described at :c:func:`media_create_pad_link()` and 142:c:func:`media_create_intf_link()`. 143 144Graph traversal 145^^^^^^^^^^^^^^^ 146 147The media framework provides APIs to iterate over entities in a graph. 148 149To iterate over all entities belonging to a media device, drivers can use 150the media_device_for_each_entity macro, defined in 151``include/media/media-device.h``. 152 153.. code-block:: c 154 155 struct media_entity *entity; 156 157 media_device_for_each_entity(entity, mdev) { 158 // entity will point to each entity in turn 159 ... 160 } 161 162Drivers might also need to iterate over all entities in a graph that can be 163reached only through enabled links starting at a given entity. The media 164framework provides a depth-first graph traversal API for that purpose. 165 166.. note:: 167 168 Graphs with cycles (whether directed or undirected) are **NOT** 169 supported by the graph traversal API. To prevent infinite loops, the graph 170 traversal code limits the maximum depth to ``MEDIA_ENTITY_ENUM_MAX_DEPTH``, 171 currently defined as 16. 172 173Drivers initiate a graph traversal by calling 174:c:func:`media_graph_walk_start()` 175 176The graph structure, provided by the caller, is initialized to start graph 177traversal at the given entity. 178 179Drivers can then retrieve the next entity by calling 180:c:func:`media_graph_walk_next()` 181 182When the graph traversal is complete the function will return ``NULL``. 183 184Graph traversal can be interrupted at any moment. No cleanup function call 185is required and the graph structure can be freed normally. 186 187Helper functions can be used to find a link between two given pads, or a pad 188connected to another pad through an enabled link 189(:c:func:`media_entity_find_link()`, :c:func:`media_pad_remote_pad_first()`, 190:c:func:`media_entity_remote_source_pad_unique()` and 191:c:func:`media_pad_remote_pad_unique()`). 192 193Use count and power handling 194^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 195 196Due to the wide differences between drivers regarding power management 197needs, the media controller does not implement power management. However, 198the struct media_entity includes a ``use_count`` 199field that media drivers 200can use to track the number of users of every entity for power management 201needs. 202 203The :c:type:`media_entity<media_entity>`.\ ``use_count`` field is owned by 204media drivers and must not be 205touched by entity drivers. Access to the field must be protected by the 206:c:type:`media_device`.\ ``graph_mutex`` lock. 207 208Links setup 209^^^^^^^^^^^ 210 211Link properties can be modified at runtime by calling 212:c:func:`media_entity_setup_link()`. 213 214Pipelines and media streams 215^^^^^^^^^^^^^^^^^^^^^^^^^^^ 216 217A media stream is a stream of pixels or metadata originating from one or more 218source devices (such as a sensors) and flowing through media entity pads 219towards the final sinks. The stream can be modified on the route by the 220devices (e.g. scaling or pixel format conversions), or it can be split into 221multiple branches, or multiple branches can be merged. 222 223A media pipeline is a set of media streams which are interdependent. This 224interdependency can be caused by the hardware (e.g. configuration of a second 225stream cannot be changed if the first stream has been enabled) or by the driver 226due to the software design. Most commonly a media pipeline consists of a single 227stream which does not branch. 228 229When starting streaming, drivers must notify all entities in the pipeline to 230prevent link states from being modified during streaming by calling 231:c:func:`media_pipeline_start()`. 232 233The function will mark all the pads which are part of the pipeline as streaming. 234 235The struct media_pipeline instance pointed to by 236the pipe argument will be stored in every pad in the pipeline. 237Drivers should embed the struct media_pipeline 238in higher-level pipeline structures and can then access the 239pipeline through the struct media_pad 240pipe field. 241 242Calls to :c:func:`media_pipeline_start()` can be nested. 243The pipeline pointer must be identical for all nested calls to the function. 244 245:c:func:`media_pipeline_start()` may return an error. In that case, 246it will clean up any of the changes it did by itself. 247 248When stopping the stream, drivers must notify the entities with 249:c:func:`media_pipeline_stop()`. 250 251If multiple calls to :c:func:`media_pipeline_start()` have been 252made the same number of :c:func:`media_pipeline_stop()` calls 253are required to stop streaming. 254The :c:type:`media_entity`.\ ``pipe`` field is reset to ``NULL`` on the last 255nested stop call. 256 257Link configuration will fail with ``-EBUSY`` by default if either end of the 258link is a streaming entity. Links that can be modified while streaming must 259be marked with the ``MEDIA_LNK_FL_DYNAMIC`` flag. 260 261If other operations need to be disallowed on streaming entities (such as 262changing entities configuration parameters) drivers can explicitly check the 263media_entity stream_count field to find out if an entity is streaming. This 264operation must be done with the media_device graph_mutex held. 265 266Link validation 267^^^^^^^^^^^^^^^ 268 269Link validation is performed by :c:func:`media_pipeline_start()` 270for any entity which has sink pads in the pipeline. The 271:c:type:`media_entity`.\ ``link_validate()`` callback is used for that 272purpose. In ``link_validate()`` callback, entity driver should check 273that the properties of the source pad of the connected entity and its own 274sink pad match. It is up to the type of the entity (and in the end, the 275properties of the hardware) what matching actually means. 276 277Subsystems should facilitate link validation by providing subsystem specific 278helper functions to provide easy access for commonly needed information, and 279in the end provide a way to use driver-specific callbacks. 280 281Media Controller Device Allocator API 282^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 283 284When the media device belongs to more than one driver, the shared media 285device is allocated with the shared struct device as the key for look ups. 286 287The shared media device should stay in registered state until the last 288driver unregisters it. In addition, the media device should be released when 289all the references are released. Each driver gets a reference to the media 290device during probe, when it allocates the media device. If media device is 291already allocated, the allocate API bumps up the refcount and returns the 292existing media device. The driver puts the reference back in its disconnect 293routine when it calls :c:func:`media_device_delete()`. 294 295The media device is unregistered and cleaned up from the kref put handler to 296ensure that the media device stays in registered state until the last driver 297unregisters the media device. 298 299**Driver Usage** 300 301Drivers should use the appropriate media-core routines to manage the shared 302media device life-time handling the two states: 3031. allocate -> register -> delete 3042. get reference to already registered device -> delete 305 306call :c:func:`media_device_delete()` routine to make sure the shared media 307device delete is handled correctly. 308 309**driver probe:** 310Call :c:func:`media_device_usb_allocate()` to allocate or get a reference 311Call :c:func:`media_device_register()`, if media devnode isn't registered 312 313**driver disconnect:** 314Call :c:func:`media_device_delete()` to free the media_device. Freeing is 315handled by the kref put handler. 316 317API Definitions 318^^^^^^^^^^^^^^^ 319 320.. kernel-doc:: include/media/media-device.h 321 322.. kernel-doc:: include/media/media-devnode.h 323 324.. kernel-doc:: include/media/media-entity.h 325 326.. kernel-doc:: include/media/media-request.h 327 328.. kernel-doc:: include/media/media-dev-allocator.h 329