1.. SPDX-License-Identifier: GPL-2.0 2 3========================================== 4ARM Mali-C55 Image Signal Processor driver 5========================================== 6 7Introduction 8============ 9 10This file documents the driver for ARM's Mali-C55 Image Signal Processor. The 11driver is located under drivers/media/platform/arm/mali-c55. 12 13The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from 14sensors through either a parallel interface or a memory bus before processing it 15and outputting it through an internal DMA engine. Two output pipelines are 16possible (though one may not be fitted, depending on the implementation). These 17are referred to as "Full resolution" and "Downscale", but the naming is historic 18and both pipes are capable of cropping/scaling operations. The full resolution 19pipe is also capable of outputting RAW data, bypassing much of the ISP's 20processing. The downscale pipe cannot output RAW data. An integrated test 21pattern generator can be used to drive the ISP and produce image data in the 22absence of a connected camera sensor. The driver module is named mali_c55, and 23is enabled through the CONFIG_VIDEO_MALI_C55 config option. 24 25The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and 26expects camera sensors connected to the ISP to have V4L2 subdevice interfaces. 27 28Mali-C55 ISP hardware 29===================== 30 31A high level functional view of the Mali-C55 ISP is presented below. The ISP 32takes input from either a live source or through a DMA engine for memory input, 33depending on the SoC integration.:: 34 35 +---------+ +----------+ +--------+ 36 | Sensor |--->| CSI-2 Rx | "Full Resolution" | DMA | 37 +---------+ +----------+ |\ Output +--->| Writer | 38 | | \ | +--------+ 39 | | \ +----------+ +------+---> Streaming I/O 40 +------------+ +------->| | | | | 41 | | | |-->| Mali-C55 |--+ 42 | DMA Reader |--------------->| | | ISP | | 43 | | | / | | | +---> Streaming I/O 44 +------------+ | / +----------+ | | 45 |/ +------+ 46 | +--------+ 47 +--->| DMA | 48 "Downscaled" | Writer | 49 Output +--------+ 50 51Media Controller Topology 52========================= 53 54An example of the ISP's topology (as implemented in a system with an IMX415 55camera sensor and generic CSI-2 receiver) is below: 56 57 58.. kernel-figure:: mali-c55-graph.dot 59 :alt: mali-c55-graph.dot 60 :align: center 61 62The driver has 4 V4L2 subdevices: 63 64- `mali_c55 isp`: Responsible for configuring input crop and color space 65 conversion 66- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor. 67- `mali_c55 resizer fr`: The Full-Resolution pipe resizer 68- `mali_c55 resizer ds`: The Downscale pipe resizer 69 70The driver has 3 V4L2 video devices: 71 72- `mali-c55 fr`: The full-resolution pipe's capture device 73- `mali-c55 ds`: The downscale pipe's capture device 74- `mali-c55 3a stats`: The 3A statistics capture device 75 76Frame sequences are synchronised across to two capture devices, meaning if one 77pipe is started later than the other the sequence numbers returned in its 78buffers will match those of the other pipe rather than starting from zero. 79 80Idiosyncrasies 81-------------- 82 83**mali-c55 isp** 84The `mali-c55 isp` subdevice has a single sink pad to which all sources of data 85should be connected. The active source is selected by enabling the appropriate 86media link and disabling all others. The ISP has two source pads, reflecting the 87different paths through which it can internally route data. Tap points within 88the ISP allow users to divert data to avoid processing by some or all of the 89hardware's processing steps. The diagram below is intended only to highlight how 90the bypassing works and is not a true reflection of those processing steps; for 91a high-level functional block diagram see ARM's developer page for the 92ISP [3]_:: 93 94 +--------------------------------------------------------------+ 95 | Possible Internal ISP Data Routes | 96 | +------------+ +----------+ +------------+ | 97 +---+ | | | | | Colour | +---+ 98 | 0 |--+-->| Processing |->| Demosaic |->| Space |--->| 1 | 99 +---+ | | | | | | Conversion | +---+ 100 | | +------------+ +----------+ +------------+ | 101 | | +---+ 102 | +---------------------------------------------------| 2 | 103 | +---+ 104 | | 105 +--------------------------------------------------------------+ 106 107 108.. flat-table:: 109 :header-rows: 1 110 111 * - Pad 112 - Direction 113 - Purpose 114 115 * - 0 116 - sink 117 - Data input, connected to the TPG and camera sensors 118 119 * - 1 120 - source 121 - RGB/YUV data, connected to the FR and DS V4L2 subdevices 122 123 * - 2 124 - source 125 - RAW bayer data, connected to the FR V4L2 subdevices 126 127The ISP is limited to both input and output resolutions between 640x480 and 1288192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt() 129operations. 130 131**mali-c55 resizer fr** 132The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different 133insertion points in the hardware (either RAW or demosaiced data): 134 135.. flat-table:: 136 :header-rows: 1 137 138 * - Pad 139 - Direction 140 - Purpose 141 142 * - 0 143 - sink 144 - Data input connected to the ISP's demosaiced stream. 145 146 * - 1 147 - source 148 - Data output connected to the capture video device 149 150 * - 2 151 - sink 152 - Data input connected to the ISP's raw data stream 153 154The data source in use is selected through the routing API; two routes each of a 155single stream are available: 156 157.. flat-table:: 158 :header-rows: 1 159 160 * - Sink Pad 161 - Source Pad 162 - Purpose 163 164 * - 0 165 - 1 166 - Demosaiced data route 167 168 * - 2 169 - 1 170 - Raw data route 171 172 173If the demosaiced route is active then the FR pipe is only capable of output 174in RGB/YUV formats. If the raw route is active then the output reflects the 175input (which may be either Bayer or RGB/YUV data). 176 177Using the driver to capture video 178================================= 179 180Using the media controller APIs we can configure the input source and ISP to 181capture images in a variety of formats. In the examples below, configuring the 182media graph is done with the v4l-utils [1]_ package's media-ctl utility. 183Capturing the images is done with yavta [2]_. 184 185Configuring the input source 186---------------------------- 187 188The first step is to set the input source that we wish by enabling the correct 189media link. Using the example topology above, we can select the TPG as follows: 190 191.. code-block:: none 192 193 media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]" 194 media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]" 195 196Configuring which video devices will stream data 197------------------------------------------------ 198 199The driver will wait for all video devices to have their VIDIOC_STREAMON ioctl 200called before it tells the sensor to start streaming. To facilitate this we need 201to enable links to the video devices that we want to use. In the example below 202we enable the links to both of the image capture video devices 203 204.. code-block:: none 205 206 media-ctl -l "'mali-c55 resizer fr':1->'mali-c55 fr':0[1]" 207 media-ctl -l "'mali-c55 resizer ds':1->'mali-c55 ds':0[1]" 208 209Capturing bayer data from the source and processing to RGB/YUV 210-------------------------------------------------------------- 211 212To capture 1920x1080 bayer data from the source and push it through the ISP's 213full processing pipeline, we configure the data formats appropriately on the 214source, ISP and resizer subdevices and set the FR resizer's routing to select 215processed data. The media bus format on the resizer's source pad will be either 216RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or 217YUV. The ISP's debayering block outputs RGB data natively, setting the source 218pad format to YUV10_1X30 enables the colour space conversion block. 219 220In this example we target RGB565 output, so select RGB121212_1X36 as the resizer 221source pad's format: 222 223.. code-block:: none 224 225 # Set formats on the TPG and ISP 226 media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB20_1X20/1920x1080]" 227 media-ctl -V "'mali-c55 isp':0[fmt:SRGGB20_1X20/1920x1080]" 228 media-ctl -V "'mali-c55 isp':1[fmt:SRGGB20_1X20/1920x1080]" 229 230 # Set routing on the FR resizer 231 media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]" 232 233 # Set format on the resizer, must be done AFTER the routing. 234 media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]" 235 236The downscale output can also be used to stream data at the same time. In this 237case since only processed data can be captured through the downscale output no 238routing need be set: 239 240.. code-block:: none 241 242 # Set format on the resizer 243 media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]" 244 245Following which images can be captured from both the FR and DS output's video 246devices (simultaneously, if desired): 247 248.. code-block:: none 249 250 yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 251 yavta -f RGB565 -s 1920x1080 -c10 /dev/video1 252 253Cropping the image 254~~~~~~~~~~~~~~~~~~ 255 256Both the full resolution and downscale pipes can crop to a minimum resolution of 257640x480. To crop the image simply configure the resizer's sink pad's crop and 258compose rectangles and set the format on the video device: 259 260.. code-block:: none 261 262 media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]" 263 media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" 264 yavta -f RGB565 -s 640x480 -c10 /dev/video0 265 266Downscaling the image 267~~~~~~~~~~~~~~~~~~~~~ 268 269Both the full resolution and downscale pipes can downscale the image by up to 8x 270provided the minimum 640x480 output resolution is adhered to. For the best image 271result the scaling ratio for each direction should be the same. To configure 272scaling we use the compose rectangle on the resizer's sink pad: 273 274.. code-block:: none 275 276 media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]" 277 media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" 278 yavta -f RGB565 -s 640x480 -c10 /dev/video0 279 280Capturing images in YUV formats 281~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 282 283If we need to output YUV data rather than RGB the color space conversion block 284needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the 285resizer's source pad. We can then configure a capture format like NV12 (here in 286its multi-planar variant) 287 288.. code-block:: none 289 290 media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]" 291 yavta -f NV12M -s 1920x1080 -c10 /dev/video0 292 293Capturing RGB data from the source and processing it with the resizers 294---------------------------------------------------------------------- 295 296The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this 297case although none of the image quality blocks would be used it can still 298crop/scale the data in the usual way. For this reason RGB data input to the ISP 299still goes through the ISP subdevice's pad 1 to the resizer. 300 301To achieve this, the ISP's sink pad's format is set to 302MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to 303work with the ISP. Converting the camera sensor's output to that format is the 304responsibility of external hardware. 305 306In this example we ask the test pattern generator to give us RGB data instead of 307bayer. 308 309.. code-block:: none 310 311 media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]" 312 media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]" 313 314Cropping or scaling the data can be done in exactly the same way as outlined 315earlier. 316 317Capturing raw data from the source and outputting it unmodified 318----------------------------------------------------------------- 319 320The ISP can additionally capture raw data from the source and output it on the 321full resolution pipe only, completely unmodified. In this case the downscale 322pipe can still process the data normally and be used at the same time. 323 324To configure raw bypass the FR resizer's subdevice's routing table needs to be 325configured, followed by formats in the appropriate places: 326 327.. code-block:: none 328 329 media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]" 330 media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]" 331 media-ctl -V "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080]" 332 media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]" 333 334 # Set format on the video device and stream 335 yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 336 337.. _mali-c55-3a-stats: 338 339Capturing ISP Statistics 340======================== 341 342The ISP is capable of producing statistics for consumption by image processing 343algorithms running in userspace. These statistics can be captured by queueing 344buffers to the `mali-c55 3a stats` V4L2 Device whilst the ISP is streaming. Only 345the :ref:`V4L2_META_FMT_MALI_C55_STATS <v4l2-meta-fmt-mali-c55-stats>` 346format is supported, so no format-setting need be done: 347 348.. code-block:: none 349 350 # We assume the media graph has been configured to support RGB565 capture 351 # from the mali-c55 fr V4L2 Device, which is at /dev/video0. The statistics 352 # V4L2 device is at /dev/video3 353 354 yavta -f RGB565 -s 1920x1080 -c32 /dev/video0 && \ 355 yavta -c10 -F /dev/video3 356 357The layout of the buffer is described by :c:type:`mali_c55_stats_buffer`, 358but broadly statistics are generated to support three image processing 359algorithms; AEXP (Auto-Exposure), AWB (Auto-White Balance) and AF (Auto-Focus). 360These stats can be drawn from various places in the Mali C55 ISP pipeline, known 361as "tap points". This high-level block diagram is intended to explain where in 362the processing flow the statistics can be drawn from:: 363 364 +--> AEXP-2 +----> AEXP-1 +--> AF-0 365 | +----> AF-1 | 366 | | | 367 +---------+ | +--------------+ | +--------------+ | 368 | Input +-+-->+ Digital Gain +---+-->+ Black Level +---+---+ 369 +---------+ +--------------+ +--------------+ | 370 +-----------------------------------------------------------------+ 371 | 372 | +--------------+ +---------+ +----------------+ 373 +-->| Sinter Noise +-+ White +--+--->| Lens Shading +--+---------------+ 374 | Reduction | | Balance | | | | | | 375 +--------------+ +---------+ | +----------------+ | | 376 +---> AEXP-0 (A) +--> AEXP-0 (B) | 377 +--------------------------------------------------------------------------+ 378 | 379 | +----------------+ +--------------+ +----------------+ 380 +-->| Tone mapping +-+--->| Demosaicing +->+ Purple Fringe +-+-----------+ 381 | | | +--------------+ | Correction | | | 382 +----------------+ +-> AEXP-IRIDIX +----------------+ +---> AWB-0 | 383 +----------------------------------------------------------------------------+ 384 | +-------------+ +-------------+ 385 +------------------->| Colour +---+--->| Output | 386 | Correction | | | Pipelines | 387 +-------------+ | +-------------+ 388 +--> AWB-1 389 390By default all statistics are drawn from the 0th tap point for each algorithm; 391I.E. AEXP statistics from AEXP-0 (A), AWB statistics from AWB-0 and AF 392statistics from AF-0. This is configurable for AEXP and AWB statsistics through 393programming the ISP's parameters. 394 395.. _mali-c55-3a-params: 396 397Programming ISP Parameters 398========================== 399 400The ISP can be programmed with various parameters from userspace to apply to the 401hardware before and during video stream. This allows userspace to dynamically 402change values such as black level, white balance and lens shading gains and so 403on. 404 405The buffer format and how to populate it are described by the 406:ref:`V4L2_META_FMT_MALI_C55_PARAMS <v4l2-meta-fmt-mali-c55-params>` format, 407which should be set as the data format for the `mali-c55 3a params` video node. 408 409References 410========== 411.. [1] https://git.linuxtv.org/v4l-utils.git/ 412.. [2] https://git.ideasonboard.org/yavta.git 413.. [3] https://developer.arm.com/Processors/Mali-C55 414