1.. SPDX-License-Identifier: GPL-2.0 2 3.. include:: <isonum.txt> 4 5=============================================================== 6Intel Image Processing Unit 3 (IPU3) Imaging Unit (ImgU) driver 7=============================================================== 8 9Copyright |copy| 2018 Intel Corporation 10 11Introduction 12============ 13 14This file documents the Intel IPU3 (3rd generation Image Processing Unit) 15Imaging Unit drivers located under drivers/media/pci/intel/ipu3 (CIO2) as well 16as under drivers/staging/media/ipu3 (ImgU). 17 18The Intel IPU3 found in certain Kaby Lake (as well as certain Sky Lake) 19platforms (U/Y processor lines) is made up of two parts namely the Imaging Unit 20(ImgU) and the CIO2 device (MIPI CSI2 receiver). 21 22The CIO2 device receives the raw Bayer data from the sensors and outputs the 23frames in a format that is specific to the IPU3 (for consumption by the IPU3 24ImgU). The CIO2 driver is available as drivers/media/pci/intel/ipu3/ipu3-cio2* 25and is enabled through the CONFIG_VIDEO_IPU3_CIO2 config option. 26 27The Imaging Unit (ImgU) is responsible for processing images captured 28by the IPU3 CIO2 device. The ImgU driver sources can be found under 29drivers/staging/media/ipu3 directory. The driver is enabled through the 30CONFIG_VIDEO_IPU3_IMGU config option. 31 32The two driver modules are named ipu3_csi2 and ipu3_imgu, respectively. 33 34The drivers has been tested on Kaby Lake platforms (U/Y processor lines). 35 36Both of the drivers implement V4L2, Media Controller and V4L2 sub-device 37interfaces. The IPU3 CIO2 driver supports camera sensors connected to the CIO2 38MIPI CSI-2 interfaces through V4L2 sub-device sensor drivers. 39 40CIO2 41==== 42 43The CIO2 is represented as a single V4L2 subdev, which provides a V4L2 subdev 44interface to the user space. There is a video node for each CSI-2 receiver, 45with a single media controller interface for the entire device. 46 47The CIO2 contains four independent capture channel, each with its own MIPI CSI-2 48receiver and DMA engine. Each channel is modelled as a V4L2 sub-device exposed 49to userspace as a V4L2 sub-device node and has two pads: 50 51.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}| 52 53.. flat-table:: 54 55 * - pad 56 - direction 57 - purpose 58 59 * - 0 60 - sink 61 - MIPI CSI-2 input, connected to the sensor subdev 62 63 * - 1 64 - source 65 - Raw video capture, connected to the V4L2 video interface 66 67The V4L2 video interfaces model the DMA engines. They are exposed to userspace 68as V4L2 video device nodes. 69 70Capturing frames in raw Bayer format 71------------------------------------ 72 73CIO2 MIPI CSI2 receiver is used to capture frames (in packed raw Bayer format) 74from the raw sensors connected to the CSI2 ports. The captured frames are used 75as input to the ImgU driver. 76 77Image processing using IPU3 ImgU requires tools such as raw2pnm [#f1]_, and 78yavta [#f2]_ due to the following unique requirements and / or features specific 79to IPU3. 80 81-- The IPU3 CSI2 receiver outputs the captured frames from the sensor in packed 82raw Bayer format that is specific to IPU3. 83 84-- Multiple video nodes have to be operated simultaneously. 85 86Let us take the example of ov5670 sensor connected to CSI2 port 0, for a 872592x1944 image capture. 88 89Using the media contorller APIs, the ov5670 sensor is configured to send 90frames in packed raw Bayer format to IPU3 CSI2 receiver. 91 92# This example assumes /dev/media0 as the CIO2 media device 93 94export MDEV=/dev/media0 95 96# and that ov5670 sensor is connected to i2c bus 10 with address 0x36 97 98export SDEV=$(media-ctl -d $MDEV -e "ov5670 10-0036") 99 100# Establish the link for the media devices using media-ctl [#f3]_ 101media-ctl -d $MDEV -l "ov5670:0 -> ipu3-csi2 0:0[1]" 102 103# Set the format for the media devices 104media-ctl -d $MDEV -V "ov5670:0 [fmt:SGRBG10/2592x1944]" 105 106media-ctl -d $MDEV -V "ipu3-csi2 0:0 [fmt:SGRBG10/2592x1944]" 107 108media-ctl -d $MDEV -V "ipu3-csi2 0:1 [fmt:SGRBG10/2592x1944]" 109 110Once the media pipeline is configured, desired sensor specific settings 111(such as exposure and gain settings) can be set, using the yavta tool. 112 113e.g 114 115yavta -w 0x009e0903 444 $SDEV 116 117yavta -w 0x009e0913 1024 $SDEV 118 119yavta -w 0x009e0911 2046 $SDEV 120 121Once the desired sensor settings are set, frame captures can be done as below. 122 123e.g 124 125yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin \ 126 -f IPU3_SGRBG10 $(media-ctl -d $MDEV -e "ipu3-cio2 0") 127 128With the above command, 10 frames are captured at 2592x1944 resolution, with 129sGRBG10 format and output as IPU3_SGRBG10 format. 130 131The captured frames are available as /tmp/frame-#.bin files. 132 133ImgU 134==== 135 136The ImgU is represented as two V4L2 subdevs, each of which provides a V4L2 137subdev interface to the user space. 138 139Each V4L2 subdev represents a pipe, which can support a maximum of 2 streams. 140This helps to support advanced camera features like Continuous View Finder (CVF) 141and Snapshot During Video(SDV). 142 143The ImgU contains two independent pipes, each modelled as a V4L2 sub-device 144exposed to userspace as a V4L2 sub-device node. 145 146Each pipe has two sink pads and three source pads for the following purpose: 147 148.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}| 149 150.. flat-table:: 151 152 * - pad 153 - direction 154 - purpose 155 156 * - 0 157 - sink 158 - Input raw video stream 159 160 * - 1 161 - sink 162 - Processing parameters 163 164 * - 2 165 - source 166 - Output processed video stream 167 168 * - 3 169 - source 170 - Output viewfinder video stream 171 172 * - 4 173 - source 174 - 3A statistics 175 176Each pad is connected to a corresponding V4L2 video interface, exposed to 177userspace as a V4L2 video device node. 178 179Device operation 180---------------- 181 182With ImgU, once the input video node ("ipu3-imgu 0/1":0, in 183<entity>:<pad-number> format) is queued with buffer (in packed raw Bayer 184format), ImgU starts processing the buffer and produces the video output in YUV 185format and statistics output on respective output nodes. The driver is expected 186to have buffers ready for all of parameter, output and statistics nodes, when 187input video node is queued with buffer. 188 189At a minimum, all of input, main output, 3A statistics and viewfinder 190video nodes should be enabled for IPU3 to start image processing. 191 192Each ImgU V4L2 subdev has the following set of video nodes. 193 194input, output and viewfinder video nodes 195---------------------------------------- 196 197The frames (in packed raw Bayer format specific to the IPU3) received by the 198input video node is processed by the IPU3 Imaging Unit and are output to 2 video 199nodes, with each targeting a different purpose (main output and viewfinder 200output). 201 202Details onand the Bayer format specific to the IPU3 can be found in 203:ref:`v4l2-pix-fmt-ipu3-sbggr10`. 204 205The driver supports V4L2 Video Capture Interface as defined at :ref:`devices`. 206 207Only the multi-planar API is supported. More details can be found at 208:ref:`planar-apis`. 209 210Parameters video node 211--------------------- 212 213The parameters video node receives the ImgU algorithm parameters that are used 214to configure how the ImgU algorithms process the image. 215 216Details on processing parameters specific to the IPU3 can be found in 217:ref:`v4l2-meta-fmt-params`. 218 2193A statistics video node 220------------------------ 221 2223A statistics video node is used by the ImgU driver to output the 3A (auto 223focus, auto exposure and auto white balance) statistics for the frames that are 224being processed by the ImgU to user space applications. User space applications 225can use this statistics data to compute the desired algorithm parameters for 226the ImgU. 227 228Configuring the Intel IPU3 229========================== 230 231The IPU3 ImgU pipelines can be configured using the Media Controller, defined at 232:ref:`media_controller`. 233 234Running mode and firmware binary selection 235------------------------------------------ 236 237ImgU works based on firmware, currently the ImgU firmware support run 2 pipes in 238time-sharing with single input frame data. Each pipe can run at certain mode - 239"VIDEO" or "STILL", "VIDEO" mode is commonly used for video frames capture, and 240"STILL" is used for still frame capture. However, you can also select "VIDEO" to 241capture still frames if you want to capture images with less system load and 242power. For "STILL" mode, ImgU will try to use smaller BDS factor and output 243larger bayer frame for further YUV processing than "VIDEO" mode to get high 244quality images. Besides, "STILL" mode need XNR3 to do noise reduction, hence 245"STILL" mode will need more power and memory bandwidth than "VIDEO" mode. TNR 246will be enabled in "VIDEO" mode and bypassed by "STILL" mode. ImgU is running at 247“VIDEO” mode by default, the user can use v4l2 control V4L2_CID_INTEL_IPU3_MODE 248(currently defined in drivers/staging/media/ipu3/include/intel-ipu3.h) to query 249and set the running mode. For user, there is no difference for buffer queueing 250between the "VIDEO" and "STILL" mode, mandatory input and main output node 251should be enabled and buffers need be queued, the statistics and the view-finder 252queues are optional. 253 254The firmware binary will be selected according to current running mode, such log 255"using binary if_to_osys_striped " or "using binary if_to_osys_primary_striped" 256could be observed if you enable the ImgU dynamic debug, the binary 257if_to_osys_striped is selected for "VIDEO" and the binary 258"if_to_osys_primary_striped" is selected for "STILL". 259 260 261Processing the image in raw Bayer format 262---------------------------------------- 263 264Configuring ImgU V4L2 subdev for image processing 265~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 266 267The ImgU V4L2 subdevs have to be configured with media controller APIs to have 268all the video nodes setup correctly. 269 270Let us take "ipu3-imgu 0" subdev as an example. 271 272media-ctl -d $MDEV -r 273 274media-ctl -d $MDEV -l "ipu3-imgu 0 input":0 -> "ipu3-imgu 0":0[1] 275 276media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "ipu3-imgu 0 output":0[1] 277 278media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "ipu3-imgu 0 viewfinder":0[1] 279 280media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "ipu3-imgu 0 3a stat":0[1] 281 282Also the pipe mode of the corresponding V4L2 subdev should be set as desired 283(e.g 0 for video mode or 1 for still mode) through the control id 0x009819a1 as 284below. 285 286yavta -w "0x009819A1 1" /dev/v4l-subdev7 287 288Certain hardware blocks in ImgU pipeline can change the frame resolution by 289cropping or scaling, these hardware blocks include Input Feeder(IF), Bayer Down 290Scaler (BDS) and Geometric Distortion Correction (GDC). 291There is also a block which can change the frame resolution - YUV Scaler, it is 292only applicable to the secondary output. 293 294RAW Bayer frames go through these ImgU pipeline hardware blocks and the final 295processed image output to the DDR memory. 296 297.. kernel-figure:: ipu3_rcb.svg 298 :alt: ipu3 resolution blocks image 299 300 IPU3 resolution change hardware blocks 301 302**Input Feeder** 303 304Input Feeder gets the Bayer frame data from the sensor, it can enable cropping 305of lines and columns from the frame and then store pixels into device's internal 306pixel buffer which are ready to readout by following blocks. 307 308**Bayer Down Scaler** 309 310Bayer Down Scaler is capable of performing image scaling in Bayer domain, the 311downscale factor can be configured from 1X to 1/4X in each axis with 312configuration steps of 0.03125 (1/32). 313 314**Geometric Distortion Correction** 315 316Geometric Distortion Correction is used to performe correction of distortions 317and image filtering. It needs some extra filter and envelop padding pixels to 318work, so the input resolution of GDC should be larger than the output 319resolution. 320 321**YUV Scaler** 322 323YUV Scaler which similar with BDS, but it is mainly do image down scaling in 324YUV domain, it can support up to 1/12X down scaling, but it can not be applied 325to the main output. 326 327The ImgU V4L2 subdev has to be configured with the supported resolutions in all 328the above hardware blocks, for a given input resolution. 329For a given supported resolution for an input frame, the Input Feeder, Bayer 330Down Scaler and GDC blocks should be configured with the supported resolutions 331as each hardware block has its own alignment requirement. 332 333You must configure the output resolution of the hardware blocks smartly to meet 334the hardware requirement along with keeping the maximum field of view. The 335intermediate resolutions can be generated by specific tool - 336 337https://github.com/intel/intel-ipu3-pipecfg 338 339This tool can be used to generate intermediate resolutions. More information can 340be obtained by looking at the following IPU3 ImgU configuration table. 341 342https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master 343 344Under baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss 345directory, graph_settings_ov5670.xml can be used as an example. 346 347The following steps prepare the ImgU pipeline for the image processing. 348 3491. The ImgU V4L2 subdev data format should be set by using the 350VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained above. 351 3522. The ImgU V4L2 subdev cropping should be set by using the 353VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the target, 354using the input feeder height and width. 355 3563. The ImgU V4L2 subdev composing should be set by using the 357VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the target, 358using the BDS height and width. 359 360For the ov5670 example, for an input frame with a resolution of 2592x1944 361(which is input to the ImgU subdev pad 0), the corresponding resolutions 362for input feeder, BDS and GDC are 2592x1944, 2592x1944 and 2560x1920 363respectively. 364 365Once this is done, the received raw Bayer frames can be input to the ImgU 366V4L2 subdev as below, using the open source application v4l2n [#f1]_. 367 368For an image captured with 2592x1944 [#f4]_ resolution, with desired output 369resolution as 2560x1920 and viewfinder resolution as 2560x1920, the following 370v4l2n command can be used. This helps process the raw Bayer frames and produces 371the desired results for the main output image and the viewfinder output, in NV12 372format. 373 374v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4 375--fmt=type:VIDEO_OUTPUT_MPLANE,width=2592,height=1944,pixelformat=0X47337069 376--reqbufs=type:VIDEO_OUTPUT_MPLANE,count:1 --pipe=1 --output=/tmp/frames.out 377--open=/dev/video5 378--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12 379--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=2 --output=/tmp/frames.vf 380--open=/dev/video6 381--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12 382--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=3 --open=/dev/video7 383--output=/tmp/frames.3A --fmt=type:META_CAPTURE,? 384--reqbufs=count:1,type:META_CAPTURE --pipe=1,2,3,4 --stream=5 385 386You can also use yavta [#f2]_ command to do same thing as above: 387 388.. code-block:: none 389 390 yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \ 391 --file=frame-#.out-f NV12 /dev/video5 & \ 392 yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \ 393 --file=frame-#.vf -f NV12 /dev/video6 & \ 394 yavta --data-prefix -Bmeta-capture -c10 -n5 -I \ 395 --file=frame-#.3a /dev/video7 & \ 396 yavta --data-prefix -Boutput-mplane -c10 -n5 -I -s2592x1944 \ 397 --file=/tmp/frame-in.cio2 -f IPU3_SGRBG10 /dev/video4 398 399where /dev/video4, /dev/video5, /dev/video6 and /dev/video7 devices point to 400input, output, viewfinder and 3A statistics video nodes respectively. 401 402Converting the raw Bayer image into YUV domain 403---------------------------------------------- 404 405The processed images after the above step, can be converted to YUV domain 406as below. 407 408Main output frames 409~~~~~~~~~~~~~~~~~~ 410 411raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.ppm 412 413where 2560x1920 is output resolution, NV12 is the video format, followed 414by input frame and output PNM file. 415 416Viewfinder output frames 417~~~~~~~~~~~~~~~~~~~~~~~~ 418 419raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.ppm 420 421where 2560x1920 is output resolution, NV12 is the video format, followed 422by input frame and output PNM file. 423 424Example user space code for IPU3 425================================ 426 427User space code that configures and uses IPU3 is available here. 428 429https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master/ 430 431The source can be located under hal/intel directory. 432 433Overview of IPU3 pipeline 434========================= 435 436IPU3 pipeline has a number of image processing stages, each of which takes a 437set of parameters as input. The major stages of pipelines are shown here: 438 439.. kernel-render:: DOT 440 :alt: IPU3 ImgU Pipeline 441 :caption: IPU3 ImgU Pipeline Diagram 442 443 digraph "IPU3 ImgU" { 444 node [shape=box] 445 splines="ortho" 446 rankdir="LR" 447 448 a [label="Raw pixels"] 449 b [label="Bayer Downscaling"] 450 c [label="Optical Black Correction"] 451 d [label="Linearization"] 452 e [label="Lens Shading Correction"] 453 f [label="White Balance / Exposure / Focus Apply"] 454 g [label="Bayer Noise Reduction"] 455 h [label="ANR"] 456 i [label="Demosaicing"] 457 j [label="Color Correction Matrix"] 458 k [label="Gamma correction"] 459 l [label="Color Space Conversion"] 460 m [label="Chroma Down Scaling"] 461 n [label="Chromatic Noise Reduction"] 462 o [label="Total Color Correction"] 463 p [label="XNR3"] 464 q [label="TNR"] 465 r [label="DDR", style=filled, fillcolor=yellow, shape=cylinder] 466 s [label="YUV Downscaling"] 467 t [label="DDR", style=filled, fillcolor=yellow, shape=cylinder] 468 469 { rank=same; a -> b -> c -> d -> e -> f -> g -> h -> i } 470 { rank=same; j -> k -> l -> m -> n -> o -> p -> q -> s -> t} 471 472 a -> j [style=invis, weight=10] 473 i -> j 474 q -> r 475 } 476 477The table below presents a description of the above algorithms. 478 479======================== ======================================================= 480Name Description 481======================== ======================================================= 482Optical Black Correction Optical Black Correction block subtracts a pre-defined 483 value from the respective pixel values to obtain better 484 image quality. 485 Defined in :c:type:`ipu3_uapi_obgrid_param`. 486Linearization This algo block uses linearization parameters to 487 address non-linearity sensor effects. The Lookup table 488 table is defined in 489 :c:type:`ipu3_uapi_isp_lin_vmem_params`. 490SHD Lens shading correction is used to correct spatial 491 non-uniformity of the pixel response due to optical 492 lens shading. This is done by applying a different gain 493 for each pixel. The gain, black level etc are 494 configured in :c:type:`ipu3_uapi_shd_config_static`. 495BNR Bayer noise reduction block removes image noise by 496 applying a bilateral filter. 497 See :c:type:`ipu3_uapi_bnr_static_config` for details. 498ANR Advanced Noise Reduction is a block based algorithm 499 that performs noise reduction in the Bayer domain. The 500 convolution matrix etc can be found in 501 :c:type:`ipu3_uapi_anr_config`. 502DM Demosaicing converts raw sensor data in Bayer format 503 into RGB (Red, Green, Blue) presentation. Then add 504 outputs of estimation of Y channel for following stream 505 processing by Firmware. The struct is defined as 506 :c:type:`ipu3_uapi_dm_config`. 507Color Correction Color Correction algo transforms sensor specific color 508 space to the standard "sRGB" color space. This is done 509 by applying 3x3 matrix defined in 510 :c:type:`ipu3_uapi_ccm_mat_config`. 511Gamma correction Gamma correction :c:type:`ipu3_uapi_gamma_config` is a 512 basic non-linear tone mapping correction that is 513 applied per pixel for each pixel component. 514CSC Color space conversion transforms each pixel from the 515 RGB primary presentation to YUV (Y: brightness, 516 UV: Luminance) presentation. This is done by applying 517 a 3x3 matrix defined in 518 :c:type:`ipu3_uapi_csc_mat_config` 519CDS Chroma down sampling 520 After the CSC is performed, the Chroma Down Sampling 521 is applied for a UV plane down sampling by a factor 522 of 2 in each direction for YUV 4:2:0 using a 4x2 523 configurable filter :c:type:`ipu3_uapi_cds_params`. 524CHNR Chroma noise reduction 525 This block processes only the chrominance pixels and 526 performs noise reduction by cleaning the high 527 frequency noise. 528 See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`. 529TCC Total color correction as defined in struct 530 :c:type:`ipu3_uapi_yuvp2_tcc_static_config`. 531XNR3 eXtreme Noise Reduction V3 is the third revision of 532 noise reduction algorithm used to improve image 533 quality. This removes the low frequency noise in the 534 captured image. Two related structs are being defined, 535 :c:type:`ipu3_uapi_isp_xnr3_params` for ISP data memory 536 and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for vector 537 memory. 538TNR Temporal Noise Reduction block compares successive 539 frames in time to remove anomalies / noise in pixel 540 values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params` and 541 :c:type:`ipu3_uapi_isp_tnr3_params` are defined for ISP 542 vector and data memory respectively. 543======================== ======================================================= 544 545Other often encountered acronyms not listed in above table: 546 547 ACC 548 Accelerator cluster 549 AWB_FR 550 Auto white balance filter response statistics 551 BDS 552 Bayer downscaler parameters 553 CCM 554 Color correction matrix coefficients 555 IEFd 556 Image enhancement filter directed 557 Obgrid 558 Optical black level compensation 559 OSYS 560 Output system configuration 561 ROI 562 Region of interest 563 YDS 564 Y down sampling 565 YTM 566 Y-tone mapping 567 568A few stages of the pipeline will be executed by firmware running on the ISP 569processor, while many others will use a set of fixed hardware blocks also 570called accelerator cluster (ACC) to crunch pixel data and produce statistics. 571 572ACC parameters of individual algorithms, as defined by 573:c:type:`ipu3_uapi_acc_param`, can be chosen to be applied by the user 574space through struct :c:type:`ipu3_uapi_flags` embedded in 575:c:type:`ipu3_uapi_params` structure. For parameters that are configured as 576not enabled by the user space, the corresponding structs are ignored by the 577driver, in which case the existing configuration of the algorithm will be 578preserved. 579 580References 581========== 582 583.. [#f5] drivers/staging/media/ipu3/include/intel-ipu3.h 584 585.. [#f1] https://github.com/intel/nvt 586 587.. [#f2] http://git.ideasonboard.org/yavta.git 588 589.. [#f3] http://git.ideasonboard.org/?p=media-ctl.git;a=summary 590 591.. [#f4] ImgU limitation requires an additional 16x16 for all input resolutions 592
