xref: /linux/Documentation/admin-guide/media/ipu6-isys.rst (revision ba124c8cf3b775c1575fc557203527e211d1fdc5)
1*ba124c8cSBingbu Cao.. SPDX-License-Identifier: GPL-2.0
2*ba124c8cSBingbu Cao
3*ba124c8cSBingbu Cao.. include:: <isonum.txt>
4*ba124c8cSBingbu Cao
5*ba124c8cSBingbu Cao========================================================
6*ba124c8cSBingbu CaoIntel Image Processing Unit 6 (IPU6) Input System driver
7*ba124c8cSBingbu Cao========================================================
8*ba124c8cSBingbu Cao
9*ba124c8cSBingbu CaoCopyright |copy| 2023--2024 Intel Corporation
10*ba124c8cSBingbu Cao
11*ba124c8cSBingbu CaoIntroduction
12*ba124c8cSBingbu Cao============
13*ba124c8cSBingbu Cao
14*ba124c8cSBingbu CaoThis file documents the Intel IPU6 (6th generation Image Processing Unit)
15*ba124c8cSBingbu CaoInput System (MIPI CSI2 receiver) drivers located under
16*ba124c8cSBingbu Caodrivers/media/pci/intel/ipu6.
17*ba124c8cSBingbu Cao
18*ba124c8cSBingbu CaoThe Intel IPU6 can be found in certain Intel SoCs but not in all SKUs:
19*ba124c8cSBingbu Cao
20*ba124c8cSBingbu Cao* Tiger Lake
21*ba124c8cSBingbu Cao* Jasper Lake
22*ba124c8cSBingbu Cao* Alder Lake
23*ba124c8cSBingbu Cao* Raptor Lake
24*ba124c8cSBingbu Cao* Meteor Lake
25*ba124c8cSBingbu Cao
26*ba124c8cSBingbu CaoIntel IPU6 is made up of two components - Input System (ISYS) and Processing
27*ba124c8cSBingbu CaoSystem (PSYS).
28*ba124c8cSBingbu Cao
29*ba124c8cSBingbu CaoThe Input System mainly works as MIPI CSI-2 receiver which receives and
30*ba124c8cSBingbu Caoprocesses the image data from the sensors and outputs the frames to memory.
31*ba124c8cSBingbu Cao
32*ba124c8cSBingbu CaoThere are 2 driver modules - intel-ipu6 and intel-ipu6-isys. intel-ipu6 is an
33*ba124c8cSBingbu CaoIPU6 common driver which does PCI configuration, firmware loading and parsing,
34*ba124c8cSBingbu Caofirmware authentication, DMA mapping and IPU-MMU (internal Memory mapping Unit)
35*ba124c8cSBingbu Caoconfiguration. intel_ipu6_isys implements V4L2, Media Controller and V4L2
36*ba124c8cSBingbu Caosub-device interfaces. The IPU6 ISYS driver supports camera sensors connected
37*ba124c8cSBingbu Caoto the IPU6 ISYS through V4L2 sub-device sensor drivers.
38*ba124c8cSBingbu Cao
39*ba124c8cSBingbu Cao.. Note:: See Documentation/driver-api/media/drivers/ipu6.rst for more
40*ba124c8cSBingbu Cao	  information about the IPU6 hardware.
41*ba124c8cSBingbu Cao
42*ba124c8cSBingbu CaoInput system driver
43*ba124c8cSBingbu Cao===================
44*ba124c8cSBingbu Cao
45*ba124c8cSBingbu CaoThe Input System driver mainly configures CSI-2 D-PHY, constructs the firmware
46*ba124c8cSBingbu Caostream configuration, sends commands to firmware, gets response from hardware
47*ba124c8cSBingbu Caoand firmware and then returns buffers to user.  The ISYS is represented as
48*ba124c8cSBingbu Caoseveral V4L2 sub-devices as well as video nodes.
49*ba124c8cSBingbu Cao
50*ba124c8cSBingbu Cao.. kernel-figure::  ipu6_isys_graph.svg
51*ba124c8cSBingbu Cao   :alt: ipu6 isys media graph with multiple streams support
52*ba124c8cSBingbu Cao
53*ba124c8cSBingbu Cao   IPU6 ISYS media graph with multiple streams support
54*ba124c8cSBingbu Cao
55*ba124c8cSBingbu CaoThe graph has been produced using the following command:
56*ba124c8cSBingbu Cao
57*ba124c8cSBingbu Cao.. code-block:: none
58*ba124c8cSBingbu Cao
59*ba124c8cSBingbu Cao   fdp -Gsplines=true -Tsvg < dot > dot.svg
60*ba124c8cSBingbu Cao
61*ba124c8cSBingbu CaoCapturing frames with IPU6 ISYS
62*ba124c8cSBingbu Cao-------------------------------
63*ba124c8cSBingbu Cao
64*ba124c8cSBingbu CaoIPU6 ISYS is used to capture frames from the camera sensors connected to the
65*ba124c8cSBingbu CaoCSI2 ports. The supported input formats of ISYS are listed in table below:
66*ba124c8cSBingbu Cao
67*ba124c8cSBingbu Cao.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
68*ba124c8cSBingbu Cao
69*ba124c8cSBingbu Cao.. flat-table::
70*ba124c8cSBingbu Cao    :header-rows: 1
71*ba124c8cSBingbu Cao
72*ba124c8cSBingbu Cao    * - IPU6 ISYS supported input formats
73*ba124c8cSBingbu Cao
74*ba124c8cSBingbu Cao    * - RGB565, RGB888
75*ba124c8cSBingbu Cao
76*ba124c8cSBingbu Cao    * - UYVY8, YUYV8
77*ba124c8cSBingbu Cao
78*ba124c8cSBingbu Cao    * - RAW8, RAW10, RAW12
79*ba124c8cSBingbu Cao
80*ba124c8cSBingbu Cao.. _ipu6_isys_capture_examples:
81*ba124c8cSBingbu Cao
82*ba124c8cSBingbu CaoExamples
83*ba124c8cSBingbu Cao~~~~~~~~
84*ba124c8cSBingbu Cao
85*ba124c8cSBingbu CaoHere is an example of IPU6 ISYS raw capture on Dell XPS 9315 laptop. On this
86*ba124c8cSBingbu Caomachine, ov01a10 sensor is connected to IPU ISYS CSI-2 port 2, which can
87*ba124c8cSBingbu Caogenerate images at sBGGR10 with resolution 1280x800.
88*ba124c8cSBingbu Cao
89*ba124c8cSBingbu CaoUsing the media controller APIs, we can configure ov01a10 sensor by
90*ba124c8cSBingbu Caomedia-ctl [#f1]_ and yavta [#f2]_ to transmit frames to IPU6 ISYS.
91*ba124c8cSBingbu Cao
92*ba124c8cSBingbu Cao.. code-block:: none
93*ba124c8cSBingbu Cao
94*ba124c8cSBingbu Cao    # Example 1 capture frame from ov01a10 camera sensor
95*ba124c8cSBingbu Cao    # This example assumes /dev/media0 as the IPU ISYS media device
96*ba124c8cSBingbu Cao    export MDEV=/dev/media0
97*ba124c8cSBingbu Cao
98*ba124c8cSBingbu Cao    # Establish the link for the media devices using media-ctl
99*ba124c8cSBingbu Cao    media-ctl -d $MDEV -l "\"ov01a10 3-0036\":0 -> \"Intel IPU6 CSI2 2\":0[1]"
100*ba124c8cSBingbu Cao
101*ba124c8cSBingbu Cao    # Set the format for the media devices
102*ba124c8cSBingbu Cao    media-ctl -d $MDEV -V "ov01a10:0 [fmt:SBGGR10/1280x800]"
103*ba124c8cSBingbu Cao    media-ctl -d $MDEV -V "Intel IPU6 CSI2 2:0 [fmt:SBGGR10/1280x800]"
104*ba124c8cSBingbu Cao    media-ctl -d $MDEV -V "Intel IPU6 CSI2 2:1 [fmt:SBGGR10/1280x800]"
105*ba124c8cSBingbu Cao
106*ba124c8cSBingbu CaoOnce the media pipeline is configured, desired sensor specific settings
107*ba124c8cSBingbu Cao(such as exposure and gain settings) can be set, using the yavta tool.
108*ba124c8cSBingbu Cao
109*ba124c8cSBingbu Caoe.g
110*ba124c8cSBingbu Cao
111*ba124c8cSBingbu Cao.. code-block:: none
112*ba124c8cSBingbu Cao
113*ba124c8cSBingbu Cao    # and that ov01a10 sensor is connected to i2c bus 3 with address 0x36
114*ba124c8cSBingbu Cao    export SDEV=$(media-ctl -d $MDEV -e "ov01a10 3-0036")
115*ba124c8cSBingbu Cao
116*ba124c8cSBingbu Cao    yavta -w 0x009e0903 400 $SDEV
117*ba124c8cSBingbu Cao    yavta -w 0x009e0913 1000 $SDEV
118*ba124c8cSBingbu Cao    yavta -w 0x009e0911 2000 $SDEV
119*ba124c8cSBingbu Cao
120*ba124c8cSBingbu CaoOnce the desired sensor settings are set, frame captures can be done as below.
121*ba124c8cSBingbu Cao
122*ba124c8cSBingbu Caoe.g
123*ba124c8cSBingbu Cao
124*ba124c8cSBingbu Cao.. code-block:: none
125*ba124c8cSBingbu Cao
126*ba124c8cSBingbu Cao    yavta --data-prefix -u -c10 -n5 -I -s 1280x800 --file=/tmp/frame-#.bin \
127*ba124c8cSBingbu Cao            -f SBGGR10 $(media-ctl -d $MDEV -e "Intel IPU6 ISYS Capture 0")
128*ba124c8cSBingbu Cao
129*ba124c8cSBingbu CaoWith the above command, 10 frames are captured at 1280x800 resolution with
130*ba124c8cSBingbu CaosBGGR10 format. The captured frames are available as /tmp/frame-#.bin files.
131*ba124c8cSBingbu Cao
132*ba124c8cSBingbu CaoHere is another example of IPU6 ISYS RAW and metadata capture from camera
133*ba124c8cSBingbu Caosensor ov2740 on Lenovo X1 Yoga laptop.
134*ba124c8cSBingbu Cao
135*ba124c8cSBingbu Cao.. code-block:: none
136*ba124c8cSBingbu Cao
137*ba124c8cSBingbu Cao    media-ctl -l "\"ov2740 14-0036\":0 -> \"Intel IPU6 CSI2 1\":0[1]"
138*ba124c8cSBingbu Cao    media-ctl -l "\"Intel IPU6 CSI2 1\":1 -> \"Intel IPU6 ISYS Capture 0\":0[5]"
139*ba124c8cSBingbu Cao    media-ctl -l "\"Intel IPU6 CSI2 1\":2 -> \"Intel IPU6 ISYS Capture 1\":0[5]"
140*ba124c8cSBingbu Cao
141*ba124c8cSBingbu Cao    # set routing
142*ba124c8cSBingbu Cao    media-ctl -v -R "\"Intel IPU6 CSI2 1\" [0/0->1/0[1],0/1->2/1[1]]"
143*ba124c8cSBingbu Cao
144*ba124c8cSBingbu Cao    media-ctl -v "\"Intel IPU6 CSI2 1\":0/0 [fmt:SGRBG10/1932x1092]"
145*ba124c8cSBingbu Cao    media-ctl -v "\"Intel IPU6 CSI2 1\":0/1 [fmt:GENERIC_8/97x1]"
146*ba124c8cSBingbu Cao    media-ctl -v "\"Intel IPU6 CSI2 1\":1/0 [fmt:SGRBG10/1932x1092]"
147*ba124c8cSBingbu Cao    media-ctl -v "\"Intel IPU6 CSI2 1\":2/1 [fmt:GENERIC_8/97x1]"
148*ba124c8cSBingbu Cao
149*ba124c8cSBingbu Cao    CAPTURE_DEV=$(media-ctl -e "Intel IPU6 ISYS Capture 0")
150*ba124c8cSBingbu Cao    ./yavta --data-prefix -c100 -n5 -I -s1932x1092 --file=/tmp/frame-#.bin \
151*ba124c8cSBingbu Cao        -f SGRBG10 ${CAPTURE_DEV}
152*ba124c8cSBingbu Cao
153*ba124c8cSBingbu Cao    CAPTURE_META=$(media-ctl -e "Intel IPU6 ISYS Capture 1")
154*ba124c8cSBingbu Cao    ./yavta --data-prefix -c100 -n5 -I -s97x1 -B meta-capture \
155*ba124c8cSBingbu Cao        --file=/tmp/meta-#.bin -f GENERIC_8 ${CAPTURE_META}
156*ba124c8cSBingbu Cao
157*ba124c8cSBingbu CaoReferences
158*ba124c8cSBingbu Cao==========
159*ba124c8cSBingbu Cao
160*ba124c8cSBingbu Cao.. [#f1] https://git.ideasonboard.org/media-ctl.git
161*ba124c8cSBingbu Cao.. [#f2] https://git.ideasonboard.org/yavta.git
162