1.. SPDX-License-Identifier: GPL-2.0 2 3======================== 4Linux Color Pipeline API 5======================== 6 7What problem are we solving? 8============================ 9 10We would like to support pre-, and post-blending complex color 11transformations in display controller hardware in order to allow for 12HW-supported HDR use-cases, as well as to provide support to 13color-managed applications, such as video or image editors. 14 15It is possible to support an HDR output on HW supporting the Colorspace 16and HDR Metadata drm_connector properties, but that requires the 17compositor or application to render and compose the content into one 18final buffer intended for display. Doing so is costly. 19 20Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other 21operations to support color transformations. These operations are often 22implemented in fixed-function HW and therefore much more power efficient than 23performing similar operations via shaders or CPU. 24 25We would like to make use of this HW functionality to support complex color 26transformations with no, or minimal CPU or shader load. The switch between HW 27fixed-function blocks and shaders/CPU must be seamless with no visible 28difference when fallback to shaders/CPU is neceesary at any time. 29 30 31How are other OSes solving this problem? 32======================================== 33 34The most widely supported use-cases regard HDR content, whether video or 35gaming. 36 37Most OSes will specify the source content format (color gamut, encoding transfer 38function, and other metadata, such as max and average light levels) to a driver. 39Drivers will then program their fixed-function HW accordingly to map from a 40source content buffer's space to a display's space. 41 42When fixed-function HW is not available the compositor will assemble a shader to 43ask the GPU to perform the transformation from the source content format to the 44display's format. 45 46A compositor's mapping function and a driver's mapping function are usually 47entirely separate concepts. On OSes where a HW vendor has no insight into 48closed-source compositor code such a vendor will tune their color management 49code to visually match the compositor's. On other OSes, where both mapping 50functions are open to an implementer they will ensure both mappings match. 51 52This results in mapping algorithm lock-in, meaning that no-one alone can 53experiment with or introduce new mapping algorithms and achieve 54consistent results regardless of which implementation path is taken. 55 56Why is Linux different? 57======================= 58 59Unlike other OSes, where there is one compositor for one or more drivers, on 60Linux we have a many-to-many relationship. Many compositors; many drivers. 61In addition each compositor vendor or community has their own view of how 62color management should be done. This is what makes Linux so beautiful. 63 64This means that a HW vendor can now no longer tune their driver to one 65compositor, as tuning it to one could make it look fairly different from 66another compositor's color mapping. 67 68We need a better solution. 69 70 71Descriptive API 72=============== 73 74An API that describes the source and destination colorspaces is a descriptive 75API. It describes the input and output color spaces but does not describe 76how precisely they should be mapped. Such a mapping includes many minute 77design decision that can greatly affect the look of the final result. 78 79It is not feasible to describe such mapping with enough detail to ensure the 80same result from each implementation. In fact, these mappings are a very active 81research area. 82 83 84Prescriptive API 85================ 86 87A prescriptive API describes not the source and destination colorspaces. It 88instead prescribes a recipe for how to manipulate pixel values to arrive at the 89desired outcome. 90 91This recipe is generally an ordered list of straight-forward operations, 92with clear mathematical definitions, such as 1D LUTs, 3D LUTs, matrices, 93or other operations that can be described in a precise manner. 94 95 96The Color Pipeline API 97====================== 98 99HW color management pipelines can significantly differ between HW 100vendors in terms of availability, ordering, and capabilities of HW 101blocks. This makes a common definition of color management blocks and 102their ordering nigh impossible. Instead we are defining an API that 103allows user space to discover the HW capabilities in a generic manner, 104agnostic of specific drivers and hardware. 105 106 107drm_colorop Object 108================== 109 110To support the definition of color pipelines we define the DRM core 111object type drm_colorop. Individual drm_colorop objects will be chained 112via the NEXT property of a drm_colorop to constitute a color pipeline. 113Each drm_colorop object is unique, i.e., even if multiple color 114pipelines have the same operation they won't share the same drm_colorop 115object to describe that operation. 116 117Note that drivers are not expected to map drm_colorop objects statically 118to specific HW blocks. The mapping of drm_colorop objects is entirely a 119driver-internal detail and can be as dynamic or static as a driver needs 120it to be. See more in the Driver Implementation Guide section below. 121 122Each drm_colorop has three core properties: 123 124TYPE: An enumeration property, defining the type of transformation, such as 125* enumerated curve 126* custom (uniform) 1D LUT 127* 3x3 matrix 128* 3x4 matrix 129* 3D LUT 130* etc. 131 132Depending on the type of transformation other properties will describe 133more details. 134 135BYPASS: A boolean property that can be used to easily put a block into 136bypass mode. The BYPASS property is not mandatory for a colorop, as long 137as the entire pipeline can get bypassed by setting the COLOR_PIPELINE on 138a plane to '0'. 139 140NEXT: The ID of the next drm_colorop in a color pipeline, or 0 if this 141drm_colorop is the last in the chain. 142 143An example of a drm_colorop object might look like one of these:: 144 145 /* 1D enumerated curve */ 146 Color operation 42 147 ├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 1D enumerated curve 148 ├─ "BYPASS": bool {true, false} 149 ├─ "CURVE_1D_TYPE": enum {sRGB EOTF, sRGB inverse EOTF, PQ EOTF, PQ inverse EOTF, …} 150 └─ "NEXT": immutable color operation ID = 43 151 152 /* custom 4k entry 1D LUT */ 153 Color operation 52 154 ├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 1D LUT 155 ├─ "BYPASS": bool {true, false} 156 ├─ "SIZE": immutable range = 4096 157 ├─ "DATA": blob 158 └─ "NEXT": immutable color operation ID = 0 159 160 /* 17^3 3D LUT */ 161 Color operation 72 162 ├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 3D LUT 163 ├─ "BYPASS": bool {true, false} 164 ├─ "SIZE": immutable range = 17 165 ├─ "DATA": blob 166 └─ "NEXT": immutable color operation ID = 73 167 168drm_colorop extensibility 169------------------------- 170 171Unlike existing DRM core objects, like &drm_plane, drm_colorop is not 172extensible. This simplifies implementations and keeps all functionality 173for managing &drm_colorop objects in the DRM core. 174 175If there is a need one may introduce a simple &drm_colorop_funcs 176function table in the future, for example to support an IN_FORMATS 177property on a &drm_colorop. 178 179If a driver requires the ability to create a driver-specific colorop 180object they will need to add &drm_colorop func table support with 181support for the usual functions, like destroy, atomic_duplicate_state, 182and atomic_destroy_state. 183 184 185COLOR_PIPELINE Plane Property 186============================= 187 188Color Pipelines are created by a driver and advertised via a new 189COLOR_PIPELINE enum property on each plane. Values of the property 190always include object id 0, which is the default and means all color 191processing is disabled. Additional values will be the object IDs of the 192first drm_colorop in a pipeline. A driver can create and advertise none, 193one, or more possible color pipelines. A DRM client will select a color 194pipeline by setting the COLOR PIPELINE to the respective value. 195 196NOTE: Many DRM clients will set enumeration properties via the string 197value, often hard-coding it. Since this enumeration is generated based 198on the colorop object IDs it is important to perform the Color Pipeline 199Discovery, described below, instead of hard-coding color pipeline 200assignment. Drivers might generate the enum strings dynamically. 201Hard-coded strings might only work for specific drivers on a specific 202pieces of HW. Color Pipeline Discovery can work universally, as long as 203drivers implement the required color operations. 204 205The COLOR_PIPELINE property is only exposed when the 206DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE is set. Drivers shall ignore any 207existing pre-blend color operations when this cap is set, such as 208COLOR_RANGE and COLOR_ENCODING. If drivers want to support COLOR_RANGE 209or COLOR_ENCODING functionality when the color pipeline client cap is 210set, they are expected to expose colorops in the pipeline to allow for 211the appropriate color transformation. 212 213Setting of the COLOR_PIPELINE plane property or drm_colorop properties 214is only allowed for userspace that sets this client cap. 215 216An example of a COLOR_PIPELINE property on a plane might look like this:: 217 218 Plane 10 219 ├─ "TYPE": immutable enum {Overlay, Primary, Cursor} = Primary 220 ├─ … 221 └─ "COLOR_PIPELINE": enum {0, 42, 52} = 0 222 223 224Color Pipeline Discovery 225======================== 226 227A DRM client wanting color management on a drm_plane will: 228 2291. Get the COLOR_PIPELINE property of the plane 2302. iterate all COLOR_PIPELINE enum values 2313. for each enum value walk the color pipeline (via the NEXT pointers) 232 and see if the available color operations are suitable for the 233 desired color management operations 234 235If userspace encounters an unknown or unsuitable color operation during 236discovery it does not need to reject the entire color pipeline outright, 237as long as the unknown or unsuitable colorop has a "BYPASS" property. 238Drivers will ensure that a bypassed block does not have any effect. 239 240An example of chained properties to define an AMD pre-blending color 241pipeline might look like this:: 242 243 Plane 10 244 ├─ "TYPE" (immutable) = Primary 245 └─ "COLOR_PIPELINE": enum {0, 44} = 0 246 247 Color operation 44 248 ├─ "TYPE" (immutable) = 1D enumerated curve 249 ├─ "BYPASS": bool 250 ├─ "CURVE_1D_TYPE": enum {sRGB EOTF, PQ EOTF} = sRGB EOTF 251 └─ "NEXT" (immutable) = 45 252 253 Color operation 45 254 ├─ "TYPE" (immutable) = 3x4 Matrix 255 ├─ "BYPASS": bool 256 ├─ "DATA": blob 257 └─ "NEXT" (immutable) = 46 258 259 Color operation 46 260 ├─ "TYPE" (immutable) = 1D enumerated curve 261 ├─ "BYPASS": bool 262 ├─ "CURVE_1D_TYPE": enum {sRGB Inverse EOTF, PQ Inverse EOTF} = sRGB EOTF 263 └─ "NEXT" (immutable) = 47 264 265 Color operation 47 266 ├─ "TYPE" (immutable) = 1D LUT 267 ├─ "SIZE": immutable range = 4096 268 ├─ "DATA": blob 269 └─ "NEXT" (immutable) = 48 270 271 Color operation 48 272 ├─ "TYPE" (immutable) = 3D LUT 273 ├─ "DATA": blob 274 └─ "NEXT" (immutable) = 49 275 276 Color operation 49 277 ├─ "TYPE" (immutable) = 1D enumerated curve 278 ├─ "BYPASS": bool 279 ├─ "CURVE_1D_TYPE": enum {sRGB EOTF, PQ EOTF} = sRGB EOTF 280 └─ "NEXT" (immutable) = 0 281 282 283Color Pipeline Programming 284========================== 285 286Once a DRM client has found a suitable pipeline it will: 287 2881. Set the COLOR_PIPELINE enum value to the one pointing at the first 289 drm_colorop object of the desired pipeline 2902. Set the properties for all drm_colorop objects in the pipeline to the 291 desired values, setting BYPASS to true for unused drm_colorop blocks, 292 and false for enabled drm_colorop blocks 2933. Perform (TEST_ONLY or not) atomic commit with all the other KMS 294 states it wishes to change 295 296To configure the pipeline for an HDR10 PQ plane and blending in linear 297space, a compositor might perform an atomic commit with the following 298property values:: 299 300 Plane 10 301 └─ "COLOR_PIPELINE" = 42 302 303 Color operation 42 304 └─ "BYPASS" = true 305 306 Color operation 44 307 └─ "BYPASS" = true 308 309 Color operation 45 310 └─ "BYPASS" = true 311 312 Color operation 46 313 └─ "BYPASS" = true 314 315 Color operation 47 316 ├─ "DATA" = Gamut mapping + tone mapping + night mode 317 └─ "BYPASS" = false 318 319 Color operation 48 320 ├─ "CURVE_1D_TYPE" = PQ EOTF 321 └─ "BYPASS" = false 322 323 324Driver Implementer's Guide 325========================== 326 327What does this all mean for driver implementations? As noted above the 328colorops can map to HW directly but don't need to do so. Here are some 329suggestions on how to think about creating your color pipelines: 330 331- Try to expose pipelines that use already defined colorops, even if 332 your hardware pipeline is split differently. This allows existing 333 userspace to immediately take advantage of the hardware. 334 335- Additionally, try to expose your actual hardware blocks as colorops. 336 Define new colorop types where you believe it can offer significant 337 benefits if userspace learns to program them. 338 339- Avoid defining new colorops for compound operations with very narrow 340 scope. If you have a hardware block for a special operation that 341 cannot be split further, you can expose that as a new colorop type. 342 However, try to not define colorops for "use cases", especially if 343 they require you to combine multiple hardware blocks. 344 345- Design new colorops as prescriptive, not descriptive; by the 346 mathematical formula, not by the assumed input and output. 347 348A defined colorop type must be deterministic. The exact behavior of the 349colorop must be documented entirely, whether via a mathematical formula 350or some other description. Its operation can depend only on its 351properties and input and nothing else, allowed error tolerance 352notwithstanding. 353 354 355Driver Forward/Backward Compatibility 356===================================== 357 358As this is uAPI drivers can't regress color pipelines that have been 359introduced for a given HW generation. New HW generations are free to 360abandon color pipelines advertised for previous generations. 361Nevertheless, it can be beneficial to carry support for existing color 362pipelines forward as those will likely already have support in DRM 363clients. 364 365Introducing new colorops to a pipeline is fine, as long as they can be 366bypassed or are purely informational. DRM clients implementing support 367for the pipeline can always skip unknown properties as long as they can 368be confident that doing so will not cause unexpected results. 369 370If a new colorop doesn't fall into one of the above categories 371(bypassable or informational) the modified pipeline would be unusable 372for user space. In this case a new pipeline should be defined. 373 374 375References 376========== 377 3781. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/