1.. SPDX-License-Identifier: GPL-2.0+ 2 3========================================= 4Automated testing of the DRM subsystem 5========================================= 6 7Introduction 8============ 9 10Making sure that changes to the core or drivers don't introduce regressions can 11be very time-consuming when lots of different hardware configurations need to 12be tested. Moreover, it isn't practical for each person interested in this 13testing to have to acquire and maintain what can be a considerable amount of 14hardware. 15 16Also, it is desirable for developers to check for regressions in their code by 17themselves, instead of relying on the maintainers to find them and then 18reporting back. 19 20There are facilities in gitlab.freedesktop.org to automatically test Mesa that 21can be used as well for testing the DRM subsystem. This document explains how 22people interested in testing it can use this shared infrastructure to save 23quite some time and effort. 24 25 26Relevant files 27============== 28 29drivers/gpu/drm/ci/gitlab-ci.yml 30-------------------------------- 31 32This is the root configuration file for GitLab CI. Among other less interesting 33bits, it specifies the specific version of the scripts to be used. There are 34some variables that can be modified to change the behavior of the pipeline: 35 36DRM_CI_PROJECT_PATH 37 Repository that contains the Mesa software infrastructure for CI 38 39DRM_CI_COMMIT_SHA 40 A particular revision to use from that repository 41 42UPSTREAM_REPO 43 URL to git repository containing the target branch 44 45TARGET_BRANCH 46 Branch to which this branch is to be merged into 47 48IGT_VERSION 49 Revision of igt-gpu-tools being used, from 50 https://gitlab.freedesktop.org/drm/igt-gpu-tools 51 52drivers/gpu/drm/ci/testlist.txt 53------------------------------- 54 55IGT tests to be run on all drivers (unless mentioned in a driver's \*-skips.txt 56file, see below). 57 58drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-fails.txt 59---------------------------------------------------------- 60 61Lists the known failures for a given driver on a specific hardware revision. 62 63drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-flakes.txt 64----------------------------------------------------------- 65 66Lists the tests that for a given driver on a specific hardware revision are 67known to behave unreliably. These tests won't cause a job to fail regardless of 68the result. They will still be run. 69 70Each new flake entry must be associated with a link to the email reporting the 71bug to the author of the affected driver or the relevant GitLab issue. The entry 72must also include the board name or Device Tree name, the first kernel version 73affected, the IGT version used for tests, and an approximation of the failure rate. 74 75They should be provided under the following format:: 76 77 # Bug Report: $LORE_URL_OR_GITLAB_ISSUE 78 # Board Name: broken-board.dtb 79 # Linux Version: 6.6-rc1 80 # IGT Version: 1.28-gd2af13d9f 81 # Failure Rate: 100 82 flaky-test 83 84Use the appropriate link below to create a GitLab issue: 85amdgpu driver: https://gitlab.freedesktop.org/drm/amd/-/issues 86i915 driver: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues 87msm driver: https://gitlab.freedesktop.org/drm/msm/-/issues 88xe driver: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues 89 90drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt 91----------------------------------------------------------- 92 93Lists the tests that won't be run for a given driver on a specific hardware 94revision. These are usually tests that interfere with the running of the test 95list due to hanging the machine, causing OOM, taking too long, etc. 96 97 98How to enable automated testing on your tree 99============================================ 100 1011. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one 102yet, by forking https://gitlab.freedesktop.org/drm/kernel (this allows GitLab 103to internally track that these are the same git objects). 104 1052. In your kernel repo's configuration (eg. 106https://gitlab.freedesktop.org/janedoe/linux/-/settings/ci_cd), change the 107CI/CD configuration file from .gitlab-ci.yml to 108drivers/gpu/drm/ci/gitlab-ci.yml. 109 1103. Request to be added to the drm/ci-ok group so that your user has the 111necessary privileges to run the CI on https://gitlab.freedesktop.org/drm/ci-ok 112 1134. Next time you push to this repository, you will see a CI pipeline being 114created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines) 115 1165. The various jobs will be run and when the pipeline is finished, all jobs 117should be green unless a regression has been found. 118 1196. Warnings in the pipeline indicate that lockdep 120(see Documentation/locking/lockdep-design.rst) issues have been detected 121during the tests. 122 123 124How to update test expectations 125=============================== 126 127If your changes to the code fix any tests, you will have to remove one or more 128lines from one or more of the files in 129drivers/gpu/drm/ci/${DRIVER_NAME}_*_fails.txt, for each of the test platforms 130affected by the change. 131 132 133How to expand coverage 134====================== 135 136If your code changes make it possible to run more tests (by solving reliability 137issues, for example), you can remove tests from the flakes and/or skips lists, 138and then the expected results if there are any known failures. 139 140If there is a need for updating the version of IGT being used (maybe you have 141added more tests to it), update the IGT_VERSION variable at the top of the 142gitlab-ci.yml file. 143 144 145How to test your changes to the scripts 146======================================= 147 148For testing changes to the scripts in the drm-ci repo, change the 149DRM_CI_PROJECT_PATH and DRM_CI_COMMIT_SHA variables in 150drivers/gpu/drm/ci/gitlab-ci.yml to match your fork of the project (eg. 151janedoe/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/. 152 153 154How to incorporate external fixes in your testing 155================================================= 156 157Often, regressions in other trees will prevent testing changes local to the 158tree under test. These fixes will be automatically merged in during the build 159jobs from a branch in the target tree that is named as 160${TARGET_BRANCH}-external-fixes. 161 162If the pipeline is not in a merge request and a branch with the same name 163exists in the local tree, commits from that branch will be merged in as well. 164 165 166How to deal with automated testing labs that may be down 167======================================================== 168 169If a hardware farm is down and thus causing pipelines to fail that would 170otherwise pass, one can disable all jobs that would be submitted to that farm 171by editing the file at 172https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml. 173