1.. SPDX-License-Identifier: GPL-2.0 2 3====================== 4 USB4 and Thunderbolt 5====================== 6USB4 is the public specification based on Thunderbolt 3 protocol with 7some differences at the register level among other things. Connection 8manager is an entity running on the host router (host controller) 9responsible for enumerating routers and establishing tunnels. A 10connection manager can be implemented either in firmware or software. 11Typically PCs come with a firmware connection manager for Thunderbolt 3 12and early USB4 capable systems. Apple systems on the other hand use 13software connection manager and the later USB4 compliant devices follow 14the suit. 15 16The Linux Thunderbolt driver supports both and can detect at runtime which 17connection manager implementation is to be used. To be on the safe side the 18software connection manager in Linux also advertises security level 19``user`` which means PCIe tunneling is disabled by default. The 20documentation below applies to both implementations with the exception that 21the software connection manager only supports ``user`` security level and 22is expected to be accompanied with an IOMMU based DMA protection. 23 24Security levels and how to use them 25----------------------------------- 26The interface presented here is not meant for end users. Instead there 27should be a userspace tool that handles all the low-level details, keeps 28a database of the authorized devices and prompts users for new connections. 29 30More details about the sysfs interface for Thunderbolt devices can be 31found in Documentation/ABI/testing/sysfs-bus-thunderbolt. 32 33Those users who just want to connect any device without any sort of 34manual work can add following line to 35``/etc/udev/rules.d/99-local.rules``:: 36 37 ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1" 38 39This will authorize all devices automatically when they appear. However, 40keep in mind that this bypasses the security levels and makes the system 41vulnerable to DMA attacks. 42 43Starting with Intel Falcon Ridge Thunderbolt controller there are 4 44security levels available. Intel Titan Ridge added one more security level 45(usbonly). The reason for these is the fact that the connected devices can 46be DMA masters and thus read contents of the host memory without CPU and OS 47knowing about it. There are ways to prevent this by setting up an IOMMU but 48it is not always available for various reasons. 49 50Some USB4 systems have a BIOS setting to disable PCIe tunneling. This is 51treated as another security level (nopcie). 52 53The security levels are as follows: 54 55 none 56 All devices are automatically connected by the firmware. No user 57 approval is needed. In BIOS settings this is typically called 58 *Legacy mode*. 59 60 user 61 User is asked whether the device is allowed to be connected. 62 Based on the device identification information available through 63 ``/sys/bus/thunderbolt/devices``, the user then can make the decision. 64 In BIOS settings this is typically called *Unique ID*. 65 66 secure 67 User is asked whether the device is allowed to be connected. In 68 addition to UUID the device (if it supports secure connect) is sent 69 a challenge that should match the expected one based on a random key 70 written to the ``key`` sysfs attribute. In BIOS settings this is 71 typically called *One time saved key*. 72 73 dponly 74 The firmware automatically creates tunnels for Display Port and 75 USB. No PCIe tunneling is done. In BIOS settings this is 76 typically called *Display Port Only*. 77 78 usbonly 79 The firmware automatically creates tunnels for the USB controller and 80 Display Port in a dock. All PCIe links downstream of the dock are 81 removed. 82 83 nopcie 84 PCIe tunneling is disabled/forbidden from the BIOS. Available in some 85 USB4 systems. 86 87The current security level can be read from 88``/sys/bus/thunderbolt/devices/domainX/security`` where ``domainX`` is 89the Thunderbolt domain the host controller manages. There is typically 90one domain per Thunderbolt host controller. 91 92If the security level reads as ``user`` or ``secure`` the connected 93device must be authorized by the user before PCIe tunnels are created 94(e.g the PCIe device appears). 95 96Each Thunderbolt device plugged in will appear in sysfs under 97``/sys/bus/thunderbolt/devices``. The device directory carries 98information that can be used to identify the particular device, 99including its name and UUID. 100 101Authorizing devices when security level is ``user`` or ``secure`` 102----------------------------------------------------------------- 103When a device is plugged in it will appear in sysfs as follows:: 104 105 /sys/bus/thunderbolt/devices/0-1/authorized - 0 106 /sys/bus/thunderbolt/devices/0-1/device - 0x8004 107 /sys/bus/thunderbolt/devices/0-1/device_name - Thunderbolt to FireWire Adapter 108 /sys/bus/thunderbolt/devices/0-1/vendor - 0x1 109 /sys/bus/thunderbolt/devices/0-1/vendor_name - Apple, Inc. 110 /sys/bus/thunderbolt/devices/0-1/unique_id - e0376f00-0300-0100-ffff-ffffffffffff 111 112The ``authorized`` attribute reads 0 which means no PCIe tunnels are 113created yet. The user can authorize the device by simply entering:: 114 115 # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized 116 117This will create the PCIe tunnels and the device is now connected. 118 119If the device supports secure connect, and the domain security level is 120set to ``secure``, it has an additional attribute ``key`` which can hold 121a random 32-byte value used for authorization and challenging the device in 122future connects:: 123 124 /sys/bus/thunderbolt/devices/0-3/authorized - 0 125 /sys/bus/thunderbolt/devices/0-3/device - 0x305 126 /sys/bus/thunderbolt/devices/0-3/device_name - AKiTiO Thunder3 PCIe Box 127 /sys/bus/thunderbolt/devices/0-3/key - 128 /sys/bus/thunderbolt/devices/0-3/vendor - 0x41 129 /sys/bus/thunderbolt/devices/0-3/vendor_name - inXtron 130 /sys/bus/thunderbolt/devices/0-3/unique_id - dc010000-0000-8508-a22d-32ca6421cb16 131 132Notice the key is empty by default. 133 134If the user does not want to use secure connect they can just ``echo 1`` 135to the ``authorized`` attribute and the PCIe tunnels will be created in 136the same way as in the ``user`` security level. 137 138If the user wants to use secure connect, the first time the device is 139plugged a key needs to be created and sent to the device:: 140 141 # key=$(openssl rand -hex 32) 142 # echo $key > /sys/bus/thunderbolt/devices/0-3/key 143 # echo 1 > /sys/bus/thunderbolt/devices/0-3/authorized 144 145Now the device is connected (PCIe tunnels are created) and in addition 146the key is stored on the device NVM. 147 148Next time the device is plugged in the user can verify (challenge) the 149device using the same key:: 150 151 # echo $key > /sys/bus/thunderbolt/devices/0-3/key 152 # echo 2 > /sys/bus/thunderbolt/devices/0-3/authorized 153 154If the challenge the device returns back matches the one we expect based 155on the key, the device is connected and the PCIe tunnels are created. 156However, if the challenge fails no tunnels are created and error is 157returned to the user. 158 159If the user still wants to connect the device they can either approve 160the device without a key or write a new key and write 1 to the 161``authorized`` file to get the new key stored on the device NVM. 162 163De-authorizing devices 164---------------------- 165It is possible to de-authorize devices by writing ``0`` to their 166``authorized`` attribute. This requires support from the connection 167manager implementation and can be checked by reading domain 168``deauthorization`` attribute. If it reads ``1`` then the feature is 169supported. 170 171When a device is de-authorized the PCIe tunnel from the parent device 172PCIe downstream (or root) port to the device PCIe upstream port is torn 173down. This is essentially the same thing as PCIe hot-remove and the PCIe 174toplogy in question will not be accessible anymore until the device is 175authorized again. If there is storage such as NVMe or similar involved, 176there is a risk for data loss if the filesystem on that storage is not 177properly shut down. You have been warned! 178 179DMA protection utilizing IOMMU 180------------------------------ 181Recent systems from 2018 and forward with Thunderbolt ports may natively 182support IOMMU. This means that Thunderbolt security is handled by an IOMMU 183so connected devices cannot access memory regions outside of what is 184allocated for them by drivers. When Linux is running on such system it 185automatically enables IOMMU if not enabled by the user already. These 186systems can be identified by reading ``1`` from 187``/sys/bus/thunderbolt/devices/domainX/iommu_dma_protection`` attribute. 188 189The driver does not do anything special in this case but because DMA 190protection is handled by the IOMMU, security levels (if set) are 191redundant. For this reason some systems ship with security level set to 192``none``. Other systems have security level set to ``user`` in order to 193support downgrade to older OS, so users who want to automatically 194authorize devices when IOMMU DMA protection is enabled can use the 195following ``udev`` rule:: 196 197 ACTION=="add", SUBSYSTEM=="thunderbolt", ATTRS{iommu_dma_protection}=="1", ATTR{authorized}=="0", ATTR{authorized}="1" 198 199Upgrading NVM on Thunderbolt device, host or retimer 200---------------------------------------------------- 201Since most of the functionality is handled in firmware running on a 202host controller or a device, it is important that the firmware can be 203upgraded to the latest where possible bugs in it have been fixed. 204Typically OEMs provide this firmware from their support site. 205 206Currently, recommended method of updating firmware is through "fwupd" tool. 207It uses LVFS (Linux Vendor Firmware Service) portal by default to get the 208latest firmware from hardware vendors and updates connected devices if found 209compatible. For details refer to: https://github.com/fwupd/fwupd. 210 211Before you upgrade firmware on a device, host or retimer, please make 212sure it is a suitable upgrade. Failing to do that may render the device 213in a state where it cannot be used properly anymore without special 214tools! 215 216Host NVM upgrade on Apple Macs is not supported. 217 218Fwupd is installed by default. If you don't have it on your system, simply 219use your distro package manager to get it. 220 221To see possible updates through fwupd, you need to plug in a Thunderbolt 222device so that the host controller appears. It does not matter which 223device is connected (unless you are upgrading NVM on a device - then you 224need to connect that particular device). 225 226Note an OEM-specific method to power the controller up ("force power") may 227be available for your system in which case there is no need to plug in a 228Thunderbolt device. 229 230Updating firmware using fwupd is straightforward - refer to official 231readme on fwupd github. 232 233If firmware image is written successfully, the device shortly disappears. 234Once it comes back, the driver notices it and initiates a full power 235cycle. After a while device appears again and this time it should be 236fully functional. 237 238Device of interest should display new version under "Current version" 239and "Update State: Success" in fwupd's interface. 240 241Upgrading firmware manually 242--------------------------------------------------------------- 243If possible, use fwupd to updated the firmware. However, if your device OEM 244has not uploaded the firmware to LVFS, but it is available for download 245from their side, you can use method below to directly upgrade the 246firmware. 247 248Manual firmware update can be done with 'dd' tool. To update firmware 249using this method, you need to write it to the non-active parts of NVM 250of the host or device. Example on how to update Intel NUC6i7KYK 251(Skull Canyon) Thunderbolt controller NVM:: 252 253 # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem 254 255Once the operation completes we can trigger NVM authentication and 256upgrade process as follows:: 257 258 # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate 259 260If no errors are returned, device should behave as described in previous 261section. 262 263We can verify that the new NVM firmware is active by running the following 264commands:: 265 266 # cat /sys/bus/thunderbolt/devices/0-0/nvm_authenticate 267 0x0 268 # cat /sys/bus/thunderbolt/devices/0-0/nvm_version 269 18.0 270 271If ``nvm_authenticate`` contains anything other than 0x0 it is the error 272code from the last authentication cycle, which means the authentication 273of the NVM image failed. 274 275Note names of the NVMem devices ``nvm_activeN`` and ``nvm_non_activeN`` 276depend on the order they are registered in the NVMem subsystem. N in 277the name is the identifier added by the NVMem subsystem. 278 279Upgrading on-board retimer NVM when there is no cable connected 280--------------------------------------------------------------- 281If the platform supports, it may be possible to upgrade the retimer NVM 282firmware even when there is nothing connected to the USB4 283ports. When this is the case the ``usb4_portX`` devices have two special 284attributes: ``offline`` and ``rescan``. The way to upgrade the firmware 285is to first put the USB4 port into offline mode:: 286 287 # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline 288 289This step makes sure the port does not respond to any hotplug events, 290and also ensures the retimers are powered on. The next step is to scan 291for the retimers:: 292 293 # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan 294 295This enumerates and adds the on-board retimers. Now retimer NVM can be 296upgraded in the same way than with cable connected (see previous 297section). However, the retimer is not disconnected as we are offline 298mode) so after writing ``1`` to ``nvm_authenticate`` one should wait for 2995 or more seconds before running rescan again:: 300 301 # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan 302 303This point if everything went fine, the port can be put back to 304functional state again:: 305 306 # echo 0 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline 307 308Upgrading NVM when host controller is in safe mode 309-------------------------------------------------- 310If the existing NVM is not properly authenticated (or is missing) the 311host controller goes into safe mode which means that the only available 312functionality is flashing a new NVM image. When in this mode, reading 313``nvm_version`` fails with ``ENODATA`` and the device identification 314information is missing. 315 316To recover from this mode, one needs to flash a valid NVM image to the 317host controller in the same way it is done in the previous chapter. 318 319Tunneling events 320---------------- 321The driver sends ``KOBJ_CHANGE`` events to userspace when there is a 322tunneling change in the ``thunderbolt_domain``. The notification carries 323following environment variables:: 324 325 TUNNEL_EVENT=<EVENT> 326 TUNNEL_DETAILS=0:12 <-> 1:20 (USB3) 327 328Possible values for ``<EVENT>`` are: 329 330 activated 331 The tunnel was activated (created). 332 333 changed 334 There is a change in this tunnel. For example bandwidth allocation was 335 changed. 336 337 deactivated 338 The tunnel was torn down. 339 340 low bandwidth 341 The tunnel is not getting optimal bandwidth. 342 343 insufficient bandwidth 344 There is not enough bandwidth for the current tunnel requirements. 345 346The ``TUNNEL_DETAILS`` is only provided if the tunnel is known. For 347example, in case of Firmware Connection Manager this is missing or does 348not provide full tunnel information. In case of Software Connection Manager 349this includes full tunnel details. The format currently matches what the 350driver uses when logging. This may change over time. 351 352Networking over Thunderbolt cable 353--------------------------------- 354Thunderbolt technology allows software communication between two hosts 355connected by a Thunderbolt cable. 356 357It is possible to tunnel any kind of traffic over a Thunderbolt link but 358currently we only support Apple ThunderboltIP protocol. 359 360If the other host is running Windows or macOS, the only thing you need to 361do is to connect a Thunderbolt cable between the two hosts; the 362``thunderbolt-net`` driver is loaded automatically. If the other host is 363also Linux you should load ``thunderbolt-net`` manually on one host (it 364does not matter which one):: 365 366 # modprobe thunderbolt-net 367 368This triggers module load on the other host automatically. If the driver 369is built-in to the kernel image, there is no need to do anything. 370 371The driver will create one virtual ethernet interface per Thunderbolt 372port which are named like ``thunderbolt0`` and so on. From this point 373you can either use standard userspace tools like ``ifconfig`` to 374configure the interface or let your GUI handle it automatically. 375 376Forcing power 377------------- 378Many OEMs include a method that can be used to force the power of a 379Thunderbolt controller to an "On" state even if nothing is connected. 380If supported by your machine this will be exposed by the WMI bus with 381a sysfs attribute called "force_power", see 382Documentation/ABI/testing/sysfs-platform-intel-wmi-thunderbolt for details. 383 384Note: it's currently not possible to query the force power state of a platform. 385