1.. SPDX-License-Identifier: GPL-2.0 2.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. 3 4============================================= 5GPU interrupt handling: GIN and the GSP event 6============================================= 7 8This document describes how nova-core receives interrupts from the GPU on Turing 9and later chipsets. It covers the GPU Interrupt and Notification unit (GIN), 10which is the GPU's interrupt controller, and the GSP event, the interrupt that 11nova-core services in normal operation. 12 13Throughout, *CPU* means the CPU and the nova-core driver running on it. The GPU 14also has on-chip processors that run their own firmware and receive their own 15interrupts. The GSP (GPU System Processor) is one of them. 16 17Register names are the names from the GPU hardware reference headers. The 18pre-Hopper headers call the controller ``NV_CTRL`` and the Hopper-plus headers 19call it ``NV_GIN``. This document calls it GIN throughout, because the tree that 20nova-core services is the same on every supported chipset. OpenRM, NVIDIA's 21open-source GPU kernel driver, is cited wherever nova-core follows it. 22 23Terminology 24=========== 25 26The three levels of the controller, innermost first: 27 28leaf 29 One ``LEAF`` register. Each of its 32 bits is the pending bit of one 30 interrupt source. A Turing, Ampere, or Ada tree has 8 leaves. A Hopper or 31 Blackwell tree has 16. 32 33subtree 34 Two consecutive leaves, summarized by one bit of ``TOP``. A subtree is the 35 unit of enabling at ``TOP``, and under MSI-X it is the unit of delivery: 36 every interrupt from one subtree arrives on one MSI-X entry. 37 38tree 39 One ``TOP`` register and the leaves under it. Every PCIe function has its 40 own tree, and nova-core services the CPU tree of one function. 41 42The hardware headers, OpenRM, and the Linux PCI API all use the word "vector", 43each for a different number. This document gives each one its own name, and a 44bare "vector" always means a GIN vector. 45 46GIN vector 47 The GPU-internal interrupt source number. It addresses one bit of one leaf. 48 A 16-leaf tree holds vectors 0 through 511, and an 8-leaf tree holds 0 49 through 255. The CPU doorbell is vector 129 and the GSP event is vector 50 155. 51 52MSI-X entry 53 An index into the device's MSI-X table. One entry serves one subtree. 54 55PCI vector 56 One of the interrupts that ``pci_alloc_irq_vectors()`` allocates: an MSI-X 57 entry, or the single MSI message. 58 59Linux IRQ number 60 What ``request_irq()`` takes, obtained from ``pci_irq_vector()`` for a PCI 61 vector. Linux's ``struct msix_entry`` calls this number ``.vector`` as 62 well. 63 64The remaining terms, each named for the register or the specification that 65defines it: 66 67enable, disable a vector 68 Writes to ``LEAF_EN_SET`` and ``LEAF_EN_CLEAR``. 69 70enable, disable a subtree 71 Writes to ``TOP_EN_SET`` and ``TOP_EN_CLEAR``. 72 73serviced subtree 74 A subtree that nova-core enables and has a handler for. 75 76rearm 77 Restoring PCI interrupt delivery after servicing an interrupt. See 78 "Rearming PCI interrupt delivery". 79 80mask 81 Reserved for the two places where hardware and the PCI specification use 82 the word: the MSI-X per-entry Vector Control mask bit, which Linux 83 controls, and the falcon interrupt masks. It never names a GIN enable. 84 85latched, pending 86 Two names for one state, a set ``LEAF`` bit. The vector's source sets the 87 bit whether or not the vector is enabled. 88 89clear a vector 90 Write a 1 to the vector's bit in ``LEAF``. OpenRM calls the same operation 91 ``intrClearLeafVector_HAL``. 92 93pending bits 94 The plain 32-bit value read from a ``LEAF`` register. 95 96notification 97 An interrupt whose only content is that something happened, such as a 98 posted message. Servicing a notification means reading what it announces. 99 The unit that raised it needs no attention. The GSP event is one. 100 101unit 102 Any block that raises an interrupt. "Engine" is reserved for the blocks 103 that do user work: GR, CE, NVDEC, and the like. 104 105falcon 106 One of the GPU's microcontrollers (see 107 Documentation/gpu/nova/core/falcon.rst). The GSP runs on the RISC-V core 108 inside its falcon. A falcon latches each of its interrupt causes and routes 109 it either to the host, meaning the CPU, or to its own core. 110 111The GIN controller 112================== 113 114A GPU has many interrupt sources: the GSP, the copy engines, the graphics 115engine, video decode and encode, the MMU fault path, timers, and others. GIN 116records which of them are pending and raises the PCI interrupt to the CPU. 117 118Trees 119----- 120 121GIN keeps one tree for each destination it can deliver an interrupt to. The CPU 122has one tree per PCIe function, so the physical function and each virtual 123function have their own. The GSP has a tree, and so do the other on-chip 124processors that receive interrupts. Every tree has the same two-level layout, 125and a function reaches its own tree through the per-function register aperture. 126 127nova-core services the CPU tree of one function. A virtual function's tree 128belongs to that function's driver, and a processor's tree belongs to the 129firmware running on that processor. 130 131The two-level tree 132------------------ 133 134A tree is a set of ``LEAF`` registers and one ``TOP`` register. 135 136* ``LEAF(i)`` is a 32-bit register that holds the pending bits of vectors 137 ``32i`` through ``32i + 31``. A set bit is a pending vector. 138* ``TOP`` is a 32-bit read-only register. Bit ``N`` summarizes subtree ``N``, 139 which is ``LEAF(2N)`` and ``LEAF(2N + 1)``. The bit is set when an enabled 140 vector is pending in either leaf. 141 142A tree with L leaves has L / 2 subtrees and uses TOP bits 0 through L / 2 - 1. 143The other TOP bits read 0. The leaves and subtrees that a chipset has are its 144implemented leaves and subtrees, and "Per-architecture differences" gives the 145counts. For an 8-leaf tree:: 146 147 TOP bit 0 -> subtree 0 -> LEAF(0), LEAF(1) vectors 0..63 148 TOP bit 1 -> subtree 1 -> LEAF(2), LEAF(3) vectors 64..127 149 TOP bit 2 -> subtree 2 -> LEAF(4), LEAF(5) vectors 128..191 150 TOP bit 3 -> subtree 3 -> LEAF(6), LEAF(7) vectors 192..255 151 152 LEAF(4), one bit per vector, holds vectors 128..159: 153 154 bit 1 = vector 129 (CPU doorbell) 155 bit 27 = vector 155 (GSP event) 156 157A vector's number fixes its place in the tree:: 158 159 leaf = vector / 32 160 bit = vector % 32 161 subtree = leaf / 2 162 163Registers 164--------- 165 166nova-core defines the tree's registers in the ``irq`` module's ``regs.rs``. The 167leaf registers are arrays indexed by leaf number. 168 169* ``LEAF(i)`` reads as the pending bits of leaf ``i``. Writing a 1 to a bit 170 clears that vector, and a 0 leaves the bit as it was. 171* ``LEAF_EN_SET(i)`` and ``LEAF_EN_CLEAR(i)`` enable and disable the vectors 172 of leaf ``i``, one bit per vector. 173* ``TOP_EN_SET`` and ``TOP_EN_CLEAR`` enable and disable subtrees, one bit per 174 subtree. 175* ``LEAF_TRIGGER`` takes a vector number and latches that vector, exactly as 176 the vector's own source would. It is write-only. The self-test uses it. 177 178nova-core does not read ``TOP``. "Servicing the tree" says why. 179 180Every set and clear register acts per bit: a 1 performs the action for that 181bit, and a 0 leaves the bit alone. No register needs a read-modify-write. 182 183GIN delivers a vector to the CPU only when its leaf enable bit and its 184subtree's TOP enable bit are both set. The enables do not affect the latch. The 185source of a disabled vector still sets its ``LEAF`` bit. ``TOP`` does not show 186that bit, so reading the leaf is the only way to see it. 187 188How a unit interrupt reaches the CPU 189------------------------------------ 190 191A unit does not write a ``LEAF`` register. Each unit has an interrupt control 192register that GSP firmware programs. The control register holds the unit's 193vector, the GFID that identifies the PCIe function whose tree receives the 194interrupt, and one enable bit per destination: the CPU, the GSP, and the other 195on-chip processors. When the unit has an event:: 196 197 1. The unit sends GIN an interrupt message carrying the vector, the GFID, 198 and the destination enables from its control register. 199 2. In the tree of each destination that the message selects, GIN sets bit 200 (vector % 32) of LEAF(vector / 32). 201 3. If the vector and its subtree are enabled in the CPU tree, GIN raises 202 the PCI interrupt. 203 204Because firmware assigns the vectors, nova-core does not hardcode which vector 205belongs to which unit, with two exceptions. The hardware headers of every 206supported chipset define the GSP event as vector 155, and GSP firmware's own 207interrupt table uses that definition. The CPU doorbell is vector 129. Pre-Hopper 208hardware fixes that number, and GSP firmware keeps it on Hopper and later. 209nova-core names both by number. A driver can fetch the full unit-to-vector 210table from the GSP by RPC, and nouveau does. nova-core does not, because a 211fixed vector needs no lookup. 212 213Edge-triggered delivery 214----------------------- 215 216A ``LEAF`` bit is a latch. Its source sets it on a rising edge, and it stays set 217until the CPU clears it. A source that stays high does not set the bit again. 218 219GIN raises the PCI interrupt for a subtree when the subtree's enabled pending 220state goes from low to high:: 221 222 Per vector, in leaf i at bit b: 223 LEAF(i)[b] AND LEAF_EN(i)[b] 224 225 Per subtree N, across leaves 2N and 2N + 1: 226 OR of every enabled pending bit -> TOP[N] 227 228 Delivery for subtree N: 229 TOP[N] AND TOP_EN[N] -> rising edge -> PCI interrupt 230 231``TOP_EN`` applies after the summary, so disabling a subtree stops delivery 232without changing what ``TOP`` reports. 233 234Three consequences: 235 236* Code that must find every pending vector reads the leaves. A vector that 237 latched while disabled is not in ``TOP``. 238* Writing ``TOP_EN_SET`` for a subtree with an enabled pending bit produces a 239 new edge. GIN delivers an interrupt for a pending bit left uncleared as soon 240 as its subtree is enabled again. 241* A source that holds its signal high produces no new edge after the CPU 242 clears the leaf bit. Such a source has to re-emit its interrupt. The falcons 243 do that through ``INTR_RETRIGGER`` (see "Retriggering a falcon"). 244 245Delivery over PCI 246================= 247 248GIN delivers the tree's interrupts to the CPU as MSI or MSI-X, whichever Linux 249grants. nova-core requests MSI-X first and falls back to MSI, and never uses 250INTx. 251 252MSI has a single message, and every subtree raises that one message, so one 253Linux IRQ serves the whole tree. 254 255MSI-X gives each subtree its own table entry, at the index equal to the subtree 256number. Linux masks every entry until a driver requests its Linux IRQ number, 257and a masked entry sends no message: the GPU records the interrupt in the MSI-X 258pending bit array, where it stays until Linux unmasks the entry. An entry that 259the driver never requests is never unmasked. A driver that enables a subtree 260without requesting that subtree's entry loses every interrupt from that 261subtree, with nothing reported: the leaf and TOP registers show the vector 262pending and enabled while no handler runs. 263 264The serviced-subtree invariant 265------------------------------ 266 267Every subtree enabled at ``TOP`` has an allocated PCI vector with a registered 268handler. 269 270MSI satisfies this with its single message. MSI-X needs one allocated entry per 271serviced subtree, and a PCI allocation cannot be sparse, so nova-core requests 272entries 0 through the highest serviced subtree:: 273 274 MSI-X, with subtree 2 serviced: 275 276 subtree 0 -> entry 0 allocated, no handler, stays masked 277 subtree 1 -> entry 1 allocated, no handler, stays masked 278 subtree 2 -> entry 2 handler here, and its rearm covers subtree 2 279 280 MSI, with any serviced set: 281 282 every serviced subtree -> the one allocated PCI vector, whose 283 handler's rearm covers the whole serviced set 284 285An allocated entry whose subtree nova-core does not service costs nothing. The 286entry stays masked, and a disabled subtree raises no interrupt. 287 288nova-core services one subtree. The GSP event, vector 155, is in leaf 4, which 289is in subtree 2. OpenRM's headers place its UVM_SHARED interrupt category in 290subtree 2 on every chipset that nova-core supports. The self-test doorbell, 291vector 129, is in the same leaf, and the test allocates its own vectors for it. 292 293Rearming PCI interrupt delivery 294------------------------------- 295 296A message-signaled interrupt is delivered once per edge, and the PCI side 297delivers no further interrupt until the CPU rearms it. The rearm operation 298depends on the GPU family and on the interrupt type that Linux granted: 299 300================== ===== =========================================== 301Architecture Type Rearm operation 302================== ===== =========================================== 303Turing through Ada MSI write the configuration-mirror EOI register 304Hopper and later MSI clear then set the serviced TOP enables 305Any MSI-X clear then set the handler's own TOP enable 306================== ===== =========================================== 307 308The end-of-interrupt register is ``NV_XVE_CYA_2`` in the BAR0 mirror of PCI 309configuration space, and the value written does not matter. The ``TOP_EN`` 310cycle produces a new delivery edge. The MSI forms cover every serviced subtree, 311because one message serves all of them. The MSI-X form covers one subtree, 312because each serviced subtree has its own entry and its own handler. 313 314A handler rearms once per delivered interrupt, on every path, including the 315path where it finds its vector not pending. A handler that skips the rearm 316receives no further interrupts. 317 318OpenRM makes the same split. It writes the configuration-space EOI for MSI on 319pre-Hopper chipsets, and cycles the TOP enables of the subtrees that it 320services for Hopper-plus MSI and for MSI-X. 321 322Servicing the tree 323================== 324 325Servicing a leaf has a required order: read its pending bits, then clear them. 326Clearing a leaf before reading it discards every vector latched in it, and no 327register reports the loss. In nova-core, reading a leaf produces the handle that 328clears it, so that the wrong order does not compile. The handle clears exactly 329the bits that it read, so a vector that latched after the read stays pending. 330 331A handler clears its bit before it services the vector. Clearing afterwards 332would discard an interrupt that the source raised while the handler ran. 333 334nova-core services the tree in two ways. 335 336The notification path services one vector. It reads the vector's leaf, clears 337only the vector's bit, and rearms. The subtree stays enabled, and a vector 338pending beside it in the same leaf keeps its bit set for the code that services 339that vector. The GSP event handler takes this path, and so does the self-test 340handler. 341 342The startup drain walks the whole tree, because it must clear whatever is 343pending across every subtree rather than one known vector. It disables the 344serviced subtrees at ``TOP``, reads and clears every implemented leaf, and 345leaves the subtrees disabled for its caller to enable once the caller is ready 346for deliveries. The drain reads every leaf rather than descending from ``TOP``, 347because sources latch vectors during boot while those vectors are disabled, and 348``TOP`` does not show them. OpenRM's stall-interrupt path reads every leaf for 349the same reason. 350 351The two paths as register operations:: 352 353 Startup drain, run once during probe: 354 write TOP_EN_CLEAR = serviced stop new deliveries 355 for each implemented leaf i: 356 pending = read LEAF(i) 357 write LEAF(i) = pending clear what was read 358 (returns with TOP_EN still clear) 359 360 Notification, the subtree stays enabled: 361 pending = read LEAF(leaf) is the handler's bit set? 362 write LEAF(leaf) = bit clear that one bit 363 rearm PCI interrupt delivery 364 365The drain clears every pending bit, including bits that nova-core never 366services. An uncleared bit holds its subtree in the pending state, and enabling 367that subtree again would deliver an interrupt for a vector that no handler 368services. 369 370The drain's ``TOP_EN_CLEAR`` is not a rearm, and pre-Hopper MSI rearms through 371the configuration mirror, which the drain never writes. An interrupt delivered 372before probe had no handler to rearm it, so the tree reset rearms explicitly 373after the drain. A TOP-enable rearm leaves the serviced subtrees enabled, so 374the reset then disables them again. 375 376nova-core does not serialize access to the tree. The GSP event handler 377touches only its own leaf, and the drain runs during probe, before that handler 378is registered. 379 380Per-architecture differences 381============================ 382 383The tree is the same on every supported GPU except for its size, which changes 384at Hopper: 385 386=================== ====== ======== ==================== 387GPUs Leaves Subtrees Implemented subtrees 388=================== ====== ======== ==================== 389Turing, Ampere, Ada 8 4 ``0x0f`` 390Hopper, Blackwell 16 8 ``0xff`` 391=================== ====== ======== ==================== 392 393The interrupt HAL provides the leaf count, and the subtree count and the 394implemented-subtree set derive from it. A subtree that the chipset does not 395implement has no TOP bit, so building a tree that services one fails with 396``EINVAL``. Vectors 129 and 155 are in the 8-leaf tree, so every supported 397chipset has them. 398 399OpenRM's headers assign every interrupt category of a 16-leaf tree to leaves 0 400through 11. The drain reads all 16, because a vector can be latched in any 401implemented leaf. 402 403The HAL performs the rearm as well (see "Rearming PCI interrupt delivery"). Two 404falcon properties also differ by family, and the falcon HAL carries them: 405Turing falcons have no ``INTR_RETRIGGER``, and the RISC-V routing registers 406moved at GA102 (see "Retriggering a falcon"). 407 408The GSP event 409============= 410 411When the GSP has output for the CPU, it writes messages into the GSP-to-CPU 412queue in shared memory and raises SWGEN0, one of the software-generated 413interrupt causes of the GSP falcon. SWGEN0 is routed to the host, at vector 414155, leaf 4 bit 27, in subtree 2. 415 416The queue carries notifications (log records, error records, lifecycle events) 417and command replies. A thread waiting for a reply reads the queue itself, so 418the interrupt is only the trigger to drain the queue (see "Draining the 419GSP-to-CPU queue"). 420 421The falcon latches every cause that it raises, SWGEN0 among them, in its 422``IRQSTAT`` register. The handler services the host-routed causes and clears 423their latches, and then it writes ``INTR_RETRIGGER`` so that the falcon re-emits 424any cause that latched in the meantime. "Retriggering a falcon" has the details. 425 426Draining the queue takes the command-queue mutex and walks shared memory, so it 427cannot run in hard interrupt context. nova-core registers a threaded handler, 428under the name ``nova-core`` in ``/proc/interrupts``. The top half runs in hard 429interrupt context and reads and writes only registers, and it wakes the IRQ 430thread to drain the queue:: 431 432 GSP writes messages into the GSP-to-CPU queue 433 GSP raises SWGEN0 434 GIN sets bit 27 of LEAF(4), and subtree 2 becomes pending 435 PCI interrupt -> Linux IRQ -> top half, in hard interrupt context: 436 read LEAF(4), and if bit 27 is clear, rearm and return 437 clear bit 27 (the subtree stays enabled) 438 read the falcon causes routed to the host, clearing SWGEN0 if set 439 for every other host cause: log it, clear its latch, and read the 440 host causes back 441 if the clear ended all of them: retrigger the falcon 442 otherwise: disable vector 155 at its leaf and skip the retrigger 443 rearm PCI interrupt delivery 444 wake the IRQ thread if SWGEN0 was set 445 IRQ thread, which may sleep: 446 take the command-queue mutex and drain the GSP-to-CPU queue 447 448A halt and a posted message can be pending together, so the top half services 449every cause that the status reports. 450 451A drain fails when a message's framing or checksum is bad (see "Draining the 452GSP-to-CPU queue"). The message stays at the queue head, so every later drain 453would fail the same way, and the IRQ thread disables vector 155 and logs the 454failure, which leaves the queue unserviced until the device is reset. 455 456The handler, the self-test, and the rest of the driver read BAR0 through one 457shared mapping. nova-core unregisters an interrupt handler when the device 458unbinds, so a handler runs only while the mapping exists. 459 460Retriggering a falcon 461--------------------- 462 463A falcon signals the tree when its set of host-routed causes goes from empty to 464non-empty. A cause left latched keeps the set non-empty, so no later cause 465signals the tree, and the vector is lost. For a cause that stays latched, the 466handler can clear the tree leaf first or the falcon latch first, and the loss 467is the same. 468 469``IRQSTAT`` latches every cause in the falcon, including the causes routed to 470the falcon's own RISC-V core and owned by the firmware running on it. A host 471handler owns only the causes that ``PRISCV_RISCV_IRQMASK`` and 472``PRISCV_RISCV_IRQDEST`` both select, so it intersects ``IRQSTAT`` with both 473before it reads or clears a cause. OpenRM computes the same intersection in 474``kflcnRiscvReadIntrStatus``. GA100 keeps the Turing offsets of the two routing 475registers and GA102 moves them, so the offsets change at GA102 rather than at 476the Ampere boundary. The handler masks no cause: ``PRISCV_RISCV_IRQMASK`` is 477read-only to the host, and ``FALCON_IRQMASK`` has no effect on host routing on 478a RISC-V falcon. 479 480``INTR_RETRIGGER`` makes the falcon re-emit its host-routed causes into the 481tree, which supplies the transition that clearing the leaf lost. The handler 482writes ``INTR_RETRIGGER`` only on a path where it ended every cause that it 483read, because a re-emitted cause that stays set arrives again at once and 484on every pass after that. 485 486``IRQSCLR`` ends a latch and does not end the source behind it, so a cause 487driven from outside the falcon stays set after the write. On Blackwell the 488fault-containment and ECC causes are driven that way: they appear in 489``IRQSTAT`` but come from ``PRISCV_RISCV_FAULT_CONTAINMENT_SRCSTAT`` and 490``PGSP_ECC_INTR_STATUS``, and only a device reset ends them. So the handler 491clears the latch of every host cause other than SWGEN0, reads the host causes 492back, and retriggers only when the read-back is empty. When a cause is still 493set, the handler disables vector 155 instead and reports that the device needs 494a reset. Disabling loses no notification: the cause that is still set holds the 495host-routed set non-empty, so the falcon would signal nothing further either 496way. OpenRM makes the same choice, and ``kgspService_TU102`` skips 497``kflcnIntrRetrigger`` once it has recorded a fatal error. 498 499The handler cannot distinguish a fault cause that arrives after the clear from 500one that the clear failed to end, so it disables the vector in that case too. 501Both mean that the GSP has faulted. 502 503Turing falcons have no ``INTR_RETRIGGER``, so a Turing handler cannot re-create 504a transition that it has lost. It must leave no host cause latched: it reads the 505host-routed status once and takes every cause that the status reports, rather 506than stopping at the first one that it recognizes. One window stays open. A 507cause that arrives after the handler has read the status is not in the value 508that the handler clears, so it stays latched after the leaf has been cleared, 509and no later cause from that falcon signals the tree. OpenRM has the same window 510on Turing, where ``kflcnIntrRetrigger`` does nothing. 511 512Enabling the GSP event 513---------------------- 514 515SWGEN0 is a latch, and the GSP drives no new edge into the tree while it stays 516set. nova-core's GSP boot code consumes the GSP's notifications by polling the 517queue, which leaves the latch set and leaves pending bits in the tree. The 518handoff from polling to interrupts has a required order:: 519 520 disable every implemented vector drop enables left by boot, or by a 521 driver that ran before this one 522 drain the tree clear stale pending bits 523 rearm PCI interrupt delivery required under pre-Hopper MSI, which 524 no later step rearms 525 disable subtree 2 at TOP a TOP-enable rearm enabled it again 526 clear the SWGEN0 latch so the next message makes an edge 527 register the threaded handler no delivery can reach it yet 528 enable vector 155 at LEAF(4) the subtree is still disabled 529 enable subtree 2 at TOP deliveries become possible here 530 drain the GSP-to-CPU queue messages posted before the clear 531 532The first four steps are the tree reset. nova-core runs them, and clears the 533latch, before it registers the handler. Registering unmasks the PCI interrupt, 534and GIN would then deliver a vector that boot left enabled to a handler that 535services one vector and has no way to service any other. OpenRM clears every 536leaf enable at the same point for the same reason. 537 538nova-core clears the latch after the drain. If nova-core cleared the latch 539first, a message posted before the drain could set it again, along with bit 27 540of LEAF(4). The drain would then clear the leaf bit while the latch stays set, 541and no later message would signal the tree. Clearing after the drain can 542instead leave the leaf bit pending with the latch already clear. Enabling the 543subtree then delivers one interrupt whose ``IRQSTAT`` reads zero. The top half 544clears the leaf bit, rearms, and does not wake the IRQ thread, and the queue 545drain that follows reads the message. 546 547Clearing the latch makes the first interrupt possible. A message that the GSP 548posted before that clear produces no interrupt, so the sequence ends by 549draining the queue. 550 551The subtree is enabled at ``TOP`` only once the handler is registered. A 552TOP-enable rearm sets the enables that it cycles, so the reset disables the 553serviced subtrees again after its rearm, and the tree reaches the registration 554with subtree 2 disabled under every rearm method. On teardown nova-core disables 555the vector at its leaf, so that no vector in the subtree can be delivered, then 556calls ``free_irq()``, and disables the subtree last. Disabling the subtree 557earlier would let a handler still in flight enable it again through the 558``TOP_EN`` cycle of its rearm, which would leave the subtree enabled with no 559handler registered. The driver's ``Gpu`` object, which owns the tree, holds the 560subtree enable, and a handler's registration holds only the enable of its own 561vector, so that tearing down one handler does not disable a subtree that another 562handler shares. nova-core tears down the registration before it frees the queue 563that the handler drains and the falcon that the handler reads, and before it 564unloads the GSP. 565 566Draining the GSP-to-CPU queue 567----------------------------- 568 569The queue carries command replies and unsolicited events, and a message's 570function code says which it is. 571 572* A function code that matches the awaited reply: the message is decoded and 573 returned to the caller that sent the command. 574* Anything else is an event. An OS error record and a robust-channel record 575 are logged at error level, and an unrecognized function code at warning 576 level. The other known events (GSP logs, libos prints, assertion records, 577 lifecycle notices) need no action and get no line of their own, because the 578 receive trace at debug level already records every message's arrival with 579 its sequence number, function code, and length. 580 581The sequence number takes no part in the match, because the GSP does not echo 582the command's sequence number on every reply. On r570 the reply to 583``UnloadingGuestDriver`` carries sequence 0. 584 585The read pointer advances past every message that passes framing and checksum 586validation, whether it matched or was an event. A matched message that is too 587short to decode is the exception, and it stays at the queue head. 588 589A message's length is inside the region that the checksum covers, so once the 590framing or the checksum fails there is no trustworthy length with which to skip 591the message. Such a message also stays at the queue head, and every later 592receive fails on it with ``EIO`` until the device is reset. 593 594The polling path and the IRQ thread both read the queue under the command-queue 595mutex. Replies and events share one queue and one read pointer, so one lock is 596held across the whole drain. A thread waiting for a reply logs each event that 597arrives before the reply and keeps waiting. One deadline of 5 seconds applies to 598the whole wait, rather than a fresh timeout after each message, and the thread 599holds the mutex for the whole wait, so that no other caller consumes the message 600that it waits for. 601 602With one lock, a drain waits for an in-flight command's receive to finish or 603time out. For log and error records that delay does not matter. 604