Changes in Kconfig

a23e1966932464e1c5226cb9ac4ce1d5fc10ba22 - Merge branch 'next' into for-linus

Mon, 15 Jul 2024 23:03:44 +0200

Merge branch 'next' into for-linusPrepare input updates for 6.11 merge window. List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

6f47c7ae8c7afaf9ad291d39f0d3974f191a7946 - Merge tag 'v6.9' into next

Tue, 28 May 2024 06:37:18 +0200

Merge tag 'v6.9' into nextSync up with the mainline to bring in the new cleanup API. List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

60a2f25de7b8b785baee2932db932ae9a5b8c86d - Merge drm/drm-next into drm-intel-gt-next

Thu, 16 May 2024 09:33:01 +0200

Merge drm/drm-next into drm-intel-gt-nextSome display refactoring patches are needed in order to allow conflict-less merging.Signed-off-by: Tvrtko Ursulin List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

594ce0b8a998aa4d05827cd7c0d0dcec9a1e3ae2 - Merge topic branches 'clkdev' and 'fixes' into for-linus

Mon, 10 Jun 2024 13:03:21 +0200

Merge topic branches 'clkdev' and 'fixes' into for-linus List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

b228ab57e51b62663a80ca820c87ba2650583f08 - Merge branch 'master' into mm-stable

Mon, 18 Mar 2024 17:55:12 +0100

Merge branch 'master' into mm-stable List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

79790b6818e96c58fe2bffee1b418c16e64e7b80 - Merge drm/drm-next into drm-xe-next

Fri, 12 Apr 2024 15:14:25 +0200

Merge drm/drm-next into drm-xe-nextBackmerging drm-next in order to get up-to-date and in particularto access commit 9ca5facd0400f610f3f7f71aeb7fc0b949a48c67.Signed-off-by: Thomas Hellström List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

3e5a516f3bf1a33f2bf219f570e9b5c031616f6a - Merge tag 'phy_dp_modes_6.10' into msm-next-lumag

Mon, 08 Apr 2024 17:35:51 +0200

Merge tag 'phy_dp_modes_6.10' into msm-next-lumagMerge DisplayPort subnode API in order to allow DisplayPort driver toconfigure the PHYs either to the DP or eDP mode, depending on hardwareconfiguration.Signed-off-by: Dmitry Baryshkov List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

5add703f6acad1c63f8a532b6de56e50d548e904 - Merge drm/drm-next into drm-intel-next

Tue, 02 Apr 2024 14:17:13 +0200

Merge drm/drm-next into drm-intel-nextCatching up on 6.9-rc2Signed-off-by: Rodrigo Vivi List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

0d21364c6e8dc1f62c34bbc49d49935c8b01844c - Merge drm/drm-next into drm-misc-next

Tue, 02 Apr 2024 09:51:30 +0200

Merge drm/drm-next into drm-misc-nextBackmerging to get v6.9-rc2 changes into drm-misc-next.Signed-off-by: Thomas Zimmermann List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

b7e1e969c887c897947fdc3754fe9b0c24acb155 - Merge branch 'topic/sound-devel-6.10' into for-next

Tue, 26 Mar 2024 12:19:11 +0100

Merge branch 'topic/sound-devel-6.10' into for-next List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

537c2e91d3549e5d6020bb0576cf9b54a845255f - Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net

Fri, 22 Mar 2024 00:14:13 +0100

Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/netCross-merge networking fixes after downstream PR.Signed-off-by: Jakub Kicinski List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

f4566a1e73957800df75a3dd2dccee8a4697f327 - Merge tag 'v6.9-rc1' into sched/core, to pick up fixes and to refresh the branch

Mon, 25 Mar 2024 11:32:29 +0100

Merge tag 'v6.9-rc1' into sched/core, to pick up fixes and to refresh the branchSigned-off-by: Ingo Molnar List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

100c85421b52e41269ada88f7d71a6b8a06c7a11 - Merge tag 'asoc-fix-v6.9-rc2' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-linus

Fri, 05 Apr 2024 08:48:12 +0200

Merge tag 'asoc-fix-v6.9-rc2' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-linusASoC: Fixes for v6.9A relatively large set of fixes here, the biggest piece of it is aseries correcting some problems with the delay reporting for Intel SOFcards but there's a bunch of other things. Everything here is driverspecific except for a fix in the core for an issue with sign extensionhandling volume controls. List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

36a1818f5a1e50b805317ba13f827067d50f6970 - Merge drm/drm-fixes into drm-misc-fixes

Mon, 25 Mar 2024 21:11:58 +0100

Merge drm/drm-fixes into drm-misc-fixesBackmerging to get drm-misc-fixes to the state of v6.9-rc1.Signed-off-by: Thomas Zimmermann List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

4138f02288333cb596885e9af03dd3ea2de845cb - Merge tag 'vfio-v6.9-rc1' of https://github.com/awilliam/linux-vfio

Fri, 15 Mar 2024 21:21:13 +0100

Merge tag 'vfio-v6.9-rc1' of https://github.com/awilliam/linux-vfioPull VFIO updates from Alex Williamson: - Add warning in unlikely case that device is not captured with driver_override (Kunwu Chan) - Error handling improvements in mlx5-vfio-pci to detect firmware tracking object error states, logging of firmware error syndrom, and releasing of firmware resources in aborted migration sequence (Yishai Hadas) - Correct an un-alphabetized VFIO MAINTAINERS entry (Alex Williamson) - Make the mdev_bus_type const and also make the class struct const for a couple of the vfio-mdev sample drivers (Ricardo B. Marliere) - Addition of a new vfio-pci variant driver for the GPU of NVIDIA's Grace-Hopper superchip. During initialization of the chip-to-chip interconnect in this hardware module, the PCI BARs of the device become unused in favor of a faster, coherent mechanism for exposing device memory. This driver primarily changes the VFIO representation of the device to masquerade this coherent aperture to replace the physical PCI BARs for userspace drivers. This also incorporates use of a new vma flag allowing KVM to use write combining attributes for uncached device memory (Ankit Agrawal) - Reset fixes and cleanups for the pds-vfio-pci driver. Save and restore files were previously leaked if the device didn't pass through an error state, this is resolved and later re-fixed to prevent access to the now freed files. Reset handling is also refactored to remove the complicated deferred reset mechanism (Brett Creeley) - Remove some references to pl330 in the vfio-platform amba driver (Geert Uytterhoeven) - Remove twice redundant and ugly code to unpin incidental pins of the zero-page (Alex Williamson) - Deferred reset logic is also removed from the hisi-acc-vfio-pci driver as a simplification (Shameer Kolothum) - Enforce that mlx5-vfio-pci devices must support PRE_COPY and remove resulting unnecessary code. There is no device firmware that has been available publicly without this support (Yishai Hadas) - Switch over to using the .remove_new callback for vfio-platform in support of the broader transition for a void remove function (Uwe Kleine-König) - Resolve multiple issues in interrupt code for VFIO bus drivers that allow calling eventfd_signal() on a NULL context. This also remove a potential race in INTx setup on certain hardware for vfio-pci, races with various mechanisms to mask INTx, and leaked virqfds in vfio-platform (Alex Williamson)* tag 'vfio-v6.9-rc1' of https://github.com/awilliam/linux-vfio: (29 commits) vfio/fsl-mc: Block calling interrupt handler without trigger vfio/platform: Create persistent IRQ handlers vfio/platform: Disable virqfds on cleanup vfio/pci: Create persistent INTx handler vfio: Introduce interface to flush virqfd inject workqueue vfio/pci: Lock external INTx masking ops vfio/pci: Disable auto-enable of exclusive INTx IRQ vfio/pds: Refactor/simplify reset logic vfio/pds: Make sure migration file isn't accessed after reset vfio/platform: Convert to platform remove callback returning void vfio/mlx5: Enforce PRE_COPY support vfio/mbochs: make mbochs_class constant vfio/mdpy: make mdpy_class constant hisi_acc_vfio_pci: Remove the deferred_reset logic Revert "vfio/type1: Unpin zero pages" vfio/nvgrace-gpu: Convey kvm to map device memory region as noncached vfio: amba: Rename pl330_ids[] to vfio_amba_ids[] vfio/pds: Always clear the save/restore FDs on reset vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper vfio/pci: rename and export range_intersect_range ... List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig

701ab935859fcfd4a8c8a97f3ee4fb5294a9d481 - vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

Tue, 20 Feb 2024 12:50:55 +0100

vfio/nvgrace-gpu: Add vfio pci variant module for grace hopperNVIDIA's upcoming Grace Hopper Superchip provides a PCI-like devicefor the on-chip GPU that is the logical OS representation of theinternal proprietary chip-to-chip cache coherent interconnect.The device is peculiar compared to a real PCI device in that whilstthere is a real 64b PCI BAR1 (comprising region 2 & region 3) on thedevice, it is not used to access device memory once the fasterchip-to-chip interconnect is initialized (occurs at the time of hostsystem boot). The device memory is accessed instead using the chip-to-chipinterconnect that is exposed as a contiguous physically addressableregion on the host. This device memory aperture can be obtained from hostACPI table using device_property_read_u64(), according to the FWspecification. Since the device memory is cache coherent with the CPU,it can be mmap into the user VMA with a cacheable mapping usingremap_pfn_range() and used like a regular RAM. The device memoryis not added to the host kernel, but mapped directly as this reducesmemory wastage due to struct pages.There is also a requirement of a minimum reserved 1G uncached region(termed as resmem) to support the Multi-Instance GPU (MIG) feature [1].This is to work around a HW defect. Based on [2], the requisite properties(uncached, unaligned access) can be achieved through a VM mapping (S1)of NORMAL_NC and host (S2) mapping with MemAttr[2:0]=0b101. To providea different non-cached property to the reserved 1G region, it needs tobe carved out from the device memory and mapped as a separate regionin Qemu VMA with pgprot_writecombine(). pgprot_writecombine() sets theQemu VMA page properties (pgprot) as NORMAL_NC.Provide a VFIO PCI variant driver that adapts the unique device memoryrepresentation into a more standard PCI representation facing userspace.The variant driver exposes these two regions - the non-cached reserved(resmem) and the cached rest of the device memory (termed as usemem) asseparate VFIO 64b BAR regions. This is divergent from the baremetalapproach, where the device memory is exposed as a device memory region.The decision for a different approach was taken in view of the fact thatit would necessiate additional code in Qemu to discover and insert thoseregions in the VM IPA, along with the additional VM ACPI DSDT changes tocommunicate the device memory region IPA to the VM workloads. Moreover,this behavior would have to be added to a variety of emulators (beyondtop of tree Qemu) out there desiring grace hopper support.Since the device implements 64-bit BAR0, the VFIO PCI variant drivermaps the uncached carved out region to the next available PCI BAR (i.e.comprising of region 2 and 3). The cached device memory aperture isassigned BAR region 4 and 5. Qemu will then naturally generate a PCIdevice in the VM with the uncached aperture reported as BAR2 region,the cacheable as BAR4. The variant driver provides emulation for thesefake BARs' PCI config space offset registers.The hardware ensures that the system does not crash when the memoryis accessed with the memory enable turned off. It synthesis ~0 readsand dropped writes on such access. So there is no need to support thedisablement/enablement of BAR through PCI_COMMAND config space register.The memory layout on the host looks like the following: devmem (memlength)|--------------------------------------------------||-------------cached------------------------|--NC--|| |usemem.memphys resmem.memphysPCI BARs need to be aligned to the power-of-2, but the actual memory on thedevice may not. A read or write access to the physical address from thelast device PFN up to the next power-of-2 aligned physical addressresults in reading ~0 and dropped writes. Note that the GPU devicedriver [6] is capable of knowing the exact device memory size throughseparate means. The device memory size is primarily kept in the systemACPI tables for use by the VFIO PCI variant module.Note that the usemem memory is added by the VM Nvidia device driver [5]to the VM kernel as memblocks. Hence make the usable memory size memblock(MEMBLK_SIZE) aligned. This is a hardwired ABI value between the GPU FW andVFIO driver. The VM device driver make use of the same value for itscalculation to determine USEMEM size.Currently there is no provision in KVM for a S2 mapping withMemAttr[2:0]=0b101, but there is an ongoing effort to provide the same [3].As previously mentioned, resmem is mapped pgprot_writecombine(), thatsets the Qemu VMA page properties (pgprot) as NORMAL_NC. Using theproposed changes in [3] and [4], KVM marks the region withMemAttr[2:0]=0b101 in S2.If the device memory properties are not present, the driver registers thevfio-pci-core function pointers. Since there are no ACPI memory propertiesgenerated for the VM, the variant driver inside the VM will only usethe vfio-pci-core ops and hence try to map the BARs as non cached. Thisis not a problem as the CPUs have FWB enabled which blocks the VMmapping's ability to override the cacheability set by the host mapping.This goes along with a qemu series [6] to provides the necessaryimplementation of the Grace Hopper Superchip firmware specification sothat the guest operating system can see the correct ACPI modeling forthe coherent GPU device. Verified with the CUDA workload in the VM.[1] https://www.nvidia.com/en-in/technologies/multi-instance-gpu/[2] section D8.5.5 of https://developer.arm.com/documentation/ddi0487/latest/[3] https://lore.kernel.org/all/20240211174705.31992-1-ankita@nvidia.com/[4] https://lore.kernel.org/all/20230907181459.18145-2-ankita@nvidia.com/[5] https://github.com/NVIDIA/open-gpu-kernel-modules[6] https://lore.kernel.org/all/20231203060245.31593-1-ankita@nvidia.com/Reviewed-by: Kevin Tian Reviewed-by: Yishai Hadas Reviewed-by: Zhi Wang Signed-off-by: Aniket Agashe Signed-off-by: Ankit Agrawal Link: https://lore.kernel.org/r/20240220115055.23546-4-ankita@nvidia.comSigned-off-by: Alex Williamson List of files: /linux/drivers/vfio/pci/nvgrace-gpu/Kconfig