Changes in thpmaps

a23e1966932464e1c5226cb9ac4ce1d5fc10ba22 - Merge branch 'next' into for-linus

Mon, 15 Jul 2024 23:03:44 +0200

Merge branch 'next' into for-linusPrepare input updates for 6.11 merge window. List of files: /linux/tools/mm/thpmaps

6f47c7ae8c7afaf9ad291d39f0d3974f191a7946 - Merge tag 'v6.9' into next

Tue, 28 May 2024 06:37:18 +0200

Merge tag 'v6.9' into nextSync up with the mainline to bring in the new cleanup API. List of files: /linux/tools/mm/thpmaps

60a2f25de7b8b785baee2932db932ae9a5b8c86d - Merge drm/drm-next into drm-intel-gt-next

Thu, 16 May 2024 09:33:01 +0200

Merge drm/drm-next into drm-intel-gt-nextSome display refactoring patches are needed in order to allow conflict-less merging.Signed-off-by: Tvrtko Ursulin List of files: /linux/tools/mm/thpmaps

594ce0b8a998aa4d05827cd7c0d0dcec9a1e3ae2 - Merge topic branches 'clkdev' and 'fixes' into for-linus

Mon, 10 Jun 2024 13:03:21 +0200

Merge topic branches 'clkdev' and 'fixes' into for-linus List of files: /linux/tools/mm/thpmaps

79790b6818e96c58fe2bffee1b418c16e64e7b80 - Merge drm/drm-next into drm-xe-next

Fri, 12 Apr 2024 15:14:25 +0200

Merge drm/drm-next into drm-xe-nextBackmerging drm-next in order to get up-to-date and in particularto access commit 9ca5facd0400f610f3f7f71aeb7fc0b949a48c67.Signed-off-by: Thomas Hellström List of files: /linux/tools/mm/thpmaps

3e5a516f3bf1a33f2bf219f570e9b5c031616f6a - Merge tag 'phy_dp_modes_6.10' into msm-next-lumag

Mon, 08 Apr 2024 17:35:51 +0200

Merge tag 'phy_dp_modes_6.10' into msm-next-lumagMerge DisplayPort subnode API in order to allow DisplayPort driver toconfigure the PHYs either to the DP or eDP mode, depending on hardwareconfiguration.Signed-off-by: Dmitry Baryshkov List of files: /linux/tools/mm/thpmaps

5add703f6acad1c63f8a532b6de56e50d548e904 - Merge drm/drm-next into drm-intel-next

Tue, 02 Apr 2024 14:17:13 +0200

Merge drm/drm-next into drm-intel-nextCatching up on 6.9-rc2Signed-off-by: Rodrigo Vivi List of files: /linux/tools/mm/thpmaps

0d21364c6e8dc1f62c34bbc49d49935c8b01844c - Merge drm/drm-next into drm-misc-next

Tue, 02 Apr 2024 09:51:30 +0200

Merge drm/drm-next into drm-misc-nextBackmerging to get v6.9-rc2 changes into drm-misc-next.Signed-off-by: Thomas Zimmermann List of files: /linux/tools/mm/thpmaps

b7e1e969c887c897947fdc3754fe9b0c24acb155 - Merge branch 'topic/sound-devel-6.10' into for-next

Tue, 26 Mar 2024 12:19:11 +0100

Merge branch 'topic/sound-devel-6.10' into for-next List of files: /linux/tools/mm/thpmaps

537c2e91d3549e5d6020bb0576cf9b54a845255f - Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net

Fri, 22 Mar 2024 00:14:13 +0100

Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/netCross-merge networking fixes after downstream PR.Signed-off-by: Jakub Kicinski List of files: /linux/tools/mm/thpmaps

f4566a1e73957800df75a3dd2dccee8a4697f327 - Merge tag 'v6.9-rc1' into sched/core, to pick up fixes and to refresh the branch

Mon, 25 Mar 2024 11:32:29 +0100

Merge tag 'v6.9-rc1' into sched/core, to pick up fixes and to refresh the branchSigned-off-by: Ingo Molnar List of files: /linux/tools/mm/thpmaps

100c85421b52e41269ada88f7d71a6b8a06c7a11 - Merge tag 'asoc-fix-v6.9-rc2' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-linus

Fri, 05 Apr 2024 08:48:12 +0200

Merge tag 'asoc-fix-v6.9-rc2' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-linusASoC: Fixes for v6.9A relatively large set of fixes here, the biggest piece of it is aseries correcting some problems with the delay reporting for Intel SOFcards but there's a bunch of other things. Everything here is driverspecific except for a fix in the core for an issue with sign extensionhandling volume controls. List of files: /linux/tools/mm/thpmaps

36a1818f5a1e50b805317ba13f827067d50f6970 - Merge drm/drm-fixes into drm-misc-fixes

Mon, 25 Mar 2024 21:11:58 +0100

Merge drm/drm-fixes into drm-misc-fixesBackmerging to get drm-misc-fixes to the state of v6.9-rc1.Signed-off-by: Thomas Zimmermann List of files: /linux/tools/mm/thpmaps

902861e34c401696ed9ad17a54c8790e7e8e3069 - Merge tag 'mm-stable-2024-03-13-20-04' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Fri, 15 Mar 2024 01:43:30 +0100

Merge tag 'mm-stable-2024-03-13-20-04' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mmPull MM updates from Andrew Morton: - Sumanth Korikkar has taught s390 to allocate hotplug-time page frames from hotplugged memory rather than only from main memory. Series "implement "memmap on memory" feature on s390". - More folio conversions from Matthew Wilcox in the series "Convert memcontrol charge moving to use folios" "mm: convert mm counter to take a folio" - Chengming Zhou has optimized zswap's rbtree locking, providing significant reductions in system time and modest but measurable reductions in overall runtimes. The series is "mm/zswap: optimize the scalability of zswap rb-tree". - Chengming Zhou has also provided the series "mm/zswap: optimize zswap lru list" which provides measurable runtime benefits in some swap-intensive situations. - And Chengming Zhou further optimizes zswap in the series "mm/zswap: optimize for dynamic zswap_pools". Measured improvements are modest. - zswap cleanups and simplifications from Yosry Ahmed in the series "mm: zswap: simplify zswap_swapoff()". - In the series "Add DAX ABI for memmap_on_memory", Vishal Verma has contributed several DAX cleanups as well as adding a sysfs tunable to control the memmap_on_memory setting when the dax device is hotplugged as system memory. - Johannes Weiner has added the large series "mm: zswap: cleanups", which does that. - More DAMON work from SeongJae Park in the series "mm/damon: make DAMON debugfs interface deprecation unignorable" "selftests/damon: add more tests for core functionalities and corner cases" "Docs/mm/damon: misc readability improvements" "mm/damon: let DAMOS feeds and tame/auto-tune itself" - In the series "mm/mempolicy: weighted interleave mempolicy and sysfs extension" Rakie Kim has developed a new mempolicy interleaving policy wherein we allocate memory across nodes in a weighted fashion rather than uniformly. This is beneficial in heterogeneous memory environments appearing with CXL. - Christophe Leroy has contributed some cleanup and consolidation work against the ARM pagetable dumping code in the series "mm: ptdump: Refactor CONFIG_DEBUG_WX and check_wx_pages debugfs attribute". - Luis Chamberlain has added some additional xarray selftesting in the series "test_xarray: advanced API multi-index tests". - Muhammad Usama Anjum has reworked the selftest code to make its human-readable output conform to the TAP ("Test Anything Protocol") format. Amongst other things, this opens up the use of third-party tools to parse and process out selftesting results. - Ryan Roberts has added fork()-time PTE batching of THP ptes in the series "mm/memory: optimize fork() with PTE-mapped THP". Mainly targeted at arm64, this significantly speeds up fork() when the process has a large number of pte-mapped folios. - David Hildenbrand also gets in on the THP pte batching game in his series "mm/memory: optimize unmap/zap with PTE-mapped THP". It implements batching during munmap() and other pte teardown situations. The microbenchmark improvements are nice. - And in the series "Transparent Contiguous PTEs for User Mappings" Ryan Roberts further utilizes arm's pte's contiguous bit ("contpte mappings"). Kernel build times on arm64 improved nicely. Ryan's series "Address some contpte nits" provides some followup work. - In the series "mm/hugetlb: Restore the reservation" Breno Leitao has fixed an obscure hugetlb race which was causing unnecessary page faults. He has also added a reproducer under the selftest code. - In the series "selftests/mm: Output cleanups for the compaction test", Mark Brown did what the title claims. - Kinsey Ho has added the series "mm/mglru: code cleanup and refactoring". - Even more zswap material from Nhat Pham. The series "fix and extend zswap kselftests" does as claimed. - In the series "Introduce cpu_dcache_is_aliasing() to fix DAX regression" Mathieu Desnoyers has cleaned up and fixed rather a mess in our handling of DAX on archiecctures which have virtually aliasing data caches. The arm architecture is the main beneficiary. - Lokesh Gidra's series "per-vma locks in userfaultfd" provides dramatic improvements in worst-case mmap_lock hold times during certain userfaultfd operations. - Some page_owner enhancements and maintenance work from Oscar Salvador in his series "page_owner: print stacks and their outstanding allocations" "page_owner: Fixup and cleanup" - Uladzislau Rezki has contributed some vmalloc scalability improvements in his series "Mitigate a vmap lock contention". It realizes a 12x improvement for a certain microbenchmark. - Some kexec/crash cleanup work from Baoquan He in the series "Split crash out from kexec and clean up related config items". - Some zsmalloc maintenance work from Chengming Zhou in the series "mm/zsmalloc: fix and optimize objects/page migration" "mm/zsmalloc: some cleanup for get/set_zspage_mapping()" - Zi Yan has taught the MM to perform compaction on folios larger than order=0. This a step along the path to implementaton of the merging of large anonymous folios. The series is named "Enable >0 order folio memory compaction". - Christoph Hellwig has done quite a lot of cleanup work in the pagecache writeback code in his series "convert write_cache_pages() to an iterator". - Some modest hugetlb cleanups and speedups in Vishal Moola's series "Handle hugetlb faults under the VMA lock". - Zi Yan has changed the page splitting code so we can split huge pages into sizes other than order-0 to better utilize large folios. The series is named "Split a folio to any lower order folios". - David Hildenbrand has contributed the series "mm: remove total_mapcount()", a cleanup. - Matthew Wilcox has sought to improve the performance of bulk memory freeing in his series "Rearrange batched folio freeing". - Gang Li's series "hugetlb: parallelize hugetlb page init on boot" provides large improvements in bootup times on large machines which are configured to use large numbers of hugetlb pages. - Matthew Wilcox's series "PageFlags cleanups" does that. - Qi Zheng's series "minor fixes and supplement for ptdesc" does that also. S390 is affected. - Cleanups to our pagemap utility functions from Peter Xu in his series "mm/treewide: Replace pXd_large() with pXd_leaf()". - Nico Pache has fixed a few things with our hugepage selftests in his series "selftests/mm: Improve Hugepage Test Handling in MM Selftests". - Also, of course, many singleton patches to many things. Please see the individual changelogs for details.* tag 'mm-stable-2024-03-13-20-04' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (435 commits) mm/zswap: remove the memcpy if acomp is not sleepable crypto: introduce: acomp_is_async to expose if comp drivers might sleep memtest: use {READ,WRITE}_ONCE in memory scanning mm: prohibit the last subpage from reusing the entire large folio mm: recover pud_leaf() definitions in nopmd case selftests/mm: skip the hugetlb-madvise tests on unmet hugepage requirements selftests/mm: skip uffd hugetlb tests with insufficient hugepages selftests/mm: dont fail testsuite due to a lack of hugepages mm/huge_memory: skip invalid debugfs new_order input for folio split mm/huge_memory: check new folio order when split a folio mm, vmscan: retry kswapd's priority loop with cache_trim_mode off on failure mm: add an explicit smp_wmb() to UFFDIO_CONTINUE mm: fix list corruption in put_pages_list mm: remove folio from deferred split list before uncharging it filemap: avoid unnecessary major faults in filemap_fault() mm,page_owner: drop unnecessary check mm,page_owner: check for null stack_record before bumping its refcount mm: swap: fix race between free_swap_and_cache() and swapoff() mm/treewide: align up pXd_leaf() retval across archs mm/treewide: drop pXd_large() ... List of files: /linux/tools/mm/thpmaps

2444172cfde45a3d6e655f50c620727c76bab4a2 - tools/mm: add thpmaps script to dump THP usage info

Tue, 16 Jan 2024 15:12:35 +0100

tools/mm: add thpmaps script to dump THP usage infoWith the proliferation of large folios for file-backed memory, and morerecently the introduction of multi-size THP for anonymous memory, it isbecoming useful to be able to see exactly how large folios are mapped intoprocesses. For some architectures (e.g. arm64), if most memory is mappedusing contpte-sized and -aligned blocks, TLB usage can be optimized soit's useful to see where these requirements are and are not being met.thpmaps is a Python utility that reads /proc//smaps,/proc//pagemap and /proc/kpageflags to print information about howtransparent huge pages (both file and anon) are mapped to a specifiedprocess or cgroup. It aims to help users debug and optimize theirworkloads. In future we may wish to introduce stats directly into thekernel (e.g. smaps or similar), but for now this provides a short termsolution without the need to introduce any new ABI.Run with help option for a full listing of the arguments: # ./thpmaps --help--8<--usage: thpmaps [-h] [--pid pid | --cgroup path] [--rollup] [--cont size[KMG]] [--inc-smaps] [--inc-empty] [--periodic sleep_ms]Prints information about how transparent huge pages are mapped, eithersystem-wide, or for a specified process or cgroup.When run with --pid, the user explicitly specifies the set of pids toscan. e.g. "--pid 10 [--pid 134 ...]". When run with --cgroup, the userpasses either a v1 or v2 cgroup and all pids that belong to the cgroupsubtree are scanned. When run with neither --pid nor --cgroup, the fullset of pids on the system is gathered from /proc and scanned as if theuser had provided "--pid 1 --pid 2 ...".A default set of statistics is always generated for THP mappings. However, it is also possible to generate additional statistics for"contiguous block mappings" where the block size is user-defined.Statistics are maintained independently for anonymous and file-backed(pagecache) memory and are shown both in kB and as a percentage of eithertotal anonymous or total file-backed memory as appropriate.THP Statistics--------------Statistics are always generated for fully- and contiguously-mapped THPswhose mapping address is aligned to their size, for each supportedby the system. Separate counters describe THPs mapped by PTE vs thosemapped by PMD. (Although note a THP can only be mapped by PMD if it isPMD-sized):- anon-thp-pte-aligned-kB- file-thp-pte-aligned-kB- anon-thp-pmd-aligned-kB- file-thp-pmd-aligned-kBSimilarly, statistics are always generated for fully- and contiguously-mapped THPs whose mapping address is *not* aligned to their size, for each supported by the system. Due to the unaligned mapping, it isimpossible to map by PMD, so there are only PTE counters for this case:- anon-thp-pte-unaligned-kB- file-thp-pte-unaligned-kBStatistics are also always generated for mapped pages that belong to a THPbut where the is THP is *not* fully- and contiguously- mapped. These"partial" mappings are all counted in the same counter regardless of thesize of the THP that is partially mapped:- anon-thp-pte-partial- file-thp-pte-partialContiguous Block Statistics---------------------------An optional, additional set of statistics is generated for everycontiguous block size specified with `--cont `. These statisticsshow how much memory is mapped in contiguous blocks of and alsoaligned to . A given contiguous block must all belong to the sameTHP, but there is no requirement for it to be the *whole* THP. Separatecounters describe contiguous blocks mapped by PTE vs those mapped by PMD:- anon-cont-pte-aligned-kB- file-cont-pte-aligned-kB- anon-cont-pmd-aligned-kB- file-cont-pmd-aligned-kBAs an example, if monitoring 64K contiguous blocks (--cont 64K), there area number of sources that could provide such blocks: a fully- andcontiguously-mapped 64K THP that is aligned to a 64K boundary wouldprovide 1 block. A fully- and contiguously-mapped 128K THP that isaligned to at least a 64K boundary would provide 2 blocks. Or a 128K THPthat maps its first 100K, but contiguously and starting at a 64K boundarywould provide 1 block. A fully- and contiguously-mapped 2M THP wouldprovide 32 blocks. There are many other possible permutations.options: -h, --help show this help message and exit --pid pid Process id of the target process. Maybe issued multiple times to scan multiple processes. --pid and --cgroup are mutually exclusive. If neither are provided, all processes are scanned to provide system-wide information. --cgroup path Path to the target cgroup in sysfs. Iterates over every pid in the cgroup and its children. --pid and --cgroup are mutually exclusive. If neither are provided, all processes are scanned to provide system-wide information. --rollup Sum the per-vma statistics to provide a summary over the whole system, process or cgroup. --cont size[KMG] Adds stats for memory that is mapped in contiguous blocks of and also aligned to . May be issued multiple times to track multiple sized blocks. Useful to infer e.g. arm64 contpte and hpa mappings. Size must be a power-of-2 number of pages. --inc-smaps Include all numerical, additive /proc//smaps stats in the output. --inc-empty Show all statistics including those whose value is 0. --periodic sleep_ms Run in a loop, polling every sleep_ms milliseconds.Requires root privilege to access pagemap and kpageflags.--8<--Example command to summarise fully and partially mapped THPs and 64Kcontiguous blocks over all VMAs in all processes in the system(--inc-empty forces printing stats that are 0): # ./thpmaps --cont 64K --rollup --inc-empty--8<--anon-thp-pmd-aligned-2048kB: 139264 kB ( 6%)file-thp-pmd-aligned-2048kB: 0 kB ( 0%)anon-thp-pte-aligned-16kB: 0 kB ( 0%)anon-thp-pte-aligned-32kB: 0 kB ( 0%)anon-thp-pte-aligned-64kB: 72256 kB ( 3%)anon-thp-pte-aligned-128kB: 0 kB ( 0%)anon-thp-pte-aligned-256kB: 0 kB ( 0%)anon-thp-pte-aligned-512kB: 0 kB ( 0%)anon-thp-pte-aligned-1024kB: 0 kB ( 0%)anon-thp-pte-aligned-2048kB: 0 kB ( 0%)anon-thp-pte-unaligned-16kB: 0 kB ( 0%)anon-thp-pte-unaligned-32kB: 0 kB ( 0%)anon-thp-pte-unaligned-64kB: 0 kB ( 0%)anon-thp-pte-unaligned-128kB: 0 kB ( 0%)anon-thp-pte-unaligned-256kB: 0 kB ( 0%)anon-thp-pte-unaligned-512kB: 0 kB ( 0%)anon-thp-pte-unaligned-1024kB: 0 kB ( 0%)anon-thp-pte-unaligned-2048kB: 0 kB ( 0%)anon-thp-pte-partial: 63232 kB ( 3%)file-thp-pte-aligned-16kB: 809024 kB (47%)file-thp-pte-aligned-32kB: 43168 kB ( 3%)file-thp-pte-aligned-64kB: 98496 kB ( 6%)file-thp-pte-aligned-128kB: 17536 kB ( 1%)file-thp-pte-aligned-256kB: 0 kB ( 0%)file-thp-pte-aligned-512kB: 0 kB ( 0%)file-thp-pte-aligned-1024kB: 0 kB ( 0%)file-thp-pte-aligned-2048kB: 0 kB ( 0%)file-thp-pte-unaligned-16kB: 21712 kB ( 1%)file-thp-pte-unaligned-32kB: 704 kB ( 0%)file-thp-pte-unaligned-64kB: 896 kB ( 0%)file-thp-pte-unaligned-128kB: 44928 kB ( 3%)file-thp-pte-unaligned-256kB: 0 kB ( 0%)file-thp-pte-unaligned-512kB: 0 kB ( 0%)file-thp-pte-unaligned-1024kB: 0 kB ( 0%)file-thp-pte-unaligned-2048kB: 0 kB ( 0%)file-thp-pte-partial: 9252 kB ( 1%)anon-cont-pmd-aligned-64kB: 139264 kB ( 6%)file-cont-pmd-aligned-64kB: 0 kB ( 0%)anon-cont-pte-aligned-64kB: 100672 kB ( 4%)file-cont-pte-aligned-64kB: 161856 kB ( 9%)--8<--Link: https://lkml.kernel.org/r/20240116141235.960842-1-ryan.roberts@arm.comSigned-off-by: Ryan Roberts Tested-by: Barry Song Cc: Alistair Popple Cc: David Hildenbrand Cc: John Hubbard Cc: Kefeng Wang Cc: Matthew Wilcox (Oracle) Cc: William Kucharski Cc: Zenghui Yu Cc: Zi Yan Signed-off-by: Andrew Morton List of files: /linux/tools/mm/thpmaps