Adrian Huang, profile picture
Uploaded byAdrian Huang
2,147 views

Memory Management with Page Folios

The document describes a memory management system using memory folios to address problems with legacy page caching and compound pages. Memory folios provide a unified interface for accessing pages and simplify operations on high-order and compound pages. Folios also improve page cache performance by maintaining a shorter LRU list with one entry per folio rather than per page.

Software◦
Related topics:
Memory Management•

Embed presentation

Downloaded 66 times
* Based on kernel 6.3 (x86_64) – QEMU * 2-socket CPUs (4 cores/socket) * 16GB memory * Kernel parameter: nokaslr norandmaps * KASAN: disabled * Userspace: ASLR is disabled * Legacy BIOS Memory Management with Page Folios Adrian Huang | May, 2023
Agenda • Problem description ✓[Background] Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
Normal high-order page & compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page Page Page Page Normal high-order page pages = alloc_pages(GFP_KERNEL, 2); Four physically contiguous pages: Init compound page metadata during page allocation pages = alloc_pages(GFP_KERNEL | __GFP_COMP, 2); Four physically contiguous pages: not a compound page
Compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) Compound page – Use Cases • Mainly used in huge page ✓ hugetlbfs (also called HugeTLB Pages or persistent huge pages) ➢ Reserved inside the kernel and cannot be used for other purposes. ➢ Cannot be swapped out. ➢ Two allocation methods: • Pre-allocated to the kernel huge page pool with appending kernel parameter. • [Dynamically allocated huge pages of the default size] Example: `echo 10 > /proc/sys/vm/nr_hugepages` ➢ User application calls the mmap system call or shared memory system calls (shmget and shmat) to request the huge page allocation. ➢ Used by database for many years ➢ Manual configuration for hugetlb pages is required. ➢ Application change is required. (via open/mmap)
Compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head)
Compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) Compound page – Use Cases • Mainly used in huge page ✓ Transparent Huge Page (THP) ➢ Support the automatic promotion and demotion of page sizes ➢ Transparent to the application: No need to modify application. ➢ Control via /sys/kernel/mm/transparent_hugepage/enabled:
Compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) Compound page – Use Cases • Mainly used in huge page • kmalloc: allocation size > 8192 bytes o Check kmalloc_order() • Memory folio When to configure compound page? • Condition: page order >= 1 && __GFP_COMP allocation flag is set • alloc_pages -> … -> prep_new_page -> prep_compound_page
Compound page: Problem Description #1 Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) [Problem Description] No unified interface: Ambiguity • Some functions may deal with PAGES_SIZE unit (4KB): They’re unaware of compound pages and huge pages • Some functions accept the page head *only* • Some functions accept the page head or page tail ✓ Call compound_head() to get the page head: waste instructions to get the page head → Performance impact ✓ compound_head() users: ➢ get_page(): This function is called quite frequently ➢ put_page(): This function is called quite frequently ➢ …
Legacy page cache 4KB 4KB 512B 512B 512B 512B [file] file->f_pos (continuous file position) Page cache 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B sector .. Disk 4KB Page Cache Buffer Cache (Buffer head) Kernel Space read()/write()/sendfile() User Space mmap()
Legacy page cache: Problem Description #2 1. Page cache occupies most of memory pages!!! 2. Each page cache (no “compound page” concept) is added to active/inactive lru list • Long lru list: lock contention & cache misses
Agenda • Problem description ✓Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
Memory folio’s goal • Unified interface ✓All accesses via folio struct (head page/tail page in a compound page) • [Page Cache] Shorter LRU list ✓Original: one struct page per 4KB to be added to LRU list ✓Folio: one struct page (page head) per 8KB, 16KB, 32KB, 64KB, 128KB….and so on (include THP) to be added to LRU list • [Anonymous page] THP support ✓create_huge_pmd -> do_huge_pmd_anonymous_page -> __do_huge_pmd_anonymous_page * THP: Transparent Huge Page
Agenda • Problem description ✓Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
Page cache with folio 4KB 512B … 512B [file] file->f_pos (continuous file position) folio sector … … .. Disk page page page page 4KB 4KB 4KB 4KB 4KB 4KB 4KB 512B … 512B 512B … 512B 512B … 512B 512B … 512B page page page page 512B … 512B 512B … 512B 512B … 512B Kernel Space read()/write()/sendfile() User Space mmap() • Folio is the container of struct page(s) ✓ All accesses via folio struct ✓ No tail page → fewer run-time checks
Page cache with folio 4KB 512B … 512B [file] file->f_pos (continuous file position) folio sector … … .. Disk page page page page 4KB 4KB 4KB 4KB 4KB 4KB 4KB 512B … 512B 512B … 512B 512B … 512B 512B … 512B page page page page 512B … 512B 512B … 512B 512B … 512B Kernel Space read()/write()/sendfile() User Space mmap() Folio’s page order: readahead mechanism • CONFIG_TRANSPARENT_HUGEPAGE is enabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 9 (512 pages) • CONFIG_TRANSPARENT_HUGEPAGE is disabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 8 (256 pages) • Commit 793917d997df (“mm/readahead: Add large folio readahead”): merged in 5.18 kernel • Default readahead size: 128KB (32 pages)
Page cache with folio 4KB 512B … 512B [file] file->f_pos (continuous file position) folio sector … … .. Disk page page page page 4KB 4KB 4KB 4KB 4KB 4KB 4KB 512B … 512B 512B … 512B 512B … 512B 512B … 512B page page page page 512B … 512B 512B … 512B 512B … 512B Kernel Space read()/write()/sendfile() User Space mmap() 1. Short LRU list: Only the head page of folio is added LRU list → Performance improvement 2. 45% improvement for lru-file-mmap-read (vm-scalability): Matthew Wilcox’s PDF file Folio’s page order: readahead mechanism • CONFIG_TRANSPARENT_HUGEPAGE is enabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 9 (512 pages) • CONFIG_TRANSPARENT_HUGEPAGE is disabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 8 (256 pages) • Commit 793917d997df (“mm/readahead: Add large folio readahead”): merged in 5.18 kernel • Default readahead size: 128KB (32 pages)
Page cache with folio: backtrace * kernel: 6.3
Agenda • Problem description ✓Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Page #0 (head) flags … Page #1 (tail) flags compound_head compound_dtor compound_order compound_mapcount compound_nr … Page #N (tail) flags _compound_pad_1 hpage_pinned_refcount deferred_list . . . page struct vs folio struct Compound pages
folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Page #0 (head) flags … Page #1 (tail) flags compound_head compound_dtor compound_order compound_mapcount compound_nr … Page #2 (tail) flags _compound_pad_1 hpage_pinned_refcount deferred_list . . . page struct vs folio struct Compound pages folio’s benefit • [Example] 512KB compound page ✓ page struct: Need to maintain 128 page structs (1 head page and 127 tail pages) ✓ folio struct: Maintain 3 page structs regardless of the size of compound pages.
folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 → won’t be used _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 → won’t be used struct struct list_head _deferred_list page struct vs folio struct folio struct’s members • _entire_mapcount ✓ The compound page is mapped via a single PMD (huge page). • _nr_pages_mapped ✓ Number of individual subpages (PTE: 4KB pages) are mapped. ✓ Scenario: Two processes map the same memory range ✓ One process maps the entire 2MB compound page (Transparent Huge Page - THP): mapped via a single PMD ✓ The other process maps some 4KB pages within this 2MB memory area: mapped via PTEs ✓ Benefit about THP: No need to split the huge page if other processes map 4KB pages within the same memory area. • _folio_nr_pages ✓ Number of pages in this folio. ✓ _folio_nr_pages = 1 << order, where order > 0.
folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Page #0 (head) flags … Page #1 (tail) flags compound_head compound_dtor compound_order compound_mapcount compound_nr … Page #2 (tail) flags _compound_pad_1 hpage_pinned_refcount deferred_list . . . folio struct vs legacy compound page Page #1 of folio and legacy page struct has the same mapping
Agenda • Problem description ✓Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
[kernel v5.11] total_mapcount() page Page cache and anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 1: Singleton page(s) Get _mapcount directly total_mapcount() users: • Huge page • rmap (reverse mapping)
page (head) Page cache and anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 2: Compound page && hugetlb (hugetlbfs) page page (first tail) compound_head struct union compound_dtor compound_order compound_mapcount compound_nr = 1 << compound_nr struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . page (second tail) _compound_pad_1 struct union hpage_pinned_refcount deferred_list struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . . . . _compound_pad_1 (compound_head) page (second tail) Get compound_mapcount directly compound_mapcount: • Map count of the whole compound page (does not include mapped sub-pages) Steps: 1. Get the head page based on any page (page head or page tail) 2. Read ‘compound_mapcount’ of the first tail page A. page[1].compound_mapcount [kernel v5.11] total_mapcount()
page (head) Page cache and anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 3: [Anonymous page] Compound page && transparent huge page page (first tail) compound_head struct union compound_dtor compound_order compound_mapcount compound_nr = 1 << compound_nr struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . page (second tail) _compound_pad_1 struct union hpage_pinned_refcount deferred_list struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . . . . _compound_pad_1 (compound_head) page (second tail) Steps: 1. Get the head page based on any page (page head or page tail) 2. Read ‘compound_mapcount’ of the first tail page A. page[1].compound_mapcount: Map count of the whole compound page 3. `Accumulate each subpage._mapcount`: A. One process maps 2MB range as a single huge page (a single PMD) B. Another process maps 512 individual PTEs 4. `Accumulate each subpage._mapcount` + page[1].compound_mapcount [kernel v5.11] total_mapcount()
page (head) Page cache and anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 3: [Page cache] Compound page && transparent huge page page (first tail) compound_head struct union compound_dtor compound_order compound_mapcount compound_nr = 1 << compound_nr struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . page (second tail) _compound_pad_1 struct union hpage_pinned_refcount deferred_list struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . . . . _compound_pad_1 (compound_head) page (second tail) Steps: 1. Get the head page based on any page (page head or page tail) 2. Read ‘compound_mapcount’ of the first tail page A. page[1].compound_mapcount: Map count of the whole compound page 3. `Accumulate each subpage._mapcount`: A. One process maps 2MB range as a single huge page (a single PMD) B. Another process maps 512 individual PTEs 4. `Accumulate each subpage._mapcount` + page[1].compound_mapcount - page[1].compound_mapcount * page[1].compound_nr A. File pages has compound_mapcount included in _mapcount [kernel v5.11] total_mapcount()
[kernel v6.3] total_mapcount() and folio_mapcount() folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 … void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Case 1: Singleton page – Not a compound page 1 2
[kernel v6.3] total_mapcount() and folio_mapcount() folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 … void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Case 2: Compound page is mapped via PMD (huge page) 1 3 2 Get _entire_mapcount 4 5 Get _nr_pages_mapped
[kernel v6.3] total_mapcount() and folio_mapcount() folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 … void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Case 3: Compound page is mapped via PMD (huge page) and some subpages are mapped by PTE 1 2 mapcount = folio’s _entire_mapcount + sum(each subpage’s _mapcount)
Reference • Memory Folios • LWN - A memory-folio update • LWN - An introduction to compound pages • LWN - Huge pages part 1 (Introduction) • Documentation/mm/transhuge.rst
backup
Learn new C standard (C11) from folio: Generic Selection * Reference from: ISO/IEC 9899:201x • C99 defines type-generic macros in the standardized library: the type of argument is detected automatically, and the corresponding function is invoked based on that type. ✓ Example: sqrt(X), ➢ X is double → invoke sqrt() ➢ X is float → invoke sqrtf() ➢ X is long double → invoke sqrtl() • However, programmers cannot define their own type- generic macros in C99. • In C11, programmers can define their own type-generic macros:
Some Functions

More Related Content

PDF
Linux Kernel - Virtual File System
byAdrian Huang
 
PDF
Memory Compaction in Linux Kernel.pdf
byAdrian Huang
 
PDF
Reverse Mapping (rmap) in Linux Kernel
byAdrian Huang
 
PDF
Decompressed vmlinux: linux kernel initialization from page table configurati...
byAdrian Huang
 
PDF
Anatomy of the loadable kernel module (lkm)
byAdrian Huang
 
PDF
Physical Memory Models.pdf
byAdrian Huang
 
PPTX
Slab Allocator in Linux Kernel
byAdrian Huang
 
PDF
Page cache in Linux kernel
byAdrian Huang
 
Linux Kernel - Virtual File System
byAdrian Huang
 
Memory Compaction in Linux Kernel.pdf
byAdrian Huang
 
Reverse Mapping (rmap) in Linux Kernel
byAdrian Huang
 
Decompressed vmlinux: linux kernel initialization from page table configurati...
byAdrian Huang
 
Anatomy of the loadable kernel module (lkm)
byAdrian Huang
 
Physical Memory Models.pdf
byAdrian Huang
 
Slab Allocator in Linux Kernel
byAdrian Huang
 
Page cache in Linux kernel
byAdrian Huang
 

What's hot

PDF
Memory Mapping Implementation (mmap) in Linux Kernel
byAdrian Huang
 
PDF
Process Address Space: The way to create virtual address (page table) of user...
byAdrian Huang
 
PDF
Physical Memory Management.pdf
byAdrian Huang
 
PDF
malloc & vmalloc in Linux
byAdrian Huang
 
PPTX
qemu + gdb + sample_code: Run sample code in QEMU OS and observe Linux Kernel...
byAdrian Huang
 
PPTX
Linux MMAP & Ioremap introduction
byGene Chang
 
PPTX
Linux Memory Management
byNi Zo-Ma
 
PDF
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
byAdrian Huang
 
PPTX
Linux Kernel Booting Process (2) - For NLKB
byshimosawa
 
PDF
How to use KASAN to debug memory corruption in OpenStack environment- (2)
byGavin Guo
 
PDF
Memory management in Linux kernel
byVadim Nikitin
 
PDF
spinlock.pdf
byAdrian Huang
 
PDF
semaphore & mutex.pdf
byAdrian Huang
 
PDF
Vmlinux: anatomy of bzimage and how x86 64 processor is booted
byAdrian Huang
 
PDF
BPF Internals (eBPF)
byBrendan Gregg
 
ODP
Memory management in Linux
byRaghu Udiyar
 
PPTX
Linux Initialization Process (2)
byshimosawa
 
PPTX
Understanding eBPF in a Hurry!
byRay Jenkins
 
PDF
Process' Virtual Address Space in GNU/Linux
byVarun Mahajan
 
PDF
The linux networking architecture
byhugo lu
 
Memory Mapping Implementation (mmap) in Linux Kernel
byAdrian Huang
 
Process Address Space: The way to create virtual address (page table) of user...
byAdrian Huang
 
Physical Memory Management.pdf
byAdrian Huang
 
malloc & vmalloc in Linux
byAdrian Huang
 
qemu + gdb + sample_code: Run sample code in QEMU OS and observe Linux Kernel...
byAdrian Huang
 
Linux MMAP & Ioremap introduction
byGene Chang
 
Linux Memory Management
byNi Zo-Ma
 
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
byAdrian Huang
 
Linux Kernel Booting Process (2) - For NLKB
byshimosawa
 
How to use KASAN to debug memory corruption in OpenStack environment- (2)
byGavin Guo
 
Memory management in Linux kernel
byVadim Nikitin
 
spinlock.pdf
byAdrian Huang
 
semaphore & mutex.pdf
byAdrian Huang
 
Vmlinux: anatomy of bzimage and how x86 64 processor is booted
byAdrian Huang
 
BPF Internals (eBPF)
byBrendan Gregg
 
Memory management in Linux
byRaghu Udiyar
 
Linux Initialization Process (2)
byshimosawa
 
Understanding eBPF in a Hurry!
byRay Jenkins
 
Process' Virtual Address Space in GNU/Linux
byVarun Mahajan
 
The linux networking architecture
byhugo lu
 

Similar to Memory Management with Page Folios

PDF
Page Cache in Linux 2.6.pdf
byycelgemici1
 
PPTX
Linux Memory Management with CMA (Contiguous Memory Allocator)
byPankaj Suryawanshi
 
PDF
Understand
byKalimuthu Velappan
 
PDF
Practical ,Transparent Operating System Support For Superpages
byNadeeshani Hewage
 
PDF
AOS Lab 7: Page tables
byZubair Nabi
 
PDF
Vmreport
bymeru2ks
 
PPT
Windows memory manager internals
bySisimon Soman
 
PPT
Chapter 04
by Google
 
PDF
Buddy system
byChenYiHuang5
 
PDF
Кирилл Шутемов aka “kas” - О Transparent Hugepages и Huge zero page
byMinsk Linux User Group
 
PPTX
Virtual Memory Managementddddddddffffffffffffff.pptx
byananyachouhan70
 
PPTX
Abhaycavirtual memory and the pagehit.pptx
bywemoji5816
 
PDF
Unit 5
bypm_ghate
 
PPT
Segmentation with paging methods and techniques
bynikhilrana24112003
 
PPT
Paging and Segmentation
bysathish sak
 
PPT
Linux memory
byericrain911
 
PPT
Memory Management from operating system module 4
byTejH2
 
PPT
Linux Memory Management
byRajan Kandel
 
PPT
Memory Management
byRamasubbu .P
 
PPT
memory
byKishore Kumar
 
Page Cache in Linux 2.6.pdf
byycelgemici1
 
Linux Memory Management with CMA (Contiguous Memory Allocator)
byPankaj Suryawanshi
 
Understand
byKalimuthu Velappan
 
Practical ,Transparent Operating System Support For Superpages
byNadeeshani Hewage
 
AOS Lab 7: Page tables
byZubair Nabi
 
Vmreport
bymeru2ks
 
Windows memory manager internals
bySisimon Soman
 
Chapter 04
by Google
 
Buddy system
byChenYiHuang5
 
Кирилл Шутемов aka “kas” - О Transparent Hugepages и Huge zero page
byMinsk Linux User Group
 
Virtual Memory Managementddddddddffffffffffffff.pptx
byananyachouhan70
 
Abhaycavirtual memory and the pagehit.pptx
bywemoji5816
 
Unit 5
bypm_ghate
 
Segmentation with paging methods and techniques
bynikhilrana24112003
 
Paging and Segmentation
bysathish sak
 
Linux memory
byericrain911
 
Memory Management from operating system module 4
byTejH2
 
Linux Memory Management
byRajan Kandel
 
Memory Management
byRamasubbu .P
 
memory
byKishore Kumar
 

Recently uploaded

PDF
Water Delivery App Development Solutions.pdf
byeSiteWorld TechnoLabs Pvt. Ltd.
 
PPTX
HackYourBrain_JFokusConference_03022026.pptx
bySimonedeGijt
 
PDF
Why I Volunteer at FOSDEM and You Should Too
byImma Valls Bernaus
 
PDF
Monitoring your MySQL Server's Health Trends
byIgor Donchovski
 
PDF
LogiNext Media Kit & Company Overview 2025
bysmritipatil055
 
PPTX
Marketo API Deep Dive From Basics to AI-Ready Foundations - January 2026.pptx
bybbedford2
 
PDF
Windows Server 2025 Administration Fundamentals 4ED
byraynasolomon136
 
PDF
Image and video processing: From Mars to Hollywood with a stop at the hospita...
byHarinath Palavalli
 
PDF
Attendance Dashboard – Time & Attendance Analytics
byHajir Pro
 
DOCX
The Rise of Headless CMS in Modern Web Development.docx
bywork4noahmiller
 
PDF
2026 Safety Roadmap: Systems, Skills, and Scale for EHS Leaders
byTECH EHS Solution
 
PDF
Smart ways to se AI in your business , presentation
byAI n Dot Net
 
PDF
Introducing MySQL HeatWave Migration Assistant
byAlexander Hitrov
 
PDF
Accelerating Data Validation with QuerySurge AI
byRTTS
 
PDF
SLIDE PROPARTNER SUMMIT 2025 - ITALY.pdf
bygiuo
 
PDF
Manual Testing Still Matters: 7 Areas Where Humans Beat Automation
byKiwiQA Services
 
PDF
Visualizing Your Data Lake with Grafana - Amsterdam
byImma Valls Bernaus
 
PDF
LogiNext Media Kit & Company Overview 2025
bypriyajoshi2929
 
PDF
AI-native development: from Prompt to Platform
byMaxim Salnikov
 
PDF
Physiotherapist App Development for Modern Rehabilitation
byV3CUBE TECHNOLABS
 
Water Delivery App Development Solutions.pdf
byeSiteWorld TechnoLabs Pvt. Ltd.
 
HackYourBrain_JFokusConference_03022026.pptx
bySimonedeGijt
 
Why I Volunteer at FOSDEM and You Should Too
byImma Valls Bernaus
 
Monitoring your MySQL Server's Health Trends
byIgor Donchovski
 
LogiNext Media Kit & Company Overview 2025
bysmritipatil055
 
Marketo API Deep Dive From Basics to AI-Ready Foundations - January 2026.pptx
bybbedford2
 
Windows Server 2025 Administration Fundamentals 4ED
byraynasolomon136
 
Image and video processing: From Mars to Hollywood with a stop at the hospita...
byHarinath Palavalli
 
Attendance Dashboard – Time & Attendance Analytics
byHajir Pro
 
The Rise of Headless CMS in Modern Web Development.docx
bywork4noahmiller
 
2026 Safety Roadmap: Systems, Skills, and Scale for EHS Leaders
byTECH EHS Solution
 
Smart ways to se AI in your business , presentation
byAI n Dot Net
 
Introducing MySQL HeatWave Migration Assistant
byAlexander Hitrov
 
Accelerating Data Validation with QuerySurge AI
byRTTS
 
SLIDE PROPARTNER SUMMIT 2025 - ITALY.pdf
bygiuo
 
Manual Testing Still Matters: 7 Areas Where Humans Beat Automation
byKiwiQA Services
 
Visualizing Your Data Lake with Grafana - Amsterdam
byImma Valls Bernaus
 
LogiNext Media Kit & Company Overview 2025
bypriyajoshi2929
 
AI-native development: from Prompt to Platform
byMaxim Salnikov
 
Physiotherapist App Development for Modern Rehabilitation
byV3CUBE TECHNOLABS
 

Memory Management with Page Folios

  • 1.
    * Based onkernel 6.3 (x86_64) – QEMU * 2-socket CPUs (4 cores/socket) * 16GB memory * Kernel parameter: nokaslr norandmaps * KASAN: disabled * Userspace: ASLR is disabled * Legacy BIOS Memory Management with Page Folios Adrian Huang | May, 2023
  • 2.
    Agenda • Problem description ✓[Background]Normal high-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
  • 3.
    Normal high-order page& compound page Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page Page Page Page Normal high-order page pages = alloc_pages(GFP_KERNEL, 2); Four physically contiguous pages: Init compound page metadata during page allocation pages = alloc_pages(GFP_KERNEL | __GFP_COMP, 2); Four physically contiguous pages: not a compound page
  • 4.
    Compound page Page (head page) flags|= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) Compound page – Use Cases • Mainly used in huge page ✓ hugetlbfs (also called HugeTLB Pages or persistent huge pages) ➢ Reserved inside the kernel and cannot be used for other purposes. ➢ Cannot be swapped out. ➢ Two allocation methods: • Pre-allocated to the kernel huge page pool with appending kernel parameter. • [Dynamically allocated huge pages of the default size] Example: `echo 10 > /proc/sys/vm/nr_hugepages` ➢ User application calls the mmap system call or shared memory system calls (shmget and shmat) to request the huge page allocation. ➢ Used by database for many years ➢ Manual configuration for hugetlb pages is required. ➢ Application change is required. (via open/mmap)
  • 5.
    Compound page Page (head page) flags|= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head)
  • 6.
    Compound page Page (head page) flags|= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) Compound page – Use Cases • Mainly used in huge page ✓ Transparent Huge Page (THP) ➢ Support the automatic promotion and demotion of page sizes ➢ Transparent to the application: No need to modify application. ➢ Control via /sys/kernel/mm/transparent_hugepage/enabled:
  • 7.
    Compound page Page (head page) flags|= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) Compound page – Use Cases • Mainly used in huge page • kmalloc: allocation size > 8192 bytes o Check kmalloc_order() • Memory folio When to configure compound page? • Condition: page order >= 1 && __GFP_COMP allocation flag is set • alloc_pages -> … -> prep_new_page -> prep_compound_page
  • 8.
    Compound page: ProblemDescription #1 Page (head page) flags |= PG_head Page (Tail page) compound_head compound_dtor compound_order compound_mapcount compound_nr First Tail Page only Page (Tail page) _compound_pad_1 (compound_head) hpage_pinned_refcount deferred_list 2nd Tail Page only . . . Compound Page Page (Tail page) _compound_pad_1 (compound_head) Page (Tail page) _compound_pad_1 (compound_head) [Problem Description] No unified interface: Ambiguity • Some functions may deal with PAGES_SIZE unit (4KB): They’re unaware of compound pages and huge pages • Some functions accept the page head *only* • Some functions accept the page head or page tail ✓ Call compound_head() to get the page head: waste instructions to get the page head → Performance impact ✓ compound_head() users: ➢ get_page(): This function is called quite frequently ➢ put_page(): This function is called quite frequently ➢ …
  • 9.
    Legacy page cache 4KB4KB 512B 512B 512B 512B [file] file->f_pos (continuous file position) Page cache 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B 512B sector .. Disk 4KB Page Cache Buffer Cache (Buffer head) Kernel Space read()/write()/sendfile() User Space mmap()
  • 10.
    Legacy page cache:Problem Description #2 1. Page cache occupies most of memory pages!!! 2. Each page cache (no “compound page” concept) is added to active/inactive lru list • Long lru list: lock contention & cache misses
  • 11.
    Agenda • Problem description ✓Normalhigh-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
  • 12.
    Memory folio’s goal •Unified interface ✓All accesses via folio struct (head page/tail page in a compound page) • [Page Cache] Shorter LRU list ✓Original: one struct page per 4KB to be added to LRU list ✓Folio: one struct page (page head) per 8KB, 16KB, 32KB, 64KB, 128KB….and so on (include THP) to be added to LRU list • [Anonymous page] THP support ✓create_huge_pmd -> do_huge_pmd_anonymous_page -> __do_huge_pmd_anonymous_page * THP: Transparent Huge Page
  • 13.
    Agenda • Problem description ✓Normalhigh-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
  • 14.
    Page cache withfolio 4KB 512B … 512B [file] file->f_pos (continuous file position) folio sector … … .. Disk page page page page 4KB 4KB 4KB 4KB 4KB 4KB 4KB 512B … 512B 512B … 512B 512B … 512B 512B … 512B page page page page 512B … 512B 512B … 512B 512B … 512B Kernel Space read()/write()/sendfile() User Space mmap() • Folio is the container of struct page(s) ✓ All accesses via folio struct ✓ No tail page → fewer run-time checks
  • 15.
    Page cache withfolio 4KB 512B … 512B [file] file->f_pos (continuous file position) folio sector … … .. Disk page page page page 4KB 4KB 4KB 4KB 4KB 4KB 4KB 512B … 512B 512B … 512B 512B … 512B 512B … 512B page page page page 512B … 512B 512B … 512B 512B … 512B Kernel Space read()/write()/sendfile() User Space mmap() Folio’s page order: readahead mechanism • CONFIG_TRANSPARENT_HUGEPAGE is enabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 9 (512 pages) • CONFIG_TRANSPARENT_HUGEPAGE is disabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 8 (256 pages) • Commit 793917d997df (“mm/readahead: Add large folio readahead”): merged in 5.18 kernel • Default readahead size: 128KB (32 pages)
  • 16.
    Page cache withfolio 4KB 512B … 512B [file] file->f_pos (continuous file position) folio sector … … .. Disk page page page page 4KB 4KB 4KB 4KB 4KB 4KB 4KB 512B … 512B 512B … 512B 512B … 512B 512B … 512B page page page page 512B … 512B 512B … 512B 512B … 512B Kernel Space read()/write()/sendfile() User Space mmap() 1. Short LRU list: Only the head page of folio is added LRU list → Performance improvement 2. 45% improvement for lru-file-mmap-read (vm-scalability): Matthew Wilcox’s PDF file Folio’s page order: readahead mechanism • CONFIG_TRANSPARENT_HUGEPAGE is enabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 9 (512 pages) • CONFIG_TRANSPARENT_HUGEPAGE is disabled ✓ Minimum: order 2 (4 pages) ✓ Maximum: order 8 (256 pages) • Commit 793917d997df (“mm/readahead: Add large folio readahead”): merged in 5.18 kernel • Default readahead size: 128KB (32 pages)
  • 17.
    Page cache withfolio: backtrace * kernel: 6.3
  • 18.
    Agenda • Problem description ✓Normalhigh-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
  • 19.
    folio flags struct list_head lru void*__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Page #0 (head) flags … Page #1 (tail) flags compound_head compound_dtor compound_order compound_mapcount compound_nr … Page #N (tail) flags _compound_pad_1 hpage_pinned_refcount deferred_list . . . page struct vs folio struct Compound pages
  • 20.
    folio flags struct list_head lru void*__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Page #0 (head) flags … Page #1 (tail) flags compound_head compound_dtor compound_order compound_mapcount compound_nr … Page #2 (tail) flags _compound_pad_1 hpage_pinned_refcount deferred_list . . . page struct vs folio struct Compound pages folio’s benefit • [Example] 512KB compound page ✓ page struct: Need to maintain 128 page structs (1 head page and 127 tail pages) ✓ folio struct: Maintain 3 page structs regardless of the size of compound pages.
  • 21.
    folio flags struct list_head lru void*__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 → won’t be used _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 → won’t be used struct struct list_head _deferred_list page struct vs folio struct folio struct’s members • _entire_mapcount ✓ The compound page is mapped via a single PMD (huge page). • _nr_pages_mapped ✓ Number of individual subpages (PTE: 4KB pages) are mapped. ✓ Scenario: Two processes map the same memory range ✓ One process maps the entire 2MB compound page (Transparent Huge Page - THP): mapped via a single PMD ✓ The other process maps some 4KB pages within this 2MB memory area: mapped via PTEs ✓ Benefit about THP: No need to split the huge page if other processes map 4KB pages within the same memory area. • _folio_nr_pages ✓ Number of pages in this folio. ✓ _folio_nr_pages = 1 << order, where order > 0.
  • 22.
    folio flags struct list_head lru void*__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 _flags_2 _head_2 void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Page #0 (head) flags … Page #1 (tail) flags compound_head compound_dtor compound_order compound_mapcount compound_nr … Page #2 (tail) flags _compound_pad_1 hpage_pinned_refcount deferred_list . . . folio struct vs legacy compound page Page #1 of folio and legacy page struct has the same mapping
  • 23.
    Agenda • Problem description ✓Normalhigh-order page & compound page ✓Legacy page cache • Memory folio’s goal • Solution ✓Page cache with memory folio ✓page struct vs folio struct ✓[Example] total_mapcount() implementation difference between legacy approach and folio
  • 24.
    [kernel v5.11] total_mapcount() page Pagecache and anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 1: Singleton page(s) Get _mapcount directly total_mapcount() users: • Huge page • rmap (reverse mapping)
  • 25.
    page (head) Page cacheand anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 2: Compound page && hugetlb (hugetlbfs) page page (first tail) compound_head struct union compound_dtor compound_order compound_mapcount compound_nr = 1 << compound_nr struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . page (second tail) _compound_pad_1 struct union hpage_pinned_refcount deferred_list struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . . . . _compound_pad_1 (compound_head) page (second tail) Get compound_mapcount directly compound_mapcount: • Map count of the whole compound page (does not include mapped sub-pages) Steps: 1. Get the head page based on any page (page head or page tail) 2. Read ‘compound_mapcount’ of the first tail page A. page[1].compound_mapcount [kernel v5.11] total_mapcount()
  • 26.
    page (head) Page cacheand anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 3: [Anonymous page] Compound page && transparent huge page page (first tail) compound_head struct union compound_dtor compound_order compound_mapcount compound_nr = 1 << compound_nr struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . page (second tail) _compound_pad_1 struct union hpage_pinned_refcount deferred_list struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . . . . _compound_pad_1 (compound_head) page (second tail) Steps: 1. Get the head page based on any page (page head or page tail) 2. Read ‘compound_mapcount’ of the first tail page A. page[1].compound_mapcount: Map count of the whole compound page 3. `Accumulate each subpage._mapcount`: A. One process maps 2MB range as a single huge page (a single PMD) B. Another process maps 512 individual PTEs 4. `Accumulate each subpage._mapcount` + page[1].compound_mapcount [kernel v5.11] total_mapcount()
  • 27.
    page (head) Page cacheand anonymous pages struct union page_pool used by netstack struct slab, slob and slub struct Tail pages of compound page struct Second tail page of compound page struct Page table pages: 1. PMD huge PTE 2. x86 pgd page <-> mm_struct struct ZONE_DEVICE pages struct rcu_head: free a page by RCU struct union atomic_t _mapcount: the number of this page is referenced by page table unsigned int page_type unsigned int active: used by slab int units: used by slob atomic_t _refcount … Case 3: [Page cache] Compound page && transparent huge page page (first tail) compound_head struct union compound_dtor compound_order compound_mapcount compound_nr = 1 << compound_nr struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . page (second tail) _compound_pad_1 struct union hpage_pinned_refcount deferred_list struct union atomic_t _mapcount unsigned int page_type unsigned int active int units atomic_t _refcount … . . . struct . . . . . . _compound_pad_1 (compound_head) page (second tail) Steps: 1. Get the head page based on any page (page head or page tail) 2. Read ‘compound_mapcount’ of the first tail page A. page[1].compound_mapcount: Map count of the whole compound page 3. `Accumulate each subpage._mapcount`: A. One process maps 2MB range as a single huge page (a single PMD) B. Another process maps 512 individual PTEs 4. `Accumulate each subpage._mapcount` + page[1].compound_mapcount - page[1].compound_mapcount * page[1].compound_nr A. File pages has compound_mapcount included in _mapcount [kernel v5.11] total_mapcount()
  • 28.
    [kernel v6.3] total_mapcount()and folio_mapcount() folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 … void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Case 1: Singleton page – Not a compound page 1 2
  • 29.
    [kernel v6.3] total_mapcount()and folio_mapcount() folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 … void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Case 2: Compound page is mapped via PMD (huge page) 1 3 2 Get _entire_mapcount 4 5 Get _nr_pages_mapped
  • 30.
    [kernel v6.3] total_mapcount()and folio_mapcount() folio flags struct list_head lru void *__filler mlock_count struct address_space *mapping pgoff_t index union void *private atomic_t _mapcount atomic_t _refcount unsigned long memcg_data struct page page struct struct union _flags_1 _head_1 unsigned char _folio_dtor unsigned char _folio_order atomic_t _entire_mapcount atomic_t _nr_pages_mapped atomic_t _pincount unsigned int _folio_nr_pages struct union struct page __page_1 … void *_hugetlb_subpool void *_hugetlb_cgroup void *_hugetlb_cgroup_rsvd void *_hugetlb_hwpoison _flags_2a _head_2a struct union struct page __page_2 struct struct list_head _deferred_list Case 3: Compound page is mapped via PMD (huge page) and some subpages are mapped by PTE 1 2 mapcount = folio’s _entire_mapcount + sum(each subpage’s _mapcount)
  • 31.
    Reference • Memory Folios •LWN - A memory-folio update • LWN - An introduction to compound pages • LWN - Huge pages part 1 (Introduction) • Documentation/mm/transhuge.rst
  • 32.
  • 33.
    Learn new Cstandard (C11) from folio: Generic Selection * Reference from: ISO/IEC 9899:201x • C99 defines type-generic macros in the standardized library: the type of argument is detected automatically, and the corresponding function is invoked based on that type. ✓ Example: sqrt(X), ➢ X is double → invoke sqrt() ➢ X is float → invoke sqrtf() ➢ X is long double → invoke sqrtl() • However, programmers cannot define their own type- generic macros in C99. • In C11, programmers can define their own type-generic macros:
  • 34.