Linux –
Memory Management
    D3 Tsung-Han “Johnny” Lin
12/4/26   Linux - Memory Management   2
Outline
•   Memory in Linux
     –     Dynamic Allocation
     –     Contiguous Allocation




•Memory Allocation
     – Buddy System
     – Zone
     – SLAB Allocator
•   Per-CPU data




         12/4/26                   Linux - Memory Management   3
MMU - Simple History

Early Linux on x86
                                                        Linux 2.5.46
       (MMU)




          Embedded Developer?!




                                     µClinux
                           (Linux for microconcroller)

                                 Jeff Dionne et al.1998

12/4/26                     Linux - Memory Management                  4
All about PAGE


• Physical page -> page frame
     – Page descriptor, page

     – All pages, mem_map




12/4/26          Linux - Memory Management   5
Initialization

                 query or invoke              Kernel
          BIOS




                             machine_specific_memory_setup()

                             setup_memory_map()

                             struct page* mem_map



                   Physical Memory

12/4/26           Linux - Memory Management                    6
Memory




12/4/26   Linux - Memory Management   7
How pages are allocated?



 12/4/26   Linux - Memory Management   8
Kernel API
• alloc_pages

• __get_free_pages

• kmalloc

• …

• But, seriously

12/4/26      Linux - Memory Management   9
HOW?

12/4/26   Linux - Memory Management   10
Hierarchy Overview
App w/ glibc heap




                                                                                   Zone Allocator
                                                            Slab Allocator
                              glibc




                                                                                                       Buddy Allocator
                                                                                                                         MMU




                                          Kernel                                                                                Physical
                                        Subsystems         VM Subsystem                                                         Memory

                                                                               kswapd
App w/ heap




                                                                             bdflush                 Disk
                                                                                                    Driver


User Space                            Kernel Space                                                                       Disk
                    12/4/26                          Linux - Memory Management                                                      11
Memory Zone
Physical Memory (mem_map)




                                                       0x37999999, 896MB

                            ZONE_DMA               ZONE_NORMAL                        ZONE_HIGHMEM
             12/4/26            0x01000000, 16MB    Linux - Memory Management                        12
                                                                                0x40000000, 1GB
Buddy System
zone descriptor
                  free_area[10]                  1024 pages                 ……
      ………




                      ………




                                                  ………
  free_area[11]
                  free_area[4]
                  free_area[3]

                  free_area[2]                     4 pages                  ……
      ………




                  free_area[1]              2 pages                   2 pages         ……
                  free_area[0]          1 page               1 page     1 page   ……

   12/4/26             Linux - Memory Management                                 13
Zone, again


• ZONE_DMA, 0 ~ 16 MB

          896MB ?!
• ZONE_NORMAL, 16 MB ~ 896 MB

• ZONE_HIGHMEM, rest of them …



12/4/26     Linux - Memory Management   14
Kernel View
                                 Linear – 4 GB                               Physical
                     TASK_SIZE
                                                 0x0

                                     3 GB

PAGE_OFFSET                                                                  ZONE_HIGHMEM
0xc0000000                                                 0x40000000, 1GB
                                     Kernel
              8 MB                    Image                                   Reserved
                                                         0x37999999, 896MB

                                     mem_map           128 MB

                                                                             ZONE_NORMAL    1 GB
                                      …




                                                          0x01000000, 16MB
          896 MB                   ZONE_NORMAL                                ZONE_DMA
                                        +
                                    ZONE_DMA
                                                 0xffffffff
    12/4/26                                    Linux - Memory Management                     15
Do we need this large?!




12/4/26   Linux - Memory Management   16
SLAB Allocator
• First introduced in
                                    大辞泉
     SunOS 5.4
                              slab/slǽb/
• Implemented in Linux
                              名 詞
     2.2                      1 (石木材金属などの)広い厚板;(材木
                              の)背板.
•    SLAB system              2 (食べ物などの)厚切り
      – SLAB                  a thick slab of apple pie|厚切りのアップル
                              パイ.
      – SLOB, embedded system 3⦅ 俗語⦅     〘野球〙ピッチャーズプレート.
      – SLUB, efficient ver.  4⦅ ~⦅⦅
                                 the        俗語⦅   (病院などの)死体置
                              き台
      – SLQB, per-CPU scaling on the slab|(死体となって)安置されて.
• kmalloc()!

    12/4/26              Linux - Memory Management
                                                     Cache !!   17
Design of SLAB
cache_chain

                slab_full
                                   ……
                                                          page   struct
   cache
                                                                  slab
                                         slab
                                                                 object   Page 1
                                                          page




                                                                  …
    cache
               slab_partial              slab
(kmem_cache)                                                     object
                                                          page




                                                                  …
                                         slab
                                                                 object   Page 2
   cache
                                                          page
               slab_partial
                                   ……

   12/4/26                    Linux - Memory Management                   18
Per-CPU Allocation
•   Request/Release single page
     –    Frequently

•   Per-CPU
     –    Reducing cache invalidation
     –    Reducing locking
     –    A simple data array

•   In 2.4 kernel
     –    get_cpu(), preemption disabled
     –    Access array directly

•   In 2.6 kernel
     –    Wrapper
     –    percpu interface
     –    Cache alignment



12/4/26                           Linux - Memory Management   19
Review
App w/ glibc heap




                                                                                   Zone Allocator
                                                            Slab Allocator
                              glibc




                                                                                                       Buddy Allocator
                                                                                                                         MMU




                                          Kernel                                                                                Physical
                                        Subsystems         VM Subsystem                                                         Memory

                                                                               kswapd
App w/ heap




                                                                             bdflush                 Disk
                                                                                                    Driver


User Space                            Kernel Space                                                                       Disk
                    12/4/26                          Linux - Memory Management                                                      20
Reference
• Linux Kernel Development, 3rd
• Understanding the Linux Kernel, 3rd
     – http://www.makelinux.net/books/ulk3/t
          oc
• Understanding the Virtual Memory
    Manager
     – http://www.kernel.org/doc/gorman/htm
          l/understand/index.html


12/4/26              Linux - Memory Management   21

Linux Memory Management

  • 1.
    Linux – Memory Management D3 Tsung-Han “Johnny” Lin
  • 2.
    12/4/26 Linux - Memory Management 2
  • 3.
    Outline • Memory in Linux – Dynamic Allocation – Contiguous Allocation •Memory Allocation – Buddy System – Zone – SLAB Allocator • Per-CPU data 12/4/26 Linux - Memory Management 3
  • 4.
    MMU - SimpleHistory Early Linux on x86 Linux 2.5.46 (MMU) Embedded Developer?! µClinux (Linux for microconcroller) Jeff Dionne et al.1998 12/4/26 Linux - Memory Management 4
  • 5.
    All about PAGE •Physical page -> page frame – Page descriptor, page – All pages, mem_map 12/4/26 Linux - Memory Management 5
  • 6.
    Initialization query or invoke Kernel BIOS machine_specific_memory_setup() setup_memory_map() struct page* mem_map Physical Memory 12/4/26 Linux - Memory Management 6
  • 7.
    Memory 12/4/26 Linux - Memory Management 7
  • 8.
    How pages areallocated? 12/4/26 Linux - Memory Management 8
  • 9.
    Kernel API • alloc_pages •__get_free_pages • kmalloc • … • But, seriously 12/4/26 Linux - Memory Management 9
  • 10.
    HOW? 12/4/26 Linux - Memory Management 10
  • 11.
    Hierarchy Overview App w/glibc heap Zone Allocator Slab Allocator glibc Buddy Allocator MMU Kernel Physical Subsystems VM Subsystem Memory kswapd App w/ heap bdflush Disk Driver User Space Kernel Space Disk 12/4/26 Linux - Memory Management 11
  • 12.
    Memory Zone Physical Memory(mem_map) 0x37999999, 896MB ZONE_DMA ZONE_NORMAL ZONE_HIGHMEM 12/4/26 0x01000000, 16MB Linux - Memory Management 12 0x40000000, 1GB
  • 13.
    Buddy System zone descriptor free_area[10] 1024 pages …… ……… ……… ……… free_area[11] free_area[4] free_area[3] free_area[2] 4 pages …… ……… free_area[1] 2 pages 2 pages …… free_area[0] 1 page 1 page 1 page …… 12/4/26 Linux - Memory Management 13
  • 14.
    Zone, again • ZONE_DMA,0 ~ 16 MB 896MB ?! • ZONE_NORMAL, 16 MB ~ 896 MB • ZONE_HIGHMEM, rest of them … 12/4/26 Linux - Memory Management 14
  • 15.
    Kernel View Linear – 4 GB Physical TASK_SIZE 0x0 3 GB PAGE_OFFSET ZONE_HIGHMEM 0xc0000000 0x40000000, 1GB Kernel 8 MB Image Reserved 0x37999999, 896MB mem_map 128 MB ZONE_NORMAL 1 GB … 0x01000000, 16MB 896 MB ZONE_NORMAL ZONE_DMA + ZONE_DMA 0xffffffff 12/4/26 Linux - Memory Management 15
  • 16.
    Do we needthis large?! 12/4/26 Linux - Memory Management 16
  • 17.
    SLAB Allocator • Firstintroduced in 大辞泉 SunOS 5.4 slab/slǽb/ • Implemented in Linux 名 詞 2.2 1 (石木材金属などの)広い厚板;(材木 の)背板. • SLAB system 2 (食べ物などの)厚切り – SLAB a thick slab of apple pie|厚切りのアップル パイ. – SLOB, embedded system 3⦅ 俗語⦅ 〘野球〙ピッチャーズプレート. – SLUB, efficient ver. 4⦅ ~⦅⦅ the 俗語⦅ (病院などの)死体置 き台 – SLQB, per-CPU scaling on the slab|(死体となって)安置されて. • kmalloc()! 12/4/26 Linux - Memory Management Cache !! 17
  • 18.
    Design of SLAB cache_chain slab_full …… page struct cache slab slab object Page 1 page … cache slab_partial slab (kmem_cache) object page … slab object Page 2 cache page slab_partial …… 12/4/26 Linux - Memory Management 18
  • 19.
    Per-CPU Allocation • Request/Release single page – Frequently • Per-CPU – Reducing cache invalidation – Reducing locking – A simple data array • In 2.4 kernel – get_cpu(), preemption disabled – Access array directly • In 2.6 kernel – Wrapper – percpu interface – Cache alignment 12/4/26 Linux - Memory Management 19
  • 20.
    Review App w/ glibcheap Zone Allocator Slab Allocator glibc Buddy Allocator MMU Kernel Physical Subsystems VM Subsystem Memory kswapd App w/ heap bdflush Disk Driver User Space Kernel Space Disk 12/4/26 Linux - Memory Management 20
  • 21.
    Reference • Linux KernelDevelopment, 3rd • Understanding the Linux Kernel, 3rd – http://www.makelinux.net/books/ulk3/t oc • Understanding the Virtual Memory Manager – http://www.kernel.org/doc/gorman/htm l/understand/index.html 12/4/26 Linux - Memory Management 21