Transparent Hugepages in RHEL 6

24,439 views

Published on

Introduction to linux memory management and advantages of using huge pages for certain applications, and final wrap-up with benefits of Transparent hugepages available in RHEL 6.

Published in: Technology, Education
2 Comments
36 Likes
Statistics
Notes
No Downloads
Views
Total views
24,439
On SlideShare
0
From Embeds
0
Number of Embeds
79
Actions
Shares
0
Downloads
0
Comments
2
Likes
36
Embeds 0
No embeds

No notes for slide

Transparent Hugepages in RHEL 6

  1. 1. Transparent Hugepages in RHEL 6 Raghu Udiyar 22 July 2011
  2. 2. Transparent Huge Pages <ul><li>Written by Andrea Arcangeli (Red Hat kernel developer)
  3. 3. Merged into upstream kernel 2.6.38
  4. 4. Available in RHEL 6.0 on GA </li></ul>
  5. 5. <ul><li>A page is the smallest unit of memory that can be manipulated
  6. 6. Memory is divided into pages of equal size </li></ul>Transparent Huge Pages Pages Memory
  7. 7. Page : Memory Management <ul><li>Page size varies with the architecture, x86 = 4KiB
  8. 8. For other arch's : ia64 = 8KiB, ppc64 = 4,8 and 64KiB </li></ul>4KiB Page x86
  9. 9. Page : Memory Management Physical Memory <ul><li>Aka : RAM – memory installed on the system e.g. 2GiB
  10. 10. Programs never work directly with physical memory
  11. 11. A physical page is called a Page Frame </li></ul>Virtual Memory <ul><li>Not real memory : 32bit – 4Gb, 64bit – more then 128T
  12. 12. Programs always allocate virtual memory
  13. 13. A virtual page is called simply a Page </li></ul>
  14. 14. Page : Virtual and Physical <ul><li>Each page is mapped to a page frame using tables </li></ul>Physical Virtual Page Page Frame
  15. 15. Page : Memory Management Process 2 Process 1
  16. 16. Page : Memory Management <ul><li>Virtual to Physical address translations are stored in page tables
  17. 17. These page tables are stored in memory
  18. 18. Looking up these page tables is a processes called a
  19. 19. Page Walk </li></ul>
  20. 20. Page : Simplified view of the Page Table Virtual address <ul>Multiple tables are involved in a Page Walk </ul>
  21. 21. Page : Memory Management <ul><li>Every memory access requires a Page Walk
  22. 22. Hardware assisted but it is still expensive
  23. 23. Typically, takes 10-100 cpu clock cycles
  24. 24. The obvious optimization is to cache the results </li></ul>
  25. 25. Page : TLB - Translation look-aside buffer TLB caches Virtual->Physical Translations <ul><ul><li>Check TLB first
  26. 26. If TLB hit, return address
  27. 27. If TLB miss, do a Page Walk and store translation in TLB </li></ul></ul><ul><ul><li>Using TLB, increases performance by as much as 15%
  28. 28. Look up takes 0.5 – 1 clock cycl e </li></ul></ul>
  29. 29. Page : TLB - Translation look-aside buffer <ul><li>CAM (content addressable memory) is used that is quite expensive to manufacture and have other limitations
  30. 30. But they are very fast
  31. 31. Typically, a TLB can only hold 8 to 1024 entries
  32. 32. Plus, TLB cache has to be flushed on every context switch
  33. 33. Therefore, TLB is a scarce resource and every attempt has to be made to maximize its usage </li></ul>
  34. 34. Page : Large Memory Applications <ul><li>Applications touching greater then 10Gb
  35. 35. 4KiB page size produces a large overhead to manage these pages
  36. 36. Example :
  37. 37. 17GiB Oracle SGA with 1000 Connection : </li><ul><li>~4.4 million pages
  38. 38. Page table size : 16GiB </li></ul><li>TLB can only hold at most 1024 entries
  39. 39. BIG performance penalty </li></ul>
  40. 40. Page : Questions? <ul><li>Q1 : How can page walk work, if we don't know where the page tables are (as they are stored in memory) ?
  41. 41. Ans : The address of the root table is stored in a special CR3 register. </li></ul>
  42. 42. <ul>Huge Pages are pages of larger sizes e.g. on x86 it's 2MiB </ul>Transparent Huge Pages X 512 4 KiB 2 MiB
  43. 43. Huge Pages : Benefits <ul><li>Now each TLB entry can effectively store 512 normal pages
  44. 44. Therefore, Increases TLB hit ratio
  45. 45. Decrease in number of pages to manage
  46. 46. Previous Oracle example :
  47. 47. Using huge pages : </li><ul><li>17GiB SGA uses 17*1024/2 = 8704 Huge Pages (compared to ~4.4 million! Normal pages)
  48. 48. Page Table size? ~33MiB (1000 processes)
  49. 49. (compared to 16GiB) </li></ul><li>Huge improvement! </li></ul>
  50. 50. Huge Pages : Configuration <ul><li>Set “vm.nr_hugepages = 10” in /etc/sysctl.conf
  51. 51. To verify :
  52. 52. # cat /proc/meminfo | grep Huge
  53. 53. HugePages_Total: 10
  54. 54. HugePages_Free: 10
  55. 55. HugePages_Rsvd: 0
  56. 56. Hugepagesize: 2048 kB
  57. 57. A reboot may be required if not enough contiguous physical pages are available.
  58. 58. Refer to : https://access.redhat.com/kb/docs/DOC-56553 </li></ul>
  59. 59. Huge Pages : Issues <ul><li>Huge pages have to be reserved beforehand
  60. 60. Not available for normal applications
  61. 61. Explicit application support is required
  62. 62. Cannot be swapped to disk – locked in memory
  63. 63. Only really used by large database applications, such as Oracle and Sybase </li></ul>
  64. 64. Huge Pages : Questions? <ul><li>Why can't we have all normal pages as huge pages i.e. Increase from 4KiB to 2MiB
  65. 65. Ans : This causes fragmentation. </li><ul><ul><li>Small page size causes large management overhead
  66. 66. Large page size causes internal and external fragmentation </li></ul></ul><li>What we need is a best of both worlds </li></ul>
  67. 67. <ul><li>This builds up on the limitations of normal Huge pages
  68. 68. THP is enabled by default for all applications
  69. 69. Transparent to the application – No configuration required
  70. 70. All applications benefit from using Huge pages
  71. 71. Up to 10% performance improvement
  72. 72. Therefore, we successfully have a mixed or hybrid page size scenario </li></ul>Transparent Huge Pages
  73. 73. THP: How does it work? <ul><li>Hugepages are allocated whenever possible
  74. 74. Example : 2MiB memory requests
  75. 75. THP are swappable, by breaking a hugepage to 4KiB pages
  76. 76. A kernel thread Khugepaged will scan running processes and attempt to switch normal pages with Hugepages
  77. 77. THP tries to maximizes Huge page use
  78. 78. Kernel itself maps it's memory using hugepages </li></ul>
  79. 79. THP: Work in Progress <ul><li>Only enabled for anonymous memory, for now
  80. 80. Page cache and shared memory still in progress
  81. 81. To check THP usage on RHEL 6 or F14 and above :
  82. 82. # grep AnonHugePages /proc/meminfo
  83. 83. AnonHugePages: 632832 kB </li></ul>
  84. 84. THP : Questions? <ul><li>Refer for more details :
  85. 85. https://access.redhat.com/kb/docs/DOC-49562 </li></ul>
  86. 86. Thank You Raghu Udiyar 22 July 2011

×