UNIVERSITY OF WISC ONSIN–MADISON
5/17/2016 University of Wisconsin–Madison 1
Mitigating Redis Latency Issues
with Transpar...
UNIVERSITY OF WISC ONSIN–MADISON
5/17/2016 University of Wisconsin–Madison 2
UNIVERSITY OF WISC ONSIN–MADISON
5/17/2016 University of Wisconsin–Madison 3
WARNING you have Transparent Huge Pages (THP)...
UNIVERSITY OF WISC ONSIN–MADISON
4
Agenda
• Virtual Memory Refresher
• Large Pages: h/w and s/w support + benefits
• Copy-...
UNIVERSITY OF WISC ONSIN–MADISON
5
Process1Process2
Virtual Address Space
Physical Memory
Page Table
Challenge:
How to red...
UNIVERSITY OF WISC ONSIN–MADISON
6
TLB
(Translation Lookaside Buffer)
Process1Process2
Virtual Address Space
Physical Memo...
UNIVERSITY OF WISC ONSIN–MADISON
7Jim Gray, netstore99
Data Access Times
UNIVERSITY OF WISC ONSIN–MADISON
8
CPU Event Time Time
1 CPU cycle 0.3 ns 1 s
Level 1 cache access 0.9 ns 3 s
Level 2 cach...
UNIVERSITY OF WISC ONSIN–MADISON
9*Inflation-adjusted 2011 USD, from: jcmit.com
0
0
0
1
10
100
1,000
10,000
1980 1990 2000...
UNIVERSITY OF WISC ONSIN–MADISON
10
Virtual
Memory
VPN0 PFN0
TLB
Physical
Memory
UNIVERSITY OF WISC ONSIN–MADISON
11
Virtual
Memory
Physical
Memory
[Transparent Huge Pages and libhugetlbfs]
VPN0 PFN0
La...
UNIVERSITY OF WISC ONSIN–MADISON
12
4 Level Page Table
The Leaf Nodes are
Page Table Entries.
Each PTE points to a
base pa...
UNIVERSITY OF WISC ONSIN–MADISON
13RMM ISCA ‘15
Processor Dual-socket Intel Xeon E5-2430 (Sandy Bridge)
L1 Data TLB
4 KB p...
UNIVERSITY OF WISC ONSIN–MADISON
• 4KB: Baseline using 4KB paging
• THP: Transparent Huge Pages using 2MB paging
[Transpar...
UNIVERSITY OF WISC ONSIN–MADISON
15
0.00%
0.25%
0.40%
0.00%
0.14%
0.06%
0.26%
1.06%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
...
UNIVERSITY OF WISC ONSIN–MADISON
16
Fork Performance
Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
RMM ISCA ‘15
UNIVERSITY OF WISC ONSIN–MADISON
17
Operating System Support
• Linux
• Two Implementations:
• libhugetlbfs
• Transparent H...
UNIVERSITY OF WISC ONSIN–MADISON
Large Pages not the Panacea
• Alignment in Virtual and Physical Address
Spaces
• Various ...
UNIVERSITY OF WISC ONSIN–MADISON
Copy-on-Write: An Overview
Virtual
Memory
Area
Physical
Page Copy entire
page2
Change
map...
UNIVERSITY OF WISC ONSIN–MADISON
Copy
20
Physical
Page
Write
Copy entire
page2
Change
mapping
3 1
Allocate
new page
Copy-o...
UNIVERSITY OF WISC ONSIN–MADISON
Virtual
Memory
Area
Copy
21
Physical
Page Copy entire
page2
Change
mapping
3 1
Allocate
n...
UNIVERSITY OF WISC ONSIN–MADISON
Write
Copy-on-Write: An Overview
22
Virtual
Memory
Area
Physical
Page Copy entire
page2
C...
UNIVERSITY OF WISC ONSIN–MADISON
Redis: Snapshot Mechanism
23
Physical
Page
Page
Tables
Server
Process
Virtual
page
Virtua...
UNIVERSITY OF WISC ONSIN–MADISON
Redis: Latency spikes with THP
24
THP Enabled THP Disabled
892401.19 requests per second ...
UNIVERSITY OF WISC ONSIN–MADISON
Evaluation
25
Mean Latencies
UNIVERSITY OF WISC ONSIN–MADISON
Implementation Challenges
• Copy-On-Write Demotion
• Collapse by Relocation
• Maintaining...
UNIVERSITY OF WISC ONSIN–MADISONCopy-on-Write Demotion
27
Parent Virtual
Page
Physical Page Child Virtual
Page
P0
P1
P2
P3...
UNIVERSITY OF WISC ONSIN–MADISONCollapse by Relocation
28
Physical Page
P0
P1
P2
P3
P4
P5
P2’
khugepaged
Allocate
Copy
Cha...
UNIVERSITY OF WISC ONSIN–MADISON
Maintaining Memory Contiguity
29
Physical Page
P0
P1
P2
P3
P4
P5
P2
P2
Write
Change Page
...
UNIVERSITY OF WISC ONSIN–MADISON
Physical Page
In-Place Promotion
Parent Virtual
Page
P2
Child Virtual
Page
P2
Change Page...
UNIVERSITY OF WISC ONSIN–MADISON
Evaluation
31
THP No Collapse THP with Collapse
UNIVERSITY OF WISC ONSIN–MADISON
Evaluation
32
THP No Collapse THP with Collapse
UNIVERSITY OF WISC ONSIN–MADISON
Evaluation: Large Page Composition
33
Amount of Memory backed by Large Pages
UNIVERSITY OF WISC ONSIN–MADISON
Conclusion
34
• MCGA Policies: Latency Improvements at the cost
of throughput and CPU cyc...
UNIVERSITY OF WISC ONSIN–MADISON
Suggestions
35
• Try to avoid disabling THP system wide
• Use prctl (PR_SET_THP_DISABLE)
...
UNIVERSITY OF WISC ONSIN–MADISON
Suggestions
36
• Disable defrag for THP in page fault
handling path
echo never >
/sys/ker...
UNIVERSITY OF WISC ONSIN–MADISON
Future Work
37
• Short Term Goals
• Port changes to BSD and test performance
• Long Term ...
UNIVERSITY OF WISC ONSIN–MADISON
Questions ?
38
Upcoming SlideShare
Loading in …5
×

Solving Redis Latency Issues with Transparent Huge Pages: Make Copy-On-Write Great Again -Mihir Shete, Vikas Goel, Abhinav Mehra, University of Wisconsin–Madison

43 views

Published on

Most computer architectures provide support for large pages along with the normal base pages. Large pages increase TLB reach allowing applications with a bigger working set to reduce TLB misses and improve performance. However, high costs for copy-on-write faults for large pages cause Redis to face performance penalties while taking
snapshots. Hence, Redis’ documentation recommends turning off Transparent Huge Page support in Linux for low latency operation. We introduce new policies in the Linux kernel for demoting and re-promoting large pages on copy-on-write faults
to improve the latency issues seen when using Redis with Transparent Huge Pages. These policies allow Redis to reap the benefits of using large pages and reduce the latency both during snapshotting and in normal operation.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
43
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Benefits
    + Most flexible
    + Only 4KB alignment required

    Disadvantage
    ─ Requires more TLB entries
    ─ Limited TLB reach
  • Benefits
    + Increases TLB reach
    + 2MB and 1GB page size in x86-64


    Disadvantages
    ─ Size alignment restriction
    ─ Contiguity required
  • Make this animated and add points to raise.
  • Solving Redis Latency Issues with Transparent Huge Pages: Make Copy-On-Write Great Again -Mihir Shete, Vikas Goel, Abhinav Mehra, University of Wisconsin–Madison

    1. 1. UNIVERSITY OF WISC ONSIN–MADISON 5/17/2016 University of Wisconsin–Madison 1 Mitigating Redis Latency Issues with Transparent Huge Pages Mihir Shete, Vikas Goel, Abhinav Mehra, Sejal Chauhan
    2. 2. UNIVERSITY OF WISC ONSIN–MADISON 5/17/2016 University of Wisconsin–Madison 2
    3. 3. UNIVERSITY OF WISC ONSIN–MADISON 5/17/2016 University of Wisconsin–Madison 3 WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
    4. 4. UNIVERSITY OF WISC ONSIN–MADISON 4 Agenda • Virtual Memory Refresher • Large Pages: h/w and s/w support + benefits • Copy-On-Write: Overview + Case Study • Redis Latency Issues with Large Pages • Make CoW Great Again! • Conclusion • Suggestions
    5. 5. UNIVERSITY OF WISC ONSIN–MADISON 5 Process1Process2 Virtual Address Space Physical Memory Page Table Challenge: How to reduce costly page walks? Cache it! Virtual Memory Refresher
    6. 6. UNIVERSITY OF WISC ONSIN–MADISON 6 TLB (Translation Lookaside Buffer) Process1Process2 Virtual Address Space Physical Memory Page Table Virtual Memory Refresher
    7. 7. UNIVERSITY OF WISC ONSIN–MADISON 7Jim Gray, netstore99 Data Access Times
    8. 8. UNIVERSITY OF WISC ONSIN–MADISON 8 CPU Event Time Time 1 CPU cycle 0.3 ns 1 s Level 1 cache access 0.9 ns 3 s Level 2 cache access 2.8 ns 9 s Level 3 cache access 12.9 ns 43 s Main memory access 120 ns 6 min Solid-state disk I/O 50-150 μs 2-6 days Rotational disk I/O 1-10 ms 1-12 months The Infinite Space Between Words, Systems Performance: Enterprise and the Cloud Data Access Times
    9. 9. UNIVERSITY OF WISC ONSIN–MADISON 9*Inflation-adjusted 2011 USD, from: jcmit.com 0 0 0 1 10 100 1,000 10,000 1980 1990 2000 2010 Memorysize Years Memory capacity for $10,000* MBGBTB 1 1 0 10 0 1 1 0 1 0 0 1 1 0 TLB reach is limited Year Processor L1 DTLB entries 1999 Pent. III 72 2001 Pent. 4 64 2008 Nehalem 96 2012 IvyBridge 100 2015 Broadwell 100
    10. 10. UNIVERSITY OF WISC ONSIN–MADISON 10 Virtual Memory VPN0 PFN0 TLB Physical Memory
    11. 11. UNIVERSITY OF WISC ONSIN–MADISON 11 Virtual Memory Physical Memory [Transparent Huge Pages and libhugetlbfs] VPN0 PFN0 Large Page TLB Large Pages
    12. 12. UNIVERSITY OF WISC ONSIN–MADISON 12 4 Level Page Table The Leaf Nodes are Page Table Entries. Each PTE points to a base page (4KB) If PSE bit is set in the Middle Entry then it can point to a 2MB Large Page *conditions apply Virtual Address Physical Address Representing Large Pages
    13. 13. UNIVERSITY OF WISC ONSIN–MADISON 13RMM ISCA ‘15 Processor Dual-socket Intel Xeon E5-2430 (Sandy Bridge) L1 Data TLB 4 KB pages: 64-entry, 4-way associative 2 MB pages: 32-entry, 4-way associative 1 GB pages: 4-entry, fully associative L1 Instruction TLB 4 KB pages: 128-entry, 4-way associative 2 MB pages: 8-entry, fully associative L2 Data TLB 4 KB and 2 MB pages: 512-entry, 4-way associative
    14. 14. UNIVERSITY OF WISC ONSIN–MADISON • 4KB: Baseline using 4KB paging • THP: Transparent Huge Pages using 2MB paging [Transparent Huge Pages] 14
    15. 15. UNIVERSITY OF WISC ONSIN–MADISON 15 0.00% 0.25% 0.40% 0.00% 0.14% 0.06% 0.26% 1.06% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 4KB CTLB THP DS RMM 4KB CTLB THP DS RMM 4KB CTLB THP DS RMM 4KB CTLB THP DS RMM 4KB CTLB THP DS RMM cactusADM canneal graph500 mcf tigr ExecutionTimeOverhead
    16. 16. UNIVERSITY OF WISC ONSIN–MADISON 16 Fork Performance Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz RMM ISCA ‘15
    17. 17. UNIVERSITY OF WISC ONSIN–MADISON 17 Operating System Support • Linux • Two Implementations: • libhugetlbfs • Transparent Huge Pages (THP) • FreeBSD • Based on the seminal work of Navarro et al.
    18. 18. UNIVERSITY OF WISC ONSIN–MADISON Large Pages not the Panacea • Alignment in Virtual and Physical Address Spaces • Various trade-offs • Promotion • Demotion • Swapping 18
    19. 19. UNIVERSITY OF WISC ONSIN–MADISON Copy-on-Write: An Overview Virtual Memory Area Physical Page Copy entire page2 Change mapping 3 1 Allocate new page Copy-on- Write Copy Page TablesVirtual Address Space 19 Physical Address Space fork() Write Animation Courtesy: V. Seshadri, Page Overlays, ISCA `15
    20. 20. UNIVERSITY OF WISC ONSIN–MADISON Copy 20 Physical Page Write Copy entire page2 Change mapping 3 1 Allocate new page Copy-on- Write Page Tables Virtual Address Space Physical Address Space Virtual Memory Area Page Allocation Latency Copy-on-Write: An Overview Animation Courtesy: V. Seshadri, Page Overlays, ISCA `15
    21. 21. UNIVERSITY OF WISC ONSIN–MADISON Virtual Memory Area Copy 21 Physical Page Copy entire page2 Change mapping 3 1 Allocate new page Copy-on- Write Page Tables Virtual Address Space Physical Address Space Page Copy Latency Write Copy-on-Write: An Overview Animation Courtesy: V. Seshadri, Page Overlays, ISCA `15
    22. 22. UNIVERSITY OF WISC ONSIN–MADISON Write Copy-on-Write: An Overview 22 Virtual Memory Area Physical Page Copy entire page2 Change mapping 3 1 Allocate new page Copy-on- Write Copy Page Tables Virtual Address Space Physical Address Space TLB Shootdown Latency Latencies introduced per CoW operation: 1. Page Allocation Latency 2. Page Copy Latency 3. TLB Shootdown Latency (negligible) Page Size Page Allocation Page Copy 4KB 1us 1us 2MB 6us 385us Animation Courtesy: V. Seshadri, Page Overlays, ISCA `15
    23. 23. UNIVERSITY OF WISC ONSIN–MADISON Redis: Snapshot Mechanism 23 Physical Page Page Tables Server Process Virtual page Virtual page Snapshotting Process Page Tables Physical Page Snapshotting Process Completed!
    24. 24. UNIVERSITY OF WISC ONSIN–MADISON Redis: Latency spikes with THP 24 THP Enabled THP Disabled 892401.19 requests per second 607805.25 requests per second
    25. 25. UNIVERSITY OF WISC ONSIN–MADISON Evaluation 25 Mean Latencies
    26. 26. UNIVERSITY OF WISC ONSIN–MADISON Implementation Challenges • Copy-On-Write Demotion • Collapse by Relocation • Maintaining Memory Contiguity • In-place Promotion 26
    27. 27. UNIVERSITY OF WISC ONSIN–MADISONCopy-on-Write Demotion 27 Parent Virtual Page Physical Page Child Virtual Page P0 P1 P2 P3 P4 P5 P2P2 Write Change Page Mappings Allocate and Copy
    28. 28. UNIVERSITY OF WISC ONSIN–MADISONCollapse by Relocation 28 Physical Page P0 P1 P2 P3 P4 P5 P2’ khugepaged Allocate Copy Change Page Mappings P0 P1 P2’ P3 P4 P5 Parent Virtual Page New Physical Page Child Virtual Page
    29. 29. UNIVERSITY OF WISC ONSIN–MADISON Maintaining Memory Contiguity 29 Physical Page P0 P1 P2 P3 P4 P5 P2 P2 Write Change Page Mappings Allocate and Copy Faulting Page Parent Virtual Page Child Virtual Page
    30. 30. UNIVERSITY OF WISC ONSIN–MADISON Physical Page In-Place Promotion Parent Virtual Page P2 Child Virtual Page P2 Change Page Mappings Allocate and Copy Faulting Page 30 P0 P1 P2 P3 P4 P5 • Make a compound page • Remove all PTE entries • Add the PMD entry • Large page promotion completed khugepaged
    31. 31. UNIVERSITY OF WISC ONSIN–MADISON Evaluation 31 THP No Collapse THP with Collapse
    32. 32. UNIVERSITY OF WISC ONSIN–MADISON Evaluation 32 THP No Collapse THP with Collapse
    33. 33. UNIVERSITY OF WISC ONSIN–MADISON Evaluation: Large Page Composition 33 Amount of Memory backed by Large Pages
    34. 34. UNIVERSITY OF WISC ONSIN–MADISON Conclusion 34 • MCGA Policies: Latency Improvements at the cost of throughput and CPU cycles. • Maintaining memory contiguity for in-place promotion reduce CPU load but increase implementation complexity. • Not all applications benefit from the new policies.
    35. 35. UNIVERSITY OF WISC ONSIN–MADISON Suggestions 35 • Try to avoid disabling THP system wide • Use prctl (PR_SET_THP_DISABLE) • Does not need root permissions • Applies only to the calling process • Can be programmatically controlled
    36. 36. UNIVERSITY OF WISC ONSIN–MADISON Suggestions 36 • Disable defrag for THP in page fault handling path echo never > /sys/kernel/mm/transparent_hugepages/defrag
    37. 37. UNIVERSITY OF WISC ONSIN–MADISON Future Work 37 • Short Term Goals • Port changes to BSD and test performance • Long Term Goals • Work on Memory Contiguity for large memory systems
    38. 38. UNIVERSITY OF WISC ONSIN–MADISON Questions ? 38

    ×