Enlightening the I/O Path: A Holistic Approach for Application Performance (USENIX FAST'17)
•
0 likes•315 views
This document presents an approach called Request-centric I/O Prioritization (RCP) that holistically prioritizes I/Os along the entire I/O path to improve application performance. Existing I/O schedulers operate independently across different layers, which can cause I/O priority inversion and arbitrarily delay foreground tasks. RCP identifies critical I/Os using an enlightenment API and inherits I/O priorities through task and I/O dependencies to enforce critical I/O prioritization in caching, block allocation and queueing layers. Evaluation with PostgreSQL, MongoDB and Redis shows RCP improves throughput by 12-49% and reduces tail latencies by 5-20x compared to other schedulers.
Recommended
More Related Content
Viewers also liked
Viewers also liked (10)
Similar to Enlightening the I/O Path: A Holistic Approach for Application Performance (USENIX FAST'17)
Similar to Enlightening the I/O Path: A Holistic Approach for Application Performance (USENIX FAST'17) (20)
Recently uploaded
Recently uploaded (20)
Enlightening the I/O Path: A Holistic Approach for Application Performance (USENIX FAST'17)
- 1. Enlightening the I/O Path: A Holistic Approach for Application Performance Sangwook Kim13, Hwanju Kim2, Joonwon Lee3, and Jinkyu Jeong3 Apposha1 Dell EMC2 Sungkyunkwan University3
- 3. Data-Intensive Applications • Common structure 3 Storage Device Operating System T1 Client T2 I/O T3 T4 Request Response I/O I/O I/O Application Application performance * Example: MongoDB - Client (foreground) - Checkpointer - Log writer - Eviction worker - …
- 4. Data-Intensive Applications • Common structure 4 Storage Device Operating System T1 Client T2 I/O T3 T4 Request Response I/O I/O I/O Application Application performance * Example: MongoDB - Server (client) - Checkpointer - Log writer - Evict worker - … Background tasks are problematic for application performance
- 5. Application Impact 5 • Illustrative experiment • YCSB update-heavy workload against MongoDB
- 6. Application Impact 6 • Illustrative experiment • YCSB update-heavy workload against MongoDB 0 5000 10000 15000 20000 25000 30000 0 200 400 600 800 1000 1200 1400 1600 1800 Operationthroughput (ops/sec) Elapsed time (sec) CFQ Regular checkpoint task 30 seconds latency at 99.99th percentile
- 7. 0 5000 10000 15000 20000 25000 30000 0 200 400 600 800 1000 1200 1400 1600 1800 Operationthroughput (ops/sec) Elapsed time (sec) CFQ CFQ-IDLE Application Impact 7 • Illustrative experiment • YCSB update-heavy workload against MongoDB I/O priority does not help
- 8. 0 5000 10000 15000 20000 25000 30000 0 200 400 600 800 1000 1200 1400 1600 1800 Operationthroughput (ops/sec) Elapsed time (sec) CFQ CFQ-IDLE SPLIT-A SPLIT-D QASIO Application Impact 8 • Illustrative experiment • YCSB update-heavy workload against MongoDB State-of-the-art schedulers do not help much
- 9. What’s the Problem? • Independent policies in multiple layers • Each layer processes I/Os w/ limited information • I/O priority inversion • Background I/Os can arbitrarily delay foreground tasks 9
- 10. What’s the Problem? • Independent policies in multiple layers • Each layer processes I/Os w/ limited information • I/O priority inversion • Background I/Os can arbitrarily delay foreground tasks 10
- 11. Multiple Independent Layers • Independent I/O processing 11 Storage Device Caching Layer Application File System Layer Block Layer Abstraction
- 12. Multiple Independent Layers 12 Storage Device Caching Layer Application File System Layer Block Layer Buffer Cache read() write() admission control Abstraction • Independent I/O processing
- 13. Multiple Independent Layers • Independent I/O processing 13 Storage Device Caching Layer Application File System Layer Block Layer Buffer Cache read() write() Block-level Q admission control Abstraction
- 14. Multiple Independent Layers • Independent I/O processing 14 Storage Device Caching Layer Application File System Layer Block Layer Abstraction Buffer Cache read() write() reorder FG FG BGBG
- 15. Multiple Independent Layers • Independent I/O processing 15 Storage Device Caching Layer Application File System Layer Block Layer Abstraction Buffer Cache read() write() FG FGBG Device-internal Q admission control
- 16. Multiple Independent Layers • Independent I/O processing 16 Storage Device Caching Layer Application File System Layer Block Layer Abstraction Buffer Cache read() write() FG FGBG BG FG BGBG reorder
- 17. What’s the Problem? • Independent policies in multiple layers • Each layer processes I/Os w/ limited information • I/O priority inversion • Background I/Os can arbitrarily delay foreground tasks 17
- 18. I/O Priority Inversion • Task dependency 18 Storage Device Caching Layer Application File System Layer Block Layer Locks Condition variables
- 19. I/O Priority Inversion • Task dependency 19 Storage Device Caching Layer Application File System Layer Block Layer Condition variables I/OFG lock BG wait
- 20. I/O Priority Inversion • Task dependency 20 Storage Device Caching Layer Application File System Layer Block Layer I/OFG lock BG wait FG wait wait BGvar wake
- 21. I/O Priority Inversion • Task dependency 21 Storage Device Caching Layer Application File System Layer Block Layer I/O FG wait wait BGuser var wake FG wait
- 22. I/O Priority Inversion • I/O dependency 22 Storage Device Caching Layer Application File System Layer Block Layer Outstanding I/Os
- 23. I/O Priority Inversion • I/O dependency 23 Storage Device Caching Layer Application File System Layer Block Layer I/OFG wait For ensuring consistency and/or durability
- 24. 24 100 ms latency at 99.99th percentile 0 5000 10000 15000 20000 25000 30000 35000 0 200 400 600 800 1000 1200 1400 1600 1800 Operationthroughput (ops/sec) Elapsed time (sec) CFQ CFQ-IDLE SPLIT-A SPLIT-D QASIO RCP • Request-centric I/O prioritization (RCP) • Critical I/O: I/O in the critical path of request handling • Policy: holistically prioritizes critical I/Os along the I/O path Our Approach
- 25. Challenges • How to accurately identify I/O criticality • How to effectively enforce I/O criticality 25
- 26. Critical I/O Detection • Enlightenment API • Interface for tagging foreground tasks • I/O priority inheritance • Handling task dependency • Handling I/O dependency 26
- 27. I/O Priority Inheritance • Handling task dependency • Locks • Condition variables 27 FG lock BG I/OFG inherit BG submit complete FG BG unlock FG wait BG register BG inherit FG BGI/O submit complete wake CV CV CV
- 28. I/O Priority Inheritance • Handling I/O dependency 28
- 29. I/O Priority Inheritance • Handling I/O dependency 29 Block Layer I/OI/O
- 30. I/O Priority Inheritance • Handling I/O dependency 30 Block Layer Q admission stage I/OI/O Sched queueing stage
- 31. I/O Priority Inheritance • Handling I/O dependency 31 Block Layer PER-DEV ROOT NCIO NCIO NCIO NCIO Q admission stage I/OI/O Sched queueing stage Non-critical I/O tracking Descriptor Location Resolver Sector #
- 32. I/O Priority Inheritance • Handling I/O dependency 32 Block Layer Descriptor Location Resolver Sector # allocate Q admission stage I/O I/O I/O Sched queueing stage Non-critical I/O tracking PER-DEV ROOT NCIO NCIO NCIO NCIO
- 33. I/O Priority Inheritance • Handling I/O dependency 33 Block Layer Q admission stage I/O I/O I/O Sched queueing stage Non-critical I/O tracking update Descriptor Location Resolver Sector # PER-DEV ROOT NCIO NCIO NCIO NCIO
- 34. I/O Priority Inheritance • Handling I/O dependency 34 Block Layer Q admission stage I/O I/O Sched queueing stage I/O Non-critical I/O tracking update Descriptor Location Resolver Sector # PER-DEV ROOT NCIO NCIO NCIO NCIO
- 35. I/O Priority Inheritance • Handling I/O dependency 35 Block Layer Q admission stage I/O I/O Sched queueing stage I/O Non-critical I/O tracking update Descriptor Location Resolver Sector # PER-DEV ROOT NCIO NCIO NCIO NCIO
- 36. I/O Priority Inheritance • Handling I/O dependency 36 Block Layer Q admission stage I/O I/O Sched queueing stage I/O Non-critical I/O tracking update Descriptor Location Resolver Sector # PER-DEV ROOT NCIO NCIO NCIO NCIO
- 37. I/O Priority Inheritance • Handling I/O dependency 37 Block Layer Q admission stage I/O I/O Sched queueing stage I/O Non-critical I/O tracking Descriptor Location Resolver Sector # PER-DEV ROOT NCIO NCIO NCIO NCIO delete on completion
- 38. Handling Transitive Dependency • Possible states of dependent task 38 FG inherit BG BG Blocked on task I/OFG inherit BG wait wait Blocked on I/O FG inherit BG wait Blocked at admission stage
- 39. Handling Transitive Dependency • Recording blocking status 39 FG inherit BG BG Blocked on task I/OFG inherit BG wait wait Blocked on I/O FG inherit BG wait Blocked at admission stage
- 40. Handling Transitive Dependency • Recording blocking status 40 FG inherit BG BG Task is recorded I/OFG inherit BG wait Blocked on I/O FG inherit BG wait Blocked at admission stage inherit
- 41. Handling Transitive Dependency • Recording blocking status 41 FG inherit BG BG I/OFG inherit BG reprio I/O is recorded FG inherit BG wait Blocked at admission stage Task is recorded inherit
- 42. Handling Transitive Dependency • Recording blocking status 42 FG inherit BG BG I/OFG inherit BG FG inherit BG retryreprio I/O is recorded Task is recorded inherit
- 43. Challenges • How to accurately identify I/O criticality • Enlightenment API • I/O priority inheritance • Recording blocking status • How to effectively enforce I/O criticality 43
- 44. Criticality-Aware I/O Prioritization • Caching layer • Apply low dirty ratio for non-critical writes (1% by default) • Block layer • Isolate allocation of block queue slots • Maintain 2 FIFO queues • Schedule critical I/O first • Limit # of outstanding non-critical I/Os (1 by default) • Support queue promotion to resolve I/O dependency 44
- 45. Evaluation • Implementation on Linux 3.13 w/ ext4 • Application studies • PostgreSQL relational database • Backend processes as foreground tasks • I/O priority inheritance on LWLocks (semop) • MongoDB document store • Client threads as foreground tasks • I/O priority inheritance on Pthread mutex and condition vars (futex) • Redis key-value store • Master process as foreground task 45
- 46. Evaluation • Experimental setup • 2 Dell PowerEdge R530 (server & client) • 1TB Micron MX200 SSD • I/O prioritization schemes • CFQ (default), CFQ-IDLE • SPLIT-A (priority), SPLIT-D (deadline) [SOSP’15] • QASIO [FAST’15] • RCP 46
- 47. Application Throughput • PostgreSQL w/ TPC-C workload 47 0 1000 2000 3000 4000 5000 6000 7000 10GB dataset 60GB dataset 200GB dataset Transactionthroughput (trx/sec) CFQ CFQ-IDLE
- 48. Application Throughput • PostgreSQL w/ TPC-C workload 48 0 1000 2000 3000 4000 5000 6000 7000 10GB dataset 60GB dataset 200GB dataset Transactionthroughput (trx/sec) CFQ CFQ-IDLE SPLIT-A
- 49. Application Throughput • PostgreSQL w/ TPC-C workload 49 0 1000 2000 3000 4000 5000 6000 7000 10GB dataset 60GB dataset 200GB dataset Transactionthroughput (trx/sec) CFQ CFQ-IDLE SPLIT-A SPLIT-D
- 50. Application Throughput • PostgreSQL w/ TPC-C workload 50 0 1000 2000 3000 4000 5000 6000 7000 10GB dataset 60GB dataset 200GB dataset Transactionthroughput (trx/sec) CFQ CFQ-IDLE SPLIT-A SPLIT-D QASIO
- 51. Application Throughput • PostgreSQL w/ TPC-C workload 51 0 1000 2000 3000 4000 5000 6000 7000 10GB dataset 60GB dataset 200GB dataset Transactionthroughput (trx/sec) CFQ CFQ-IDLE SPLIT-A SPLIT-D QASIO RCP 37% 31% 28%
- 52. Application Throughput • Impact on background task 52 0 5 10 15 20 25 30 35 0 200 400 600 800 1000 1200 1400 1600 1800 Transactionlogsize(GB) Elapsed time (sec) CFQ CFQ-IDLE QASIO
- 53. Application Throughput • Impact on background task 53 0 5 10 15 20 25 30 35 0 200 400 600 800 1000 1200 1400 1600 1800 Transactionlogsize(GB) Elapsed time (sec) CFQ CFQ-IDLE SPLIT-A SPLIT-D QASIO
- 54. Application Throughput • Impact on background task 54 Our scheme improves application throughput w/o penalizing background tasks 0 5 10 15 20 25 30 35 0 200 400 600 800 1000 1200 1400 1600 1800 Transactionlogsize(GB) Elapsed time (sec) CFQ CFQ-IDLE SPLIT-A SPLIT-D QASIO RCP
- 55. Application Latency • PostgreSQL w/ TPC-C workload 55 1.E-05 1.E-04 1.E-03 1.E-02 1.E-01 1.E+00 0 1000 2000 3000 4000 5000 6000 CCDFP[X>=x] Transaction latency (msec) CFQ CFQ-IDLE SPLIT-A SPLIT-D QASIO 100 10-1 10-2 10-3 10-4 10-5 Over 2 sec at 99.9th 0th 90th 99th 99.9th 99.99th 99.999th
- 56. Application Latency • PostgreSQL w/ TPC-C workload 56 1.E-05 1.E-04 1.E-03 1.E-02 1.E-01 1.E+00 0 1000 2000 3000 4000 5000 6000 CCDFP[X>=x] Transaction latency (msec) CFQ CFQ-IDLE SPLIT-A SPLIT-D QASIO RCP 100 10-1 10-2 10-3 10-4 10-5 300 msec at 99.999th Our scheme is effective for improving tail latency Over 2 sec at 99.9th 100 10-1 10-2 10-3 10-4 10-5 0th 90th 99th 99.9th 99.99th 99.999th
- 57. Summary of Other Results • Performance results • MongoDB: 12%-201% throughput, 5x-20x latency at 99.9th • Redis: 7%-49% throughput, 2x-20x latency at 99.9th • Analysis results • System latency analysis using LatencyTOP • System throughput vs. Application latency • Need for holistic approach 57
- 58. Conclusions • Key observation • All the layers in the I/O path should be considered as a whole with I/O priority inversion in mind for effective I/O prioritization • Request-centric I/O prioritization • Enlightens the I/O path solely for application performance • Improves throughput and latency of real applications • Ongoing work • Practicalizing implementation • Applying RCP to database cluster with multiple replicas 58
- 59. Thank You! • Questions and comments • Contact • sangwook@apposha.io 59