Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[Pgday.Seoul 2018] PostgreSQL 성능을 위해 개발된 라이브러리 OS 소개 apposha

85 views

Published on

[Pgday.Seoul 2018] PostgreSQL 성능을 위해 개발된 라이브러리 OS 소개 apposha - 김상욱

Published in: Software
  • Be the first to comment

  • Be the first to like this

[Pgday.Seoul 2018] PostgreSQL 성능을 위해 개발된 라이브러리 OS 소개 apposha

  1. 1. PostgreSQL 성능을 위해 개발된 라이브러리 OS 소개
  2. 2. Special application on top of general OS (Linux) Problem 2 Linux PostgreSQL Hardware General purpose: Web server, office, game, … Bottleneck for DB performance Special need: ACID, high concurrency, fast data access…
  3. 3. AppOS Solution Linux PostgreSQL Hardware Specialized for DB performance Special need: ACID, high concurrency, fast data access… Special application on top of special OS (AppOS) 3
  4. 4. Solution V1 Special application on top of special OS (AppOS) Modified Linux PostgreSQL Hardware V2 Linux PostgreSQL Hardware Module Module V3 Linux PostgreSQL Hardware AppOS 4
  5. 5. Merits 5 Portable Performant Extensible
  6. 6. AppOS provides system call APIs PostgreSQL is built on top of Linux system calls libhook libapp libcore syscall PostgreSQL AppOS AppOS seamlessly runs by hooking system calls open() read()... libhook libapp libcore PostgreSQL AppOS open() read()... 6 AppOS doesn’t require modifications Technology: Portability
  7. 7. Technology: Portability AppOS is highly portable because it is built on top of Linux ABIs LinuxLinux Linux LinuxLinux 7 AppOS runs in diverse environments
  8. 8. Technology: Performance Virtual File System libcore libapp libhook AppOS Page Cache Physical File Mapping mongodbpostgresql Memory Mgmt Utility … I/O Engine • libapp • Per-DB optimizations • libcore • DB-common optimizations 8 AppOS consists of user-level libraries
  9. 9. Technology: Performance 9 Linux doesn’t know DB-internal semantics
  10. 10. Technology: Performance AppOS infers semantics by observing system call patterns 10 AppOS understands DB-internal semantics
  11. 11. Technology: Performance write() write() T2 Linux File System MetadataFile Journal Linux Page Cache T1 Page T3 T4 rename() T5 write() write() Only one thread can write to a file Complex locking behaviors for file system consistency 11 Linux imposes unnecessary overheads for file accesses
  12. 12. Technology: Performance write() write() T2 Linux File System AppOS Page Cache T1 Page T4 rename() T5 write() Multiple threads can concurrently write to different positions of a file Simple locking based on data consistency guarantees provided by DB Page Asynchronous direct I/Os by I/O workers 12 AppOS decouples page cache from file system
  13. 13. Technology: Performance 13 Linux I/O path delays high-priority disk I/Os Storage Device Caching Layer Application File System Layer Block Layer Abstraction
  14. 14. Technology: Performance 14 Linux I/O path delays high-priority disk I/Os Storage Device Caching Layer Application File System Layer Block Layer Abstraction Buffer Cache read() write() admission control
  15. 15. Technology: Performance 15 Linux I/O path delays high-priority disk I/Os Storage Device Caching Layer Application File System Layer Block Layer Abstraction Buffer Cache read() write() admission control Block-level Q
  16. 16. Technology: Performance 16 Linux I/O path delays high-priority disk I/Os Application Storage Device Caching Layer File System Layer Block Layer Abstraction Buffer Cache reorder FG FG BGBG read() write()
  17. 17. Technology: Performance 17 Linux I/O path delays high-priority disk I/Os Storage Device Caching Layer Application File System Layer Block Layer Abstraction Buffer Cache read() write() FG FGBG Device-internal Q admission control
  18. 18. Technology: Performance 18 Linux I/O path delays high-priority disk I/Os Storage Device Caching Layer Application File System Layer Block Layer Abstraction Buffer Cache read() write() FG FGBG BG FG BGBG reorder
  19. 19. Technology: Performance 19 Linux I/O path delays high-priority disk I/Os Storage Device Caching Layer Application File System Layer Block Layer Locks Condition variables
  20. 20. Technology: Performance 20 Linux I/O path delays high-priority disk I/Os Storage Device Caching Layer Application File System Layer Block Layer Condition variables I/OFG lock BG wait
  21. 21. Technology: Performance 21 Linux I/O path delays high-priority disk I/Os Storage Device Caching Layer Application File System Layer Block Layer I/OFG lock BG wait FG wait wait BGvar wake
  22. 22. Technology: Performance 22 Linux I/O path delays high-priority disk I/Os Storage Device Caching Layer Application File System Layer Block Layer I/O FG wait wait BGuser var wake FG wait
  23. 23. Technology: Performance AppOS libapp Storage (I/O class, share)(latency, 1000) (throughput, 500)(latency, 1000) AppOS I/O Engine WAL write Foreground data read Background read/write (e.g., checkpoint, compaction) Dispatch I/Os based on priority only if there is no storage congestion 23 AppOS schedules I/Os based on DB-internal priority
  24. 24. Technology: Performance Linux Cache PostgreSQL Cache Block 1 Data block Block 2 Storage 1. Crash happens Block 1 Block 2 Torn page problem Block 1 Data block Block 2 2. Crash happens Block 1 Block 2 WAL Full page write 1. Write block to WAL file 3. Recover upon reboot 2. Data block is corrupted 24 Linux doesn’t support atomic write 2X Write
  25. 25. Technology: Performance AppOS manages file mapping for eliminating write amplification 25 AppOS supports atomic write
  26. 26. Technology: Performance 26 Benchmarks show better performance for OLTP * Host - AWS ec2 - c4.8xlarge - 36 cores, 60GB mem - 1000GB SSD-backed EBS - PostgreSQL 10.3 - 24GB shared buffers - 10GB max wal * Client - AWS ec2 - c4.xlarge - 4 cores, 7.5GB mem - OLTP-Bench - TPC-C with 200 scale factor (전자상거래 트랜잭션 처리)
  27. 27. Technology: Performance 27 Benchmarks show better performance for OLTP * Host - AWS ec2 - c4.8xlarge - 36 cores, 60GB mem - 1000GB SSD-backed EBS - PostgreSQL 10.3 - 24GB shared buffers - 10GB max wal * Client - AWS ec2 - c4.xlarge - 4 cores, 7.5GB mem - OLTP-Bench - TPC-C with 200 scale factor (전자상거래 트랜잭션 처리)
  28. 28. Technology: Extensibility PostgreSQLMongoDB : 70 syscalls : 74 syscalls Common : 50 syscalls 28 Used system calls are similar from DB to DB
  29. 29. Technology: Extensibility PostgreSQL 9.5 PostgreSQL 9.6 PostgreSQL 10.0 poll() -> epoll() semop() -> futex() 29 Used system calls are not changed a lot
  30. 30. Technology: Extensibility libcore libapp libhook AppOS postgresql PostgreSQL libcore libapp libhook AppOS mongodb MongoDB libcore libapp libhook AppOS mysql MySQL libcore libapp libhook AppOS redis Redis ... 30 AppOS can easily support various databases
  31. 31. Roadmap 31 Efficient cache management Linux Cache PostgreSQL Cache Block 1 Block 2 Storage read Block 1 Block N Duplicated LRU Block 1 Block 2 Block 1 Block N DB cache-aware Block 1 next victim read Block 3 Block 1 Block 3 next victimread read
  32. 32. Roadmap 32 Parallel query optimization
  33. 33. Roadmap 33 Parallel query optimization
  34. 34. Roadmap 34 Parallel query optimization
  35. 35. Roadmap 35 Cooperative CPU scheduling
  36. 36. Vision 36 Support diverse types of applications
  37. 37. https://apposha.io sangwook@apposha.io

×