XtraDB 5.6: Key Performance Algorithms / PLMCE 2014

620 views
448 views

Published on

Percona Live 2014 presentation

https://www.percona.com/live/mysql-conference-2014/sessions/xtradb-56-key-performance-algorithms

Published in: Software
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
620
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

XtraDB 5.6: Key Performance Algorithms / PLMCE 2014

  1. 1. XtraDB 5.6 Key Performance Algorithms Laurynas Biveinis Alexey Stroganov 2014-04-02
  2. 2. Agenda • Background: InnoDB and XtraDB in 5.5 • XtraDB 5.6 mk I: InnoDB with trx descriptors • XtraDB 5.6 mk II: 5.5 performance patches revisited • XtraDB 5.6 mk III: flushing rework • XtraDB 5.6: the rest and benchmarks 2
  3. 3. 5.5 • InnoDB 5.5 & XtraDB 5.5 3
  4. 4. InnoDB 5.5: Buffer Pool 4 Flush list: … LRU list: … Free list: … Page hash: flush list mutex buffer pool mutex Accessed on: Page dirtying. Master thread checkpoint age flush Page access. Query thread flush to make space in buffer pool Flush to make space in buffer pool Page lookup
  5. 5. MySQL solution for the contention 5 Flush list 1: … LRU list 1: … Free list 1: … Page hash 1: flush list mutex 1 buffer pool mutex 1 Flush list 2: … LRU list 2: … Free list 2: … Page hash 2: flush list mutex 2 buffer pool mutex 2
  6. 6. XtraDB BP mutex split solution 6 Flush list: … LRU list: … Free list: … Page hash: flush list mutex LRU list mutex free list mutex page hash latch
  7. 7. The solutions may be combined 7 Flush list: … LRU list 1: … Free list 1: … Page hash 1: Flush list: … LRU list 2: … Free list 2: … Page hash 2:
  8. 8. InnoDB 5.5: trx list 8 1. TRX_ACTIVE kernel_mutex: 2. TRX_ACTIVE 3. TRX_COMMITTED_IN_MEMORY trx_sys->trx_list: … Create Read View list scan view->trx_ids: 1 2 …
  9. 9. XtraDB 5.5: trx descriptors 9 kernel_mutex: trx_sys->descriptorsCreate Read view memcpy view->descriptors: 1 2 … 1 2 • http://bugs.mysql.com/bug.php?id=49169 • http://bit.ly/pstrx1 • http://bit.ly/pstrxdesc2
  10. 10. InnoDB 5.5: AHI 10 4 2 3 . 7 1 5 64 8 97 . . t1 d b c . g a e fd h jg . . t2 t1.3 t1.5 t1.7 t2.c t2.g btr_search_sys-> hash_index btr_search_latch:
  11. 11. XtraDB 5.5: AHI partitions 11 4 2 3 . 7 1 5 64 8 97 . . t1 d b c . g a e fd h jg . . t2 t1.3 t1.5 t1.7 t2.c t2.g btr_search_sys-> hash_tables[0] btr_search_latch[0]: btr_search_latch[1]: btr_search_sys-> hash_tables[1]
  12. 12. XtraDB 5.6 mk I • Can we drop the scalability patches? – InnoDB 5.6 introduced cleaner thread, http://bit.ly/ibclean – What it does: ● Flush LRU tail, somehow, innodb_lru_scan_depth ● Flush flush list, adaptively, innodb_io_capacity[_max] ● Sleep until 1 s completes ● Goto 1 – http://bit.ly/markexplainslru ● We knew we could not drop the trx descriptor patch 12
  13. 13. XtraDB 5.6 mk I: InnoDB / trx desc 13
  14. 14. XtraDB 5.6 mk I: InnoDB / trx desc 14
  15. 15. XtraDB 5.6 mk I: InnoDB / trx desc 15 • Why XtraDB 5.6 mk I scales worse than XtraDB 5.5? – AHI contention ● OK, should have ported it too at once, no changes there in 5.6 ● http://bugs.mysql.com/bug.php?id=62018 – Buffer pool mutex contention ● No problem, we have a patch, right?
  16. 16. XtraDB 5.6 mk II: 5.5 patches • Ported the AHI partition patch – Improved it during the port. http://bugs.mysql.com/bug.php?id=70216 – Worked as expected • Ported the buffer pool mutex split –Attempted to improve it during the port too –InnoDB 5.6 split off page hash latch –And it did not help much/at all 16
  17. 17. BP Access Concurrency Analysis, 5.5 17 master threadtime flush list flush flush list flush flush list flush query thread 1 Make page young Make page young LRU list flush query thread 2 LRU list flush Make page young Make page young
  18. 18. BP Access Concurrency Analysis, 5.6 18 cleaner threadtime LRU list flush flush list flush LRU list flush query thread 1 Make page young Make page young LRU list flush query thread 2 LRU list flush Make page young Make page young flush list flush
  19. 19. XtraDB 5.6 mk III: flushing rework • Issue 1: cleaner runs the flushing, but does the cleaner itself run? • One of the reasons innodb_thread_concurrency helps... 19 100% CPU run time share few query threads cleaner many query threads c
  20. 20. XtraDB 5.6 mk III: flushing rework • Issue 1 solution: increase the cleaner priority 20 100% CPU run time share few query threads cleaner many query threads cleaner
  21. 21. XtraDB 5.6 mk III: flushing rework • setpriority() • --innodb_sched_priority_cleaner=n 21 100% CPU run time share few query threads cleaner many query threads cleaner
  22. 22. XtraDB 5.6 mk III: flushing rework 22
  23. 23. XtraDB 5.6 mk III: flushing rework • Issue 2: What happens in the case of an empty free list in some instance? • Query thread: – Lock (buffer pool|free list) mutex – If free list empty, unlock the mutex, do something (*), goto 1 – * - query thread eviction / LRU flush attempt, sleep before looping 23
  24. 24. XtraDB 5.6 mk III: flushing rework • Issue 2. Still no free page, so given enough query threads... – Thread 1 waiting for the mutex – Thread 2 has the mutex, checks for empty free list – Thread n waiting for the mutex 24
  25. 25. XtraDB 5.6 mk III: flushing rework • Issue 2. Cleaner produced a free page! – Thread 1 waiting for the mutex – Thread 2 has the mutex, checks for empty free list – Thread n waiting for the mutex – Cleaner thread waiting for the mutex 25
  26. 26. XtraDB 5.6 mk III: flushing rework • Issue 2. Cleaner produced a free page, so what? – Thread 2 unlocked the mutex – Thundering herd of 1, 3..n, and cleaner threads, mutex_signal_object() – The chance that cleaner thread gets the mutex is roughly 1 / n – No mutex, no free page, n likely to increase 26
  27. 27. XtraDB 5.6 mk III: flushing rework • Issue 2. Solution: obviously, the cleaner must get the free list mutex before anybody else • Adapted the sync array code • mutex->high_priority_event • mutex_enter_first() 27
  28. 28. XtraDB 5.6 mk III: flushing rework • Issue 1+2: what applies to cleaner, applies to other background threads as well – Under high concurrency, background threads must have priority over query threads to keep the server in the steady state – State is: checkpoint age, history length, ... 28
  29. 29. XtraDB 5.6 mk III: flushing rework • Issue 3. Why do query threads try to (LRU|flush list) flush anyway? • Because the cleaner thread failed to do so in time • Is it a good idea to increase mutex pressure if the cleaner thread is already in trouble? 29
  30. 30. XtraDB 5.6 mk III: flushing rework • Issue 3 solution: do not let query threads flush • --innodb_empty_free_list_algorithm =backoff • --innodb_foreground_preflush =exponential_backoff • http://bit.ly/noqueryflush 30
  31. 31. XtraDB 5.6 mk III: flushing rework 31
  32. 32. XtraDB 5.6 mk III: flushing rework • Issue 4: why do we have empty free lists in the first place? • mysql -uroot -e'show engine innodb statusG'| grep Free • Free buffers 1142 • Free buffers 1235 • Free buffers 1288 • Free buffers 318 • Free buffers 279 • Free buffers 938 • Free buffers 0 • Free buffers 406 32 • Free buffers 312 • Free buffers 687 • Free buffers 696 • Free buffers 910 • Free buffers 1029 • Free buffers 993 • Free buffers 537
  33. 33. XtraDB 5.6 mk III: flushing rework • Issue 4: buffer pool instance usage is not uniform! – Mikael Ronstrom: “First the accesses to the buffer pools is in no way evenly spread out.” – http://bit.ly/bpsplit ● But 5.6 flushing does not recognize that! – LRU/flush list flush request for x pages across y instances: ● Request x/y flush for each instance ● Assuming uniformity where there is none 33
  34. 34. XtraDB 5.6 mk III: flushing rework • Current issue 4 solution attempt – “pseudoparallel” flushing: instead of issuing all requests to each instance before advancing to the next one, issue requests across instances – keep flushing an instance in a loop if its free list nearly empty – innodb_free_list_lwm, default 100, UNIV_PERF_DEBUG tuning var 34
  35. 35. XtraDB 5.6 mk III: flushing rework • Proper issue 4 solution: parallel LRU flushers for each instance – Future work... – 5.7 MT flushing is a step to right direction but it must be ensured that it does the right thing 35
  36. 36. XtraDB 5.6 mk III: flushing rework • Talking of parallel LRU flushers... • Busy cleaner thread: 36 flush list flush LRU list flush flush list flush LRU list flush ... Empty free list danger here Max checkpoint age danger here
  37. 37. XtraDB 5.6 mk III: flushing rework • Solution is separate flush list and LRU flush threads 37 flush list flush LRU list flush flush list flush LRU list flush flush list flush
  38. 38. XtraDB 5.6: the rest • Last but not least: – Adaptive flushing formula tweak to maintain higher checkpoint age – Efficient log block checksums – http://bugs.mysql.com/bug.php? id=70453 38
  39. 39. XtraDB 5.6 39

×