Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
High Load on
Alibaba MySQL
Xiaobin Lin
About Me
• Xiaobin Lin
• RDS @ Aliyun, Alibaba
• Database source code team
• Work around MySQL more than 7 years
Catalog
• hotspot updating
• unexpected long transaction
• high concurrency
• slave latency
• thread holding
• performance...
hotspot updating
• Problem
• reduce stock of hot commodities
• hotspot updating on a single InnoDB rows
• dead lock detect...
hotspot updating
0
5000
10000
15000
20000
1 12 48 192 768
tps
tps
Thread tps
1 6100
6 19160
12 14000
24 10000
48 9800
96 7...
hotspot updating
Dead lock detection
hotspot updating
0
5000
10000
15000
20000
1 6 12 24 48 96 192 384 768 1536
TPS1 TPS2
Disable dead lock detection
hotspot updating
• Solution: Queue on
primary key
hotspot updating
0
5000
10000
15000
20000
1 6 12 24 48 96 192 384 768 1536
TPS1 TPS2
TPS3
unexpected long transaction
simplest transaction
begin;
update table_name set …… where id=N;
commit;
unexpected long transaction
• Problem
• unexpected long transaction on high
load
• long time waiting for “commit“
• busy c...
hotspot updating
• Last Case: Queue on primary
key
unexpected long transaction
• Solution
• Commit/rollback transactions as early as
possible
• start transaction;
update
/*!...
high concurrency
• Problem
• more than 5w active thread
• cpu cost on switch
• lot of timeout in client
• out of MySQL’s c...
high concurrency
• Solution: actions on Low/High level
watermark
Slave latency
• Problem
• High load on master means the same
on slave
• speed of applying event on slave VS
generating eve...
Slave latency
• Solution
• multi-thread applying event, based on
table and row
• relay fetch
• bulk apply
Thread holding
• Problem
• Version upgrade / machine maintenance
need to shutdown master
• user connection broken
Thread holding
• Solution
• proxy layer
• thread holding
performance enhancement
• async redo log copy and write
• innodb redo crc32 checksum
• reuse dummy index on change buffer
...
Q&A
HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)
HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)
HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)
Upcoming SlideShare
Loading in …5
×

HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)

1,351 views

Published on

In Alibaba, almost all of the busiest business are builded on MySQL. It aquires MySQL can support high load. The challenge comes from sucn as complex transactions, huge number of parallel connections and press on network, cpu, memory and disk.

To deal with the high press, we have some solutions on database level, such as some patches on MySQL source and the relevant change in application code.

This topics will discuss about the solutions, focus on what problems we met, how solutions were designed and the result of them.

Published in: Engineering
  • Be the first to comment

HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)

  1. 1. High Load on Alibaba MySQL Xiaobin Lin
  2. 2. About Me • Xiaobin Lin • RDS @ Aliyun, Alibaba • Database source code team • Work around MySQL more than 7 years
  3. 3. Catalog • hotspot updating • unexpected long transaction • high concurrency • slave latency • thread holding • performance enhancement
  4. 4. hotspot updating • Problem • reduce stock of hot commodities • hotspot updating on a single InnoDB rows • dead lock detecting
  5. 5. hotspot updating 0 5000 10000 15000 20000 1 12 48 192 768 tps tps Thread tps 1 6100 6 19160 12 14000 24 10000 48 9800 96 7000 192 3600 384 2000 768 600 1536 150
  6. 6. hotspot updating Dead lock detection
  7. 7. hotspot updating 0 5000 10000 15000 20000 1 6 12 24 48 96 192 384 768 1536 TPS1 TPS2 Disable dead lock detection
  8. 8. hotspot updating • Solution: Queue on primary key
  9. 9. hotspot updating 0 5000 10000 15000 20000 1 6 12 24 48 96 192 384 768 1536 TPS1 TPS2 TPS3
  10. 10. unexpected long transaction simplest transaction begin; update table_name set …… where id=N; commit;
  11. 11. unexpected long transaction • Problem • unexpected long transaction on high load • long time waiting for “commit“ • busy client • network • rowlock causes problem
  12. 12. hotspot updating • Last Case: Queue on primary key
  13. 13. unexpected long transaction • Solution • Commit/rollback transactions as early as possible • start transaction; update /*! commit_on_success rollback_on_fail*/ table_name set ……
  14. 14. high concurrency • Problem • more than 5w active thread • cpu cost on switch • lot of timeout in client • out of MySQL’s capability
  15. 15. high concurrency • Solution: actions on Low/High level watermark
  16. 16. Slave latency • Problem • High load on master means the same on slave • speed of applying event on slave VS generating event on master
  17. 17. Slave latency • Solution • multi-thread applying event, based on table and row • relay fetch • bulk apply
  18. 18. Thread holding • Problem • Version upgrade / machine maintenance need to shutdown master • user connection broken
  19. 19. Thread holding • Solution • proxy layer • thread holding
  20. 20. performance enhancement • async redo log copy and write • innodb redo crc32 checksum • reuse dummy index on change buffer • remove read-only transaction list • cache transaction read-view • reduce overload of read-only transaction • split lock_grant lock • …… • 80% improvement on TPS/QPS
  21. 21. Q&A

×