0
High-Availability MySQL with DR:BD and Heartbeat: MTV Japan mobile services




©2008 MTV Networks Japan K.K.
Patrick Bolduan
                                      patrick.bolduan@mtvn.jp




                                   Syste...
Let’s get started


©2008 MTV Networks Japan K.K.
About MTVNJ
                                Company
                                Profile
                              ...
MTVNJ Mobile Brands




©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project intro




©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                        DB Selection




                                  ...
MTV Flux/MTV Mobile DB project
                                        DB Selection


            Project summary:
       ...
MTV Flux/MTV Mobile DB project
                                          DB Selection

                                   ...
MTV Flux/MTV Mobile DB project
                                        DB Selection

                                   We...
MTV Flux/MTV Mobile DB project
                                     MySQL Architectures

                                M...
MTV Flux/MTV Mobile DB project
                                         MySQL Architectures

            Single/multi Mast...
MTV Flux/MTV Mobile DB project
                                      MySQL Architectures

            Master/Slave
       ...
MTV Flux/MTV Mobile DB project
                                              MySQL Architectures

             Cluster
   ...
MTV Flux/MTV Mobile DB project
                                      MySQL Architectures

            DR:BD/Heartbeat HA M...
MTV Flux/MTV Mobile DB project
                                MySQL HA Master




©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                                HA Master

                                ...
MTV Flux/MTV Mobile DB project
                                              HA Master

            Application           ...
MTV Flux/MTV Mobile DB project
                                                HA Master

                                ...
MTV Flux/MTV Mobile DB project
                                                HA Master

                                ...
MTV Flux/MTV Mobile DB project
              HA MySQL install




©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                              HA Master Install


  Old DB
  architecture

...
MTV Flux/MTV Mobile DB project
                                          HA Master Install


   New DB
   architecture
   ...
MTV Flux/MTV Mobile DB project
                                       HA Master Install

           Database migration wor...
MTV Flux/MTV Mobile DB project
                                                HA Master Install



  Migration           ...
MTV Flux/MTV Mobile DB project
                                       HA Master Install

           Actual migration work ...
MTV Flux/MTV Mobile DB project
                                HA MySQL lessons learned




©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                      Lessons learned


        Impact on our application:
...
MTV Flux/MTV Mobile DB project
                                      Lessons learned


        mysqld:
        • can take ...
MTV Flux/MTV Mobile DB project
                                      Lessons learned

        Overall – good points:
     ...
HA MySQL in other MTVNJ environments




©2008 MTV Networks Japan K.K.
Questions?




     DRBD related links
     http://www.drbd.org/users-guide/s-heartbeat-r1.html
     http://www.drbd.org/u...
Upcoming SlideShare
Loading in...5
×

High Availability MySQL with DRBD and Heartbeat MTV Japan Mobile Service

1,313

Published on

这应该是08年mysql大会上的一个ppt,
从scribd.com上老叶yejr那挖过来的,
scribd已被▄︻┻┳═一……

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,313
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
20
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "High Availability MySQL with DRBD and Heartbeat MTV Japan Mobile Service"

  1. 1. High-Availability MySQL with DR:BD and Heartbeat: MTV Japan mobile services ©2008 MTV Networks Japan K.K.
  2. 2. Patrick Bolduan patrick.bolduan@mtvn.jp System Manager MTV Networks Japan K.K. ©2008 MTV Networks Japan K.K.
  3. 3. Let’s get started ©2008 MTV Networks Japan K.K.
  4. 4. About MTVNJ Company Profile Company Name MTV Networks Japan K.K. President Peter Robert Bullard Address 2-8-2 Jingumae, Shibuya-ku, Tokyo 150-0001 TEL: 03-6434-3111 FAX: 03-6434-3150 Business MTV Networks Japan K.K. (MTVNJ) operates the mobile and PC Description digital media services for all MTV Networks brands including MTV and Nickelodeon in Japan. In addition to digital media services, MTVNJ also operates a 24-hour Nickelodeon channel on CS/CATV, and the consumer products business for Nickelodeon's character franchises in Japan. ©2008 MTV Networks Japan K.K.
  5. 5. MTVNJ Mobile Brands ©2008 MTV Networks Japan K.K.
  6. 6. MTV Flux/MTV Mobile DB project intro ©2008 MTV Networks Japan K.K.
  7. 7. MTV Flux/MTV Mobile DB project DB Selection {database} ©2008 MTV Networks Japan K.K.
  8. 8. MTV Flux/MTV Mobile DB project DB Selection Project summary: Merge all the database services for the MTV Flux and MTV Mobile services on to a single database platform. Database requirements: • flexibility • performance • scalability • redundancy • support • cost ©2008 MTV Networks Japan K.K.
  9. 9. MTV Flux/MTV Mobile DB project DB Selection MySQL MySQL reqs/DB Oracle MS SQL Community Enterprise flexibility △/ × × ○ ○ performance ○ ○ ○ ○ scalability ○ /△ ○ ○ ○ redundancy ○ /△ ○ ○ ○ support ○ ○ × ○ cost × × ◎ △ ©2008 MTV Networks Japan K.K.
  10. 10. MTV Flux/MTV Mobile DB project DB Selection We chose MySQL Enterprise ©2008 MTV Networks Japan K.K.
  11. 11. MTV Flux/MTV Mobile DB project MySQL Architectures MySQL Architectures: • Single/multi Master • Master/Slave • Cluster • DR:BD/Heartbeat HA Master What we looked for: • flexibility • performance • scalability • redundancy • support • cost ©2008 MTV Networks Japan K.K.
  12. 12. MTV Flux/MTV Mobile DB project MySQL Architectures Single/multi Master www www www • cheap(!) • easy configuration × • load ↑ ⇒ performance ↓ • high risk (if master dies, no more DB) Master flexibility performance scalability redundancy support cost ○ ○/△ × × ○ ◎ ©2008 MTV Networks Japan K.K.
  13. 13. MTV Flux/MTV Mobile DB project MySQL Architectures Master/Slave www www www • commonly used architecture × • some redundancy (read/writes separate) • still some risk (operator needed for outage recovery) • manual failover for outages Master Slave flexibility performance scalability redundancy support cost ○ △ ○ △ ○ ◎ ©2008 MTV Networks Japan K.K.
  14. 14. MTV Flux/MTV Mobile DB project MySQL Architectures Cluster www www www • redundant! • can scale △ • complicated configuration • limited support in Japan (2007) Cluster flexibility performance scalability redundancy support cost ○ ○ ◎ ◎ ○/△ △ ©2008 MTV Networks Japan K.K.
  15. 15. MTV Flux/MTV Mobile DB project MySQL Architectures DR:BD/Heartbeat HA Master www www www • redundant (master fully redundant) • can scale (add salves) • reduced outage risk ◎ • can use failover for maintenance tasks Primary Secondary • complicated configuration flexibility performance scalability redundancy support cost ◎ ○/△ ○/△ ◎ ○ ○ ©2008 MTV Networks Japan K.K.
  16. 16. MTV Flux/MTV Mobile DB project MySQL HA Master ©2008 MTV Networks Japan K.K.
  17. 17. MTV Flux/MTV Mobile DB project HA Master Application Application VIP heartbeat /etc/my.cnf /etc/my.cnf local local /var/lib/mysql drbd /var/lib/mysql DRBD DRBD Primary Secondary ©2008 MTV Networks Japan K.K.
  18. 18. MTV Flux/MTV Mobile DB project HA Master Application Web site or whatever VIP Manages/monitors DB process and mounts: • provides application connection point (Virtual IP) • monitors network status heartbeat • watches the mysqld process • manages the /var/lib/mysql mount point mysqld (the database) DRBD Synchronous device layer data replication ©2008 MTV Networks Japan K.K.
  19. 19. MTV Flux/MTV Mobile DB project HA Master Application 3 – heartbeat does: Application • 4 – Failoverdir unmount pDB mysql complete! • DRBD2switch Failover × 1 – Application looses – heartbeat • sDB mysqlthe mount detects dir problem connection to mysqld VIP process • sDB mysqld start network or mysqld failure heartbeat /etc/my.cnf /etc/my.cnf local local /var/lib/mysql drbd drbd /var/lib/mysql DRBD DRBD Primary Secondary ©2008 MTV Networks Japan K.K.
  20. 20. MTV Flux/MTV Mobile DB project HA Master Application Application 3 – heartbeat does: 4 – Failover complete! Failover • sDB 32 – drbd mount mysql dir – heartbeat 1 – primary server fails! • sDBdetects outage mysqld start VIP fails over process server outage heartbeat /etc/my.cnf /etc/my.cnf local local /var/lib/mysql drbd /var/lib/mysql DRBD DRBD Primary Secondary ©2008 MTV Networks Japan K.K.
  21. 21. MTV Flux/MTV Mobile DB project HA MySQL install ©2008 MTV Networks Japan K.K.
  22. 22. MTV Flux/MTV Mobile DB project HA Master Install Old DB architecture A Master B Master Master local disk local disk local disk Dual Master Single Master ©2008 MTV Networks Japan K.K.
  23. 23. MTV Flux/MTV Mobile DB project HA Master Install New DB architecture VIP heartbeat /etc/my.cnf /etc/my.cnf local local /var/lib/mysql drbd /var/lib/mysql DRBD DRBD Primary Secondary ©2008 MTV Networks Japan K.K.
  24. 24. MTV Flux/MTV Mobile DB project HA Master Install Database migration work plan: • stop production services • perform full data backup (mysqldump) • Replace MySQL architecture • restore data from backup • test database failover • test application • re-open production services ©2008 MTV Networks Japan K.K.
  25. 25. MTV Flux/MTV Mobile DB project HA Master Install Migration 2 – install/configure 5 – test application 3 – install 1 – remove old VIP 4 – configure VIP process database components drbd and mysql connectivity heartbeat heartbeat A Master B Master Master × /etc/my.cnf local disk local × local disk /var/lib/mysql drbd /var/lib/mysql /etc/my.cnf local disk local DRBD DRBD Primary Secondary ©2008 MTV Networks Japan K.K.
  26. 26. MTV Flux/MTV Mobile DB project HA Master Install Actual migration work times: •production services stop – 00:10 • full data backup (mysqldump) – 00:05 • MySQL architecture replacement – 01:30 • data restore – 00:10 • database failover test – 00:15 • application test – 03:00 • production services fully restored! ©2008 MTV Networks Japan K.K.
  27. 27. MTV Flux/MTV Mobile DB project HA MySQL lessons learned ©2008 MTV Networks Japan K.K.
  28. 28. MTV Flux/MTV Mobile DB project Lessons learned Impact on our application: • essentially the same as a single master • can use same HA for: • single master - master/slave - multi-master • failover fast enough that DB outages don’t affect application Heartbeat: • no room for configuration mistakes • pay special attention to peer connectivity • run lots of tests before going to production!!! ©2008 MTV Networks Japan K.K.
  29. 29. MTV Flux/MTV Mobile DB project Lessons learned mysqld: • can take advantage of redundancy to swap out nodes • never try to run both primary and secondary at once!! • leave all the redundancy to heartbeat DR:BD: • requires separate partition (can be LVM) • like heartbeat – test your configurations!!! • always check your DRBD status after a failover • peers sometime require a manual re-sync ©2008 MTV Networks Japan K.K.
  30. 30. MTV Flux/MTV Mobile DB project Lessons learned Overall – good points: • we were able to deploy HA in a short maintenance window • commercial support available (MySQL – LINBIT) • MySQL consultative services were a big help • also usable with MySQL Community • can be part of other architectures • multi-master - master/slave Overall – weak points: • lots to learn, lots to test (heartbeat and DRBD) • configuration and failover is complex • mysqld restart requires some extra steps • all admins/vendors need to understand failover basics ©2008 MTV Networks Japan K.K.
  31. 31. HA MySQL in other MTVNJ environments ©2008 MTV Networks Japan K.K.
  32. 32. Questions? DRBD related links http://www.drbd.org/users-guide/s-heartbeat-r1.html http://www.drbd.org/users-guide/s-heartbeat-crm.html MTVNJ links http://www.mtvn.jp/en/index.html http://www.mtvn.jp/en/service.html Patrick Bolduan MTV Networks Japan KK 2-8-2 Jingu-mae, Shibuya-ku, Tokyo 150-0001 patrick.bolduan@mtvn.jp http://www.mtvjapan.com ©2008 MTV Networks Japan K.K.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×