High Availability?
Setting Up MySQL
Our Approach
DBI Framework
Questions
High Availability DBI & MySQL
YAPC::NA 2005
High ...
High Availability?
Setting Up MySQL
Our Approach
DBI Framework
Questions
High Availability DBI & MySQL
Slide 3
High Availability?
• What is High Availability?
• Ever have an application fail because a
mission critical server ...
Slide 4
High Availability?
• How do I achieve HA?:
• Analyze your apps
• Most important components?
• Remove single points...
Slide 5
High Availability?
• This talk…
• Introduces one approach to building HA
Perl apps that rely on MySQL.
• We’ve use...
High Availability?
Setting Up MySQL
Our Approach
DBI Framework
Questions
High Availability DBI & MySQL
Slide 7
Setting Up MySQL
• Some typical architectures…
Slide 8
Setting Up MySQL
• Ye olde basic setup
• Not HA.
Application Server
MySQL
Server
Slide 9
Setting Up MySQL
• One master, one read-only slave
• Good for load balancing
• Still not really HA!
Application Se...
Slide 10
Setting Up MySQL
• Two masters
• It’s HA, but…
• Flaky, tricky to work with!
Application Server
MySQL
MasterTwo-w...
Slide 11
MySQL
Slave
Setting Up MySQL
• Two masters, multiple slaves
• Only one master active at a time
• HA
Application S...
Slide 12
Setting Up MySQL
• Your HA MySQL architecture will depend on:
• SLAs
• Application requirements
• Load
• Physical...
Slide 13
Setting Up MySQL
• Load balancing
• Hardware?
• Software?
• Master Failover
• On Slaves:
CHANGE MASTER TO …
• Man...
Slide 14
Setting Up MySQL
• Recommended reading:
• High Performance MySQL
- Jeremey Zawodny & Derek Balling
• MySQL Replic...
Slide 15
Setting Up MySQL
• If you have any trouble:
• MySQL mailing list
High Availability?
Setting Up MySQL
Our Approach
DBI Framework
Questions
High Availability DBI & MySQL
Slide 17
Our Approach
• Our requirements:
• Read-heavy apps
• Over 750 clients, many with SLAs
• Reliability:
• Read avail...
Slide 18
Our Approach
Application
Server
Application
Server
Application
Server
MySQL
Slave
MySQL
Slaves
MySQL
Slaves
Appli...
Slide 19
Farm 2
Farm 3
Farm 1
Our Approach
Application
Server
Application
Server
MySQL
Slave
Application
Server
MySQL
Mast...
Slide 20
Our Approach
• DBI Framework:
• Load balancing
• Server selection
• Read / Write query?
• Failover
Application
DB...
Slide 21
Our Approach
• Load balancing
• On connect
• Weighted server selection
• Availability, number of processes,
user ...
Slide 22
Our Approach
• Failover
• Persistent connection to
all masters
• Automatic query retry
• Automatic fallback
Appli...
High Availability?
Setting Up MySQL
Our Approach
DBI Framework
Questions
High Availability DBI & MySQL
Slide 24
DBI Framework
• Two wrapper functions:
• dbconnect( 'read' | 'write' )
• Read: Select slave to connect to
• Write...
Slide 25
DBI Framework
• dbconnect( 'read' | 'write' )
• Read:
For each slave
weight = 0, next if can’t ping
check number ...
Slide 26
DBI Framework
• $sth = sql( $query )
• Determine query type…
• Read:
Execute query.
Failover to next slave on err...
Slide 27
DBI Framework
• Major Drawbacks:
• Using DBI-based CPAN modules is hard!
Slide 28
DBI Framework
• Looking Ahead…
• Push HA logic into the DBI layer
• Write DBD::MysqlHA ?
• DBIx::HA ?
• DBD::Mult...
High Availability?
Setting Up MySQL
Our Approach
DBI Framework
Questions
High Availability DBI & MySQL
Slide 30
DBI Frameworks on CPAN
• If there’s time…
Slide 31
DBI Frameworks on CPAN
• The ones I know a bit about:
• DBIx::HA
• DBD::Multiplex
• DBIx::DBCluster
Slide 32
DBI Frameworks on CPAN
• DBIx::HA
• Generic HA solution
(written & tested for Sybase)
• Configurable by db name (...
Slide 33
DBIx::HA
Pros
• Does Failover
• On query failure & dbh
disconnected
• Does timeouts:
• connect, query execute
• S...
Slide 34
DBIx::HA
• Also: written & tested for Sybase!
• Potential DBD::Mysql problems:
• auto_reconnect
we may reconnect ...
Slide 35
DBI Frameworks on CPAN
• DBD::Multiplex
• send requests to multiple dsn's
• Configure servers to use in $dsn
(pip...
Slide 36
DBD::Multiplex
Pros
• Supports master/slave
setup:
• differentiates between
reads / writes
• Connects to all dsns...
Slide 37
DBI Frameworks on CPAN
• How could we re-use CPAN modules?
• DBIx::HA
• Sub-class to introduce MySQL specific fun...
Slide 38
DBI Frameworks on CPAN
• Maybe the best way to reuse them is to nick their
ideas?
• DBD::MysqlHA ?
• Combination ...
Upcoming SlideShare
Loading in …5
×

High Availability Perl DBI + MySQL

2,129 views
1,974 views

Published on

A talk I gave @ YAPC::NA in 2005 on implementing High Availability for Perl applications that span multiple datacenters.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,129
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
16
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Start off by asking who knows what HA is, and get a feel for how much experience they’ve got with HA.
  • Did your clients notice?
    Did you have an SLA?
    Were you in bed at the time?
    Did you have a backup server?
    Did you have a backup of the data?
    How long did it take to get the backup server with the backup data back online?
  • HA for reads, but not for writes.
  • Due to the way MySQL replication works - conflicting writes (typically auto increment keys)
    Mention FS replication
  • Include further links
  • Confirm with andrew on # requests a day.
  • Speak about minimizing costs:
    replication across farms.
    Query caching run on all slaves out of memory to remove io bottleneck
  • S/w Loadbalancing
  • Mention Tear down / setup of connections.
  • HA dsn:
    dsn1: Multiplex to primary master + many slaves
    dsn2: Multiplex to secondary master + many slaves
    If all slaves go down, will all reads go to master?
  • High Availability Perl DBI + MySQL

    1. 1. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL YAPC::NA 2005 High Availability DBI & MySQL Steve Purkis
    2. 2. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
    3. 3. Slide 3 High Availability? • What is High Availability? • Ever have an application fail because a mission critical server crashed? • High Availability is an attempt to provide zero application downtime.
    4. 4. Slide 4 High Availability? • How do I achieve HA?: • Analyze your apps • Most important components? • Remove single points of failure (SPOFs): • Add redundant servers • Replicate data (backups, realtime) • Load balancing • Failover when things go pear-shaped
    5. 5. Slide 5 High Availability? • This talk… • Introduces one approach to building HA Perl apps that rely on MySQL. • We’ve used similar techniques with Oracle • They work for us… • They may not be right for you!
    6. 6. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
    7. 7. Slide 7 Setting Up MySQL • Some typical architectures…
    8. 8. Slide 8 Setting Up MySQL • Ye olde basic setup • Not HA. Application Server MySQL Server
    9. 9. Slide 9 Setting Up MySQL • One master, one read-only slave • Good for load balancing • Still not really HA! Application Server MySQL Master MySQL Slave Data Replication reads writes Load Balancer
    10. 10. Slide 10 Setting Up MySQL • Two masters • It’s HA, but… • Flaky, tricky to work with! Application Server MySQL MasterTwo-way Replication MySQL Master Load Balancer
    11. 11. Slide 11 MySQL Slave Setting Up MySQL • Two masters, multiple slaves • Only one master active at a time • HA Application Server MySQL Slaves Data Replication reads writes Primary Master 2nd ary Master Master Failover Load Balancer • HA (unless your hosting center goes down)
    12. 12. Slide 12 Setting Up MySQL • Your HA MySQL architecture will depend on: • SLAs • Application requirements • Load • Physical spread of servers • Budget • Caveats: • Are your apps read or write heavy? • Beware of replication lag time • Common in high-latency networks • Do your apps need the latest data? • Transactions?
    13. 13. Slide 13 Setting Up MySQL • Load balancing • Hardware? • Software? • Master Failover • On Slaves: CHANGE MASTER TO … • Manual? • Automate? • Cron? Application? SLB? • On App servers: • Connect to Secondary • Not required if using a hot spare w/same ip
    14. 14. Slide 14 Setting Up MySQL • Recommended reading: • High Performance MySQL - Jeremey Zawodny & Derek Balling • MySQL Replication docs • MySQL Cluster white paper • For Linux: • Linux Virtual Server Project • Linux-HA
    15. 15. Slide 15 Setting Up MySQL • If you have any trouble: • MySQL mailing list
    16. 16. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
    17. 17. Slide 17 Our Approach • Our requirements: • Read-heavy apps • Over 750 clients, many with SLAs • Reliability: • Read availability most important • Must work if a server farm goes down • Speed is of essence: • Millions of requests a day • Don’t need transactions • Minimize costs
    18. 18. Slide 18 Our Approach Application Server Application Server Application Server MySQL Slave MySQL Slaves MySQL Slaves Application Server MySQL Master MySQL 2nd ary MasterFailover Data Replication 95% 5% reads writes
    19. 19. Slide 19 Farm 2 Farm 3 Farm 1 Our Approach Application Server Application Server MySQL Slave Application Server MySQL Master MySQL 2nd ary MasterFailover In-Farm Data Replication Application Server MySQL Slave MySQL Slaves MySQL Slave MySQL Slaves Query caching
    20. 20. Slide 20 Our Approach • DBI Framework: • Load balancing • Server selection • Read / Write query? • Failover Application DBI Framework ----------------------------------- snip: how to tell reads from writes? --------------------------------- my ($action, $dbh) = $query =~ /Aselect|A(select|Ashow|Adesc/i ? ("read", $dbh_read) : ("write", $dbh_write); ----------------------------------------------------------------------------------------------------------
    21. 21. Slide 21 Our Approach • Load balancing • On connect • Weighted server selection • Availability, number of processes, user weights, which farm • MySQL idle timeout • Failover • Connect to next slave • Automatic query retry Application DBI Framework MySQL Slave MySQL Slave reads
    22. 22. Slide 22 Our Approach • Failover • Persistent connection to all masters • Automatic query retry • Automatic fallback Application Server DBI Framework Primary Master 2nd ary Master MasterFailover writes
    23. 23. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
    24. 24. Slide 24 DBI Framework • Two wrapper functions: • dbconnect( 'read' | 'write' ) • Read: Select slave to connect to • Write: Connect to all masters • $sth = sql( $query ) • Read / write dbh selection • Failover & fallback • Pseudo code?
    25. 25. Slide 25 DBI Framework • dbconnect( 'read' | 'write' ) • Read: For each slave weight = 0, next if can’t ping check number of processes with mysqladmin weight = no. processes / user weighting Connect to slave with lowest weight Sanity check: run a simple query Try next slave if that failed • Write: For each master Connect Sanity check: run a simple query Set $write_dbh to primary master
    26. 26. Slide 26 DBI Framework • $sth = sql( $query ) • Determine query type… • Read: Execute query. Failover to next slave on error, and retry query. • Write: Fallback? If not using primary, and we failed over X seconds ago, try reconnecting to master. Execute query. Failover to next master on error, and retry query.
    27. 27. Slide 27 DBI Framework • Major Drawbacks: • Using DBI-based CPAN modules is hard!
    28. 28. Slide 28 DBI Framework • Looking Ahead… • Push HA logic into the DBI layer • Write DBD::MysqlHA ? • DBIx::HA ? • DBD::Multiplex ? • DBIx::DBCluster ? • SQL Relay ? • MySQL Cluster?
    29. 29. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
    30. 30. Slide 30 DBI Frameworks on CPAN • If there’s time…
    31. 31. Slide 31 DBI Frameworks on CPAN • The ones I know a bit about: • DBIx::HA • DBD::Multiplex • DBIx::DBCluster
    32. 32. Slide 32 DBI Frameworks on CPAN • DBIx::HA • Generic HA solution (written & tested for Sybase) • Configurable by db name (%DATABASE::conf) • Looks well thought out
    33. 33. Slide 33 DBIx::HA Pros • Does Failover • On query failure & dbh disconnected • Does timeouts: • connect, query execute • Safe signals • Connect all dsns on init • Supports Apache::DBI Cons • No read / write distinction • No way to choose which dbh to use on failover • (want to use a mysql- specific algorithm) • Timeouts with SIGALRM • (but how else, really?) • No ping checks for non- Apache::DBI • (uses $dbh->ping anyways - we prefer ICMP ping)
    34. 34. Slide 34 DBIx::HA • Also: written & tested for Sybase! • Potential DBD::Mysql problems: • auto_reconnect we may reconnect to a db when we should be failing over
    35. 35. Slide 35 DBI Frameworks on CPAN • DBD::Multiplex • send requests to multiple dsn's • Configure servers to use in $dsn (pipe-separated list)
    36. 36. Slide 36 DBD::Multiplex Pros • Supports master/slave setup: • differentiates between reads / writes • Connects to all dsns (good for fast master failover) • Does failover: • default behaviour reads: first_success writes: first_error Cons • Can only specify one master • (though if you specify none, writes can go to all with 'first_error' exit mode) • Connects to all dsns • (don't want to connect to all slaves) • No customizable slave failover algorithm • No reconnects • No fallback!
    37. 37. Slide 37 DBI Frameworks on CPAN • How could we re-use CPAN modules? • DBIx::HA • Sub-class to introduce MySQL specific functionality? • Introduce a callback for server selection on failover? • Use in conjunction with DBD::Multiplex for read/write dbh selection? • There could be problems though… • In general: • Backwards compat • sql() wrapper (for backwards compat) • Custom logging? • Our db wrappers do other things too…
    38. 38. Slide 38 DBI Frameworks on CPAN • Maybe the best way to reuse them is to nick their ideas? • DBD::MysqlHA ? • Combination of DBIx::HA and DBD::Multiplex • MySQL specific: • Customizeable server selection algorithm • Persistent connections

    ×