Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

High Availability Perl DBI + MySQL

2,556 views

Published on

A talk I gave @ YAPC::NA in 2005 on implementing High Availability for Perl applications that span multiple datacenters.

Published in: Technology
  • Be the first to comment

High Availability Perl DBI + MySQL

  1. 1. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL YAPC::NA 2005 High Availability DBI & MySQL Steve Purkis
  2. 2. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
  3. 3. Slide 3 High Availability? • What is High Availability? • Ever have an application fail because a mission critical server crashed? • High Availability is an attempt to provide zero application downtime.
  4. 4. Slide 4 High Availability? • How do I achieve HA?: • Analyze your apps • Most important components? • Remove single points of failure (SPOFs): • Add redundant servers • Replicate data (backups, realtime) • Load balancing • Failover when things go pear-shaped
  5. 5. Slide 5 High Availability? • This talk… • Introduces one approach to building HA Perl apps that rely on MySQL. • We’ve used similar techniques with Oracle • They work for us… • They may not be right for you!
  6. 6. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
  7. 7. Slide 7 Setting Up MySQL • Some typical architectures…
  8. 8. Slide 8 Setting Up MySQL • Ye olde basic setup • Not HA. Application Server MySQL Server
  9. 9. Slide 9 Setting Up MySQL • One master, one read-only slave • Good for load balancing • Still not really HA! Application Server MySQL Master MySQL Slave Data Replication reads writes Load Balancer
  10. 10. Slide 10 Setting Up MySQL • Two masters • It’s HA, but… • Flaky, tricky to work with! Application Server MySQL MasterTwo-way Replication MySQL Master Load Balancer
  11. 11. Slide 11 MySQL Slave Setting Up MySQL • Two masters, multiple slaves • Only one master active at a time • HA Application Server MySQL Slaves Data Replication reads writes Primary Master 2nd ary Master Master Failover Load Balancer • HA (unless your hosting center goes down)
  12. 12. Slide 12 Setting Up MySQL • Your HA MySQL architecture will depend on: • SLAs • Application requirements • Load • Physical spread of servers • Budget • Caveats: • Are your apps read or write heavy? • Beware of replication lag time • Common in high-latency networks • Do your apps need the latest data? • Transactions?
  13. 13. Slide 13 Setting Up MySQL • Load balancing • Hardware? • Software? • Master Failover • On Slaves: CHANGE MASTER TO … • Manual? • Automate? • Cron? Application? SLB? • On App servers: • Connect to Secondary • Not required if using a hot spare w/same ip
  14. 14. Slide 14 Setting Up MySQL • Recommended reading: • High Performance MySQL - Jeremey Zawodny & Derek Balling • MySQL Replication docs • MySQL Cluster white paper • For Linux: • Linux Virtual Server Project • Linux-HA
  15. 15. Slide 15 Setting Up MySQL • If you have any trouble: • MySQL mailing list
  16. 16. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
  17. 17. Slide 17 Our Approach • Our requirements: • Read-heavy apps • Over 750 clients, many with SLAs • Reliability: • Read availability most important • Must work if a server farm goes down • Speed is of essence: • Millions of requests a day • Don’t need transactions • Minimize costs
  18. 18. Slide 18 Our Approach Application Server Application Server Application Server MySQL Slave MySQL Slaves MySQL Slaves Application Server MySQL Master MySQL 2nd ary MasterFailover Data Replication 95% 5% reads writes
  19. 19. Slide 19 Farm 2 Farm 3 Farm 1 Our Approach Application Server Application Server MySQL Slave Application Server MySQL Master MySQL 2nd ary MasterFailover In-Farm Data Replication Application Server MySQL Slave MySQL Slaves MySQL Slave MySQL Slaves Query caching
  20. 20. Slide 20 Our Approach • DBI Framework: • Load balancing • Server selection • Read / Write query? • Failover Application DBI Framework ----------------------------------- snip: how to tell reads from writes? --------------------------------- my ($action, $dbh) = $query =~ /Aselect|A(select|Ashow|Adesc/i ? ("read", $dbh_read) : ("write", $dbh_write); ----------------------------------------------------------------------------------------------------------
  21. 21. Slide 21 Our Approach • Load balancing • On connect • Weighted server selection • Availability, number of processes, user weights, which farm • MySQL idle timeout • Failover • Connect to next slave • Automatic query retry Application DBI Framework MySQL Slave MySQL Slave reads
  22. 22. Slide 22 Our Approach • Failover • Persistent connection to all masters • Automatic query retry • Automatic fallback Application Server DBI Framework Primary Master 2nd ary Master MasterFailover writes
  23. 23. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
  24. 24. Slide 24 DBI Framework • Two wrapper functions: • dbconnect( 'read' | 'write' ) • Read: Select slave to connect to • Write: Connect to all masters • $sth = sql( $query ) • Read / write dbh selection • Failover & fallback • Pseudo code?
  25. 25. Slide 25 DBI Framework • dbconnect( 'read' | 'write' ) • Read: For each slave weight = 0, next if can’t ping check number of processes with mysqladmin weight = no. processes / user weighting Connect to slave with lowest weight Sanity check: run a simple query Try next slave if that failed • Write: For each master Connect Sanity check: run a simple query Set $write_dbh to primary master
  26. 26. Slide 26 DBI Framework • $sth = sql( $query ) • Determine query type… • Read: Execute query. Failover to next slave on error, and retry query. • Write: Fallback? If not using primary, and we failed over X seconds ago, try reconnecting to master. Execute query. Failover to next master on error, and retry query.
  27. 27. Slide 27 DBI Framework • Major Drawbacks: • Using DBI-based CPAN modules is hard!
  28. 28. Slide 28 DBI Framework • Looking Ahead… • Push HA logic into the DBI layer • Write DBD::MysqlHA ? • DBIx::HA ? • DBD::Multiplex ? • DBIx::DBCluster ? • SQL Relay ? • MySQL Cluster?
  29. 29. High Availability? Setting Up MySQL Our Approach DBI Framework Questions High Availability DBI & MySQL
  30. 30. Slide 30 DBI Frameworks on CPAN • If there’s time…
  31. 31. Slide 31 DBI Frameworks on CPAN • The ones I know a bit about: • DBIx::HA • DBD::Multiplex • DBIx::DBCluster
  32. 32. Slide 32 DBI Frameworks on CPAN • DBIx::HA • Generic HA solution (written & tested for Sybase) • Configurable by db name (%DATABASE::conf) • Looks well thought out
  33. 33. Slide 33 DBIx::HA Pros • Does Failover • On query failure & dbh disconnected • Does timeouts: • connect, query execute • Safe signals • Connect all dsns on init • Supports Apache::DBI Cons • No read / write distinction • No way to choose which dbh to use on failover • (want to use a mysql- specific algorithm) • Timeouts with SIGALRM • (but how else, really?) • No ping checks for non- Apache::DBI • (uses $dbh->ping anyways - we prefer ICMP ping)
  34. 34. Slide 34 DBIx::HA • Also: written & tested for Sybase! • Potential DBD::Mysql problems: • auto_reconnect we may reconnect to a db when we should be failing over
  35. 35. Slide 35 DBI Frameworks on CPAN • DBD::Multiplex • send requests to multiple dsn's • Configure servers to use in $dsn (pipe-separated list)
  36. 36. Slide 36 DBD::Multiplex Pros • Supports master/slave setup: • differentiates between reads / writes • Connects to all dsns (good for fast master failover) • Does failover: • default behaviour reads: first_success writes: first_error Cons • Can only specify one master • (though if you specify none, writes can go to all with 'first_error' exit mode) • Connects to all dsns • (don't want to connect to all slaves) • No customizable slave failover algorithm • No reconnects • No fallback!
  37. 37. Slide 37 DBI Frameworks on CPAN • How could we re-use CPAN modules? • DBIx::HA • Sub-class to introduce MySQL specific functionality? • Introduce a callback for server selection on failover? • Use in conjunction with DBD::Multiplex for read/write dbh selection? • There could be problems though… • In general: • Backwards compat • sql() wrapper (for backwards compat) • Custom logging? • Our db wrappers do other things too…
  38. 38. Slide 38 DBI Frameworks on CPAN • Maybe the best way to reuse them is to nick their ideas? • DBD::MysqlHA ? • Combination of DBIx::HA and DBD::Multiplex • MySQL specific: • Customizeable server selection algorithm • Persistent connections

×