Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

High Availability Perl DBI + MySQL

on

  • 2,009 views

A talk I gave @ YAPC::NA in 2005 on implementing High Availability for Perl applications that span multiple datacenters.

A talk I gave @ YAPC::NA in 2005 on implementing High Availability for Perl applications that span multiple datacenters.

Statistics

Views

Total Views
2,009
Views on SlideShare
2,006
Embed Views
3

Actions

Likes
2
Downloads
14
Comments
0

1 Embed 3

http://www.slashdocs.com 3

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Start off by asking who knows what HA is, and get a feel for how much experience they’ve got with HA.
  • Did your clients notice? Did you have an SLA? Were you in bed at the time? Did you have a backup server? Did you have a backup of the data? How long did it take to get the backup server with the backup data back online?
  • HA for reads, but not for writes.
  • Due to the way MySQL replication works - conflicting writes (typically auto increment keys) Mention FS replication
  • Include further links
  • Confirm with andrew on # requests a day.
  • Speak about minimizing costs: replication across farms. Query caching run on all slaves out of memory to remove io bottleneck
  • S/w Loadbalancing
  • Mention Tear down / setup of connections.
  • HA dsn: dsn1: Multiplex to primary master + many slaves dsn2: Multiplex to secondary master + many slaves If all slaves go down, will all reads go to master?

High Availability Perl DBI + MySQL High Availability Perl DBI + MySQL Presentation Transcript

  • High Availability DBI & MySQL Steve Purkis YAPC::NA 2005
  •  
  • High Availability?
    • What is High Availability?
      • Ever have an application fail because a mission critical server crashed?
      • High Availability is an attempt to provide zero application downtime.
  • High Availability?
    • How do I achieve HA?:
      • Analyze your apps
        • Most important components?
      • Remove single points of failure (SPOFs):
        • Add redundant servers
        • Replicate data (backups, realtime)
        • Load balancing
        • Failover when things go pear-shaped
  • High Availability?
    • This talk…
      • Introduces one approach to building HA Perl apps that rely on MySQL.
      • We’ve used similar techniques with Oracle
      • They work for us…
        • They may not be right for you!
  •  
  • Setting Up MySQL
    • Some typical architectures…
  • Setting Up MySQL
    • Ye olde basic setup
      • Not HA.
    Application Server MySQL Server
  • Setting Up MySQL
    • One master , one read-only slave
      • Good for load balancing
      • Still not really HA!
    Application Server MySQL Master MySQL Slave Data Replication reads writes Load Balancer
  • Setting Up MySQL
    • Two masters
      • It’s HA, but…
      • Flaky, tricky to work with!
    Application Server MySQL Master Two-way Replication MySQL Master Load Balancer
  • Setting Up MySQL
    • Two masters , multiple slaves
      • Only one master active at a time
      • HA
    MySQL Slave Application Server MySQL Slaves Data Replication reads writes Primary Master 2 nd ary Master Master Failover Load Balancer
      • HA (unless your hosting center goes down)
  • Setting Up MySQL
    • Your HA MySQL architecture will depend on:
      • SLAs
      • Application requirements
      • Load
      • Physical spread of servers
      • Budget
    • Caveats:
      • Are your apps read or write heavy?
      • Beware of replication lag time
        • Common in high-latency networks
        • Do your apps need the latest data?
      • Transactions?
  • Setting Up MySQL
    • Load balancing
      • Hardware?
      • Software?
    • Master Failover
      • On Slaves:
        • CHANGE MASTER TO …
        • Manual?
        • Automate?
          • Cron? Application? SLB?
      • On App servers:
        • Connect to Secondary
      • Not required if using a hot spare w/same ip
  • Setting Up MySQL
    • Recommended reading:
      • High Performance MySQL
        • - Jeremey Zawodny & Derek Balling
      • MySQL Replication docs
      • MySQL Cluster white paper
    • For Linux:
      • Linux Virtual Server Project
      • Linux-HA
  • Setting Up MySQL
    • If you have any trouble:
      • MySQL mailing list
  •  
  • Our Approach
    • Our requirements:
      • Read-heavy apps
      • Over 750 clients, many with SLAs
      • Reliability:
        • Read availability most important
        • Must work if a server farm goes down
      • Speed is of essence:
        • Millions of requests a day
      • Don’t need transactions
      • Minimize costs
  • Our Approach Application Server Application Server Application Server MySQL Slave MySQL Slaves MySQL Slaves Application Server MySQL Master MySQL 2 nd ary Master Failover Data Replication 95% 5% reads writes
  • Our Approach Farm 2 Farm 3 Farm 1 Application Server Application Server MySQL Slave Application Server MySQL Master MySQL 2 nd ary Master Failover In-Farm Data Replication Application Server MySQL Slave MySQL Slaves MySQL Slave MySQL Slaves Query caching
  • Our Approach
    • DBI Framework:
      • Load balancing
      • Server selection
        • Read / Write query?
      • Failover
    Application DBI Framework ----------------------------------- snip: how to tell reads from writes? --------------------------------- my ( $action , $dbh ) = $query =~ /Aselect|A(select|Ashow|Adesc/i ? ( "read" , $dbh_read ) : ( "write" , $dbh_write ); ----------------------------------------------------------------------------------------------------------
  • Our Approach
    • Load balancing
      • On connect
      • Weighted server selection
        • Availability, number of processes, user weights, which farm
      • MySQL idle timeout
    • Failover
      • Connect to next slave
      • Automatic query retry
    Application DBI Framework MySQL Slave MySQL Slave reads
  • Our Approach
    • Failover
      • Persistent connection to all masters
      • Automatic query retry
      • Automatic fallback
    Application Server DBI Framework Primary Master 2 nd ary Master Master Failover writes
  •  
  • DBI Framework
    • Two wrapper functions:
      • dbconnect( 'read' | 'write' )
        • Read: Select slave to connect to
        • Write: Connect to all masters
      • $sth = sql( $query )
        • Read / write dbh selection
        • Failover & fallback
      • Pseudo code?
  • DBI Framework
    • dbconnect( 'read' | 'write' )
      • Read:
          • For each slave
          • weight = 0, next if can’t ping
          • check number of processes with mysqladmin
          • weight = no. processes / user weighting
          • Connect to slave with lowest weight
          • Sanity check: run a simple query
          • Try next slave if that failed
      • Write:
          • For each master
          • Connect
          • Sanity check: run a simple query
          • Set $write_dbh to primary master
  • DBI Framework
    • $sth = sql( $query )
      • Determine query type…
      • Read:
          • Execute query.
          • Failover to next slave on error, and retry query.
      • Write:
          • Fallback?
          • If not using primary, and we failed over X seconds ago, try reconnecting to master.
          • Execute query.
          • Failover to next master on error, and retry query.
  • DBI Framework
    • Major Drawbacks:
      • Using DBI-based CPAN modules is hard !
  • DBI Framework
    • Looking Ahead…
      • Push HA logic into the DBI layer
        • Write DBD::MysqlHA ?
        • DBIx::HA ?
        • DBD::Multiplex ?
        • DBIx::DBCluster ?
        • SQL Relay ?
      • MySQL Cluster?
  •  
  • DBI Frameworks on CPAN
    • If there’s time…
  • DBI Frameworks on CPAN
    • The ones I know a bit about:
      • DBIx::HA
      • DBD::Multiplex
      • DBIx::DBCluster
  • DBI Frameworks on CPAN
    • DBIx::HA
      • Generic HA solution
      • (written & tested for Sybase)
      • Configurable by db name (%DATABASE::conf)
      • Looks well thought out
  • DBIx::HA
    • Pros
    • Does Failover
      • On query failure & dbh disconnected
    • Does timeouts:
      • connect, query execute
      • Safe signals
    • Connect all dsns on init
    • Supports Apache::DBI
    • Cons
    • No read / write distinction
    • No way to choose which dbh to use on failover
      • (want to use a mysql-specific algorithm)
    • Timeouts with SIGALRM
      • (but how else, really?)
    • No ping checks for non-Apache::DBI
      • (uses $dbh->ping anyways - we prefer ICMP ping)
  • DBIx::HA
    • Also: written & tested for Sybase!
    • Potential DBD::Mysql problems:
      • auto_reconnect
      • we may reconnect to a db when we should be failing over
  • DBI Frameworks on CPAN
    • DBD::Multiplex
      • send requests to multiple dsn's
      • Configure servers to use in $dsn
      • (pipe-separated list)
  • DBD::Multiplex
    • Pros
    • Supports master/slave setup:
      • differentiates between reads / writes
      • Connects to all dsns
      • (good for fast master failover)
    • Does failover:
      • default behaviour
      • reads: first_success
      • writes: first_error
    • Cons
    • Can only specify one master
      • (though if you specify none, writes can go to all with 'first_error' exit mode)
    • Connects to all dsns
      • (don't want to connect to all slaves)
    • No customizable slave failover algorithm
    • No reconnects
      • No fallback!
  • DBI Frameworks on CPAN
    • How could we re-use CPAN modules?
    • DBIx::HA
      • Sub-class to introduce MySQL specific functionality?
      • Introduce a callback for server selection on failover?
      • Use in conjunction with DBD::Multiplex for read/write dbh selection?
        • There could be problems though…
    • In general:
      • Backwards compat
      • sql() wrapper (for backwards compat)
      • Custom logging?
      • Our db wrappers do other things too…
  • DBI Frameworks on CPAN
    • Maybe the best way to reuse them is to nick their ideas?
    • DBD::MysqlHA ?
      • Combination of DBIx::HA and DBD::Multiplex
      • MySQL specific:
        • Customizeable server selection algorithm
      • Persistent connections