0
Scaling Databases with     DBIx::Router        Perrin Harkins       We Also Walk Dogs
What is DBIx::Router?Load-balancingFailoverShardingTransparent   (Mostly.)
Why would you need this?Web and app servers are easy to scale   Just add another dozen boxes
Why would you need this?Databases not so much   Big iron   Commercial clustering solutions   Human sacrifice
Advice from Experts       Brad Fitzpatrick,                Cal Henderson,               Jeremy Zawodny“Inside LiveJournal’...
CachingKeep your hot data in a fast cacheYou get this one for free, thanks to DBI::GoferCan use Cache::FastMmap, memcached...
Read-only CopiesReplication to local server or remote slavesBe careful of replication lag                (from MySQL 5.1 d...
ShardingLarge data is split across multiple machines   Users A-L, M-Z   Logs by month   Consistent hashing algorithmMay in...
This is going to mean a custom database layerAnd rewriting all your old code to use itDBIx::Router tries to separate this ...
Shoulders of GiantsEnter DBI::Gofer   “A scalable stateless proxy architecture for DBI”   Bundles up requests, sends them ...
Shoulders of GiantsUsed by shopzilla.com and petfinder.com to pool connectionsLots of good stuff like caching, timeouts, tr...
Shoulders of GiantsCPAN makes everything easy   SQL::Statement parses SQL (!)   Config::Any solves the XML problem
How does it work?DataSources   Individual (DSNs) or Group   Group handles failover   Also load-balancing
Single DataSource  {       name       =>   Master1,       dsn        =>   dbi:Pg:dbname=lolcats,       user       =>   ‘ic...
Group DataSource  {       name        => ReadCluster,       class       => random,       datasources => [ ‘Slave1’, ‘Slave...
Group DataSource  {       name        => ReadCluster,       class       => roundrobin,       datasources => [ ‘Slave1’, ‘S...
Group DataSource  {       name          =>   ReadCluster,       class         =>   roundrobin,       datasources   =>   [ ...
Group DataSource  {       name        => ReadWriteCluster,       class       => repeater,       datasources => [ ‘Master1’...
Group DataSource    {         name      =>   StoreShards,         class     =>   shard,         type      =>   list,      ...
Subclass DataSourcesCustom auth schemes for the paranoidYe olde insane load-balancing schemeShards with a directory server
RulesMap queries to DataSourcesOrganized in RuleLists   Most specific to least   Can fall back to pass-through
Rule: regex  {       class        =>   regex,       datasource   =>   ReadCluster,       match        =>   [^ s* SELECT b ...
Rule: readonly  {       class      => readonly,       datasource => ReadCluster,  },
Rule: parser  {        class => parser,        match => [            {                structure => tables,                ...
Rule: not  {       class      => not,       rule       => { class => ‘readonly’ }       datasource => Master1,  },
Rule: default  {       class      => default,       datasource => Master1,  },
{    datasources => [        {            name => Master1,            dsn => dbi:mysql:dbname=lolcats,            user => ...
How do you run it?DBI_AUTOPROXY=”dbi:Gofer:   transport=DBIx::Router;   conf=/path/to/conf.pl”$dbh = DBI->connect("dbi:Gof...
What’s the bad news?AutoCommit   But Tim plans to fix thatNo streaming results   Also fixableFailover and sharding are tough...
StatusHosted on Google CodeMostly there, but some bits need work   Sharding   FailoverNeeds user feedbackNeeds more tests
Future DirectionsMake routing decisions optionally stickySupport explicit hints in method call attributes   Helps with sha...
Thanks!http://code.google.com/p/dbix-router/
What else is out there?Mostly faux DBD drivers   DBD::Multi   DBD::Multiplex   DBIx::HADBI::Role
Upcoming SlideShare
Loading in...5
×

Scaling Databases with DBIx::Router

1,406

Published on

This talk was presented at OSCON 2008 and YAPC::NA 2008.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,406
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Scaling Databases with DBIx::Router"

  1. 1. Scaling Databases with DBIx::Router Perrin Harkins We Also Walk Dogs
  2. 2. What is DBIx::Router?Load-balancingFailoverShardingTransparent (Mostly.)
  3. 3. Why would you need this?Web and app servers are easy to scale Just add another dozen boxes
  4. 4. Why would you need this?Databases not so much Big iron Commercial clustering solutions Human sacrifice
  5. 5. Advice from Experts Brad Fitzpatrick, Cal Henderson, Jeremy Zawodny“Inside LiveJournal’s Backend” “Building Scalable Websites” and Derek J Balling, “High Performance MySQL”
  6. 6. CachingKeep your hot data in a fast cacheYou get this one for free, thanks to DBI::GoferCan use Cache::FastMmap, memcached, etc. through CHI
  7. 7. Read-only CopiesReplication to local server or remote slavesBe careful of replication lag (from MySQL 5.1 docs)
  8. 8. ShardingLarge data is split across multiple machines Users A-L, M-Z Logs by month Consistent hashing algorithmMay involve directory serverJOINs are now the programmer’s problem
  9. 9. This is going to mean a custom database layerAnd rewriting all your old code to use itDBIx::Router tries to separate this plumbing
  10. 10. Shoulders of GiantsEnter DBI::Gofer “A scalable stateless proxy architecture for DBI” Bundles up requests, sends them over a transport, executes them, sends back results
  11. 11. Shoulders of GiantsUsed by shopzilla.com and petfinder.com to pool connectionsLots of good stuff like caching, timeouts, tracingDBIx::Router is a Gofer transport But it executes the calls locally
  12. 12. Shoulders of GiantsCPAN makes everything easy SQL::Statement parses SQL (!) Config::Any solves the XML problem
  13. 13. How does it work?DataSources Individual (DSNs) or Group Group handles failover Also load-balancing
  14. 14. Single DataSource { name => Master1, dsn => dbi:Pg:dbname=lolcats, user => ‘icanhas’, password => ‘ch33zeburger’, },
  15. 15. Group DataSource { name => ReadCluster, class => random, datasources => [ ‘Slave1’, ‘Slave2’ ], },
  16. 16. Group DataSource { name => ReadCluster, class => roundrobin, datasources => [ ‘Slave1’, ‘Slave2’ ], },
  17. 17. Group DataSource { name => ReadCluster, class => roundrobin, datasources => [ ‘Slave1’, ‘Slave2’ ], failover => 1, timeout => 8, },
  18. 18. Group DataSource { name => ReadWriteCluster, class => repeater, datasources => [ ‘Master1’, ‘Master2’ ], },
  19. 19. Group DataSource { name => StoreShards, class => shard, type => list, table => orders, column => store_id, shards => [ { values => [ 1, 3, 5 ], datasource => EastCoast, }, { values => [ 2, 4, 6 ], datasource => WestCoast, }, ], },
  20. 20. Subclass DataSourcesCustom auth schemes for the paranoidYe olde insane load-balancing schemeShards with a directory server
  21. 21. RulesMap queries to DataSourcesOrganized in RuleLists Most specific to least Can fall back to pass-through
  22. 22. Rule: regex { class => regex, datasource => ReadCluster, match => [^ s* SELECT b ], not_match => [b FOR s+ UPDATE b ], },
  23. 23. Rule: readonly { class => readonly, datasource => ReadCluster, },
  24. 24. Rule: parser { class => parser, match => [ { structure => tables, operator => all, tokens => [order_history’] }, }, Operators: all, any, none, only Structures: command, tables, columns
  25. 25. Rule: not { class => not, rule => { class => ‘readonly’ } datasource => Master1, },
  26. 26. Rule: default { class => default, datasource => Master1, },
  27. 27. { datasources => [ { name => Master1, dsn => dbi:mysql:dbname=lolcats, user => undef, password => undef, }, { name => Slave1, dsn => dbi:mysql:dbname=zomg1, user => undef, password => undef, }, { name => Slave2, dsn => dbi:mysql:dbname=zomg2, user => undef, password => undef, }, { name => ReadCluster, class => random, datasources => [ Slave1, Slave2, ], failover => 1, timeout => 8, }, ], rules => [ { class => readonly, datasource => ReadCluster, }, { class => default, datasource => Master1, }, ],}
  28. 28. How do you run it?DBI_AUTOPROXY=”dbi:Gofer: transport=DBIx::Router; conf=/path/to/conf.pl”$dbh = DBI->connect("dbi:Gofer: transport=DBIx::Router; conf=/path/to/conf.pl; dsn=$original_dsn", $user, $passwd, %attributes);
  29. 29. What’s the bad news?AutoCommit But Tim plans to fix thatNo streaming results Also fixableFailover and sharding are tough to generalize
  30. 30. StatusHosted on Google CodeMostly there, but some bits need work Sharding FailoverNeeds user feedbackNeeds more tests
  31. 31. Future DirectionsMake routing decisions optionally stickySupport explicit hints in method call attributes Helps with sharding Workaround for tricky queries that fool SQL::Parser
  32. 32. Thanks!http://code.google.com/p/dbix-router/
  33. 33. What else is out there?Mostly faux DBD drivers DBD::Multi DBD::Multiplex DBIx::HADBI::Role
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×