Sharding Architectures

Sharding Architectures International PHP Conference 2008

Who the f*** is talking? David Soria Parra <dsp@php.net> PHP since 7 years Software Engineer at Sun Microsystems Sun Microsystems Web Stack PHP Source Committer Various other OSS Projects (Mercurial, etc)

Agenda Problem Possible Solutions ...and what not What is sharding? Implementation Optimization Migration

Problem Scalability What kind of scalability? High read and write trafﬁc Huge amount of data

Solutions

Master/Slave replication Master Slave Slave

Master/Master replication Master Master Master Slave Slave

Master/Master Master/Slave Master / Master unstable / no ofﬁcial support replication not distribution Master / Slave no write optimization replication not distribution

Sharding Shard Database Shard Shard Aggregation

What is sharding? Application side implementation Application side aggregation Splitting data across databases Parallelization

Sharding is not... sharding is... NOT replication / backup NOT clustering NOT just spreading

Implementation

shard 0 PHP shard 1 application shard 2

shard 0 user ID: 9 PHP shard 1 application shard 2

9%3=0 shard 0 user ID: 9 PHP shard 1 application shard 2

Shard Tables with a lot of writes Tables with a lot of reads Related data on one shard

Splitting: Tables User Feeds Database Posts

Splitting: Tables User Feeds Shard1 Shard2 Posts

Splitting: Data User Conﬁg Database Thread Post

Splitting: Data User Conﬁg User Conﬁg Shard1 Shard2 Thread Thread Post Post even user id odd user id

Data distribution

Criteria Fast lookup Few database connections Inexpensive reorganization Equal distribution

Modulo Shard Shard Shard Uid: 1-9 Uid: 10-19 Uid: 20-30

Modulo 2 4 6 Shard Shard Shard Shard Uid: 1-7 Uid: 8-15 Uid: 16-23 Uid: 24-30

Lookup table Shard Uid: 1-9 Lookup Shard Uid: 10-18 Shard Uid: 19-28

Optimizing

Optimizing caches (memcache) model optimization normalization? no! connection pools persistant connections (using mysqli+mysqlnd)

Optimizing II aggregation daemons persistent MySQL Proxy with LUA asynchronous queries (mysqlnd)

Optimizing III Shard 0 App Aggregation Shard 1 App Shard 2

Problems Joins, Unions, Intersections Grouping Selecting and projecting on groups Aggregation Primary keys Referential integrity (foreign keys) (De-)Normalization

Problems Global tables Sharing Tagging Invitations Search Lots of relations (due to normalization) between tables

Migration Find the bottleneck Refactoring of DB model needed? Prepare implementation prepare DBA layer write unit tests split existing data

Migration II Setup shards Migrate your code Test, Test, Test Check if everything runs smooth MySQL proxy DTrace

Scale!

Some code $conn->getConnection($userId)->query(); $conn->getGlobalConnection()->query();

Questions? david.soriaparra@sun.com http://blog.experimentalworks.net

http://www.codefutures.com	1915
http://blog.experimentalworks.net	183
http://www.planet-php.net	77
http://www.slideshare.net	77
http://dinamicas.emvideos.com.br	24
http://planet-php.org	14
http://www.phpeye.com	11
http://www.planet-php.org	10
http://codefutures.com	10
http://lj-toys.com	4
http://webcache.googleusercontent.com	4
http://translate.googleusercontent.com	2
http://static.slideshare.net	2
http://helwr.com	1
https://www.vtunnel.com	1
http://209.85.173.132	1
http://127.0.0.1:8795	1

Sharding Architectures

by guest0e6d5e on Oct 30, 2008

Accessibility

Categories

Tags

Upload Details

Usage Rights

17 Embeds 2,337

Statistics

Sharding Architectures — Presentation Transcript