2go ScaleConf 2012

Introducing 2go• Instant messenger for Java phones• Send messages to friends• Share photos and ﬁles• Connect with other IM networks• Meet others in chat rooms• And more...

Why it’s popular• Cheaper than SMS (1 cent/message)• Fast on slow networks• Designed for mobile phones• It’s social!

The Beginning• Founded by Alan Wolff, Ashley Peter•6 months of coding, learning, not studying• Clueless about ‘scaling’•1 desktop PC as server• Launched in March 2007

Registered Users20 1610 1Day

Registered Users300 254150 1 Day 1 Week

Registered Users1 500 1 159 750 1 Week 1 Month

Registered Users150 000 124 500 75 000 1 Month 1 Year

Registered Users15 100 000 15 000 000+ 7 550 000 1 Year Today

2go Today• Over 15 million users• Users in over 150 countries • Mostly in Africa (Nigeria, South Africa, Kenya)• Fastest rising Google search term in Nigeria and Kenya in 2010• 200 million messages per day• 20 million logins per day• 45 thousand signups per day

Why we’re at ScaleConf• Raise awareness about scaling • Not a common problem • Not covered in formal education• Learn from others• Share our lessons with you

Scaling

What is ‘scaling’? Ability to accommodategrowing volumes of users on your network.

Why is scaling important? Consequences of not scaling: Slow service OR No service

Why is scaling important?All high demand services will eventually face scaling challenges.

3 scaling techniques1.Vertical scaling • Upgrade hardware2.Parallelism • Execute in parallel3.Horizontal scaling • Division

How does 2go scale?1.Vertical scaling • Upgrade hardware USE A2.Parallelism LL TH • Execute R in parallel EE!3.Horizontal scaling • Division

Data layerApplication layerOS & network layer

Traditional Websites DataJohn Server John’s Data

Traditional Websites Data• Users interact with own data• Users expect RAM speeds • Easy to keep ‘hot’ data in RAM• Normally 1-2% of users are concurrent• Website grows, add more servers

Traditional Websites Data John’s Sam’sJohn Server Sam Server Data Data Mary’s Elina’sMary Server Elina Server Data Data

Social Networks Data John’s Data Sara’s Data James’ DataJohn Server Julie’s Data ... Chris’ Data

Social Networks Data John’s DataJohn Server Sara’s DataSara Server James’ Data Anni’s DataJames Server ...Anni Server Chris’ Data

Social Networks Data• Users have many (100+) friends• Users interact with friends’ data• Data access is geometric

Quick Example• 600 users login per second• Each user has 100 friends = 600*100 = 60k• Get 60k users’ data (name, status, image) = 60k * 3•= 180k objects per second• Not possible on 1 or 2 servers• Need 10+ DB servers!

Social Networks Data• Users have many (100+) friends• Users interact with friends’ data• Data access is geometric• Accessing180k objects in 1 second means hitting many DB servers• Difficult to keep ‘hot’ data in RAM

How we store & retrieve data MySQL persistent, disk based Memcached volatile, RAM based

Why do we use MySQL?• Reliable • Never had data corruption• Simple• Free• Good, helpful community• Widely used, well understood

How do we scale MySQL?• Vertical Scaling: • Disks, RAID• Parallelism: • Multiple connections to MySQL • MyISAM (default) has table locking, use InnoDB for row locking • Replication (scales reads)

MySQL Replication DB Master 500 reads/s 200 writes/s

MySQL ReplicationDB Master DB Master DB Slave1500 reads/s 250 reads/s 250 reads/s200 writes/s 200 writes/s 200 writes/s

MySQL ReplicationDB Master DB Slave1 DB Slave2 DB Slave32 reads/s 2 reads/s 2 reads/s 2 reads/s698 writes/s 698 writes/s 698 writes/s 698 writes/s Write saturation

How do we scale MySQL?• Parallelism: • Multiple connections to MySQL • MyISAM (default) has table locking, user InnoDB for row locking • Replication (scales reads)• Horizontal Scaling: • Split data onto multiple masters. ‘sharding’. (scales writes) • Bye bye relational DB. Joining data moves to application level

MySQL The biggest issue:MySQL stores data on disk.Users expect RAM speeds. We have a problem...

Enter Memcached...

What is Memcached?• Developed by Brad Fitzpatrick at LiveJournal• In-memory LRU distributed hash table• ‘Hot’ data stored in the cache• Manually managed cache

Why do we use Memcached?• It’s fast • Really.• Alleviates DB load• Distributed• Low latency• Also, it’s fast

Issues with Memcache:• Manually managed • Stale cache is bad • Manage with caution• Race conditions• Serialise data • Storing strings is inefficient • Use binary protocol• New connection overhead • Use connection pools or UDP• Multiget

Data Overviewquery = SELECT name from Users WHERE userId= 1234;result = getFromMemcache(user_name_1234);// Return cached result (FAST!)if (result != null) return result;// Get from DB (slow...)result = getFromDB(query);// Add to cache (fast next time!)putInMemcache(user_name_1234, result);return result;

Data layerApplication layerOS & Network layer

Application Layer• We use Java • Works well • Learn to tune JVM• Different backend services • Some services have multiple instances• Services communicate with messaging protocol

How do we scale applications?• Vertical Scaling: • Multicore CPUs, faster cores, more RAM• Parallelism: • Multithreaded applications • Connection pools• Horizontal Scaling: • Different services, split by functionality • Some services have multiple instances • Load balancing to instances via LVS

Data layerApplication layerOS & network layer

OS & network layers• We use Linux • Follow kernel developments • Apply relevant patches• Puppet• Tune Linux and shell to handle many connections • C10K problem• Experiment with virtualization

Data layerApplication layerOS & Network layer

General scaling tips• Start simple, don’t over engineer• Scaling is problem solving

Scaling cycle Problem!Apply ﬁx Isolate cause Understand cause

More scaling tips• Understand and consider the entire stack • Hardware (CPU, RAM, Disks, NIC) • OS (Paging, memory allocation, kernel) • Application layer (DB, language)• Step back• Look forward • Fix tomorrow’s problems before they become today’s

Wrapping up• It’s been an interesting journey• We’re opening in Cape Town • ...we’re recruiting!• We’d love to hear feedback

Thanksashley@2go.imwww.2go.im

http://vrindavijayan.blogspot.in	4
http://us-w1.rockmelt.com	4
http://a0.twimg.com	3
http://lanyrd.com	3
http://www.onlydoo.com	1

2go ScaleConf 2012

by 2go on Jan 31, 2012

Accessibility

Categories

Tags

Upload Details

Usage Rights

5 Embeds 15

Statistics

2go ScaleConf 2012 — Presentation Transcript