Redis to the Rescue?
Upcoming SlideShare
Loading in...5
×
 

Redis to the Rescue?

on

  • 19,098 views

Based in Berlin, wooga is the leading European social games developer. Social games offer interesting scaling challenges—when they become popular, the user base can grow quickly, often by 50.000 ...

Based in Berlin, wooga is the leading European social games developer. Social games offer interesting scaling challenges—when they become popular, the user base can grow quickly, often by 50.000 people per day or more.

This case study recounts how we successfully scaled up two Facebook games to one million daily active users each, why we decided to replace MySQL (once partially, once fully) with Redis, what difficulties we encountered on the way and how we solved them.

As an in-memory database, Redis offers an order-of-magnitude reduction in query roundtrip latency, but also introduces new challenges: How can you guarantee durability of data in the case of server outages? How do you best structure your data when there are no ad-hoc query capabilities?

This talk will go into technical details of our backend architecture and discuss both its advantages and disadvantages, how it stacks up against other possible setups, and what lessons we have learned.

Statistics

Views

Total Views
19,098
Views on SlideShare
9,500
Embed Views
9,598

Actions

Likes
32
Downloads
273
Comments
4

33 Embeds 9,598

http://www.wooga.com 5066
http://blog.nosqlfan.com 2579
http://nosql.mypopescu.com 1644
http://trumcuoi.com 113
http://www.dcshi.com 43
http://wiki 40
http://www.dedecms.com 28
url_unknown 22
http://paper.li 10
http://www.slideshare.net 7
http://news.google.com 5
http://translate.googleusercontent.com 5
http://bo.lt 4
http://www.hanrss.com 4
http://feeds.feedburner.com 3
http://webcache.googleusercontent.com 3
http://twitter.com 3
http://trunk.ly 2
http://www.zhuaxia.com 2
http://www.cit.cn 2
http://www.liebao.cn 1
https://www.google.ro 1
http://youssef.bo.lt 1
http://www.dakaifa.cn 1
http://a0.twimg.com 1
http://cache.baidu.com 1
http://www.fdselections.nl 1
http://xianguo.com 1
http://zhuaxia.com 1
http://www.trunk.ly 1
http://feedproxy.google.com 1
http://blog.slideshare.net 1
http://www.techgig.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Thanks for sharing
    Are you sure you want to
    Your message goes here
    Processing…
  • Facebook游戏开发商wooga的架构:基于MySQL和Redis
    Are you sure you want to
    Your message goes here
    Processing…
  • Those graphics are great :D Good presentation, I've been wanting to hear more real world stories on redis, thanks!
    Are you sure you want to
    Your message goes here
    Processing…
  • Awesome deck! Thanks for sharing...
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Redis to the Rescue? Redis to the Rescue? Presentation Transcript

  • Redis to the Rescue?O’Reilly MySQL Conference2011-04-13
  • Who?• Tim Lossen / @tlossen• Berlin, Germany• backend developer at wooga
  • Redis IntroCase 1: Monster WorldCase 2: Happy HospitalDiscussion
  • Redis IntroCase 1: Monster WorldCase 2: Happy HospitalDiscussion
  • What?• key-value-store• in-memory database• “data structure server”
  • Data Types• strings (integers)
  • Data Types• strings (integers)• lists• hashes
  • Data Types• strings (integers)• lists• hashes• sets• sorted sets
  • flickr.com/photos/atzu/2645776918
  • Performance• same for reads / writes
  • Performance• same for reads / writes• 50 K ops/second - regular notebook, EC2 instance
  • Performance• same for reads / writes• 50 K ops/second - regular notebook, EC2 instance• 200 K ops/second - intel core i7 X980 (3.33 GHz)
  • Durability• snapshots• append-only log
  • Other Features• master-slave replication• virtual memory• ...
  • Redis IntroCase 1: Monster WorldCase 2: Happy HospitalDiscussion
  • Daily Active UsersApril May June July Aug Sept Oct 2010
  • Daily Active UsersApril May June July Aug Sept Oct 2010
  • Challenge• traffic growing rapidly
  • Challenge• traffic growing rapidly• bottleneck: write throughput - EBS volumes pretty slow
  • Challenge• traffic growing rapidly• bottleneck: write throughput - EBS volumes pretty slow• MySQL already sharded - 4 x 2 = 8 shards
  • Idea• move write-itensive data to Redis
  • Idea• move write-itensive data to Redis• first candidate: inventory - integer fields - frequently changing
  • Solution• inventory = Redis hash - atomic increment / decrement !
  • Solution• inventory = Redis hash - atomic increment / decrement !• on-demand migration of users - with batch roll-up
  • Results• quick win - implemented in 2 weeks - 10% less load on MySQL servers
  • Results• quick win - implemented in 2 weeks - 10% less load on MySQL servers• decision: move over more data
  • But ...• “honeymoon soon over”
  • But ...• “honeymoon soon over”• growing memory usage (fragmentation) - servers need periodic “refresh” - replication dance
  • Current Status• hybrid setup - 4 MySQL master-slave pairs - 4 Redis master-slave pairs
  • Current Status• hybrid setup - 4 MySQL master-slave pairs - 4 Redis master-slave pairs• evaluating other alternatives - Riak, Membase
  • Redis IntroCase 1: Monster WorldCase 2: Happy HospitalDiscussion
  • Challenge• expected peak load: - 16000 concurrent users - 4000 requests/second - mostly writes
  • “Memory is the new Disk, Disk is the new Tape.” ⎯ Jim Gray
  • Idea• use Redis as main database - excellent (write) performance - virtual memory for capacity
  • Idea• use Redis as main database - excellent (write) performance - virtual memory for capacity• no sharding = simple operations
  • Data Model• user = single Redis hash - each entity stored in hash field (serialized to JSON)
  • Data Model• user = single Redis hash - each entity stored in hash field (serialized to JSON)• custom Ruby mapping layer (“Remodel”)
  • {“level”: 4,1220032045 u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1,1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
  • {“level”: 4,1220032045 u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1,1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
  • {“level”: 4,1220032045 u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1,1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
  • {“level”: 4,1220032045 u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1,1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
  • {“level”: 4,1220032045 u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1,1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
  • {“level”: 4,1220032045 u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1,1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
  • Virtual Memory• server: 24 GB RAM, 500 GB disk - can only keep “hot data” in RAM
  • Virtual Memory• server: 24 GB RAM, 500 GB disk - can only keep “hot data” in RAM• 380 GB swap file - 50 mio. pages, 8 KB each
  • Dec 2010: Crisis• memory usage growing fast
  • Dec 2010: Crisis• memory usage growing fast• cannot take snapshots any more - cannot start new slaves
  • Dec 2010: Crisis• memory usage growing fast• cannot take snapshots any more - cannot start new slaves• random crashes
  • Analysis• Redis virtual memory not compatible with: - persistence - replication
  • Analysis• Redis virtual memory not compatible with: - persistence - replication• need to implement our own!
  • Workaround• “dumper” process - tracks active users - every 10 minutes, writes them into YAML file
  • ruby redis disk
  • ruby redis disk
  • ruby redis disk
  • ruby redis disk
  • ruby redis disk
  • Workaround• in case of Redis crash - start with empty database - restore users on demand from YAML files
  • Real Solution• Redis “diskstore” - keeps all data on disk - swaps data into memory as needed
  • Real Solution• Redis “diskstore” - keeps all data on disk - swaps data into memory as needed• under development (expected Q2)
  • Results• average response time: 10 ms
  • Results• average response time: 10 ms• peak traffic: - 1500 requests/second - 15000 Redis ops/second
  • Current Status• very happy with setup - simple, robust, fast - easy to operate• still lots of spare capacity
  • Redis IntroCase 1: Monster WorldCase 2: Happy HospitalDiscussion
  • Advantages• order-of-magnitude performance improvement - removes main bottleneck - enables simple architecture
  • Disadvantages• main challenge: durability - diskstore very promising
  • Disadvantages• main challenge: durability - diskstore very promising• no ad-hoc queries - think hard about data model - hybrid approach?
  • Conclusion• Ruby + Redis = killer combo
  • Q&A
  • redis.iogithub.com/tlossen/remodelwooga.com/jobs