Your SlideShare is downloading. ×
Utopia Kingdoms scaling case. From 4 users to 50.000+
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Utopia Kingdoms scaling case. From 4 users to 50.000+


Published on

Describing the real life case of Utopia Kingdoms, an online game. The game had initially problems scaling on production environment and had to be greatly refactored to support large number of players. …

Describing the real life case of Utopia Kingdoms, an online game. The game had initially problems scaling on production environment and had to be greatly refactored to support large number of players. This includes use of caching, profiling, queuing system and the migration of database from Amazon SimpleDB to MongoDB.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. ● scaling case. From 4 users to 90k+ ● ● Jaime Buelta ● Soft. Developer at ●
  • 2. The Game
  • 3. Get image from game
  • 4. Utopia Kingdoms● Fantasy strategy game● Build your own Kingdom● Create armies and attack other Kingdoms● Join other Kingdoms in an Alliance● Manage resources● Available in Facebook and Kongregate
  • 5. Technology stack
  • 6. Technology Stack - Backend Python Cherrypy framework Amazon SimpleDB Linux in Amazon EC2
  • 7. Stack of technologies - Frontend HTML( generated by Genshi templates) jQuery
  • 8. Stack of technologies - Frontend HTML( generated by Genshi templates) jQuery
  • 9. Some points of interest (will discuss them later)● Your resources (population, gold, food, etc) grows with time● You actions (build something, attack a player) typically takes some time● Players are ranked against the rest● You can add friends and enemies
  • 10. Do not guessMeasure
  • 11. Measurement tools● OS tools ● Task manager (top) ● IO Monitor (iostat)● Monitoring tools (Munin, Nagios)● Logs ● Needs to find a good compromise detailed/relevance● Profiling
  • 12. Youve got to love profiling● Generate profiles with cProfile module Profile whole application with python -m cProfile -o (not very useful in a web app)● If youre using a framework, profile only your functions to reduce noise
  • 13. Profile decorator (example)def profile_this(func): import cProfile prof = cProfile.Profile() retval = prof.runcall(func) filename = profile-{ts}.prof.format(time.time()) prof.dumpstats(filename) return retval
  • 14. Analyzing profile● gprof2dot ● Using dot, convert to graph gprof2dot -f pstats | dot -Tpng -o file.png ● Good for workflows● RunSnakeRun ● Good for cumulative times
  • 15. Example of RunSnakeRun RAZR
  • 16. Example of gprof2dot
  • 17. The power of cache
  • 18. All static should be out of python● Use a good web server to serve all static content (Static HTML, CSS, JavaScript code)● Some options ● Apache ● Nginx ● Cherokee ● Amazon S3
  • 19. Use memcached(and share the cache between your servers)
  • 20. Example● Asking for friends/enemies to DB ● Costly request in SimpleDB (using SQL statement)● On each request● Cache the friends on memcache for 1 hour● Invalidate the cache if adding/removing friends or enemies
  • 21. Caching caveats● Cache only after knowing there is a problem● Do not trust in cache for storage● Take a look on size of cached data● Choosing a good cache time can be difcult / Invalidate cache can be complex● Some data is too dynamic to be cached
  • 22. Caching is not just memcached● More options available: ● Get on memory on start ● File cache ● Cache client side
  • 23. Parse templates just once● The template rendering modules have options to parse the templates just once● Be sure to activate it in production● In development, youll most likely want to parse them each time● Same apply to regex, specially complex ones
  • 24. More problems
  • 25. Rankings● Sort players on the DB is slow when you grow the number of players● Solution: ● Independent ranking server (operates just in memory) ● Works using binary trees ● Small Django project, communicates using xmlrpc● Inconvenient: ● Data is not persistent, if the rankings server goes down, needs time to reconstruct the rankings
  • 26. Database pulling - Resources● There was a process just taking care of the growth of resources. ● It goes element by element, and increasing the values ● It pulls the DB constantly, even when the user has their values to maximum● Increment the resources of a user just the next time is accessed (by himself or by others) ● No usage of DB when the user is not in use ● The request already reads from DB the user
  • 27. Database pulling - Actions● Lots of actions are delayed. Recruit a unit, buildings, raids...● A process check each user if an action has to be done NOW. ● Tons of reads just to check “not now” ● Great delay in some actions, as they are not executed in time
  • 28. Database pulling - Actions● Implement a queue to execute the actions at the proper time: ● Beanstalk (allows deferred extraction) ● A process listen to this queue and performs the action, independently from request servers. ● The process can be launched in a diferent machine. ● Multiple process can extract actions faster.
  • 29. DataBase Issues
  • 30. Amazon SimpleDB● Key – Value storage● Capable of SQL queries● Store a dictionary (schemaless, multiple columns)● All the values are strings● Access through boto module● Pay per use
  • 31. Problems with SimpleDB● Lack of control ● Cant use local copy – In development, you must access Amazon servers (slow and costly) ● Cant backup except manually ● Cant analyze or change DB (e.g. cant define indexes) ● Cant monitor DB
  • 32. Problems with SimpleDB● Bad tool support● Slow and high variability (especially on SQL queries) ● Sometime, the queries just timeout and had to be repeated.
  • 33. Migrate to MongoDB
  • 34. MongoDB● NoSQL● Schemaless● Fast● Allow complex queries● Retain control (backups, measure queries, etc)● Previous experience using it from ChampMan
  • 35. Requisites of the migration● Low-level approach● Objects are basically dictionaries● Be able to save dirty fields (avoid saving unchanged values)● Log queries to measure performance
  • 36. MongoSpell● Thin wrap over pymongo● Objects are just dictionary-like elements● Minimal schema● Fast!● Able to log queries● It will probably be released soon as Open Source
  • 37. Definition of collectionsclass Spell(Document): collection_name = spells needed_fields = [name, cost, duration] optional_fields = [ elemental, ] activate_dirty_fields = True indexes = [name__unique, cost]
  • 38. Querying from DBSpell.get_from_db(name=fireball)Spell.filter()Spell.filter(sort=name)Spell.filter(name__in=[fireball, magic missile])Spell.filter(elemental__fire__gt=2)Spell.filter(duration__gt=2, cost=3, hint=cost)Spell.filter(name=fireball, only=cost)
  • 39. Some features● Dirty fields● No type checks● Query logs● 10x faster than SimpleDB!!!
  • 40. Query logs[07:46:06]- 2.6 ms – get_from_db - Reinforcement -[07:46:06]- 4.3 ms - get_from_db - Player -[07:46:10]- 0.1 ms - filter - Membership-[07:46:10]- 1.3 ms - get_from_db - Reinforcement[07:46:10]- 1.4 ms - get_from_db - Notifications - (56)
  • 41. Scalability vs Efciency
  • 42. Scalable vs Efcient Scalable Efficient● Can support more ● Can support more users adding more users with the same elements elements Work on both to achieve your goals
  • 43. Keep measuring and improving!(and monitor production to be proactive)
  • 44. Thank you for your interest! Questions?