Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Utopia Kindgoms scaling case: From 4 to 50K users


Published on

PyCon Ireland Talk 2011

  • Be the first to comment

  • Be the first to like this

Utopia Kindgoms scaling case: From 4 to 50K users

  1. 1. ● scaling case. From 4 users to 90k+ ● ● Jaime Buelta ● Soft. Developer at ●
  2. 2. The Game
  3. 3. Get image from game
  4. 4. Utopia Kingdoms● Fantasy strategy game● Build your own Kingdom● Create armies and attack other Kingdoms● Join other Kingdoms in an Alliance● Manage resources● Available in Facebook and Kongregate
  5. 5. Technology stack
  6. 6. Technology Stack - Backend Python Cherrypy framework Amazon SimpleDB Linux in Amazon EC2
  7. 7. Stack of technologies - Frontend HTML( generated by Genshi templates) jQuery
  8. 8. Stack of technologies - Frontend HTML( generated by Genshi templates) jQuery
  9. 9. Some points of interest (will discuss them later)● Your resources (population, gold, food, etc) grows with time● You actions (build something, attack a player) typically takes some time● Players are ranked against the rest● You can add friends and enemies
  10. 10. Do not guessMeasure
  11. 11. Measurement tools● OS tools ● Task manager (top) ● IO Monitor (iostat)● Monitoring tools (Munin, Nagios)● Logs ● Needs to find a good compromise detailed/relevance● Profiling
  12. 12. Youve got to love profiling● Generate profiles with cProfile module Profile whole application with python -m cProfile -o (not very useful in a web app)● If youre using a framework, profile only your functions to reduce noise
  13. 13. Profile decorator (example)def profile_this(func): import cProfile prof = cProfile.Profile() retval = prof.runcall(func) filename = profile-{ts}.prof.format(time.time()) prof.dumpstats(filename) return retval
  14. 14. Analyzing profile● gprof2dot ● Using dot, convert to graph gprof2dot -f pstats | dot -Tpng -o file.png ● Good for workflows● RunSnakeRun ● Good for cumulative times
  15. 15. Example of RunSnakeRun RAZR
  16. 16. Example of gprof2dot
  17. 17. The power of cache
  18. 18. All static should be out of python● Use a good web server to serve all static content (Static HTML, CSS, JavaScript code)● Some options ● Apache ● Nginx ● Cherokee ● Amazon S3
  19. 19. Use memcached(and share the cache between your servers)
  20. 20. Example● Asking for friends/enemies to DB ● Costly request in SimpleDB (using SQL statement)● On each request● Cache the friends on memcache for 1 hour● Invalidate the cache if adding/removing friends or enemies
  21. 21. Caching caveats● Cache only after knowing there is a problem● Do not trust in cache for storage● Take a look on size of cached data● Choosing a good cache time can be difcult / Invalidate cache can be complex● Some data is too dynamic to be cached
  22. 22. Caching is not just memcached● More options available: ● Get on memory on start ● File cache ● Cache client side
  23. 23. Parse templates just once● The template rendering modules have options to parse the templates just once● Be sure to activate it in production● In development, youll most likely want to parse them each time● Same apply to regex, specially complex ones
  24. 24. More problems
  25. 25. Rankings● Sort players on the DB is slow when you grow the number of players● Solution: ● Independent ranking server (operates just in memory) ● Works using binary trees ● Small Django project, communicates using xmlrpc● Inconvenient: ● Data is not persistent, if the rankings server goes down, needs time to reconstruct the rankings
  26. 26. Database pulling - Resources● There was a process just taking care of the growth of resources. ● It goes element by element, and increasing the values ● It pulls the DB constantly, even when the user has their values to maximum● Increment the resources of a user just the next time is accessed (by himself or by others) ● No usage of DB when the user is not in use ● The request already reads from DB the user
  27. 27. Database pulling - Actions● Lots of actions are delayed. Recruit a unit, buildings, raids...● A process check each user if an action has to be done NOW. ● Tons of reads just to check “not now” ● Great delay in some actions, as they are not executed in time
  28. 28. Database pulling - Actions● Implement a queue to execute the actions at the proper time: ● Beanstalk (allows deferred extraction) ● A process listen to this queue and performs the action, independently from request servers. ● The process can be launched in a diferent machine. ● Multiple process can extract actions faster.
  29. 29. DataBase Issues
  30. 30. Amazon SimpleDB● Key – Value storage● Capable of SQL queries● Store a dictionary (schemaless, multiple columns)● All the values are strings● Access through boto module● Pay per use
  31. 31. Problems with SimpleDB● Lack of control ● Cant use local copy – In development, you must access Amazon servers (slow and costly) ● Cant backup except manually ● Cant analyze or change DB (e.g. cant define indexes) ● Cant monitor DB
  32. 32. Problems with SimpleDB● Bad tool support● Slow and high variability (especially on SQL queries) ● Sometime, the queries just timeout and had to be repeated.
  33. 33. Migrate to MongoDB
  34. 34. MongoDB● NoSQL● Schemaless● Fast● Allow complex queries● Retain control (backups, measure queries, etc)● Previous experience using it from ChampMan
  35. 35. Requisites of the migration● Low-level approach● Objects are basically dictionaries● Be able to save dirty fields (avoid saving unchanged values)● Log queries to measure performance
  36. 36. MongoSpell● Thin wrap over pymongo● Objects are just dictionary-like elements● Minimal schema● Fast!● Able to log queries● It will probably be released soon as Open Source
  37. 37. Definition of collectionsclass Spell(Document): collection_name = spells needed_fields = [name, cost, duration] optional_fields = [ elemental, ] activate_dirty_fields = True indexes = [name__unique, cost]
  38. 38. Querying from DBSpell.get_from_db(name=fireball)Spell.filter()Spell.filter(sort=name)Spell.filter(name__in=[fireball, magic missile])Spell.filter(elemental__fire__gt=2)Spell.filter(duration__gt=2, cost=3, hint=cost)Spell.filter(name=fireball, only=cost)
  39. 39. Some features● Dirty fields● No type checks● Query logs● 10x faster than SimpleDB!!!
  40. 40. Query logs[07:46:06]- 2.6 ms – get_from_db - Reinforcement -[07:46:06]- 4.3 ms - get_from_db - Player -[07:46:10]- 0.1 ms - filter - Membership-[07:46:10]- 1.3 ms - get_from_db - Reinforcement[07:46:10]- 1.4 ms - get_from_db - Notifications - (56)
  41. 41. Scalability vs Efciency
  42. 42. Scalable vs Efcient Scalable Efficient● Can support more ● Can support more users adding more users with the same elements elements Work on both to achieve your goals
  43. 43. Keep measuring and improving!(and monitor production to be proactive)
  44. 44. Thank you for your interest! Questions?