Scalarium and CouchDB

2,341 views

Published on

How and why Scalarium is using CouchDB to control and manage thousands of machines on Amazon EC2

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,341
On SlideShare
0
From Embeds
0
Number of Embeds
21
Actions
Shares
0
Downloads
12
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Scalarium and CouchDB

  1. 1. Scalarium and CouchDBJonathan Weiss@jweiss@scalariumPeritor GmbH
  2. 2. Scalarium.comEC2 Cluster Management   Automatic & dynamic configuration over lifetime of a cluster via Chef runs   Self healing clusters   Auto scaling clusters   One click deployment   One click cluster cloning   Monitoring & alerting   Logs & accounting   Firewalls & backups   Access & rights management 2
  3. 3. Use case wooga.com One of wooga’s Facebook games   1.5 million DAU (Daily Active Users)   ~ 130 servers   ~ 150.000 requests per minute on average   Complex stack with different databases and replication   Multiple deployments of the game on multiple clusters 3
  4. 4. Scalarium.comhttp://www.flickr.com/photos/av8pix/3391322123/ 4
  5. 5. DataRather static but important Dynamic and non-critical   Cluster configuration   Accounting & events   Server descriptions   Monitoring and metering and current state   Lifecycle events: setup, deploy, configure, shutdown   Applications and deployment definitions   EC2 assets like EBS, elastic IPs, SSH keys 5
  6. 6. DataRather static but important Dynamic and non-critical 6
  7. 7. Why CouchDB? High Availability Easy Replication Clustering Robustness Short Recovery Time Jan Lehnardt 7
  8. 8. Architecture 8
  9. 9. Architecture 9
  10. 10. Interaction 10
  11. 11. SimplyStoredRuby library with ActiveRecord inspired capabilities   Models   Associations  Generates map & reduce   Validations   Callbacks   Dynamic finders   S3 attachments   Paranoid deletehttp://github.com/peritor/simply_stored 11
  12. 12. SimplyStored 12
  13. 13. SimplyStored 13
  14. 14. Conflict & Resolution 14
  15. 15. Conflict & ResolutionRe-load and retry: Automatically done by SimplyStoredDiscard and report: Internal report 15
  16. 16. RockingChairIn-memory CouchDB as a Ruby library   Just a big Hash   Understands views generated by SimplyStored   Speeds up your tests   Tests can run in parallel   Nice for debugginghttp://github.com/jweiss/rocking_chair 16
  17. 17. Operations 17
  18. 18. BackupSlaves in different EC2 availability zones and regionsOffsite copy of data and index files 18
  19. 19. Hot Standby 19
  20. 20. Cold Standbyhttp://www.flickr.com/photos/dnkemontoh/2680822579 20
  21. 21. Keeping views fresh Cronjobs for crawling all design docs and calling all views On master & standby! Rollout new views without using them first And while we are there, compact views&dbs 21
  22. 22. Performance 22
  23. 23. I/OCouchDB heavily depends on file system I/O   EBS RAIDs for better performance   Different volumes for data and indices   Be generous with RAM & CPUhttp://www.flickr.com/photos/walkn/5471945439 23
  24. 24. include_docs Nice feature to load related data in one go…but slow  random I/O on read Emit the data you need to the view index 24
  25. 25. Design Documents Maintained in a framework/model-layer Using md5 of content to decide if update needed Whenever one view definition changed, all views got recomputed 25
  26. 26. Design DocumentsSeparate design document per viewFine granular control   Update   Delete   Compact 26
  27. 27. One DB to rule them all 27
  28. 28. Separate DBs 28
  29. 29. Wishlist Build-in caching include_docs to load from index Improved performance of building views Build-in solution for automatic sharding Automatic compaction Metrics & monitoring 29
  30. 30. Couchbase 2.0 30
  31. 31. Q&ATalk to us if you want to use Scalarium 3 month for free @scalarium 31

×