Scalarium and CouchDB

  • 1,731 views
Uploaded on

How and why Scalarium is using CouchDB to control and manage thousands of machines on Amazon EC2

How and why Scalarium is using CouchDB to control and manage thousands of machines on Amazon EC2

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,731
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
10
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Scalarium and CouchDBJonathan Weiss@jweiss@scalariumPeritor GmbH
  • 2. Scalarium.comEC2 Cluster Management   Automatic & dynamic configuration over lifetime of a cluster via Chef runs   Self healing clusters   Auto scaling clusters   One click deployment   One click cluster cloning   Monitoring & alerting   Logs & accounting   Firewalls & backups   Access & rights management 2
  • 3. Use case wooga.com One of wooga’s Facebook games   1.5 million DAU (Daily Active Users)   ~ 130 servers   ~ 150.000 requests per minute on average   Complex stack with different databases and replication   Multiple deployments of the game on multiple clusters 3
  • 4. Scalarium.comhttp://www.flickr.com/photos/av8pix/3391322123/ 4
  • 5. DataRather static but important Dynamic and non-critical   Cluster configuration   Accounting & events   Server descriptions   Monitoring and metering and current state   Lifecycle events: setup, deploy, configure, shutdown   Applications and deployment definitions   EC2 assets like EBS, elastic IPs, SSH keys 5
  • 6. DataRather static but important Dynamic and non-critical 6
  • 7. Why CouchDB? High Availability Easy Replication Clustering Robustness Short Recovery Time Jan Lehnardt 7
  • 8. Architecture 8
  • 9. Architecture 9
  • 10. Interaction 10
  • 11. SimplyStoredRuby library with ActiveRecord inspired capabilities   Models   Associations  Generates map & reduce   Validations   Callbacks   Dynamic finders   S3 attachments   Paranoid deletehttp://github.com/peritor/simply_stored 11
  • 12. SimplyStored 12
  • 13. SimplyStored 13
  • 14. Conflict & Resolution 14
  • 15. Conflict & ResolutionRe-load and retry: Automatically done by SimplyStoredDiscard and report: Internal report 15
  • 16. RockingChairIn-memory CouchDB as a Ruby library   Just a big Hash   Understands views generated by SimplyStored   Speeds up your tests   Tests can run in parallel   Nice for debugginghttp://github.com/jweiss/rocking_chair 16
  • 17. Operations 17
  • 18. BackupSlaves in different EC2 availability zones and regionsOffsite copy of data and index files 18
  • 19. Hot Standby 19
  • 20. Cold Standbyhttp://www.flickr.com/photos/dnkemontoh/2680822579 20
  • 21. Keeping views fresh Cronjobs for crawling all design docs and calling all views On master & standby! Rollout new views without using them first And while we are there, compact views&dbs 21
  • 22. Performance 22
  • 23. I/OCouchDB heavily depends on file system I/O   EBS RAIDs for better performance   Different volumes for data and indices   Be generous with RAM & CPUhttp://www.flickr.com/photos/walkn/5471945439 23
  • 24. include_docs Nice feature to load related data in one go…but slow  random I/O on read Emit the data you need to the view index 24
  • 25. Design Documents Maintained in a framework/model-layer Using md5 of content to decide if update needed Whenever one view definition changed, all views got recomputed 25
  • 26. Design DocumentsSeparate design document per viewFine granular control   Update   Delete   Compact 26
  • 27. One DB to rule them all 27
  • 28. Separate DBs 28
  • 29. Wishlist Build-in caching include_docs to load from index Improved performance of building views Build-in solution for automatic sharding Automatic compaction Metrics & monitoring 29
  • 30. Couchbase 2.0 30
  • 31. Q&ATalk to us if you want to use Scalarium 3 month for free @scalarium 31