Scalarium and CouchDB

Scalarium and CouchDBJonathan Weiss@jweiss@scalariumPeritor GmbH

Scalarium.comEC2 Cluster Management  Automatic & dynamic conﬁguration over lifetime of a cluster via Chef runs  Self healing clusters  Auto scaling clusters  One click deployment  One click cluster cloning  Monitoring & alerting  Logs & accounting  Firewalls & backups  Access & rights management 2

Use case wooga.com One of wooga’s Facebook games  1.5 million DAU (Daily Active Users)  ~ 130 servers  ~ 150.000 requests per minute on average  Complex stack with different databases and replication  Multiple deployments of the game on multiple clusters 3

Scalarium.comhttp://www.ﬂickr.com/photos/av8pix/3391322123/ 4

DataRather static but important Dynamic and non-critical  Cluster configuration  Accounting & events  Server descriptions  Monitoring and metering and current state  Lifecycle events: setup, deploy, configure, shutdown  Applications and deployment definitions  EC2 assets like EBS, elastic IPs, SSH keys 5

DataRather static but important Dynamic and non-critical 6

Why CouchDB? High Availability Easy Replication Clustering Robustness Short Recovery Time Jan Lehnardt 7

Architecture 8

Architecture 9

Interaction 10

SimplyStoredRuby library with ActiveRecord inspired capabilities  Models  Associations  Generates map & reduce  Validations  Callbacks  Dynamic ﬁnders  S3 attachments  Paranoid deletehttp://github.com/peritor/simply_stored 11

SimplyStored 12

SimplyStored 13

Conﬂict & Resolution 14

Conﬂict & ResolutionRe-load and retry: Automatically done by SimplyStoredDiscard and report: Internal report 15

RockingChairIn-memory CouchDB as a Ruby library  Just a big Hash  Understands views generated by SimplyStored  Speeds up your tests  Tests can run in parallel  Nice for debugginghttp://github.com/jweiss/rocking_chair 16

Operations 17

BackupSlaves in different EC2 availability zones and regionsOffsite copy of data and index ﬁles 18

Hot Standby 19

Cold Standbyhttp://www.ﬂickr.com/photos/dnkemontoh/2680822579 20

Keeping views fresh Cronjobs for crawling all design docs and calling all views On master & standby! Rollout new views without using them ﬁrst And while we are there, compact views&dbs 21

Performance 22

I/OCouchDB heavily depends on ﬁle system I/O  EBS RAIDs for better performance  Different volumes for data and indices  Be generous with RAM & CPUhttp://www.ﬂickr.com/photos/walkn/5471945439 23

include_docs Nice feature to load related data in one go…but slow  random I/O on read Emit the data you need to the view index 24

Design Documents Maintained in a framework/model-layer Using md5 of content to decide if update needed Whenever one view deﬁnition changed, all views got recomputed 25

Design DocumentsSeparate design document per viewFine granular control  Update  Delete  Compact 26

One DB to rule them all 27

Separate DBs 28

Wishlist Build-in caching include_docs to load from index Improved performance of building views Build-in solution for automatic sharding Automatic compaction Metrics & monitoring 29

Couchbase 2.0 30

Q&ATalk to us if you want to use Scalarium 3 month for free @scalarium 31

http://lanyrd.com	14
http://twitter.com	1
http://tweetedtimes.com	1

Scalarium and CouchDB

by Jonathan Weiss

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 16

Statistics

Scalarium and CouchDB Presentation Transcript