CouchConf-Chicago-Couchbase-in-production

Couchbase Server 2.0 in Production Matt Ingenthron Couchbase Inc. 1

Couchbase Server 2.0: Overview = Simple. Fast. Elastic. • Membase + CouchDB • Managed memory caching layer = super high performance • Clustering and online data redistribution (Rebalancing) • Indexing and Querying via JSON Map-Reduce • New SDK’s and client libraries • Developer Preview available today! • http://www.couchbase.com/downloads 2

Let’s build a social game… www.facebook.com/farm_town_wars Load Balancer Web Servers Couchbase Servers 3

COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE Test Deploy Maintain 4

COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE Initial Setup Test Deploy View Development Maintain 4

COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE Client setup Initial Setup Test Deploy View Development Sizing Maintain 4

COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE Client setup Initial Setup Test Deploy View Development Sizing Monitor Grow Maintain Upgrade Backup/Restore Failures 4

COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE Initial Setup Test View Development 4

Couchbase Server 2.0 in Production: Initial Setup Extremely easy to get up and running: • RPM/deb/OSX/exe installation • Simple Web UI and setup wizard 5

Couchbase Server 2.0 in Production: Data and Indexes• Data goes “in and out” via the Memcached protocol• Queries/indexes are created and accessed via an HTTP protocol• Both are available separately, Couchbase-provided SDK’s will expose a single API to the developer and abstract the actual traffic 6

Couchbase Server 2.0 in Production: Data and Indexes• Indexes/views are based on incremental map-reduce: • Indexes are updated with incremental changes (not batch)• View processing is done per-vbucket: • Parallel processing on subset of data • Couchbase provides scatter-gather aggregation 7

Couchbase Server 2.0 in Production: Data and Indexes• Views are “developed” off of a random (or speciﬁc) sampling of the overall dataset and then deployed • Faster • Less load on system• Updated views can be applied without rescanning entire dataset 8

Couchbase Server 2.0 in Production: Data and Indexes (Demo) 9

Couchbase Server 2.0 in Production: Data and IndexesExample JSON Document:{ "_id": "Keith4540", "jsonType": "player", "name": "Keith4540", "level": 4, … …} (Demo) 9

Couchbase Server 2.0 in Production: Data and IndexesExample JSON Document:{ "_id": "Keith4540", "jsonType": "player", "name": "Keith4540", "level": 4, + … …} (Demo) 9

Couchbase Server 2.0 in Production: Data and IndexesExample JSON Document: Example view:{ function (doc) "_id": "Keith4540", { if (doc.jsonType == "player") "jsonType": "player", + { "name": "Keith4540", emit(["Level", doc.level], doc._id); "level": 4, } } … …} (Demo) 9

Couchbase Server 2.0 in Production: Data and IndexesExample JSON Document: Example view:{ function (doc) "_id": "Keith4540", { if (doc.jsonType == "player") "jsonType": "player", + { "name": "Keith4540", emit(["Level", doc.level], doc._id); "level": 4, } } … …} = (Demo) 9

Couchbase Server 2.0 in Production: Data and IndexesExample JSON Document: Example view:{ function (doc) "_id": "Keith4540", { if (doc.jsonType == "player") "jsonType": "player", + { "name": "Keith4540", emit(["Level", doc.level], doc._id); "level": 4, } } … …} = Secondary index of players, by level (Demo) 9

COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE Client setup Deploy Sizing 10

COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE 10

Couchbase Server 2.0 in production: Sizing Sizing Question: How many nodes do I need? Considerations: RAM Disk Network Data Safety 11

Couchbase Server 2.0 in production: Sizing Sizing Question: How many nodes do I need? Considerations: RAM • Metadata • Working set Active+Replica • Buffer/overhead 12

Couchbase Server 2.0 in production: Sizing 500,000 keys to begin with: -20 bytes in length -average value size of 2k -metadata of about 150 bytes per key = ~1Gb to store active data, an extra 1Gb to store replica dataAdding in some headroom: Give 3GB RAM to Couchbase to start and grow with 13

Couchbase Server 2.0 in production: Sizing Sizing Question: How many nodes do I need? Considerations: Disk: • Sustained write rate • Index generation (space and IO) • Append-only B-Tree • Compaction 14

Couchbase Server 2.0 in production: Sizing Sizing Question: How many nodes do I need? Considerations: Network: • Client traffic • Replication • Rebalancing 15

Couchbase Server 2.0 in production: Sizing Sizing Question: How many nodes do I need? Considerations: Data Safety1 node = single point of failure (bad)2 nodes = 1 replica copy (better)3+ nodes = 1 replica copy AND data/load distribution 16

Couchbase Server 2.0 in production: Sizing Sizing Question: How many nodes do I need? Considerations: RAM Disk Network Data Safety 17

Couchbase Server 2.0 in production: Client-side Deployment Farm Town Wars Farm Town Wars App Code App Code Application server Application server Couchbase PHP Client Couchbase Java Library Client library Moxi (Couchbase proxy) OR 11210 5984 11210 5984 Query API Query API Couchbase Server Couchbase Couchbase Couchbase Server Client-side Moxi (“smart”) library 18

COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE Monitor Grow Maintain Upgrade Backup/Restore Failures 19

COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE 19

Couchbase Server 2.0 in production: MonitoringIMMENSE amount of information available•Real-time traffic graphs•REST API accessible•Per bucket, per node and aggregate statistics•Application and inter-node traffic•RAM <-> Disk•Inter-system timing 20

Couchbase Server 2.0 in production: Growth (Demo) 22

Couchbase Server 2.0 in production: GrowthGoing from 1 hundred users to 2 million users… – RAM usage is growing: • Ejecting data to and fetching data from disk • Resident item ratios decreasing – might impact failover – Disk write queue grows higher than usual (Demo) 22

Couchbase Server 2.0 in production: GrowthGoing from 1 hundred users to 2 million users… – RAM usage is growing: • Ejecting data to and fetching data from disk • Resident item ratios decreasing – might impact failover – Disk write queue grows higher than usualNeed to add a few more nodes... (Demo) 22

Couchbase Server 2.0 in production: GrowthGoing from 1 hundred users to 2 million users… – RAM usage is growing: • Ejecting data to and fetching data from disk • Resident item ratios decreasing – might impact failover – Disk write queue grows higher than usualNeed to add a few more nodes...Now we have more RAM and more disk throughput without anydowntime (Demo) 22

Couchbase Server 2.0 in production: General Maintenance• Persistence is CouchDB: – Append-only B-tree – Reliability and data integrity – Constantly growing on-disk ﬁles• Compaction is the answer: – Automatic – “Fragmentation” data – Scheduled compaction – Partial compaction via per-vbucket database (saves disk space and time) 23

Couchbase Server 2.0 in production: General Maintenance• Backup/Restore: – Backup is as simple as a ﬁle-level copy (thanks CouchDB!) – Server will automatically “warmup” from disk ﬁles upon reboot• Upgrade: – Add nodes of new version, rebalance… – Remove nodes of old version, rebalance… – Done! – No disruption – Upgrade from existing Membase 1.7 installations to Couchbase 2.0 24

Couchbase Server 2.0 in production: Failures • Failures happen: – Hardware – Network – Bugs • Failover to replica data for immediate access • Remove and rebalance “malfunctioning” node (Demo)25

COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE Client setup Initial Setup Test Deploy View Development Sizing Monitor Grow Maintain Upgrade Backup/Restore Failures 26

COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE Initial Setup Test View Development 26

THANK YOU! Q&AMatt Ingenthron - matt@couchbase.com @ingenthr 27

CouchConf-Chicago-Couchbase-in-production

by Couchbase on Nov 22, 2011

Accessibility

Categories

Upload Details

Usage Rights

Statistics

CouchConf-Chicago-Couchbase-in-production Presentation Transcript