Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn

4. ©2016 Couchbase Inc. 4 Michael Kehoe Staff Site Reliability Engineer (SRE) - LinkedIn mkehoe@linkedin.com • Production-SRE team • Member of CBVT • Australian! • Contact • linkedin.com/in/michaelkkehoe • @matrixtek GOES HERE

9. ©2016 Couchbase Inc. 9 MonitoringTools – Couchbase REST API • http://docs.couchbase.com/admin/admin/REST/rest-bucket-stats.html • GET /pools/default/buckets/[bucket-name]/stats • JSON output format • 60 collections per metric

10. ©2016 Couchbase Inc. 10 MonitoringTools - cbstats • http://docs.couchbase.com/admin/admin/CLI/cbstats-intro.html • Command Line tool for viewing stats • 333+ Available stats • Cumulative and Snapshot

11. ©2016 Couchbase Inc. 11 MonitoringTools - cbstats • Average value size = ep_value_size/(curr_items_tot-ep_num_non_resident) • ep_value_size = Amount of RAM used to hold values in this bucket for this node • Curr_items_tot =Total count of active/replica items in this bucket for this node • Ep_num_non_resident =Total number of items not resident in RAM • 9567135872 / ( 28733039 – 26582747 ) = 4449.22 bytes

13. ©2016 Couchbase Inc. 13 MonitoringTools - cbstats • CbstatsTimings • Histogram that shows the timing of a number of internal operations • Commit to disk, background IO operations, GET ops • http://docs.couchbase.com/admin/admin/CLI/CBstats/cbstats-timing.html

22. ©2016 Couchbase Inc. 22 Key Statistics – Couchbase Server Metrics to Consider: • Operations • Cache miss (ep_cache_miss_rate) • Active/Replica vbuckets (vb_active_num/vb_replica_num) • Percentage of items in memory (vb_active_resident_items_ratio) • Disk Queue (ep_diskqueue_items) • Misdirected Requests (ep_num_not_my_vbuckets)

23. ©2016 Couchbase Inc. 23 Key Statistics – Couchbase Client Metrics to Consider: • Call-time latency • Measure GET’s/ SET’s separately • Hit-rate • Is the hit-rate what you expected • Errors • Timeouts retrieving objects • Unable to reach Couchbase-Server • See http://developer.couchbase.com/documentation/server/4.0/sdks/java-2.2/event-bus- metrics.html

27. ©2016 Couchbase Inc. 27 Key Statistics – Network – Connectivity • Ping - simple network connectivity test • Firewalls – make sure you have the correct ports open • See http://developer.couchbase.com/documentation/server/current/install/install-ports.html

29. ©2016 Couchbase Inc. 29 Key Statistics – Network – Capacity/ Utilization • Practical network capacity is ~85-90% of theoretical • E.g. 1Gb/s network interface can do 850-900Mb/s • Congested networks are problematic • Higher latency on responses • Slower replication • Collect stats from /proc/net/dev

30. ©2016 Couchbase Inc. 30 Key Statistics – Network – Capacity/ Utilization • Practical network capacity is ~85-90% of theoretical (1250 Mb/s) • E.g. 1Gb/s network interface can do 850-900Mb/s Average object size (bytes) 4,096 ID length (bytes) 32 Meta data size (bytes) 56 Reads 100,000 Writes 60,000 Replica count 1 Read network utilization 421,600,000 Write network utilizaation 502,080,000 Total network utilization 923,680,000 1.25 billion theoretical max remaining bandwidth 276,320,000