Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to How Rackspace Cloud Monitoring uses Cassandra
Similar to How Rackspace Cloud Monitoring uses Cassandra (20)
More from gdusbabek
More from gdusbabek (8)
Recently uploaded
Recently uploaded (20)
How Rackspace Cloud Monitoring uses Cassandra
- 1. A Cassandra Usage Example Gary Dusbabek @gdusbabek
- 2. Goals CM Overview Control Cluster Data Cluster
- 3. CM Overview Thousands of servers Pre-existing solutions Lessons learned from Cloudkick Internal versus external Millions of checks
- 4. Terminology Entity Something with an IP address or host name
- 5. Terminology Check Tied to a entity Is an action Produces metrics
- 6. Terminology Alarm Tied to an entity Evaluates check output Specified by DSL
- 7. Terminology Notification Plan Determines what to do for various alarm states
- 8. Features Remote checks Five datacenters collectors in five processing in three Alerting
- 9. 1,000 words
- 10. All REST all the time More Ops friendly Features Agent Metrics https://github.com/racker/virgo
- 11. Future Automation Prediction Support hooks Agent expansion Correlation Aggregation Entity Spanning
- 12. Two Clusters Data Control
- 13. Control Cluster Metadata State Three datacenters High RF Wide rows Easy dump & load https://github.com/racker/cassandra-syncer
- 14. Data Model Rich but simple Objects used together stored together Simple parent-child relations One row per customer (tenant) Composite column names
- 15. Data Model Good: Bad: Single Parent/Child Complex Acyclic Multiple Parent/Child
- 16. Data Model Object types: Foo, Bar, Baz Relationship: 1:*
- 17. Data Model New Foo: 123 Row Column Value Key Column Name
- 18. Data Model New Foo: 124
- 19. Data Model New Bar attached to fo123. ID is “fo123” ID is “fo123:ba001”
- 20. Data Model New Bar attached to fo123.
- 21. Data Model New Bar attached to fo124.
- 22. Data Model New Bar attached to fo123. ID is “fo123:ba003”
- 23. Data Model Baz attached to ba003 ID is “fo123:ba001:bz001”
- 24. Data Model Select Foo children for 123: select „fo123‟..‟fo123:x7f‟; Select single Bar object attached to fo123: select „fo123:ba001‟..‟fo123:ba001:x7f‟; Object hierarchy reduced to a single slice. Parent keys available from context (no extra reads or indexes)
- 25. Control Cluster API server is Node.js Javascript ORM library • Define object model in JS • Read/write entire objects • Never think about CQL node-cassandra-client https://github.com/racker/node-cassandra-client
- 26. Control Cluster
- 27. Data Cluster The fun starts here
- 28. Data Cluster Goal: Fast graphs Time series data Fewer data points OK to shave resolution Recent data is most important
- 30. Granularity Full, 5m, 20m, 80m, 320m, 1280m time-to-live (TTL)
- 31. Rollup Concepts Slot Pegged at 4096 slots Circular buffer of metrics names that have come into the system Key is granularity + slot Columns index keys in rollup table
- 32. Keyed by ascii Bigint column names Blob column values JDBC Rollups
- 33. Arrival Full Resolution! time, name, several metrics metric = name, type, value Create a locator assigned to slot Insert metrics Single Cassandra APPLY BATCH;
- 34. Process: Rolling up • rollup(src_gran, from, to); • from, to determine slots • For each slot, get locator row slice – Cols tell us which metrics are active (need roll up) – For each column in locator row slice, slice in data row to get metrics • For each column (metric) in slice, add to calculations • Store rolled up calculation in dst_gran CF • Store locator column for slot, dst_gran • Also rollup(metric, src_gran, from, to);
- 35. Rollup operations are idempotent* Simplifies availability Rollups are easily parallelized Partition the locator space
- 36. But… What if data arrives after rollup is performed? More than 24hrs late: don‟t care, forget it Figure out which slot the data belongs it, flag for each granularity in a meta row Recalculate rollups
- 37. http://www.flickr.com/photos/nateone/3768979925 / HBase @gdusbabek
- 38. Image Credits soccer http://www.flickr.com/photos/ollesvensson/4252196844/ apples http://www.flickr.com/photos/jean_koulev/2697677595/ clock http://www.flickr.com/photos/ollesvensson/4252196844/ corridor http://www.flickr.com/photos/orinrobertjohn/4354716077/ car http://www.flickr.com/photos/ellenm1/3541270451 controller http://www.flickr.com/photos/germanium/117612088/ model http://www.flickr.com/photos/mikeschinkel/2703438152/ objects 1 http://www.flickr.com/photos/radder86/389283265/ objects 2 http://www.flickr.com/photos/radder86/389283262 robot http://www.flickr.com/photos/kb35/430976324/ fun http://www.flickr.com/photos/pierrebedat/1095337445/ blobs http://www.flickr.com/photos/inl/5097547405/ dish http://www.flickr.com/photos/mrpbps/2862208028/ sand http://www.flickr.com/photos/backkratze/3480338854/ slot http://www.flickr.com/photos/andresrueda/2925383781 sushi http://www.flickr.com/photos/basykes/4348613931/ airplane http://www.flickr.com/photos/fhashemi/72489620/ scale http://www.flickr.com/photos/puuikibeach/4765115333 hourglass http://www.flickr.com/photos/aidanmorgan/2331754875/ street http://www.flickr.com/photos/nateone/3768979925/
Editor's Notes
- Would not scale to all.
- Self serviceExtensive dashboard12 platformsDeep analysis
- Secondary, nice to have, but not critical to monitoring.