Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How Rackspace Cloud Monitoring uses Cassandra


Published on

Presented at Cassandra EU on 28 March 2012

Published in: Technology, Business
  • Be the first to comment

How Rackspace Cloud Monitoring uses Cassandra

  1. 1. A Cassandra Usage ExampleGary Dusbabek@gdusbabek
  2. 2. Goals CM Overview Control Cluster Data Cluster
  3. 3. CM OverviewThousands of serversPre-existing solutionsLessons learned from CloudkickInternal versus externalMillions of checks
  4. 4. TerminologyEntitySomething with an IP address or host name
  5. 5. TerminologyCheckTied to a entityIs an actionProduces metrics
  6. 6. TerminologyAlarmTied to an entityEvaluates check outputSpecified by DSL
  7. 7. TerminologyNotification PlanDetermines what to do for various alarm states
  8. 8. Features Remote checksFive datacenters collectors in five processing in three Alerting
  9. 9. 1,000 words
  10. 10. All REST all the time More Ops friendlyFeatures Agent Metrics
  11. 11. Future Automation Prediction Support hooksAgent expansion Correlation AggregationEntity Spanning
  12. 12. Two Clusters Data Control
  13. 13. Control ClusterMetadataStateThree datacentersHigh RFWide rowsEasy dump & load
  14. 14. Data ModelRich but simpleObjects used together stored together Simple parent-child relationsOne row per customer (tenant)Composite column names
  15. 15. Data Model Good: Bad:Single Parent/Child Complex Acyclic Multiple Parent/Child
  16. 16. Data Model Object types:Foo, Bar, Baz Relationship: 1:*
  17. 17. Data Model New Foo: 123Row Column Value Key Column Name
  18. 18. Data ModelNew Foo: 124
  19. 19. Data Model New Bar attached to fo123.ID is “fo123” ID is “fo123:ba001”
  20. 20. Data ModelNew Bar attached to fo123.
  21. 21. Data ModelNew Bar attached to fo124.
  22. 22. Data Model New Bar attached to fo123.ID is “fo123:ba003”
  23. 23. Data Model Baz attached to ba003ID is “fo123:ba001:bz001”
  24. 24. Data ModelSelect Foo children for 123:select „fo123‟..‟fo123:x7f‟;Select single Bar object attached to fo123:select „fo123:ba001‟..‟fo123:ba001:x7f‟;Object hierarchy reduced to a single slice.Parent keys available from context (no extra reads or indexes)
  25. 25. Control ClusterAPI server is Node.jsJavascript ORM library• Define object model in JS• Read/write entire objects• Never think about CQLnode-cassandra-client
  26. 26. Control Cluster
  27. 27. Data ClusterThe fun starts here
  28. 28. Data ClusterGoal: Fast graphsTime series dataFewer data pointsOK to shave resolutionRecent data is most important
  29. 29. GranularityFull, 5m, 20m, 80m, 320m, 1280m time-to-live (TTL)
  30. 30. Rollup ConceptsSlotPegged at 4096 slotsCircular buffer of metrics names thathave come into the systemKey is granularity + slotColumns index keys in rollup table
  31. 31. Keyed by asciiBigint column names Blob column values JDBC Rollups
  32. 32. ArrivalFull Resolution!time, name, several metricsmetric = name, type, valueCreate a locator assigned to slotInsert metricsSingle Cassandra APPLY BATCH;
  33. 33. Process: Rolling up• rollup(src_gran, from, to);• from, to determine slots• For each slot, get locator row slice – Cols tell us which metrics are active (need roll up) – For each column in locator row slice, slice in data row to get metrics • For each column (metric) in slice, add to calculations • Store rolled up calculation in dst_gran CF • Store locator column for slot, dst_gran• Also rollup(metric, src_gran, from, to);
  34. 34. Rollup operations are idempotent* Simplifies availabilityRollups are easily parallelized Partition the locator space
  35. 35. But…What if data arrives after rollup is performed?More than 24hrs late: don‟t care, forget itFigure out which slot the data belongs it, flag for each granularity in a meta rowRecalculate rollups
  36. 36. / HBase@gdusbabek
  37. 37. Image Creditssoccer clock corridor car controller objects 1 objects 2 robot fun blobs dish sand slot sushi airplane scale hourglass street