Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Update on OpenTSDB and AsyncHBase


Published on

This year we'll talk about the joys of the HBase Fuzzy Row Filter, new TSDB filters, expression support, Graphite functions and running OpenTSDB on top of Google’s hosted Bigtable. AsyncHBase now includes per-RPC timeouts, append support, Kerberos auth, and a beta implementation in Go.

Published in: Software
  • Be the first to comment

Update on OpenTSDB and AsyncHBase

  1. 1. HBaseCon Update Distributed, Scalable Time Series Database Chris Larsen
  2. 2. Who Am I? (no really, who am I?) Chris Larsen Maintainer for OpenTSDB Software Engineer @ Yahoo! Monitoring Team
  3. 3. What Is OpenTSDB? Open Source Time Series Database Store trillions of data points Sucks up all data and keeps going Never lose precision Scales using HBase, Cassandra Or Bigtable
  4. 4. What good is it? Systems Monitoring & Measurement Servers Networks Sensor Data The Internet of Things SCADA Financial Data Scientific Experiment Results
  5. 5. Use Cases Backing store for Argus: Open source monitoring and alerting system 15 HBase Servers 6 month retention 10M writes per minute 95p query latency < 30 days = 200ms Moving to 200 node cluster writing at 100M/m
  6. 6. Use Cases ●Monitoring system, network and application performance and statistics 110 region servers, 10M writes/s ~ 2PB Multi-tenant and Kerberos secure HBase ~200k writes per second per TSD Central monitoring for all Yahoo properties Over 2 billion time series served
  7. 7. Some Other Users
  8. 8. What Are Time Series? Time Series: data points for an identity over time Typical Identity: Dotted string: web01.sys.cpu.user.0 OpenTSDB Identity: Metric: sys.cpu.user Tags (name/value pairs): host=web01 cpu=0
  9. 9. What Are Time Series? Data Point: Metric + Tags + Value: 42 + Timestamp: 1234567890 sys.cpu.user 1234567890 42 host=web01 cpu=0 ^ a data point ^
  10. 10. How it Works
  11. 11. Writing Data 1) Open Telnet style socket, write: put sys.cpu.user 1234567890 42 host=web01 cpu=0 2) ..or, post JSON to: http://<host>:<port>/api/put 3) .. or import big files with CLI No schema definition No RRD file creation Just write!
  12. 12. Querying Data Graph with the GUI CLI tools HTTP API Aggregate multiple series Simple query language To average all CPUs on host: start=1h-ago avg sys.cpu.user host=web01
  13. 13. HBase Data Tables tsdb - Data point table. Massive tsdb-uid - Name to UID and UID to name mappings tsdb-meta - Time series index and meta-data tsdb-tree - Config and index for hierarchical naming schema
  14. 14. Data Table Schema Row key is a concatenation of UIDs and time: metric + timestamp + tagk1 + tagv1… + tagkN + tagvN sys.cpu.user 1234567890 42 host=web01 cpu=0 x00x00x01x49x95xFBx70x00x00x01x00x00x01x00x00x02x00x00x02 Timestamp normalized on 1 hour boundaries All data points for an hour are stored in one row Enables fast scans of all time series for a metric …or pass a row key filter for specific time series with particular tags
  15. 15. New for OpenTSDB 2.2 ● Append writes (no more need for TSD Compactions) ● Row salting and random metric IDs ● Downsampling Fill Policies ● Query filters (wildcard, regex, group by or not) ● Storage Exception plugin for retrying writes ● Released February 2016
  16. 16. New for OpenTSDB 2.3 ● Graphite style expressions ● Cross-metric expressions ● Calendar based downsampling ● New data stores ● UID assignment plugin interface ● Datapoint write filter plugin interface ● RC1 released May 2016 ● New Committer, Jonathan Creasy
  17. 17. Fuzzy Row Filter How do you find a single time series out of 1 million? For a day? For a month?
  18. 18. Fuzzy Row Filter Instead of running a regex string comparator over each byte array formatted key… (?s)^.{9}(?:.{8})*Qx00x00x00x02 E(?:Q)x00x0F‡x42x2BE)(?:.{8})*$ TSDB query takes 1.6 seconds for 89,726 rows KEY Match -> m t1 tagk1 tagv1 No Match -> m t1 tagk1 tagv2 No Match -> m t1 tagk1 tagv1 tagk2 tagv3 No Match -> m t1 tagk1 tagv2 tagk2 tagv4 No Match -> m t1 tagk3 tagv5 No Match -> m t1 tagk3 tagv6 Match -> m t2 tagk tagv1 No Match -> m t2 tagk tagv2
  19. 19. Fuzzy Row Filter Use a byte mask! ● Use the bloom filter to skip-scan to the next candidate row. ● Combine with regex (after fuzzy filter) to filter further. FuzzyFilter{[FuzzyFilterPair{row_key=[18, 68, -3, -82, 120, 87, 56, -15, 96, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0], mask=[0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1]}]} Now it takes 0.239 seconds KEY Match -> m t1 tagk1 tagv1 No Match -> m t1 tagk1 tagv2 Skip -> m t1 tagk1 tagv1 tagk2 tagv3 m t1 tagk1 tagv2 tagk2 tagv4 m t1 tagk3 tagv5 m t1 tagk3 tagv6 Match -> m t2 tagk tagv1 No Match -> m t2 tagk tagv2
  20. 20. Fuzzy Row Filter Pros: ● Can improve scan latency by orders of magnitude ● Combines nicely with other filters Cons: ● All row keys for the match have to be the same, fixed length ● Doesn’t help much when matching the majority of a set OR if a set has uniform key lengths ● Doesn’t support bitmasks, only byte masks
  21. 21. AsyncHBase AsyncHBase is a fully asynchronous, multi- threaded HBase client Supports HBase 0.90 to 1.x Faster and less resource intensive than the native HBase client Support for scanner filters, META prefetch, “fail-fast” RPCs
  22. 22. AsyncHBase in YCSB ● New Yahoo! Cloud Serving Benchmark (YCSB) module for testing AsyncHBase ● Test Params: ○ 1 YCSB worker thread with workload A for run and load ○ Ran consecutive Async -> HBase -> Async -> HBase… (new YCSB JVM each run) 50 times ○ HBase 1.0.0 stock Apache with default configs ○ Local host, Macbook Pro ○ 10K rows written/read ○ Async writes for both
  23. 23. AsyncHBase in YCSB HBase Client Threads: 238 AsyncHBase Client Threads: 22
  24. 24. AsyncHBase in YCSB
  25. 25. AsyncHBase in YCSB
  26. 26. Upcoming in 1.8 ●Reverse Scanning ●Multi-Get requests ●Netty 4 ●Lots of bug fixes ○Stuck NSRE bugs ○Region client resource leaks
  27. 27. OpenTSDB on Bigtable ● Bigtable ○Hosted Google Service ○Client uses HTTP2 and GRPC for communication ● OpenTSDB heads home ○Based on a time series store on Bigtable at Google ○Identical schema as HBase ○Same filter support (fuzzy filters are coming)
  28. 28. OpenTSDB on Bigtable ● AsyncBigtable ○Implementation of AsyncHBase’s API for drop-in use ○ ○Uses HTable API ○Moving to native Bigtable API ● Thanks to Christos of Pythian, Solomon, Carter, Misha, and the rest of the Google Bigtable team ● bigtable/#
  29. 29. OpenTSDB on Cassandra ● AsyncCassandra - Implementation of AsyncHBase’s API for drop-in use ● Wraps Netflix’s Astyanax for asynchronous calls ● Requires the ByteOrderedPartitioner and legacy API ● Same schema as HBase/Bigtable ● Scan filtering performed client side ● May not work with future Cassandra versions if they drop the API
  30. 30. Community Salesforce Argus ●Time series monitoring and alerting ●Multi-series annotations ●Dashboards Thanks to Tom Valine and the Salesforce engineers alerting-d2941f67864#.ez7mbo3ek
  31. 31. Community Turn Splicer ●API to shard TSDB queries ●Locality advantage hosting TSDs on region servers ●Query caching Thanks to Jonathan Creasy and the Turn engineers
  32. 32. The Future of OpenTSDB
  33. 33. The Future Reworked query pipeline for selective ordering of operations Histogram support Flexible query caching framework Distributed queries Greater data store abstraction
  34. 34. More Information Thank you to everyone who has helped test, debug and add to OpenTSDB 2.2 and 2.3 including, but not limited to: Kyle, Ivan, Davide, Liu, Utkarsh, Andy, Anna, Camden, Can, Carlos, Hugo, Isaih, Kevin, Ping, Jonathan Contribute at Website: Documentation: Mailing List: Images