Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

HBaseConEast2016: OpenTSDB+BigTable

399 views

Published on

Danil Zburivsky, Big Data Practice Lead, Pythian and Max Luebbe, SRE, Google presentation from HBaseConEast2016: http://www.meetup.com/HBase-NYC/events/233024937/

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

HBaseConEast2016: OpenTSDB+BigTable

  1. 1. OpenTSDB + Bigtable Integrating time series database with Google Cloud Bigtable Danil Zburivsky, Big Data Practice Lead, Pythian — zburivsky@pythian.com Max Luebbe, SRE, Google — maxluebbe@google.com
  2. 2. Pythian specializes in design, implementation, and management of systems that directly contribute to revenue and business success. History 19 years in business Growing at 30+% per year 400+ employees 300+ customers worldwide HQ Ottawa, Canada - global reach Technology agnostic = trusted advisor Deep expertise: Oracle, Oracle Apps, MySQL, AWS, SQL Server, Cassandra/DataStax, Azure, PostgreSQL, Cloudera, MapR, Hortonworks etc. Google Premier Partner Status (as of end Aug) 5 Certified Developers (soon to be 12) Dedicated Google Technical Champion Launch partner for: Kubernetes, Dataflow, Cloud SQL, Dataproc Integrated OpenTSDB with Bigtable DW Explorers Program Partner Upcoming BigQuery & Cloud ML Launch Partner
  3. 3. • (time, metric, value) • OS and apps metrics • Industrial equipment • Web traffic Time series data
  4. 4. • Volume can be explosive • Data arrival and access patterns are different Storing time series data is a challenge
  5. 5. • Volume can be explosive • Data arrival and access patterns are different Storing time series data is a challenge
  6. 6. OpenTSDB Architecture Server Server Server Server TSD TSD Hbase or Bigtable TSD RPC Hbase RPC or Hbase API Web UI Scripts/Alerting http TSD RPC
  7. 7. • Open source • Uses HBase as a data store • Data model optimized for TS • REST API OpenTSDB <metric_uid><timestamp><tagk1><tagv1>[...<tagkN><tagvN>] <col_t+1>[...<col_t+N>]
  8. 8. Origins of Bigtable • Updating search index in bulk: too slow! • Rather than a file system (GFS), needed random access • Generic & scalable, now powers many diverse products
  9. 9. Bigtable decouples compute & storage... Node Distributed filesystem Node Client Client Client Client Client Client Processing Storage Clients A B C D E
  10. 10. ...making scalability fast & seamless Node Distributed filesystem Node Client Client Client Client Client Client Processing Storage Clients A B C D E Node
  11. 11. Google Cloud Bigtable Cloud Bigtable Bigtable Service
  12. 12. async-bigtable ● Uses standard HBase 1.0 API ● BufferedMutator ● Thread pool https://github.com/OpenTSDB/asyncbigtable
  13. 13. Future work ● Native Bigtable API ● Fully asynchronous ● Performance
  14. 14. Demo time!

×