Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

HBaseCon 2013: OpenTSDB at Box

7,485 views

Published on

Presented by: Jonathan Creasy (Box) and Geoffrey Anderson (Box)

Published in: Technology, Business
  • Be the first to comment

HBaseCon 2013: OpenTSDB at Box

  1. 1. OpenTSDB at Box #HBaseCon2013 Jonathan Creasy Geoffrey Anderson @geodbz
  2. 2. Jonathan Creasy • SysAdmin @ Box, Inc. • Hadoop for Analytics
  3. 3. Geoffrey Anderson • DBA @ Box, Inc. • Tooling for MySQL and HBase • #DBHangOps
  4. 4. The Situation
  5. 5. •Storing •RRDs •Flat files •Pre-defined •Graphs •Data to collect •Poll model These are problematic because...
  6. 6. Enter OpenTSDB
  7. 7. OpenTSDB is... • Distributed • Scalable • Time Series Database • Runs on HBase • Created By Benoit Sigoure HBase TSD for Querying mydb.example.com HAProxy fe1.example.com TSD for Storing Push Metrics Query via API
  8. 8. • FAST • EASY to Scale • EASY to Populate • EASY to collect data • EASY to Query Why OpenTSDB?
  9. 9. Collecting Data
  10. 10. #!/usr/bin/env bash timestamp=$(date +%s) mysql -ss -e "SHOW GLOBAL STATUS" | while read var val do echo "mysql.$var $timestamp $val host=$HOSTNAME" done ganderson@mydb.example.com:~$ _./mysql_collector.sh mysql.Aborted_connects 1366399993 0 host=mydb.example.com mysql.Binlog_cache_disk_use 1366399993 0 host=mydb.example.com mysql.Binlog_cache_use 1366399993 0 host=mydb.example.com mysql.Binlog_stmt_cache_disk_use 1366399993 0 host=mydb.example.com mysql.Binlog_stmt_cache_use 1366399993 0 host=mydb.example.com mysql.Bytes_received 1366399993 19453687 host=mydb.example.com mysql.Bytes_sent 1366399993 1238166682 host=mydb.example.com mysql.Com_admin_commands 1366399993 1 host=mydb.example.com mysql.Com_assign_to_keycache 1366399993 0 host=mydb.example.com ... Example: mysql_collector.sh
  11. 11. #!/usr/bin/env bash timestamp=$(date +%s) mysql -ss -e "SHOW GLOBAL STATUS" | while read var val do echo "mysql.$var $timestamp $val host=$HOSTNAME" done ganderson@mydb.example.com:~$ _./mysql_collector.sh mysql.Aborted_connects 1366399993 0 host=mydb.example.com mysql.Binlog_cache_disk_use 1366399993 0 host=mydb.example.com mysql.Binlog_cache_use 1366399993 0 host=mydb.example.com mysql.Binlog_stmt_cache_disk_use 1366399993 0 host=mydb.example.com mysql.Binlog_stmt_cache_use 1366399993 0 host=mydb.example.com mysql.Bytes_received 1366399993 19453687 host=mydb.example.com mysql.Bytes_sent 1366399993 1238166682 host=mydb.example.com mysql.Com_admin_commands 1366399993 1 host=mydb.example.com mysql.Com_assign_to_keycache 1366399993 0 host=mydb.example.com ... Example: mysql_collector.sh Metric name Timestamp Value “Tags” (key=val)
  12. 12. * * * * * mysql_collector.sh | nc opentsdb.example.com 4242 Example: adding a cron for OpenTSDB
  13. 13. ganderson@mydb.example.com:tcollector$ tree . |-- collectors | |-- 0 | | |-- ifstat.py | | |-- iostat.py | | |-- procnettcp.py | | |-- procstats.py | |-- 15 | | `-- dfstat.py | |-- 30 | | |-- mysql_collector.sh | |-- 300 | | `-- ptTcpModel.sh | `-- etc | |-- config.py |-- config |-- startstop `-- tcollector.py Run forever Run every 15 seconds Run every 5 minutes Run every 30 seconds
  14. 14. Querying Data
  15. 15. http://opentsdb.example.com /#start=2013/06/05-17:00:00 &end=2013/06/05-19:00:00 &m=sum:hadoop.hbase.regionserver.requests {server_type=dwh-data} &o=axis x1y1 &m=sum:proc.stat.cpu.percentage_iowait {server_type=dwh-data,dc=lv7,host=data08} &o=axis x1y2 &ylabel=HBase Requests &y2label=&CPU IOWait &yrange=[0:] &wxh=1475x600
  16. 16. http://opentsdb.example.com /q?start=2013/06/05-17:00:00 &end=2013/06/05-19:00:00 &m=sum:hadoop.hbase.regionserver.requests {server_type=dwh-data} &o=axis x1y1 &m=sum:proc.stat.cpu.percentage_iowait {server_type=dwh-data,dc=lv7,host=data08} &o=axis x1y2 &ylabel=HBase Requests &y2label=&CPU IOWait &yrange=[0:] &wxh=1475x600 &ascii
  17. 17. How does this change things?
  18. 18. In all seriousness, though... • Easily see aggregate graphs • Easily build graphs on-the-fly • Full granularity forever • API request for raw data • Cluster-wide nagios checks with check_tsd
  19. 19. Challenges Switching • Aggregates are the default • Mouse-zooming (patched!) • Auto-suggest for metrics • “The graphs aren’t pretty” • Migrating from proof of concept • Plan for 5+ machines • Data pruning may be required
  20. 20. Some Quick Numbers OpenTSDB @ Box • 24,448 metrics • 79 tag keys • 5,371,701 tag values • 150,000 data points per second
  21. 21. To store metric data for anything that is measurable Collection Philosophy
  22. 22. Next Steps
  23. 23. Enjoy #Hbasecon2013! https://www.box.com/about-us/careers/ jcreasy@box.com geoff@box.com We’re Hiring!
  24. 24. Image credits • http://upload.wikimedia.org/wikipedia/commons/7/7b/Batelco_Network_Operations_Centre_(NOC).JPG • http://www.flickr.com/photos/hoyvinmayvin/5873697252/ • http://www.percona.com/doc/percona-monitoring-plugins • http://www.2cto.com/uploadfile/2012/0731/20120731112415744.jpg • http://media.tumblr.com/tumblr_lvfspoenWU1qi19a2.png • http://img.izismile.com/img/img4/20110527/640/you_can_be_a_superhero_640_01.jpg • http://openclipart.org/image/250px/svg_to_png/26427/Anonymous_notebook.png • http://images.alphacoders.com/768/2560-1600-76893.jpg • http://www.flickr.com/photos/in365/4861180503/ • http://openclipart.org/image/250px/svg_to_png/130915/Prohibido_3D.png • http://www.flickr.com/photos/61114149@N02/5566484951/ • http://opentsdb.net/img/tsd-sample.png • http://images2.wikia.nocookie.net/__cb20080911160202/bttf/images/5/57/WhatdidItellyou-HQ.jpg • http://www.flickr.com/photos/lisakayaks/3028350539/ • http://www.flickr.com/photos/25566302@N00/1472400115 • http://www.flickr.com/photos/grandmaitre/5846058698/ • http://www.flickr.com/photos/7518432@N06/2673347604/

×