Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
SHOW ME
THE
METRI...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
Born in 1999
UK, ...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
OpenTSDB &
OpenTS...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 78
What is OpenTS...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 79
What is OpenTS...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 80
OpenTSDB Archi...
TSDB
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 81
OpenTSDB ...
Original
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 82
OpenT...
Source :http://mo.nitor.me/hadoop-hbase-opentsdb/
Original
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 84
OpenT...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 85
Original
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 86
OpenT...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 87
tcollector
collect-os
collect-jmx
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting E...
•  TSDB Use Cases
•  Troubleshooting / Root Cause analysis
•  Performance Analysis
CONFIDENTIAL and not for reproduction w...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 90
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 91
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 92
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 93
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 94
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 95
•  TSDB Use Cases
•  Troubleshooting / Root Cause analysis
•  Performance Analysis
•  Monitoring and Alerting
•  Limitatio...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 97
•  TSDB Use Cases
•  Troubleshooting / Root Cause analysis
•  Performance Analysis
•  Monitoring and Alerting
•  Limitatio...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 99
OpenTSP
tcollector
collect-os
collect-jmx
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting E...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 101
OpenTSP Archi...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 102
OpenTSP Archi...
poller
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 103
OpenTS...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 104
OpenTSP Archi...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 105
OpenTSP Archi...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 106
[james@ohlook...
Visualise
r
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 107
T...
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 108
Open Source
•...
james.brooks@betfair.com
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange L...
Upcoming SlideShare
Loading in …5
×

James Brooks (Betfair) - Show me the Metrics

1,081 views

Published on

Time Series metrics can be an important part of a comprehensive monitoring solution. Betfair will present a talk on their experiences running OpenTSDB and a new open source tool called OpenTSP, designed to streamline the process of gathering and delivering system metrics quickly and reliably to multiple endpoints so that you can use any of your favourite tools to analyse the stream.

Video: https://www.youtube.com/watch?v=a2zBTD-pveY

Join DevOps Exchange London here: http://www.meetup.com/DevOps-Exchange-London
Follow DOXLON on twitter http://www.twitter.com/doxlon

Published in: Technology
  • Be the first to comment

  • Be the first to like this

James Brooks (Betfair) - Show me the Metrics

  1. 1. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. SHOW ME THE METRICS James Brooks
  2. 2. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. Born in 1999 UK, Romania, Portugal, Ireland, Malta, Gibraltar, USA Engineering Blog: www.betsandbits.com 700+ Engineers and Growing Exchange / Sportsbook /Games ~750k active users ~70M daily transactions ~2.5Bn daily API calls ~100k/s Monitoring Points E2E P95 transaction times <4ms One of UK’s most successful .coms with ~£2Bn Market Cap 76
  3. 3. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. OpenTSDB & OpenTSP at Betfair 77
  4. 4. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 78 What is OpenTSDB?
  5. 5. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 79 What is OpenTSDB? A database optimised for storing and querying Time Series Metrics os.filesystem.size 1413978032 912234340 host=fe-108 cluster=bf_c mount=/ type=Free Name Timestamp Value metadata
  6. 6. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 80 OpenTSDB Architecture Hadoop • TSDB Read • TSDB Write Frontend Server • TSDB Read • TSDB Write Frontend Server • TSDB Read • TSDB Write Frontend Server LoadBalancer Metrics Queries
  7. 7. TSDB CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 81 OpenTSDB Architecture Metrics Queries
  8. 8. Original CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 82 OpenTSDB Architecture (ctd) TSDB
  9. 9. Source :http://mo.nitor.me/hadoop-hbase-opentsdb/
  10. 10. Original CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 84 OpenTSDB Architecture (ctd) Metrilyx TSDB
  11. 11. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 85
  12. 12. Original CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 86 OpenTSDB Architecture (ctd) Metrilyx TSDB Visualiser
  13. 13. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 87
  14. 14. tcollector collect-os collect-jmx CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 88 OpenTSDB Architecture (ctd) TSDB tcollector collect-os collect-jmx tcollector collect-os collect-jmx
  15. 15. •  TSDB Use Cases •  Troubleshooting / Root Cause analysis •  Performance Analysis CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 89
  16. 16. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 90
  17. 17. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 91
  18. 18. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 92
  19. 19. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 93
  20. 20. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 94
  21. 21. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 95
  22. 22. •  TSDB Use Cases •  Troubleshooting / Root Cause analysis •  Performance Analysis •  Monitoring and Alerting •  Limitations •  Load •  Recency CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 96
  23. 23. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 97
  24. 24. •  TSDB Use Cases •  Troubleshooting / Root Cause analysis •  Performance Analysis •  Monitoring and Alerting •  Limitations •  Load •  Recency •  TSP CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 98
  25. 25. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 99 OpenTSP
  26. 26. tcollector collect-os collect-jmx CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 100 OpenTSP TSDB tcollector collect-os collect-jmx forwarder collect-os collect-jmx
  27. 27. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 101 OpenTSP Architecture: Forwarder TAG FILTER collect-statse
  28. 28. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 102 OpenTSP Architecture: Statse application collect-statse count mean median p95 p99 forwarder collect-statse count mean median p95 p99 forwarder application collect-statse count mean median p95 p99 forwarder application
  29. 29. poller CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 103 OpenTSP Architecture: Poller collect-snmp collect-f5 collect-netscaler switches routers firewalls load balancers storage
  30. 30. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 104 OpenTSP Architecture: Aggregator forwarder forwarder poller aggregator collect-site
  31. 31. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 105 OpenTSP Architecture: Controller controller collectors aggregator
  32. 32. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 106 [james@ohlookahostname~]$1CONTROL_HOST=tspconfig.inf.betfair1tspDforwarder1Dt1 {1 11111111"Filter":1[1 1111111111111111{1 111111111111111111111111"Match":1[1 11111111111111111111111111111111"",1 11111111111111111111111111111111"host",1 11111111111111111111111111111111"^$"1 111111111111111111111111],1 111111111111111111111111"Set":1[1 11111111111111111111111111111111"",1 11111111111111111111111111111111"host",1 11111111111111111111111111111111“ohlookahostname.inf.betfair"1 111111111111111111111111]1 1111111111111111}1 11111111],1 11111111"LogPath":1"/var/log/tsp/forwarder.log",1 11111111"Relay":1{1 1111111111111111"aggregator":1{1 111111111111111111111111"DropRepeats":1false,1 111111111111111111111111"Host":1“tspaggregator.inf.betfair",1 111111111111111111111111"MaxConnsPerHost":11,1 111111111111111111111111"OnQueueFull":1"Drop"1 1111111111111111},1 1111111111111111"tsd":1{1 111111111111111111111111"DropRepeats":1true,1 111111111111111111111111"Host":1"tsdb.inf.betfair",1 111111111111111111111111"MaxConnsPerHost":11,1 111111111111111111111111"OnQueueFull":1"Drop"1 1111111111111111}1 11111111},1 11111111"CollectPath":1"/etc/tsp/collect.d"1 }1
  33. 33. Visualise r CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 107 TSP at Betfaircontroller collectors TSDB Visualiser tsdb2mysql mysqld feedhealth Riemann future? aggregator aggregator
  34. 34. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 108 Open Source •  tsp-forwarder •  collect-statse •  tsp-poller •  collect-f5 •  collect-netscaler •  collect-snmp •  tsp-aggregator •  tsp-controller Apache Licence v2 http://github.com/betfair/opentsp
  35. 35. james.brooks@betfair.com CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 109

×