SlideShare a Scribd company logo
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
SHOW ME
THE
METRICS James Brooks
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
Born in 1999
UK, Romania, Portugal, Ireland, Malta, Gibraltar, USA
Engineering Blog: www.betsandbits.com
700+ Engineers and Growing
Exchange / Sportsbook /Games
~750k active users
~70M daily transactions
~2.5Bn daily API calls
~100k/s Monitoring Points
E2E P95 transaction times <4ms
One of UK’s most successful .coms with ~£2Bn Market Cap
76
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.
OpenTSDB &
OpenTSP at Betfair
77
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 78
What is OpenTSDB?
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 79
What is OpenTSDB?
A database optimised for
storing and querying Time
Series Metrics
os.filesystem.size 1413978032 912234340
host=fe-108 cluster=bf_c mount=/ type=Free
Name Timestamp Value
metadata
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 80
OpenTSDB Architecture
Hadoop
• TSDB Read
• TSDB Write
Frontend
Server
• TSDB Read
• TSDB Write
Frontend
Server
• TSDB Read
• TSDB Write
Frontend
Server
LoadBalancer
Metrics
Queries
TSDB
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 81
OpenTSDB Architecture
Metrics
Queries
Original
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 82
OpenTSDB Architecture (ctd)
TSDB
Source :http://mo.nitor.me/hadoop-hbase-opentsdb/
Original
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 84
OpenTSDB Architecture (ctd)
Metrilyx
TSDB
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 85
Original
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 86
OpenTSDB Architecture (ctd)
Metrilyx
TSDB Visualiser
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 87
tcollector
collect-os
collect-jmx
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 88
OpenTSDB Architecture (ctd)
TSDB
tcollector
collect-os
collect-jmx
tcollector
collect-os
collect-jmx
•  TSDB Use Cases
•  Troubleshooting / Root Cause analysis
•  Performance Analysis
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 89
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 90
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 91
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 92
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 93
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 94
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 95
•  TSDB Use Cases
•  Troubleshooting / Root Cause analysis
•  Performance Analysis
•  Monitoring and Alerting
•  Limitations
•  Load
•  Recency
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 96
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 97
•  TSDB Use Cases
•  Troubleshooting / Root Cause analysis
•  Performance Analysis
•  Monitoring and Alerting
•  Limitations
•  Load
•  Recency
•  TSP
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 98
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 99
OpenTSP
tcollector
collect-os
collect-jmx
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 100
OpenTSP
TSDB
tcollector
collect-os
collect-jmx
forwarder
collect-os
collect-jmx
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 101
OpenTSP Architecture: Forwarder
TAG
FILTER
collect-statse
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 102
OpenTSP Architecture: Statse
application collect-statse
count
mean
median
p95
p99
forwarder
collect-statse
count
mean
median
p95
p99
forwarder
application collect-statse
count
mean
median
p95
p99
forwarder
application
poller
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 103
OpenTSP Architecture: Poller
collect-snmp
collect-f5
collect-netscaler
switches
routers
firewalls
load balancers
storage
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 104
OpenTSP Architecture: Aggregator
forwarder
forwarder
poller
aggregator
collect-site
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 105
OpenTSP Architecture: Controller
controller
collectors aggregator
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 106
[james@ohlookahostname~]$1CONTROL_HOST=tspconfig.inf.betfair1tspDforwarder1Dt1
{1
11111111"Filter":1[1
1111111111111111{1
111111111111111111111111"Match":1[1
11111111111111111111111111111111"",1
11111111111111111111111111111111"host",1
11111111111111111111111111111111"^$"1
111111111111111111111111],1
111111111111111111111111"Set":1[1
11111111111111111111111111111111"",1
11111111111111111111111111111111"host",1
11111111111111111111111111111111“ohlookahostname.inf.betfair"1
111111111111111111111111]1
1111111111111111}1
11111111],1
11111111"LogPath":1"/var/log/tsp/forwarder.log",1
11111111"Relay":1{1
1111111111111111"aggregator":1{1
111111111111111111111111"DropRepeats":1false,1
111111111111111111111111"Host":1“tspaggregator.inf.betfair",1
111111111111111111111111"MaxConnsPerHost":11,1
111111111111111111111111"OnQueueFull":1"Drop"1
1111111111111111},1
1111111111111111"tsd":1{1
111111111111111111111111"DropRepeats":1true,1
111111111111111111111111"Host":1"tsdb.inf.betfair",1
111111111111111111111111"MaxConnsPerHost":11,1
111111111111111111111111"OnQueueFull":1"Drop"1
1111111111111111}1
11111111},1
11111111"CollectPath":1"/etc/tsp/collect.d"1
}1
Visualise
r
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 107
TSP at Betfaircontroller
collectors TSDB Visualiser
tsdb2mysql mysqld
feedhealth
Riemann
future?
aggregator
aggregator
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 108
Open Source
•  tsp-forwarder
•  collect-statse
•  tsp-poller
•  collect-f5
•  collect-netscaler
•  collect-snmp
•  tsp-aggregator
•  tsp-controller
Apache Licence v2
http://github.com/betfair/opentsp
james.brooks@betfair.com
CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 109

More Related Content

Similar to James Brooks (Betfair) - Show me the Metrics

Wellington APAC Groundbreakers tour - Upgrading to the 12c Optimizer
Wellington APAC Groundbreakers tour - Upgrading to the 12c OptimizerWellington APAC Groundbreakers tour - Upgrading to the 12c Optimizer
Wellington APAC Groundbreakers tour - Upgrading to the 12c Optimizer
Connor McDonald
 
EMMMV - Caging the Gorilla
EMMMV - Caging the GorillaEMMMV - Caging the Gorilla
EMMMV - Caging the Gorilla
The Open Group SA
 
How to Take Advantage of Optimizer Improvements in MySQL 8.0
How to Take Advantage of Optimizer Improvements in MySQL 8.0How to Take Advantage of Optimizer Improvements in MySQL 8.0
How to Take Advantage of Optimizer Improvements in MySQL 8.0
Norvald Ryeng
 
NTM Corp: Mobil SHC Presentation on power cost reduction using mobil shc - cu...
NTM Corp: Mobil SHC Presentation on power cost reduction using mobil shc - cu...NTM Corp: Mobil SHC Presentation on power cost reduction using mobil shc - cu...
NTM Corp: Mobil SHC Presentation on power cost reduction using mobil shc - cu...
japjaca
 
Agilis_CorporateBrochure
Agilis_CorporateBrochureAgilis_CorporateBrochure
Agilis_CorporateBrochureBrett Stoia
 
PDO Predictive Analytics Share for the Annual Research Forum 2015
PDO Predictive Analytics Share for the Annual Research Forum 2015PDO Predictive Analytics Share for the Annual Research Forum 2015
PDO Predictive Analytics Share for the Annual Research Forum 2015
Faris Al-Kharusi
 
RESUME - ALOK SAXENA - CNC - ELECTRONICS ENGINEER
RESUME - ALOK SAXENA - CNC - ELECTRONICS ENGINEERRESUME - ALOK SAXENA - CNC - ELECTRONICS ENGINEER
RESUME - ALOK SAXENA - CNC - ELECTRONICS ENGINEERAlok Saxena
 
Julius job details report
Julius job details report Julius job details report
Julius job details report
Olusegun Julius Adegboyega
 
Pattern Matching with SQL - APEX World Rotterdam 2019
Pattern Matching with SQL - APEX World Rotterdam 2019Pattern Matching with SQL - APEX World Rotterdam 2019
Pattern Matching with SQL - APEX World Rotterdam 2019
Connor McDonald
 
GPS 2023
GPS 2023GPS 2023
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
Altinity Ltd
 
Zentech Manufacturing Capabilities
Zentech Manufacturing CapabilitiesZentech Manufacturing Capabilities
Zentech Manufacturing CapabilitiesTransline Technology
 
Top 10 SQL Performance tips & tricks for Java Developers
Top 10 SQL Performance tips & tricks for Java DevelopersTop 10 SQL Performance tips & tricks for Java Developers
Top 10 SQL Performance tips & tricks for Java Developers
gvenzl
 
Acceptance Test-Driven Development: Mastering Agile Testing
Acceptance Test-Driven Development: Mastering Agile TestingAcceptance Test-Driven Development: Mastering Agile Testing
Acceptance Test-Driven Development: Mastering Agile Testing
TechWell
 
WorldLine Digital-New Technology for Transit Advertising
WorldLine Digital-New Technology for Transit Advertising WorldLine Digital-New Technology for Transit Advertising
WorldLine Digital-New Technology for Transit Advertising
minh đức
 
Trillium Flow Technologies: Alloa Service Centre Capabilities
Trillium Flow Technologies: Alloa Service Centre CapabilitiesTrillium Flow Technologies: Alloa Service Centre Capabilities
Trillium Flow Technologies: Alloa Service Centre Capabilities
Matthew Hewett
 
byteLAKE's CFD Suite (AI-accelerated CFD) (2024-02)
byteLAKE's CFD Suite (AI-accelerated CFD) (2024-02)byteLAKE's CFD Suite (AI-accelerated CFD) (2024-02)
byteLAKE's CFD Suite (AI-accelerated CFD) (2024-02)
byteLAKE
 
stackconf 2023 | How to reduce expenses on monitoring with VictoriaMetrics by...
stackconf 2023 | How to reduce expenses on monitoring with VictoriaMetrics by...stackconf 2023 | How to reduce expenses on monitoring with VictoriaMetrics by...
stackconf 2023 | How to reduce expenses on monitoring with VictoriaMetrics by...
NETWAYS
 
Brochure Services-AFTER
Brochure Services-AFTERBrochure Services-AFTER
Brochure Services-AFTER
Tina Thomas
 

Similar to James Brooks (Betfair) - Show me the Metrics (20)

Wellington APAC Groundbreakers tour - Upgrading to the 12c Optimizer
Wellington APAC Groundbreakers tour - Upgrading to the 12c OptimizerWellington APAC Groundbreakers tour - Upgrading to the 12c Optimizer
Wellington APAC Groundbreakers tour - Upgrading to the 12c Optimizer
 
EMMMV - Caging the Gorilla
EMMMV - Caging the GorillaEMMMV - Caging the Gorilla
EMMMV - Caging the Gorilla
 
How to Take Advantage of Optimizer Improvements in MySQL 8.0
How to Take Advantage of Optimizer Improvements in MySQL 8.0How to Take Advantage of Optimizer Improvements in MySQL 8.0
How to Take Advantage of Optimizer Improvements in MySQL 8.0
 
NTM Corp: Mobil SHC Presentation on power cost reduction using mobil shc - cu...
NTM Corp: Mobil SHC Presentation on power cost reduction using mobil shc - cu...NTM Corp: Mobil SHC Presentation on power cost reduction using mobil shc - cu...
NTM Corp: Mobil SHC Presentation on power cost reduction using mobil shc - cu...
 
Agilis_CorporateBrochure
Agilis_CorporateBrochureAgilis_CorporateBrochure
Agilis_CorporateBrochure
 
PDO Predictive Analytics Share for the Annual Research Forum 2015
PDO Predictive Analytics Share for the Annual Research Forum 2015PDO Predictive Analytics Share for the Annual Research Forum 2015
PDO Predictive Analytics Share for the Annual Research Forum 2015
 
RESUME - ALOK SAXENA - CNC - ELECTRONICS ENGINEER
RESUME - ALOK SAXENA - CNC - ELECTRONICS ENGINEERRESUME - ALOK SAXENA - CNC - ELECTRONICS ENGINEER
RESUME - ALOK SAXENA - CNC - ELECTRONICS ENGINEER
 
Julius job details report
Julius job details report Julius job details report
Julius job details report
 
Pattern Matching with SQL - APEX World Rotterdam 2019
Pattern Matching with SQL - APEX World Rotterdam 2019Pattern Matching with SQL - APEX World Rotterdam 2019
Pattern Matching with SQL - APEX World Rotterdam 2019
 
GPS 2023
GPS 2023GPS 2023
GPS 2023
 
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Kh...
 
Zentech Manufacturing Capabilities
Zentech Manufacturing CapabilitiesZentech Manufacturing Capabilities
Zentech Manufacturing Capabilities
 
Top 10 SQL Performance tips & tricks for Java Developers
Top 10 SQL Performance tips & tricks for Java DevelopersTop 10 SQL Performance tips & tricks for Java Developers
Top 10 SQL Performance tips & tricks for Java Developers
 
Acceptance Test-Driven Development: Mastering Agile Testing
Acceptance Test-Driven Development: Mastering Agile TestingAcceptance Test-Driven Development: Mastering Agile Testing
Acceptance Test-Driven Development: Mastering Agile Testing
 
WorldLine Digital-New Technology for Transit Advertising
WorldLine Digital-New Technology for Transit Advertising WorldLine Digital-New Technology for Transit Advertising
WorldLine Digital-New Technology for Transit Advertising
 
Trillium Flow Technologies: Alloa Service Centre Capabilities
Trillium Flow Technologies: Alloa Service Centre CapabilitiesTrillium Flow Technologies: Alloa Service Centre Capabilities
Trillium Flow Technologies: Alloa Service Centre Capabilities
 
byteLAKE's CFD Suite (AI-accelerated CFD) (2024-02)
byteLAKE's CFD Suite (AI-accelerated CFD) (2024-02)byteLAKE's CFD Suite (AI-accelerated CFD) (2024-02)
byteLAKE's CFD Suite (AI-accelerated CFD) (2024-02)
 
stackconf 2023 | How to reduce expenses on monitoring with VictoriaMetrics by...
stackconf 2023 | How to reduce expenses on monitoring with VictoriaMetrics by...stackconf 2023 | How to reduce expenses on monitoring with VictoriaMetrics by...
stackconf 2023 | How to reduce expenses on monitoring with VictoriaMetrics by...
 
Brochure Services-AFTER
Brochure Services-AFTERBrochure Services-AFTER
Brochure Services-AFTER
 
GTC EMS Bro 12-15
GTC EMS Bro 12-15GTC EMS Bro 12-15
GTC EMS Bro 12-15
 

More from Outlyer

Murat Karslioglu, VP Solutions @ OpenEBS - Containerized storage for containe...
Murat Karslioglu, VP Solutions @ OpenEBS - Containerized storage for containe...Murat Karslioglu, VP Solutions @ OpenEBS - Containerized storage for containe...
Murat Karslioglu, VP Solutions @ OpenEBS - Containerized storage for containe...
Outlyer
 
How & When to Feature Flag
How & When to Feature FlagHow & When to Feature Flag
How & When to Feature Flag
Outlyer
 
Why You Need to Stop Using "The" Staging Server
Why You Need to Stop Using "The" Staging ServerWhy You Need to Stop Using "The" Staging Server
Why You Need to Stop Using "The" Staging Server
Outlyer
 
How GitHub combined with CI empowers rapid product delivery at Credit Karma
How GitHub combined with CI empowers rapid product delivery at Credit Karma How GitHub combined with CI empowers rapid product delivery at Credit Karma
How GitHub combined with CI empowers rapid product delivery at Credit Karma
Outlyer
 
Packaging Services with Nix
Packaging Services with NixPackaging Services with Nix
Packaging Services with Nix
Outlyer
 
Minimum Viable Docker: our journey towards orchestration
Minimum Viable Docker: our journey towards orchestrationMinimum Viable Docker: our journey towards orchestration
Minimum Viable Docker: our journey towards orchestration
Outlyer
 
Ops is dead. long live ops.
Ops is dead. long live ops.Ops is dead. long live ops.
Ops is dead. long live ops.
Outlyer
 
The service mesh: resilient communication for microservice applications
The service mesh: resilient communication for microservice applicationsThe service mesh: resilient communication for microservice applications
The service mesh: resilient communication for microservice applications
Outlyer
 
Microservices: Why We Did It (and should you?)
Microservices: Why We Did It (and should you?) Microservices: Why We Did It (and should you?)
Microservices: Why We Did It (and should you?)
Outlyer
 
Renan Dias: Using Alexa to deploy applications to Kubernetes
Renan Dias: Using Alexa to deploy applications to KubernetesRenan Dias: Using Alexa to deploy applications to Kubernetes
Renan Dias: Using Alexa to deploy applications to Kubernetes
Outlyer
 
Alex Dias: how to build a docker monitoring solution
Alex Dias: how to build a docker monitoring solution Alex Dias: how to build a docker monitoring solution
Alex Dias: how to build a docker monitoring solution
Outlyer
 
How to build a container monitoring solution - David Gildeh, CEO and Co-Found...
How to build a container monitoring solution - David Gildeh, CEO and Co-Found...How to build a container monitoring solution - David Gildeh, CEO and Co-Found...
How to build a container monitoring solution - David Gildeh, CEO and Co-Found...
Outlyer
 
Heresy in the church of - Corey Quinn, Principal at The Quinn Advisory Group
Heresy in the church of - Corey Quinn, Principal at The Quinn Advisory Group Heresy in the church of - Corey Quinn, Principal at The Quinn Advisory Group
Heresy in the church of - Corey Quinn, Principal at The Quinn Advisory Group
Outlyer
 
Anatomy of a real-life incident -Alex Solomon, CTO and Co-Founder of PagerDuty
Anatomy of a real-life incident -Alex Solomon, CTO and Co-Founder of PagerDutyAnatomy of a real-life incident -Alex Solomon, CTO and Co-Founder of PagerDuty
Anatomy of a real-life incident -Alex Solomon, CTO and Co-Founder of PagerDuty
Outlyer
 
A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering...
A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering...A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering...
A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering...
Outlyer
 
The Network Knows—Avi Freedman, CEO & Co-Founder of Kentik
The Network Knows—Avi Freedman, CEO & Co-Founder of Kentik The Network Knows—Avi Freedman, CEO & Co-Founder of Kentik
The Network Knows—Avi Freedman, CEO & Co-Founder of Kentik
Outlyer
 
Building a production-ready, fully-scalable Docker Swarm using Terraform & Pa...
Building a production-ready, fully-scalable Docker Swarm using Terraform & Pa...Building a production-ready, fully-scalable Docker Swarm using Terraform & Pa...
Building a production-ready, fully-scalable Docker Swarm using Terraform & Pa...
Outlyer
 
Zero Downtime Postgres Upgrades
Zero Downtime Postgres UpgradesZero Downtime Postgres Upgrades
Zero Downtime Postgres Upgrades
Outlyer
 
DOXLON November 2016: Facebook Engineering on cgroupv2
DOXLON November 2016: Facebook Engineering on cgroupv2DOXLON November 2016: Facebook Engineering on cgroupv2
DOXLON November 2016: Facebook Engineering on cgroupv2
Outlyer
 
DOXLON November 2016 - ELK Stack and Beats
DOXLON November 2016 - ELK Stack and Beats DOXLON November 2016 - ELK Stack and Beats
DOXLON November 2016 - ELK Stack and Beats
Outlyer
 

More from Outlyer (20)

Murat Karslioglu, VP Solutions @ OpenEBS - Containerized storage for containe...
Murat Karslioglu, VP Solutions @ OpenEBS - Containerized storage for containe...Murat Karslioglu, VP Solutions @ OpenEBS - Containerized storage for containe...
Murat Karslioglu, VP Solutions @ OpenEBS - Containerized storage for containe...
 
How & When to Feature Flag
How & When to Feature FlagHow & When to Feature Flag
How & When to Feature Flag
 
Why You Need to Stop Using "The" Staging Server
Why You Need to Stop Using "The" Staging ServerWhy You Need to Stop Using "The" Staging Server
Why You Need to Stop Using "The" Staging Server
 
How GitHub combined with CI empowers rapid product delivery at Credit Karma
How GitHub combined with CI empowers rapid product delivery at Credit Karma How GitHub combined with CI empowers rapid product delivery at Credit Karma
How GitHub combined with CI empowers rapid product delivery at Credit Karma
 
Packaging Services with Nix
Packaging Services with NixPackaging Services with Nix
Packaging Services with Nix
 
Minimum Viable Docker: our journey towards orchestration
Minimum Viable Docker: our journey towards orchestrationMinimum Viable Docker: our journey towards orchestration
Minimum Viable Docker: our journey towards orchestration
 
Ops is dead. long live ops.
Ops is dead. long live ops.Ops is dead. long live ops.
Ops is dead. long live ops.
 
The service mesh: resilient communication for microservice applications
The service mesh: resilient communication for microservice applicationsThe service mesh: resilient communication for microservice applications
The service mesh: resilient communication for microservice applications
 
Microservices: Why We Did It (and should you?)
Microservices: Why We Did It (and should you?) Microservices: Why We Did It (and should you?)
Microservices: Why We Did It (and should you?)
 
Renan Dias: Using Alexa to deploy applications to Kubernetes
Renan Dias: Using Alexa to deploy applications to KubernetesRenan Dias: Using Alexa to deploy applications to Kubernetes
Renan Dias: Using Alexa to deploy applications to Kubernetes
 
Alex Dias: how to build a docker monitoring solution
Alex Dias: how to build a docker monitoring solution Alex Dias: how to build a docker monitoring solution
Alex Dias: how to build a docker monitoring solution
 
How to build a container monitoring solution - David Gildeh, CEO and Co-Found...
How to build a container monitoring solution - David Gildeh, CEO and Co-Found...How to build a container monitoring solution - David Gildeh, CEO and Co-Found...
How to build a container monitoring solution - David Gildeh, CEO and Co-Found...
 
Heresy in the church of - Corey Quinn, Principal at The Quinn Advisory Group
Heresy in the church of - Corey Quinn, Principal at The Quinn Advisory Group Heresy in the church of - Corey Quinn, Principal at The Quinn Advisory Group
Heresy in the church of - Corey Quinn, Principal at The Quinn Advisory Group
 
Anatomy of a real-life incident -Alex Solomon, CTO and Co-Founder of PagerDuty
Anatomy of a real-life incident -Alex Solomon, CTO and Co-Founder of PagerDutyAnatomy of a real-life incident -Alex Solomon, CTO and Co-Founder of PagerDuty
Anatomy of a real-life incident -Alex Solomon, CTO and Co-Founder of PagerDuty
 
A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering...
A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering...A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering...
A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering...
 
The Network Knows—Avi Freedman, CEO & Co-Founder of Kentik
The Network Knows—Avi Freedman, CEO & Co-Founder of Kentik The Network Knows—Avi Freedman, CEO & Co-Founder of Kentik
The Network Knows—Avi Freedman, CEO & Co-Founder of Kentik
 
Building a production-ready, fully-scalable Docker Swarm using Terraform & Pa...
Building a production-ready, fully-scalable Docker Swarm using Terraform & Pa...Building a production-ready, fully-scalable Docker Swarm using Terraform & Pa...
Building a production-ready, fully-scalable Docker Swarm using Terraform & Pa...
 
Zero Downtime Postgres Upgrades
Zero Downtime Postgres UpgradesZero Downtime Postgres Upgrades
Zero Downtime Postgres Upgrades
 
DOXLON November 2016: Facebook Engineering on cgroupv2
DOXLON November 2016: Facebook Engineering on cgroupv2DOXLON November 2016: Facebook Engineering on cgroupv2
DOXLON November 2016: Facebook Engineering on cgroupv2
 
DOXLON November 2016 - ELK Stack and Beats
DOXLON November 2016 - ELK Stack and Beats DOXLON November 2016 - ELK Stack and Beats
DOXLON November 2016 - ELK Stack and Beats
 

Recently uploaded

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
Cheryl Hung
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
Thijs Feryn
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)
Ralf Eggert
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Thierry Lestable
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
ThousandEyes
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
Paul Groth
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
Product School
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
Fwdays
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
OnBoard
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
BookNet Canada
 

Recently uploaded (20)

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
 

James Brooks (Betfair) - Show me the Metrics

  • 1. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. SHOW ME THE METRICS James Brooks
  • 2. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. Born in 1999 UK, Romania, Portugal, Ireland, Malta, Gibraltar, USA Engineering Blog: www.betsandbits.com 700+ Engineers and Growing Exchange / Sportsbook /Games ~750k active users ~70M daily transactions ~2.5Bn daily API calls ~100k/s Monitoring Points E2E P95 transaction times <4ms One of UK’s most successful .coms with ~£2Bn Market Cap 76
  • 3. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. OpenTSDB & OpenTSP at Betfair 77
  • 4. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 78 What is OpenTSDB?
  • 5. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 79 What is OpenTSDB? A database optimised for storing and querying Time Series Metrics os.filesystem.size 1413978032 912234340 host=fe-108 cluster=bf_c mount=/ type=Free Name Timestamp Value metadata
  • 6. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 80 OpenTSDB Architecture Hadoop • TSDB Read • TSDB Write Frontend Server • TSDB Read • TSDB Write Frontend Server • TSDB Read • TSDB Write Frontend Server LoadBalancer Metrics Queries
  • 7. TSDB CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 81 OpenTSDB Architecture Metrics Queries
  • 8. Original CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 82 OpenTSDB Architecture (ctd) TSDB
  • 10. Original CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 84 OpenTSDB Architecture (ctd) Metrilyx TSDB
  • 11. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 85
  • 12. Original CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 86 OpenTSDB Architecture (ctd) Metrilyx TSDB Visualiser
  • 13. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 87
  • 14. tcollector collect-os collect-jmx CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 88 OpenTSDB Architecture (ctd) TSDB tcollector collect-os collect-jmx tcollector collect-os collect-jmx
  • 15. •  TSDB Use Cases •  Troubleshooting / Root Cause analysis •  Performance Analysis CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 89
  • 16. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 90
  • 17. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 91
  • 18. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 92
  • 19. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 93
  • 20. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 94
  • 21. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 95
  • 22. •  TSDB Use Cases •  Troubleshooting / Root Cause analysis •  Performance Analysis •  Monitoring and Alerting •  Limitations •  Load •  Recency CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 96
  • 23. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 97
  • 24. •  TSDB Use Cases •  Troubleshooting / Root Cause analysis •  Performance Analysis •  Monitoring and Alerting •  Limitations •  Load •  Recency •  TSP CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 98
  • 25. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 99 OpenTSP
  • 26. tcollector collect-os collect-jmx CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 100 OpenTSP TSDB tcollector collect-os collect-jmx forwarder collect-os collect-jmx
  • 27. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 101 OpenTSP Architecture: Forwarder TAG FILTER collect-statse
  • 28. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 102 OpenTSP Architecture: Statse application collect-statse count mean median p95 p99 forwarder collect-statse count mean median p95 p99 forwarder application collect-statse count mean median p95 p99 forwarder application
  • 29. poller CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 103 OpenTSP Architecture: Poller collect-snmp collect-f5 collect-netscaler switches routers firewalls load balancers storage
  • 30. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 104 OpenTSP Architecture: Aggregator forwarder forwarder poller aggregator collect-site
  • 31. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 105 OpenTSP Architecture: Controller controller collectors aggregator
  • 32. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 106 [james@ohlookahostname~]$1CONTROL_HOST=tspconfig.inf.betfair1tspDforwarder1Dt1 {1 11111111"Filter":1[1 1111111111111111{1 111111111111111111111111"Match":1[1 11111111111111111111111111111111"",1 11111111111111111111111111111111"host",1 11111111111111111111111111111111"^$"1 111111111111111111111111],1 111111111111111111111111"Set":1[1 11111111111111111111111111111111"",1 11111111111111111111111111111111"host",1 11111111111111111111111111111111“ohlookahostname.inf.betfair"1 111111111111111111111111]1 1111111111111111}1 11111111],1 11111111"LogPath":1"/var/log/tsp/forwarder.log",1 11111111"Relay":1{1 1111111111111111"aggregator":1{1 111111111111111111111111"DropRepeats":1false,1 111111111111111111111111"Host":1“tspaggregator.inf.betfair",1 111111111111111111111111"MaxConnsPerHost":11,1 111111111111111111111111"OnQueueFull":1"Drop"1 1111111111111111},1 1111111111111111"tsd":1{1 111111111111111111111111"DropRepeats":1true,1 111111111111111111111111"Host":1"tsdb.inf.betfair",1 111111111111111111111111"MaxConnsPerHost":11,1 111111111111111111111111"OnQueueFull":1"Drop"1 1111111111111111}1 11111111},1 11111111"CollectPath":1"/etc/tsp/collect.d"1 }1
  • 33. Visualise r CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 107 TSP at Betfaircontroller collectors TSDB Visualiser tsdb2mysql mysqld feedhealth Riemann future? aggregator aggregator
  • 34. CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 108 Open Source •  tsp-forwarder •  collect-statse •  tsp-poller •  collect-f5 •  collect-netscaler •  collect-snmp •  tsp-aggregator •  tsp-controller Apache Licence v2 http://github.com/betfair/opentsp
  • 35. james.brooks@betfair.com CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited. 109