Uploaded byBoyan Dimitrov
Monitoring microservices platform
The document details the monitoring micro-services platform at Hailo, highlighting its architecture built on AWS with over 160 services and key challenges faced in scaling and instrumentation. It outlines the evolution of monitoring practices, including the transition to tools like Grafana and the implementation of service-level health checks for improved performance insights. The mission involves defining platform metrics, enhancing automatic failover capabilities, and integrating machine learning for advanced event correlation.
More Related Content
Similar to Monitoring microservices platform
Monitoring microservices platform
- 1. Monitoring micro-services platform Boyan Dimitrov, Platform Engineering @ Hailo @nathariel
- 2. Outline • Introto the Hailo world • Platform Overview • Monitoring Evolution
- 4. The Platform Trolla platform by Swinsto101 / CC BY-SA 3.0 / Desaturated from original
- 5. Platform specifics •SOA based on Go ( and Java… ) • 1000+ AWS instances spanning multiple regions • 160+ services in production • Designed specifically for the cloud – different building blocks and components will constantly be in flux, broken or unavailable.
- 6. eu-west-1 Proxy Layer Message Bus+ Go Services Java Services C* us-east-1 Proxy Layer Message Bus+ Go Services Java C* Services
- 7. Provisioning Service CIPipeline (Janky/Jenkins) Amazon S3 Provisioning Service Provisioning Service Provisioning Manager Docker Registry Inside an environment
- 8. A micro-service underthe hood Handler platform-layer Logic Storage Library for abstracting service-to- service comms service-layer Self-configuring external service adapters Service Any service gets for free: • Provisioning • Discovery • Configuration • Authentication/Authorization • A/B testing capabilities • Self-configuring connectivity to third-party services • Monitoring • Instrumentation
- 9. Mission: Define highlevel platform and business metrics Gather as many insights as possible Add automatic failover and recovery capabilities "A[ollo 8 Launch Control Room” by Tfawls / Desaturated from original
- 10. PHP Java HostInstance Graphite Zabbix Aspiration vs Reality CloudWatch Zabbix Agent StatsD Carbon
- 11. Challenges • SingleStatsD instance and generic graphite setup cannot cope with all the traffic (surprise!) • No easy way of generating and searching for graphs quickly • We didn’t instrument everything • “Traditional” monitoring systems can only give basic app insights • Se#ing up app templates is a manual daunting process and does not scale • No in-depth visibility into our main KPIs • No way of identifying platform / release / config / cloud infrastructure changes
- 12. Instrumentation++ “Airplaine board”by Smithore / Desaturated from original
- 13. Host Instance Graphite Cache Zabbix Iterate on what we already know Relay CloudWatch CollectD StatsD Cache Cache Zabbix Agent
- 14. Result • Scalingup graphite and moving StatsD to every box allowed us to collect millions of metrics • Instrumenting everything gives us a lot of insights. • Grafana allows us to quickly build, store and search for important graphs. Widely adopted by the whole development team! Tip: Focus on upper 95th and 99th percentiles and work out from there.
- 15.
- 16.
- 17. Provisioning Service Messagebus Monitoring Service New Service Publish Healthchecks Host Instance Provisioning Manager Binding Discovery Provisioning Service Host Instance Monitoring V2
- 18. healthcheck.Register(&healthcheck.HealthCheck{! Id: “MyHCId”,! ServiceName: ServiceName,! ServiceVersion: ServiceVersion,! Hostname: Hostname,! InstanceId: InstanceID,! Interval: time.Minute,! Checker: myCallbackFunc,! Priority: hc.Warning,! })!
- 19. Service level healthchecks
- 20. Result • Servicehealth checks give us in-depth service performance details • The monitoring service has a holistic view of our platform health and can identify degraded availability zones • Developers can identify what is important for their service and track & alert on it.
- 21. Trace++ Monitoring & Instrumentation “Abstract conception of network and communication” by Leszekglasner / Desaturated from original
- 22. Trace Architecture CollectDStatsD Zabbix Agent Provisioning Service Host Instance Phosphor Publish Trace Service Dashboards Monitoring In-memory Aggregates Optional persistant storage Async UDP
- 23.
- 24.
- 25.
- 26. Result • Traceincoming requests and pinpoint bo#lenecks & SLA offenders • Easily identify problems on the request/response path • Quickly find out exactly which services participate on the request path
- 27.
- 28.
- 29. Result • Identifybusiness impacting issues immediately • Highlight the service on the critical path that is most likely responsible for the problems
- 30. Event Correlation “Connection”by A2bb5s / Desaturated from the original
- 31. CollectD StatsD Zabbix Agent Provisioning Service Host Instance Phosphor Publish c Dashboards Monitoring Persistent Storage SNS Platform Events Whisper Service c Platform events
- 34. Result • Answerto the most important “Did anything change?” question • Audit trail for any platform changes • Holistic view of our platform status
- 35. It is notover yet! ++ Machine Learning ++ Event source weighting
- 36. Thanks! PS. We’rehiring! @nathariel boyan@hailocab.com London DevOps