Download as PDF, PPTX
Uploaded byOpenStack
Monitoring Uptime on the NeCTAR Research Cloud - Andy Botting, University of Melbourne
The document outlines a comprehensive monitoring system for the Nectar Research Cloud, detailing components such as uptime checks, health checks for APIs and services, and integration testing via Tempest. It highlights various monitoring tools like Nagios and ELK for logging and alert notifications, while also emphasizing configuration management through Puppet. The goal is to ensure all services function correctly and to facilitate quick diagnostics of any issues across multiple sites.
More Related Content
Similar to Monitoring Uptime on the NeCTAR Research Cloud - Andy Botting, University of Melbourne
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
Monitoring Uptime on the NeCTAR Research Cloud - Andy Botting, University of Melbourne
- 1. Monitoring uptime onthe Nectar Research Cloud
- 2.
- 3. Load BalancerLoad Balancer Tier1 Service API Tier 1 Service API Tier 1 Service API Tier 2 Service API Message Queue Message Queue Message Queue Database Tier 1 Service API Tier 1 Service Engine Tier 1 Service API Tier 2 Service Engine Load BalancerDashboard Nectar core services
- 4. Test everything ● APIsand dashboard are running ● Services are working correctly ● Existing resources are happy (e.g. instances, networks) ● New resources can be created successfully ● Across all sites
- 5. Control plane hosts Nagios ●Ping ● SSH ● NTP ● Filesystem ● Uptime ● Puppet Ganglia metrics ● CPU ● Memory ● Network ● Disk I/O
- 6. Control plane services ●Service ports and processes ● HTTP endpoint ● API process ● Oslo middleware healthcheck ● Consistent /healthcheck URL for all services ● Called by load balancers ● More complex tests ● Request token from Keystone ● Check glance for image
- 7. Environment Canary instance ineach AZ ● Ping ● DHCP ● Metadata Not an exhaustive test, but a good indicator
- 8. Instance boot test Exercisethe whole stack with Tempest ● Fetch a token ● Create a keypair ● Create security groups and rules ● Create instance ● Ping instance ● SSH to instance ● Destroy/clean up all resources Tempest
- 9. Instance boot test Instanceboot for each AZ ● Tiny CirrOS image for speed ● Help identify site specific issues Instance boot for each flavour ● Enough capacity for large flavours? ● Scheduler working properly ● Can be problematic with cells v1 Tempest
- 10. Tempest ● OpenStack integrationtesting suite ● Jobs launched by Jenkins with custom wrapper script ● Result pushed (passive) to Nagios via NRDP ● Lots more can be done here (e.g testing more services) Tempest
- 11.
- 12.
- 13.
- 14. Alerts Nagios ● Configured byPuppet ● Notifications delivered by email and Slack ● Site specific alerts sent to site ops team
- 15. Analysing logs ELK ● ElasticSearch,Logstash and Kibana ● Service and LB access logs sent to central syslog server ● Pretty dashboards ● Great for diagnosing issues
- 16.
- 17. Tying it alltogether ● Define nagios_host and nagios_service resources in Puppet ● Nagios configuration built by Naginator from PuppetDB ● Deploy Ganglia ● Custom scripts to extract data from Nagios for dashboard and reports
- 18. Thanks Andy Botting Nectar CoreServices andrew.botting@unimelb.edu.au