Nagios Conference 2014 - Dave Williams - Multi-Tenant Nagios Monitoring
•Download as PPT, PDF•
1 like•1,159 views
Dave Williams presentation on Multi-Tenant Nagios Monitoring. The presentation was given during the Nagios World Conference North America held Oct 13th - Oct 16th, 2014 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/conference
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Nagios Conference 2014 - Dave Williams - Multi-Tenant Nagios Monitoring
Similar to Nagios Conference 2014 - Dave Williams - Multi-Tenant Nagios Monitoring (20)
More from Nagios
More from Nagios (20)
Nagios Conference 2014 - Dave Williams - Multi-Tenant Nagios Monitoring
- 1. October 14th 2014 Dave Williams Technical Architect Multi-Tenant Nagios Monitoring © Bull, 2014 1
- 2. Agenda Background Multi-Tenant Monitoring Why Multi-Tenant Multi-Tenant Design Service Catalogue Futures & ‘Blue Sky thinking’ Questions © Bull, 2014 2
- 3. Background UK based Mainframe (IBM & Honeywell) Unix (HP-UX, AIX, Solaris) Linux (RedHat, SLES, Debian) Network (CASE, 3COM, CISCO) Working for Bull French Computer Manufacturer Mainframes, Unix, HPC, Security, Managed Services, Advisory Services © Bull, 2014 3
- 4. Background System Monitoring OpenView Netview Open Master Open Source Monitoring NetSaint on AIX Nagios © Bull, 2014 4
- 5. Why Multi-Tenant ? Outsourcing Support & Monitoring Multiple Customers –Different Levels of security –Different Hardware / Software Platforms One Support Team –Only need to know about real problems –Can be driven by support ticket not Nagios Required 365 x 24 –Infrastructure must survive all outages without loss of service © Bull, 2014 5
- 6. Multi-Tenant Design Each customer may have 2-3000 hosts 10-100 services per host Real time monitoring Customer profile SLA Reporting Batch Event completion Different SLA’s for each Business Process per customer Different alerting & escalation methods per customer © Bull, 2014 6
- 7. Multi-Tenant Design Hardware Platform – Central Support Virtualised Platform (Intel based) –XenServer Hypervisor Allows clustering with shared storage Inexpensive Licensing Shared Storage –NAS Using QNAP Appliances with underlying RAID-5 & Hot Spare protection Network connection using dual interfaces bound across multiple switches Could have used FreeNas LAN Infrastructure –Dual connections to all hardware –SNMP managed switches © Bull, 2014 7
- 8. Hardware Platform – Basic Schematic © Bull, 2014 8
- 9. Multi-Tenant Design Hardware Platform – Resilience Virtualised Platform (Intel based) –XenServer Hypervisor Allows clustering with shared storage If Primary node fails cluster will ‘spin up’ image on 2nd node Same data / logs (Shared storage) LAN Infrastructure –Dual connections to all hardware Bonded interfaces for NAS access – no data loss / access loss with failure SNMP managed switches © Bull, 2014 9
- 10. Hardware Setup © Bull, 2014 10
- 11. Multi-Tenant Design Hardware Platform – Recovery Virtualised Platform (Intel based) –XenServer Hypervisor Allows clustering with shared storage If Primary Site fails will spin up image Internet Access fails over – using BGP Shared Storage – replicated from Prime Site –NAS Using QNAP Appliances with underlying RAID-5 & Hot Spare protection Using RTRR (Real Time Remote Replication) between sites Network connection using dual interfaces bound across multiple switches LAN Infrastructure –Dual connections to all hardware Bonded interfaces for NAS access – no data loss / access loss with failure SNMP managed switches © Bull, 2014 11
- 12. Hardware Platform - Resilience © Bull, 2014 12
- 13. Hardware Platform – Customer Site Using generic netbooks Minimum requirement –1Gb Memory , Atom processor, Ethernet Port –Running Centos 6.4 64 bit Operating System Can use Raspberry Pi for small customers –512K Memory , Arm processor , Ethernet Port –Running Raspbian Operating System © Bull, 2014 13
- 14. Software Platform – Central Site Nagios – Core Running latest 4.0.8 Using MK Livestatus for interfacing Using Thruk for Visualisation Graylog2 / Elastic Search Store all logs & Syslog in ‘Big Data’ repository using MongoDB Asterisk PBX Allow all alerting to use standard dial-up with speech synthesis + IVR SMS-Client Still using TAPI to SMS Text contacts © Bull, 2014 14
- 15. Software Platform – Central Site (contd) NRPE Running 2.1.5 NSCA &NSCA-ng Using NSCA for external communication Using NSCA-ng for issuing remote commands Postfix / Procmail Used to generate emails but also handle responses. Routes unsolicited alerting emails (HP Insight, Pingdom) OTRS Record alerts, track issues © Bull, 2014 15
- 16. Software Platform – Remote Site Nagios – Core Running latest 4.0.8 NRPE Running 2.14 NSCA Using NSCA for external communication OpenVPN Communication via IPSec VPN © Bull, 2014 16
- 17. Customer Multi-Tenant © Bull, 2014 17
- 18. Multi Tenant Schematic © Bull, 2014 18
- 19. Service Catalogue ITIL Flavour Really just services & their characteristics © Bull, 2014 19
- 20. Service Catalogue Agreed list of servers / services With importance levels With alerting paths With escalation paths Recovery options Feeds into Service Level Agreements and Operational Level Agreements Basis of agreed reporting structures © Bull, 2014 20
- 21. Examples Basic Spreadsheet plus Shell script Usually easy to create, Shell script is different for each customer based on a initial standard script Chef or Puppet Use Exported Resources Nagios Cookbook – Nagios Conference 2012 Presentation © Bull, 2014 21
- 22. Multi Tenant Issues Naming conventions Every customer has a server01 Customers naming conventions are obscure Customers have multiple physical locations or levels of security –This gives rise to different nagios names to actual names: –Custloc1-swfeltsw01 –Custloc2-nwfeltsw01 Not so smart when a non-Nagios originated alert is received, –‘swfeltsw01 – RAID battery backup failure’ from HP Insight for example –The external alert processor has to perform table lookups before building the appropriate NSCA command for example © Bull, 2014 22
- 23. Futures & Blue Sky thinking The Nagios Visualisation is resource heavy All Customers want their own Dashboard All Customers want a different screen layout Why not move the visualisation into the cloud ? Use a Amazon EC2 image to access central Livestatus via https Allow end user to authenticate Customer portal allows ‘spin up’ & ‘spin down’ of images –Move billing to the customer –Scale horizontally for Visualisation © Bull, 2014 23
- 24. Load Sharing Using plugins like check_wmi_plus put a strain on the monitoring system, large number of queries that take wall clock time to complete and parse. Better to have ‘worker nodes’ via Merlin or Mod Gearman similar to perform these functions – Raspberry Pi for example. No great expense to add 2/3 Pi’s to customer site configurations, easy fall back if they fail – no unique locally stored data © Bull, 2014 24
- 25. BPI Example © Bull, 2014 25
- 26. Dashboard Example © Bull, 2014 26
- 27. Questions ? © Bull, 2014 27
- 28. © Bull, 2014 28
Editor's Notes
- Click to add comments
- First some background about me – where I’ve been and where I am Then some description of the Nagios multi tenant solution – why we needed it and some of the design decisions made
- Over 35 years working in the IT industry Coding real time systems and then on to Operating system support GeCOS & Transaction Processing – assembler / machine code Network Processor software development Customer Facing IBM Mainframes MVS / VM – SNA VTAM / NCP – Bureau Environment / Service Orientated Bull – Honeywell Bull / CIIHB / Honeywell - worked at R&D in Toronto / Minneapolis / Grenoble
- Bull Toronto was R&D centre for system monitoring SNMp & Graphical systems Openview on HP kit for IBM Bureau ISDN / Dial-In systems (from FT / Legal document systems) Netview for IBM SNA networks Open Master Openview like system written by Bull – AIX based Saw Netsaint 0.6 finally complied by Chris Rothecker, tried it decided I could do a better port – did it. – built AIX installp file that contained everything GD etc. released via the Bull freeware site. Followed project and continued to work with Nagios under AIX. Later evangelized Nagios in Bull and worked on the Linux based versions
- Support multiple customers throughout Europe – different hardware / software sets / different level of involvement – simple reporting / OS patching / Database / Citrix support / Application support. Only 1 3rd line support team – therefore need system to tell them what is happening before the Service Desk gets involved. Need for monitoriing 24x7 365/6 days a year – keeping relevant history for problem analysis.
- All sizes and shapes 2 – 3000 hosts maybe lots of services – maybe only a few (but very important) SLA reporting – the service must as a minimum achieve the service levels contracted Each Customer is unique in terms of alerting and reporting.
- Virtual central system. XenServer because it’s ‘free’ and supports clusters. QNAP for storage because it was there, could use FreeNAS or other shareable storage – with replication. No need to implement Nagios clustering or multiple nodes as its all taken care of at hardware & hypervisor level
- The usual things to allow the system to stay up – bonded interfaces, multiple switches , multiple nodes in hardware cluster, hardware replication of disk storage.
- Need to cater for site loss – data replicated to 2nd geographic site, identical hardware ready to start up – IP addresses swapped by telecoms provider
- Each customer needs at least one appliance to do the actual monitoring. The size and number of the appliances depends on the needs of the customer. Raspberry Pi is good for the smaller end of SME’s , the Netbook serves the larger customers well. By ‘Gold disk’ the appliance and pulling the Nagios config files back to the centre every day recovery of the appliance is easy – most large sites hold a spare appliance.
- Using Core with LiveStatus to provide data access. Speeds local access and gives better filter facilities. LiveStatus also allows multiple remote Nagios inputs - https Using Thruk for Visual representation – derives data from livestatus, gives SLA reporting in PDF format and a Dashboard. Graylog2 / Elastic search used to absorb Nagios logs + Syslog (Cisco devices sure can generate syslog) + Windows Event logs. Allows searching and some correlation analysis. Asterisk used to alert ‘wetware’ – SMS or emails easily ignored, a voice in the dark reading out error messages is a lot harder (can hit landlines as well as mobiles !)
- NSCA-ng used because I wanted to submit commands other than ENABLE/DISABLE_NOTIFICATIONS and ACKNOWLEDGE_SVC_PROBLEM – had ACKNOWLEDGE_HOST_PROBLEM in my sights as well. Email handling a whole new topic – sending is easy but routing inbound emails a lot harder – acks , new alerts If nothing else available (on a customer basis) use OTRS to hold alerts and commentary
- Just usual stuff running on the remote platform. OpenVPN because we use Cisco IPSeC vpn connection but have used other software when the Cisco ports are blocked (https anyone ?)
- Slightly stretched version of ITIL Service Catalogue – services and their characteristics , processes are related mainly to change control and configuration item relationships Service catalogues are implemented in a manner that facilitate the registration, discovery, request, execution, and tracking of desired services for catalogue users. Each service within the catalogue typically includes traits and elements such as: Clear ownership of and accountability for the service (a person and often an organization) A name or identification label for the service A description of the service A service categorization or type that allows it to be grouped with other similar services Related service request types Any supporting or underpinning services Service level agreement (SLA) data and information that helps service providers set expectations for their service requestors Who is entitled to request/view the service Associated costs (if any) How to request the service and how its delivery is fulfilled Escalation points and key contacts The more descriptive the service details are, the easier it is for end users of the service catalog to find and invoke the services they desire.
- The agreed list is the hardest to get, pulling teeth… no one wants to be alerted (or the guy that does is no use) , if there is an automated way of doing it – please do it…
- Really hard to get people to do this – ‘job preservation’ – work smarter….. Will get round to Puppet soon……
- Joys of multi –tenant – naming conventions & physical locations combine to make it hard, then change control for CI’s is important.
- ‘Cloud’ – it’s a lovely buzzword – sometimes it fits a solution like a glove – sometimes not. Horizontal scaling and moving the costs directly back to the customer is good. Cloud geographic issues don’t apply – no local data held.
- Engineer around real life problems. For $40 why spend weeks / month tuning & developing when simple hardware / software add on will do the job
- Business process by drag & drop – simple but effective. Explains why a process is at warning and not critical for example.
- Dashboard by drag & drop – can restrict to be locked down for a particular user – only screen they can see (Nagios contacts based)
- Question time….
- Click to add comments