The document discusses improving the readability and usefulness of monitoring check outputs. It recommends making outputs readable through formatting, using specific language, and only including necessary information. Checks should aggregate relevant data and change outputs between normal and warning/critical states. The overall goal is to design outputs that provide operators with the key details needed to address issues.

1. About Me
Hi!
I’m Mattis Haase
Sysadmin with C-Store
Responsible for config management and
monitoring
Mattis Haase Icinga Camp Berlin 2017

2. Let’s do an experiment
Next slide is a real world RAID check output
One hard drive is broken
Imagine getting the following mail 3 am in the
morning.
As soon as you figure out which drive is broken,
raise your hand.
Mattis Haase Icinga Camp Berlin 2017

3. OK ServeRAIDM5110e.controllerSerial=XXXXXX ServeRAIDM5110e.controllerCache=365MB
Drive0.disk_Model=RandomHarddriveName Drive1.disk_Model=RandomHarddriveName
Drive2.disk_Model=RandomHarddriveName Drive3.disk_Model=RandomHarddriveName
Drive4.disk_Model=RandomHarddriveName Drive5.disk_Model=RandomHarddriveName
Drive6.disk_Model=RandomHarddriveName Drive7.disk_Model=RandomHarddriveName
Drive0.disk_Port=0 Drive1.disk_Port=1 Drive2.disk_Port=2 Drive3.disk_Port=3 Drive4.disk_Port=4
Drive5.disk_Port=5 Drive6.disk_Port=6 Drive7.disk_Port=7 Drive0.disk_AttachedType=SATA
Drive1.disk_AttachedType=SATA Drive2.disk_AttachedType=SATA Drive3.disk_AttachedType=SATA
Drive4.disk_AttachedType=SATA Drive5.disk_AttachedType=SATA Drive6.disk_AttachedType=SATA
Drive7.disk_AttachedType=SATA Drive0.disk_Type=SSD Drive1.disk_Type=SSD
Drive2.disk_Type=SSD Drive3.disk_Type=SSD Drive4.disk_Type=SSD Drive5.disk_Type=SSD
Drive6.disk_Type=SSD Drive7.disk_Type=SSD Drive0.disk_Bandwidth=6.0Gb/s
Drive1.disk_Bandwidth=6.0Gb/s Drive2.disk_Bandwidth=6.0Gb/s Drive3.disk_Bandwidth=6.0Gb/s
Drive4.disk_Bandwidth=6.0Gb/s Drive5.disk_Bandwidth=6.0Gb/s Drive6.disk_Bandwidth=6.0Gb/s
Drive7.disk_Bandwidth=6.0Gb/s Drive0.disk_Status=1 Drive1.disk_Status=1 Drive2.disk_Status=0
Drive3.disk_Status=1 Drive4.disk_Status=1 Drive5.disk_Status=1 Drive6.disk_Status=1
Drive7.disk_Status=1 Drive0.disk_Size=465.762GB Drive1.disk_Size=465.762GB
Drive2.disk_Size=465.762GB Drive3.disk_Size=465.762GB Drive4.disk_Size=465.762GB
Drive5.disk_Size=465.762GB Drive6.disk_Size=465.762GB Drive7.disk_Size=465.762GB
Drive0.disk_Serial=XXX Drive1.disk_Serial=XXX Drive2.disk_Serial=XXX Drive3.disk_Serial=XXX
Drive4.disk_Serial=XXX Drive5.disk_Serial=XXX Drive6.disk_Serial=XXX Drive7.disk_Serial=XXX
Volume0.volumeRaidLevel=RAID6 Drive6.disk_Temperature=25.00 Drive7.disk_Temperature=25.00
Drive5.disk_Temperature=26.00 Drive4.disk_Temperature=27.00 Drive1.disk_Temperature=27.00
Drive2.disk_Temperature=25.00 Drive3.disk_Temperature=25.00 Drive0.disk_Temperature=26.00
Mattis Haase Icinga Camp Berlin 2017

4. That sucked!
Why did it suck?
● For many reasons
● It is an unreadable mess
1.Difficult to quickly skip between values
2.Easy to get lost
Mattis Haase Icinga Camp Berlin 2017

5. Rule 1: Make it readable (formatting)
How to make it readable?
● USE NEWLINE
● html tags: <b> <i> <u> <br>
● html tables
What you cannot do:
● No CSS
● No emojis
● No javascript
Mattis Haase Icinga Camp Berlin 2017

6. Rule 1: Make it readable (language)
But wait, there is more!
Don’t use ambigious language:
Raid_Status=1
Write it out instead:
Raid_Status=DEGRADED
Mattis Haase Icinga Camp Berlin 2017

7. Experiment MK2
Next slide is a real world RAID check output
One hard drive is broken
Imagine getting the following mail 3 am in the
morning.
As soon as you figure out which drive is broken,
raise your hand.
Mattis Haase Icinga Camp Berlin 2017

8. Mattis Haase Icinga Camp Berlin 2017

9. That (still) sucked!
Why did it suck?
Redundant information!
1.Disk ID = Disk Port
2.Multiple disks with
same information
3.Some info repeated in
perfdata
Mattis Haase Icinga Camp Berlin 2017

10. More reasons why that (still)
sucked!
Useless information!
1.Serial # might be in
inventory system
2.Things that are working
are not interesting
Mattis Haase Icinga Camp Berlin 2017

11. Rule 2: Show the minimum info
necessary
1. Avoid redundant information
2. Don’t show information you don’t need
3. Group similar things together
Mattis Haase Icinga Camp Berlin 2017

12. Experiment MK3
Next slide is a real world RAID check output
One hard drive is broken
Imagine getting the following mail 3 am in the
morning.
As soon as you figure out which drive is broken,
raise your hand.
Mattis Haase Icinga Camp Berlin 2017

13. Mattis Haase Icinga Camp Berlin 2017

14. OK ServeRAIDM5110e.controllerSerial=XXXXXX ServeRAIDM5110e.controllerCache=365MB
Drive0.disk_Model=RandomHarddriveName Drive1.disk_Model=RandomHarddriveName
Drive2.disk_Model=RandomHarddriveName Drive3.disk_Model=RandomHarddriveName
Drive4.disk_Model=RandomHarddriveName Drive5.disk_Model=RandomHarddriveName
Drive6.disk_Model=RandomHarddriveName Drive7.disk_Model=RandomHarddriveName
Drive0.disk_Port=0 Drive1.disk_Port=1 Drive2.disk_Port=2 Drive3.disk_Port=3 Drive4.disk_Port=4
Drive5.disk_Port=5 Drive6.disk_Port=6 Drive7.disk_Port=7 Drive0.disk_AttachedType=SATA
Drive1.disk_AttachedType=SATA Drive2.disk_AttachedType=SATA Drive3.disk_AttachedType=SATA
Drive4.disk_AttachedType=SATA Drive5.disk_AttachedType=SATA Drive6.disk_AttachedType=SATA
Drive7.disk_AttachedType=SATA Drive0.disk_Type=SSD Drive1.disk_Type=SSD
Drive2.disk_Type=SSD Drive3.disk_Type=SSD Drive4.disk_Type=SSD Drive5.disk_Type=SSD
Drive6.disk_Type=SSD Drive7.disk_Type=SSD Drive0.disk_Bandwidth=6.0Gb/s
Drive1.disk_Bandwidth=6.0Gb/s Drive2.disk_Bandwidth=6.0Gb/s Drive3.disk_Bandwidth=6.0Gb/s
Drive4.disk_Bandwidth=6.0Gb/s Drive5.disk_Bandwidth=6.0Gb/s Drive6.disk_Bandwidth=6.0Gb/s
Drive7.disk_Bandwidth=6.0Gb/s Drive0.disk_Status=1 Drive1.disk_Status=1 Drive2.disk_Status=0
Drive3.disk_Status=1 Drive4.disk_Status=1 Drive5.disk_Status=1 Drive6.disk_Status=1
Drive7.disk_Status=1 Drive0.disk_Size=465.762GB Drive1.disk_Size=465.762GB
Drive2.disk_Size=465.762GB Drive3.disk_Size=465.762GB Drive4.disk_Size=465.762GB
Drive5.disk_Size=465.762GB Drive6.disk_Size=465.762GB Drive7.disk_Size=465.762GB
Drive0.disk_Serial=XXX Drive1.disk_Serial=XXX Drive2.disk_Serial=XXX Drive3.disk_Serial=XXX
Drive4.disk_Serial=XXX Drive5.disk_Serial=XXX Drive6.disk_Serial=XXX Drive7.disk_Serial=XXX
Volume0.volumeRaidLevel=RAID6 Drive6.disk_Temperature=25.00 Drive7.disk_Temperature=25.00
Drive5.disk_Temperature=26.00 Drive4.disk_Temperature=27.00 Drive1.disk_Temperature=27.00
Drive2.disk_Temperature=25.00 Drive3.disk_Temperature=25.00 Drive0.disk_Temperature=26.00
Mattis Haase Icinga Camp Berlin 2017

15. Software exists to aid man
Mattis Haase Icinga Camp Berlin 2017

16. Software exists to aid man
Mattis Haase Icinga Camp Berlin 2017

17. Rule 3: Let the check do work for
you
1. Think about what information an employee
would need to fix a problem
2. Aggregate that information into the check
output
Mattis Haase Icinga Camp Berlin 2017

18. Experiment MK4
Next slide is a real world RAID check output
One hard drive is broken
Imagine getting the following mail 3 am in the
morning.
As soon as you figure out which drive is broken,
raise your hand.
Mattis Haase Icinga Camp Berlin 2017

19. Mattis Haase Icinga Camp Berlin 2017

20. Summary
1. Make check outputs readable by using
formatting and specific language
2. Only show information needed, grouping
information together where appropiate
3. Have the check accumulate relevant
information
4. Change check output between OK and
WARNING/CRITICAL states where appropiate
Mattis Haase Icinga Camp Berlin 2017