Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Icinga Camp Berlin 2018 - Automated Configuration with Icinga Director

0 views

Published on

The world out there is neither perfect nor uniform, and that’s good as it is. You’re using VMware, but also running KVM. A little bit of AWS is a must, and something has been deployed to Azure. Evaluation projects for Mesos/Marathon and Kubernetes are on the run, some of them already running in production. A lot of information is in your Active Directory, but some departments are only half-way in. A lot of orphaned entries are to be found. Some use Puppet, experiments with other tools are going on, and quite some things are still under manual control. There are three CMDBs, but none of those are complete. There is an Excel sheet for IP address reservations. Oh, and by the way, network people are of course using their very own tool-chain.
In such kinds of environments, Icinga Director is in full force. Given concrete implementations from daily practice, this presentation shows how to build a fully automated monitoring system based on varying data sources. Optionally, you can have different degrees of automation to accommodate varying speeds within individual teams.
This shouldn’t be an introduction to Director. Given dedicated solutions for specific problems in real projects, the possibilities of this software will be shown.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Icinga Camp Berlin 2018 - Automated Configuration with Icinga Director

  1. 1. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf AUTOMATED MONITORING IN HETEROGENEOUS ENVIRONMENTS
  2. 2. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf THOMAS GELF ● Principal Consultant @netways ● Lead Architect @icinga SELF-INTRODUCTION
  3. 3. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf HI BERLIN! WHO IS…?
  4. 4. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf DISCLAIMER NOT EVERYTHING SHOULD BE TAKEN LITERALLY.
  5. 5. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf AUTOMATION IMPORT → SYNC → DEPLOYMENT. JOBS.
  6. 6. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf AUTOMATION OVERVIEW
  7. 7. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf AUTOMATION ● Any Data Source. Shipped with Director or from a Module ● Fills "deduplicted" tables, isolated from others ● Doesn‘t disturb other tasks and daily business ● Therefore slower sources are no problem at all ● Import is atomic -> all or nothing IMPORT
  8. 8. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf AUTOMATION ● Decoupled from Import ● Builds old and new objects in Memory ● Writes Diff to Database ● Single Transaction - all or nothing, once again ● But: Activity Log is decoupled for performance reasons SYNC
  9. 9. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf AUTOMATION ● Jobs run as background services ● Configurable independently ● No defined order ● Jobs can trigger Import, Sync and Deployments JOBS
  10. 10. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf SOME NUMBERS HINT: DIRECTOR v1.0.0 HAS BEEN TAGGED ON MARCH 24th, 2016
  11. 11. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf SOME NUMBERS ● Active since slightly more than 2 years. ● 150k single Import Runs (currently 3-600 a day) ● 16k Sync Runs (currently about 70 every day) ● 5.7k Deployments (currently 30+ automated ones every day) ● 660k Activity Log entries ● 93k Services, 3.3k Hosts, average check interval 1min FROM A PRODUCTIVE ENVIRONMENT, YESTERDAY
  12. 12. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf SCENARIO 1 IDO SYNC
  13. 13. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf IDO SYNC ● Running in parallel during Migration ● Pretty often: Icinga 1 is running, Icinga 2 being prepared ● Comfortable Option: Import from the IDO ● Preferrably only Hosts with attributes ● Groups and memberships ● Build Service checks from scratch to benefit from Icinga 2 TASK / CHALLENGE
  14. 14. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf IDO SYNC ● Important for IDO: always check icinga_objects.is_active A FIRST QUERY
  15. 15. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf IDO SYNC ADD CUSTOM VARS TO THE MIX
  16. 16. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf AUTOMATION EVEN MORE CUSTOM VARS?
  17. 17. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf AUTOMATON ● Emty Strings ● Invalid values CLEAN UP DIRTY DATA
  18. 18. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CONCLUSION NICE. WHAT ELSE SPEAKS SQL? HEY, THERE IS A CMDB!
  19. 19. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf SCENARIO 2 DATA CLEANUP
  20. 20. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf WE HAVE A CMDB IMPORT IS RUNNING. BUT.
  21. 21. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf INSIGHT OUR CMDB DATA IS GARBAGE
  22. 22. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CHALLENGES ● Location ● Nürnberg ● Nuernberg ● Nuremberg ● Hostname: ● spooler ● PRINTSRV ● FILESERVER.example.com ● localhost ● dbserver.example.com OUR CMDB DATA IS GARBAGE
  23. 23. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf DECISION NOT WITH US, WE‘LL VALIDATE ALL THE VALUES
  24. 24. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CONCLUSION BEING STRICT DOESN‘T WORK. SOCIAL ENGINEERING DIDN‘T HELP EITHER.
  25. 25. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CONCLUSION ● Some people are slower ● It always worked that way ● Others are not willing to do extra work ● Those who want to work have insufficient permissions THIS DOESN‘T WORK
  26. 26. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf DECISION WE‘LL FIX IT ON OUR OWN. LET‘S MAP THAT DIRTY DATA!
  27. 27. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
  28. 28. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf ● We can still be strict ● New variant? ● New list entry!
  29. 29. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CONCLUSION KEEPING THAT MAP UP TO DATE IS NO FUN
  30. 30. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CONCLUSION ● That‘s a job for the apprentice ● Director provides no "Access to a single list" restriction ● The apprentice shouldn‘t be granted other permissions KEEPING THAT MAP UP TO DATE IS NO FUN
  31. 31. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf DECISION GIVE HIM A CSV-FILE, WE‘RE USING IT AS AN IMPORT SOURCE
  32. 32. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
  33. 33. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CONCLUSION WHEN SOMETHING GOES WRONG WE HAVE TO TELL THE APPRENTICE
  34. 34. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CONCLUSION ● We are not willing to care ● This shouldn‘t bother us ● It‘s time for a Health-Check WHEN SOMETHING GOES WRONG WE HAVE TO TELL THE APPRENTICE
  35. 35. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CONCLUSION ● Notifications? ● Directly to the apprentice! ...WE HAVE TO TELL THE APPRENTICE
  36. 36. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf INSIGHT OUR APPRENTICE IS SUPER INTELLIGENT
  37. 37. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf INSIGHT ● He makes the most creative excuses ● „My dog ate the Notification“ ● Automation can be blocked for a day or so ● Physical violence could improve his motivation ● HR-people don‘t want us to practice that OUR APPRENTICE IS SUPER INTELLIGENT
  38. 38. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CONCLUSION WE NEED TO ESCALATE IN TIME
  39. 39. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
  40. 40. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf SCENARIO 3 MONITOR ALL THE THINGS.
  41. 41. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf MONITOR ALL THE THINGS ● Disclaimer: I do not consider this being useful at all ● Some people continue to claim this being an essential feature ● So let me show you that we could. ● If we would. NOT A REAL CHALLENGE
  42. 42. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf MONITOR ALL... ● git clone ● module enable ● Configure an Import Source NOT A REAL CHALLENGE
  43. 43. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf MONITOR ALL... ● ...that‘s it. NOT A REAL CHALLENGE
  44. 44. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CONCLUSION THIS RARELY MAKES ANY SENSE. ANYWAYS, WE CAN.
  45. 45. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf SCENARIO 4 CONTACT WITH THE DARK SIDE
  46. 46. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf WINDOWS MONITORING ● We were successful. Success sucks. ● Have been told to also monitor our Windows Servers ● Not in the main CMDB, they are running their own one ● Their CMDB is not available, ongoing migration project TASK / CHALLENGE
  47. 47. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf INSIGHT WINDOWS SERVERS ARE TO BE FOUND IN THE ACTIVE DIRECTORY
  48. 48. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
  49. 49. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf USE PROPERTY MODIFIERS SOME HAVE BEEN BUILT FOR ACTIVE DIRECTORY – USE THEM
  50. 50. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf BITMASK MATCH A SPECIAL PROPERTY MODIFIER
  51. 51. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf SCENARIO 5 EVERYTHING IS GOING TO BE VIRTUALIZED
  52. 52. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf VIRTUALIZE ALL THE THINGS ● All Servers are going to be virtualized ● On our metal there is running only ESX TASK / CHALLENGE
  53. 53. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf VIRTUALIZE ALL THE THINGS ● https://github.com/Icinga/icingaweb2-module-vsphere ● Enable the module ● Configure an Import Source TASK / CHALLENGE
  54. 54. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
  55. 55. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf VIRTUALIZE ALL THE THINGS ● Import works fine ● Lots of single checks are hammering our VCenter ● Checks based on SDK eat a lot of memory ● Many of them in parallel waste lots of resources ● It‘s a little bit boring and annoying. CONCLUSION
  56. 56. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CONCLUSION THIS MUST BECOME MORE FANCY.
  57. 57. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
  58. 58. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
  59. 59. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
  60. 60. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
  61. 61. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
  62. 62. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
  63. 63. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf VIRTUALIZE ALL THE THINGS https://github.com/Thomas-Gelf/icingaweb2-module-vspheredb It‘s done when it is done. But as you could see, we are very very close. TASK / CHALLENGE
  64. 64. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf INSIGHT THAT‘S AMAZING. BUT THERE IS STILL HARDWARE. THAT‘S SOOOO 90‘s
  65. 65. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf SCENARIO 6 LET‘S MOVE EVERYTHING INTO THE CLOUD
  66. 66. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CONCLUSION EVERYBODY IS DOING AWS – WE MUST FOLLOW THEM
  67. 67. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf WE ARE IN THE CLOUD ● https://github.com/Icinga/icingaweb2-module-aws ● Enable the module ● Define an Import Source IMPORT FROM AWS
  68. 68. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
  69. 69. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf ● You could import all your instances, but this rarely makes any sense ● Focus on your applications ● Autoscaling Groups are usually a good match
  70. 70. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf INSIGHT THEY GOT ALL OUR DATA. WHO EXPECTED THAT?
  71. 71. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CONCLUSION LET‘S DO CONTAINERS. EVERYBODY DOES.
  72. 72. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf WE ARE CONTAINER ● Marathon/Mesos https://github.com/b0e/icingaweb2-module-marathon ● Proxmox https://github.com/nbuchwitz/icingaweb2-module-pve ● Kubernetes? IMPORT FROM...
  73. 73. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
  74. 74. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf INSIGHT IT‘S EASY. DON‘T BE SCARED. THINK APPLICATIONS.
  75. 75. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf SCENARIO X BEYOND THE CLOUD
  76. 76. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf SCENARIO X ● Everybody is doing „a little bit of Cloud“ ● Life in the real world out there is rough ● Time-tested configuration management systems ● Kind of standard in Enterprise environments ● Let‘s have a look at the related OSI Model BEYOND THE CLOUD
  77. 77. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf
  78. 78. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf CONFIGURATION MANAGEMENT YOU GROW ALONG WITH YOUR TASKS
  79. 79. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf THANK YOU!
  80. 80. #IcingaCamp | March 8th, 2018 | Berlin Thomas Gelf QUESTIONS? THANK YOU!

×