Handout: 'Open Source Tools & Resources'


Published on

21 people attended the July 2014 program meeting hosted by BDPA Cincinnati chapter. The topic was 'Open Source Tools and Resources'. The guest speaker was Greg Greenlee (Blacks In Technology).

'Open source' refers to a computer program in which the source code is available to the general public for use or modification from its original design. Open source code is typically created as a collaborative effort in which programmers improve upon the code and share the changes within the community. Open source sprouted in the technological community as a response to proprietary software owned by corporations. Over 85% of enterprises are using open source software. Managers are quickly realizing the benefit that community-based development can have on their businesses. This month, we put on our geek hats and detective gloves to learn how we can monitor our computers’ environments using open source tools. This meetup covered some of the most popular ‘Free and Open Source Software’ (FOSS) tools used to monitor various aspects of your computer environment.

Published in: Education
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Handout: 'Open Source Tools & Resources'

  1. 1. Monitoring your Infrastructure using Open Source ToolsSource Tools by Greg Greenlee
  2. 2. greg@blacksintechnology:~$ whoami ● Founder of Blacks In Technology organization ● Systems Engineer● Systems Engineer ● FOSS lover ● Avid comic book reader (Marvel) ● Father ● Husband
  3. 3. What is Open Source Software? ● Source code is openly shared ● Licensed to allow freedom to modify, copy,● Licensed to allow freedom to modify, copy, study
  4. 4. Why Open Source? ● Pros o Free (as in beer) and sometimes not o code open to modification o vulns are detected and fixed quickero vulns are detected and fixed quicker o entire community working on it (depends on popularity) ● Cons o documentation sometimes lacks o may prove difficult to implement
  5. 5. Why monitor? ● You want to know when something goes wrong ● Insight into your environment o proactive ● Business needs● Business needs o analytics o trending data ● Event correlation
  6. 6. Tools to monitor your infrastructure ● Nagios (monitoring and alerting) ● ELK Stack (log aggregation, search and analysis) o ElasticSearch (search engine based on Lucene) Logstasho Logstash (log aggregator and manipulator) o Kibana (front end) ● Cacti (data graphing) ● NagVis (visualization) ● nTopNG (protocol analyzer)
  7. 7. Nagios ● Created by Ethan Galstad (1999) - originally called NetSaint ● http://www.nagios.org ● http://www.nagios.org/download ● Very customizable ● Nagios XI o standard and enterprise edition (http://www.nagios.com/products/nagiosxi/edition-comparison) o aids in large scale configuration, reporting o includes support services (http://www.nagios.com/products/nagiosxi/pricing) ● Nagios core o free o can purchase support
  8. 8. What does Nagios do? Monitors and alerts
  9. 9. What can Nagios monitor? 1. Applications 2. Network services (smtp, http, dns, ssh….) 3. Hosts 4. Host resources (disk, cpu, memory, processes)4. Host resources (disk, cpu, memory, processes) 5. Web sites 6. Sensors (temperature, humidity, power consumption..) 7. Just about anything you can write a script for!!!
  10. 10. How does Nagios monitor? ● Local plugins o network services are running ● Remote plugins (NRPE NSClient++) o disk utilizationo disk utilization o memory utilization o cpu utiliziation ● Custom Scripts (Perl, Python, Bash, etc.) ● More plugins and nagios goodness o http://exchange.nagios.org/#/
  11. 11. How does it work? Uses a host to service relationship ● Hosts (switch, router, server, sensors) ● Service (http, dns, temperature, ping, ftp) Uses traffic light system to indicate stateUses traffic light system to indicate state ● Red - critical ● Yellow - warning ● Green - ok
  12. 12. How does it work cont’d Makes use of configuration files ● templates ● hosts and host groups● hosts and host groups ● services and services groups ● contacts and contact groups ● notifications ● escalations
  13. 13. Host and service config file # Host definition define host{ use generic-host ; Name of host template to use host_name domain- server-1 alias Name Server address x.x.x.x check_command check-host- alive hostgroups fakegroup # Service definition define service{ use generic-service ; Name of service template to use host_name domain-server-1 service_description DNS is_volatile 0 check_period 24x7 max_check_attempts 3 normal_check_interval 1 retry_check_interval 1 # 'check_dns' command definition define command{ command_name check_dns command_line $USER1$/check_dns -H www.profitability.net -s $HOSTADDRESS$ } hostgroups fakegroup contact_groups novell- admins max_check_attempts 10 notification_interval 30 notification_period 24x7 notification_options d,u,r parents internalswitch-4-3-7-1 } retry_check_interval 1 contact_groups novell-admins notification_interval 30 notification_period 24x7 notification_options w,u,c,r,f,s check_command check_dns }
  14. 14. How does it look? ping check ping check ● service check ○ http ○ ftp
  15. 15. How do I get alerts? email text IMIM
  16. 16. Nagios demoNagios demo
  17. 17. ELK Stack (log aggregation and analysis) ● Elasticsearch (Apache Lucene search engine) ● Logstash (collects and parses logs)● Logstash ● Kibana (front end gui allows you to visualize data)
  18. 18. Why do we need centralized log server? ● Easy to find stuff ● Everything has logs o operating systemso operating systems o applications o devices ● Correlation of events ● Insight into environment
  19. 19. Elasticsearch ● http://www.elasticsearch.org/ ● based on Apache Lucene ● indexes data ● full text search ● distributed (scales out) ● highly available (can build clusters) ● schema free (tries to structure data) ● real time data
  20. 20. Elasticsearch DemoElasticsearch Demo look at Elasticsearch Head
  21. 21. Logstash Created by Jordan Sissel Allows you to ingest data from anywhere (input)Allows you to ingest data from anywhere (input) Centralizes and parses log data (filter) Display data (output)
  22. 22. logstash ● input o syslog o file o tcp o udp o eventlog o twitter o irco irc ● filter (modifies) o grok (parses unstructured data) o mutate (mutates your events such as renaming, replacing, modifying fields) ● output o csv o email o file o elasticsearch o http
  23. 23. flow of data input (tcp, syslog, file) filter (grok, mutate) output (stdout, elasticsearch)
  24. 24. Logstash format input { tcp { port => 5000 type => syslog } udp { port => 5000 type => syslog } } filter { if [type] == "syslog" {if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } syslog_pri { } date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } } output { elasticsearch { host => localhost } stdout { codec => rubydebug } }
  25. 25. Kibana ● visualization tool ● interact with your data in real time● interact with your data in real time ● drill down and view only data you want ● make sense of your data o bar or line graphs, maps and pie charts
  26. 26. ELK stack demoELK stack demo
  27. 27. NagVis ● http://www.nagvis.org/ ● Nagios add on visualisation tool o allows you to visualize Nagios data ● Allows you to create maps of your IT environment● Allows you to create maps of your IT environment ● Allows you to visualize work flows and processes ● create graphic out of performance data o disk usage o cpu usage
  28. 28. Nagvis screenshots
  29. 29. Nagvis demoNagvis demo
  30. 30. Cacti ● http://www.cacti.net/ ● front end for RRDTool (Round Robin Database) o records data in intervals o graphs data pointso graphs data points ● Useful for seeing data trends o bandwidth o power consumption o cpu usage o memory usage
  31. 31. Cacti demoCacti demo
  32. 32. nTOPng http://www.ntop.org/ Network traffic probe and analyzer Based on original ntop Supported platforms (Windows and Unix) ● Sort network traffic according to many protocols ● Show network traffic and IPv4/v6 active hosts ● Store on disk persistent traffic statistics in RRD format ● Geolocate hosts ● Analyse IP traffic and sort it according to the source/destination ● Display IP Traffic Subnet matrix (who’s talking to who?) ● Report IP protocol usage sorted by protocol type ● Act as a NetFlow/sFlow collector for flows generated by routers (e.g. Cisco and Juniper) or switches (e.g. Foundry Networks) when used together with nProbe. ● Produce HTML5/AJAX network traffic statistics
  33. 33. What can you do with nTOPng? ● Network troubleshooting ● Drill down into traffic flows o source and destination ip address o source and destination porto source and destination port o bandwidth o protocol o application
  34. 34. nTOPng screen shotsnTOPng screen shots
  35. 35. nTopNG set up
  36. 36. Thank you!!! email: greg@blacksintechnology.net twitter: @blkintechnology facebook: http://www.facebook.com/blacksintechnology website: http://www.blacksintechnology.netwebsite: http://www.blacksintechnology.net LinkedIn: Blacks In Technology
  37. 37. ????????????