Recommended
More Related Content
Similar to Elk for Sysadmins
Similar to Elk for Sysadmins (20)
Recently uploaded
Recently uploaded (20)
Elk for Sysadmins
- 1. ELK Not just for InfoSec any more
- 2. Who are we? Russel Havens ● Monitoring Architect for Adobe Digital Marketing Business Unit ● Adjunct Professor, BYU ● 25 years in IT Operations, 2 years consulting, 5 years in software development Hayden Panike ● Former log analytics undergraduate researcher ● Storage Engineer Tanner Lund ● Former log analytics undergraduate researcher ● Service Engineer
- 3. ELK
- 4. ● As an developer, I write logs ● As an operations engineer, I live in the log files while troubleshooting ● However: in most organizations where I’ve worked, formal log aggregation and analysis is owned by InfoSec, and access to those were limited (Splunk or Syslog) ELK background and approach
- 5. ● Monitoring is a broad area ○ Up/Down ○ Historical trending ○ Log Analysis ● Using ELK for managing logs of 60+ Nagios servers, web servers, etc. ● Worked with a BYU Capstone team 2013- 2014 ELK approach
- 6. Hayden & Tanner - Took over log management project from previous team - Expanded partnership with BYU OIT to gather logs from servers, SANs, and network equipment (including Wifi hotspots) - Expanded cluster size massively Our ELK History
- 7. We do what we must...
- 8. Partnership with BYU OIT Production ELK Deployment Every 60 seconds at BYU we ingest event logs:* -2,431 network -430,000 IDS -59,000 wireless 62,300,000 total a day*
- 12. This process ought to be formalized Proactive monitoring Logs and Enterprise Monitoring
- 13. Monitoring as a Discipline SOURCES OF DATA -SNMP -stdout -/proc *stat -Logs SYSTEM PIECES -Collector agents -Aggregator nodes -Analysis platform PRINCIPLES -Aggregation -Cause Analysis (reactive) -Behavior Analysis (proactive)
- 14. Queries: Simple Yet Elegant ● As Operation Engineers, we already know lots of keywords. These can be easily leveraged for simple queries. ● error ● failure ● root ● port flapping ● memory
- 15. Simple Yet Elegant ● A simple search for the word “memory” over a 30 day period.
- 16. Simple search ● Nagios 4 can be set for a maximum number of concurrent processes. Here we see 2 overloaded monitoring servers.
- 19. Simple Bin and Count
- 20. Simple Bin and Count
- 21. Simple Bin and Count ● Campus wide phone OS stats
- 22. Simple Bin and Count Business School Administrative Building
- 23. Simple Bin and Count
- 24. Simple Bin and Count
- 25. Simple Bin and Count
- 26. Simple Bin and Count
- 27. -We are at the tip of a transformative iceberg -Machine Learning and Statisticians needed A Call to Arms!
Editor's Notes
- Presentation ideas Who are we What is ELK (everybody there knows this, but this seems obligatory) Background of our approach to the topic, including a short history of what lead to this work SysAdmins use logs for detailed troubleshooting, but usually, formalized log collection and analysis is owned by the InfoSec team. SysAdmins should collect and formally analyze logs as well! (This topic will be large-ish.) RH - Capstone team 2013-2014 HP/TL project this year 1-4: 3 mins OIT partnership give scope of log collection - number & types of logs, teams utilizing the system, etc. Benefits of Simple Search <lots of examples> Simple Bin and Count metrics <some more examples> 5-7: 7 mins Call to use more advanced statistical techniques <one example?> Apply machine learning. Wrap this all in a process 8-9: remaining time Ideas: Lire Lots of Kibana Screenshots Dashboards Wiki/Knowledge Base Statistical Crunching of Data
- This slide should tailor to Prof. Havens and his background with ELK.
- You can start with something as simple as this. ...okay, maybe not quite THIS simple, but simple. One laptop running the whole system.
- This is what you are more likely to deploy as you start implementing ELK.
- And this is what a big, highly available enterprise architecture might look like. You can be as simple or minimalist or as professional/robust as you choose.
- Let’s address monitoring as a discipline SOURCES You have your common SNMP data, which is the core of most modern monitoring There are also more anecdotal or specific sources such as stdout and /proc Lastly there’s logs, which in fact already include some of the information in these other sources -ELK allows us to look at log data AT SCALE, like we can do for SNMP SYSTEM PIECES You must collect data, put it all in one place, and then use some sort of tool to make sense of it. -Maybe it sends out alerts on certain thresholds. Maybe it has a nice GUI. Maybe it just lets you poke around the data. The nice thing about ELK is PASSIVE COLLECTION -Don’t need to install agents in many cases (though you can), rsyslog will do the trick -No requests required, logs are collected constantly in the background. PRINCIPLES You must gather your data, and then analyze it. -When something goes wrong, we dig into whatever sources of data we have to find the root cause. This is reactive. -ELK encourages and enables proactive analysis, since we have a wealth of data at all times. At any given moment, we can open up ELK and look for existing problems, as well as the EARLY WARNING SIGNS of what might be future problems, helping catch issues before they get out of hand or trigger alerts.
- These two monitoring servers are in a data center which has been growing faster than expected. Seeing that they were heavily loaded, we were able to order hardware in to deal with that growth.
- This is a simple count of Nagios NOTIFICATION log entries by nagios_status (OK, WARNING, CRITICAL, UNKNOWN). Digging into this spike (not shown for company security reasons), we found that almost 75% were from a London data center, one of our smaller ones. Filtering on that data center (with one click) and slicing by alert service, we found that 72% of the alerts in the time frame were SSL certificate related. It turned out that a name server had failed, and its redundant backup was responding too slowly.