Splunk .conf2011: Search Language: Intermediate


Published on

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • How can you leverage Splunk?
  • How can you leverage Splunk?
  • Splunk .conf2011: Search Language: Intermediate

    1. 1. Search Language - Intermediate Karen Hodges, Sr. Instructor
    2. 2. <ul><li>Karen Hodges – Senior Instructor – Splunk </li></ul><ul><li>Over 20 years of experience in software training and education in: </li></ul><ul><ul><li>UNIX System Administration </li></ul></ul><ul><ul><li>Intergraph GIS Systems </li></ul></ul><ul><ul><li>Relational Database Management Systems </li></ul></ul><ul><ul><li>BMC Remedy </li></ul></ul><ul><ul><li>Mortgage Fraud Detection </li></ul></ul><ul><ul><li>Real Property Title Search </li></ul></ul><ul><ul><li>Splunk </li></ul></ul>Your presenter . . .
    3. 3. <ul><li>Knowledge Objects </li></ul><ul><ul><li>Tags </li></ul></ul><ul><ul><li>Event types </li></ul></ul><ul><ul><li>Saved searches and alerts </li></ul></ul><ul><li>Advanced searching techniques </li></ul><ul><ul><li>Comparison operators </li></ul></ul><ul><ul><li>The search pipeline </li></ul></ul>Topics
    4. 4. Knowledge Objects
    5. 5. <ul><li>Type in keywords, hit return, get results . . . </li></ul>Splunk as “Search Engine”
    6. 6. <ul><li>Splunk allows you to “store” knowledge alongside your IT data </li></ul><ul><li>Institutional knowledge </li></ul><ul><ul><li>For example: server function or device location </li></ul></ul><ul><li>Learned knowledge </li></ul><ul><ul><li>For example: identify crash precursors or suspicious activity patterns </li></ul></ul><ul><li>You store these in Splunk using Knowledge Objects </li></ul>So Much More than a “Search Engine”
    7. 7. <ul><li>Server names aren’t always very helpful! </li></ul><ul><li>Sometimes they pack too much information into the name </li></ul><ul><li>Sometimes they make them reflect their hobbies/obsessions </li></ul>Scenario – Confusing Server Names
    8. 8. <ul><li>Tags are metadata you can add to field values </li></ul>Knowledge Objects – Tags to the Rescue
    9. 9. <ul><li>Search all hosts tagged as “ webfarm ” </li></ul>Using Tags
    10. 10. <ul><li>IT data is full of strange and confusing message </li></ul><ul><li>Some are alarming! </li></ul><ul><li>Some are low key, but should be alarming </li></ul>Scenario – So Many Different Needles and Hays
    11. 11. <ul><li>Event types are fields based on a search – similar to a saved search </li></ul>Knowledge Objects – Event Types
    12. 12. <ul><li>For example: 2 events in linux_secure </li></ul><ul><li>Save event types to differentiate these 2 events </li></ul><ul><ul><li>pwd_fail_known and pwd_fail_unknown </li></ul></ul>Event Type Example - Different Events
    13. 13. <ul><li>For example: 2 different types of firewalls </li></ul><ul><ul><li>CheckPoint firewall “action=reject” </li></ul></ul><ul><ul><li>Netscreen firewall “action=deny” </li></ul></ul>Event Type Example – Same Event
    14. 14. Using Event Types <ul><li>Use the eventtype as you would any other field </li></ul>
    15. 15. <ul><li>Servers and devices run 24/7 </li></ul><ul><li>Hackers, bugs and crashes (oh my!) are lurking 24/7 </li></ul><ul><li>Humans aren’t 24/7 – they need things like sleep, vacations, lunch, or just a few minutes away from staring at a screen in a freezing cold server room! </li></ul>Scenario – 24/7 Monitoring
    16. 16. <ul><li>Searches can be run on a schedule and be setup to “do something” based on the results </li></ul><ul><li>We call these Alerts </li></ul>Splunk Alerts Never Sleep!
    17. 17. <ul><li>Hackers need a user name AND a password to log in to your systems </li></ul><ul><li>Public web pages often contain names of CEOs, sales folks, etc. splunk.com is no exception </li></ul>Alerting Scenario – Public User Logins
    18. 18. <ul><li>Since only certain users appear on the web page, we can give those users the tag=publicID </li></ul><ul><li>We can use the “ pwd_fail_known ” Event Type we created earlier </li></ul>Leverage Tagging and Event Types
    19. 19. <ul><li>Craft the search that searches for login attempts from public users then create the alert </li></ul><ul><li>Click next to define alert conditions </li></ul>Craft Your Search and Create the Alert
    20. 20. <ul><li>You can specify alert conditions which will trigger the alert </li></ul><ul><li>In our case we are looking for four or more login attempts since after that legitimate users are locked out </li></ul>Alert Conditions
    21. 21. <ul><li>Can send email, create RSS feed, or trigger shell script </li></ul><ul><li>We have opted to have the results included in our email so we can evaluate the severity of the attack easily </li></ul><ul><li>Tracking allows us to view fired alerts in the Alert manager </li></ul>Alert Actions
    22. 22. <ul><li>Use the Alerts menu item in the main Splunk navigation to display the Alerts manager window. </li></ul><ul><li>Click Results to view the events that triggered the alert </li></ul><ul><li>Click Edit to edit the alert settings </li></ul>Alert Manager Failed Logins Failed Logins
    23. 23. Advanced Searching Techniques
    24. 24. <ul><li>Comparison operators make your searches more exacting </li></ul><ul><li>Splunk’s full-featured search language permits you to organize and analyze data in amazing ways! </li></ul>So Much More than a “Search Engine”- Part II
    25. 25. <ul><li>Comparison operators </li></ul><ul><ul><li>!= > < <= >= </li></ul></ul>Towards More Sophisticated Searches
    26. 26. <ul><li>Search is a data generating command </li></ul><ul><li>You can organize and analyze data using the search pipeline </li></ul>The Search Pipeline sourcetype=syslog ERROR | top user | fields - percent Fetch events from disk that match Remove column showing percentage Intermediate results table Intermediate results table Final results table Disk Summarize into table of top 10 users
    27. 27. <ul><li>After the search command use the “|” symbol to pipe your search results to a subsequent command </li></ul><ul><li>For example here we are changing the sort order to sort by user name descending – grouping all the logins together </li></ul>Organize and Analyze Your Data
    28. 28. <ul><li>We’ve already seen sort, there are many MANY more . . . </li></ul><ul><ul><li>dedup removes duplicates </li></ul></ul><ul><ul><ul><li>Weeding out duplicate entries makes results easier to use AND keeps statistical operations more pure </li></ul></ul></ul><ul><ul><li>regex allows you filter your results using a regular expression </li></ul></ul><ul><ul><ul><li>REGEX gurus can filter using all the ?’s and *’s they want! </li></ul></ul></ul><ul><ul><li>transaction allows you to group your events by a certain field and time range </li></ul></ul><ul><ul><ul><li>See all the web pages your boss visited in the past hour from your proxy data </li></ul></ul></ul>Data Processing Commands
    29. 29. <ul><li>When you type in a command after the | symbol Splunk’s Search Assistant provides an instant mini “man page” </li></ul>Splunk Makes Using its Search Language Easy
    30. 30. <ul><li>The table command is useful for visually organizing events </li></ul><ul><li>Columns are displayed in the same order of fields entered in the command </li></ul><ul><ul><li>Column headers are field names </li></ul></ul><ul><ul><li>Rows are field values </li></ul></ul><ul><ul><li>Each row represents an event </li></ul></ul>View Events in a Table
    31. 31. <ul><li>The top command finds the most common values of a given field </li></ul><ul><ul><li>Returns top 10 results by default </li></ul></ul><ul><li>Automatically returns a count and percentage </li></ul><ul><li>Adding limit=# after the top command returns the specified number of results </li></ul>Top Scenario – Getting Top Site Visitors
    32. 32. <ul><li>count returns the number of occurrences of a given field </li></ul><ul><li>The by clause returns a count for each field value of a named field </li></ul>Stats Scenario – Counting Product Sales
    33. 33. <ul><li>Online trading activity is captured in a log file which includes each trader’s unique identification </li></ul><ul><li>Company policy requires that we monitor each trader’s activity in hourly chunks, but the trades are all jumbled up together making it hard to spot patterns in each trader’s trades </li></ul>Transaction Scenario – Monitor Trading Activity
    34. 34. <ul><li>Use transaction to group each trade by TradeID </li></ul><ul><li>Set your time span to an hour and your max pause to one hour in case some traders only have one or two trades per hour </li></ul>Use Transaction to Group Your Trades
    35. 35. <ul><li>Event types and tags are excellent ways to capture existent knowledge as well as knowledge learned from using Splunk </li></ul><ul><li>Splunk’s search language includes many powerful commands which allow you to organize and analyze your data easily </li></ul>Summary
    36. 36. <ul><li>You’ve just seen some of the many ways Splunk can be used to leverage the intelligence in your IT data </li></ul><ul><li>Further your Splunk education with official Splunk training </li></ul><ul><li>Using Splunk – Gets deeper into basic search, alerts, knowledge objects, quick reports and more… </li></ul><ul><li>Searching and Reporting with Splunk – Takes you to the next level leveraging statistical operations and reporting in Splunk </li></ul>Congratulations!
    37. 37. August 15, 2011 Questions? Karen Hodges, Sr. Instructor