Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

TSTAS, the Life of a Splunk Trainer and using DevOps in Splunk Development

497 views

Published on

TSTAS, the Life of a Splunk Trainer and using DevOps in Splunk Development
Some interesting talks about using TSTATS and the internal Splunk logs, have a Splunk Trainer share his journey with Splunk and how he's managed to achieve every possible Splunk certification (over 10!), and a short discussion about emerging thoughts of using development/release frameworks with Splunk deployments.

Published in: Data & Analytics
  • Be the first to comment

TSTAS, the Life of a Splunk Trainer and using DevOps in Splunk Development

  1. 1. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Splunk User Group Edinburgh
  2. 2. © 2018 SPLUNK INC. Harry McLaren ● Alumnus of Edinburgh Napier (Active Mentor) ● Managing Consultant at ECS ● Leader of the Splunk User Group Edinburgh
  3. 3. © 2018 SPLUNK INC. Introduction to ECS Splunk Partner - UK – Type: Security / IT Operations / Managed Services (SOC / Splunk) – Awards: Splunk Revolution Award & Splunk Partner of the Year
  4. 4. © 2018 SPLUNK INC. Agenda • Housekeeping: Event Overview & House Rules • Tstats, _internals and Me (Andrew McManus) • Journey of a Splunk Trainer (Tom Wise) • Development & Release Life-cycles with Splunk (Harry McLaren)
  5. 5. © 2018 SPLUNK INC. Splunk [Official] User Group “The overall goal is to create an authentic, ongoing user group experience for our users, where they contribute and get involved” ● Technical Discussions ● Sharing Environment ● Build Trust ● No Sales!
  6. 6. © 2018 SPLUNK INC.© 2018 SPLUNK INC. | tstats, _internals and Me Or How I Stopped Worrying About Expensive Search Queries, and Learned to Love Interrogating the tsidx Andrew McManus
  7. 7. © 2018 SPLUNK INC. About Myself ● Associate Security Consultant at ECS ● Prior - Senior/Security Operations Center Analyst at ECS ● Credentials: Power User (Admin this weekend, hopefully) ● Current tasks at <redacted>: Troubleshooting configuration and investigation.
  8. 8. © 2018 SPLUNK INC. Why talk about this? ● No (Splunk) Environment is perfect. – Scheduling, search run times, missing data sources ● Insight into environment required ● Internal logs provides insight into performance ● | tstats provides insight into data ● Combined, issues can be detected and squashed.
  9. 9. © 2018 SPLUNK INC. What Splunk logs about itself? ● Splunk logs and indexes everything about itself ● _audit – user activity – Login attempts, user searches, configuration changes – Everything thrown together in one sourcetype, audittrail ● _internal – many sourcetypes containing various logs – Scheduler logs, CPU/Mem usage, license usage and more. – Sourcetype per log type. – Events can have log_levels (INFO, WARN, ERROR) ● _introspection – system metrics. – Per Process metrics ● _fishbucket – checkpoints for ingested files
  10. 10. © 2018 SPLUNK INC. Scenario 1 – Licenser Issues ● Alert came in the prior day regarding your Licenser ● Don’t have access to DMC but _internal ● Use licenser logs to determine what data blew the license: ● Demo 1): – index=_internal source="/opt/splunk/var/log/splunk/license_usage.log" st=* | bucket _time span=1d | eventstats sum(b) as total_used, values(poolsz) as limit by _time | eval over_limit=if(total_used>limit, "YES", "NO") | stats sum(eval(b/1024/1024)) as mb values(over_limit) by st, _time ● (Can also check per_index_thruput metrics for indexers) – index=_internal metrics kb series!=_* group="per_index_thruput" | eval indexed_GB = (kb / 1024 / 1024) | timechart eval(round((sum(indexed_GB)),2)) as TotalGB fixedrange=t span=1d
  11. 11. © 2018 SPLUNK INC. Scenario 2 – Missing Data ● Problem: ● Scheduled search gets data from number of logs w/ UTC timestamps ● Analyst notices that log source never shows up in search ● Search re-ran over same period the next day: ● Missing logs present when manually searched later than scheduled.
  12. 12. © 2018 SPLUNK INC. Scenario 2 – Missing Data (cont) ● Investigation: ● Use _internal index to investigate scheduler logs to see when search runs and what time-frame it scans. – Or _audit ● Find events that aren’t indexed in time for search with tstats
  13. 13. © 2018 SPLUNK INC. | tstats ● Uses tsidx files to report on: – Indexed values – Accelerated data models ● If field is not indexed, cannot be used in search ● Fast ● Seriously fast.
  14. 14. © 2018 SPLUNK INC. Tsidx file ● Can use walklex to investigate what’s in a file. – Returns unique id (term ID), how many times the term occurs and the term. ● Example:
  15. 15. © 2018 SPLUNK INC. | tstats ● Can find event counts per index, sourcetype or source for example. ● Metrics with indexing times can be determined. ● Fast stats on Accelerated Data Models can be determined in seconds.
  16. 16. © 2018 SPLUNK INC. Example – TSTATS count events by host | stats countindex = main by host| tstats count where|tstats count where index=main by host, _time span=1h
  17. 17. © 2018 SPLUNK INC. How Fast? Without Tstats: 9.002s. With Tstats: 0.183s 48x decrease in time taken.
  18. 18. © 2018 SPLUNK INC. Revisiting Scenario 2 ● Using tstats, you can see every hour, events from source comes in up to a hour later. – | tstats earliest(_indextime) as it, latest(_indextime) as lit where index=<index> sourcetype=<sourcetype> by source, _time span=1h | convert ctime(it) ctime(lit) ● This source is logging later than expected – Another more complex search: – | tstats earliest(_indextime) as earliest_indexed, latest(_indextime) as latest_indexed where index=<index> sourcetype=<sourcetype> by source, _time span=1h | rename _time as time_window | join time_window [ search index=_audit info=completed savedsearch_name=<savedsearch_name> | eval time_window=round(api_et,0) | rename _time as search_time | table search_time time_window savedsearch_name] | eval data_missing = if(latest_indexed>=search_time, "YES", "NO") | table savedsearch_name search_time time_window source data_missing earliest_indexed latest_indexed | sort search_time source | convert ctime(search_time) ctime(time_window) ctime(earliest_indexed) ctime(latest_indexed) ● Solution ● Change Scheduling for alert until after all logs come in.
  19. 19. © 2018 SPLUNK INC. Other examples: ● Quick Glass pane view of event counts per index/host/source etc…
  20. 20. © 2018 SPLUNK INC. Compare Event Counts to Run Times
  21. 21. © 2018 SPLUNK INC. Scenario 1 in tstats? ● Internals by default appear to be put into a datamodel. ● index=_internal source="/opt/splunk/var/log/splunk/license_usage.log" st=* | timechart span=1h sum(eval(b/1024/1024/1024)) as gb by st ● Link: Search ● Or: ● | tstats sum(server.licenser.daily_usage.gb) as usage from datamodel=internal_server where nodename=server.licenser.quota groupby server.st, _time span=1h ● Link: Search
  22. 22. © 2018 SPLUNK INC. Further Reading ▶ .conf2017: Searching FAST: How to Start Using tstats and Other Acceleration Techniques - David Veuve • https://conf.splunk.com/files/2017/slides/searching-fast-how-to-start-using-tstats-and-other- acceleration-techniques.pdf • Covers Data Acceleration Models in greater detail. ▶ .conf2017: Worst Practices... And How To Fix Them - Jeff Champagne • https://conf.splunk.com/files/2017/slides/worst-practicesand-how-to-fix-them.pdf • (2016 the same content, except talks about Virtualisation)
  23. 23. © 2018 SPLUNK INC. Any Questions? ● Have you made anything cool with tstats or _internal indexes? Contact Me: Andrew McManus – andrew.mcmanus@ecs.co.uk Slack – On Splunk and SecurityScotland Slack Channels
  24. 24. My Journey
  25. 25. $ whoami  Tom Wise  Security Consultant & Splunk Specialist @ ECS  Splunk Consultant II  Splunk Architect II  Splunk Trainer
  26. 26. Certifications….so far!
  27. 27. Accreditations…so far! Splunk Accredited Sales Rep I Splunk Accredited Sales Rep II Splunk Accredited IT & App Sales Rep Splunk Accredited Sales Engineer I Splunk Accredited Implementation Fundamentals Splunk Accredited Core Implementation Splunk Accredited ES Implementation
  28. 28. Origin Story  Every Splunk hero has one!  Spanned 2 Continents  Varsity Baby!  Boom…Injury  Return to roots.  Man up & save the world one event at a time!
  29. 29. 1st Foray into IT Wilderness
  30. 30. Space…the next frontier!
  31. 31. Personal Achievements
  32. 32. UK Challenge 2015 Lake District 6th / 60 Most Sporting Team 2015
  33. 33. Put that in your | and Splunk it!  Started with ECS in June 2016  Consultant II by November 2016  ITSI Specialist since 2017  Enterprise Security Implementation gained in 2018  Splunk Trainer since January 2018
  34. 34. The Padawan to Master  2 x Interviews with Heads of Splunk Training (US)  Competence and Personality tested  1 x Train the Trainer session  Show the logistic tooling & processes  Present a module to the Trainer
  35. 35. A Day in the Life…  Access to the training center to confirm numbers and attendees.  All pre-class actions are automated:  Links to user watermarked documents sent to individuals & full list to trainer 2 days before the course.  Lab credentials created and sent to instructor.  WebEx session automatically configured ready to be joined.  Question etiquette  Best to save questions, unless pertinent to continuing, until at least after the 2nd lab. Generally a lot of time is spent before and after first lab answering access issues.  Where possible ask the question to all participants as it will help in the learning and is not as easy for the trainer to miss.
  36. 36.  Labs should be done in order as some activities rely on previous labs being completed.  However don’t rush! The labs do not need to be completed in the allotted time, only before the entire course finishes.  If in an office location, ensure that access over HTTPS and SSH is available out of the network.  If not then this can be worked around, but will take some time for the instructor to implement.  Test your access before the class (if possible)
  37. 37. Enablement  This position allows me to:  Offer more internal training along side the standard path.  Provide Splunk training to ECS clients.  Keep in touch with any changes coming in.
  38. 38. Tips  Don’t eat yellow snow  Install, Install, Install  Until it is muscle memory  Where possible script  Helps develop logic and defensive coding practices.  Always Best Practice in Labs & Exams!  Implementing Best Practice even if not specifically asked can gain additional points/kudos.
  39. 39. Questions?
  40. 40. © 2018 SPLUNK INC. Development & Release Cycles Using SDLC for Controlled Splunk-based Success
  41. 41. © 2018 SPLUNK INC. Software Development Life Cycle
  42. 42. © 2018 SPLUNK INC. Continuous Development, Deployment & Integration “DevOps is a culture, and not a collection of technology or role”
  43. 43. © 2018 SPLUNK INC. Splunk is “Agile” by Default Search Processing Language (SPL) Empowerment to Users Web User Interface Creation & Sharing of Knowledge Objects
  44. 44. © 2018 SPLUNK INC. Where DIY Could Harm the Business Security • Tuning of Rules / Alerts • Disabling of Saved Searches IT Monitoring • Manually Setting Thresholds • Forgetting Tactical Changes Business Analytics • Changes to Business Logic • Requirements Not Fully Understood
  45. 45. © 2018 SPLUNK INC. ▶ Local Work in Progress (WIP) ▶ Environment Specific Branches? • Local > Dev > Test > Release ▶ Tags ▶ Integration into Workflow Version Control Systems (VCS)
  46. 46. © 2018 SPLUNK INC. Configuration Management ● Manual vs. Automatic ● Agent & Agentless ● Actions, Roles, Groups ● Complemented by VCS
  47. 47. © 2018 SPLUNK INC. Best of Both: Splunk Route-to-Live New Requirement Create Feature in DevEnv Commit to Dev Branch Validate / Collect Features Merge into Test Branch Testing Process in TestEnv Change / Governance Merge into Prod Branch Release into ProdEnv
  48. 48. © 2018 SPLUNK INC. Resources ● A Visual Guide to Version Control ● Version Control using Git and GitLab ● Deploying Splunk Securely with Ansible Config Management ● Configuration Management 101: Writing Ansible Playbooks ● What is DevOps? ● DevOps is a culture, not a role!
  49. 49. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Thank You
  50. 50. © 2018 SPLUNK INC. Get Involved! ● Splunk User Group Edinburgh – https://usergroups.splunk.com/group/splunk-user-group-edinburgh.html – https://www.linkedin.com/groups/12013212 ● Splunk’s Slack Group – Register via http://splunk-usergroups.signup.team/ – Channel: #edinburgh ● Present & Share at the User Group? Connect: ‣ Harry McLaren | harry.mclaren@ecs.co.uk | @cyberharibu | harrymclaren.co.uk ‣ ECS | enquiries@ecs.co.uk | @ECS_IT | ecs.co.uk
  51. 51. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Thank You

×