SFBA Splunk Usergroup meeting December 14, 2023: Writing the Fine (Splunk) Manual

1. © 2019 SPLUNK INC.
Welcome to the December SF Bay
Area Splunk User Group Meeting!
Glad you could join us!
The meeting will start at 11:10 am PST, so we’ll kick things off soon.
Notes:
● The meeting will start off with a welcome & announcements before our speakers take the floor.

2. © 2019 SPLUNK INC.
Welcome to the November SF Bay
Area Splunk User Group Meeting!
SFBA User Group Leaders/Facilitator:
Becky Burwell, Sr. Production Engineer, Yahoo
burwell@yahooinc.com
Manan Grover, Splunk

3. © 2019 SPLUNK INC.
Agenda
● Welcome!
● Announcements
● Writing the Fine (Splunk)
Manual
● Questions/Discussion

4. © 2019 SPLUNK INC.
Announcements
● Interested in giving a talk at a future meeting?
○ Becky burwell@yahooinc.com
Join the global Splunk Community on Slack @
splk.it/slack
○ Our user group channel is #ug_sfba

5. © 2019 SPLUNK INC.
2024 SFBA
User Group
Meeting
Schedule
Planned meeting dates for 2023:
● January, 2024: skipping
● February, 2024: in-person in San Jose
● March, 2024: virtual

6. © 2019 SPLUNK INC.
Writing the Fine (Splunk) Manual
Mark McCullough, Cyber Security Architect, SLAC

7. Writing The Fine (Splunk>®)
Manual

8. Ground Rules

9. Ask your
questions
• Curious?
• Don't wait for the landing

10. Why?
SA-5
The right
thing
So you can
take
vacation
Pitchfork
avoidance

11. SA-5
a. Obtain or develop administrator documentation for the system, system component, or system
service that describes:
1. Secure configuration, installation, and operation of the system, component, or service;
2. Effective use and maintenance of security and privacy functions and mechanisms; and
3. Known vulnerabilities regarding configuration and use of administrative or privileged functions;
b. Obtain or develop user documentation for the system, system component, or system service that
describes:
1. User-accessible security and privacy functions and mechanisms and how to effectively use those functions and
mechanisms;
2. Methods for user interaction, which enables individuals to use the system, component, or service in a more
secure manner and protect individual privacy; and
3. User responsibilities in maintaining the security of the system, component, or service and privacy of individuals;
c. Document attempts to obtain system, system component, or system service documentation when
such documentation is either unavailable or nonexistent and take [Assignment: organization-defined
actions] in response; and
d. Distribute documentation to [Assignment: organization-defined personnel or roles].

12. SA-5 in Plain English
Write the Fine Manual!

13. Do The Right Thing
Documenting is
good

14. Vacations are good
Interrupted vacation is bad

15. Pitchfork Avoidance
Your successor
may know where
you live

16. What to do

17. The Service Run Book
•Enough detail to
rebuild
•No git details
•No credentials
The
whole
Splunk
Infra

18. Overview
What is this service? What is
the value to the organization?

19. Usage
How do we login? (the URLs)
• SHC
• ES
• DS
• MC

20. Architecture
Assumptions Systems
Network
connectivity
IAM
Service
configuration

21. Assumptions
What are the key assumptions of the service?
• SHs are intended to be available to multiple teams for their logs
• Users are self-service, but only comfortable using the GUI for editing dashboads or
saved searches
• Downtime tolerance for a search head is no more than two hours
• Data ingestion downtime tolerance for forwarded logs is approximately 30 minutes
• Data ingestion downtime for pulled logs (e.g. modular inputs, scripted inputs) is four
hours
• Content in user private space (not app shared) is non-production and may be safely
deleted when a user leaves the organization

22. Systems
Hostname CNAME Role CPU RAM Storage Type Notes
sec-splunk-sh01 splunk SH 72 256G /: 116G
/boot: 2G
/opt: 95G
Dell Poweredge R640 Primary SH
Asset Tag:
PC12345
sec-splunk-test0
1
SH-t
est
12 24G /: 20G
/boot:
700M
VMware VM Cluster baz

23. Systems in AWS
Hostname CNAME Role Storage Type Notes
cc01023 splunk SH /: 116G
/boot: 2G
/opt: 95G
c6i.8xl SHC
cc01026 IDX /: 20G
/opt/splunk: 15T
i3en.6xl Storage is
ephemeral
cc01043 splunk-dev SH /: 20G
/opt/splunk: 150G
t3.m Dev SH,
low CPU

24. Network Connectivity
•Firewall rules?
•Key ports?
•https://www.aplura.com/cheats
heets/splunk_network_ports.ht
ml

25. IAM
Key roles in your shop
How to request access
Any shop specifics on granting access?

26. Service Configuration
Local custom apps
• Include pointers to their docs
Any Splunkbase apps?
• Include the ID number
Where's your git repos?

27. Maintenance
Training Support
Standard
Procedures
Known
Issues

28. Training
Where to get it

29. Support
Enough detail to file a new ticket

30. Standard Procedures
Anything site
local
Don't rewrite
docs.splunk.com

33. Known Issues
Need more than ten minutes
to solve? Document!

36. Alerts

37. Audience
Those who
receive the alert

38. Content of alert documentation
What is
it?
Why does
it matter?
What do
you do?
Validation

39. What is it?
Explain the alert

42. What to do?
Details required

44. Validation
Know it is fixed
correctly

47. Process Tips

48. Iterate Documentation
Add
notes
Improve
notes
Add
notes
Improve
notes

49. Docs First!
Write the alert
documentation
Get signoff -
include your SOC!
Build alerts

50. Everyone's an editor
No guardian on the
edits
Make sure
everyone who
receives the alert
can update TFM

51. Is this still current?
Check alerts
periodically
even if one
per month

52. Now you can Write The Fine Manual

53. © 2019 SPLUNK INC.
Thank You!

54. © 2019 SPLUNK INC.
Title and Content
Phasellus et nisi lacus, mauris ultricies arcu faucibus orci sit
Donec fermentum sollicitudin neque, nec viverra neque lacinia eu
Donec mattis tortor vitae egestas pulvinar
• Vivamus eu dignissim turpis
Nunc eu cursus est, at ullamcorper dui
Optional subtitle