SlideShare a Scribd company logo
1 of 54
Download to read offline
© 2019 SPLUNK INC.
Welcome to the December SF Bay
Area Splunk User Group Meeting!
Glad you could join us!
The meeting will start at 11:10 am PST, so we’ll kick things off soon.
Notes:
● The meeting will start off with a welcome & announcements before our speakers take the floor.
© 2019 SPLUNK INC.
Welcome to the November SF Bay
Area Splunk User Group Meeting!
SFBA User Group Leaders/Facilitator:
Becky Burwell, Sr. Production Engineer, Yahoo
burwell@yahooinc.com
Manan Grover, Splunk
© 2019 SPLUNK INC.
Agenda
● Welcome!
● Announcements
● Writing the Fine (Splunk)
Manual
● Questions/Discussion
© 2019 SPLUNK INC.
Announcements
● Interested in giving a talk at a future meeting?
○ Becky burwell@yahooinc.com
Join the global Splunk Community on Slack @
splk.it/slack
○ Our user group channel is #ug_sfba
© 2019 SPLUNK INC.
2024 SFBA
User Group
Meeting
Schedule
Planned meeting dates for 2023:
● January, 2024: skipping
● February, 2024: in-person in San Jose
● March, 2024: virtual
© 2019 SPLUNK INC.
Writing the Fine (Splunk) Manual
Mark McCullough, Cyber Security Architect, SLAC
Writing The Fine (Splunk>®)
Manual
Ground Rules
Ask your
questions
• Curious?
• Don't wait for the landing
Why?
SA-5
The right
thing
So you can
take
vacation
Pitchfork
avoidance
SA-5
a. Obtain or develop administrator documentation for the system, system component, or system
service that describes:
1. Secure configuration, installation, and operation of the system, component, or service;
2. Effective use and maintenance of security and privacy functions and mechanisms; and
3. Known vulnerabilities regarding configuration and use of administrative or privileged functions;
b. Obtain or develop user documentation for the system, system component, or system service that
describes:
1. User-accessible security and privacy functions and mechanisms and how to effectively use those functions and
mechanisms;
2. Methods for user interaction, which enables individuals to use the system, component, or service in a more
secure manner and protect individual privacy; and
3. User responsibilities in maintaining the security of the system, component, or service and privacy of individuals;
c. Document attempts to obtain system, system component, or system service documentation when
such documentation is either unavailable or nonexistent and take [Assignment: organization-defined
actions] in response; and
d. Distribute documentation to [Assignment: organization-defined personnel or roles].
SA-5 in Plain English
Write the Fine Manual!
Do The Right Thing
Documenting is
good
Vacations are good
Interrupted vacation is bad
Pitchfork Avoidance
Your successor
may know where
you live
What to do
The Service Run Book
•Enough detail to
rebuild
•No git details
•No credentials
The
whole
Splunk
Infra
Overview
What is this service? What is
the value to the organization?
Usage
How do we login? (the URLs)
• SHC
• ES
• DS
• MC
Architecture
Assumptions Systems
Network
connectivity
IAM
Service
configuration
Assumptions
What are the key assumptions of the service?
• SHs are intended to be available to multiple teams for their logs
• Users are self-service, but only comfortable using the GUI for editing dashboads or
saved searches
• Downtime tolerance for a search head is no more than two hours
• Data ingestion downtime tolerance for forwarded logs is approximately 30 minutes
• Data ingestion downtime for pulled logs (e.g. modular inputs, scripted inputs) is four
hours
• Content in user private space (not app shared) is non-production and may be safely
deleted when a user leaves the organization
Systems
Hostname CNAME Role CPU RAM Storage Type Notes
sec-splunk-sh01 splunk SH 72 256G /: 116G
/boot: 2G
/opt: 95G
Dell Poweredge R640 Primary SH
Asset Tag:
PC12345
sec-splunk-test0
1
SH-t
est
12 24G /: 20G
/boot:
700M
VMware VM Cluster baz
Systems in AWS
Hostname CNAME Role Storage Type Notes
cc01023 splunk SH /: 116G
/boot: 2G
/opt: 95G
c6i.8xl SHC
cc01026 IDX /: 20G
/opt/splunk: 15T
i3en.6xl Storage is
ephemeral
cc01043 splunk-dev SH /: 20G
/opt/splunk: 150G
t3.m Dev SH,
low CPU
Network Connectivity
•Firewall rules?
•Key ports?
•https://www.aplura.com/cheats
heets/splunk_network_ports.ht
ml
IAM
Key roles in your shop
How to request access
Any shop specifics on granting access?
Service Configuration
Local custom apps
• Include pointers to their docs
Any Splunkbase apps?
• Include the ID number
Where's your git repos?
Maintenance
Training Support
Standard
Procedures
Known
Issues
Training
Where to get it
Support
Enough detail to file a new ticket
Standard Procedures
Anything site
local
Don't rewrite
docs.splunk.com
Known Issues
Need more than ten minutes
to solve? Document!
Alerts
Audience
Those who
receive the alert
Content of alert documentation
What is
it?
Why does
it matter?
What do
you do?
Validation
What is it?
Explain the alert
What to do?
Details required
Validation
Know it is fixed
correctly
Process Tips
Iterate Documentation
Add
notes
Improve
notes
Add
notes
Improve
notes
Docs First!
Write the alert
documentation
Get signoff -
include your SOC!
Build alerts
Everyone's an editor
No guardian on the
edits
Make sure
everyone who
receives the alert
can update TFM
Is this still current?
Check alerts
periodically
even if one
per month
Now you can Write The Fine Manual
© 2019 SPLUNK INC.
Thank You!
© 2019 SPLUNK INC.
Title and Content
Phasellus et nisi lacus, mauris ultricies arcu faucibus orci sit
Donec fermentum sollicitudin neque, nec viverra neque lacinia eu
Donec mattis tortor vitae egestas pulvinar
• Vivamus eu dignissim turpis
Nunc eu cursus est, at ullamcorper dui
Optional subtitle

More Related Content

Similar to SFBA Splunk Usergroup meeting December 14, 2023

Using Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at SplunkUsing Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at Splunk
Docker, Inc.
 

Similar to SFBA Splunk Usergroup meeting December 14, 2023 (20)

Best Practices For Workflow
Best Practices For WorkflowBest Practices For Workflow
Best Practices For Workflow
 
Using Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at SplunkUsing Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at Splunk
 
Building Business Service Intelligence with ITSI
Building Business Service Intelligence with ITSIBuilding Business Service Intelligence with ITSI
Building Business Service Intelligence with ITSI
 
online blogging system
online blogging systemonline blogging system
online blogging system
 
Architecting a Large Software Project - Lessons Learned
Architecting a Large Software Project - Lessons LearnedArchitecting a Large Software Project - Lessons Learned
Architecting a Large Software Project - Lessons Learned
 
Distributed teams
Distributed teamsDistributed teams
Distributed teams
 
Distributed_teams
Distributed_teamsDistributed_teams
Distributed_teams
 
Oracle Management Cloud
Oracle Management Cloud Oracle Management Cloud
Oracle Management Cloud
 
Oracle Management Cloud
Oracle Management CloudOracle Management Cloud
Oracle Management Cloud
 
Azure + DataStax Enterprise (DSE) Powers Office365 Per User Store
Azure + DataStax Enterprise (DSE) Powers Office365 Per User StoreAzure + DataStax Enterprise (DSE) Powers Office365 Per User Store
Azure + DataStax Enterprise (DSE) Powers Office365 Per User Store
 
Splunk4Rookies - Attendee - May 2023.pdf
Splunk4Rookies - Attendee - May 2023.pdfSplunk4Rookies - Attendee - May 2023.pdf
Splunk4Rookies - Attendee - May 2023.pdf
 
What's New in Grizzly & Deploying OpenStack with Puppet
What's New in Grizzly & Deploying OpenStack with PuppetWhat's New in Grizzly & Deploying OpenStack with Puppet
What's New in Grizzly & Deploying OpenStack with Puppet
 
Geek Sync | Planning a SQL Server to Azure Migration in 2021 - Brent Ozar
Geek Sync | Planning a SQL Server to Azure Migration in 2021 - Brent OzarGeek Sync | Planning a SQL Server to Azure Migration in 2021 - Brent Ozar
Geek Sync | Planning a SQL Server to Azure Migration in 2021 - Brent Ozar
 
Automating secure server baselines with Chef
Automating secure server baselines with ChefAutomating secure server baselines with Chef
Automating secure server baselines with Chef
 
Facilitating Release Planning Event
Facilitating Release Planning EventFacilitating Release Planning Event
Facilitating Release Planning Event
 
OpenStack Glance Project Update
OpenStack Glance Project UpdateOpenStack Glance Project Update
OpenStack Glance Project Update
 
Getting Started with Splunk Enterprise
Getting Started with Splunk EnterpriseGetting Started with Splunk Enterprise
Getting Started with Splunk Enterprise
 
A Lap Around Developer Awesomeness in Splunk 6.3
A Lap Around Developer Awesomeness in Splunk 6.3A Lap Around Developer Awesomeness in Splunk 6.3
A Lap Around Developer Awesomeness in Splunk 6.3
 
Ebook9
Ebook9Ebook9
Ebook9
 
Sql interview question part 9
Sql interview question part 9Sql interview question part 9
Sql interview question part 9
 

More from Becky Burwell

More from Becky Burwell (12)

SFBA_SUG_2023-08-02.pdf
SFBA_SUG_2023-08-02.pdfSFBA_SUG_2023-08-02.pdf
SFBA_SUG_2023-08-02.pdf
 
SFBA Splunk Usergroup meeting May 3, 2023
SFBA Splunk Usergroup meeting May 3, 2023SFBA Splunk Usergroup meeting May 3, 2023
SFBA Splunk Usergroup meeting May 3, 2023
 
SFBA Splunk User Group Meeting February 2023
SFBA Splunk User Group Meeting February 2023SFBA Splunk User Group Meeting February 2023
SFBA Splunk User Group Meeting February 2023
 
SFBA Splunk Usergroup meeting December 2022
SFBA Splunk Usergroup meeting December 2022SFBA Splunk Usergroup meeting December 2022
SFBA Splunk Usergroup meeting December 2022
 
SFBA Usergroup meeting November 2, 2022
SFBA Usergroup meeting November 2, 2022SFBA Usergroup meeting November 2, 2022
SFBA Usergroup meeting November 2, 2022
 
SF Bay Area Splunk User Group Meeting October 5, 2022
SF Bay Area Splunk User Group Meeting October 5, 2022SF Bay Area Splunk User Group Meeting October 5, 2022
SF Bay Area Splunk User Group Meeting October 5, 2022
 
SFBA Splunk User Group Meeting August 10, 2022
SFBA Splunk User Group Meeting August 10, 2022SFBA Splunk User Group Meeting August 10, 2022
SFBA Splunk User Group Meeting August 10, 2022
 
SFBA Splunk Usergroup meeting July 13, 2022
SFBA Splunk Usergroup meeting July 13, 2022SFBA Splunk Usergroup meeting July 13, 2022
SFBA Splunk Usergroup meeting July 13, 2022
 
designing-resilient-cloud-native-splunk-arch-in-aws-austin-rose.pdf
designing-resilient-cloud-native-splunk-arch-in-aws-austin-rose.pdfdesigning-resilient-cloud-native-splunk-arch-in-aws-austin-rose.pdf
designing-resilient-cloud-native-splunk-arch-in-aws-austin-rose.pdf
 
Splunking configfiles 20211208_daniel_wilson
Splunking configfiles 20211208_daniel_wilsonSplunking configfiles 20211208_daniel_wilson
Splunking configfiles 20211208_daniel_wilson
 
Getting Started with Splunk Observability September 8, 2021
Getting Started with Splunk Observability September 8, 2021Getting Started with Splunk Observability September 8, 2021
Getting Started with Splunk Observability September 8, 2021
 
Advanced Outlier Detection and Noise Reduction with Splunk & MLTK August 11, ...
Advanced Outlier Detection and Noise Reduction with Splunk & MLTK August 11, ...Advanced Outlier Detection and Noise Reduction with Splunk & MLTK August 11, ...
Advanced Outlier Detection and Noise Reduction with Splunk & MLTK August 11, ...
 

Recently uploaded

Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
amitlee9823
 
Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...
Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...
Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...
amitlee9823
 
Mg Road Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Banga...
Mg Road Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Banga...Mg Road Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Banga...
Mg Road Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Banga...
amitlee9823
 
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
amitlee9823
 
Call Girls In Bellandur ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Bellandur ☎ 7737669865 🥵 Book Your One night StandCall Girls In Bellandur ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Bellandur ☎ 7737669865 🥵 Book Your One night Stand
amitlee9823
 
Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...
amitlee9823
 
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get CytotecAbortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Riyadh +966572737505 get cytotec
 
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
amitlee9823
 
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
amitlee9823
 
Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...
amitlee9823
 

Recently uploaded (20)

Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
 
Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...
Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...
Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...
 
Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 night
Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 nightCheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 night
Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 night
 
Invezz.com - Grow your wealth with trading signals
Invezz.com - Grow your wealth with trading signalsInvezz.com - Grow your wealth with trading signals
Invezz.com - Grow your wealth with trading signals
 
Mg Road Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Banga...
Mg Road Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Banga...Mg Road Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Banga...
Mg Road Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Banga...
 
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...
 
Call Girls In Bellandur ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Bellandur ☎ 7737669865 🥵 Book Your One night StandCall Girls In Bellandur ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Bellandur ☎ 7737669865 🥵 Book Your One night Stand
 
April 2024 - Crypto Market Report's Analysis
April 2024 - Crypto Market Report's AnalysisApril 2024 - Crypto Market Report's Analysis
April 2024 - Crypto Market Report's Analysis
 
Predicting Loan Approval: A Data Science Project
Predicting Loan Approval: A Data Science ProjectPredicting Loan Approval: A Data Science Project
Predicting Loan Approval: A Data Science Project
 
Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...
 
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get CytotecAbortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
 
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
 
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
 
Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...
 
Call me @ 9892124323 Cheap Rate Call Girls in Vashi with Real Photo 100% Secure
Call me @ 9892124323  Cheap Rate Call Girls in Vashi with Real Photo 100% SecureCall me @ 9892124323  Cheap Rate Call Girls in Vashi with Real Photo 100% Secure
Call me @ 9892124323 Cheap Rate Call Girls in Vashi with Real Photo 100% Secure
 
Mature dropshipping via API with DroFx.pptx
Mature dropshipping via API with DroFx.pptxMature dropshipping via API with DroFx.pptx
Mature dropshipping via API with DroFx.pptx
 
Generative AI on Enterprise Cloud with NiFi and Milvus
Generative AI on Enterprise Cloud with NiFi and MilvusGenerative AI on Enterprise Cloud with NiFi and Milvus
Generative AI on Enterprise Cloud with NiFi and Milvus
 
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
 
Sampling (random) method and Non random.ppt
Sampling (random) method and Non random.pptSampling (random) method and Non random.ppt
Sampling (random) method and Non random.ppt
 
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
 

SFBA Splunk Usergroup meeting December 14, 2023

  • 1. © 2019 SPLUNK INC. Welcome to the December SF Bay Area Splunk User Group Meeting! Glad you could join us! The meeting will start at 11:10 am PST, so we’ll kick things off soon. Notes: ● The meeting will start off with a welcome & announcements before our speakers take the floor.
  • 2. © 2019 SPLUNK INC. Welcome to the November SF Bay Area Splunk User Group Meeting! SFBA User Group Leaders/Facilitator: Becky Burwell, Sr. Production Engineer, Yahoo burwell@yahooinc.com Manan Grover, Splunk
  • 3. © 2019 SPLUNK INC. Agenda ● Welcome! ● Announcements ● Writing the Fine (Splunk) Manual ● Questions/Discussion
  • 4. © 2019 SPLUNK INC. Announcements ● Interested in giving a talk at a future meeting? ○ Becky burwell@yahooinc.com Join the global Splunk Community on Slack @ splk.it/slack ○ Our user group channel is #ug_sfba
  • 5. © 2019 SPLUNK INC. 2024 SFBA User Group Meeting Schedule Planned meeting dates for 2023: ● January, 2024: skipping ● February, 2024: in-person in San Jose ● March, 2024: virtual
  • 6. © 2019 SPLUNK INC. Writing the Fine (Splunk) Manual Mark McCullough, Cyber Security Architect, SLAC
  • 7. Writing The Fine (Splunk>®) Manual
  • 9. Ask your questions • Curious? • Don't wait for the landing
  • 10. Why? SA-5 The right thing So you can take vacation Pitchfork avoidance
  • 11. SA-5 a. Obtain or develop administrator documentation for the system, system component, or system service that describes: 1. Secure configuration, installation, and operation of the system, component, or service; 2. Effective use and maintenance of security and privacy functions and mechanisms; and 3. Known vulnerabilities regarding configuration and use of administrative or privileged functions; b. Obtain or develop user documentation for the system, system component, or system service that describes: 1. User-accessible security and privacy functions and mechanisms and how to effectively use those functions and mechanisms; 2. Methods for user interaction, which enables individuals to use the system, component, or service in a more secure manner and protect individual privacy; and 3. User responsibilities in maintaining the security of the system, component, or service and privacy of individuals; c. Document attempts to obtain system, system component, or system service documentation when such documentation is either unavailable or nonexistent and take [Assignment: organization-defined actions] in response; and d. Distribute documentation to [Assignment: organization-defined personnel or roles].
  • 12. SA-5 in Plain English Write the Fine Manual!
  • 13. Do The Right Thing Documenting is good
  • 17. The Service Run Book •Enough detail to rebuild •No git details •No credentials The whole Splunk Infra
  • 18. Overview What is this service? What is the value to the organization?
  • 19. Usage How do we login? (the URLs) • SHC • ES • DS • MC
  • 21. Assumptions What are the key assumptions of the service? • SHs are intended to be available to multiple teams for their logs • Users are self-service, but only comfortable using the GUI for editing dashboads or saved searches • Downtime tolerance for a search head is no more than two hours • Data ingestion downtime tolerance for forwarded logs is approximately 30 minutes • Data ingestion downtime for pulled logs (e.g. modular inputs, scripted inputs) is four hours • Content in user private space (not app shared) is non-production and may be safely deleted when a user leaves the organization
  • 22. Systems Hostname CNAME Role CPU RAM Storage Type Notes sec-splunk-sh01 splunk SH 72 256G /: 116G /boot: 2G /opt: 95G Dell Poweredge R640 Primary SH Asset Tag: PC12345 sec-splunk-test0 1 SH-t est 12 24G /: 20G /boot: 700M VMware VM Cluster baz
  • 23. Systems in AWS Hostname CNAME Role Storage Type Notes cc01023 splunk SH /: 116G /boot: 2G /opt: 95G c6i.8xl SHC cc01026 IDX /: 20G /opt/splunk: 15T i3en.6xl Storage is ephemeral cc01043 splunk-dev SH /: 20G /opt/splunk: 150G t3.m Dev SH, low CPU
  • 24. Network Connectivity •Firewall rules? •Key ports? •https://www.aplura.com/cheats heets/splunk_network_ports.ht ml
  • 25. IAM Key roles in your shop How to request access Any shop specifics on granting access?
  • 26. Service Configuration Local custom apps • Include pointers to their docs Any Splunkbase apps? • Include the ID number Where's your git repos?
  • 29. Support Enough detail to file a new ticket
  • 31.
  • 32.
  • 33. Known Issues Need more than ten minutes to solve? Document!
  • 34.
  • 35.
  • 38. Content of alert documentation What is it? Why does it matter? What do you do? Validation
  • 39. What is it? Explain the alert
  • 40.
  • 41.
  • 43.
  • 44. Validation Know it is fixed correctly
  • 45.
  • 46.
  • 49. Docs First! Write the alert documentation Get signoff - include your SOC! Build alerts
  • 50. Everyone's an editor No guardian on the edits Make sure everyone who receives the alert can update TFM
  • 51. Is this still current? Check alerts periodically even if one per month
  • 52. Now you can Write The Fine Manual
  • 53. © 2019 SPLUNK INC. Thank You!
  • 54. © 2019 SPLUNK INC. Title and Content Phasellus et nisi lacus, mauris ultricies arcu faucibus orci sit Donec fermentum sollicitudin neque, nec viverra neque lacinia eu Donec mattis tortor vitae egestas pulvinar • Vivamus eu dignissim turpis Nunc eu cursus est, at ullamcorper dui Optional subtitle