phantom playbook practice for Automation

1. This presentation may contain forward-looking statements regarding future events, plans or the
expected financial performance of our company, including our expectations regarding our products,
technology, strategy, customers, markets, acquisitions and investments. These statements reflect
management’s current expectations, estimates and assumptions based on the information currently
available to us. These forward-looking statements are not guarantees of future performance and
involve significant risks, uncertainties and other factors that may cause our actual results,
performance or achievements to be materially different from results, performance or achievements
expressed or implied by the forward-looking statements contained in this presentation.
For additional information about factors that could cause actual results to differ materially from those
described in the forward-looking statements made in this presentation, please refer to our periodic
reports and other filings with the SEC, including the risk factors identified in our most recent quarterly
reports on Form 10-Q and annual reports on Form 10-K, copies of which may be obtained by visiting
the Splunk Investor Relations website at www.investors.splunk.com or the SEC's website at
www.sec.gov. The forward-looking statements made in this presentation are made as of the time and
date of this presentation. If reviewed after the initial presentation, even if made available by us, on our
website or otherwise, it may not contain current or accurate information. We disclaim any obligation to
update or revise any forward-looking statement based on new information, future events or otherwise,
except as required by applicable law.
In addition, any information about our roadmap outlines our general product direction and is subject to
change at any time without notice. It is for informational purposes only and shall not be incorporated
into any contract or other commitment. We undertake no obligation either to develop the features or
functionalities described, in beta or in preview (used interchangeably), or to include any such feature
or functionality in a future release.
Splunk, Splunk> and Turn Data Into Doing are trademarks and registered trademarks of Splunk Inc. in the United States and other countries.
All other brand names, product names or trademarks belong to their respective owners. © 2023 Splunk Inc. All rights reserved.
Forward-
Looking
Statements
03.16.23-15:28

2. © 2023 SPLUNK INC.
12 Angry Analysts
Tuning Splunk®
SOAR Events To Keep
Your Analysts Happy
(Or at Least Content)
SEC1406C
Gregory Rivas
Chief SOAR Dude | Accenture

3. © 2023 SPLUNK INC.
Gregory Rivas
Chief SOAR Dude | Accenture

4. © 2023 SPLUNK INC.
Otto
Noted Lazy Theropod | AutoSOARus rex

5. © 2023 SPLUNK INC.
Overview
1. An Overview of Phases
2. The Problem
3. An Answer
4. A Scalable Answer
5. A Mature Answer
6. Demo
What we’ll cover
They told you not to do an
overview slide

6. © 2023 SPLUNK INC.
Think About Playbooks in PHASES
Ingest
Initial container
standardization
Enhancement
Reach out to other
services to give
your container
some context
Triage
Make a decision
Response
Take action
Groups of playbooks should be categorized according to the functions they
play in one phase or another
tldr: Keep your playbooks in one phase or another and stay modular
Report
Tell people
Splunk SOAR
found or did a
thing
Closure
Archive it for later
maybe

7. © 2023 SPLUNK INC.
Ingest
Initial container
standardization
Enhancement
Reach out to other
services to give
your container
some context
Triage
Make a decision
Response
Take action
Report
Tell people
Splunk®
SOAR
found or did a
thing
Closure
Archive it for later
maybe
Your Sales Demo Probably
Focused Here

8. © 2023 SPLUNK INC.
Ingest
Initial container
standardization
Enhancement
Reach out to other
services to give
your container
some context
Triage
Make a decision
Response
Take action
Report
Tell people
Splunk®
SOAR
found or did a
thing
Closure
Archive it for later
maybe
Check out Conf22 for SEC1266B for information about the others
(link at the end)
Our Talk Today Will Focus
Here

9. © 2023 SPLUNK INC.
Triage
1. Focus Playbooks on deciding
I can’t read what’s on the phone

10. © 2023 SPLUNK INC.
Why is this even on a phone?
Triage
1. Focus Playbooks on deciding
2. Put your Humans Here
a. Human time is expensive, and slow. Keep prompts, and decisions in
‘Triage’, and to a lesser extent in ‘Response’

11. © 2023 SPLUNK INC.
Triage
1. Focus Playbooks on deciding
2. Put your Humans Here
a. Human time is expensive, and slow. Keep prompts, and decisions in
‘Triage’, and to a lesser extent in ‘Response’
3. Automation should still play a role in this phase
a. Use Splunk®
SOAR to do obvious steps
b. Use Humans to help with less obvious outcomes

12. © 2023 SPLUNK INC.
Triage
1. Focus Playbooks on deciding
2. Put your Humans Here
a. Human time is expensive, and slow. Keep prompts, and decisions in
‘Triage’, and to a lesser extent in ‘Response’
3. Automation should still play a role in this phase
a. Use Splunk®
SOAR to do obvious steps
b. Use Humans to help with less obvious outcomes
4. Consider relevant artifacts created during your enrichment phase

13. © 2023 SPLUNK INC.
The Problem

14. © 2023 SPLUNK INC.
The Problem
The analysts keep getting alerts and they hate it.
They somehow think it's our responsibility to close
some of them

15. © 2023 SPLUNK INC.
© 2023 SPLUNK INC.
Done When:
• Detection == ‘Interesting External Successful
Authentication’
• User == greg.rivas
• IP == Within South Korea
Why does the dog get
a hat but not me?
The Problem

16. © 2023 SPLUNK INC.
SOAR Problems
The SOC keeps bothering us, and we
don't like it. They are always over
caffeinated and generally smell bad

17. © 2023 SPLUNK INC.
Done When:
• SOC leaves us alone
• SOAR team doesn’t have to do work
SOAR Problems

18. © 2023 SPLUNK INC.
Solution 1
Let's do something obvious!

19. © 2023 SPLUNK INC.
• Use a Decision!
• It’s quick!
• It’s easy!
Oh no phone anymore?
Solution 1

20. © 2023 SPLUNK INC.
IF:
Event Detection == ‘Interesting External
Successful Authentication’
AND
Event User == greg.rivas
AND
Event Country IP == Republic of Korea
Solution 1

21. © 2023 SPLUNK INC.
hah this was too easy!
Solution 1

22. © 2023 SPLUNK INC.

23. © 2023 SPLUNK INC.

24. © 2023 SPLUNK INC.

25. © 2023 SPLUNK INC.

26. © 2023 SPLUNK INC.
Solution 1
Lets TRY!

27. © 2023 SPLUNK INC.
This isn't working
Solution 1

28. © 2023 SPLUNK INC.
The Problem with Solution 1
It's just not scaling well

29. © 2023 SPLUNK INC.
Like REALLY not
scaling well

30. © 2023 SPLUNK INC.
• Solution 1 SOLVES the Analyst’s Problem.
• Solution 1 does NOT solve ANY of our SOAR problems:
1) SOC leaves us alone
2) We don’t want to do work
1) Detection== ‘Interesting External Successful Authentication’
2) User == ‘Greg.Rivas’
3) IP == ‘Within South Korea’
Can we make this solution lazier??
The Problem with Solution 1

31. © 2023 SPLUNK INC.
• Lookups!
• We’ll make a table of Country and User pairs
• We can now call up this table and return a
boolean if a value matches!
• SOC can also now ADD ROWS to this lookup by
themselves when they get more requests
Solution 2

32. © 2023 SPLUNK INC.
• Solution 2 empowers the SOC to make entries
outside of the SOAR dev cycle
• It also allows SOC to very rapidly scale up or
down the number of exceptions
• Our playbook is now easier to read, and no
longer changes exception to exception
• Exceptions can now trivially scale many times
over
Solution 2

33. © 2023 SPLUNK INC.
Insert a custom function behind our
fateful decision
(This Custom Function will be available for download
at the end)

34. © 2023 SPLUNK INC.
Let’s Configure it
• Custom List Name
• Value to Search
• Column Header to search in
(This Custom Function will be available for download
at the end)
Those arrows aren't even

35. © 2023 SPLUNK INC.
Solution 2

36. © 2023 SPLUNK INC.
Now we can revisit our SINGLE decision

37. © 2023 SPLUNK INC.
Let’s Configure it!
• Expected src_country == Event src_country
• Expected user == Event user
If no match, we treat as normal container
If it DOES match…

38. © 2023 SPLUNK INC.

39. © 2023 SPLUNK INC.
We did it!! …. Right??

40. © 2023 SPLUNK INC.

41. © 2023 SPLUNK INC.

42. © 2023 SPLUNK INC.

43. © 2023 SPLUNK INC.

44. © 2023 SPLUNK INC.

45. © 2023 SPLUNK INC.
We are not doing 200 playbooks.

46. © 2023 SPLUNK INC.
The Problem with Solution 2
• SOC now wants us to make even more playbooks in direct violation of SOAR rule 2:
‘We don’t want to do work’
• SOC did NOT mention getting in trouble for mixing up ‘IP Addresses’ and ‘Users’, playing the blame
game for who edited the exception lookup we wrote them.
• While this is objectively funny to you, you still feel bad because you gave them enough Cat5 cable to
DOS themselves
Solution 2 was a victim of its own success. Originally solving both SOAR problems,
The solution worked so well the SOC wants it for other detections now.

47. © 2023 SPLUNK INC.
• Solution 2 SOLVES the Analyst’s Problem.
• Solution 2 does solve ONE of our SOAR problems:
1) SOC leaves us alone
2) We don’t want to do work
1) Detection== ‘Interesting External Successful Authentication’
2) User == ‘Greg.Rivas’
3) IP == ‘Within South Korea’
Can we make this solution lazier??
The Problem with Solution 2

48. © 2023 SPLUNK INC.
Solution 3
Let's Revisit Solution 2 and see if
we can make some improvements

49. © 2023 SPLUNK INC.
• Lookups!
Solution 3

50. © 2023 SPLUNK INC.
• Lookups!
Solution 3

51. © 2023 SPLUNK INC.
• Lookups!
• We’ll make a table of IP and User pairs
Solution 3

52. © 2023 SPLUNK INC.
• Lookups!
• We’ll make a table of IP and User pairs with relevant field names
Solution 3

53. © 2023 SPLUNK INC.
• Lookups!
• We’ll make a table of IP and User pairs with relevant field names
• We can now call up this table and return a boolean if a value matches!
Solution 3

54. © 2023 SPLUNK INC.
• Lookups!
• We’ll make a table of IP and User pairs with relevant field names
• We can now call up this table and return a boolean if a value matches!
• Playbook now allows field values to be RegEx compliant, allowing for wild
card matches or other patterns
Solution 3

55. © 2023 SPLUNK INC.
• Lookups!
• We’ll make a table of IP and User pairs with relevant field names
• We can now call up this table and return a boolean if a value matches!
• Playbook now allows field values to be RegEx compliant, allowing for wild
card matches or other patterns
• SOC can also now ADD ROWS to this lookup by themselves when they
get another request
Solution 3

56. © 2023 SPLUNK INC.
• Lookups!
• We’ll make a table of IP and User pairs with relevant field names
• We can now call up this table and return a boolean if a value matches!
• Playbook now allows field values to be RegEx compliant, allowing for wild
card matches or other patterns
• SOC can also now ADD ROWS to this lookup by themselves when they
get another request
• New meta fields allow for a JIRA ticket number to be included, line match
statistics, and some resolution language
Solution 3

57. © 2023 SPLUNK INC.
• Lookups!
• We’ll make a table of IP and User pairs with relevant field names
• We can now call up this table and return a boolean if a value matches!
• Playbook now allows field values to be RegEx compliant, allowing for wild
card matches or other patterns
• SOC can also now ADD ROWS to this lookup by themselves when they
get another request
• New meta fields allow for a JIRA ticket number to be included, line match
statistics, and some resolution language
• List now includes timestamp of first and last rule match
Solution 3

58. © 2023 SPLUNK INC.
• Lookups!
• We’ll make a table of IP and User pairs with relevant field names
• We can now call up this table and return a boolean if a value matches!
• Playbook now allows field values to be RegEx compliant, allowing for wild
card matches or other patterns
• SOC can also now ADD ROWS to this lookup by themselves when they
get another request
• New meta fields allow for a JIRA ticket number to be included, line match
statistics, and some resolution language
• List now includes timestamp of first and last rule match
• Playbook echoes a copy of the matching rule and line in closure notes and
as a note in the container in case rule changes
Solution 3

59. © 2023 SPLUNK INC.

60. © 2023 SPLUNK INC.

61. © 2023 SPLUNK INC.
This is literally the only thing the
analysts care about…
So uncultured!

62. © 2023 SPLUNK INC.
Solution 3

63. © 2023 SPLUNK INC.
• Solution 3 SOLVES the Analyst’s Problem.
• Solution 3 SOLVES the SOAR problems:
1) SOC leaves us alone
2) We dont want to do work
1) Detection== ‘Interesting External Successful
Authentication’
2) User == ‘Greg.Rivas’
3) IP == ‘Within South Korea’
Solution 3

64. © 2023 SPLUNK INC.
SOC now:
• Has only one lookup table to worry about
Solution 3

65. © 2023 SPLUNK INC.
SOC now:
• Has only one lookup table to worry about
• Has a table that can scale to hundreds of unique rules
Solution 3

66. © 2023 SPLUNK INC.
SOC now:
• Has only one lookup table to worry about
• Has a table that can scale to hundreds of unique rules
• Can write an autoclosure lookup rule for ANY detection that exists or could exist
Solution 3

67. © 2023 SPLUNK INC.
My favorite one!
SOC now:
• Has only one lookup table to worry about
• Has a table that can scale to hundreds of unique rules
• Can write an autoclosure lookup rule for ANY detection that exists or could exist
• Can write their own rules without bothering us SOAR folk
Solution 3

68. © 2023 SPLUNK INC.
SOC now:
• Has only one lookup table to worry about
• Has a table that can scale to hundreds of unique rules
• Can write an autoclosure lookup rule for ANY detection that exists or could exist
• Can write their own rules without bothering us SOAR folk
• Are now required to have a JIRA ticket history for the rule they write, which is used in the closure
notes
Solution 3

69. © 2023 SPLUNK INC.
SOC now:
• Has only one lookup table to worry about
• Has a table that can scale to hundreds of unique rules
• Can write an autoclosure lookup rule for ANY detection that exists or could exist
• Can write their own rules without bothering us SOAR folk
• Are now required to have a JIRA ticket history for the rule they write, which is used in the closure
notes
• Can auto close tickets using text contained in the lookup, not a generic closure note
Solution 3

70. © 2023 SPLUNK INC.
SOC now:
• Has only one lookup table to worry about
• Has a table that can scale to hundreds of unique rules
• Can write an autoclosure lookup rule for ANY detection that exists or could exist
• Can write their own rules without bothering us SOAR folk
• Are now required to have a JIRA ticket history for the rule they write, which is used in the closure
notes
• Can auto close tickets using text contained in the lookup, not a generic closure note
• Gets a note auto-created when a rule matches, containing the matching line from the lookup
Solution 3

71. © 2023 SPLUNK INC.
But SOAR gets some wins
from this too!
Solution 3

72. © 2023 SPLUNK INC.
SOAR Now:
• Does not have to worry about modifying playbooks for every exception from SOC
Solution 3

73. © 2023 SPLUNK INC.
SOAR Now:
• Does not have to worry about modifying playbooks for every exception from SOC
• Has an audit trail in JIRA of such modifications or additions to the table
Solution 3

74. © 2023 SPLUNK INC.
SOAR Now:
• Does not have to worry about modifying playbooks for every exception from SOC
• Has an audit trail in JIRA of such modifications or additions to the table
• Knows exactly what rule matched and closed the alert at the time of auto-closure
Solution 3

75. © 2023 SPLUNK INC.
SOAR Now:
• Does not have to worry about modifying playbooks for every exception from SOC
• Has an audit trail in JIRA of such modifications or additions to the table
• Knows exactly what rule matched and closed the alert at the time of auto-closure
Can you summarize this
better?
Solution 3

76. © 2023 SPLUNK INC.
Thank You
Using Splunk®
SOAR, we were able to
empower SOC to move at speed, outside
the confines that limit standard DevOps

77. © 2023 SPLUNK INC.
WAY too Salesy

78. © 2023 SPLUNK INC.
Thank You
We don’t have to listen to the SOC
complain at SOAR about false positives
anymore!

79. © 2023 SPLUNK INC.
Demo Video

80. © 2023 SPLUNK INC.
Git Link is listed below
Greg Rivas
https://beacons.ai/not_greg

81. © 2023 SPLUNK INC.
Thank You