SFBA Usergroup meeting November 2, 2022: Route, filter, and mask data at ingest time with Ingest Actions

1. During the course of this presentation, we may make forward-looking statements
regarding future events or the expected performance of the company. We caution
you that such statements reflect our current expectations and estimates sampled
on factors currently known to us and that actual events or results could differ
materially. For important factors that may cause actual results to differ from those
contained in our forward-looking statements, please review our filings with the
SEC.
The forward-looking statements made in this presentation are being made as of
the time and date of its live presentation. If reviewed after its live
presentation, this presentation may not contain current or accurate information.
We do not assume any obligation to update any forward-looking statements
we may make. In addition, any information about our roadmap outlines our
general product direction and is subject to change at any time without notice. It is
for informational purposes only and shall not be incorporated into any contract or
other commitment. Splunk undertakes no obligation either to develop the features
or functionality described or to include any such feature or functionality in a future
release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk
Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk
Inc. in the United States and other countries. All other brand names, product
Forward-
Looking
Statements
© 2022 SPLUNK INC.

2. © 2022 SPLUNK INC.
Introducing Ingest
Actions: Filter, Mask,
Route, Repeat
San Francisco Bay Area Splunk User Group
Nov 2, 2022
Divya Vijayan
Software Engineer | Splunk Inc.
Samat Jain
Principal Software Engineer | Splunk Inc.

3. © 2022 SPLUNK INC.
Software Engineer | Splunk Inc.
Divya Vijayan
Principal Software Engineer | Splunk Inc.
Samat Jain

4. © 2022 SPLUNK INC.
Thanks to…
Senior Product Manager |
Splunk Inc.
Felix Jiang
Cloud Solutions Architect |
Splunk Inc.
Russell Uman
Product Management
Director | Splunk Inc.
Izzy Park

5. © 2022 SPLUNK INC.
$
Value
of
Data
Age of Data
Potential Splunk Use Cases
Common Splunk Use Cases
Real and Near-Real Time
Ad Hoc
Data Lake and Archive
Forensics, Summaries, and Data Retention Compliance
<1 sec <1 min
<10 sec 1 week 1 year
1 month
1 hour 1 day 10 year
Data Value Changes With Age

6. © 2022 SPLUNK INC.
Data
Optimization
Brings a
Value-Based
Approach to
Data Strategy
TIER A
TIER B
TIER C
Higher Value Low Volume
Low Value High Volume
Use Cases
Monitoring, Investigation, RCA,
Premium Solutions
Value / Volume Ratio
High Value, Med-High Volume
Use Cases
Troubleshooting, Forensic
Investigation, Forensic Analysis
Value / Volume Ratio
Low Value, Med-High Volume
Use Cases
Compliance, Future Proofing
Value / Volume Ratio
Low Value, High Volume
B
U
S
I
N
E
S
S
C
R
I
T
I
C
A
L
L
O
W
S
I
G
N
A
L
L
O
W
V
A
L
U
E

7. © 2022 SPLUNK INC.
Data Tiering Flows - GDI
HF / IDX
IDXC DDAS Flex Index S3: RFS
S3: DDSS
(Frozen)
DDAA
Syslog
Frozen
Hadoop
Splunk
INGEST
INDEX
ARCHIVE

8. © 2022 SPLUNK INC.
What Can Admins Accomplish before
Ingest Actions?
Edit props.conf:
[source::/var/log/messages]
TRANSFORMS-null= setnull
Edit transforms.conf:
[setnull]
REGEX = DEBUG
DEST_KEY = queue
FORMAT = nullQueue
Filtering and masking data
involves:
• Memorizing syntax
• Handwriting stanzas
• Expensive iteration
• Editing of many conf files
• Manual deployment Source:
https://www.memesmonkey.com/topic/confused+dog

9. © 2022 SPLUNK INC.
A new user interface and backend enhancements to enable admins to easily author
and deploy rules on existing Splunk Enterprise-derived infrastructure.
This means you can now:
• Filter: discard unwanted events
– Remove noisy events, DEBUG logs, etc
• Mask: change the contents of events
– Mask PII, IP addresses, usernames
• Route: Events can be routed to any combination
of original Splunk index, different Splunk index,
clone, or sent to Amazon S3
• Use the UI to preview and validate rules / logic
– Does my regex work?
– How did one rule interact
with others?
How Do Ingest Actions Achieve This?

10. © 2022 SPLUNK INC.
Save Time, Save $
Less iteration time between authoring and deployment in prod
Filtering and routing events do not count against the ingest license
meter
Why?
• We heard you loud and clear
• In the long term, we still want to help you operate and derive value on your most
mission-critical data

11. © 2022 SPLUNK INC.
IA enables masking with PCRE regex compatibility
For audit & compliance contexts, store unmasked data on S3 for compliance, but
mask and de-identify for everyday search and reporting
Address Compliance Related Use Cases
with Ingest Actions

12. © 2022 SPLUNK INC.
Platform and Licensing Support
Deployment Customer-Managed
(Splunk®
Enterprise)
Splunk-Managed
(Splunk®
Cloud)
Licensing - Ingest
- vCPU
*No new SKU required for IA
- Ingest
- SVC
*No new SKU required for IA
Stack - N/A - Upgraded (“Victoria”) Stacks
- Classic Stacks (excluding GCP and
FedRAMP in 8.2.2203)
Platform Tier - Forwarding Tier: Deployment Server
→ Heavyweight Forwarder via app
distribution to all clients
- Indexing Tier: Cluster Manager →
Indexers via cluster bundle push
- Indexing Tier: Rules deployed via
Splunk Cloud Platform internal
mechanisms
UI location - Forwarding Tier: Deployment Server
- Indexing Tier: Cluster Manager
- Indexing Tier: Search Head

13. © 2022 SPLUNK INC.
Demo!

14. © 2022 SPLUNK INC.
Demo Architecture
Splunk cloud
Search Head
Self-Managed
Forwarding Tier

15. © 2022 SPLUNK INC.
What Do Rulesets Look Like?

16. © 2022 SPLUNK INC.
Where is IA configuration written?
DS
$SPLUNK_HOME/etc/
deployment-apps/
splunk_ingest_actions
Standalone
(incl. HWF)
$SPLUNK_HOME/etc/
apps/
splunk_ingest_actions
SH, CM
$SPLUNK_HOME/etc/
manager-apps/
splunk_ingest_actions

17. © 2022 SPLUNK INC.
Changes to props.conf and
transforms.conf
props.conf
o RULESET-*
Works the same as TRANSFORMS-* class,
but will run transforms on cooked data
o RULESET_DESC-*
Description of ruleset
transforms.conf
o STOP_PROCESSING_IF
Used for certain types of rules to
conditionally stop processing subsequent
rules
o Basically, routing rules will have events “exit
early”
STOP_PROCESSING_IF = <evaluator expression>
* An evaluator expression that the regexreplacement processor uses to determine
whether or not further processing is to occur for this event.
* If you set STOP_PROCESSING_IF, and the regexreplacement processor evaluates the
expression that you supply to be true, then the processor stops further
processing of this event.
* When you set STOP_PROCESSING_IF, like INGEST_EVAL, this setting overrides
all of the other index-time settings (such as REGEX, DEST_KEY, etc) except
for INGEST_EVAL. STOP_PROCESSING_IF executes after INGEST_EVAL.

18. © 2022 SPLUNK INC.
Changes to outputs.conf
[rfs:s3]
path = s3://data-actions-ingest/data-actions-service-acct/
remote.s3.endpoint = https://s3.us-west-2.amazonaws.com
remote.s3.access_key = key
remote.s3.secret_key = secret
Note:
● If on Standalone (incl. HWF), use the UI!
● If on IDXC, use the UI on the CM or SH!
● If on DS, no UI yet (9.1), must configure HWF
directly
● Much of SmartStore’s configuration from
indexes.conf works in this stanza

19. © 2022 SPLUNK INC.
Where do rulesets execute?
Ingest Action Rulesets are executed after existing transforms, e.g. TAs
Universal
Forwarder
Indexer
Heavy Forwarder Indexer
Universal
Forwarder
Unparsed Data
Unparsed Data Parsed Data
● Parsing
● Merging
● Typing
○ TRANSFORMS
○ RULESET
● Ruleset
○ RULESET
● Parsing
● Merging
● Typing
○ TRANSFORMS
○ RULESET
Parse Boundary

20. © 2022 SPLUNK INC.
Samat: Updated Masa

21. © 2022 SPLUNK INC.
What’s on S3?

22. © 2022 SPLUNK INC.
File Format is Valid JSON, “HEC JSON”

23. © 2022 SPLUNK INC.
Configuring Metrics
# transforms.conf
[_ruleset:global_settings]
metrics.disabled = false
metrics.report_interval = 30s
metrics.rule_filter = *<your rule name>*
Turned off by default

24. © 2022 SPLUNK INC.
What metrics are logged
Metrics - group=transforms, name=typing, rule="_rule:ruleset_splunkd_ui_access:mask:m7yeuix8",
sourcetype="splunkd", hit=216, cpu_seconds=0.1 in=38426, out.splunk=38000, out.drop=426
● rule is the name of the rule, and can be mapped to a rule in an Ingest Action
ruleset
● hit is the times the rule is hit in the report interval (number of events)
● cpu_seconds is the cpu time spent by the rule during the report interval
● in is the raw bytes the rule processes in the report interval
● out.x is the raw bytes the rule routes to each destination
● Ingest Action rules have a special prefix _rule
○ _rule:ruleset_splunkd_ui_access:mask:m7yeuix8
New

25. © 2022 SPLUNK INC.
Key new concepts for users who already have
experience with props/transforms, pipelines
● In Ingest Actions rulesets only, filtering and routing rules will stop further processing on
events, by default & by design
● A new pipeline “ruleset” was added
○ The pipeline will accept “cooked” data from HFs (by design)
○ This also means there’s another queue to monitor, previously only needed to mntiro
● A new output “rfs” pipeline was created for S3
○ Using output to S3 is not immune to issues such as backpressure
https://confluence.splunk.com/display/PROD/Data+Actions+Performance+Plan

26. © 2022 SPLUNK INC.
How Do I Get Started?
(1) Capability prereqs:
• list_ingest_ruleset: list existing rulesets
• edit_ingest_ruleset: create / edit rulesets
*Admins get these capabilities automatically
(2) Create your first ruleset!

27. © 2022 SPLUNK INC.
New Since
.conf’s 9.0
● “Set Index”: Route events to different
Splunk indexes
● Health Report for S3 destinations

28. Thank You
© 2022 SPLUNK INC.

29. © 2022 SPLUNK INC.
Additional Resources
1. What is a Victoria Cloud Stack?
2. Monitoring vCPU Consumption
3. Monitoring SVC Consumption
4. Using Ingest Actions to improve the data input process