First Prague Splunk User Group kick-off session held on Jan 22 2024 in Prague (Splunk WeWork office). Main topic "Onboarding Best Practices".

1. © 2023 SPLUNK INC.
Onboarding Data
Best Practices
Prague Splunk User Group 22/1/2024
Tomáš Moser
Ingrid Němečková
Radek Filip

2. © 2023 SPLUNK INC.
16:00 - 16:20 (20 min) Checking-in and networking, refreshment
16:20 - 16:50 (30 min) Welcome and Introductions (round table)
16:50 - 17:45 (55 min) Presentation: Data Onboarding Best Practices (1/2)
17:45 - 18:00 (15 min) Break, refreshment
18:00- 18:45 (45 min) Presentation: Data Onboarding Best Practices (2/2)
18:45 - 19:00 (15 min) Wrap-up and Q&A
19:00 - 19:15 (15 min) Transfer to the pub “Kozlovna U Paukerta” - across the street
19:15 - ? Beer time
Prague Splunk User Group 22/1/2024
Program

3. © 2023 SPLUNK INC.
Splunk User Group Community
From Splunkers To Splunkers
✓ No sales
✓ No marketing
✓ It’s about You!
✓ Speak up!

4. © 2023 SPLUNK INC.
Who Are We?
Tomas
Moser
Sales Engineer,
Splunk CZ
Technical Support
Engineer, Splunk UK
Ingrid
Nemeckova
Splunk Consultant,
Alef Nula
Radek Filip
Sales Engineer,
Splunk CZ
Michal
Skorczewski

5. © 2023 SPLUNK INC.
Who Are You?
● Name
● Company
● Why Splunk?
● What are you interested in?
Round table :-)

6. © 2023 SPLUNK INC.
GDI is very large and too
complex topic. To cover all
the details and intricacies
we could talk …
… because of that we will
give you both high-level
guidance and best
practices you can apply
immediately and easily in
practice

7. © 2023 SPLUNK INC.
Why
Onboarding?
Optional subtitle
Click to add text

8. © 2023 SPLUNK INC.
Data Onboarding - Why it matters?
● Most important activity - “Splunk is all about data”
● Most undervalued, neglected or ignored prerequisite for any Splunk success
● Takes most of your project time (up to 90%)
● Affects quality of any outcome (“garbage in - garbage out”)
● Affects performance
● Affects cost
Foundation of any data processing engine
Getting Data In
Data Ingestion
Data Onboarding
GDI

9. © 2023 SPLUNK INC.
Splunk - Tiered Architecture
Collection tier is a foundation of the pyramid

10. © 2023 SPLUNK INC.
ONBOARDING
IS ALWAYS
HARD WORK

11. © 2023 SPLUNK INC.
Step by Step
High Level Process
Click to add text

12. © 2023 SPLUNK INC.
Step 1 - What’s the Business Case?
● GDI may be hard and it always takes time - Always!
● Use your time and other resources effectively - Onboard only data you really
need
● Get it right from the start - “Rework is expensive”
● Use case (problem) drives what data (information) you need.
Business is a sponsor
Example: Data Leak

13. © 2023 SPLUNK INC.
Step 2 - Identify System
● Any Data is provided by a Source system
● Identify technical systems that has the data to solve our business problem
● Research takes time
IT supports business
Example: DLP system, Email proxy, Web Proxy, DNS server (proxy), OS logs, EDR logs …

14. © 2023 SPLUNK INC.
Step 3 - Identify System Component
● Today’s systems are complex
● It’s not always is clear where exactly the data
● In more complex systems perhaps one or multiple components can source the data
● Research takes time
Identify integration needs
Example: Agent, Management console, existing collector, existing log storage (eg. S3)

15. © 2023 SPLUNK INC.
Step 4 - What data source(s) I need?
● Systems may share multiple different types of data (data sources, logs)
● Identify only relevant to our business case
● Non-existing Splunk technology add-on (TA) does not mean there is not a
important data source we need!
○ Often TAs don’t cover every Data source available from the Component
● Research takes time!
Identity data sources
Example:
Cisco ESA: Authentication, Textmail, HTTP, Consolidated event, Bounce log, Delivery logs,
Antispam logs, …
Cisoc ESA manual documents 40 different log sources!

16. © 2023 SPLUNK INC.
Step 5 - Do I Get All Information I Need?
● Does the data source contain expected information (anything missing?)
○ eg. “user” field is missing
● Is information in the right format?
○ e.g user is defined as an ID (e.g. “1234”) instead of a login name (e.g. “jdoe”,
“john.doe@help.com”)
● Do I have access to the system to modify logging configuration? Will external
team help?
● Research takes time!
Is information in the log sufficient?
Example: In Cisco WSA manual - up to 32 custom time fields can be added to its W3C
type Access log - there are more formats available (Squid, W3C)!

17. © 2023 SPLUNK INC.
Step 6 - How Is The Data Shared?
● Source systems can have different data sharing capabilities (per data source?)
● Type of sharing directly determines (limits) collection options
● Collection method (that we pick) impacts collection tier architecture
○ push - syslog stream (text), snmp trap stream (binary), HTTP stream
○ pull - REST call, SQL query, custom API call
○ read file on the disk (not that easy)
○ scripted output
● Any requirements (policy)?
○ eg. encryption (TLS?)
How to get data out?
Example: Cisco WSA logs:
Files on the disk:
Q. How do you guarantee files are not read multiple times or missed?

18. © 2023 SPLUNK INC.
Step 7 - Chose Collector & method
● We know how to get the data OUT of System the way we may prefer or require
● We need to determine how to get data TO Splunk
● Native Splunk solutions (direct connection to Splunk) + helpers
○ UF, HF, HEC, SC4S, SC4SNMP, Splunk Otel, Splunk Otel for K8S
○ Splunk Stream
○ Splunk Cloud Data Manager
● Helper tools:
○ SSH/SCP, FTP, etc.
● 3rd party solutions - (usually) no direct connection to Splunk (combinations of tools)
○ SyslogNG, rsyslog, Net-SNMP
○ Logstash
How to get data to Splunk?
Example:

19. © 2023 SPLUNK INC.
Step 8 - Does Collector Talk To Splunk?
● Collector may not be able (or allowed) to send data directly to Splunk
○ Incompatible protocols (e.g. Syslog to Splunk Cloud)
○ Not following best practices (e.g. terminate syslog on Splunk)
● Processing “trains” of multiple components might be necessary
● Many options - choose what fit your needs
Native communication to Splunk?
Example:
● Syslog:
[syslog server + UF] -> Splunk
[syslog server] -> (via HEC) -> Splunk
[SC4S] -> (via HEC) -> Splunk
● HTTP stream:
Custom HTTP stream -> AWS API Gateway -> (via HEC) -> Splunk
● SNMP traps:
[Net-SNMP + UF] -> Splunk
[SNMP TA + HF] -> Splunk
● SC4SNMP (K8s) -> Otel -> (via HEC) -> Splunk

20. © 2023 SPLUNK INC.
Step 9 - Do You Need To Manipulate Events?
● Any need to transform “raw event” before it gets ingested to Splunk?
○ e.g. compliance reasons, license optimisation, performance optimisation
● Transformation
○ mask data
○ strip
○ filter events
○ reformat events (KV, JSON, syslog, XML, unstructured text, structured - CSV|TSV, …)
■ standardisation, resource usage optimisation
● Enrichment
○ e.g. add new context: IP-DNS lookup
● Routing
○ eg. filter data to Splunk but send all data to a cheaper S3 storage
● License Optimisation
○ Splunk uses ingestion type licensing (daily calculation)
○ Not all data is equally important
○ Saved license portion may be filled with more important data
Modify data before it gets ingested to Splunk

21. © 2023 SPLUNK INC.
Step 10 - Where To Apply Transformation?
● Best practice - “as close to the source as possible” - usually easiest, most efficient
There are multiple options
Example:
1. Source system (system, application, …)
2. Syslog server
3. UF (index-time extractions)
4. HF | Splunk Edge Processor
5. 3rd party streaming processor
6. Indexer or Splunk Cloud
Custom event :-)

22. © 2023 SPLUNK INC.
Step 10 - Any Technology add-on exists?
● Technology add-on (TA) helps with parsing (getting fields out of events)
○ index-time, search-time
● Check Splunkbase (apps.splunk.com) if any TA exists
● Use the TA if exists - do NOT reinvent the wheel!
○ may people already invested a lot of their time!
● However, world is not perfect - Review first!
○ Last updated, cadence of updates, number of downloads, support level, answers.splunk.com, …
○ Unpack .spl or .tgz files and check config files manually
○ Check parsing - index-time/search-time parsing rules, …
● Not all TAs have the same level of quality
○ sometimes rework pays off
● A lot of research that takes time!
There are multiple options

23. © 2023 SPLUNK INC.
Summary
Customer:
How much time does it take to onboard “this” data source?
Splunker:
Hard to say. It depends. Could be hours, days, weeks. I don’t know every
data source in the log universe. I would say …

24. © 2023 SPLUNK INC.
Magic 8
Optimise indexing performance
Click to add text

25. © 2023 SPLUNK INC.
Splunk Event
Data onboarding affects performance
Before indexing event Splunk needs to know
● what the event looks like
● meta information (fields):
_time, host, source, sourcetype

26. © 2023 SPLUNK INC.
Performance vs. Flexibility Trade-Off
● All pre-indexing pipelines are
expensive at default settings
● Flexibility at a cost
● If you’re looking for performance,
minimize generality via props.conf
Higher flexibility at a cost of lower performance
Flexibility Trade-Off
Grow performance = lower resource usage

27. © 2023 SPLUNK INC.
“Magic 8”
Settings to maximise index-time performance
Set per sourcetype in props.conf
Set A - Parsing phase
1. SHOULD_LINEMERGE = false (always false)
2. LINE_BREAKER = regular expression for event breaks)
3. TIME_PREFIX = regex of the text that leads up to the
timestamp
4. MAX_TIMESTAMP_LOOKAHEAD = how many characters for the
timestamp
5. TIME_FORMAT = strptime format of the timestamp
6. TRUNCATE = 999999 (always a high number, default 10K)
Set B - Input phase
7. EVENT_BREAKER_ENABLE = true*
8. EVENT_BREAKER = regular expression for event breaks*

28. © 2023 SPLUNK INC.
Magic 6 - Example
Real example
props.conf for Auth0
[auth0]
LINE_BREAKER = ([rn]*){"log_id
SHOULD_LINEMERGE = 0
TIME_PREFIX = "date":"
MAX_TIMESTAMP_LOOKAHEAD = 24
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3Q%Z
TRUNCATE = 999999
Time Variables:
https://docs.splunk.com/Documentation/Splunk/9.1.2/SearchReference/Commontimeformatvariables
Sourcetype
Regex; ([rn]+)
Regex
Time variables
Best practices
- Every Timestamp (TS) contains Timezone (TZ) - ideally UTC
- Standardize on one TS format (ISO 8601): 2022-04-13T14:00:15.000Z

29. © 2023 SPLUNK INC.
Safe 40+% Of Your HW Resource
Testing indexing performance
.conf talk 2020: PLA1486 - Understanding Splunk Performance and Making Hardware
(Physical/Virtual) Choices

30. © 2023 SPLUNK INC.
Where to Apply?
Data onboarding affects performance
Splunk phases
Input -> Parsing -> Indexing ->
Searching
Index-time rules: only on nodes
supporting “parsing phase”:
- Indexer
- Heavy Forwarder
- Exception: Universal Forwarder
(indexed-extractions)
Input Parsing
Indexing
Docs: Configuration parameters and the data pipelines

31. © 2023 SPLUNK INC.
Dry Run (Data
Preview)
Testing Indexing Process
Click to add text

32. © 2023 SPLUNK INC.
Test Indexing On Sample Events
● After indexing NOTHING can be changed!
● Use sandbox
○ safe environment to test your props.conf/transforms.conf configs
● Many free or commercial options
○ VMWare Workstation/Fusion, Virtualbox, Parallels, …
● Your environment
○ you can use any time or break it!
● Use “test index” (eg. main, test, etc.)
● Use Splunk UI (Data Preview)
○ simulate index-time onboarding process - live modifications in real-time
Verify index-time parsing rules in a safe environment first

33. © 2023 SPLUNK INC.
Data Preview Workflow
1Can we
1.
2.
3. 4.
5.
6.

34. © 2023 SPLUNK INC.
Test your REGEX in UI
regex101.com
● PCRE 2.0
● Check # of steps
● …

35. © 2023 SPLUNK INC.
Save Or Export Your Configuration
Two options: app/local/props.conf | clipboard
Application <app> must exists before saving to <app>/local/props.conf!

36. © 2023 SPLUNK INC.
Private apps
Creating private configuration apps
Click to add text

37. © 2023 SPLUNK INC.
Package Your Custom Configuration
● Using only Splunk UI sooner or later your configuration will end up scattered all over
the Splunk installation - example:
○ etc/system/local/props.conf
○ etc/apps/search/local/props.conf
○ etc/apps/Splunk_TA_windows/props.conf
● Because of internal configuration file precedence new changes might not apply
● Best practice - manage your configuration manually - via private apps
● Private apps
○ separate custom changes from default settings (TA from Splunkbase) - independent
○ helps to keep configuration organized
○ In distributed environment - faster deployment, less resource intensive, well organisation, scales
○ require rigorous naming convention
● Private app naming conventions
○ it’s clear what is where
○ assures all changes apply when they should (add, modify, disable, etc.)
Separate custom changes from default settings

38. © 2023 SPLUNK INC.
App Naming Convention - Example
● Every customer may have their own preferences
● Too complex doesn’t mean better.
● If it suits its purpose it’s OK
Data onboarding affects performance
App naming template for GDI
<org>_<vendor>-<product|component|log>_
<config_file>[_<node>]
Examples:
● tom_windows-security_inputs
● tom_windows-security_inputs_hf
● tom_linux-auditd_inputs_slg
● tom_linux-auditd_props_sh
● tom_linux-auditd_props_idx
Node names
● SH - Search Head
● DEP - SHC Deployer
● IDX - Indexer
● MN - IDX cluster Manager Node
HF - Heavy Forwarder
● MC - Monitoring Console
● DEP - Deployment Server
● SLG - Syslog Server
Choose whatever
works for YOU!
This one proved
working :-)

39. © 2023 SPLUNK INC.
Search-time precedence (reverse-lexicographical order: t, s, S, B, A, 1)
1. tom_windows_security_props_sh/local/props.conf (highest precedence)
2. some_app/local/props.conf
3. Splunk_TA_windows/default/props.conf (lowest precedence)
4. 1tom_windows_security_props_sh/local/props.conf (lowest precedence)
Configuration Files Precedence
Data onboarding affects performance
Configuration files precedence
https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/Wheretofindtheconfigurationfiles
Index-time precedence (lexicographical order: 1, A, B, S, s, t)!
1. 1tom_windows_security_props_idx/local/props.conf (highest precedence)
2. Splunk_TA_windows/default/props.conf (lowest precedence)
3. some_app/local/props.conf
Lower
case first
Numbers
first

40. © 2023 SPLUNK INC.
Deployment Apps Example
245 deployment apps (3 environments) and growing …
● Git repository
● Automation (CI/CD)
○ Gitlab pipelines
○ Azure CLI
○ Bash
● Deployment time 6 min

41. © 2023 SPLUNK INC.
Btool
My config isn’t working …
Click to add text

42. © 2023 SPLUNK INC.
Btool - Holy Grail to Troubleshoot Configs
● Unsupported CLI command
● Helps to locate configuration option (which file)
● Shows merged configurations (taking configuration precedence into account)
● Doesn’t show loaded configuration - “only what it would look like (after
restart/reload)”
Understand configuration merge process
Troubleshooting Manual: Use btool to troubleshoot configurations
splunk btool <CONF_FILE> list [options]
splunk btool inputs list
splunk btool props list --debug
splunk btool indexes list
…

43. © 2023 SPLUNK INC.
Btool - Example props.conf”
22:26 $ /opt/splunk/bin/splunk btool props list auth0 --debug
/opt/splunk/etc/apps/TA-auth0/local/props.conf [auth0]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/apps/TA-auth0/local/props.conf EVAL-app = "Auth0"
/opt/splunk/etc/apps/TA-auth0/local/props.conf EVAL-authentication_service = 'data.strategy'
/opt/splunk/etc/apps/TA-auth0/local/props.conf EVAL-duration = coalesce('data.details.elapsedTime',
'data.details.prompts{}.elapsedTime')
/opt/splunk/etc/apps/TA-auth0/local/props.conf EVAL-src = src_host
/opt/splunk/etc/apps/TA-auth0/local/props.conf FIELDALIAS-aob_gen_auth0_alias_1 = data.hostname AS src_host
Props.conf merger for the sourcetype “auth0”

44. © 2023 SPLUNK INC.
Best practice
Always export your private “configuration” apps to a global context (share with the
system)
○ For any app that deals with both index-time and search-time parsing
■ e.g. props.conf, transforms.conf, eventtypes.conf, etc.
Example:
Splunk Configuration Propagation
myapp/metadata/local.meta:
[]
export = system

45. © 2023 SPLUNK INC.
Search-time
Parsing
Performance optimized indexing
Click to add text
https://splunk.lightning.force.com/lightning/r
/Deal_Support__c/a1Z5a00000PgQJMEA3
/view

46. © 2023 SPLUNK INC.
Verify Sequence of Search-time Operations
Knowledge Manager Manual
https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Searchtimeoperationssequence
Issue: Search-time parsing not
working
Best practice
Verify order of operators applied!
Watch out!
● EVAL statements in props.conf run in
parallel!
● If one references another it might not work!
!

47. © 2023 SPLUNK INC.
Data Balance
Event Distribution
Click to add text

48. © 2023 SPLUNK INC.
Verify Data Balance Across Indexers
● Splunk scales horizontally
● Adding more indexers increase the speed of searching linearly
● On one condition
● There is even event distribution across all indexers
Great 2019 .conf talk by Richard Morgan
FN1402 - Best practises for forwarder hierarchies - slides | video

49. © 2023 SPLUNK INC.
Why is Good Event Distribution Important?

50. © 2023 SPLUNK INC.
What’s Bed Event Distribution?

51. © 2023 SPLUNK INC.
Bad Event Distribution Affects Search

52. © 2023 SPLUNK INC.
Optimize UF Forwarding
Modify outputs.conf:
autoLBFrequency = <lowest value possible>
30 seconds is too long interval

53. © 2023 SPLUNK INC.
Optimize UF Forwarding (2)
Modify outputs.conf:
autoLBVolume = <lowest value possible>
try and tune

54. © 2023 SPLUNK INC.
Optimize UF Forwarding (3)
Modify inputs.conf:
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = <regex>

55. © 2023 SPLUNK INC.
Verify Real-time Ingestion and Distribution

56. © 2023 SPLUNK INC.

57. © 2023 SPLUNK INC.
Partners
Point of View
Onboarding related topics
Click to add text

58. © 2023 SPLUNK INC.
Most frequently met issues with
onboarding
Implementation partner has to deal with
Top 5:
● Mixed sourcetypes (under the same sourcetype)
● Linebreaking issues
○ multiline messages (Java stack trace)
● Timezone and timestamp recognition
○ timeformat mix for the same sourcetype
○ missing/not processed TZ information
● Event timestamp vs. Indexing time
○ offline users/workstations
● Log format definition change
○ App/OS version change
○ Add-on source/sourcetype naming conventions update

59. © 2023 SPLUNK INC.
Tools and utilities used
● Monitoring Console
How can I detect any problems with onboarding?

60. © 2023 SPLUNK INC.
Tools and utilities used (cont.)
● Broken Hosts
○ Am I still receiving data at all?
● Meta Woot by Discovered Intelligence
○ App focused on “Index time” data
○ Compliance reports of data latency and
indexing
○ Estimate license costs associated with your
data sources and hosts
○ Uses specific DM
Data streaming & latency

61. © 2023 SPLUNK INC.
Tools and utilities used (cont.)
● Custom developed tool
● CMDB cross-check host
verification
● RFC/IPV4/IPV6
compliance check
Host field validation

62. © 2023 SPLUNK INC.
Specific areas to pay attention to
● Data model
mapping
○ can easily add
another level of
complexity
○ even official
add-ons can
contain
incomplete or
incorrect DM
mapping
○ corner-cases
specific for each
company IT
environment

63. © 2023 SPLUNK INC.
TSE -
Frequent
issues with
GDI
Ingrid Němečková

64. © 2023 SPLUNK INC.
Frequent issues with data onboarding
1. What is time and why it’s important?
2. Why is Splunk blocking our data?
3. Don’t reinvent the wheel..
Technical Support Engineer view

65. © 2023 SPLUNK INC.
What is time and why it’s important?
Progression of events from the past to the present
into the future.
A timestamp allows companies to keep track of
events that take place at a particular moment.
Knowing what happened at an exact point gives
the user of the information control, and more
definitive direction on how to tackle situations of
the event that happened in that specific
time-period.
Technical Support Engineer view

66. © 2023 SPLUNK INC.
Timestamp
Technical Support Engineer view
INFO SavedSplunker - savedsearch_id= "nobody; search; nameOfAlert", search_type"*, search_streaming=0, user= "userName", appe"search",
savedsearch_name="nameOfAlert", priority default, status-success, digest_mode=0, durable_cursor=®, scheduled_time=1704714480, window_time=600,
dispatch_time=1704714497, run_time=1.836, result_count=2, alert_actions"",
sid="scheduler_RANDOMW5mLWNpZS10ZWx1bWV@cnktYWxIcnRpbmc_search__RMD57406f cbb5655a518_at_1704714480_58986_71C4975-0443-RANDOM",
suppressed=2, fired=0, skipped=2, action_time_ms=2, thread_id="AlertNotifierWorker-0", message="", workload_pool-"standard_perf"
host = sh-1-003.stackname.splunkcloud.com source = /opt/splunk/var/log/splunk/scheduler.log sourcetype = scheduler
INFO SavedSplunker - savedsearch_id= "nobody; search; nameOfAlert", search_type="", search_streaming=0, user="userName", app="search",
savedsearch_name="nameOfAlert", priority-default, status-success, digest_mode=0, durable_cursor=0, scheduled_time=1704714420, window_time=600,
dispatch_time=1704714437, run_time-1.715, result_count=2, alert_actions=", sid="scheduler_RANDOMcnktYWx1cnRpbmc_search__20_76385_RANDOM",
suppressed=1, fired=1, skipped=1, action_time_ms=3, thread_id="AlertNotifierWorker-0", message="", workload_pool="standard_perf"
host = sh-i-002.stackname.splunkcloud.com / source = /opt/splunk/var/log/splunk/scheduler.log sourcetype = scheduler
INFO SavedSplunker - savedsearch_id="nobody; nameOfAlert", search_type="", search_streaming=0, user="userName", app="search",
savedsearch_name="nameOfAlert", priority-default, status-success, digest_mode=0, durable_cursor=0, scheduled_time=1704714360, window_time=600,
dispatch_time=1704714377, run_time-1.753, result_count=2, alert_actions=*,
sid="scheduler_RANDOMNpZS10ZWxIbwV0cnktYWx1cnRpbmcsearch_RMD57406f60_85217_RANDOM", suppressed=1, fired=1, skipped=1, action_time_ms=5,
thread_id="AlertNotifierWorker-0", message="", workload_pool="standard_perf"
host = sh-i-001.stackname.splunkcloud.com | source = /opt/splunk/var/log/splunk/scheduler.log sourcetype = scheduler

67. © 2023 SPLUNK INC.
Timestamp
Technical Support Engineer view
01-08-2024 11:46:19.744 +0000 INFO SavedSplunker - savedsearch_id="nobody; nameOfAlert", search_type="", search_streaming=0, user="userName", app="search",
savedsearch_name="nameOfAlert", priority-default, status-success, digest_mode=0, durable_cursor=0, scheduled_time=1704714360, window_time=600,
dispatch_time=1704714377, run_time-1.753, result_count=2, alert_actions=*,
sid="scheduler_RANDOMNpZS10ZWxIbwV0cnktYWx1cnRpbmcsearch_RMD57406f60_85217_RANDOM", suppressed=1, fired=1, skipped=1, action_time_ms=5,
thread_id="AlertNotifierWorker-0", message="", workload_pool="standard_perf"
host = sh-i-001.stackname.splunkcloud.com | source = /opt/splunk/var/log/splunk/scheduler.log sourcetype = scheduler
01-08-2024 11:47:19.854 +0000 INFO SavedSplunker - savedsearch_id= "nobody; search; nameOfAlert", search_type="", search_streaming=0, user="userName",
app="search", savedsearch_name="nameOfAlert", priority-default, status-success, digest_mode=0, durable_cursor=0, scheduled_time=1704714420, window_time=600,
dispatch_time=1704714437, run_time-1.715, result_count=2, alert_actions=", sid="scheduler_RANDOMcnktYWx1cnRpbmc_search__20_76385_RANDOM",
suppressed=1, fired=1, skipped=1, action_time_ms=3, thread_id="AlertNotifierWorker-0", message="", workload_pool="standard_perf"
host = sh-i-002.stackname.splunkcloud.com / source = /opt/splunk/var/log/splunk/scheduler.log sourcetype = scheduler
01-08-2024 11:48:19.223 +0000 INFO SavedSplunker - savedsearch_id= "nobody; search; nameOfAlert", search_type"*, search_streaming=0, user= "userName",
appe"search", savedsearch_name="nameOfAlert", priority default, status-success, digest_mode=0, durable_cursor=®, scheduled_time=1704714480, window_time=600,
dispatch_time=1704714497, run_time=1.836, result_count=2, alert_actions"",
sid="scheduler_RANDOMW5mLWNpZS10ZWx1bWV@cnktYWxIcnRpbmc_search__RMD57406f cbb5655a518_at_1704714480_58986_71C4975-0443-RANDOM",
suppressed=2, fired=0, skipped=2, action_time_ms=2, thread_id="AlertNotifierWorker-0", message="", workload_pool-"standard_perf"
host = sh-1-003.stackname.splunkcloud.com source = /opt/splunk/var/log/splunk/scheduler.log sourcetype = scheduler

68. © 2023 SPLUNK INC.

69. © 2023 SPLUNK INC.

70. © 2023 SPLUNK INC.
Buckets

71. © 2023 SPLUNK INC.
Buckets
index="indexname_wmd"
| eval bktId=_bkt

72. © 2023 SPLUNK INC.
Buckets
|dbinspect index=indexname_wmd
|search bucketId=indexname_wmd~62~ED07611E-EBA4-4D73-BC2C-RANDOM
|convert ctime(startEpoch)
|convert ctime(endEpoch)
|table bucketId endEpoch startEpoch
bucketID endEpoch startEpoch
indexname_wmd~62-ED07611E-EBA4-4D7
3-BC2C-0272AD3DD6D9
02/17/2023 09:38:47 11/24/2022 13:05:46

73. © 2023 SPLUNK INC.
Timestamp
index=indexname source="udp:9514"
sourcetype=random_syslog
|eval indexed_time=strftime(_indextime,"%+")
|eval latency=(_indextime-_time)
|eval human=tostring(latency,"duration")
|table _raw _time indexed_time latency human

74. © 2023 SPLUNK INC.
Timestamp with HF
props.conf:
[default]
TRANSFORMS-time = time
transforms.conf:
[time]
INGEST_EVAL = hf_time=round(time(),0)
index=indexname source="udp:9514" sourcetype=random_syslog
|eval indexed_time=strftime(_indextime,"%+")
|eval latency=(_indextime-_time)
|eval human=tostring(latency,"duration")
|eval hf_time_forReal=_time+hf_time
|eval hf_time_forReal_human=strftime(hf_time_forReal,"%+")
|table _raw _time indexed_time latency human hf_time_forReal*

75. © 2023 SPLUNK INC.
Why is Splunk blocking our data?
SPLUNK IS NOT BLOCKING YOUR DATA
Technical Support Engineer view

76. © 2023 SPLUNK INC.
Don’t reinvent the wheel
Technical Support Engineer view

77. © 2023 SPLUNK INC.
Splunkbase

78. © 2023 SPLUNK INC.
Splunk Knowledge Based Articles
Written by Technical Support Engineers
https://splunk.my.site.com/customer/s/search

79. © 2023 SPLUNK INC.
Wrap-Up
Optional subtitle
Click to add text

80. © 2023 SPLUNK INC.
Resources
Best
● How indexing works (Splunk Community) - “Holy Grail!”
● Where do I configure (Splunk docs)
● Magic 8 (Aplura)
● Onboarding cheat sheet (Aplura)
● Splunk Search (Splunk.com, Splunk Product Documentation, Community questions
and answers, the Splunk Dev platform, and more)
Other
● Data Management (Splunk Lantern)

81. © 2023 SPLUNK INC.
Wrap-Up
● That’s it for today :-)
● Any Questions?
● Check your mailboxes and fill in post-event survey on 24/1
● Let’s stay in touch
○ Join Slack space “splunk-usergroups” and find channel: #prague-sug

82. © 2023 SPLUNK INC.
16:00 - 16:20 (20 min) Checking-in and networking, refreshment
16:20 - 16:50 (30 min) Introductions (round table)
16:50 - 17:45 (55 min) Presentation: Data Onboarding Best Practices (1/2)
17:45 - 18:00 (15 min) Break, refreshment
18:00- 18:45 (45 min) Presentation: Data Onboarding Best Practices (2/2)
18:45 - 19:00 (15 min) Wrap-up and Q&A
19:00 - 19:15 (15 min) Transfer to the pub “Kozlovna U Paukerta” - across the street
19:15 - ? Beer time
Before you leave … :-)
Program

83. © 2023 SPLUNK INC.
Thank You!