This document describes how Ashley Hartge implemented Splunk at their organization Verizon to improve network security monitoring and reporting. It outlines how they started with a single Splunk instance ingesting incident data to create dynamic ad-hoc reports. They then expanded Splunk to include distributed collection of SNMP data from network devices using forwarders. The document also provides examples of using Python scripts to poll thousands of devices and ingest the data into Splunk.

2.  Ashley Hartge (https://au.linkedin.com/in/ashleyhartge)
 Not a professional speaker / presenter
 19+ years full-time in the IT Industry
 7+ years in Managed Network Security space
 Senior Network Security Engineer
 Verizon – Global Commercial MSS Provider
 splunk> user since 2009 (personal & prof.)
 ActiveVerizon splunk> evangelist (APAC & US)

3.  Verizon - Global Commercial MSS Provider
 In-House SIEM product for many years
 Legacy selection of pre-canned reports
 Increasing need for dynamic ad/hoc reports
 Internal & Customer driven
 Difficult & rigid ETL process (release cycles)
 Any changes need development / release work
 Manually craft DB Query -> CSV -> Pivot table……
 Needed reporting at the speed of thought

4.  Using splunk for home / lab networks … why
not see what can be done for our reports …
 Old SOC workstation
 Download & install splunk eval
 Ingest a day’s incidents
 Build reports
 Blow management’s mind with visibility
 Splunk grabs a foothold ….The rest is history

5.  Secondary use-case after deploying splunk
 IT Ops | CMDB | Process Auditing |Tracking
 Basic SNMP polling using splunk>
 Distributed deployment with remote
collection
 Large scale scripted collection

6. Simplest deployment of single instance directly polling end device

7. https://splunkbase.splunk.com/app/1537/

9. Expand our single instance, with forwarders remotely collecting SNMP
data

10.  Setup Receiving port on Indexer
 Install Forwarder

11.  Configure Forwarding to your indexer
 Unzip & Install SNMP Modular input on
Universal Forwarder

12.  Configure /local/inputs.conf on Forwarder
with community string & OIDs for polling.

13.  Restart forwarder &
verify events being
received
 Create tags to give
recognisable names to
the hosts (this could
also be done within the
input or a lookup)

14.  Run a simple search on the sourcetype to see
if we are getting data (SNMP was configured
to poll the device every minute).
Notice our host tags we configured

15.  Quick & Dirty regex to showTEMP &
HUMIDITY from our sensors

16.  Use splunk field extractor

18.  The values are provided by the sensors but
need to be divided by 10 & rounded
 Solution = calculated field

21. Python script to poll 1000’s of devices, output to CSV, scp to splunk server,
index CSV into splunk

22. Compile a list of SNMP OIDs that provide the values we want from each platform,
then use python script to connect to management stations, snmp poll the devices &
return the result into CSV file – which is scp to the Indexer
'fortigate': {'hostname': '1.3.6.1.2.1.1.5.0', 'uptime': '1.3.6.1.2.1.1.3.0', 'model':
'SNMPv2-SMI::mib-2.47.1.1.1.1.10.1', 'version': 'SNMPv2-
SMI::enterprises.12356.101.4.1.1.0', 'serial': '1.3.6.1.4.1.12356.100.1.1.1.0', 'avsig':
'1.3.6.1.4.1.12356.101.4.2.1.0', 'idssig': '1.3.6.1.4.1.12356.101.4.2.2.0', 'go':True}
'paloalto': {'model': '1.3.6.1.4.1.25461.2.1.2.2.1.0', 'version':
'1.3.6.1.4.1.25461.2.1.2.1.1.0', 'uptime': '1.3.6.1.2.1.1.3.0', 'serial':
'.1.3.6.1.4.1.25461.2.1.2.1.3.0', 'avsig': '1.3.6.1.4.1.25461.2.1.2.1.8.0', 'idssig':
'1.3.6.1.4.1.25461.2.1.2.1.9.0', 'go':True}
'cisco-asa-ssm': {'hostname': 'SNMPv2-MIB::sysName.0', 'uptime': 'HOST-
RESOURCES-MIB::hrSystemUptime.0', 'model': 'SNMPv2-SMI::mib-
2.47.1.1.1.1.13.1', 'version': 'SNMPv2-SMI::enterprises.9.9.383.1.4.20.0', 'serial':
'SNMPv2-SMI::mib-2.47.1.1.1.1.11.1', 'license': 'SNMPv2-
SMI::enterprises.9.9.383.1.4.22.00', 'idssig': 'SNMPv2-
SMI::enterprises.9.9.383.1.4.21.0', 'go':True}

23.  Splunk easily ingests CSV data
 Because our output file does NOT contain CSV
header names on the first row – we pre-create a
sourcetype, naming our columns
 props.conf
 transforms.conf

24.  Create an input to monitor the csv

25.  Vendor signature release details (scrape)

29.  Dynamic Python Input Files
 | inputlookup asset | table ip, snmp_community, platform
| outputlookup fabricsource.csv
 Dynamically deploy to forwarders
 Deployment server to push input file to forwarder
 Scripted inputs on forwarders to replace cron
job(s) & csv-fu