John Villacres, Nationwide

1. Copyright © 2016 Splunk Inc.
Nationwide

2. 2
John Villacres
Network Automation and Tools
Nationwide

3. 3
Nationwide Overview
• We are on your side
• Network support for 40,000 employees
• 10,000+ agents and employees use VPN remote access daily
• Multiple data centers, lots of devices and applications

4. 4
My Background and Role
• Ohio State University – Go Bucks!
• First career as a pilot – Helicopters, Learjets, flight instructor
• Now Cisco routers and switches. Also FW, LB, Wifi, ACS, ISE, VPN
• Four year Splunk user
• Network Automation and Tools – Splunk, Gigamon, ExtraHop, Sniffers
• My favorite Splunk tag line: ‘I like big data and I cannot lie’
• Fun fact about me: If you attended a Woody Hayes speaking event in
1984, I probably drove him there.

5. 5
How We Got Started
• I demonstrated Splunk to architects and managers in 2012
• Free versions of Splunk in lab (10 x 500MB/day, shift stream every 2 hrs)
• Used on high priority outage calls until it caught on
• We now publish user dashboards for about 12 support teams
• Integrated Splunk with ExtraHop and Savvius

6. 6
Before Splunk
• Data is mostly there, but cumbersome to get, then difficult to interpret
in a timely manner
• Opaque hindsight after major replacements and upgrades to network
• Tools used - Syslogs, router, switch information and packet sniffers to
monitor network data
• Difficult to run/store long-term packet captures, network data

7. 7
New Process Needed
• No access to user’s connection history when they call in for assistance
• Trouble Tickets for connection issues - blocked IP addresses forwarded
to firewall team and wait for response (or start grep-ing)
• Unable to maintain visibility through packet captures/monitoring
sessions without losing wire data:
• 16TB full packet < 100GB ExtraHop select data -> Splunk!

8. 8
Finding Value With Splunk
• All the data you need to solve a user’s VPN issue in 10 seconds
• The data was already there in sylogs in folders, but data was not useable
• Are any one of the 200+ legacy firewalls blocking your app? Self service
via Firewall dashboard (12 seconds instead of a day)
• Agents can write more policies and generate revenue with enhanced
network support
• Support teams have reduced resolution time from days to minutes

9. 9
Timely, Useable Data
• Support teams can resolve firewall issues themselves in minutes
• ExtraHop trigger – 1500 byte packet -> 20 useful bytes of data -> Splunk
time chart -> determine in two seconds through JSESSIONIDs if they’re
balanced properly across JVM’s
• Months of data are now quickly available through custom dashboards
• Humans avoid activities that are difficult and have low resolution probability

10. Splunk Use Cases
Central Logging
& Visibility
Security
Dashboards
& Reporting
Threat Prevention
& Alerting
Metrics
& Searches

11. 11
How We Use Splunk
• Resolving VPN, firewall, and config change issues with custom
dashboards
• Splunk software serving support operations
• Data sources include wire data, router/switch/firewall syslogs, Cisco
ACS/ISE and others
• Who’s been messin’ with my router?
• Pursuing data the way I want; to make the decisions the way we
want (self service)

12. 12
From this… to this!
grep…arg!
Click…smile 

13. 13
From this… to this!
grep…arg!
Click…smile 

14. 14

15. 15
Growing With Splunk
• Other teams and departments have trained on, and cloned some our
dashboards for their own needs
• Because of Splunk’s efficiency, we have successfully transitioned
employees on to other tasks while continuing with productivity
• Currently reconstituting wire data capability

16. 16
Top Takeaways
• The data may already be there. But now you can access it faster, and
make more sense out of it once you have it.
• “Teach someone how to fish” with Splunk software
• MTTR times go down; sometimes way down

17. 17
Get a Good Night’s Sleep…
“What would normally take me hours or
even days to resolve takes me minutes to an
hour using Splunk.”
– John

18. Thank You