Gary Hayslip is the Deputy Director and CISO of the City of San Diego. Before implementing Splunk, the city lacked visibility and coordination across its IT systems. Splunk has provided the city with end-to-end visibility across its 24 networks and 40,000 endpoints. It uses Splunk for security reporting, threat detection, application management, analytics, and dashboards. Splunk has helped the city detect threats faster, improve operations, and protect critical data. The city plans to continue expanding its use of Splunk across departments to improve security and operations.

1. Copyright © 2016 Splunk Inc.
Gary Hayslip
Deputy Director, CISO

2. 2
About the City of San Diego
• “America’s Finest City”
• U.S. 8th largest city
• 11,000+ employees
• $4 billion business
• 1.5 Million Citizens
• 24 Networks
• 40,000 endpoints
• 4 Millions attacks per week

3. 3
About Me
• Chief Information Security Officer
(CISSP, CISA, CRISC, CCSK)
• Background in DOD and US Navy
• Responsible for developing and executing
city-wide security strategy
• Creating “risk-aware” culture that protects
city and personal information resources
I am a World of Warcraft gamer
Favorite Splunk T-shirt – “Taking the SH out
of IT”

4. 4
Before Splunk: Chaos
• No visibility, no coordination, no control
• IT was outsourced to a city-owned non-profit
• No documentation; no strategic plan
• Lacked insight into networking, data analysis,
and who was doing what
• No security operation center; the security of the
networks was uncoordinated.
• Business impact
• Inefficiencies from too many networks and
disparate technologies duct-taped together
• Extreme vulnerability to cyber threats
• Voters were insisting on managed services
“Nothing was
documented and there
was no strategic plan. It
was like the Wild, Wild
West and the city was
just throwing money at
issues.”

5. 5
Choosing Splunk
• Selection criteria:
• Prior experience and knowledge
• Good ranking on Gartner
• Able to handle our massive data streams on one
platform
• Strong track record of interfacing smoothly with
other products (we have 26+)
• Success meant bringing together disparate
systems and data into one integrated,
managed platform “Visibility = Action”
• Splunk expands to meet evolving needs
“Based on
Splunk’s track
record, I wasn’t
interested in
anybody else.”

6. Splunk at City of San Diego
• 100 GB license for production
• Splunk Enterprise Security
• 33 Splunk environments
• 1 clustered indexers, 25 forwarders
• 24 networks, a billion packets/month
40,000 endpoints, petabytes of data
• 90% of the SOC owned by city, 10% by
service provider Atos
6
25 Universal Forwarders
1 Indexers
1 Search Heads + 1 Deployment Servers

7. Use of Splunk at City of San Diego
Data / Log Visibility
Application Management
Security Reporting
Threat Detection
Analytics / Dashboards

8. 8
End-to-End Visibility
• We now route all logs and manage multiple data
sources and apps on one platform
(Tenable, ActiveDirectory, networks audit server)
• We have visibility into our operations to function
effectively
• Achieved audit capability and intel into our
networks (Varonis tool, Netskope, Netwrix)
• With Splunk, we can create dashboards for any
function and see information in real time
“Splunk has the
ability to slice
and dice the
information to
give us the
visibility we
need.”

9. 9
Security Reporting
• Splunk supports our large data volume
(100GB/hour, 1 Billion packets/month - - >
adds up to a petabyte of data)
• Security dashboard opens visibility across the
network & shows management “what’s going on
today in cyber”
• I can translate the ton of paperwork on my desk
into building meaningful metrics
• Visibility lets us protect the perimeter to its “BYOD”
end points
“Security goes
beyond my
perimeter, which
is no longer just
the firewalls; it’s
the bank of mobile
phones my
employees are
walking around
with.”

10. 10
Threat Detection & Response
• We experience 4 million attacks/week (including
international countries & “hacktivists”)
• With Splunk Enterprise Security, we:
• Detect, investigate, scope and respond to threats
• Quarantine dangerous files in minutes
• When Mayor’s office hit with TeslaCrypt attack, we:
• Detected and had machine off network within 20 minutes
• Quickly protected critical folders (treasury, fire depts)
• Completely remediated and got back up and running in 3.5
hours (prior, would have taken several days)
“After a
TeslaCrypt
attack, we
identified the
effected
machine and
pulled it off
network in
minutes.”

11. 11
Application Management
• We manage 26 apps and plug-ins in Splunk
• Able to interface with components like:
• Nessus
• Tenable
• Varonis
• Cisco
• ActiveDirectory
• SCADA, ICS networks
• For our diverse needs, there was no other
choice
“Splunk was
the right size
Lego in the
box for the
puzzle we are
building.”

12. 12
Analytics, Dashboards, Reports
• Splunk gives us (multiple teams) a single pane and
point of access for our data
• Dashboards and reports to give full views across the
environment are in progress
• Able to view employee use of city data on cloud
solutions (mobile device, cloud storage)
• Future projects - see analytic reports on public
operations and functions (emergency teams, stop
lights, traffic)
• Reports drive more effective business management
decisions and practices
“I can show my
mayor and COO
how many attacks
we’ve blocked, how
many tickets we’re
handling, what’s
going on across the
network.”

13. 13
City of San Diego Use Cases
Daily Operations
Network Behavior Analytics
Continuous Monitoring
Data Governance

14. 14
Users Across City of San Diego
Enterprise networks
HVAC systems
Libraries
Police and fire departments, 911 dispatchers
Financial, medical (HIPAA), PCI data
GPS networks
Sanitation and utilities
Golf courses

15. 15
Splunk Words to the Wise….
• Don’t underestimate the amount of support
you need—add another 25%
• Plan for growth
• Start with a trial version and take the training
• Devote the effort to get the most out of the
solution
• When building out and adding other vendors:
• Always ask, “Can you Splunk it?”, “Do you have a
Splunk App?”
• If the answer is “no,” don’t buy it

16. 16
Splunking Ahead….
• We have a 5-year road map
• We plan to share our success by
introducing Splunk to other City
departments (Splunk Day)
• We want to expand to the cloud
• Our dream is to have Splunk’s versatility
touch every aspect of City operations, for
both ingress and egress

17. 17
Splunk Successes
• “Aha” moment – Clear usage detail of our city phones
• For years our vendor couldn’t give us this information
• Took a class on using Splunk, and got a clear report showing phone charges
and details by department - That alone paid for the class!
• We have immediate visibility into our data
• High level of detail, across multiple functions and departments
• We can create reports that improve productivity and help reduce
costs
• Threat protection systems have saved the city from critical data
breaches
17

18. Thank You