You and your colleagues are all doing great things with Splunk. But you seldom come together to share ideas, apps and best practices. This session will help you take Splunk to the next level by helping you establish a Splunk Center of Excellence (CoE) at your organization. The purpose of a COE is simple - to provide Splunk users an informal venue in which they can discuss ideas, diagnose challenges, share innovations and network with peers. This session will share the best practices you need to create and maintain a successful CoE practice.

1. Copyright © 2015 Splunk Inc.
Best Practices for a
Center of Excellence (COE)
SplunkLive DC 2016
Clint Locker, Sr. PS Manager
Midatlantic and Southeast

2. 2
About Me
Clint Locker – clocker@splunk.com
Sr. PS Manager for Midatlantic and Southeast Regions
Started with Splunk 4 years ago, delivered Professional
Services for 2 years (may have deployed your
environment, sorry in advance )
Managed over 400+ Splunk successful deployments
Based in Arlington VA, lived in the area for close to 15
years, wife and I have a 8 month old boy William
2

3. 3
Agenda
• Evolution of Splunk Deployments
• Components of a Splunk COE
• Communication Framework
• Training Recommendations
• COE Assessment

4. 4
Splunk Deployments Evolve Over TimeSplunk Deployments Evolve Over Time
4
Expansion
Download
Workgroup
Enterprise
Deployment
• Enterprise standard
• Large number of users
• Many different use cases
• Many different users
• MultipleSplunk
deployments
• More sites
• More geographies
• More data sources
More data volume
InitialUser
• Specific use case
• Specific users

5. 5
Turning Machine Data Into Operational Intelligence
Search
and
Investigate
Search and
Investigate
Proactive
Monitoring
and Alerting
Operational
Visibility
Real-time
Business
Insight
Proactive
Reactive

6. 6
What is COE?
A Center of Excellence refers to a team, a shared
facility or an entity that provides leadership,
evangelization, best practices, research, support
and/or training for a focus area.

7. 7
Splunk Center of Excellence
Goals
Provide Splunk technical oversight
Drive and communicate best practices
Facilitate data on-boarding, user on-
boarding, run book documentation
Provide expertise focused on enablement
Deliver support services to Splunk
consumers

8. 8
Key Success Factors
 Program Management Office
 Executive sponsorship
 Project planning, communication, and
process
 Success Criteria Clearly Defined
 Business requirements, use cases
 Reports, alerts, dashboards
 Create Deployment Plan
 Architecture
 Data/App on-boarding
 Resourcing, staffing, training plan
 Communications
 Regular cadence from technical to executiv
teams
 Quarterly Business Reviews
 Sustainment
 Establish Splunk Center of Excellence

9. 11
COE Sample Benefits
11
Challenge Solution Benefit
What is on your mind? How can the COE help? Measure, improve, let us know!
Lack of general knowledge and
internal best practices for Splunk
led to increased support calls.
30 minutes devoted to addressing
end user needs and education of
internal processes.
Reduction of Splunk Support
interactions by over 25% and faster
time to value.
A flurry of new dashboards led to
decreased performance of system.
Focused training sessions and
advanced techniques education.
Elimination of over 24 redundant
panels and better overall system
performance.
Rapid adoption of Splunk led to a
severe backlog to on-board desired
data targets.
Open discussion on process
improvement, standards and
requirements for new sources.
Increased speed and efficiency of
data onboarding process from 2
weeks to 48 hours on average.

10. 12
Components of a Splunk COE
Architecture &
Infrastructure
Operations
Supporting
Tools
Staffing
Data
On-Boarding
User
On-Boarding
Inform

11. 13
Architecture & Infrastructure
 Appropriate hardware sizing for indexing and search load
 Physical cores only, hyperthreading does not count
 SSD provides significant performance advantage
 High performance storage
 IOPS are critical
 In distributed environments, dedicated IOPS are not cumulative
 Measure with Bonnie++, SplunkIT, IOPS App
 Current Splunk version
 Clear upgrade path and process
 Proactive capacity planning
 Understand unit of scale for hardware
 Map growth curve for data and users

12. Best Practice – Service Levels
Characteristics Staging Class C Class B Class A
Infrastructure Shared Shared Federated Dedicated
Use Case Value Low
(testing)
Low
(discovery)
Medium
(visible, supporting
tools)
High
(revenue/service
impacting)
Retention Short
(2-4 weeks)
Short
(1-3 months)
Medium
(3-6 months)
Long
(6-12 months)
Security/Access Basic Basic Moderate Strong
Chargeback None Simple Mixed Complex
SLA None Lowest Moderate High
Geography Single Single Multiple Single/Multiple
HA/DR None None Partially Resilient Fully Resilient

13. 15
Operations & Supporting Tools
 Configuration Management
 Common: Puppet/Chef
 Splunk: Deployment Server
 Change Management
 Version control
 Service ticketing
 Deployment
 System Health Monitoring
 System capacity and
performance
 Splunk tools: Unix App,
Windows App, VMware App,
NetApp App
 Splunk Health Monitoring
 Splunk on Splunk App
 Fire Brigade App
 Sanity App

14. 16
Staffing
1
A successful and scalable deployment of
Splunk relies on the orchestration of key
roles and responsibilities, primarily
centered around:
 Architecture
 Administration
 User adoption (Power User)
 Application development

15. 17
Splunk Architect Role
1
Responsibility
• Accountable for the design of the Splunk architecture
• Fully understands concepts and best practices for sizing, scaling, and deploying Splunk across your
organization so that performance meets current and future needs
• Works with power users to determine which data sources should be indexed to meet each
department’s needs
Recommendation
• 1 to 2 Splunk Architects
• Part time for < 500GB; 1 Full time for 500GB to 1TB; 2 for >1TB
• Note: if deploying Splunk Cloud, assume only 25% of above resources are required

16. 18
Splunk Admin Role
1
Responsibility
• Maintains the Splunk software and it’s infrastructure for optimal performance
• Adds data sources to the Splunk platform according to Power User needs
• Assist power users with the development of advanced dashboards, alerting and reporting
Recommendation
• 1 to 2 Splunk Admins depending on size of implementation
• Part time for < 500GB; 1 Full time for 500GB to 1TB; 2+ for >1TB
• Note: if deploying Splunk Cloud, assume only 25% of above resources are required

17. 19
Splunk Power User Role
1
Responsibility
• Works with their group to identify opportunities where Splunk can provide value
• Collaborates with the Splunk admin(s) to add new data sources to address their requirements
• Provides basic support for new and existing reports and dashboards to their group from
investigative keyword searches to creating rich reports and visualizations to becoming a Splunk
search ninja!
Recommendation
• 1 part-time power user per user group

18. 20
Splunk Developer Role
2
Responsibility
• Splunk developers are only required if applications are developed on top of the Splunk platform
• Create rich, interactive dashboards and forms, and package Splunk knowledge objects for
distribution across your organization

19. 21
Basic Communication Framework
2
Architect
Admin
Works with power users to determine
which data sources should be indexed
to meet each department’s needs
Scales the Splunk architecture to meet
business demand
Power Users Department Users
Adds data sources to the Splunk
platform according to business needs
Assist power users with the
development of advanced dashboards,
alerting and reporting
Maintains the Splunk SW and it’s
infrastructure for optimal performance
1 Power user per department
Provides basic support for new and existing reports
and dashboards
Works with their group to identify opportunities
where Splunk can provide value

20. 22
Splunk Classes
2
Splunk Roles
Using
Splunk
Splunk
Administration
Searching
and
Reporting
Creating
Knowledge
Objects
Advanced
Searching &
Reporting
Developing
Apps with
Splunk
Developing
with Splunk
SDKs
Architect Required Required Optional Optional Optional Optional Optional
Admin Required Required Optional Optional
Power User Required Required Required Optional
Developer Required Optional Required Required Optional Required Optional
for Splunk on-premises

21. 23
Splunk Classes
2
Splunk Roles
Using
Splunk
Splunk
Administration
Searching
and
Reporting
Creating
Knowledge
Objects
Advanced
Searching &
Reporting
Developing
Apps with
Splunk
Developing
with Splunk
SDKs
Architect Required Optional Optional Optional Optional Optional
Admin Required Optional Optional
Power User Required Required Required Optional
Developer Required Required Required Optional Required Optional
for Splunk Cloud

22. 24
Data On-Boarding
 Define on-boarding process
for new data sources / apps
 Repeatable, documented
process
 Provide customer interview
forum or survey
 Integrate with service
workflow
New Data Source Request
 Provide a data sample
 Describe the data’s structure
 timestamp | timezone  single-/multi-line
 sourcetype  interesting fields
 Describe initial uses for the data
 searches | alerts | reports | dashboards
 How to collect the data?
 UF | syslog | API
 How long to retain the data?
 Who should have access?
 Apply Common information Model
 Are there TA’s available?
 Validate

23. 25
User On-Boarding
 Orientation for new users
 Develop training program
 Splunk instructor-led online/onsite courses
 Get started with Splunk videos
 Advancement for experienced users
 Continuing education
 Splunk workshops
 Office Hours
 Where to get help?
 How to contact <Company> Splunk team
 Internal/external email lists, chat group
 Splunk Answers

24. 26
Inform
 Track Value and ROI
 Document Use Cases
 Expert Showcases
 Internal knowledge sharing
 Develop power users
 Tip of the Week/Month
 Contests
 Search competition
 Use case drive
 Regular Newsletter
Splunk Accelerates Troubleshooting
An expressive troubleshooting dashboard shines a
bright light on any part of the infrastructure exceeding
reasonable performance thresholds.
Less Screwing Up, More Drilling Down
Application and site performance is often dependent on
system performance. Splunk’s monitoring probes
through layers to collect high resolution CPU statistics.

25. 27
Example Meetings
 User Group
 Splunk in Action
 Ask Splunk
 Open Office Hours
 Splunk Administrators Group
 Architecture and Administration topics
 Splunk Developers Group
 App, UI and API topics
 Splunk Lunch & Learn
 Education topics
 Splunk Support Session
 Support case review
 Quarterly Business Reviews
 Vendor Management Office

26. 28
COE Success – Be Visible & Valuable
• Create a Knowledge Management Portal for Splunk resource
– Publish company specific policies & procedures
– Publish Naming Standards
– Publish Data Onboarding guidelines
– Link to Splunk.com resources
• Aggregate Training Needs from Line of Businesses
• Conduct regular meetings for Line of Business Users
– General User Group for Best Practice Sharing
– Specialized meetings for Administrators, Developers, etc.
– Lunch & Learn Sessions for informal training
2

27. 29
Use Case Documentation (Examples)
Splunk Monitors Proactively for Threat Patterns
Alongside historical trending and analysis for monthly and incident
reports, Splunk alerts the Fraud Detection team to similar patterns
emerging across systems or locales in real time. Email alerts also
promote standardization in capturing and exposing critical
information.
Splunk Secures Access for Independent Forensics
Role-based controls provide shielded views into data. Incident investigations no
longer require highly paid security professionals for pattern tracking and reporting.
Empowering customer service or individual financial institutions to research
independently and securely reduced incident response time from hours to minutes.
Splunk Detects $5M in Attempted Fraud
Correlation by transaction, time and geography identifies all elements in
the infrastructure exposed to nefarious activity originating internally or
externally. In one incident, Splunk’s transaction tracing and geoip
mapping abilities identified 15 banks located in the same region
exhibiting a similar fraud pattern. The activity was tracked to a single
shared data processing vendor which had been compromised.

28. 30
Partner with Splunk Teams
 Account Team
 Account Manager
 Sales Engineer (SE)
 Specialists (Security, IT SI, etc.)
 Support Team
 Designated Support Engineer (DSE)
 Customer Success Manager (CSM)
 Education
 Standard curriculum
(online/onsite)
 Boot camps
 Customized curriculum
 Professional Services Team
 Project-based (e.g. Deployment,
Health Checks, Upgrades, App
Development)
 Technical Advisory Services (TAS)
 Center of Excellence Advisory
Services
 Customer Advisory & Success
Teams (CAST)
 Dedicated Splunk Advisory
Engineers
 Faster time to value and adoption

29. 31
 Splunk User Groups
 Community driven
 Bootstrapped by Splunk
 Locally every 2-3 months
 SplunkLive!
 Worldwide customer events
 Technical workshops for beginner and advanced users
 Local events held yearly
 Annual Worldwide Users Conference
 September 26-29, 2016 in Orlando FL, Disney World
 3+ days, 130+ sessions, 4000+ enthusiasts
 Splunk Answers Desk, SplunkBase Lab, Chalk Talks, Search Party, Hackathon
Get Social with Splunk Events
3
www.splunk.com > Events

30. SEPT 26-29, 2016
WALT DISNEY WORLD, ORLANDO
SWAN AND DOLPHIN RESORTS
• 5000+ IT & Business Professionals
• 3 days of technical content
• 165+ sessions
• 80+ Customer Speakers
• 35+ Apps in Splunk Apps Showcase
• 75+ Technology Partners
• 1:1 networking: Ask The Experts and Security
Experts, Birds of a Feather and Chalk Talks
• NEW hands-on labs!
• Expanded show floor, Dashboards Control
Room & Clinic, and MORE!
The 7th Annual Splunk Worldwide Users’ Conference
PLUS Splunk University
• Three days: Sept 24-26, 2016
• Get Splunk Certified for FREE!
• Get CPE credits for CISSP, CAP, SSCP
• Save thousands on Splunk education!

31. Copyright © 2015 Splunk Inc.
Next Steps
COE Assessment

32. 34
Splunk Architect Training
Splunk
Architect(s)
Using
Splunk
Splunk
Administration
Searching and
Reporting
Creating
Knowledge
Objects
Advanced
Searching &
Reporting
Developing
Apps with
Splunk
Developing
with Splunk
SDKs
• # name
• # name
= Splunk training completed= Required = Optional = Training required but not completed = Optional training not completed
Instructions: List the
names and color
code the cells as
green, red or leave
blank, based on
legend below

33. 35
Splunk Admin Training
Splunk
Administrator(s)
Using
Splunk
Splunk
Administration
Searching and
Reporting
Creating
Knowledge
Objects
Advanced
Searching &
Reporting
Developing
Apps with
Splunk
Developing
with Splunk
SDKs
• #name
• #name
• #name
= Splunk training completed= Required = Optional = Training required but not completed = Optional training not completed
Instructions: List the
names and color
code the cells as
green, red or leave
blank, based on
legend below

34. 36
Splunk Power User Training
Splunk
Power User(s)
Using
Splunk
Splunk
Administration
Searching and
Reporting
Creating
Knowledge
Objects
Advanced
Searching &
Reporting
Developing
Apps with
Splunk
Developing
with Splunk
SDKs
Server Team
• # name
Network Team
• # name
Middleware Team
• # name
DBA Team
• # name
App Support Team
• # name
App Development
• # name
Security Team
• # name
= Splunk training completed= Required = Optional = Training required but not completed = Optional training not completed
Instructions: List the
names and color
code the cells as
green, red or leave
blank, based on
legend below

35. 37
Splunk Developer Training
Splunk
Developer(s)
Using
Splunk
Splunk
Administration
Searching and
Reporting
Creating
Knowledge
Objects
Advanced
Searching &
Reporting
Developing
Apps with
Splunk
Developing
with Splunk
SDKs
• # name
• # name
• # name
• # name
= Splunk training completed= Required = Optional = Training required but not completed = Optional training not completed
Instructions: This slide is optional and only applies IF there are plans to develop applications on top of Splunk.
List the names and color code the cells as green, red or leave blank, based on legend below

36. 38
Your Splunk COE
Splunk
Architect
Doug
Splunk
Administrator
Kevin
Splunk
Developer
Suzie
UX Admins
Power User
Bob
Network
Power User
Mark
DBA
Power User
Dave
ecommerce
Power User
Tony
example
= Fully Trained = Partially Trained = Not assigned
Splunk
Developer
Todd
Instructions: add / remove boxes
as needed. Include existing and
future user groups. Color code
each box based on legend below

37. 39
Splunk COE Recommendations
Roles Assignments
• A
• B
• C
Required Training
• A
• B
• C
architect
admin
developer
power user
Instructions: add recommendations to address role gaps with current and
future user groups. 1 person may carry more than 1 role, however Power
Users are usually different from team to team.
Instructions: add recommendations to address training gaps for current
and future user groups.

38. Thank You