Splunk at Ceryx Director Software Development

Splunk at Ceryx
Derek Mock
Director, Software Development

May 3, 2012

About Me…
Director, Software Development
10 years at Ceryx
Previous log management experience includes
• Late nights, lots of files and grep
Hobbies - possibly
Fav Splunk T-Shirt:
• Finding your faults, just like mom

Copyright © 2011, Splunk Inc. 2 Listen to your data.

Who we are …
A leader in providing enterprise unified communications via
the cloud model to large corporations with over 18+ years of
messaging and collaboration experience

Ceryx’s Cloud Control software dramatically reduces support
costs while enhancing/enabling customer experience.

Have developed a large integrator partner channel to address
this target market.


Ceryx Services

Cloud
Unified
Archiving Encryption Security Management Mobility
Messaging
Capabilities

Multi-Tenant (< 3000 seats) Dedicated (3000 + seats) Managed (3000 + seats)

• Shared Server Hardware • Dedicated Exchange and Active • Solution managed at customers DC (or
Directory locations of their choice)
• Complete mapping to existing • Monitored/Supported to Ceryx SLA’s
business processes with full integration to the customer’s
business process and operational
guidelines


The Initial Problem – Message Tracking
• Email logs across multiple platforms
• Growing infrastructure - increasing complexity
• New retention and compliance requirements
• Increasing number of helpdesk tickets being
opened
• Customer experience was declining - resolution
times increasing
• Training times for new staff were increasing


Message Tracking Before Splunk
Had to sift through:
Multiple layers of servers
Each layer with redundant
machines
Would run separate session
searches and try to sift through
all the data


Initial Splunk Deployment
• Used two data centers
• Deployed an indexer in each DC
• Deployed ~30 forwarders
• Setup distributed search
• Provided access to support staff only
• No customizations
• No apps


Message Tracking After Splunk “Aha”
Huge customer experience win:
End-to-end resolution now
takes 1 hour vs. 1 day
Organization has since grown,
2-3x number of servers
If we didn’t have Splunk we
would have needed to add
additional staff to support our
growth.

Why Splunk?
Securely see all logs from one place
Splunk is more cost-effective in licensing for throughput vs. per agent
or per user
Search GUI intuitive with great visualizations
Searching was quick and easy
Flexibility – can do things you can’t even think of
Apps – Can wrap our knowledge around the data


Our Environment
Deployed to multiple data centers
Deployed the forwarder to 400+ servers
6 indexers
50+ staff with Splunk access
Indexing 45+GB per day
And we aren’t done…


Machine Data Sources

MS Applications Perf Monitoring
Exchange Windows
SharePoint Linux
Lync/OCS

Custom Applications Web Servers Email Authentication Event Monitoring
App performance IIS logs Exchange MTA IAS Radius Windows event logs
and availability Apache logs Sendmail MTA (VPN/Secure ID) Linux system logs
Other MTA’s


Security & Compliance
Since deploying Splunk we have completed:
– ISO 27001
– SAS Type II
Handling security logs for both Ceryx
corporate data as well as customer data
Monitoring for violations etc.


Ceryx Security App
Custom app developed for
Security team
– System access monitoring and
alerting
– Abuse reporting
– Ad-hoc searches as required


Ceryx NOC
Custom apps developed for
Operations
– OS performance
Dashboards/Alerting
– App performance Alerting
– Event Log alerting
– Usage reports
– Capacity/Trending reports
Allows our NOC fast and
easy access to the
information they need when
responding to problems


Ceryx Service Delivery
Custom app developed for our
Support and Delivery Groups
– Assists in generation of monthly
reporting to our customers
– Respond to trends in customer
issues quicker
– Generate system usage reports


Ceryx Customer Support
Dashboard developed
by Support for Support
Enables live monitoring
of phone and ticket
queues
View ticket and call
distributions to ensure
adequate resource
utilization


Ceryx Software Development
Responsible for all in-house software development
– Cloud Control
– Monitoring Tools
 Health checks
 Availability
– Support Tools
 ITSM systems

Modifying our custom applications to log data easily ingested by Splunk
Provide log data to developers without needing server access
Increased our time to resolution on issues escalated by our Ops groups


Cloud Control App
Custom app for Cloud
Control
– Performance
– Web Analytics
– Usage
– Capacity
– Auditing


Splunk Dashboards
Wow!
Greatly accelerates creation of
weekly status reports
Share data on a large LED board
along with phone stats—a
“wow” moment for rest of org
to see live Splunk stats
CIO


Splunkbase
Some of the Apps we use:
– Splunk App for Windows
– Splunk App for Unix and Linux
– Google Maps
– Sideview Utils
– Web Intelligence
– Deployment Monitor
– Splunk for use with aaMap
– Maxmind (Geo Location)
– PDF Report Server


Other Customizations
Our NOC is very picky 
Developed additional search commands
– Custom alert formats
– Custom filtering options
– 3rd Party monitoring system integration


Splunk “Aha” Moments
Happen all the time now
Splunk graphical visualizations gave
immediate visibility
Allow us to look at data as a whole
instead of only at a component level
Easier access to historical data
makes analysis much quicker


Internal Transformation
Everyone got tired of “What was the system
doing last _____”
For many, Splunk has become the go to
application for troubleshooting issues
Show them a few searches and they are
hooked
Getting almost daily requests for “can we get
data X into Splunk?”
Increased awareness in the value of our data


Fielding Requests Beyond IT
Sales wanted to know where
users connected to Ceryx servers
throughout the world
Took a few clicks and about a
minute to generate the report
Answering questions for the
business paints IT as critical in
providing the data that shapes
business decisions.

What’s Next?
More Splunk:
– Migrating many alerting functions to be centralized in Splunk
– OS/Performance event data across the entire environment
– Continue to customize our in-house applications to log to Splunk
– The Splunk for VMware App
Give internal groups better visibility with customized Splunk apps for:
– Security-related info
– Custom application data
Give customers the ability to search Splunk data
– Leverage the API to give customers visibility and self-serve ability
– More visibility into messaging services – using data for their own internal
reporting

Hot Tips
Do a live demo with management to show non-technical people the
power of Splunk
Create a cheat-sheet for people with practical examples
Ask business owners what questions they’d like to ask/ answer, but
don’t know who/ how to ask
Storage sizing/ performance
Get involved in the Splunk Community


Toronto Splunk Users Group
Started in March 2012
Meets monthly and open to current Splunk customers
Discussions include:
– Splunk use cases
– Issues or problems people are having
– Best practices
– Beer
Splunk T-shirts
Contact me: derek.mock@ceryx.com or @derek_mock


Splunk at Ceryx Director Software Development

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Splunk at Ceryx Director Software Development

Similar to Splunk at Ceryx Director Software Development (20)

More from Splunk

More from Splunk (20)

Recently uploaded

Recently uploaded (20)

Splunk at Ceryx Director Software Development