SplunkLive! Toronto - Ceryx


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

SplunkLive! Toronto - Ceryx

  1. 1. Splunk at CeryxDerek MockDirector, Software DevelopmentMay 3, 2012
  2. 2. About Me… Director, Software Development 10 years at Ceryx Previous log management experience includes • Late nights, lots of files and grep Hobbies - possibly Fav Splunk T-Shirt: • Finding your faults, just like momCopyright © 2011, Splunk Inc. 2 Listen to your data.
  3. 3. Who we are … A leader in providing enterprise unified communications via the cloud model to large corporations with over 18+ years of messaging and collaboration experience Ceryx’s Cloud Control software dramatically reduces support costs while enhancing/enabling customer experience. Have developed a large integrator partner channel to address this target market.Copyright © 2011, Splunk Inc. 3 Listen to your data.
  4. 4. Ceryx Services Cloud Unified Archiving Encryption Security Management Mobility Messaging Capabilities Multi-Tenant (< 3000 seats) Dedicated (3000 + seats) Managed (3000 + seats) • Shared Server Hardware • Dedicated Exchange and Active • Solution managed at customers DC (or Directory locations of their choice) • Complete mapping to existing • Monitored/Supported to Ceryx SLA’s business processes with full integration to the customer’s business process and operational guidelinesCopyright © 2011, Splunk Inc. 4 Listen to your data.
  5. 5. The Initial Problem – Message Tracking• Email logs across multiple platforms• Growing infrastructure - increasing complexity• New retention and compliance requirements• Increasing number of helpdesk tickets being opened• Customer experience was declining - resolution times increasing• Training times for new staff were increasingCopyright © 2011, Splunk Inc. 5 Listen to your data.
  6. 6. Message Tracking Before Splunk Had to sift through: Multiple layers of servers Each layer with redundant machines Would run separate session searches and try to sift through all the dataCopyright © 2011, Splunk Inc. 6 Listen to your data.
  7. 7. Initial Splunk Deployment• Used two data centers• Deployed an indexer in each DC• Deployed ~30 forwarders• Setup distributed search• Provided access to support staff only• No customizations• No appsCopyright © 2011, Splunk Inc. 7 Listen to your data.
  8. 8. Message Tracking After Splunk “Aha” Huge customer experience win: End-to-end resolution now takes 1 hour vs. 1 day Organization has since grown, 2-3x number of servers If we didn’t have Splunk we would have needed to add additional staff to support our growth.Copyright © 2011, Splunk Inc. 8 Listen to your data.
  9. 9. Why Splunk? Securely see all logs from one place Splunk is more cost-effective in licensing for throughput vs. per agent or per user Search GUI intuitive with great visualizations Searching was quick and easy Flexibility – can do things you can’t even think of Apps – Can wrap our knowledge around the dataCopyright © 2011, Splunk Inc. 9 Listen to your data.
  10. 10. Our Environment Deployed to multiple data centers Deployed the forwarder to 400+ servers 6 indexers 50+ staff with Splunk access Indexing 45+GB per day And we aren’t done…Copyright © 2011, Splunk Inc. 10 Listen to your data.
  11. 11. Machine Data SourcesMS Applications Perf MonitoringExchange WindowsSharePoint LinuxLync/OCS Custom Applications Web Servers Email Authentication Event Monitoring App performance IIS logs Exchange MTA IAS Radius Windows event logs and availability Apache logs Sendmail MTA (VPN/Secure ID) Linux system logs Other MTA’s Copyright © 2011, Splunk Inc. 11 Listen to your data.
  12. 12. Security & Compliance Since deploying Splunk we have completed: – ISO 27001 – SAS Type II Handling security logs for both Ceryx corporate data as well as customer data Monitoring for violations etc.Copyright © 2011, Splunk Inc. 12 Listen to your data.
  13. 13. Ceryx Security App Custom app developed for Security team – System access monitoring and alerting – Abuse reporting – Ad-hoc searches as requiredCopyright © 2011, Splunk Inc. 13 Listen to your data.
  14. 14. Ceryx NOC Custom apps developed for Operations – OS performance Dashboards/Alerting – App performance Alerting – Event Log alerting – Usage reports – Capacity/Trending reports Allows our NOC fast and easy access to the information they need when responding to problemsCopyright © 2011, Splunk Inc. 14 Listen to your data.
  15. 15. Ceryx Service Delivery Custom app developed for our Support and Delivery Groups – Assists in generation of monthly reporting to our customers – Respond to trends in customer issues quicker – Generate system usage reportsCopyright © 2011, Splunk Inc. 15 Listen to your data.
  16. 16. Ceryx Customer Support Dashboard developed by Support for Support Enables live monitoring of phone and ticket queues View ticket and call distributions to ensure adequate resource utilizationCopyright © 2011, Splunk Inc. 16 Listen to your data.
  17. 17. Ceryx Software Development Responsible for all in-house software development – Cloud Control – Monitoring Tools  Health checks  Availability – Support Tools  ITSM systems Modifying our custom applications to log data easily ingested by Splunk Provide log data to developers without needing server access Increased our time to resolution on issues escalated by our Ops groupsCopyright © 2011, Splunk Inc. 17 Listen to your data.
  18. 18. Cloud Control App Custom app for Cloud Control – Performance – Web Analytics – Usage – Capacity – AuditingCopyright © 2011, Splunk Inc. 18 Listen to your data.
  19. 19. Splunk Dashboards Wow! Greatly accelerates creation of weekly status reports Share data on a large LED board along with phone stats—a “wow” moment for rest of org to see live Splunk stats CIOCopyright © 2011, Splunk Inc. 19 Listen to your data.
  20. 20. Splunkbase Some of the Apps we use: – Splunk App for Windows – Splunk App for Unix and Linux – Google Maps – Sideview Utils – Web Intelligence – Deployment Monitor – Splunk for use with aaMap – Maxmind (Geo Location) – PDF Report ServerCopyright © 2011, Splunk Inc. 20 Listen to your data.
  21. 21. Other Customizations Our NOC is very picky  Developed additional search commands – Custom alert formats – Custom filtering options – 3rd Party monitoring system integrationCopyright © 2011, Splunk Inc. 21 Listen to your data.
  22. 22. Splunk “Aha” Moments Happen all the time now Splunk graphical visualizations gave immediate visibility Allow us to look at data as a whole instead of only at a component level Easier access to historical data makes analysis much quickerCopyright © 2011, Splunk Inc. 22 Listen to your data.
  23. 23. Internal Transformation Everyone got tired of “What was the system doing last _____” For many, Splunk has become the go to application for troubleshooting issues Show them a few searches and they are hooked Getting almost daily requests for “can we get data X into Splunk?” Increased awareness in the value of our dataCopyright © 2011, Splunk Inc. 23 Listen to your data.
  24. 24. Fielding Requests Beyond IT Sales wanted to know where users connected to Ceryx servers throughout the world Took a few clicks and about a minute to generate the report Answering questions for the business paints IT as critical in providing the data that shapes business decisions.Copyright © 2011, Splunk Inc. 24 Listen to your data.
  25. 25. What’s Next? More Splunk: – Migrating many alerting functions to be centralized in Splunk – OS/Performance event data across the entire environment – Continue to customize our in-house applications to log to Splunk – The Splunk for VMware App Give internal groups better visibility with customized Splunk apps for: – Security-related info – Custom application data Give customers the ability to search Splunk data – Leverage the API to give customers visibility and self-serve ability – More visibility into messaging services – using data for their own internal reportingCopyright © 2011, Splunk Inc. 25 Listen to your data.
  26. 26. Hot Tips Do a live demo with management to show non-technical people the power of Splunk Create a cheat-sheet for people with practical examples Ask business owners what questions they’d like to ask/ answer, but don’t know who/ how to ask Storage sizing/ performance Get involved in the Splunk CommunityCopyright © 2011, Splunk Inc. 26 Listen to your data.
  27. 27. Toronto Splunk Users Group Started in March 2012 Meets monthly and open to current Splunk customers Discussions include: – Splunk use cases – Issues or problems people are having – Best practices – Beer Splunk T-shirts Contact me: derek.mock@ceryx.com or @derek_mockCopyright © 2011, Splunk Inc. 27 Listen to your data.
  28. 28. Questions?
  29. 29. Thank youMay 3, 2012