Cerner at SplunkLive! Minneapolis
•
2 likes•856 views
The document summarizes a company's use of Splunk for log management and analytics. It describes their Splunk environment with multiple clusters, around 1000 users, and applications of Splunk for tracking errors, anomalies, and performance. It discusses lessons learned around query impacts, timestamps, and resources for support. Future plans include search head clustering, heavy forwarders, and apps for visibility and intelligence to enable agility and improved code quality.
More Related Content
What's hot
What's hot (10)
Viewers also liked
Viewers also liked (13)
Similar to Cerner at SplunkLive! Minneapolis
Similar to Cerner at SplunkLive! Minneapolis (20)
More from Splunk
More from Splunk (20)
Recently uploaded
Recently uploaded (20)
Cerner at SplunkLive! Minneapolis
- 1. Copyright © 2015 Splunk Inc. Charlie Huggard Software Architect
- 14. Public Domain Image: The Tango! Desktop Project
- 15. Public Domain Image: The Tango! Desktop Project
- 16. By Christoph Neumüller From Wikimedia Commons
- 20. Show us your Splunk Total License: 400 GB/ day Main Environment – 3 clusters, Dev, Non-Prod, Prod UK soon – Approximately 1000 users – Search Head, 2-3 indexers, ??? forwarders each – Splunk 6.0.x, 6.2.x soon Networking – Search Head, 2 indexers Rev Cycle – Standalone, with DB Connect
- 21. Logos refer to respective products If things failWhen things fail
- 23. “We’re doing great with Splunk. We love the Splunk!”
- 24. Crash Email Alerts Image Source: https://www.flickr.com/photos/barryskeates/7717816416/ Creative Commons Attribution License 2.0 60+ / Week < 1 / Month
- 25. Tracking errors and volumes per client
- 29. Measuring and Improving Performance
- 30. Dashboards for Everyone REST API +
- 32. In 2013 350+ Issues Identified or Resolved using Splunk
- 33. Much Training
- 34. When a node is NOT configured for Splunk … “I tried to sort through the logs by hand and it was a NIGHTMARE”
- 36. Lessons Learned
- 37. Lessons Learned – Developers on Support
- 40. Lessons Learned
- 41. Lessons Learned Bad Queries can impact performance – Real-time is cool, but expensive Timestamps early in events are nice to Splunk http://wiki.splunk.com/Community:HowIndexingWorks
- 42. Good Resources #splunk on EFNet answers.splunk.com Your friendly sales engineers docs.splunk.com
- 45. Looking Ahead • Common Information Model • Splunk 6.2 • Search Head Clustering • Heavy Forwarders • Splunk App for VMWare • Web Framework • Splunk Mobile App CC BY 2.0 Image: https://www.flickr.com/photos/popculturegeek/5134039427
- 46. • Visibility into app performance across product suite • Visibility into KPIs/ dashboards for management teams • Quickly validate and troubleshoot code before it hits production • Ensure that new code doesn’t negatively impact performance or user experience • Developers focus on innovation vs. remediation • Dashboards showing operational status + outliers replacing frequent alerts • Combat alert fatigue • Capers Jones estimates $16,000 for production level defect Reduced MTTR and Alert Noise Improved Code Quality Delivering Operational Intelligence Results with Splunk Summary • Quickly identify and fix issues during continuous integration • Developers now have access to production data Enabling DevOps Agility
- 47. Thanks! Email : charlie.huggard@cerner.com Many Social Nets: acharlieh EFNet: acharfez
Editor's Notes
- I work for Cerner, a Kansas City based company
- Founded in 1979 to automate the Medical Laboratory.
- In the 35 years since, we’re a 3 billion dollar public company, hundreds of solutions now cover the entire continuum of health + care, including managed services, revenue management, wellness programs, and more 21000 associates 18000 sites 30 countries
- I started my career as primarily a Java Developer, but with an affinity toward how systems work together From that interest, about 2 years ago, I transitioned to a role where my goal is to enable teams to automate and get the most information about their systems as possible with the goal of delivering the best value for our clients. So I currently manage Cerner’s Splunk architecture and direction.
- All of which lead to putting together this gripping story, Co authored by my teddy bear
- Of Ponies, and Superheroes, entitled: Splunk> OR…
- “How I learned to stop worrying and to love my logs”
- Brown Bear, His doctor, and His insurance company. (Who better than Teddy Roosevelt to represent a Teddy Bear’s health insurance company) Peter Davidson, 5th Doctor.
- Update this for today (new doctor) The insurance company says not only do I have brown bear, but I have a few million of brown bear’s closest friends, I’ll pay you to keep them healthy.
- How would you start building such a system? What sort of things would we need
- You’d need data from all kinds of different sources, not just hospital visits, but also
- Need that data available, mobile, and in a clear fashion
- To build these apps means Petabytes of data, which means building out some form of cloud
- A really big cloud
- Lots of logs
- Looking through these logs was like this… We saw the Bears, and occasionally a Tigers. We heard there might be lions too, but * Didn’t know what we didn’t know * Some logs cycled before we could see what happened * Homegrown parsers, collectors and BI tools didn’t scale * Sporadic reporting, monitoring or alerting Yes that is a 6 foot teddy bear named Sir Winston ChurchBear
- Splunk monitors activity from code commit through production Splunk ensures real-time health and stability of continuous integration for DevOps agility and responsiveness
- There are other splunk cookbooks out there, but this is ours.
- One of our mobile teams gets alerts on EVERY crash that occurs in the application. At first this was annoying, and everyone was like this is too much – 60+ emails per week. However this methodology/process has them under control. Other teams, use email alerts to post to community. Results are posted to the community of developers via social networking app for in-line conversation about the issue
- One team put together a scripted lookup based on python UA Parser, and
- Patient Portal team - 1 Million Users Eventtypes are awesome. See Who is getting what error most frequently See what errors you haven’t seen yet!
- I think the scale is wrong on this one and those are microseconds not milliseconds. 10,000 microseconds of course being one hundreth of a second.
- The API can run searches and pull data into Dashing dashboards http://dashing.io/
- Chris Hogan and Tom Twait’s dashboards to monitoring performance and transaction volumes going to 3rd party processors
- That we can directly attribute at least Haven’t gone counting through JIRA to see for updated stats for 2014
- Splunk has been really intuitive to our users Not really much training at this point.
- Operations folks are starting to expect Splunk
- Look out over the ocean, and what Lessons have we learned
- This is a Fun Shirt…. Wearing it to a restaurant can cause awkward conversation with your waiter or waitress.
- “Splunk is just a pleasure to work with”
- As people learn about splunk, they’ll want to eat your brains!
- Leverage LDAP Authentication & Authorization Map Groups to custom roles in Splunk. Give your users their own boxes (apps & indexes) to play in.
- The Splunk Community is fun and very supportive of each other.
- The Splunk Fez is not given, it’s earned. The only way to get the Splunk fez, is to participate in the Splunk community, and to have Rachel “piebob” bestow one upon you. Rachel “Piebob” Perkins – Sr. Director of Community, and ministry of silly hats.
- It used to be at best a couple day process for developers to get logs. MTTR – we are now fixing issues in minutes that would have taken hours or potentially gone unnoticed. The overall code base is running cleaner every day. We are also opening up executive visibility to things that they have been asking for for years.