Public Domain Image and Logs Before Splunk
•
2 likes•793 views
This document summarizes a presentation about a company's use of Splunk for log management and analytics. Some key points: - The company collects 400GB of logs per day across various clusters and forwards them to Splunk. - They have 1300 user accounts accessing Splunk across development, non-production, and US/EU production environments. - Splunk is used for tracking errors and volumes, identifying anomalies, network monitoring, and measuring performance. - In 2013 alone, over 350 issues were identified or resolved using Splunk.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (7)
Similar to Public Domain Image and Logs Before Splunk
Similar to Public Domain Image and Logs Before Splunk (20)
More from Splunk
More from Splunk (20)
Recently uploaded
Recently uploaded (20)
Public Domain Image and Logs Before Splunk
- 1. Copyright © 2015 Splunk Inc. Charlie Huggard Software Architect
- 10. Public Domain Image: The Tango! Desktop Project
- 11. Public Domain Image: The Tango! Desktop Project
- 12. By Christoph Neumüller From Wikimedia Commons
- 15. “Excuse me while I whip this out…” License: 400 GB/day Main Environment – 4 clusters, Dev, Non-Prod, US & EU Prod – Search Head, 2-7 indexers, 1000s of forwarders – Splunk 6.2.5 6.3.4 or 6.4.1+ soon – 1300 user accounts in Prod – 155 Billion Events+ in Prod Networking – Search Head, 2 indexers Rev Cycle – Standalone, with DB Connect
- 16. Logos refer to respective products If things failWhen things fail
- 17. “We’re doing great with Splunk. We love the Splunk!”
- 18. Crash Email Alerts Image Source: https://www.flickr.com/photos/barryskeates/7717816416/ Creative Commons Attribution License 2.0 60+ / Week < 1 / Month
- 19. Tracking errors and volumes per client
- 23. Measuring and Improving Performance
- 24. REST + Dashboards to the people …soon (DVPL-6794)
- 25. Help Desk
- 27. In 2013 350+ Issues Identified or Resolved with Splunk
- 28. Much Training
- 29. When a node is NOT configured for Splunk … “I tried to sort through the logs by hand and it was a NIGHTMARE”
- 30. Lesson #1
- 31. https://github.com/cerner/cerner_splunk Lesson #2 - Automate
- 32. Lesson #3 – Developers on Support
- 33. + = Lesson #3 – Developers on Support
- 36. Golan Ben-Oni, CSO/SVP Network Architecture, IDT “It comes back to the fact that [the competitors] don’t have the same user community base or following. Even if they had like functionality, they wouldn’t have the same large community that I can work with… The people are open. In a nutshell, that's the major difference between Splunk and the other guys.”
- 39. Options • IRC Client & You’re In • The original: #splunk Channel • Many Helpful Folks • Request an Invite: https://splunk402.com/chat/ • Regional & Topical Channels • Monthly #office_hours • SplunkTalk chat • Many Helpful Folks
- 45. Scheduled Local User Group Meetings Kansas City – Meetup at DST – May 19th St Louis – Hackathon – May 21st Use your splunk.com account & sign up: https://usergroups.splunk.com
- 48. Copyright © 2015 Splunk Inc. Thanks! Email: charlie.huggard@cerner.com Twitter: @acharlieh Splunk Chats: teddybfez
Editor's Notes
- My name is Charlie Huggard, for most of my career I was a Java Enterprise Developer, but my interest in how things fit together lead me to a role where I am responsible for how things connect together and are monitored. I’m not sure what it is, but it has also become a tradition that every Splunk talk I give, must involve a number of bears.
- I work for Cerner, a Kansas City based company
- Founded in 1979 to automate the Medical Laboratory.
- Now over 37 years later, we’re one of the leading providers of health care software in the world Been on Forbes most innovative list several times... Standard infographic slide.
- But let’s talk about our Splunk journey… where did we come from, where are we going and why are we in this handbasket?
- About 5-6 years ago, we had a push to enable more mobility for health care providers
- And not that much later… we started seeing a need to support the tracking the health not just of individuals
- But the health of entire populations at a time… to help ensure better qualities of care and meet changing market pressures. To build such a system, this means processing a lot of data.
- To process a lot of data, this means building a cloud... A lot of data means
- A really big cloud, and a really big cloud means
- Lots of logs
- And to sift through the logs was often like this… We saw the Bears, and occasionally a Tigers. Because we’re near Kansas it’s often times there are lions too * Some logs cycled before we could see what happened * Homegrown parsers * Sporadic reporting, monitoring or alerting Yes that is a 6 foot teddy bear named Sir Winston ChurchBear
- Lions and tigers and bears… get it? :-D
- It seems like every Splunk talk has to have a slide where they show off the size of their Splunk installation, and this talk is no different. We might see about bringing the other clusters back in…
- In any case, the first thought was, as developers check in code, build code, and deploy code, they would send logs to Splunk… and if things went wrong... Sorry... When things go wrong... They could see their logs in Splunk Splunk monitors activity from code commit through production Splunk ensures real-time health and stability of continuous integration for DevOps agility and responsiveness
- And the results are pretty good. Internal survey had among the quotes
- One of our mobile teams gets alerts on EVERY crash that occurs in the application. At first this was annoying, and everyone was like this is too much – 60+ emails per week. However this methodology/process has them under control. Other teams, use email alerts to post to community. Results are posted to the community of developers via social networking app for in-line conversation about the issue If this is the only thing you’re going to do with the data, don’t use splunk. It’s too expensive.
- One team put together a scripted lookup based on python UA Parser, and
- Patient Portal team - 1 Million Users Eventtypes are awesome. See Who is getting what error most frequently See what errors you haven’t seen yet!
- I think the scale is wrong on this one and those are microseconds not milliseconds. 10,000 microseconds of course being one hundreth of a second.
- The API can run searches and pull data into Dashing dashboards http://dashing.io/
- Newer versions, there’s a secured page that holds the mapping of person to userid
- Chris Hogan and Tom Twait’s dashboards to monitoring performance and transaction volumes going to 3rd party processors
- The year my team started managing Splunk, That we can directly attribute at least In 2015, there were > 3000 JIRAs mentioning Splunk. (but some of those are people developing things for Splunk…).
- Splunk has been really intuitive to our users, but we’re developing it now. Not really much training at that point.
- Operations folks are starting to expect Splunk
- This is a Fun Shirt…. Wearing it to a restaurant can cause awkward conversation with your waiter or waitress.
- Automate your universal forwarder deployment in a way that makes sense for you. Teams are deploying their code with chef, so we wanted to make it easy to deploy their monitoring as well
- “Splunk is just a pleasure to work with”
- As people learn about splunk, they’ll want Splunk brains!
- The Splunk community is made up of Customers, Partners, and Splunkers from around the world, and is one of the most helpful groups of folks the world
- The champions of the community are those who are members of the Splunk Trust. You’ll know them by their fezzes Ask them anything, they will try to help you get an answer. 22 worldwide for 2015-2016 2016-2017 class will be inducted at .conf
- Discuss, Share, Learn, & Network with local Splunk Users Always Need Speakers Venues Publicity
- From here, there are currently 5 Splunk User Groups within 4 hours away Kansas City, St Louis Ames Des Moines Eastern Nebraska
- Clint, Would you come up here?
- Clint is Director of Product Management at Splunk Clint is active in the community Speaks at User Groups Engages regularly on IRC and Slack taking feedback and explaining things Talks on our SplunkTalk podcast