About Me Linux System Administrator Husband and Father of 2 Kids DevOps, Productivity Hacks and Tools, The Big LebowskiTuesday, May 15, 12
OH: (during an outage) I don’t want to live in a world without Splunk.Tuesday, May 15, 12
Backstory Free instance installed in 2009 by the Network Team Single Instance on Central Log server Upgrade to EnterpriseTuesday, May 15, 12I started as an IT intern my mentor had a free copy of version 2.x running on the log server.I was tasked with finding a solution for SOX & PCI requirements. (Which was mind expanding for an intern, to say the least)Worked with purchasing to get a small license for the enterprise features.My project ended up piping Splunk output into a python program that no one but I understood that printed out a text report that (I felt at least) was superior to the one in place at the time. (Bigsurprise, didn’t end up using it).
Building Blocks Split Splunk off onto dedicated instance License overwhelmed by single app Limited visibility and useTuesday, May 15, 12When I came back there was some cursory interest in the app, but no major users and no project champion.Welcome, back Tyler... Splunk Expert (by Default).I was also attached to Garmin Connect, which is our awesome fitness tracking site, after getting more comfortable in my settings, I began to integrate the site logs into Splunk
IF YOU HAVE MORE INPUTS THAN LICENSE YOU’RE GONNA HAVE A BAD TIMETuesday, May 15, 12Obvious, but this was my experience during the ﬁrst dedicated instance. We had a small license and it was all being used by Garmin Connect. It really wasn’t taking holdlike I knew it could.
Plan for Expansion Decided to make application more robust Read the Documentation Planned roll out Multiple Applications License Increase Scalable ArchitectureTuesday, May 15, 12After I became more comfortable in my position, I felt impelled to make the application more robust and widespread.I went to .conf last year, attended some training sessions and read up on the Administration documentation.
Enterprise Architecture Elements (so far) Puppet Deploy Infrastructure Layout Gotchas Future PlansTuesday, May 15, 12Overview of the Current Architecture Elements, will then go in depth a bit more on each subject.
Puppet Search, Indexer and Forwarder are “turn-key” ex: include splunk::indexer ...done Really Awesome for ForwardersTuesday, May 15, 12Puppet makes deployment simple. Servers are built with one include statement.Forwarders are split up based on role and inputs. Customize the inputs a bit if necessary and include the splunk forwarder class in the puppet node deﬁnition.
InfrastructureTuesday, May 15, 12Describe layers and functions.Search is load balanced.Search, Index and Forwarders are horizontally scalable.Network/Taiwan instances aren’t pictured but are separate dedicated instances. Will move the network index into the main infrastructure real soon now.
How We Use Splunk Web Access Logs Service Usage Metrics Feature Tracking Diagnosing Problems in Production Internal Application Audits Windows Security EventsTuesday, May 15, 12We don’t have a wide variety if inputs into Splunk at the moment.We currently use it on all of the major IT web applications to obtain service metrics, track new features and diagnosing issues in Production.The developers are also starting to cater their applications to output Splunk friendly logsWindows security events are queried via WMI ad ﬁltered to speciﬁc IDs, this helps keep the volume down while delivering value for the Windows guys.
Why I like Splunk Makes Users Happy Real Time Data No AlternativesTuesday, May 15, 12Ease of configuration, having the one stop shop for user-land configs. LDAP integration is super simple.Able to generate detailed reports and drill into the data on the fly is a killer feature and something that you simply wont find with any other application.User community and Documentation.There are no real alternatives to Splunk. Some tools touch on some of the features gained with the app, but there is no offering that matches what splunk can give you. I’vetried SEC, logwatch, Logstash, and Spiceworks. None were as user friendly and robust as Splunk.
Gotchas Don’t Index a lot of data over NFS Shared Knowledge Bundle Time Sync Tag and Search permissionsTuesday, May 15, 12Keeping up with the demand. From a license and user request perspective, I limited amount of time to handle the requests at hand. Familiar position for me at least, but a good problem tohave.Mounted Bundles must have the same time across the board.Watch your permissions on saved searches and tags. They are usually private when I share them with another user and they cannot access.
Future Plans Fix Central Logging Check Out Deployment Server More Inputs TrainingTuesday, May 15, 12Currently only one centralized syslog server, want to scale it out and put a farm of syslog servers behind a load balancer. Splunk will be the deﬁninitive timeline for syslogevents.Read about Deployment Server but passed on it at the time. Would like to pick it back up and see how it could be beneﬁcial.Add additional inputs to the applicationI’ve been tasked with training my coworkers on how to use the application. Once they pick it up and ﬁgure it out, they can do awesome things.
Tips and Advice WMI Event Filter for Windows Events Splunkbase (stack overﬂow engine)Tuesday, May 15, 12