Recommended
More Related Content
Viewers also liked
Viewers also liked (18)
Similar to SplunkLIve! Charlotte, Lumos Networks
Similar to SplunkLIve! Charlotte, Lumos Networks (20)
More from Splunk
More from Splunk (20)
Recently uploaded
Recently uploaded (20)
SplunkLIve! Charlotte, Lumos Networks
- 1. Jeremy Williams Senior IPTV Engineer
- 2. About Lumos Networks • Formerly NTelos • Fiber-based service provider in the Mid-Atlantic region • Provide leading-edge data, broadband, voice and IP services over an expanding fiber optic network Charlotte, March 29, 2012 2 Copyright © 2012, Splunk Inc.
- 3. About Me • IPTV Operations Group • Complete terminal responsibility of IPTV middleware • Splunk Ninja at Lumos • Create dashboards for Engineering and Service Operations Charlotte, March 29, 2012 3 Copyright © 2012, Splunk Inc.
- 4. Challenge Before Splunk • Middleware is Microsoft Media Room incl. MOM • Not able to correlate and analyze application event logs effectively • Impossible to prioritize problems • Events required detailed attention requiring manual processing of logs • Couldn't tell what was normal vs abnormal “ We didn't know what was wrong and what was right before Splunk ” Charlotte, March 29, 2012 4 Copyright © 2012, Splunk Inc.
- 5. How We Got Started with Splunk • Occasional user of free product • Advocating that we needed to bring Splunk in and evaluate • POC spoke for itself! “ There was complete consensus – this is what we need! Everybody just loved it! ” Charlotte, March 29, 2012 5 Copyright © 2012, Splunk Inc.
- 6. Touchstone Dashboard: Live Video Processing Charlotte, March 29, 2012 6 Copyright © 2012, Splunk Inc.
- 7. Drill-down Dashboards Missing ingress stream Service re-buffering Empty stream PIDs Exceeding configured Bitrate Charlotte, March 29, 2012 7 Copyright © 2012, Splunk Inc.
- 8. Using Server Logs to Analyze Session Retry Reports Charlotte, March 29, 2012 8 Copyright © 2012, Splunk Inc.
- 9. Field Service Operations Dashboard Charlotte, March 29, 2012 9 Copyright © 2012, Splunk Inc.
- 10. Researching Time Down From Hours to Seconds! Before Splunk... After Splunk... Charlotte, March 29, 2012 10 Copyright © 2012, Splunk Inc.
- 11. Splunk: The Solution Analytics with ad-hoc reporting (quality instead of quantity and break- fixes) – Comparing events over 400 unique services over different time periods – Trending rate of error, severity of error, frequency of error – Missing streams, rebuffering, exceeding bitrates – and more.. Drill-down and investigate as necessary by Operations and Engineering teams Role-based views for various users across the organization Charlotte, March 29, 2012 11 Copyright © 2012, Splunk Inc.
- 12. Thank You!
Editor's Notes
- Normal vs Abnormal problemsIn video world not everything 100% good all the time....Impossible to prioritize problems with MomIn mom each event requires attention (microsoft operations manager) It wasn't possible to correlate, associate events to draw better conclusions.Manually process to understand what was going onHow many of that error occurred on that channel yesterday? MOM can't address that sort of question.Mom focused on ops and event driven. Designed for errors that occur in a vacuum. Running out of disk spaceAnalyzing app event logs rate of error, distribution of error, frequency – more statistical analysis then x breaks y is responseVideo as an app we all deal with; people take video for granted – MPEG as a group of standards – is really not what you would think of as a well defined standard. It's more a set of tools and suggestions on how you can use those to construct various things.Analyzing log files that process this videoWe needed to solve for quality – mom can't do thatQuality is hard to track for:
- In MSN MOM, each event requires attentionCapability to analyze events in unified fashion or ask simple questions was missingNo way to correlate data together or associate events to get conclusionsTechnical integration problems – I had my hands tied. I had to have manually read the event logsMOM couldn’t answer the questions I was askingMOM is designed for errors that occur in a vacuum Technically interacting with video – mpeg as a group of s tandards is not well definedMore like a set of tools and suggestions on how to construct various thingsWe’re trying to analyze for quality – MOM had no concept/ capability to approach the problem that way
- Top ErrorsTop Errors ChannelsError Services Yesterday vs TodayThey can drill down into the events when they happen, see what the events werePrimary dash that both operations and engineering use as a launch pointField service also using Splunk (more later)
- We do look-ups EPG call sign, relevant IP addresses. MPEG program numbers, ingress streamDrill down dashboards – 2 channels from same provider – immediately provider issue – Service free buffering – Speficic host error – encoding gear misconfiguredOverall system health
- Dserver – instant channel change (pre-buffer very quickly) Reliable UDPSettop boxes can keep track of video stream. If packet is missing, the settop box can request packet from Dserver. WE can deliver solution to problem Just in Time. Dserver creates logs that tell us how amny and which packets they send to customers.Initially we didn't do anythign with them. We didn't knwo what was in them.Hey guys we can figure out how to analyze session retry reports. And look we can tell you cuz we index, which of your customers is having problems with delivery of videostream by requesting packets. We can figure out if certain settop boxes are having issues and requires attention. I heard about it and said, i have splunk – we figured out how to encode
- Dserver – instant channel change (pre-buffer very quickly) Reliable UDPSettop boxes can keep track of video stream. If packet is missing, the settop box can request packet from Dserver. WE can deliver solution to problem Just in Time. Dserver creates logs that tell us how amny and which packets they send to customers.Initially we didn't do anythign with them. We didn't knwo what was in them.Hey guys we can figure out how to analyze session retry reports. And look we can tell you cuz we index, which of your customers is having problems with delivery of videostream by requesting packets. We can figure out if certain settop boxes are having issues and requires attention. I heard about it and said, i have splunk – we figured out how to encode