Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

SplunkLIve! Charlotte, Lumos Networks

637
views

Published on

SplunkLIve! Charlotte, Lumos Networks

SplunkLIve! Charlotte, Lumos Networks

Published in: Technology, Business

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
637
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Normal vs Abnormal problemsIn video world not everything 100% good all the time....Impossible to prioritize problems with MomIn mom each event requires attention (microsoft operations manager) It wasn't possible to correlate, associate events to draw better conclusions.Manually process to understand what was going onHow many of that error occurred on that channel yesterday? MOM can't address that sort of question.Mom focused on ops and event driven. Designed for errors that occur in a vacuum. Running out of disk spaceAnalyzing app event logs rate of error, distribution of error, frequency – more statistical analysis then x breaks y is responseVideo as an app we all deal with; people take video for granted – MPEG as a group of standards – is really not what you would think of as a well defined standard. It's more a set of tools and suggestions on how you can use those to construct various things.Analyzing log files that process this videoWe needed to solve for quality – mom can't do thatQuality is hard to track for:
  • In MSN MOM, each event requires attentionCapability to analyze events in unified fashion or ask simple questions was missingNo way to correlate data together or associate events to get conclusionsTechnical integration problems – I had my hands tied. I had to have manually read the event logsMOM couldn’t answer the questions I was askingMOM is designed for errors that occur in a vacuum Technically interacting with video – mpeg as a group of s tandards is not well definedMore like a set of tools and suggestions on how to construct various thingsWe’re trying to analyze for quality – MOM had no concept/ capability to approach the problem that way
  • Top ErrorsTop Errors ChannelsError Services Yesterday vs TodayThey can drill down into the events when they happen, see what the events werePrimary dash that both operations and engineering use as a launch pointField service also using Splunk (more later)
  • We do look-ups EPG call sign, relevant IP addresses. MPEG program numbers, ingress streamDrill down dashboards – 2 channels from same provider – immediately provider issue – Service free buffering – Speficic host error – encoding gear misconfiguredOverall system health
  • Dserver – instant channel change (pre-buffer very quickly) Reliable UDPSettop boxes can keep track of video stream. If packet is missing, the settop box can request packet from Dserver. WE can deliver solution to problem Just in Time. Dserver creates logs that tell us how amny and which packets they send to customers.Initially we didn't do anythign with them. We didn't knwo what was in them.Hey guys we can figure out how to analyze session retry reports. And look we can tell you cuz we index, which of your customers is having problems with delivery of videostream by requesting packets. We can figure out if certain settop boxes are having issues and requires attention. I heard about it and said, i have splunk – we figured out how to encode
  • Dserver – instant channel change (pre-buffer very quickly) Reliable UDPSettop boxes can keep track of video stream. If packet is missing, the settop box can request packet from Dserver. WE can deliver solution to problem Just in Time. Dserver creates logs that tell us how amny and which packets they send to customers.Initially we didn't do anythign with them. We didn't knwo what was in them.Hey guys we can figure out how to analyze session retry reports. And look we can tell you cuz we index, which of your customers is having problems with delivery of videostream by requesting packets. We can figure out if certain settop boxes are having issues and requires attention. I heard about it and said, i have splunk – we figured out how to encode
  • Transcript

    • 1. Jeremy WilliamsSenior IPTV Engineer
    • 2. About Lumos Networks• Formerly NTelos• Fiber-based service provider in the Mid-Atlantic region• Provide leading-edge data, broadband, voice and IP services over an expanding fiber optic network Charlotte, March 29, 2012 2 Copyright © 2012, Splunk Inc.
    • 3. About Me• IPTV Operations Group• Complete terminal responsibility of IPTV middleware• Splunk Ninja at Lumos• Create dashboards for Engineering and Service Operations Charlotte, March 29, 2012 3 Copyright © 2012, Splunk Inc.
    • 4. Challenge Before Splunk• Middleware is Microsoft Media Room incl. MOM• Not able to correlate and analyze application event logs effectively• Impossible to prioritize problems• Events required detailed attention requiring manual processing of logs• Couldnt tell what was normal vs abnormal “ We didnt know what was wrong and what was right before Splunk ” Charlotte, March 29, 2012 4 Copyright © 2012, Splunk Inc.
    • 5. How We Got Started with Splunk• Occasional user of free product• Advocating that we needed to bring Splunk in and evaluate• POC spoke for itself! “ There was complete consensus – this is what we need! Everybody just loved it! ” Charlotte, March 29, 2012 5 Copyright © 2012, Splunk Inc.
    • 6. Touchstone Dashboard: Live Video Processing Charlotte, March 29, 2012 6 Copyright © 2012, Splunk Inc.
    • 7. Drill-down DashboardsMissing ingress streamService re-bufferingEmpty stream PIDsExceeding configured Bitrate Charlotte, March 29, 2012 7 Copyright © 2012, Splunk Inc.
    • 8. Using Server Logs to Analyze Session Retry Reports Charlotte, March 29, 2012 8 Copyright © 2012, Splunk Inc.
    • 9. Field Service Operations Dashboard Charlotte, March 29, 2012 9 Copyright © 2012, Splunk Inc.
    • 10. Researching Time Down From Hours to Seconds! Before Splunk... After Splunk... Charlotte, March 29, 2012 10 Copyright © 2012, Splunk Inc.
    • 11. Splunk: The SolutionAnalytics with ad-hoc reporting (quality instead of quantity and break-fixes)– Comparing events over 400 unique services over different time periods– Trending rate of error, severity of error, frequency of error– Missing streams, rebuffering, exceeding bitrates– and more..Drill-down and investigate as necessary by Operations and EngineeringteamsRole-based views for various users across the organization Charlotte, March 29, 2012 11 Copyright © 2012, Splunk Inc.
    • 12. Thank You!

    ×