Your SlideShare is downloading. ×
0
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Challenges and experiences with IPTV from a network point of view
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Challenges and experiences with IPTV from a network point of view

2,327

Published on

OpenSource IPTV MPEG2-TS analyzer. …

OpenSource IPTV MPEG2-TS analyzer.

This presentation was given at OpenSourceDays 2010 (and in earlier stages of the project at Bifrost Workshop 2009 and 2010)

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,327
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
61
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • (BCU-26 currently have 2223 active equipment, cust:1176, iptables chains:7551 rules:32015) (Amager POP, Internet customers:4892 with active equipment:8615.)
  • TALK: First I'll focus on the routing performance issue I'll come back to slow rule changes later
  • Summary for 1.0.0 libpcap release, October 27, 2008. On Linux, support new tpacket frame headers (2.6.27+)
  • Drop this slide???
  • Transcript

    • 1. Open Source Days 2010 d.6/3-2010 ComX Networks A/S by Jesper Dangaard Brouer <jdb@comx.dk> Master of Computer Science ComX Networks A/S
    • 2. Who am I
      • Name: Jesper Dangaard Brouer
        • Edu: Computer Science for Uni. Copenhagen
          • Focus on Network, Dist. sys and OS
        • Linux user since 1996, professional since 1998
          • Sysadm, Developer, Embedded
        • OpenSource projects
          • Author of
            • ADSL-optimizer
            • 3. CPAN IPTables::libiptc
          • Patches accepted into
            • Linux kernel, iproute2, iptables and Wireshark
    • 4. What will you learn?
      • This talk is about network experiences with IPTV
        • Not about: challenges to get multicast to work
        • 5. Part1: about trouble shooting IPTV network problems
        • 6. Part 2: about developing our own IPTV monitor probe
        • 7. Future plans for an IPTV shaper
    • 8. ComX: Physical surroundings
      • ComX Networks A/S: Fiber based solutions
        • variety of services (TV, IPTV, VoIP, Internet)
      • Our primary customers are
          • Apartment buildings with end-user relationship
          • 9. OEM customers, like “Syd Energi”
      • Apartment building have Ethernet switches
        • Port separation, “private” VLAN
        • 10. CPE box in apartment performs
          • service separation into VLANs
    • 11. What is this talk NOT about
      • Lot of challenges to implement multicast routing
        • Its not straight forward, BUT its not the focus of this talk.
      • Shortly, pitfalls to watch out for:
        • It can be difficult to :
          • get different vendors multicast to work together
          • 12. get even the same vendor
            • Watch out for firmware upgrades...
            • 13. E.g: New firmware (on Foundry FWSX 424 ) drops packets, MC in HW.
        • Also Remember:
            • Apartment buildings Ethernet switches, proper config.
            • 14. CPE equipment needs IGMP snooping, more Settop boxes
    • 15. The focus: Scalability
      • What happened when more channels were added
        • High-Definition channels
        • 16. More than 120 IPTV channels
        • 17. Approx 600 Mbit/s multicast traffic.
      • What happens to a realtime service
        • on a loaded ethernet switch based network?
    • 18. Basics: MPEG2 TS packets
      • Each IP-packet contains 7 Transport Stream Packets/Frames
      • 19. The Continuity Counter is very small values 0-15
        • Thus, an exact drop counter is hard as wrap around occurs quickly
    • 20. History: What was the problem?
      • Customers were experiencing bad quality,
        • due to packet drops
      • Traffic had reached 600Mbit/s
        • Drops were worse on loaded links.
        • 21. But dedicated links also showed significant drops
          • 60 percent loaded link should not have drops
      • Finding the cause of packet drops
        • Turned out to be a harder task than expected...
    • 22. History: Measure the problem
      • First objective: Be able to measure the problem
        • Used &quot;IP-probe&quot; from BridgeTech, to detect/see drops.
          • Which is an IPTV analyzer appliance
            • internally runs Linux and has a special FPGA chip
            • 23. Really nice tool, but very expensive!
      • Work under pressure: OEM Customer mad!!!
        • Network Core team was getting desperate
          • Foundry support
            • Found issues with their PIM implementation
              • Bad distribution over trunked links
              • 24. Desperate clear PIM cache drills
    • 25. What caused the drops?
      • Bursty traffic from some of the IPTV streamers.
        • Some of the &quot;cheap&quot; (100k DKK) streamers, bursting
        • 26. Worst: Linux boxes with VLC and old 2.6.18 kernel
      • Linux VLC machines easily fixed
        • by newer kernel with high resolution timer support
      • The commercial streamers were harder to &quot;fix&quot;
        • Closed source
        • 27. Slow and expensive support
        • 28. Kickoff IPTV probe/shaper project based on Linux
    • 29.
      • Important to enable
        • high resolution timers and tickless OS
          • to avoid VLC multicast bursts
      • Kernel config options
        • CONFIG_HIGH_RES_TIMERS=y
        • 30. CONFIG_NO_HZ=y
      • Watch the kernel log messages for:
      “ Switched to high resolution mode on CPU 0” or “ Could not switch to high resolution mode on CPU 0” Hint slide: Enable Linux Highres
    • 31. How did we locate the problem?
      • BridgeTech probe tool, very helpful, showing the drops
        • but could not show the burstiness directly
      • One of the Linux based streamers,
        • Alternating between 6 Mbit/s and 15 Mbit/s streams
        • 32. All channels experienced more drops during 15 Mbit/s streams
      • Detailed analysis of traffic dumps
        • Wireshark was used to show burstiness
        • 33. Wireshark IO-graph with resolution < 1 ms
        • 34. Implemented drop detect our selves in Wireshark
        • 35. (the power of open source)
    • 36. Wireshark IO-graph
      • Menu: Statistics -> IO Graph -> Tick Interval 0.001
    • 37. Wireshark drop detection
      • MPEG2 TS drop detection in Wireshark
          • Included in mainline since SVN rev. 27381
        • Not fast enough for realtime/continuous usage
        • 38. Dumping 600Mbit/s traffic
          • Standard tcpdump not fast enough
            • WARN cause “fake” packet drops (Func: “packet_rcv”)
          • Use tcpdump/pcap with Mem Mapped IO
            • Use minimum libpcap version 1.0.0
            • 39. Function: “ t packet_rcv” (See /proc/net/ptype)
    • 40. Part 1: Conclusion Watch out for bursts
      • Packets bursts experiences
        • Surprisingly bursts do cause problems on 60% loaded links
        • 41. Non-bursty streams are affected by bursty-streams
        • 42. On Linux: Use Highres timers and no tick
        • 43. Know you switch buffer sizes
          • don't place low-buffer switches in backbone
    • 44. Part 2: Develop our own IPTV probe
      • Learned
        • Need to monitor the signal
        • 45. Bursts can cause drops, on all channels
        • 46. Bursts can occur per network segment
      • Monitor each network segment
        • Approx measurement points: 12
        • 47. Price per IPTV-probe approx 80.000 DKK
          • Total cost: 960.000 DKK
        • Develop our own IPTV-probe
    • 48. Project plan: IPTV probe/shaper Develop our own MPEG2-TS analyser
      • Milestone 1 : Measure the drops – DONE
        • First objective: measure the problem
      • Milestone 2 : Measure the bursts
        • Detect potential issues, before drops occur
      • Milestone 3 : Smooth out the bursts
        • Solve the burst issue permanently
    • 49. Milestone 1: Measure drops
      • Offline analysis
        • Contribute to Wirehark, mostly prototyping
      • Need continuous monitoring (24x7x365)
        • Kernel module for detection (iptables)
        • 50. Collector daemon (Perl)
        • 51. Database design, storing events (MySQL)
        • 52. Web based frontend (PHP)
        • 53. Trick: use settop boxes as probes (bash)
    • 54. IPTV Probe architecture
    • 55. Kernel module: Efficient!
      • Developed iptables/netfilter module (mp2t)
        • For realtime/continuous drop monitoring
        • 56. Only uses 2% CPU with 600Mbit/s traffic
        • 57. on a low power ATOM 330 CPU (1.6Ghz)
        • 58. tcpdump solution used 100% and caused false drops
      • Implemented with RCU locking
        • for parallel processing on each CPU
        • 59. (requires multi-queue NICs)
      • Stats via /proc/ file
        • Collector currently polls every 10 sec
    • 60. Collector
      • Design
        • Only report on changes
      • Poll every 10 sec
        • Compare with previous data
      • Plan: Avoid polling
        • kernel report activity (via netlink)
      • Written as Perl daemon process
    • 61. Web-Frontend
      • Just finished the web-frontend
        • Its not pretty, but it works!
        • 62. PHP developers needed on the project!
    • 63. Web-frontend: Probe overview
    • 64. Web-frontend: Channel view
    • 65. Trick: Use settop box as probes
      • See what the customer sees
        • As many probes as customers
        • 66. But only one channel at a time
      • Trick: The settop box runs Linux
        • local tool support asking for “sync errors”
        • 67. install small bash script
          • periodically poll, and submit result back central
      • Still need probes, in the network
        • need to identify the network segment
    • 68. Settox boxes as probes
    • 69. Open Source Status
      • Wireshark
        • Drop patches accepted
        • 70. PCR-clock patches, not accepted :-(
      • IPTV probe tool: (Is anybody interested?)
        • Kernel module is GPL, but not released public
          • Contact me if you want a copy...
        • Collector + DB design will hopefully be GPL
          • Want the same basis for frontends
        • Frontend
          • We are looking for PHP developers...
    • 71. Future?
      • Milestone 2 : Bursts, detect issues, before drops occur
        • Currently down prioritized due to other tasks...
      • Milestone 3 : Shaper, solve issue
        • Have an implementation idea...
        • 72. Will it ever happen?
      • Progress through cooperation?
        • Anybody want to cooperate?
          • Or I am a just an one-man-army™ ?
    • 73. The End Goodbye and thank you for committing your patches... Want to try the IPTV-probe? Please contact me: [email_address] 81.161.128/0/18 195.135.216.0/22 87.72.0.0/16 82.211.224.0/19
    • 74. Switch buffer size is key!
      • Problem comes from: The switch buffer size
        • Ethernet switches, are suppose to drops packets
        • 75. QoS didn't help, all data was multicast
      • From burst size
        • Directly calculate required switch buffer size
        • 76. Fixed size packets of 1358 bytes on wire
        • 77. Know you switch buffer sizes
        • 78. don't place small-buffer switches in backbone

    ×