Open Source Days 2010 d.6/3-2010 ComX Networks A/S by Jesper Dangaard Brouer  <jdb@comx.dk> Master of Computer Science Com...
Who am I <ul><li>Name: Jesper Dangaard Brouer </li><ul><li>Edu: Computer Science for Uni. Copenhagen </li><ul><li>Focus on...
CPAN IPTables::libiptc </li></ul><li>Patches accepted into </li><ul><li>Linux kernel, iproute2, iptables and Wireshark </l...
What will you learn? <ul><li>This talk is about  network  experiences with IPTV </li><ul><li>Not about: challenges to get ...
Part1: about trouble shooting IPTV network problems
Part 2: about developing our own IPTV monitor probe
Future plans for an IPTV  shaper </li></ul></ul>
ComX: Physical surroundings <ul><li>ComX Networks A/S: Fiber based solutions </li><ul><li>variety of services (TV, IPTV, V...
OEM customers, like “Syd Energi” </li></ul></ul></ul><ul><li>Apartment building have  Ethernet switches </li><ul><li>Port ...
CPE box in apartment performs </li><ul><li>service separation into VLANs </li></ul></ul></ul>
What is this talk NOT about <ul><li>Lot of challenges to implement multicast routing </li><ul><li>Its not straight forward...
get even the same vendor </li><ul><li>Watch out for firmware upgrades...
E.g: New firmware (on  Foundry FWSX 424 ) drops packets, MC in HW. </li></ul></ul><li>Also Remember: </li><ul><ul><li>Apar...
CPE equipment needs IGMP snooping, more Settop boxes </li></ul></ul></ul></ul>
The focus: Scalability <ul><li>What happened when more channels were added </li><ul><li>High-Definition channels
More than 120 IPTV channels
Approx 600 Mbit/s multicast traffic. </li></ul><li>What happens to a realtime service </li><ul><li>on a loaded ethernet sw...
Basics: MPEG2 TS packets <ul><li>Each IP-packet contains 7 Transport Stream Packets/Frames
The Continuity Counter is very small values 0-15 </li><ul><li>Thus, an exact drop counter is hard as wrap around occurs qu...
History: What was the problem? <ul><li>Customers were experiencing bad quality, </li><ul><li>due to  packet drops </li></u...
But dedicated links also showed significant drops </li><ul><li>60 percent loaded link should not have drops </li></ul></ul...
History: Measure the problem <ul><li>First objective:  Be able to measure the problem </li><ul><li>Used &quot;IP-probe&quo...
Really nice tool, but very  expensive! </li></ul></ul></ul><li>Work under pressure: OEM Customer mad!!! </li><ul><li>Netwo...
Desperate clear PIM cache drills </li></ul></ul></ul></ul></ul>
What caused the drops? <ul><li>Bursty traffic  from some of the IPTV streamers. </li><ul><li>Some of the &quot;cheap&quot;...
Worst: Linux boxes with VLC and old 2.6.18 kernel </li></ul><li>Linux VLC machines easily fixed </li><ul><li>by newer kern...
Slow and expensive support
Kickoff IPTV probe/shaper project based on Linux </li></ul></ul>
<ul><li>Important to enable </li><ul><li>high resolution timers and tickless OS </li><ul><li>to avoid VLC multicast bursts...
CONFIG_NO_HZ=y </li></ul><li>Watch the kernel log messages for: </li></ul>“ Switched to high resolution mode on CPU 0” or ...
Upcoming SlideShare
Loading in …5
×

Challenges and experiences with IPTV from a network point of view

2,995 views

Published on

OpenSource IPTV MPEG2-TS analyzer.

This presentation was given at OpenSourceDays 2010 (and in earlier stages of the project at Bifrost Workshop 2009 and 2010)

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,995
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
70
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • (BCU-26 currently have 2223 active equipment, cust:1176, iptables chains:7551 rules:32015) (Amager POP, Internet customers:4892 with active equipment:8615.)
  • TALK: First I&apos;ll focus on the routing performance issue I&apos;ll come back to slow rule changes later
  • Summary for 1.0.0 libpcap release, October 27, 2008. On Linux, support new tpacket frame headers (2.6.27+)
  • Drop this slide???
  • Challenges and experiences with IPTV from a network point of view

    1. 1. Open Source Days 2010 d.6/3-2010 ComX Networks A/S by Jesper Dangaard Brouer <jdb@comx.dk> Master of Computer Science ComX Networks A/S
    2. 2. Who am I <ul><li>Name: Jesper Dangaard Brouer </li><ul><li>Edu: Computer Science for Uni. Copenhagen </li><ul><li>Focus on Network, Dist. sys and OS </li></ul><li>Linux user since 1996, professional since 1998 </li><ul><li>Sysadm, Developer, Embedded </li></ul><li>OpenSource projects </li><ul><li>Author of </li><ul><li>ADSL-optimizer
    3. 3. CPAN IPTables::libiptc </li></ul><li>Patches accepted into </li><ul><li>Linux kernel, iproute2, iptables and Wireshark </li></ul></ul></ul></ul>
    4. 4. What will you learn? <ul><li>This talk is about network experiences with IPTV </li><ul><li>Not about: challenges to get multicast to work
    5. 5. Part1: about trouble shooting IPTV network problems
    6. 6. Part 2: about developing our own IPTV monitor probe
    7. 7. Future plans for an IPTV shaper </li></ul></ul>
    8. 8. ComX: Physical surroundings <ul><li>ComX Networks A/S: Fiber based solutions </li><ul><li>variety of services (TV, IPTV, VoIP, Internet) </li></ul><li>Our primary customers are </li></ul><ul><ul><ul><li>Apartment buildings with end-user relationship
    9. 9. OEM customers, like “Syd Energi” </li></ul></ul></ul><ul><li>Apartment building have Ethernet switches </li><ul><li>Port separation, “private” VLAN
    10. 10. CPE box in apartment performs </li><ul><li>service separation into VLANs </li></ul></ul></ul>
    11. 11. What is this talk NOT about <ul><li>Lot of challenges to implement multicast routing </li><ul><li>Its not straight forward, BUT its not the focus of this talk. </li></ul><li>Shortly, pitfalls to watch out for: </li><ul><li>It can be difficult to : </li><ul><li>get different vendors multicast to work together
    12. 12. get even the same vendor </li><ul><li>Watch out for firmware upgrades...
    13. 13. E.g: New firmware (on Foundry FWSX 424 ) drops packets, MC in HW. </li></ul></ul><li>Also Remember: </li><ul><ul><li>Apartment buildings Ethernet switches, proper config.
    14. 14. CPE equipment needs IGMP snooping, more Settop boxes </li></ul></ul></ul></ul>
    15. 15. The focus: Scalability <ul><li>What happened when more channels were added </li><ul><li>High-Definition channels
    16. 16. More than 120 IPTV channels
    17. 17. Approx 600 Mbit/s multicast traffic. </li></ul><li>What happens to a realtime service </li><ul><li>on a loaded ethernet switch based network? </li></ul></ul>
    18. 18. Basics: MPEG2 TS packets <ul><li>Each IP-packet contains 7 Transport Stream Packets/Frames
    19. 19. The Continuity Counter is very small values 0-15 </li><ul><li>Thus, an exact drop counter is hard as wrap around occurs quickly </li></ul></ul>
    20. 20. History: What was the problem? <ul><li>Customers were experiencing bad quality, </li><ul><li>due to packet drops </li></ul><li>Traffic had reached 600Mbit/s </li><ul><li>Drops were worse on loaded links.
    21. 21. But dedicated links also showed significant drops </li><ul><li>60 percent loaded link should not have drops </li></ul></ul><li>Finding the cause of packet drops </li><ul><li>Turned out to be a harder task than expected... </li></ul></ul>
    22. 22. History: Measure the problem <ul><li>First objective: Be able to measure the problem </li><ul><li>Used &quot;IP-probe&quot; from BridgeTech, to detect/see drops. </li><ul><li>Which is an IPTV analyzer appliance </li><ul><li>internally runs Linux and has a special FPGA chip
    23. 23. Really nice tool, but very expensive! </li></ul></ul></ul><li>Work under pressure: OEM Customer mad!!! </li><ul><li>Network Core team was getting desperate </li><ul><li>Foundry support </li><ul><li>Found issues with their PIM implementation </li><ul><li>Bad distribution over trunked links
    24. 24. Desperate clear PIM cache drills </li></ul></ul></ul></ul></ul>
    25. 25. What caused the drops? <ul><li>Bursty traffic from some of the IPTV streamers. </li><ul><li>Some of the &quot;cheap&quot; (100k DKK) streamers, bursting
    26. 26. Worst: Linux boxes with VLC and old 2.6.18 kernel </li></ul><li>Linux VLC machines easily fixed </li><ul><li>by newer kernel with high resolution timer support </li></ul><li>The commercial streamers were harder to &quot;fix&quot; </li><ul><li>Closed source
    27. 27. Slow and expensive support
    28. 28. Kickoff IPTV probe/shaper project based on Linux </li></ul></ul>
    29. 29. <ul><li>Important to enable </li><ul><li>high resolution timers and tickless OS </li><ul><li>to avoid VLC multicast bursts </li></ul></ul><li>Kernel config options </li><ul><li>CONFIG_HIGH_RES_TIMERS=y
    30. 30. CONFIG_NO_HZ=y </li></ul><li>Watch the kernel log messages for: </li></ul>“ Switched to high resolution mode on CPU 0” or “ Could not switch to high resolution mode on CPU 0” Hint slide: Enable Linux Highres
    31. 31. How did we locate the problem? <ul><li>BridgeTech probe tool, very helpful, showing the drops </li><ul><li>but could not show the burstiness directly </li></ul><li>One of the Linux based streamers, </li><ul><li>Alternating between 6 Mbit/s and 15 Mbit/s streams
    32. 32. All channels experienced more drops during 15 Mbit/s streams </li></ul><li>Detailed analysis of traffic dumps </li><ul><li>Wireshark was used to show burstiness
    33. 33. Wireshark IO-graph with resolution < 1 ms
    34. 34. Implemented drop detect our selves in Wireshark
    35. 35. (the power of open source) </li></ul></ul>
    36. 36. Wireshark IO-graph <ul><li>Menu: Statistics -> IO Graph -> Tick Interval 0.001 </li></ul>
    37. 37. Wireshark drop detection <ul><li>MPEG2 TS drop detection in Wireshark </li><ul><ul><li>Included in mainline since SVN rev. 27381 </li></ul><li>Not fast enough for realtime/continuous usage
    38. 38. Dumping 600Mbit/s traffic </li><ul><li>Standard tcpdump not fast enough </li><ul><li>WARN cause “fake” packet drops (Func: “packet_rcv”) </li></ul><li>Use tcpdump/pcap with Mem Mapped IO </li><ul><li>Use minimum libpcap version 1.0.0
    39. 39. Function: “ t packet_rcv” (See /proc/net/ptype) </li></ul></ul></ul></ul>
    40. 40. Part 1: Conclusion Watch out for bursts <ul><li>Packets bursts experiences </li><ul><li>Surprisingly bursts do cause problems on 60% loaded links
    41. 41. Non-bursty streams are affected by bursty-streams
    42. 42. On Linux: Use Highres timers and no tick
    43. 43. Know you switch buffer sizes </li><ul><li>don't place low-buffer switches in backbone </li></ul></ul></ul>
    44. 44. Part 2: Develop our own IPTV probe <ul><li>Learned </li><ul><li>Need to monitor the signal
    45. 45. Bursts can cause drops, on all channels
    46. 46. Bursts can occur per network segment </li></ul><li>Monitor each network segment </li><ul><li>Approx measurement points: 12
    47. 47. Price per IPTV-probe approx 80.000 DKK </li><ul><li>Total cost: 960.000 DKK </li></ul><li>Develop our own IPTV-probe </li></ul></ul>
    48. 48. Project plan: IPTV probe/shaper Develop our own MPEG2-TS analyser <ul><li>Milestone 1 : Measure the drops – DONE </li><ul><li>First objective: measure the problem </li></ul><li>Milestone 2 : Measure the bursts </li><ul><li>Detect potential issues, before drops occur </li></ul><li>Milestone 3 : Smooth out the bursts </li><ul><li>Solve the burst issue permanently </li></ul></ul>
    49. 49. Milestone 1: Measure drops <ul><li>Offline analysis </li><ul><li>Contribute to Wirehark, mostly prototyping </li></ul><li>Need continuous monitoring (24x7x365) </li><ul><li>Kernel module for detection (iptables)
    50. 50. Collector daemon (Perl)
    51. 51. Database design, storing events (MySQL)
    52. 52. Web based frontend (PHP)
    53. 53. Trick: use settop boxes as probes (bash) </li></ul></ul>
    54. 54. IPTV Probe architecture
    55. 55. Kernel module: Efficient! <ul><li>Developed iptables/netfilter module (mp2t) </li><ul><li>For realtime/continuous drop monitoring
    56. 56. Only uses 2% CPU with 600Mbit/s traffic
    57. 57. on a low power ATOM 330 CPU (1.6Ghz)
    58. 58. tcpdump solution used 100% and caused false drops </li></ul><li>Implemented with RCU locking </li><ul><li>for parallel processing on each CPU
    59. 59. (requires multi-queue NICs) </li></ul><li>Stats via /proc/ file </li><ul><li>Collector currently polls every 10 sec </li></ul></ul>
    60. 60. Collector <ul><li>Design </li><ul><li>Only report on changes </li></ul><li>Poll every 10 sec </li><ul><li>Compare with previous data </li></ul><li>Plan: Avoid polling </li><ul><li>kernel report activity (via netlink) </li></ul><li>Written as Perl daemon process </li></ul>
    61. 61. Web-Frontend <ul><li>Just finished the web-frontend </li><ul><li>Its not pretty, but it works!
    62. 62. PHP developers needed on the project! </li></ul></ul>
    63. 63. Web-frontend: Probe overview
    64. 64. Web-frontend: Channel view
    65. 65. Trick: Use settop box as probes <ul><li>See what the customer sees </li><ul><li>As many probes as customers
    66. 66. But only one channel at a time </li></ul><li>Trick: The settop box runs Linux </li><ul><li>local tool support asking for “sync errors”
    67. 67. install small bash script </li><ul><li>periodically poll, and submit result back central </li></ul></ul><li>Still need probes, in the network </li><ul><li>need to identify the network segment </li></ul></ul>
    68. 68. Settox boxes as probes
    69. 69. Open Source Status <ul><li>Wireshark </li><ul><li>Drop patches accepted
    70. 70. PCR-clock patches, not accepted :-( </li></ul><li>IPTV probe tool: (Is anybody interested?) </li><ul><li>Kernel module is GPL, but not released public </li><ul><li>Contact me if you want a copy... </li></ul><li>Collector + DB design will hopefully be GPL </li><ul><li>Want the same basis for frontends </li></ul><li>Frontend </li><ul><li>We are looking for PHP developers... </li></ul></ul></ul>
    71. 71. Future? <ul><li>Milestone 2 : Bursts, detect issues, before drops occur </li><ul><li>Currently down prioritized due to other tasks... </li></ul><li>Milestone 3 : Shaper, solve issue </li><ul><li>Have an implementation idea...
    72. 72. Will it ever happen? </li></ul><li>Progress through cooperation? </li><ul><li>Anybody want to cooperate? </li><ul><li>Or I am a just an one-man-army™ ? </li></ul></ul></ul>
    73. 73. The End Goodbye and thank you for committing your patches... Want to try the IPTV-probe? Please contact me: [email_address] 81.161.128/0/18 195.135.216.0/22 87.72.0.0/16 82.211.224.0/19
    74. 74. Switch buffer size is key! <ul><li>Problem comes from: The switch buffer size </li><ul><li>Ethernet switches, are suppose to drops packets
    75. 75. QoS didn't help, all data was multicast </li></ul><li>From burst size </li><ul><li>Directly calculate required switch buffer size
    76. 76. Fixed size packets of 1358 bytes on wire
    77. 77. Know you switch buffer sizes
    78. 78. don't place small-buffer switches in backbone </li></ul></ul>

    ×