Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Как построить видеоплатформу на 200 Гбитс / Ольховченков Вячеслав (Integros)

1,088 views

Published on

Как грамотная архитектура и правильное планирование запросов позволяет небольшим количеством серверов достичь высокой производительности при раздаче видеоконтента.
В докладе будет рассказано об опыте развития проекта видеоплатформы, о проблемах, которые возникли на пути, и как нам удается раздавать 200 Гбит меньше чем 10 серверами.

Тезисы - http://www.highload.ru/2015/abstracts/1872.html

Published in: Engineering
  • Be the first to comment

Как построить видеоплатформу на 200 Гбитс / Ольховченков Вячеслав (Integros)

  1. 1. Building 200 Gbit/s video platform Vyacheslav Olhovchenkov
  2. 2. Load profile • Peak bandwidth 210Gbit/s • Peak bandwidth per edge 40Gbit/s • Daily traffic 1PTB • Views per day 6M • RPS to ruby app: 1000 • RPD to ruby app: 60M
  3. 3. Core components • Balancers • Edge servers • Video manager • Encoders • Storage servers • Stats server • Player w/ adv.support: IMA; VPAID, VAST 2,3 • API
  4. 4. Core components (2)
  5. 5. Request flowchart
  6. 6. Balancer (architecture)
  7. 7. Balancer (features) • Geo targeting • Stats • Edge selection • Serving player iframe • Secured manifest transfer • URL signing • Advertising module
  8. 8. Balancer (configuration) • CPU: E3-1270 V2 @ 3.50GHz • RAM: 8GB • SSD • Ubuntu 14.04 • RoR, unicorn • PostgreSQL, redis
  9. 9. Balancer • 1000 RPS at peak • 75% CPU utilization
  10. 10. Balancer (tips & tricks) • PgSQL => Redis for hot data • ORM optimization joins, indexes, pure sql • Linux/nginx optimization net.core.somaxconn=65535 net.ipv4.tcp_fin_timeout = 25… • RoR optimization worker_processes = 24 timeout = 60
  11. 11. Edge — history. V1.0. 5Gbit/s • Dual E5-2620@ 2.00GHz • RAM: 32GB • 1x10Gbit Intel NIC • ZMIRROR: – 12 SATA 3TB – L2ARC (6SSD 256GB) • ZFS default settings
  12. 12. Edge (tuning 1) Reason for multiple writes? zfs set atime=off tank 8Gbit/s
  13. 13. Edge (tuning 2) • HDS, HLS: segment size 300-500K • Too many files • Prefetch isn’t working Concatenate segments Second NIC 12 Gbit/s
  14. 14. HDD performance total = seek + rotate + transfer Seek: 9ms (datasheet) Rotate: 7200rpm, 4.17 ms half turn Transfer: BlockSize/160MB/s Total: 13ms + transfer time 128KB: 0.8 ms => 71.4 IOPS, 9.1MB/s 1MB: 6.4ms => 53.8 IOPS, 55MB/s
  15. 15. Edge (tuning 3) • zfs set recordsize=1M ZFS larger block support Matthew Ahrens • HEAD: r274673 • STABLE: r27608 • vfs.zfs.prefetch_disable=1 • Limited by CPU: 20Gbit/s
  16. 16. Edge — configuration (CPU) • Xenon E3 vs E5 E3: 32GB RAM Limit • 40Gbit => 50K clients • 1MB/client => 50GB buffers • UP E5 vs DP E5 High end => high price Less PCIe slots
  17. 17. Edge — configuration (disk) SATA average transfer: • 160MB/s linear: 40/(.160*8) = 31 SATA HDD • 50MB/s random: 40/(.050*8) = 100 SATA HDD SSD cost: • x8 expensive $/TB • x4 less size
  18. 18. Edge — configuration (final) • CPU: 2x E5-2640v3 8 core @ 2.60GHz) • RAM: 128GB • NIC: Chelsio 40Gbit • HBA: LSI2308 + onboard • HDD: SATA x18 (storage) • SSD: 477GB x 8 (cache)
  19. 19. Edge — performance • 40 Gbit • 40-50K clients • 5Mpps
  20. 20. Edge — tips & tricks • No HT • Dedicated package for apps • Dedicated package for NIC • No NUMA
  21. 21. Warm content
  22. 22. Warm content (2)
  23. 23. Warm content (3)
  24. 24. 200Gbit
  25. 25. P2P (WebRTC) • Streamroot (Fr) • Hot content: wins 67% 0. 2.25 4.5 6.75 9. 13:00 15:00 17:00 19:00 21:00 23:00 P2P CDN
  26. 26. P2P Architecture • Viewer authenticates and initializes the connection with Integros load balancer • At the same time, viewer connects to Streamroot tracker and requests peers • Tracker sends back intelligently-selected list of peers • A P2P webRTC data channel is created among viewers to exchange segments INTEGROS LOAD BALANCER STREAMROOT TRACKER PEER POOL % SERVER WebRTC WebRTC WebRTC
  27. 27. P2P Issues and Current Status • Issue Aug 2015: Significant traffic offload followed by partial playback issues • Reason: Flash-JavaScript interface: – Javascript based WebRTC module – Callback is not error resilient – Fun fact: not documented anywhere in Flash • Solution: fixed by tweaking Flash-JS interface • Next step: HTML5 first (hls.js / Dash.js) to avoid Flash associated issues
  28. 28. P2P Future • 80% Offload • More clients TODAY 2016 Chrome YES YES Firefox YES YES Opera YES YES Edge BETA YES Safari NO ? iOS SDK NO YES Android SDK NO YES STBs BETA YES
  29. 29. Links https://www.youtube.com/watch? v=tujArcDozq0 (railsclub Ruby Meetup №2) https://events.yandex.ru/lib/talks/ 2694/ (ruBSD2104) slawa@integros.com

×