Как грамотная архитектура и правильное планирование запросов позволяет небольшим количеством серверов достичь высокой производительности при раздаче видеоконтента. В докладе будет рассказано об опыте развития проекта видеоплатформы, о проблемах, которые возникли на пути, и как нам удается раздавать 200 Гбит меньше чем 10 серверами. Тезисы - http://www.highload.ru/2015/abstracts/1872.html

1. Building 200 Gbit/s
video platform
Vyacheslav
Olhovchenkov

2. Load profile
• Peak bandwidth 210Gbit/s
• Peak bandwidth per edge 40Gbit/s
• Daily traffic 1PTB
• Views per day 6M
• RPS to ruby app: 1000
• RPD to ruby app: 60M

3. Core components
• Balancers
• Edge servers
• Video manager
• Encoders
• Storage servers
• Stats server
• Player w/ adv.support: IMA; VPAID,
VAST 2,3
• API

4. Core components (2)

5. Request flowchart

6. Balancer (architecture)

7. Balancer (features)
• Geo targeting
• Stats
• Edge selection
• Serving player iframe
• Secured manifest transfer
• URL signing
• Advertising module

8. Balancer (configuration)
• CPU: E3-1270 V2 @ 3.50GHz
• RAM: 8GB
• SSD
• Ubuntu 14.04
• RoR, unicorn
• PostgreSQL, redis

9. Balancer
• 1000 RPS at peak
• 75% CPU utilization

10. Balancer (tips & tricks)
• PgSQL => Redis for hot data
• ORM optimization
joins, indexes, pure sql
• Linux/nginx optimization
net.core.somaxconn=65535
net.ipv4.tcp_fin_timeout = 25…
• RoR optimization
worker_processes = 24
timeout = 60

11. Edge — history. V1.0. 5Gbit/s
• Dual E5-2620@ 2.00GHz
• RAM: 32GB
• 1x10Gbit Intel NIC
• ZMIRROR:
– 12 SATA 3TB
– L2ARC (6SSD 256GB)
• ZFS default settings

12. Edge (tuning 1)
Reason for multiple writes?
zfs set atime=off tank
8Gbit/s

13. Edge (tuning 2)
• HDS, HLS: segment size 300-500K
• Too many files
• Prefetch isn’t working
Concatenate segments
Second NIC
12 Gbit/s

14. HDD performance
total = seek + rotate + transfer
Seek: 9ms (datasheet)
Rotate: 7200rpm, 4.17 ms half turn
Transfer: BlockSize/160MB/s
Total: 13ms + transfer time
128KB: 0.8 ms => 71.4 IOPS, 9.1MB/s
1MB: 6.4ms => 53.8 IOPS, 55MB/s

15. Edge (tuning 3)
• zfs set recordsize=1M
ZFS larger block support
Matthew Ahrens
• HEAD: r274673
• STABLE: r27608
• vfs.zfs.prefetch_disable=1
• Limited by CPU: 20Gbit/s

16. Edge — configuration (CPU)
• Xenon E3 vs E5
E3: 32GB RAM Limit
• 40Gbit => 50K clients
• 1MB/client => 50GB buffers
• UP E5 vs DP E5
High end => high price
Less PCIe slots

17. Edge — configuration (disk)
SATA average transfer:
• 160MB/s linear:
40/(.160*8) = 31 SATA HDD
• 50MB/s random:
40/(.050*8) = 100 SATA HDD
SSD cost:
• x8 expensive $/TB
• x4 less size

18. Edge — configuration (final)
• CPU: 2x E5-2640v3
8 core @ 2.60GHz)
• RAM: 128GB
• NIC: Chelsio 40Gbit
• HBA: LSI2308 + onboard
• HDD: SATA x18 (storage)
• SSD: 477GB x 8 (cache)

19. Edge — performance
• 40 Gbit
• 40-50K clients
• 5Mpps

20. Edge — tips & tricks
• No HT
• Dedicated package for apps
• Dedicated package for NIC
• No NUMA

21. Warm content

22. Warm content (2)

23. Warm content (3)

24. 200Gbit

25. P2P (WebRTC)
• Streamroot (Fr)
• Hot content: wins 67%
0.
2.25
4.5
6.75
9.
13:00 15:00 17:00 19:00 21:00 23:00
P2P
CDN

26. P2P Architecture
• Viewer authenticates and initializes the connection with Integros load
balancer
• At the same time, viewer connects to Streamroot tracker and requests peers
• Tracker sends back intelligently-selected list of peers
• A P2P webRTC data channel is created among viewers to exchange
segments
INTEGROS
LOAD
BALANCER
STREAMROOT
TRACKER
PEER POOL
% SERVER
WebRTC
WebRTC
WebRTC

27. P2P Issues and Current Status
• Issue Aug 2015: Significant traffic offload
followed by partial playback issues
• Reason: Flash-JavaScript interface:
– Javascript based WebRTC module
– Callback is not error resilient
– Fun fact: not documented anywhere in
Flash
• Solution: fixed by tweaking Flash-JS interface
• Next step: HTML5 first (hls.js / Dash.js) to
avoid Flash associated issues

28. P2P Future
• 80% Offload
• More clients
TODAY 2016
Chrome YES YES
Firefox YES YES
Opera YES YES
Edge BETA YES
Safari NO ?
iOS SDK NO YES
Android SDK NO YES
STBs BETA YES

29. Links
https://www.youtube.com/watch?
v=tujArcDozq0
(railsclub Ruby Meetup №2)
https://events.yandex.ru/lib/talks/
2694/
(ruBSD2104)
slawa@integros.com