In this talk, we will describe globo.com's live video stream architecture, which was used to broadcast events such as the FIFA World Cup (with peak of 500K concurrent users), Brazilian election debates (27 simultaneous streams) and BBB (10 cameras streaming 24/7 for 3 months) . Nginx is one of the main components of our platform, as we use it for content distribution, caching, authentication, and dynamic content. Besides our architecture, we will also discuss the Nginx and Operational System tuning that was required for a 19Gbps throughput in each node, the open source Cassandra driver for Nginx that we developed, and our recent efforts to migrate to nginx-rtmp. this was presented by leandro moreira and juarez bochi at nginx.conf in 09/23/15

1. Globo.com’s Live
Video Platform
for FIFA World Cup 14
09/23/15
#nginx #nginxconf

2. an amazing teamwork
Leandro
Moreira
other teamsDaniel
Martins
Flávio
Ribeiro
Juarez
Bochi
Hugo
Roque
Robson
Soares
Rafael
Borsani
Bernardo
Camilo
Lucas
Mundim
Juan
Almeida
more people

4. streaming rights for Brazil only

5. Agenda (4+ years of work)
2010 Architecture for FIFA World Cup 10
2012 Mobile support
2013 HLS adoption
2014 DVR
2014 Waiting room
2014 FIFA World Cup results
2015 Recap & Next steps

6. architecture in 2010
#nginx #nginxconf

7. {2010} in the beginning
encoderh264+aacrtmptcp
h264+aacrtmptcp
flash player
firewall
1935
1º
2º3º
4º
RTMP

8. {2010} good part
small delay (2-5s)

9. {2010} issue :: instrumentation
tail -f
kill -9
ps aux

10. {2010} issue :: scalability
$$$$$

11. {2010} issue :: scalability

12. {2010} some data from FIFA10
● ~= 285k simult. users *
● max bitrate 800 Kbps
*peak

13. mobile support
#nginx #nginxconf

14. {2012} and then multi-devices...
hls
h264+aacrtmptcp
h264+aacrtmptcp
h264+aacrtmptcp
hls

15. {2012} how does HLS work?
seg1.ts
seg2.ts
seg3.ts
seg4.ts
240p.m3u8
480p.m3u8
720p.m3u8
seg1.ts
seg2.ts
seg3.ts
seg4.ts
seg1.ts
seg2.ts
seg3.ts
seg4.ts
1 - GET /variant.m3u8 (text)
2 - GET /720p.m3u8 (text)
240p.m3u8
variant.m3u8
480p.m3u8 720p.m3u8
3 - GET /seg3.ts (video)
4 - GET /seg4.ts (video)
5 - GET /720p.m3u8 (text)
6 - GET /seg5.ts (video)
7 - GET /seg6.ts (video)
8 - ...
http
http
seg3.ts
seg4.ts
seg5.ts
seg6.ts
seg3.ts
seg4.ts
seg5.ts
seg6.ts
seg3.ts
seg4.ts
seg5.ts
seg6.ts

16. {2012} first hls solution
encoderh264+aacrtmptcp
h264+aachttptcp
flash player
2º3º
5º
4º
android
ios
1º
HTTP

17. {2012} generation and distribution
rtmp
video
playlists
files
location ~ /hls {
root "/vol/hls/";
}
dest = "/vol/hls";
http
BEFE segmenter

18. {2012} first solution architecture
BEFE LB SELB LB
auth & cache mostly cache hls generation
upstream upstream
http

19. HLS adoption
#nginx #nginxconf

20. {2012} one protocol to rule them all

21. {2013} buffering hls x rtmp
6x less buffering with HLS

22. {2013} instrumentation
Graphite
● status code
● request t.
● cache hit
● cpu usage
● memory
● buffer rate
● bitrate
time-series data
graph api
http
enqueue
dequeuemetric x t
snmp
metric x t
agents

23. {2013} sauron
Graphite

24. {2014} caching configuration
don’t forget to: proxy_cache_use_stale updating
011011100
110011101
101001011
011100111
1000
video0.ts
video0.ts expired
UPDATING
FE BE
proxy_cache_lock

25. {2014} OS fine tuning

26. {2014} OS fine tuning
19Gpbs maxconn: 70k, cpu affinity
15Gpbs maxconn: 40k, cpu affinity
12Gpbs maxconn: 25k, irqbalance
packet loss
4Gbps, no fine tuning
*each machine has 2 NIC bonded (10GB each one)

27. {2014} what we’ve got here is
Scalabilityport:80 UX Instrumentation
2G
4G
Wifi
1080p
720p
320p
Caching

28. {2014} farm size
FE: 80 nodes
24 cores 64 GB 1TB
10 Gbps 10 Gbps
BE: 6 nodes
SE: 2 nodes

29. DVR
#nginx #nginxconf

30. FIFA 2014 World Cup Brasil

31. {2014} what is DVR for hls?
/720p.m3u8
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:1
#EXTINF:5,
seg1.ts
#EXTINF:5,
seg2.ts
#EXTINF:5,
seg3.ts
/720p.m3u8
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:1
#EXTINF:5,
seg1.ts
#EXTINF:5,
seg2.ts
#EXTINF:5,
seg3.ts
#EXTINF:5,
seg4.ts
#EXTINF:5,
seg5.ts
#EXTINF:5,
seg6.ts
#EXTINF:5,
seg7.ts
#EXTINF:5,
seg8.ts
#EXTINF:5,
seg9.ts
#EXTINF:5,
seg10.ts

32. ingest
{2014} reviewing current solution
frontend
encoderh264+aacrtmptcp
h264+aachttptcp
flash player
2º3º
5º
4º
android
ios
1º

33. {2014} dvr challenges :: failover
didn’t have a failover option for DVR
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:10
#EXTINF:5,
seg10.ts
#EXTINF:5,
seg11.ts
#EXTINF:5,
seg12.ts
#EXTINF:5,
seg13.ts
#EXTINF:5,
seg14.ts
#EXTINF:5,
seg15.ts
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:1
#EXTINF:5,
seg1.ts

34. {2014} dvr challenges :: storage
2h * 60m * 60s / 5s * 4.5mb ~= 6.4GB
}1.6MB 1.1MB 0.8MB 0.5MB 0.3MB 0.1MB
4.5MB
per 5s
2 games + 4 other streams ~= 38.4GB
02:00:0000:00:00

35. {2014} redis as a datastore
ingest
embed-script: fetch and serve hls
frontend
daemon: mon file change and save hls

36. {2014} Brazil’s general election
too many simultaneous live streams

37. {2014} from redis to cassandra
ingest
embed-script: fetch and serve hls
frontend
daemon: mon file change and save hls

38. waiting room
#nginx #nginxconf

39. {2014} multihoming (anycast)
ISPx
ISPy
ISPk
SPBGP announcement
186.192.81.0/24
ISPx ISPy
ISPk
RJ
186.192.81.2
BGP announcement
186.192.81.0/24
PoP
PoP

40. {2014} user’s bandwidth
ISP X
ISP Y
20Gbps
5Gbps
IXP
ISP K
80Gbps
8Gbps

42. FIFA World Cup 2014 results
#nginx #nginxconf

43. FIFA 2014 World Cup results

44. FIFA 2014 World Cup Brazil data

45. ● > 500k simult. users (peak 1 game)
● ~= 125.000 video req/s (peak 1 game)
● > 640 Gbps (peak 1 game)
FIFA 2014 World Cup Brazil data

46. FIFA 2014 World Cup Brazil data
● ~= 1600 watched years (sum all)
● > 40M video views (sum all)
● ~= avg 6 Gbps, max 19 Gbps

47. ● ~= cpu usage max 10% (per node, FE)
● ~= avg bitrate 480 Kbps (crowded)
● ~= avg bitrate 1864 Kbps (less crowded)
FIFA 2014 World Cup Brazil data

48. Recap and next steps
#nginx #nginxconf

49. ● video streaming:
○ delivery and caching
○ geolocation (geofencing)
○ authorization / authentication
● microservices
○ playlist generation
○ waiting room
○ streams per account locking system
Nginx usage

50. is amazing
Lua was made in Brazil :)
TDD with busted
easier to develop
performance near C (lang)

51. OSS from this history
m3u8 parser
cassandra
driver
clappr
evostream
api
live_thumb
nginx-audio-track-module
BemTV
iprange

52. summary and future
2010
FIFA
2012
BBB
2014
FIFA
Elections
● hls
● http
● flash
● rtmp
● hls
● http
● dvr
2016
Olympic
Games
● dash ???
● ingest ???
● 4k ???

53. Questions?
THANK YOU!
leandromoreira.com.br
jbochi.me