Quality of Service is a difficult problem in WebRTC. You spend hours perfecting your design in the lab. You test in different network conditions and try your best. A week later the reports trickle in of different customers having unreproducable issues. Then when you go to research you can't even find consensus on how to solve these problems. Most companies don’t discuss how they do it, and those that do don’t agree. This talk is about my time building media experiences with WebRTC. The lessons I have learned, and mistakes along the way. This talk doesn’t discuss signaling, connectivity or security. If you are interested in those topics check out WebRTC for the Curious.

1. WebRTC: Mostly the video bits

2. MY BACKGROUND

3. 1
Pion
Collection of Go RTC libraries
- Building blocks to build your vision
- Designed to be flexible
- Community Owned/Non-Commercial
PeerConnection API in Pure Go
- CreateOffer/AddTrack/….
- User controls media pipeline
- SettingEngine
- Interceptors

4. 2
Pion
TURN as an API in Pure Go
- Callbacks for authentication
- Bring your own logger
- Run TURN and HTTPS on same port
- Embed TURN in application
Ion
- Cluster based system for building RTC
- SFU Instance
- AVP Instance (Processing real-time media)
- LIVE Instance (SIP/RTMP/HLS Endpoints)

5. 3
WebRTC for the Curious
Book on how WebRTC really works
- Not just about the public APIs!
- Deep dive on protocols
History of WebRTC
- Interviews with RFC authors
WebRTC in practice
- Debugging every piece
- Teach the sharp edges

6. REAL WORLD CONSTRAINTS

7. 4
Everyone notices quality of service
Good UI, Easy signup everything looks good!
Then people get frustrated
- Video freezes or is corrupted
- Audio drops out
- Audio/Video becomes desynchronized
- Quality fluctuates rapidly
- Disconnected….
RTC is the eternal struggle of quality vs latency

8. 5
The real world is cruel
Video is huge
- Uncompressed 1280x720 8-Bit needs ~80 MB/s
Codecs
- Compressed video needs careful transport
- Support differs everywhere
Network
- Congestion
- Errors

9. 7
Video Compression
Compression can be grouped into two categories
- Intraframe (Compression in single frame)
- Interframe (Compression across many frames)
Interframe types
- Intra-code Frame (Keyframe)
- Predicted Frame
- Bi-Directional Predicted Frame
Lots of knobs here
- Disable bframes
- Keyframe interval

10. 8

11. 9
Real World Networks
Congestion
- Can manifest as loss or latency
- Every network hop can have different behaviors
Jitter
Networks are dynamic
Errors can happen at every step
- Wifi/Cellular
- Gateways

12. HOW DOES RTC SOLVE IT

13. 10
WebRTC Media Transport
Over UDP
RTP (Media Transport)
- SSRC (Session Identifier)
- Sequence Number (Unique ID, rolls over)
- Timestamp (Relative timestamp, random start)
- Extensions
RTCP (Control Messages)
- Flexible, but a common subset is agreed upon

14. 11
Fixing Congestion
Determine available bandwidth and adjust
- Bitrate
- Resolution
- Framerate
Common Congestion Controller Algorithms
- Google Congestion Controller
- NADA
- SCreAM
All this is being actively discussed in RMCAT[0]
[0] https://datatracker.ietf.org/wg/rmcat/documents/

15. 12
Congestion Controller Flow

16. 13
Fixing Jitter
If we play as they arrive we will have stuttering
JitterBuffer
- Trade-off between latency and quality
- This concept will be re-used in the future

17. 14
Fixing Audio/Video Sync
Buffering/Network messes up your timing
Sender Reports to the rescue!
- Map each SSRC Relative Timestamp to NTP
- Then you can join each Relative Timestamp
- Also contains other useful data….

18. 15
Fixing Loss
Some loss is unavoidable, even without congestion
- Wifi
- Cellular
Not all loss is equal
- Do you care about a P-Frame before a I-Frame?

19. 16
Fixing Loss By Reacting
RTCP Negative Acknowledgments (NACK)
- SequenceNumber `I` of SSRC `J` was lost
- Recover small part of video frame
RTCP Picture Loss Indication (PLI)
- SSRC `J` is broken
- In unrecoverable state, request new keyframe
Re-use that JitterBuffer and add a delay to wait

20. 17
Fixing Loss preemptively
Forward Error Correction (FEC)
- Redundant Data is sent in another stream
- Is dynamic/protects the important bits
Less RTT then NACK, but wasting bandwidth
Still in flux, not available everywhere

21. 18
Fixing Loss After Decode
More common with audio
- Play the frame before the missing one
- Use Model-Based methods to extrapolate
Never done this myself!

22. 19
WebRTC Servers
Simulcast
- Send multiple video feeds (high, medium, low)
- Server then distributes the feed to each viewer
Scalable Video Coding
- Layered coding
- Not as common/Browser Support not great

23. 20
My Time with RTP
It's possible to ship without all this
- Don't let perfect be enemy of the good
- Do a high keyframe interval and ship
Always something you can improve
Moving any knob can effect the others
- Congestion equals more NACKs,
- More NACKs equals more congestion....
Lab/Demos are fun, but not very useful
- Network simulators are deterministic
- User metrics are what really matter

24. OPEN SOURCE SHOWCASE

25. ns-remote
21

26. kerberos.io
22

27. CloudRetro.io
23

28. 24
Neko
github.com/nurdism/neko

29. 25
TelloGo

30. 26
ascii

31. 27
cloud-morph

32. 28
ssh-p2p

33. 29
Snowflake

34. webwormhole.io
30

35. s4y/space
31

36. Project
Lightspeed
32

37. 33
github.com/pion
pion.ly/slack
twitter.com/_pion
sean@pion.ly