1.
WebRTC: Mostly the video bits

2.
MY BACKGROUND

3.
1
Pion
Collection of Go RTC libraries
- Building blocks to build your vision
- Designed to be flexible
- Community Owned/Non-Commercial
PeerConnection API in Pure Go
- CreateOffer/AddTrack/….
- User controls media pipeline
- SettingEngine
- Interceptors

4.
2
Pion
TURN as an API in Pure Go
- Callbacks for authentication
- Bring your own logger
- Run TURN and HTTPS on same port
- Embed TURN in application
Ion
- Cluster based system for building RTC
- SFU Instance
- AVP Instance (Processing real-time media)
- LIVE Instance (SIP/RTMP/HLS Endpoints)

5.
3
WebRTC for the Curious
Book on how WebRTC really works
- Not just about the public APIs!
- Deep dive on protocols
History of WebRTC
- Interviews with RFC authors
WebRTC in practice
- Debugging every piece
- Teach the sharp edges

6.
REAL WORLD CONSTRAINTS

7.
4
Everyone notices quality of service
Good UI, Easy signup everything looks good!
Then people get frustrated
- Video freezes or is corrupted
- Audio drops out
- Audio/Video becomes desynchronized
- Quality fluctuates rapidly
- Disconnected….
RTC is the eternal struggle of quality vs latency

8.
5
The real world is cruel
Video is huge
- Uncompressed 1280x720 8-Bit needs ~80 MB/s
Codecs
- Compressed video needs careful transport
- Support differs everywhere
Network
- Congestion
- Errors

9.
7
Video Compression
Compression can be grouped into two categories
- Intraframe (Compression in single frame)
- Interframe (Compression across many frames)
Interframe types
- Intra-code Frame (Keyframe)
- Predicted Frame
- Bi-Directional Predicted Frame
Lots of knobs here
- Disable bframes
- Keyframe interval

10.
8

11.
9
Real World Networks
Congestion
- Can manifest as loss or latency
- Every network hop can have different behaviors
Jitter
Networks are dynamic
Errors can happen at every step
- Wifi/Cellular
- Gateways

12.
HOW DOES RTC SOLVE IT

13.
10
WebRTC Media Transport
Over UDP
RTP (Media Transport)
- SSRC (Session Identifier)
- Sequence Number (Unique ID, rolls over)
- Timestamp (Relative timestamp, random start)
- Extensions
RTCP (Control Messages)
- Flexible, but a common subset is agreed upon

14.
11
Fixing Congestion
Determine available bandwidth and adjust
- Bitrate
- Resolution
- Framerate
Common Congestion Controller Algorithms
- Google Congestion Controller
- NADA
- SCreAM
All this is being actively discussed in RMCAT[0]
[0] https://datatracker.ietf.org/wg/rmcat/documents/

15.
12
Congestion Controller Flow

16.
13
Fixing Jitter
If we play as they arrive we will have stuttering
JitterBuffer
- Trade-off between latency and quality
- This concept will be re-used in the future

17.
14
Fixing Audio/Video Sync
Buffering/Network messes up your timing
Sender Reports to the rescue!
- Map each SSRC Relative Timestamp to NTP
- Then you can join each Relative Timestamp
- Also contains other useful data….

18.
15
Fixing Loss
Some loss is unavoidable, even without congestion
- Wifi
- Cellular
Not all loss is equal
- Do you care about a P-Frame before a I-Frame?

19.
16
Fixing Loss By Reacting
RTCP Negative Acknowledgments (NACK)
- SequenceNumber `I` of SSRC `J` was lost
- Recover small part of video frame
RTCP Picture Loss Indication (PLI)
- SSRC `J` is broken
- In unrecoverable state, request new keyframe
Re-use that JitterBuffer and add a delay to wait

20.
17
Fixing Loss preemptively
Forward Error Correction (FEC)
- Redundant Data is sent in another stream
- Is dynamic/protects the important bits
Less RTT then NACK, but wasting bandwidth
Still in flux, not available everywhere

21.
18
Fixing Loss After Decode
More common with audio
- Play the frame before the missing one
- Use Model-Based methods to extrapolate
Never done this myself!

22.
19
WebRTC Servers
Simulcast
- Send multiple video feeds (high, medium, low)
- Server then distributes the feed to each viewer
Scalable Video Coding
- Layered coding
- Not as common/Browser Support not great

23.
20
My Time with RTP
It's possible to ship without all this
- Don't let perfect be enemy of the good
- Do a high keyframe interval and ship
Always something you can improve
Moving any knob can effect the others
- Congestion equals more NACKs,
- More NACKs equals more congestion....
Lab/Demos are fun, but not very useful
- Network simulators are deterministic
- User metrics are what really matter

24.
OPEN SOURCE SHOWCASE

25.
ns-remote
21

26.
kerberos.io
22

27.
CloudRetro.io
23

28.
24
Neko
github.com/nurdism/neko

29.
25
TelloGo

30.
26
ascii

31.
27
cloud-morph

32.
28
ssh-p2p

33.
29
Snowflake

34.
webwormhole.io
30

35.
s4y/space
31

36.
Project
Lightspeed
32

37.
33
github.com/pion
pion.ly/slack
twitter.com/_pion
sean@pion.ly