1. Architecting Low Latency
Applications
Alberto Gonzalez Trastoy
WSAS 2022
Nov 15, 2022

2. @WebRTCventures
Agenda
• WebRTC, a protocol to connect them all
• Why it’s not easy to build with WebRTC
• Open Source vs CPaaS
• Basic architectures: SFUs and MCUs
• Scalability
• Optimizations
• Conclusions
Architecting Low Latency Applications!

3. WebRTC
An open framework for the web that enables Real-Time Communications (RTC)
capabilities in apps and in the browser. In 2021 WebRTC got officially standardized.
@WebRTCventures
Some companies using WebRTC Latency VS other low latency protocols

4. WebRTC in 2 steps
In theory we just need:
1) Capture/Manage media (MediaStream API)
const stream = await
navigator.mediaDevices.getUserMedia({audi
o: true, video: true});
localStream = stream;
2) Send /Receive media (RTCPeerConnection API)
pc1 = new
RTCPeerConnection(configuration);
const offer = await
pc1.createOffer(offerOptions);
await pc1.setLocalDescription(offer);
localStream.getTracks().forEach(track =>
pc1.addTrack(track, localStream));
//send offer/receive answer and
//receive incoming media
@WebRTCventures

5. But WebRTC implementations aren’t that simple…
@WebRTCventures
Many pieces are left for the
implementer to figure out and its
scalability is complex…

6. WebRTC Scaling Challenges
@WebRTCventures
Mesh network

7. WebRTC Media Servers
Media Servers will handle:
• Video/audio details
• Scaling capabilities
• Browser/Mobile support
Media
Servers
Your cloud servers
@WebRTCventures
Relationship between # client side
connections (y) and total participants (x)
- Blue line representing the connection
growth using a mesh network
- Yellow and red lines representing other
popular media server architectures

8. Open Source Media Servers
Media
Servers
Your cloud servers
janus.conf.meetecho.com
jitsi.org
@WebRTCventures
Popular examples:
Pion.ly
mediasoup.org
livekit.io

9. Types of WebRTC Media Servers: MCUs
• Multipoint Control Unit
• Central server mixes all audio and video
• Each participant only gets one downloaded stream
each for audio and video
• Heavy processing required on MCU, but more
predictable bandwidth requirements
Media Servers offering MCU capability (not a comprehensive list):
M
C
U
@WebRTCventures

10. Types of WebRTC Media Servers: SFUs
• Selective Forwarding Unit
• Routes the correct stream to each user
• More powerful and more modern option but
more complicated implementation
• Lower CPU requirements but more variable
bandwidth required (based on # of users)
• Possible to do end-to-end encryption
Media Servers offering SFU capability (not a comprehensive list):
S
F
U
@WebRTCventures

11. Scaling beyond single media server
Depends on the use case… What happens if we have 1000+ viewers?
For large broadcasting applications:
@WebRTCventures
S
F
U
S
F
U
S
F
U

12. Scaling beyond single media server
For large multiparty video conferencing applications:
@WebRTCventures
S
F
U
S
F
U

13. Orchestration and Containers in WebRTC applications to
Achieve Horizontal Scalability
Challenges
• Complex and coupled media server and application logic
• Stateful system complexities
• Autoscaling
• Overprovisioning
•
@WebRTCventures

14. WebRTC Scalability Autoscaling Rules
Planning your autoscaling rules
• Connections threshold for autoscaling
○ More accurate than CPU/bandwidth
• Maximum number of sessions/rooms per server
• Maximum users per room
○ To make sure we can predict
• Backup resources ready for spikes:
○ 1, 2 or even 10 servers ready?
@WebRTCventures
Example of users joining media servers at a different pace

15. WebRTC scalability, stickiness and persistence
Sticky Sessions
• We need all users in a call to use the same media server
• Generally needs additional app logic build to distribute traffic
accordingly
• Some approaches:
○ Cookie based load balanced sticky sessions
○ Direct routing through initial auth
Data Persistence
• All servers need to be aware of the current situation of the
connections
• DB or Cache based storage systems can be used for storing
sessions information and distribute traffic
• PubSub mechanisms can be a good addition to decouple and
scale independently
@WebRTCventures
Basic WebRTC Scalability and High
Availability Architecture

16. Application and Media Optimizations
What can you do?
• Simulcast or SVC
• Audio detection
• Adaptive bitrate based on resolution
• Opus RED and DTX (Discontinuous Transmission)
@WebRTCventures
WebRTC SFU
SVC example
WebRTC SFU
Audio #1
Audio #2
Audio #4
Audio #3
A#3
A#2
A#1
Receiving Opus RED (Redundant Audio Data) example
Missing
packet

17. Future Application and Media Optimizations
What will be recommended soon?
• AV1 video codec*
• Lyra V2 audio codec*
• Other ML optimizations (e.g: Noise Reduction or
packet loss concealment)
@WebRTCventures
*It is possible to use it but performance encoding is not great due to average hardware not being ready and some
browsers and devices don’t support it yet
Lyra v2 Google open source results:
https://opensource.googleblog.com/2022/09/lyra-v2-a-better-faster-and-more-versatile-speech-codec.html

18. Summary
• Focus on your specific use case needs. Review if you really need to use low latency streaming.
• Don’t underestimate the importance of a efficient scalable and high available WebRTC infrastructure
• Decide where you are willing to compromise
• Optimize and leverage the latest WebRTC capabilities
@WebRTCventures

19. Thank you!
Learn more about us:
https://webrtc.ventures
Follow us on Twitter:
@WebRTCventures
@lbertogon
Now you hold the Ring of Power
Questions?
Experts in live video app development for:
Telehealth, Broadcasting, Contact Centers, and More!