The document discusses the development of premium features for the video chat platform appear.in. Key points include: - appear.in wanted to offer premium features to meet user demands but needed to improve call quality for larger groups first. - Their solution was building an in-house selective forwarding unit (SFU) to handle streaming between multiple peers, improving stability and allowing more participants. - This involved extensive testing using network emulation and traffic analysis tools to understand quality issues at different bandwidths and with varying conditions. - The SFU architecture uses a global mesh of servers to discover streams and selectively forward them between peers for higher quality group calls supporting up to 12 people.

1. The making of appear.in
premium
Dag-Inge Aas
Tech lead

2. What is appear.in?
• Web-based video chat with
no download, signup or login
• An internal, independent
startup in Telenor
• Team of 20 people from 8
nationalities

3. Agenda
• How we started creating premium
• A little about how we work
• How we went about solving our problem
• What we ended up with
appear.in/awesome-cat

4. The beginning

5. Monetizing appear.in
• Had lots of requests from people wanting to pay
• We wanted to prove to ourselves that we were worth paying for
• Some clear needs emerged
• More people in the room
• Customization options
• Better screen sharing
• Improved quality
appear.in/awesome-cat

6. Companies wanted to use us
• We saw a lot of standup rooms
• Bigger corporations wanted to buy enterprise
licenses
• Support for town hall meetings,
broadcasting scenarios

7. Defining the problem we want to solve

8. What are the users problems?
• Users want to pay for appear.in, but can’t
• There isn’t enough room for the entire team in a
single appear.in room
• The quality is unreliable and unacceptable
in group calls
appear.in/awesome-cat

9. Users want to pay for appear.in
• Luxury problem when you get these requests
• User won’t pay out of the good of their hearts

10. There isn’t enough room for the entire team
• A clear need for 8+ calls
• How many is many? When does a meeting stop being a
meeting and start being a broadcast?
• Are the people spectators
or participants?
• We can’t solve this without solving
our third problem

11. Quality is unreliable and unacceptable
in group calls
• The core issue we were facing
• No use adding more people, if we couldn’t solve this
• No use charging money, if we couldn’t
solve this
appear.in/awesome-cat

12. What is quality?
• Quality can be any number of things, is it the number
of pixels of video you send? A function of latency and
jitter?
• We define latency as the experience the user has
during a conversation.
• Does it flow naturally, does it feel real? Does it work as
if you are in the same room?

13. What is bad quality?
• Latency in itself isn’t such a huge issue, the human
brain is amazing
• Variable latency is killer, the brain can’t adjust, social
clues break down
• So-so, mm, ja. We depend on small clues to keep
conversation flowing

14. Solving the problem

15. We are peer-to-peer based
• We believed peer-to-peer was
the superior technology
• Really wanted to solve the
problem without going for the
“obvious”

16. How can we improve quality?
• The Internet is inherently variable
• Jitter saves us most of the time, but when it gets too
bad, things start to fail
• Chrome estimates bandwidth by saturating the link,
leading to packet loss
• PeerConnections compete in a full-mesh network

17. RTCStats
• https://github.com/opentok/rtcstats-server
• Queries on getStats() and creates aggregates. Currently at
500 features per peer connection.
• Syncs to redshift for querying big datasets
• https://medium.com/the-making-of-appear-in/webrtc-
connection-times-and-the-power-of-playing-around-with-
data-ab11312737e9#.vrqtr3mlq

18. RTCStats

19. Network link conditioner to the rescue
• Find the limits in a controlled environment
• Not always enough, link conditioner is too static, we
are looking for big events, not constant losses

20. Idea: Controlling bandwidth
• b=AS:<bitrate>
• Can be set dynamically on the receiving end
• Set bitrate based on number of people in the room

22. Partial success
• We were able to set the bandwidth, and even network-
constrained endpoints worked. After 256kbit/s it was
more or less unusable for video
• Supersize looked bad, screen sharing ruined displayed
resolution-based bitrate modification
• Huge realisation when we compared our results to an
SFU-based model

23. Solution: Selective forwarding
• Support for more people in rooms
• Handles supersize
• Lower CPU requirement
• Better stability and flow in group calls
• It costs a lot more money

25. Our home-grown SFU
• Written in JavaScript with some C-bindings
• First working version was only 3000 lines
• Globally distributed, uses full-mesh for
discoverability of streams on different hosts
• Multiple servers in each AWS region

26. Advantages of building your own
• Full control and understanding of the component
• Can build custom scaling, simulcast rules
• Simulcast based on both network and displayed
resolution
• “Control your own core technology”

27. Disadvantages of building your own
• You have to build it all yourself
• There is a whole bunch of issues you didn’t think
about beforehand
• Learning how to count
• From first prototype to first production use took us
10 months

28. Architecture
• Each SFU provides signalling
• Every user connects to their closest SFU using latency-
based DNS (sfu.appear.in)
• Global mesh network for peer discovery on publish/
subscribe
• SFU requests stream, forwarded through Amazon network
to closest exit node

29. AWS internal network
eu-west-1 us-east-1
SFU 1 SFU 2
BobAlice
sfu.appear.in -> 84.0.0.1 sfu.appear.in -> 78.0.0.1

30. Premium

31. appear.in/awesome-cat
appear.in premium
Upgrade your room to get extra features:
• Up to 12 people in conversation
• Better quality and stability
• Improved screen sharing
• Branding of room
$12 per room/month

32. appear.in/awesome-cat
Try for free!
Use coupon code:
WEBRTC-JP
for one month free

33. ありがとうございます !