Hookflash Chief Architect, W3C ORTC Chair, Robin Raymond overview of challenges and solutions to delivering RTC (real-time communications) integrations on Mobile devices.

1. WebRTC, Mobility, Cloud, and More...
IIT REAL-TIME COMMUNICATIONS
Conference & Expo Sept 30 - Oct 2, 2014 Chicago
Delivering Real-Time
Communications with Mobile
Presented by Robin Raymond [http://about.me/robinraymond]
Chief Architect
Hookflash.com / OpenPeer.org
2014-10-01 9:00am WEST: Alumni Lounge

2. Why should I care about mobility?
Digital time spent on mobile apps exceeds the desktop:
Source: http://www.comscore.com/Insights/Blog/Major-Mobile-Milestones-in-May-Apps-Now-Drive-Half-of-All-Time-Spent-on-Digital
Mobile matters

3. Why should I care about mobility?
Mobile is where RTC is used most:
Source: http://www.comscore.com/Insights/Blog/Major-Mobile-Milestones-in-May-Apps-Now-Drive-Half-of-All-Time-Spent-on-Digital
Mobile matters

4. What's different about mobile RTC?
A mobile device is always with us…
but what makes mobile RTC so different than desktop RTC?
Mobility differences

5. What's different about mobile RTC?
Slower
CPUs
Limited
Battery Life
Mobility differences
Mobile + Wireless
Connectivity
Limited
Storage /
Memory
Small differences. Huge impact.
Screen
Sizes /
Rotation

6. CPUs Limitations
CPU limitations

7. Slower CPU and encoding/decoding
The CPU is limited. Encoding, decoding,
capturing and rendering audio/video consume the
CPU's resources.
CPU usage also impacts battery life.
CPU limitations

8. Solution #1: Some assembly required
For heavily used encoding, decoding, rendering, scaling and
other routines, some assembly code should be expected.
Expect to have to do this
optimization assembly work or find
a library that does it for you.
CPU limitations

9. Solution #2: Hardware encoders
Some mobile devices have hardware encoders/decoders
either in the CPU or on the camera.
But there are problems:
● Encoders/decoders are not ubiquitous
● Not always real-time friendly (buffered)
● Not always 100% compatible with other encoders/decoders
● May only support a limited number of simultaneous streams
● Other applications may have the hardware busy
CPU limitations

10. Solution #3: On-demand high resolution
Send lower resolution/bitrate streams for media not in focus (thus
conferencing signalling needs to be made mobile friendly)!
In-focus. Sent with high
resolution/frame rate.
CPU limitations
Not-in-focus sent with low
resolution / low bitrate.

11. Battery Life
Battery life

12. Is there anyone out there?
Mobile RTC clients need to listen for incoming RTC
connection requests from a remote clients
(typically via a server):
I AM HERE
Battery life
OK
Is someone there?
Yes, there is...

13. Maintaining a Connection Consumes Battery Life
Wifi/WWan antennas need a fair amount of power to run.
I AM HERE
Battery life
OK
Keeping this connection active to a
server consumes battery life.

14. Solution #1 - Specialized "push friendly" servers
are required
Be sure to use push friendly protocol services for mobile or
your communication will break down.
Battery life
When a RTC client application goes to the background the
connection to the server will typically be killed. The server
will assume the client is gone.
I AM HERE
Is someone there?
[X] Not any more...

15. Solution #1 - Register for push
Most mobile devices offer a "push" service. Register for a
push instead of keeping a server connection alive.
PUSH TO ME
Battery life
OK
REGISTER PUSH
Push service
Temporary connection

16. Solution #1 - Send push upon incoming request
Incoming connections cause a push to happen.
Battery life
SEND PUSH
Push service
PUSH NOTICE
Server needs to "hold" incoming connection
request while push is delivered.
OK, I AM BACK
Is someone there?

17. Solution #2 - Application as a service
Ideally have the server ping the client to see if it's alive so the
client application can go to "sleep" otherwise.
Battery life
Is someone there?
Yes, they are here
Registering the connection as a "VoIP/RTC" connection and/or
registering the application as a background service will allow
the connection to be processed when data is received.
It's not as efficient as push but it is better.
STILL THERE?
YES

18. Backgrounding and Messaging
When an application goes to sleep in the
background to save battery it can't accept
receiving directed messages anymore.
Battery life
Hi Bob!
Zzzz...

19. Backgrounding and Messaging
Protocols that "retry" sending message until they
get delivered do not work when applications are
sleeping.
Battery life
Hi Bob!
Hi Bob!
Hi Bob!
*FAILURE*
Zzzz...

20. Solution #1: Server Store + Forward
Servers need to store messages until the
receiving application wakes up.
Battery life
Hi Bob!
Zzzz...
Hi Bob!
*AWAKE*
*CONNECTED*
Some time later...

21. Solution #2: Peer-to-peer Synchronize
Messages are stored locally and synchronized
once applications are both awake and connected.
Battery life
- Hi Bob!
- How are you?
Zzzz...
Hi Bob!
*AWAKE*
*CONNECTED*
Some time later...
How are you?

22. Mobility + Wireless Connectivity
Mobility + wireless connectivity

23. Mobile 4G/3G and IPv4 Address Exhaustion
With millions of new mobile devices, mobile providers are sharing a single IPv4 for
many devices. There aren't enough IPv4 addresses for each mobile device online.
Mobility + wireless connectivity
Mobile Provider
Firewall
IP=2.3.4.5
IP=2.3.4.5
IP=2.3.4.5

24. Mobile 4G/3G and IPv4 Address Exhaustion
For efficient usage of IPv4, typically symmetric NAT firewalls are used. Every local
port gets mapped to a different firewall port per connection destination.
Mobility + wireless connectivity
2.3.4.5
Mobile Provider
Firewall
10.0.0.80
5000
7542
32812
99.98.97.96
15.16.17.18

25. Solution #1: Mobile 4G/3G requires TURN relay
Symmetric NAT firewalls aren't friendly for direct media connections and TURN
relays are often required for wwan mobile for IPv4.
2.3.4.5
Mobile Provider
Firewall
10.0.0.80
5000
7542
32812
Mobility + wireless connectivity
99.98.97.96
Media Relay
TURN=5.6.7.8

26. Solution #2: Mobile 4G/3G and IPv6
An inexhaustible supply of IPv6 addresses exist so applications should take
advantage of mobile devices and providers that support IPv6!
Mobile provider firewalls don't need to share IPs or do NAT port mappings. They
only act as a filter to block bad traffic thus TURN relays can be avoided.
Mobile Provider Filter
FE80::0202:
B3FF:FE1E:
8329
5000 5000
Mobility + wireless connectivity
2607:f0d0:
1002:51::4

27. FEC and Wireless Packet Loss
FEC = Forward Error Correction
A/V packets can include forward/backward predictive
information so lost packets can be reconstructed.
Mobility + wireless connectivity

28. But wireless loss is often bursty
What do we do in this situation?
Due to wireless buffering overflow, mobile packet loss often
happens in bursts. Single frame FEC does not work well.
Mobility + wireless connectivity

29. Solution #1 - Scalable Video Codecs
For video, a base layer video frame can be transmitted with
enhancement video resolutions/frames. Loss of enhancement
frames are non critical.
Loss of video enhancements
frames will decrease quality but
the overall video transmission is
maintained.
Mobility + wireless connectivity

30. Solution #1 - Scalable Video Codecs
NACK (Negative Acknowledgement) / RTX (Retransmission) /
other schemes are used to protect the base layer.
Lost base layer packets are
protected with NACK / RTX and
enhancements frames reference
protected base layer frame.
Mobility + wireless connectivity

31. Solution #2 - Audio Burst FEC
Burst audio loss requires a burst capable audio
codec and/or multi-packet RTP FEC modes.
In some conditions, retransmitting the same
packet multiple times increases the probability of
burst packet loss but not always.
Mobility + wireless connectivity

32. Solution #2 - Audio Burst FEC
Burst audio loss is an ongoing field of study and
there are proposals but this is not a heavily
standardized area.
See also:
http://research.google.com/pubs/pub41611.html
https://github.com/vr000m/payload-1d2d-parity-fec/blob/master/1d2d/draft-singh-payload-1d2d-parity-scheme.xml
http://tools.ietf.org/html/rfc5956
http://tools.ietf.org/html/draft-lennox-payload-ulp-ssrc-mux
Mobility + wireless connectivity

33. Limited Storage / Memory
Limited storage / memory

34. Recording and Storing Audio/Video
Locally storing raw audio/video frames on a
mobile device is not practical due to the limited
storage space.
Limited storage / memory

35. Recording and Storing Audio/Video
Encoding video locally on the mobile device
consumes the mobile CPU, meaning less CPU is
available for RTC.
Limited storage / memory

36. Solution #1: Server Encoding
Relay all media through a server and have the
server do the heavy encoding work.
Limited storage / memory
Increased latency
due to relay and
increased media cost
for hosting.

37. Solution #2: Local Encoding
Audio/Video is encoded locally and optionally
upload to a server.
Limited storage / memory
Optional to upload audio/video on
a server.
To not impact RTC bandwidth,
upload after the video recording is
complete and not in real-time.

38. Solution #2: Local Encoding Offloaded
Using mobile hardware encoders makes more sense here
since playback formats are often compatible within the device.
Hardware video encoders work well since they
need not be fully real-time and the encoding
format need not be compatible with the remote
device.
Limited storage / memory

39. Screen Sizes / Rotation
Screen sizes and rotation

40. Camera rotation / aspect ratio
Camera rotation and multi angle camera (e.g. back vs front) selection affects the
capture frame rate, aspect ratio, and scalings scalings.
Screen sizes and rotation
Don't waste bits by needless encoding and
sending frame rates and sizes beyond the
remote party's capability to render.

41. Dynamic Sizing of Render
Local and remote mobile parties can both rotate. The capture side need dynamic
updates of the render's current output capabilities. The sender should maximize
output by not needlessly encoding beyond the capabilities of the render.
NOTIFY RENDER SIZE=X,Y
NOTIFY RENDER SIZE=Y,X
Screen sizes and rotation
Signalling protocol
needs to update
sender constantly of
changing conditions.

42. Letterboxing / Pillarboxing vs Stretching
The only thing worse than black letterbox / pillarbox bars are stretched images.
Don't stretch images!
Letterbox
Screen sizes and rotation
Pillarbox

43. Letterboxing / Pillarboxing vs Cropping
Cropping can work as a solution to fill unused areas but must be done within care
and in known use cases.
Image loss = OK
Screen sizes and rotation
Image loss = NOT
OK

44. Mobile Wrap Up
● Specialized signalling helps save battery via push
● Messaging is difficult without store and forward or synchronization
● Heavily encoding / decoding optimization is needed
● Hardware encoders / decoders aren't ubiquitous across devices
● WWAN often requires TURN or IPv6 to work
● Scalable Video Codecs with base layer protection help with burst loss
● Continuous signalling updates of render capability saves CPU and
bandwidth
● Record on a server or passively encode with remaining CPU or via
onboard hardware encoders
● Choose image letterbox/pillarbox or clipping, and not stretching
Wrap up

45. Thank you
Thanks to the IIT RTC Conference and Hookflash for
allowing me to present.
http://hookflash.com/
http://openpeer.org/
C++ Open Source ORTC Library (mobile friendly)
http://ortc.org/