Lorenzo Miniero discusses the transition from legacy telecommunications devices used for hearing-impaired emergency communication to next-generation emergency services, highlighting the limitations of traditional TDD/TTY devices. He introduces Real-Time Text (RTT) and discusses its implementation over IP via the ITU-T T.140 protocol, emphasizing the need for modern communication methods that incorporate richer media. Additionally, he explores the potential of WebRTC data channels for RTT communication, addressing compatibility challenges with existing systems.

1. Bringing real-time text to WebRTC
for NG Emergency Services
Lorenzo Miniero
@lminiero@fosstodon.org
Kamailio World
June 7th 2023,

2. Who am I?
Lorenzo Miniero
• Ph.D @ UniNA
• Chairman @ Meetecho
• Main author of Janus
Contacts and info
• lorenzo@meetecho.com
• https://fosstodon.org/@lminiero
• https://www.slideshare.net/LorenzoMiniero
• https://lminiero.bandcamp.com

3. Just a few words on Meetecho
• Co-founded in 2009 as an academic spin-off
• University research efforts brought to the market
• Completely independent from the University
• Focus on real-time multimedia applications
• Strong perspective on standardization and open source
• Several activities
• Consulting services
• Commercial support and Janus licenses
• Streaming of live events (IETF, ACM, etc.)
• Proudly brewed in sunny Napoli(*), Italy

4. ((*)
A different picture, for once!)

5. Emergency Services for the Hearing Impaired
• Ad-hoc text telephones and telecommunication devices
• Originally conceived and implemented in the 60’s
• Commonly known as TDD / TTY / TT
• “Telecommunications Device for the Deaf”
• Units made of keyboard, small screen and modem
• Messages converted to electric signals
• Phone line used to deliver messages
• Widely used in legacy Emergency Services
• e.g., to allow 911 operators to engage with hard of hearing people
• Uses phone line + batteries −→ still works in power failure

6. Emergency Services for the Hearing Impaired
• Ad-hoc text telephones and telecommunication devices
• Originally conceived and implemented in the 60’s
• Commonly known as TDD / TTY / TT
• “Telecommunications Device for the Deaf”
• Units made of keyboard, small screen and modem
• Messages converted to electric signals
• Phone line used to deliver messages
• Widely used in legacy Emergency Services
• e.g., to allow 911 operators to engage with hard of hearing people
• Uses phone line + batteries −→ still works in power failure

7. Emergency Services for the Hearing Impaired
• Ad-hoc text telephones and telecommunication devices
• Originally conceived and implemented in the 60’s
• Commonly known as TDD / TTY / TT
• “Telecommunications Device for the Deaf”
• Units made of keyboard, small screen and modem
• Messages converted to electric signals
• Phone line used to deliver messages
• Widely used in legacy Emergency Services
• e.g., to allow 911 operators to engage with hard of hearing people
• Uses phone line + batteries −→ still works in power failure

8. TDD/TTY/TT (PSTN)

9. TDD/TTY/TT (PSTN)
• Different acronyms for mostly similar things
• TDD (Telecommunications Device for the Deaf)
• TTY (TeleTYpe)
• TT (Text Telephone)
• Some important limitations
• Half-duplex (GA = GO AHEAD)
• Can’t be interrupted (garbled output)
• Limited set of “characters” (e.g., Baudot)
• No specific handshake, and no error correction
• Requires specific device

10. TDD/TTY/TT (PSTN)
• Different acronyms for mostly similar things
• TDD (Telecommunications Device for the Deaf)
• TTY (TeleTYpe)
• TT (Text Telephone)
• Some important limitations
• Half-duplex (GA = GO AHEAD)
• Can’t be interrupted (garbled output)
• Limited set of “characters” (e.g., Baudot)
• No specific handshake, and no error correction
• Requires specific device

11. Next Generation Emergency Services
• Multiple efforts in different areas
• NG9-1-1 (https://www.911.gov/issues/ng911/)
• NG112 (https://eena.org/next-generation-112/)
• Attempt to switch to modern ways of communication
• Existing NG services based on PSTN
• Internet-based communication allows for richer media, though
• Location, , photos, videos, etc.

12. Next Generation Emergency Services
• Multiple efforts in different areas
• NG9-1-1 (https://www.911.gov/issues/ng911/)
• NG112 (https://eena.org/next-generation-112/)
• Attempt to switch to modern ways of communication
• Existing NG services based on PSTN
• Internet-based communication allows for richer media, though
• Location, , photos, videos, etc.

13. Next Generation Emergency Services
• Multiple efforts in different areas
• NG9-1-1 (https://www.911.gov/issues/ng911/)
• NG112 (https://eena.org/next-generation-112/)
• Attempt to switch to modern ways of communication
• Existing NG services based on PSTN
• Internet-based communication allows for richer media, though
• Location, real-time text, photos, videos, etc.

14. Next Generation Emergency Services
• Multiple efforts in different areas
• NG9-1-1 (https://www.911.gov/issues/ng911/)
• NG112 (https://eena.org/next-generation-112/)
• Attempt to switch to modern ways of communication
• Existing NG services based on PSTN
• Internet-based communication allows for richer media, though
• Location, real-time text, photos, videos, etc.

15. Real-Time Text (RTT)
• Text transmitted instantly, as it is typed or created
• TDD/TTY work that way too
• How to do it over IP, though?
• Text over IP (ToIP)
• ITU-T T.140 (Protocol for multimedia application text conversation)
• Allows for real-time editing (e.g., backspace, rewriting)
• T.140 messages transported over RTP
• RFC 4103 (RTP Payload for Text Conversation)
• Redundancy implemented via RED (RFC 2198)

16. Real-Time Text (RTT)
• Text transmitted instantly, as it is typed or created
• TDD/TTY work that way too
• How to do it over IP, though?
• Text over IP (ToIP)
• ITU-T T.140 (Protocol for multimedia application text conversation)
• Allows for real-time editing (e.g., backspace, rewriting)
• T.140 messages transported over RTP
• RFC 4103 (RTP Payload for Text Conversation)
• Redundancy implemented via RED (RFC 2198)

17. Real-Time Text (RTT)
• Text transmitted instantly, as it is typed or created
• TDD/TTY work that way too
• How to do it over IP, though?
• Text over IP (ToIP)
• ITU-T T.140 (Protocol for multimedia application text conversation)
• Allows for real-time editing (e.g., backspace, rewriting)
• T.140 messages transported over RTP
• RFC 4103 (RTP Payload for Text Conversation)
• Redundancy implemented via RED (RFC 2198)

18. T.140
• ITU-T T.140 (Protocol for multimedia application text conversation)
• https://www.itu.int/rec/T-REC-T.140/
• Protocol based on the concept of T140blocks
• Set of characters or special codes that one party is delivering to one or more others
• Participant typing −→ typed characters bundled together in a T.140 block
• Buffering size up to sender (overhead vs. latency)
• UTF-8 used for characters, including special codes (actions)
• e.g., Byte Order Mark (BOM), backspace, etc.

19. T.140
• ITU-T T.140 (Protocol for multimedia application text conversation)
• https://www.itu.int/rec/T-REC-T.140/
• Protocol based on the concept of T140blocks
• Set of characters or special codes that one party is delivering to one or more others
• Participant typing −→ typed characters bundled together in a T.140 block
• Buffering size up to sender (overhead vs. latency)
• UTF-8 used for characters, including special codes (actions)
• e.g., Byte Order Mark (BOM), backspace, etc.

20. T.140
• ITU-T T.140 (Protocol for multimedia application text conversation)
• https://www.itu.int/rec/T-REC-T.140/
• Protocol based on the concept of T140blocks
• Set of characters or special codes that one party is delivering to one or more others
• Participant typing −→ typed characters bundled together in a T.140 block
• Buffering size up to sender (overhead vs. latency)
• UTF-8 used for characters, including special codes (actions)
• e.g., Byte Order Mark (BOM), backspace, etc.

21. T.140 over RTP
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC=0 |M| T140 PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp (1000Hz) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| T.140 encoded data |
+ +---------------+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

22. Dealing with packet loss using RED

23. Dealing with packet loss using RED

24. Dealing with packet loss using RED

25. Dealing with packet loss using RED

26. Dealing with packet loss using RED

27. T.140 over RTP (with redundancy)
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC=0 |M| "RED" PT | sequence number of primary |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp of primary encoding "P" |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| T140 PT | timestamp offset of "R" | "R" block length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| T140 PT | "R" T.140 encoded redundant data |
+-+-+-+-+-+-+-+-+ +---------------+
+ | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+
| "P" T.140 encoded primary data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

28. RTT and SIP/SDP
• As an RTP-based stream, easy to negotiate via SDP
• Needs text as media in m-line
• t140 used to advertise codec in rtpmap
• Sampling rate always 1000 (also for RED)
SDP example
m=text 11000 RTP/AVP 98 100
a=rtpmap:98 t140/1000
a=rtpmap:100 red/1000
a=fmtp:100 98/98/98

29. RTT and SIP/SDP
• As an RTP-based stream, easy to negotiate via SDP
• Needs text as media in m-line
• t140 used to advertise codec in rtpmap
• Sampling rate always 1000 (also for RED)
SDP example
m=text 11000 RTP/AVP 98 100
a=rtpmap:98 t140/1000
a=rtpmap:100 red/1000
a=fmtp:100 98/98/98

30. RTT and SIP/SDP
• As an RTP-based stream, easy to negotiate via SDP
• Needs text as media in m-line
• t140 used to advertise codec in rtpmap
• Sampling rate always 1000 (also for RED)
SDP example
m=text 11000 RTP/AVP 98 100
a=rtpmap:98 t140/1000
a=rtpmap:100 red/1000
a=fmtp:100 98/98/98

31. RTT and SIP/SDP
• As an RTP-based stream, easy to negotiate via SDP
• Needs text as media in m-line
• t140 used to advertise codec in rtpmap
• Sampling rate always 1000 (also for RED)
SDP example
m=text 11000 RTP/AVP 98 100
a=rtpmap:98 t140/1000
a=rtpmap:100 red/1000
a=fmtp:100 98/98/98

32. RTT and SIP/SDP
• As an RTP-based stream, easy to negotiate via SDP
• Needs text as media in m-line
• t140 used to advertise codec in rtpmap
• Sampling rate always 1000 (also for RED)
SDP example
m=text 11000 RTP/AVP 98 100
a=rtpmap:98 t140/1000
a=rtpmap:100 red/1000
a=fmtp:100 98/98/98

33. RTT and SIP/SDP
• As an RTP-based stream, easy to negotiate via SDP
• Needs text as media in m-line
• t140 used to advertise codec in rtpmap
• Sampling rate always 1000 (also for RED)
SDP example
m=text 11000 RTP/AVP 98 100
a=rtpmap:98 t140/1000
a=rtpmap:100 red/1000
a=fmtp:100 98/98/98

34. RTT and SIP/SDP
• As an RTP-based stream, easy to negotiate via SDP
• Needs text as media in m-line
• t140 used to advertise codec in rtpmap
• Sampling rate always 1000 (also for RED)
SDP example
m=text 11000 RTP/AVP 98 100
a=rtpmap:98 t140/1000
a=rtpmap:100 red/1000
a=fmtp:100 98/98/98

35. What about WebRTC?
• WebRTC heavily based on existing VoIP technologies
• SDP (on “steroids”) is mandatory
• RTP used for media (encrypted via DTLS-SRTP)
• That said, not all RTP media is supported
• Only m=audio and m=video supported out of the box
• No support at all for m=text, so RTT won’t work
• Only m=application media that can be used is UDP/DTLS/SCTP
• WebRTC data channels
• Generic data sent via SCTP and encapsulated in DTLS

36. What about WebRTC?
• WebRTC heavily based on existing VoIP technologies
• SDP (on “steroids”) is mandatory
• RTP used for media (encrypted via DTLS-SRTP)
• That said, not all RTP media is supported
• Only m=audio and m=video supported out of the box
• No support at all for m=text, so RTT won’t work
• Only m=application media that can be used is UDP/DTLS/SCTP
• WebRTC data channels
• Generic data sent via SCTP and encapsulated in DTLS

37. What about WebRTC?
• WebRTC heavily based on existing VoIP technologies
• SDP (on “steroids”) is mandatory
• RTP used for media (encrypted via DTLS-SRTP)
• That said, not all RTP media is supported
• Only m=audio and m=video supported out of the box
• No support at all for m=text, so RTT won’t work
• Only m=application media that can be used is UDP/DTLS/SCTP
• WebRTC data channels
• Generic data sent via SCTP and encapsulated in DTLS

38. T.140 RTT over WebRTC Data Channels
• Why not use data channels for RTT, then?
• T.140 Real-Time Text Conversation over WebRTC Data Channels (RFC 8865)
• https://www.rfc-editor.org/rfc/rfc8865.html
• Sending T.140 frames directly in data channels, not on RTP
• Data channels can be configured to be ordered and reliable
• Helps avoiding hoops needed for RTP (e.g., RED)
• A specific label can be associated with a specific RTT session
• Of course, needs gatewaying to be backwards compatible
• WebRTC endpoints can’t do RTP/RTT, and legacy ones don’t do data channels
• Something needs to “translate” (media and SDP)

39. T.140 RTT over WebRTC Data Channels
• Why not use data channels for RTT, then?
• T.140 Real-Time Text Conversation over WebRTC Data Channels (RFC 8865)
• https://www.rfc-editor.org/rfc/rfc8865.html
• Sending T.140 frames directly in data channels, not on RTP
• Data channels can be configured to be ordered and reliable
• Helps avoiding hoops needed for RTP (e.g., RED)
• A specific label can be associated with a specific RTT session
• Of course, needs gatewaying to be backwards compatible
• WebRTC endpoints can’t do RTP/RTT, and legacy ones don’t do data channels
• Something needs to “translate” (media and SDP)

40. T.140 RTT over WebRTC Data Channels
• Why not use data channels for RTT, then?
• T.140 Real-Time Text Conversation over WebRTC Data Channels (RFC 8865)
• https://www.rfc-editor.org/rfc/rfc8865.html
• Sending T.140 frames directly in data channels, not on RTP
• Data channels can be configured to be ordered and reliable
• Helps avoiding hoops needed for RTP (e.g., RED)
• A specific label can be associated with a specific RTT session
• Of course, needs gatewaying to be backwards compatible
• WebRTC endpoints can’t do RTP/RTT, and legacy ones don’t do data channels
• Something needs to “translate” (media and SDP)

41. T.140/data channels SDP usage
m=application 911 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP6 2001:db8::3
a=max-message-size:1000
a=sctp-port 5000
a=setup:actpass
a=dcmap:2 label="ACME customer service";subprotocol="t140"
a=dcsa:2 fmtp:t140 cps=20
a=dcsa:2 hlang-send:es eo
a=dcsa:2 hlang-recv:es eo

42. Prototyping T.140/WebRTC in Janus
• Created a prototype in Janus a few years ago
• Old PR: https://github.com/meetecho/janus-gateway/pull/1898 (closed)
• Blog post: https://www.meetecho.com/blog/realtime-text-sip-and-webrtc/
• New PR: https://github.com/meetecho/janus-gateway/pull/3231
• Used the SIP plugin in Janus as the necessary gateway
• Negotiates text m-lines and the text/t140 format on the SDP side
• Negotiates data channels on the WebRTC side (binary)
• T140blocks are then bridged (no translation) as they are
• T140blocks decapsulated from RTP packets −→ data channels
• Data channels −→ crafting RTP packets with T140blocks
• RED optionally supported in both directions on the RTP side

43. Prototyping T.140/WebRTC in Janus
• Created a prototype in Janus a few years ago
• Old PR: https://github.com/meetecho/janus-gateway/pull/1898 (closed)
• Blog post: https://www.meetecho.com/blog/realtime-text-sip-and-webrtc/
• New PR: https://github.com/meetecho/janus-gateway/pull/3231
• Used the SIP plugin in Janus as the necessary gateway
• Negotiates text m-lines and the text/t140 format on the SDP side
• Negotiates data channels on the WebRTC side (binary)
• T140blocks are then bridged (no translation) as they are
• T140blocks decapsulated from RTP packets −→ data channels
• Data channels −→ crafting RTP packets with T140blocks
• RED optionally supported in both directions on the RTP side

44. Prototyping T.140/WebRTC in Janus
• Created a prototype in Janus a few years ago
• Old PR: https://github.com/meetecho/janus-gateway/pull/1898 (closed)
• Blog post: https://www.meetecho.com/blog/realtime-text-sip-and-webrtc/
• New PR: https://github.com/meetecho/janus-gateway/pull/3231
• Used the SIP plugin in Janus as the necessary gateway
• Negotiates text m-lines and the text/t140 format on the SDP side
• Negotiates data channels on the WebRTC side (binary)
• T140blocks are then bridged (no translation) as they are
• T140blocks decapsulated from RTP packets −→ data channels
• Data channels −→ crafting RTP packets with T140blocks
• RED optionally supported in both directions on the RTP side

45. SIP plugin in Janus
https://janus.conf.meetecho.com/docs/sip

46. SIP plugin in Janus + RTT

47. Example: SDP offer from SIP/RTT endpoint
v=0
o=Lorenzo_Miniero 1 1 IN IP4 192.168.1.74
s=Omnitor_SDP_v1.1
c=IN IP4 192.168.1.74
t=0 0
m=text 1024 RTP/AVP 99 98
a=rtpmap:99 red/1000
a=fmtp:99 98/98/98
a=rtpmap:98 t140/1000

48. Example: SDP offer to WebRTC endpoint
v=0
o=Lorenzo_Miniero 1 1 IN IP4 192.168.1.74
s=Omnitor_SDP_v1.1 t=0 0
a=group:BUNDLE data
a=msid-semantic: WMS janus
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 192.168.1.74
a=sendrecv
a=sctp-port:5000
a=mid:0
[.. ICE/DTLS details follow..]

49. Testing: TIPcon1 (SIP/RTT)
https://www.meetecho.com/blog/realtime-text-sip-and-webrtc/

50. Testing: Janus SIP demo (WebRTC)
https://www.meetecho.com/blog/realtime-text-sip-and-webrtc/

51. T.140 conversation on the wire
https://www.meetecho.com/blog/realtime-text-sip-and-webrtc/

52. What’s next?
• First of all, revive the effort
• The original PR was closed a few months ago
• Little to no feedback, and core had changed a lot in the meanwhile
• New PR: https://github.com/meetecho/janus-gateway/pull/3231
• New revision can take advantage of some new features
• e.g., RED supported in the Janus core, now
• Maybe write a prettier and more functional UI too...
• Possibly test with something more complex than TIPcon1
• TIPcon1 good for basic functional testing, but very limited
• RTT endpoints very hard to find, especially in FOSS
• Possibly integration tests with actual RTT deployments?

53. What’s next?
• First of all, revive the effort
• The original PR was closed a few months ago
• Little to no feedback, and core had changed a lot in the meanwhile
• New PR: https://github.com/meetecho/janus-gateway/pull/3231
• New revision can take advantage of some new features
• e.g., RED supported in the Janus core, now
• Maybe write a prettier and more functional UI too...
• Possibly test with something more complex than TIPcon1
• TIPcon1 good for basic functional testing, but very limited
• RTT endpoints very hard to find, especially in FOSS
• Possibly integration tests with actual RTT deployments?

54. What’s next?
• First of all, revive the effort
• The original PR was closed a few months ago
• Little to no feedback, and core had changed a lot in the meanwhile
• New PR: https://github.com/meetecho/janus-gateway/pull/3231
• New revision can take advantage of some new features
• e.g., RED supported in the Janus core, now
• Maybe write a prettier and more functional UI too...
• Possibly test with something more complex than TIPcon1
• TIPcon1 good for basic functional testing, but very limited
• RTT endpoints very hard to find, especially in FOSS
• Possibly integration tests with actual RTT deployments?

55. What’s next?
• First of all, revive the effort
• The original PR was closed a few months ago
• Little to no feedback, and core had changed a lot in the meanwhile
• New PR: https://github.com/meetecho/janus-gateway/pull/3231
• New revision can take advantage of some new features
• e.g., RED supported in the Janus core, now
• Maybe write a prettier and more functional UI too...
• Possibly test with something more complex than TIPcon1
• TIPcon1 good for basic functional testing, but very limited
• RTT endpoints very hard to find, especially in FOSS
• Possibly integration tests with actual RTT deployments?

56. What’s next?
• First of all, revive the effort
• The original PR was closed a few months ago
• Little to no feedback, and core had changed a lot in the meanwhile
• New PR: https://github.com/meetecho/janus-gateway/pull/3231
• New revision can take advantage of some new features
• e.g., RED supported in the Janus core, now
• Maybe write a prettier and more functional UI too...
• Possibly test with something more complex than TIPcon1
• TIPcon1 good for basic functional testing, but very limited
• RTT endpoints very hard to find, especially in FOSS
• Possibly integration tests with actual RTT deployments?

57. Thanks! Questions? Comments?
Get in touch!
• https://fosstodon.org/@lminiero
• https://twitter.com/elminiero
• https://twitter.com/meetecho
• https://www.meetecho.com/blog/