Download to read offline
Uploaded byAlberto González Trastoy
Experiences from incorporating sign language in customer interactions
The document discusses the integration of sign language into customer service interactions, emphasizing the use of an open-source platform for real-time communication. It outlines various applications, challenges, and future opportunities, particularly in enhancing accessibility through technology like AI and facial recognition. The author highlights the need for better interoperability among existing communication protocols and the potential for advancements in WebRTC applications.
More Related Content
Similar to Experiences from incorporating sign language in customer interactions
Experiences from incorporating sign language in customer interactions
- 1. EXPERIENCES INCORPORATING SIGN LANGUAGE IN CUSTOMER INTERACTIONS @lbertogon AlbertoGonzalez Trastoy WebRTC.ventures
- 2. 2WebRTC.ventures 2019 It allstarts with the power of accessibility
- 3. PRESENTATION OUTLINE • Context •Applications • Challenges & Opportunities • Conclusions & Future WebRTC.ventures 2019
- 4. APPLICATIONS Real time sign languageuse cases 4WebRTC.ventures 2019
- 5. USE CASES 5WebRTC.ventures 2019 Imagesources: tentacle blog post, hearinghealthmatters.org,TikTok (@lifelaughterasl), leaningOnline.xyz
- 6. CALL CENTER SIGNLANGUAGE SUPPORT 6WebRTC.ventures 2019 Description • Open Source platform • Secure • Customizable • FCC initiative Goal Allow government organizations and private sector companies to redirect callers who use American Sign Language to interact with customer service representatives
- 7.
- 8. SOME REQUIREMENTS 8WebRTC.ventures 2019 •Improvement of the call quality • Supporting both H264 and VP8 codecs • Allow inbound WebRTC to WebRTC and SIP to WebRTC • Allow outbound WebRTC to SIP and WebRTC to WebRTC • A better control of the traffic • Call recording • Call monitoring • Multiparty calling
- 9. CHALLENGESAND OPPORTUNITIES Experiences incorporating real time signlanguage 9WebRTC.ventures 2019
- 10. CHOOSING THE RIGHTOPEN SOURCE TOOL 10WebRTC.ventures 2019 Media Server candidates with media transcoding: • Kurento • FreeSWITCH Kurento FreeSWITCH
- 11. CHOOSING THE RIGHTOPEN SOURCE TOOL 11WebRTC.ventures 2019 SIP signaling candidates: • JsSIP • Sip.js • Drachtio
- 12.
- 13.
- 14. INTEROPERABILITY 14WebRTC.ventures 2019 Many providerswith some differences: • Supported resolutions • RTCP-MUX • Codecs • ICE (trickle) • ?
- 15. SECURITY 15WebRTC.ventures 2019 kurento- Fixing in themodule dependencies Deployments SSL for all communicatio Required SIP authentication DTLS-SRTP SDES support
- 16. TESTING 16WebRTC.ventures 2019 Unit testing+ Basic built-in media server functional testing KITE interoperability testing
- 17.
- 18.
- 19. WHAT TECHNOLOGIES AREYOU WORKING ON OR PLANNING TO INTEGRATE WITH YOUR WEBRTCAPPLICATION? 71% 39% 32% 25% 21% 0% 10% 20% 30% 40% 50% 60% 70% 80% AI Embedded Systems AR Blockchain VR 19 WebRTC.ventures 2019
- 20. • Image Recognition •Speech-to-Text • Real time transcription • Sentiment analysis • AI assistant in call AI IN WEBRTC APPS 20 WebRTC.ventures 2019
- 21. NEXT STEPS Real timeASL image to text: 1. With OpenCV and TensorFlow 2. With TF PoseNet Model: Pose Detection in the 21WebRTC.ventures 2019 Project sources: https://github.com/shekit/alexa-sign-language-translator and PoseNet
- 22. CONCLUSIONS AND FUTURE MoreWebRTC based call centers due to all the benefits this provides New tech = Opportunities that will let us make interactions more WebRTC interoperability with SIP devices still is a challenge in production AI/ML in RTC applications arises most of the 22 WebRTC.ventures 2019
- 23.
Editor's Notes
- #2 The diversity of customer interactions is exploding. Customer experiences that once required dedicated devices can now run on almost any phone or laptop. I’ll share our experiences from incorporating sign language in customer interactions. Programmable telecoms is not only enabling many new and better customer experiences, its making customer interactions ever more inclusive.
- #3 If you don't live with a disability, you may not appreciate how crucial and meaningful community and accessibility are: Patient center care, customer call centers…To have the same rights for everyone we need accessibility to everyone. Aprox. 1M people deaf in US One viral TikTok is showing exactly why community + accessibility: Chrissy Marshall, a content creator living in Los Angeles, was born hard of hearing and became deaf in high school. Now, she makes content on YouTube and TikTok about her disability and American Sign Language.
- #6 accessibility is crucial in: Customer call centers, Patient center care, social media, eLearning…
- #7 Allow callers who use American Sign Language to directly interact with customer service representatives. A platform to be used by call centers of any kind Some big tech companies have worked a lot in accessibility but some governmental or medium&small companies are not capable of invest in expensive solutions that support sign language interactions
- #8 Tool to provide an ASL translator in the middle would have the following stack…
- #9 Multi party is an important part to have an ASL interpreter if needed
- #11 Although FreeSWITCH is built for SIP out of the box, we already had that functionality with the Asterisk PBX in place. The flexibility of acting as MCU or SFU and the set of functionalities available for WebRTC calls declined our decision to Kurento. (we could have built it using FS)
- #12 If we pick the kurento media server, how will we handle SIP signaling from the client to Asterisk? 1) SIP over websockets and use WebRTC to provide a fully featured SIP endpoint (Older) 2) SIP.JS is a fork of JSSIP. They use SIP over websockets and use WebRTC 3) Node.js framework for SIP server applications (Younger) Drachtio was the solution that better fit our (work in node out of the box), the issue was that was a very new module with a single dev working on it. After some discussion, the client wanted to go with JsSIP because the one dev and familiarity with the JSSIP platform
- #13 I didn’t mention it before but, why not using Asterisk only? 1) We needed to allow hot transfer or multiparty. We would be able to redirect the call to an ASL interpreter for the company using the ACE Direct platform 2) Lack of flexibility implementing features 3) Complexity. Asterisk advanced features or source code changes on it can be tricky 4) Slower page load due to moving all the logic to the DOM with JSSIP
- #14 The addition of the media server required the modification of the flow on the client side and also the implementation of a signaling server that speaks SIP. We needed to customize JsSIP to work in nodejs without using getUserMedia or trying to create RTCPeerConnections Why not Asterisk only? 1) We needed to allow hot transfer or multiparty. We would be able to redirect the call to an ASL interpreter for the company using the ACE Direct platform 2) Lack of flexibility implementing features 3) Complexity. Asterisk advanced features or source code changes on it can be tricky 4) Slower page load due to moving all the logic to the DOM with JSSIP
- #15 Interoperability is perceived to be the most difficult aspect of a SIP deployment We had challenges with different devices (softphones and hardphones) Differences in the SIP implementation, supported resolutions, RTCP Muxing supported or not, different codecs or in case of H264 different profile level ID that define it. ICE (trickle) support…
- #16 1) Some modules had security vulnerabilities (e.g: Denial of Service, DoS due to excessively large websocket message) 2)SSL 3)SIP Auth through Asterisk 4) DTLS-SRTP handled when using webrtc by default, SDES in WebRTC-to-SIP interworking scenarios
- #18 Another opportunity that brings this approach is having multiple agents and a consumer reaching through a SIP client!
- #20 Which makes sense because today is all about AI and WebRTC is just a feature or technology. One technology integrates with another and so on. And today seems we are talking a lot about AI
- #21 5 main use AI in WebRTC use cases. We have implemented solutions for all those 5 main use cases
- #22 OpenCV - reading frames from the video stream and classifying them frame-by-frame and TensorFlow For example, here I have an example using an Open Source app Sign language detection for Alexa. It is implemented in the browser but that’s a valid approach In our case since we have a media server we might opt for applying image recognition on the server… ASL is not just symbols with the hand, pose matters. A combination with PoseNet would be a more accurate approach
- #23 My bet is that we will see a big company running webrtc for more than vide calls in the browser, telepresence… for example with the amazon smart home lock is just the beginning…