Experiences from Incorporating Sign Language in Customer Interactions
•Download as PPTX, PDF•
0 likes•181 views
Alberto Gonzalez, Senior Engineer, WebRTC.ventures / AgilityFeat, TADHack Global Participant; and Arin Sime, Founder/CEO WebRTC.ventures, AgilityFeat, and UniWellness Care. The diversity of customer interactions has never been greater. Customer experiences that once required dedicated devices, and hence remained niche, can now run on almost any phone or laptop. Some of you may remember the first hack to incorporate sign language in customer service interactions from TADHack Global in 2015. Four years later, adding sign language to customer interactions is becoming mainstream. We’ll share our experiences from incorporating sign language in customer experiences. Programmable telecoms is not only enabling many new and better customer experiences, its making customer interactions ever more inclusive.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Experiences from Incorporating Sign Language in Customer Interactions
Similar to Experiences from Incorporating Sign Language in Customer Interactions (20)
More from Alan Quayle
More from Alan Quayle (20)
Recently uploaded
Recently uploaded (20)
Experiences from Incorporating Sign Language in Customer Interactions
- 1. EXPERIENCES INCORPORATING SIGN LANGUAGE IN CUSTOMER INTERACTIONS @lbertogon Alberto Gonzalez Trastoy WebRTC.ventures
- 2. 2WebRTC.ventures 2019 It all starts with the power of accessibility
- 3. PRESENTATION OUTLINE • Context • Applications • Challenges & Opportunities • Conclusions & Future WebRTC.ventures 2019
- 4. APPLICATIONS Real time sign language use cases 4WebRTC.ventures 2019
- 5. USE CASES 5WebRTC.ventures 2019 Image sources: tentacle blog post, hearinghealthmatters.org,TikTok (@lifelaughterasl), leaningOnline.xyz
- 6. CALL CENTER SIGN LANGUAGE SUPPORT 6WebRTC.ventures 2019 Description • Open Source platform • Secure • Customizable • FCC initiative Goal Allow government organizations and private sector companies to redirect callers who use American Sign Language to interact with customer service representatives
- 8. SOME REQUIREMENTS 8WebRTC.ventures 2019 • Improvement of the call quality • Supporting both H264 and VP8 codecs • Allow inbound WebRTC to WebRTC and SIP to WebRTC • Allow outbound WebRTC to SIP and WebRTC to WebRTC • A better control of the traffic • Call recording • Call monitoring • Multiparty calling
- 9. CHALLENGESAND OPPORTUNITIES Experiences incorporating real time sign language 9WebRTC.ventures 2019
- 10. CHOOSING THE RIGHT OPEN SOURCE TOOL 10WebRTC.ventures 2019 Media Server candidates with media transcoding: • Kurento • FreeSWITCH Kurento FreeSWITCH
- 11. CHOOSING THE RIGHT OPEN SOURCE TOOL 11WebRTC.ventures 2019 SIP signaling candidates: • JsSIP • Sip.js • Drachtio
- 14. INTEROPERABILITY 14WebRTC.ventures 2019 Many providers with some differences: • Supported resolutions • RTCP-MUX • Codecs • ICE (trickle) • ?
- 15. SECURITY 15WebRTC.ventures 2019 kurento- Fixing in the module dependencies Deployments SSL for all communicatio Required SIP authentication DTLS-SRTP SDES support
- 16. TESTING 16WebRTC.ventures 2019 Unit testing + Basic built-in media server functional testing KITE interoperability testing
- 19. WHAT TECHNOLOGIES ARE YOU WORKING ON OR PLANNING TO INTEGRATE WITH YOUR WEBRTCAPPLICATION? 71% 39% 32% 25% 21% 0% 10% 20% 30% 40% 50% 60% 70% 80% AI Embedded Systems AR Blockchain VR 19 WebRTC.ventures 2019
- 20. • Image Recognition • Speech-to-Text • Real time transcription • Sentiment analysis • AI assistant in call AI IN WEBRTC APPS 20 WebRTC.ventures 2019
- 21. NEXT STEPS Real time ASL image to text: 1. With OpenCV and TensorFlow 2. With TF PoseNet Model: Pose Detection in the 21WebRTC.ventures 2019 Project sources: https://github.com/shekit/alexa-sign-language-translator and PoseNet
- 22. CONCLUSIONS AND FUTURE More WebRTC based call centers due to all the benefits this provides New tech = Opportunities that will let us make interactions more WebRTC interoperability with SIP devices still is a challenge in production AI/ML in RTC applications arises most of the 22 WebRTC.ventures 2019
Editor's Notes
- The diversity of customer interactions is exploding. Customer experiences that once required dedicated devices can now run on almost any phone or laptop. I’ll share our experiences from incorporating sign language in customer interactions. Programmable telecoms is not only enabling many new and better customer experiences, its making customer interactions ever more inclusive.
- If you don't live with a disability, you may not appreciate how crucial and meaningful community and accessibility are: Patient center care, customer call centers…To have the same rights for everyone we need accessibility to everyone. Aprox. 1M people deaf in US One viral TikTok is showing exactly why community + accessibility: Chrissy Marshall, a content creator living in Los Angeles, was born hard of hearing and became deaf in high school. Now, she makes content on YouTube and TikTok about her disability and American Sign Language.
- accessibility is crucial in: Customer call centers, Patient center care, social media, eLearning…
- Allow callers who use American Sign Language to directly interact with customer service representatives. A platform to be used by call centers of any kind Some big tech companies have worked a lot in accessibility but some governmental or medium&small companies are not capable of invest in expensive solutions that support sign language interactions
- Tool to provide an ASL translator in the middle would have the following stack…
- Multi party is an important part to have an ASL interpreter if needed
- Although FreeSWITCH is built for SIP out of the box, we already had that functionality with the Asterisk PBX in place. The flexibility of acting as MCU or SFU and the set of functionalities available for WebRTC calls declined our decision to Kurento. (we could have built it using FS)
- If we pick the kurento media server, how will we handle SIP signaling from the client to Asterisk? 1) SIP over websockets and use WebRTC to provide a fully featured SIP endpoint (Older) 2) SIP.JS is a fork of JSSIP. They use SIP over websockets and use WebRTC 3) Node.js framework for SIP server applications (Younger) Drachtio was the solution that better fit our (work in node out of the box), the issue was that was a very new module with a single dev working on it. After some discussion, the client wanted to go with JsSIP because the one dev and familiarity with the JSSIP platform
- I didn’t mention it before but, why not using Asterisk only? 1) We needed to allow hot transfer or multiparty. We would be able to redirect the call to an ASL interpreter for the company using the ACE Direct platform 2) Lack of flexibility implementing features 3) Complexity. Asterisk advanced features or source code changes on it can be tricky 4) Slower page load due to moving all the logic to the DOM with JSSIP
- The addition of the media server required the modification of the flow on the client side and also the implementation of a signaling server that speaks SIP. We needed to customize JsSIP to work in nodejs without using getUserMedia or trying to create RTCPeerConnections Why not Asterisk only? 1) We needed to allow hot transfer or multiparty. We would be able to redirect the call to an ASL interpreter for the company using the ACE Direct platform 2) Lack of flexibility implementing features 3) Complexity. Asterisk advanced features or source code changes on it can be tricky 4) Slower page load due to moving all the logic to the DOM with JSSIP
- Interoperability is perceived to be the most difficult aspect of a SIP deployment We had challenges with different devices (softphones and hardphones) Differences in the SIP implementation, supported resolutions, RTCP Muxing supported or not, different codecs or in case of H264 different profile level ID that define it. ICE (trickle) support…
- 1) Some modules had security vulnerabilities (e.g: Denial of Service, DoS due to excessively large websocket message) 2)SSL 3)SIP Auth through Asterisk 4) DTLS-SRTP handled when using webrtc by default, SDES in WebRTC-to-SIP interworking scenarios
- Another opportunity that brings this approach is having multiple agents and a consumer reaching through a SIP client!
- Which makes sense because today is all about AI and WebRTC is just a feature or technology. One technology integrates with another and so on. And today seems we are talking a lot about AI
- 5 main use AI in WebRTC use cases. We have implemented solutions for all those 5 main use cases
- OpenCV - reading frames from the video stream and classifying them frame-by-frame and TensorFlow For example, here I have an example using an Open Source app Sign language detection for Alexa. It is implemented in the browser but that’s a valid approach In our case since we have a media server we might opt for applying image recognition on the server… ASL is not just symbols with the hand, pose matters. A combination with PoseNet would be a more accurate approach
- My bet is that we will see a big company running webrtc for more than vide calls in the browser, telepresence… for example with the amazon smart home lock is just the beginning…