Building voice interfaces with JavaScript & Web APIs

•Download as PPTX, PDF•

2 likes•116 views

How do you build a voice interface device, like Google Home or Amazon Alexa, using only a RaspberryPi, Javascript and some web APIs? Talk given at 2017 GeekTime TechFest Code

Technology

Building a voice interface–
With JavaScript!

Hello!
I am Gal Hochberg
I’m an entrepreneur and a consultant
I ♥️ AI, NLP and conversational interfaces
Nadav Leshem
Co-Founder &
CTO, Jolt
Nir Singher
Software
Engineer,
Trusteer

“
I’m sorry Dave, I’m afraid I can’t do that
HAL 9000

Why not use
Alexa/Siri/Google?
◇ Interaction requires specific phrasing
◇ Authoring requires coding expertise
◇ “Request-Response” only

System Architecture
Physical
Device
Voice Server Authoring Interface

82.9GBBigger than your 3G plan
1036$Could’ve hired an intern instead
One month of listening…

◇ Device-local voice and hotword detection
◇ Supports custom keywords
◇ Has NodeJS bindings
◇ Runs on RaspberryPi
◇ https://snowboy.kitt.ai/

“hello” “how are you?”
<asked_hello>
Server Processing

Off-the-shelf solutions
◇Api.ai (Google)
◇Wit.ai (Facebook)
◇LUIS (Microsoft)
◇Lex (Amazon)
◇Botsify
◇Motion.ai
◇Pullstring
◇…..
◇ Intent based, not conversation based
◇ Hard to use for non-techies (And for techies too!)
◇ We didn’t need their wealth of integrations

“hello” “how are you?”
<asked_hello>
Server Processing
Speech to Text Text to Speech
Natural Language Processing

Speech to Text
◇ Not intent based
◇ Realtime results
◇ Accurate in noisy environments
◇ Easy NodeJS Bindings

NLP
◇ Rule based
◇ Intent/Slot/Entity based
◇ Statistical

Rule based NLP
◇ As long as you control the conversation, you know where its going to go
◇ You’re mostly looking for keywords
◇ Context is usually just a “state”
◇ Catchalls are important
◇ Its really hard to guess what people will say
◇ Iterate, Iterate, Iterate!

Telemetry
◇ Tracking your devices
◇ Remote admin and configuration
◇ Server-triggered conversations

Thanks!
Any questions?
You can find me at:
◇ hochbergg@gmail.com
◇ Gal Hochberg (FB)
◇ @SoftFlare

Similar to Building voice interfaces with JavaScript & Web APIs

Ruby on Google App Engine: Upgrade to Google App "Turbo" Engine

Joseph Ku

Why we are still writing?

Alexander Osin

Real speaker usa

RealSpeaker 2.0

RealSpeaker usa

RealSpeaker 2.0

Mobile cross platform development with Dojo

Nikolai Onken

HTML5 - Let’s make the WEB more powerful

Naga Harish M

DevOps Columbus Meetup Kickoff - Infrastructure as Code

Michael Ducy

Couch DB/PouchDB approach for hybrid mobile applications

Ihor Malytskyi

OK Google, it's time to bot! - Hadar Franco & Stav Levi

Hadar Franco

Ok google, it's time to bot! - Hadar Franco, Albert + Stav Levi, Monday

DroidConTLV

Hey hubballi! - Talk on "Actions on Google" #DevFestHubali

Smit Jethwa

HTML5 is the Future of Mobile, PhoneGap Takes You There Today

davyjones

Groove Readi Depth

ASaikovski

Pipeline your pipelines!

Giulio Vian

DevDays 2011- Let’s get ready for the cloud: Building your applications so th...

Robert MacLean

Marmalade: bittersweet experience

Max Klyga

解密解密

Tom Chen

Hw09 Cloudera Desktop In Detail

Cloudera, Inc.

Designing complex interactions for experiences that target XR headsets can be challenging due to the limited input schemes. Advances in Machine Learning (ML) now allow developers to easily leverage Natural Language Understanding through reusable techniques. The combination of XR+AI is a powerful integration that opens new possibilities for both gaming, entertainment and enterprise scenarios. This intermediate session is an exploration of how speech and language understanding can be used to augment Mixed Reality & VR experiences. We'll explore the use of speech recognition & Natural Language Understanding to build advanced voice commands, translate languages from within XR environments, and also look at the creation of intelligent conversation assistants to be used as interactive entities in Mixed Reality and VR apps & games. In a world where speech is the primary form of input, using Machine Learning to process language input and understand the user's intent is of paramount importance. Nick Landry - Microsoft

Designing XR Experiences with Speech & Natural Language Understanding in Unity

Unity Technologies

Going Mobile with React Native and WebRTC

Saúl Ibarra Corretgé

Similar to Building voice interfaces with JavaScript & Web APIs (20)

Ruby on Google App Engine: Upgrade to Google App "Turbo" Engine

Why we are still writing?

Real speaker usa

RealSpeaker usa

Mobile cross platform development with Dojo

HTML5 - Let’s make the WEB more powerful

DevOps Columbus Meetup Kickoff - Infrastructure as Code

Couch DB/PouchDB approach for hybrid mobile applications

OK Google, it's time to bot! - Hadar Franco & Stav Levi

Ok google, it's time to bot! - Hadar Franco, Albert + Stav Levi, Monday

Hey hubballi! - Talk on "Actions on Google" #DevFestHubali

HTML5 is the Future of Mobile, PhoneGap Takes You There Today

Groove Readi Depth

Pipeline your pipelines!

DevDays 2011- Let’s get ready for the cloud: Building your applications so th...

Marmalade: bittersweet experience

解密解密

Hw09 Cloudera Desktop In Detail

Designing XR Experiences with Speech & Natural Language Understanding in Unity

Going Mobile with React Native and WebRTC

Recently uploaded

[BuildWithAI] Introduction to Gemini.pdf

Sandro Moreira

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Passkeys: Developing APIs to enable passwordless authentication Cody Salas, Sr Developer Advocate | Solutions Architect - Yubico Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

apidays

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Whatsapp Number Escorts Call girls 8617370543 Available 24x7 Mcleodganj Call Girls Service Offer Genuine VIP Model Escorts Call Girls in Your Budget. Mcleodganj Call Girls Service Provide Real Call Girls Number. Make Your Sexual Pleasure Memorable with Our Mcleodganj Call Girls at Affordable Price. Top VIP Escorts Call Girls, High Profile Independent Escorts Call Girls, Housewife Women Escorts Call Girl, College Girls Escorts Call Girls, Russian Escorts Call girls Service in Your Budget.

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Deepika Singh

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Angeliki Cooney has spent over twenty years at the forefront of the life sciences industry, working out of Wynantskill, NY. She is highly regarded for her dedication to advancing the development and accessibility of innovative treatments for chronic diseases, rare disorders, and cancer. Her professional journey has centered on strategic consulting for biopharmaceutical companies, facilitating digital transformation, enhancing omnichannel engagement, and refining strategic commercial practices. Angeliki's innovative contributions include pioneering several software-as-a-service (SaaS) products for the life sciences sector, earning her three patents. As the Senior Vice President of Life Sciences at Avenga, Angeliki orchestrated the firm's strategic entry into the U.S. market. Avenga, a renowned digital engineering and consulting firm, partners with significant entities in the pharmaceutical and biotechnology fields. Her leadership was instrumental in expanding Avenga's client base and establishing its presence in the competitive U.S. market.

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Angeliki Cooney

Retrieval augmented generation (RAG) is the most popular style of large language model application to emerge from 2023. The most basic style of RAG works by vectorizing your data and injecting it into a vector database like Milvus for retrieval to augment the text output generated by an LLM. This is just the beginning. One of the ways that we can extend RAG, and extend AI, is through multilingual use cases. Typical RAG is done in English using embedding models that are trained in English. In this talk, we’ll explore how RAG could work in languages other than English. We’ll explore French, Chinese, and Polish.

Introduction to Multilingual Retrieval Augmented Generation (RAG)

Zilliz

Exploring Multimodal Embeddings with Milvus

Zilliz

Elevate Developer Efficiency & build GenAI Application with Amazon Q

Bhuvaneswari Subramani

MS Copilot expands with MS Graph connectors

Nanddeep Nachan

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

MINDCTI Revenue Release Quarter One 2024

MIND CTI

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Victor Rentea

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

💥 You’re lucky! We’ve found two different (lead) developers that are willing to share their valuable lessons learned about using UiPath Document Understanding! Based on recent implementations in appealing use cases at Partou and SPIE. Don’t expect fancy videos or slide decks, but real and practical experiences that will help you with your own implementations. 📕 Topics that will be addressed: • Training the ML-model by humans: do or don't? • Rule-based versus AI extractors • Tips for finding use cases • How to start 👨‍🏫👨‍💻 Speakers: o Dion Morskieft, RPA Product Owner @Partou o Jack Klein-Schiphorst, Automation Developer @Tacstone Technology

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

UiPathCommunity

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

Recently uploaded (20)

[BuildWithAI] Introduction to Gemini.pdf

Corporate and higher education May webinar.pptx

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Strategies for Landing an Oracle DBA Job as a Fresher

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Introduction to Multilingual Retrieval Augmented Generation (RAG)

Exploring Multimodal Embeddings with Milvus

Elevate Developer Efficiency & build GenAI Application with Amazon Q

MS Copilot expands with MS Graph connectors

presentation ICT roal in 21st century education

MINDCTI Revenue Release Quarter One 2024

Why Teams call analytics are critical to your entire business

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

How to Troubleshoot Apps for the Modern Connected Worker

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

FWD Group - Insurer Innovation Award 2024

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Building voice interfaces with JavaScript & Web APIs

1. Building a voice interface– With JavaScript!

2. Hello! I am Gal Hochberg I’m an entrepreneur and a consultant I ♥️ AI, NLP and conversational interfaces Nadav Leshem Co-Founder & CTO, Jolt Nir Singher Software Engineer, Trusteer

3. “ I’m sorry Dave, I’m afraid I can’t do that HAL 9000

4. “

5. Dedicated voice interfaces?

7. Why not use Alexa/Siri/Google? ◇ Interaction requires specific phrasing ◇ Authoring requires coding expertise ◇ “Request-Response” only

8. System Architecture Physical Device Voice Server Authoring Interface

9. The Physical Device 1

10.

11. 82.9GBBigger than your 3G plan 1036$Could’ve hired an intern instead One month of listening…

12.

13. ◇ Device-local voice and hotword detection ◇ Supports custom keywords ◇ Has NodeJS bindings ◇ Runs on RaspberryPi ◇ https://snowboy.kitt.ai/

14. Device State Machine Ring Buffer

15. The Voice Server 2

16. “hello” “how are you?” <asked_hello> Server Processing

17. Off-the-shelf solutions ◇Api.ai (Google) ◇Wit.ai (Facebook) ◇LUIS (Microsoft) ◇Lex (Amazon) ◇Botsify ◇Motion.ai ◇Pullstring ◇….. ◇ Intent based, not conversation based ◇ Hard to use for non-techies (And for techies too!) ◇ We didn’t need their wealth of integrations

18. “hello” “how are you?” <asked_hello> Server Processing Speech to Text Text to Speech Natural Language Processing

19. Speech to Text ◇ Not intent based ◇ Realtime results ◇ Accurate in noisy environments ◇ Easy NodeJS Bindings

20. Text to Speech ◇ Sounds good

21. NLP ◇ Rule based ◇ Intent/Slot/Entity based ◇ Statistical

22. The Authoring Interface 3

23.

24.

25. Rule based NLP ◇ As long as you control the conversation, you know where its going to go ◇ You’re mostly looking for keywords ◇ Context is usually just a “state” ◇ Catchalls are important ◇ Its really hard to guess what people will say ◇ Iterate, Iterate, Iterate!

26. Summary

27. Telemetry ◇ Tracking your devices ◇ Remote admin and configuration ◇ Server-triggered conversations

28. Demo!

29. Thanks! Any questions? You can find me at: ◇ hochbergg@gmail.com ◇ Gal Hochberg (FB) ◇ @SoftFlare

Building voice interfaces with JavaScript & Web APIs

Recommended

Recommended

More Related Content

Similar to Building voice interfaces with JavaScript & Web APIs

Similar to Building voice interfaces with JavaScript & Web APIs (20)

Recently uploaded

Recently uploaded (20)

Building voice interfaces with JavaScript & Web APIs