Latencies+within++Multimodal

1. Latencies within
Multimodal GenAI
Varun Singh
Daily.co
11 Dec 2023
Understanding Latency

2. Toolkits for
AI-powered voice
and video

3. Memory-based Networks
and Transformers
2016
Conversational AIs are
everywhere
2018
LLMs and Generative AIs:
Image/ Video/Chat
2023
2013
Neural Networks for Natural
Language Processing
AI’s new inflection point, Generative AI

4. ChatGPT and friends
Text AI Engine/LLM Text
Websocket
HTTP2/3
Websocket
HTTP2/3
Latency for chat while measurable is
not significant, because there can be
hidden with UX tools

5. Voice AI Engine Voice
Speech to
text
Text to
Speech
Multimodal Bots!
Humans are messy, AI engines need
to work with this

7. Confidential ©2023 Daily
Real-time AI experience workflow

8. Confidential ©2023 Daily
Transports
// Understanding latency
WebSockets and WebTransport
● Can stream packets, but handle audio chunking
● Maybe, HOLB
WebRTC
● Best option, currently
Irrespective of transport, things to consider
● Voice activity detection
● Audio packetization, is 20ms the best option
● Redundancy – FEC, NACK etc

9. Confidential ©2023 Daily
Latencies stack up
// Understanding latency
Speech-to-text
Text fed to LLM, LLM response
Text-to-speech

10. Confidential ©2023 Daily
Latencies stack up, but do they?
// Understanding latency
Speech-to-text
Text fed to LLM, LLM response
Text-to-speech
We hem and haw
We interrupt, speak over each other
We sometimes take a moment to
conjure and organise our thoughts

11. Confidential ©2023 Daily
Latencies stack up, but do they?
// Understanding latency
Speech-to-text
● Provide each word? Until a punctuation? Talk spurt?
● Because spoken word is not the same as written

12. Confidential ©2023 Daily
Latencies stack up, but do they?
// Understanding latency
Speech-to-text
● Provide each word? Until a punctuation? Talk spurt?
● Because spoken word is not the same as written
Text fed to LLM
● Feed fragments or complete talk spurt?
● More context means better LLM response

13. Confidential ©2023 Daily
Latencies stack up, but do they?
// Understanding latency
Speech-to-text
● Provide each word? Until a punctuation? Talk spurt?
● Because spoken word is not the same as written
Text fed to LLM
● Feed fragments or complete talk spurt?
● More context means better LLM response
Text-to-Speech
● Word by word? Sentence?
● Again spoken words have intonation, which require
context

14. Confidential ©2023 Daily
Real-time AI experience workflow

15. Download it, give it a try