- The document discusses the potential for a spoken version of Google that allows users to access information via voice commands and returns results in audio form using speech synthesis. - Currently, Google can only understand text and return text-based results, but advances in speech recognition and synthesis mean machines may soon be able to listen to and understand all spoken content online as well as the text. - A spoken version of Google could enable users to retrieve any information from the huge quantities of multimedia content on the internet via voice from any device, making the world's knowledge universally accessible.

1. Towards A Spoken Version of Google
─ making the global knowledge
accessible by voice
Lin-shan Lee
National Taiwan University

2. • Google, Amazon, Facebook, YouTube offer most of our daily life
information
• Physical knowledge archives (e.g. libraries, museums) useful in the
past are developing their virtualized versions
• Online courses, distant learning, electronic books, etc. become the
most efficient learning aids
Internet is the Only Largest Archive for Global
Human Knowledge
Internet
Real-time
Information
– weather, traffic
– flight schedule
– stock price
– sports scores
Special Services
– Google
– Facebook
–YouTube
– Amazon
Knowledge
Archieves
– digital libraries
– virtual museums
Intelligent Working
Environment
– intelligent agents
– teleconferencing
– distant learning
– electric commerce
Private Services
– personal assistants
– business databases
– home appliances
– network
entertainments

3. • Have been An Important Part of our Daily Life
Google and the Internet
• Finding any desired information (in text) over the Internet
relevant to the user instructions (in text): Information
Retrieval
• Google is able to match and find the relevance between
segments of text
Internet
Google
……………
…………..
…….
server
server
User instructions
(in text)
Documents/Information
(in text)
(in text)
(in text)

4. Google and the Internet
– for multimedia information (e.g. YouTube, online courses, including
video and audio) to be retrieved by Google, the content needs to include
text descriptions
• Google can only read text information
• Have been An Important Part of our Daily Life
Internet
……………
…………..
…….
server
server
User instructions
(in text)
Documents/Information
(in text)
Google

5. Google and the Internet
• The Whole Video (a lecture or a news story) can be retrieved,
but Not the Exact Sentence
– the text descriptions is for a video or a lecture
• Have been An Important Part of our Daily Life
Internet
……………
…………..
…….
server
server
User instructions
(in text)
Documents/Information
(in text)
Google

6. Google and the Internet
• The Audio Information (voice in a video or a lecture) in
Multimedia very often tells what is going on in the
Multimedia
• Can be used as the Key for Information Retrieval
• Have been An Important Part of our Daily Life
Internet
……………
…………..
…….
server
server
User instructions
(in text)
Documents/Information
(in text)
Google

7. Speech Technologies Advancing Very Fast Today
• Speech Recognition / Synthesis Technologies
• Machines can Listen to Voice or Read Text
• All Roles of Text can be Realized by Voice
Speech Recognition
(machine)
Speech Synthesis
(machine)
text
voice
How are you today?
Good morning!
Spoken language written language

8. Machines
listen to
the audio
part
Google
voice
information
Multimedia
Content
Internet
voice
input/
output
text
information
• Information Retrieval as Google does can be performed based on Voice
• User-Content Interaction can be Accomplished by Spoken and Multi-
modal Dialogues (including other Modes of Interaction, e.g. those with
fingers)
• Text Information found can be transformed to Voice
Google
(text-based)
Text
Content
Information
Retrieval
(voice-based)
Speech Synthesis
Spoken and
Multi-modal
Dialogue
Spoken Version of Google
Google may have a Spoken Version

9. Information Retrieval (text/voice-based)
• Both the User Instructions and Network Content can be in
form of Voice
Voice Instructions
US/China Trade Policy ?
Text Instructions
d1
Text Information
d2
d3
d1
d2
d3
Voice Information
President Donald Trump…
Google
/voice

10. Voice Interface is Convenient for All Different Kinds of
User Terminals
Internet
Text Content
Multimedia
Content
• Smart phones, Hand-held Devices, Wearable Devices (Watches,
Glasses, etc.), Notebooks, Vehicular Electronics, Home
Appliances …
• Network Access at Any Time, from Anywhere
• Small in Size, Light in Weight, Ubiquitous, Invisible…
• Voice is the Only Interface Convenient for ALL User Terminals at
Any Time, from Anywhere, and To the Point in one Utterance

11. Google and the Internet
• Have been An Important Part of our Daily Life
Internet
Google
……………
…………..
…….
server
server
User instructions
(in text)
Documents/Information
(in text)
• Finding any desired information (in text) over the Internet
relevant to the user instructions (in text): Information
Retrieval
• Google is able to match and find the relevance between
segments of text

12. Google can have a Spoken Version
• Finding any desired information (in voice) over the Internet
relevant to the user instructions (in voice): Information
Retrieval
• Spoken version of Google is able to match and find the
relevance between segments of voice
• Have been An Important Part of our Daily Life
Internet
Google
……………
…………..
…….
server
server
User instructions
(in voice)
Documents/Information
(in voice)

13. Google can have a Spoken Version
• Can Locate the Exact Time the Desired Information Appears
in the Multimedia
• No Need for Text Descriptions
• Have been An Important Part of our Daily Life
Internet
Google
……………
…………..
…….
server
server
User instructions
(in voice)
Documents/Information
(in voice)

14. Retrieval of Public TV News of Taiwan (公視新聞
搜尋)
• 2004
• Locate the Exact Relevant Utterances (Spoken Sentences)
without Text Descriptions

15. Retrieval of Public TV News of Taiwan (公視新聞
搜尋)
• 2004
• Locate the Exact Relevant Utterances (Spoken Sentences)
without Text Descriptions

16. User-Content Interaction for Spoken Content
Retrieval
• Problems
– User-content interaction always important even for text content
– Unlike text content, spoken content not easily summarized on screen,
thus retrieved results difficult to scan and select
User
Query
Spoken
Archives
Retrieved Results Spoken Version
of Google

17. User-Content Interaction for Spoken Content
Retrieval
• Possible Approaches
– Automatic summary/title generation and keyword extraction from
spoken content
– Topic structure for spoken content
– Multi-modal dialogue with improved interaction
Keyword/
Titles/Summaries
User
Query
Multi-modal
Dialogue
Spoken
Archives
Retrieved Results Spoken Version
of Google
User
Interface Topic
Structure
Spoken version of Google

18. X1
X2
X3
X4
X5
X6
Correctly recognized word
X1
X3
summary
• Selecting most representative
utterances but avoiding redundancy
Wrongly recognized word
t2
t1
Summarization of A Segment of Voice

19. User-Content Interaction for Spoken Content
Retrieval
• Possible Approaches
– Automatic summary/title generation and keyword extraction from
spoken content
– Topic structure for spoken content
– Multi-modal dialogue with improved interaction
Keyword/
Titles/Summaries
User
Query
Multi-modal
Dialogue
Spoken
Archives
Retrieved Results Spoken Version
of Google
User
Interface Topic
Structure
Spoken version of Google

20. • Example 1: retrieved results Grouped by Topics and
organized in a Two-dimensional Tree Structure
– each group of retrieved segments labeled by a set of keywords (topic)
– each group expanded into a map in the next layer
Topic Structure (1/2)

21. Broadcast News Browser (2006)
(電視新聞瀏覽器)
Demonstration
Summary NewsVideo

22. • Usually very long time required (e.g. 45 hrs) to learn from a
complete online course
– difficult for very busy people to learn pieces of knowledge
• Possible to retrieve a segment of lecture with a browser
– knowledge transfer in a course is usually sequential
– not easy to understand a lecture without listening to the previous
lectures
– not easy to find out background or related knowledge
• Possible Approaches
– consecutive audio/video for each slide taken as a segment of
multimedia (a page of spoken slide)
– each spoken slide labeled by the keywords (topic) extracted from the
audio
– relationships between keywords of the course represented by a graph
Online Courses

23. • Example 2: Keyword Graph
– each spoken slide labeled by a set of keywords (topics)
– relationships between keywords represented by a graph
Topic Structure (2/2)
-----
-----
-----
-----
---------
---------
---------
---
-------
-------
-------
----
spoken
slides
keyword
graph
Acoustic
Modeling
Viterbi
search
HMM
Language
Modeling
Perplexity

24. NTU Virtual Instructor (2009)
(台大虛擬教師)
Demonstration

25. Too Many Online Courses
752 matches
• A user enters a keyword or a key phrase to coursera

26. Machines Listening to the online Courses
Lectures with very
similar content
three courses on
some similar topic

27. sequential order for
learning (prerequisite
conditions)
Machines Listening to the online Courses
three courses on
some similar topic

28. Learning Map Produced by
Machine (2014)
(機器製作學習地圖)
Demonstration

29. • To Which Degree can Machines Understand the Spoken
Content ?
• Machines to take TOEFL Listening Comprehension Test (an
English test for students whose native language is not English)
• An Example Problem:
A Question: “ What is a possible origin of Venus’ clouds? ”
Answer options:
(A) gases released as a result of volcanic activity
(C) bursts of radio energy from the plane's surface
(D) strong winds that blow dust into the atmosphere
Recorded voice: (5min long)
Machine Comprehension of Spoken Content
(B) chemical reactions caused by high surface temperatures

30. (1) (2) (3) (4) (5)
by some hand-crafted rules without
listening to the recorded voice
Best Machine Performance: 50.0%
choosing the shortest answer
Machine Comprehension of Spoken Content
(%)
• Machine Performance 2017
Accuracy

31. What is Technically
Achievable in the Future ?

32. What can Machines do for Human ?
• Machines can listen to and understand the entire multimedia
knowledge archive over the Internet
– extracting desired information for each individual user
300hrs of videos
uploaded per min
(2015.01)
Roughly 2000 online
courses on Coursera
(2016.04)
• Nobody can go through so much multimedia
information, but Machines can
• Multimedia Content exponentially
increasing over the Internet
– best archive of global human knowledge is here
– desired information deeply buried under huge quantities
of unrelated information

33. An Example: Personalized Courses
• Machines generate desired personalized courses for each
individual user
I wish to learn some knowledge
about the masterpieces of
Wolfgang Amadeus Mozart
I am an engineer. I know nothing
about music
I can spend 3 hrs to learn
user
Thank you for your request. This is
the 3-hr personalized course for
you.
Information
from Internet

34. Spoken Version of Google
• Speech Recognition and Synthesis Technologies Make
Machines Capable of Listening to and Speaking Human Voice
– the best the machines can do may be as good as the human
• Machines can Handle Huge Quantities of Information
– much more efficient than human
• Google Reads All Text over the Internet
– similarly machines can listen to all voices over the Internet
• Internet is the Only Largest Archive for Global Human
Knowledge
– voice can be the key to that archive