Breaking Language Barriers with AI Services

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Breaking Language Barriers with AI
Dr. Ruth Bergman
Director Of Engineering
Acute Care, GE Healthcare
A I M 3 0 1
Boaz Ziniman
Technical Evangelist
Amazon Web Services
@ziniman ziniman

Natural language processing (NLP)
• Automatic Speech Recognition (ASR)
• Natural Language Understanding (NLU)
• Text to Speech
• Translation
1970 1980 1990 2000 2010 2020
HUMAN ACCURACY
50% 55%
60% 62%
70%
95%
Source: MindMeld

Common Language Use Cases
Information Bots
Education
Accessibility
Knowledge Management
Voice of Customer
Applications
Customer Service/
Call Centers
Enterprise
Digital Assistant
Semantic Search
Captioning Workflows
LocalizationPersonalization

M L F R A M E W O R K S &
I N F R A S T R U C T U R E
A I S E R V I C E S
R E K O G N I T I O N
I M A G E
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D
& C O M P R E H E N D
M E D I C A L
L E XR E K O G N I T I O N
V I D E O
Vision Speech Chatbots
A M A Z O N
S A G E M A K E R
B U I L D T R A I N
F O R E C A S TT E X T R A C T P E R S O N A L I Z E
D E P L O Y
Pre-built algorithms & notebooks
Data labeling (G R O U N D T R U T H )
One-click model training & tuning
Optimization (N E O )
One-click deployment & hosting
M L S E R V I C E S
F r a m e w o r k s I n t e r f a c e s I n f r a s t r u c t u r e
E C 2 P 3
& P 3 d n
E C 2 C 5 F P G A s G R E E N G R A S S E L A S T I C
I N F E R E N C E
Reinforcement learningAlgorithms & models ( A W S M A R K E T P L A C E
F O R M A C H I N E L E A R N I N G )
Language Forecasting Recommendations
The Amazon ML stack: Broadest & deepest set of capabilities

Pre-trained AI services that require
no ML skills or training
Easily add intelligence to your
existing apps and workflows
Quality and accuracy from
continuously-learning APIs
R E K O G N I T I O N
I M A G E
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D
& C O M P R E H E N D
M E D I C A L
L E XR E K O G N I T I O N
V I D E O
Vision Speech Chatbots
F O R E C A S TT E X T R A C T P E R S O N A L I Z E
Language Forecasting Recommendations
A I S E R V I C E S
AI Services

Turn text into lifelike speech using deep learning

Amazon Polly – Use Cases
Contact
Centers
Special Needs
AI Assistant
Voiced videos
and presentations
Language
learning
Amazon Polly
Navigation
Podcasting,
Voiced blogs
and news
articles

“Today in Tel Aviv
Israel is 19°C”
Amazon Polly: Text In, Life-like Speech Out
54 voices across 27 languages
“Today in Tel Aviv Israel is 19
degrees Celsius”
Amazon Polly

Amazon Polly: Text In, Life-like Speech Out
“Надеюсь вам
понравится саммит в
Тель-Авиве”
“Надеюсь вам
понравится саммит в
Тель-Авиве”
54 voices across 27 languages
Amazon Polly

A Focus On Voice Quality & Pronunciation
1. Automatic, Accurate Text Processing
“Today in Tel Aviv Israel is 19°C”

2. Intelligible and Easy to Understand

3. Add Semantic Meaning to Text

https://www.w3.org/TR/speech-synthesis/
<speak>
The spelling of my name is
<prosody rate='x-slow'>
<say-as interpret-as="characters">Boaz</say-as>
</prosody>
</speak>

3. Add Semantic Meaning to Text
“Richard’s number is 2122341237“
<say-as interpret-as="telephone">

Voice Modification: Vocal Tract Length
<speak>
This is Brian without any voice modifications.
<amazon:effect vocal-tract-length="+15%"> Imagine now that I got bigger… </amazon:effect>
<amazon:effect vocal-tract-length="+25%"> Suppose that I got even bigger still… </amazon:effect>
Now let's go back and hear the effect when I go in the opposite direction.
<amazon:effect vocal-tract-length="-15%"> Can you tell that I'm getting smaller? </amazon:effect>
<amazon:effect vocal-tract-length="-25%"> Now I'm even smaller than before. </amazon:effect>
</speak>

Polly API example
aws polly synthesize-speech
--output-format mp3 --voice-id Matthew --text-type ssml
--text '<speak>
<amazon:auto-breaths>
<prosody rate="x-slow" pitch="low">Here is my little secret.</prosody>
<amazon:breath duration="long" volume="x-loud"/>
<amazon:effect name="whispered">
<prosody rate="x-slow">
<prosody pitch="x-low">I</prosody>
killed Mufasa!
</prosody>
</amazon:effect>
</amazon:auto-breaths>
</speak>'
mufasa.mp3

Automatic speech recognition

Automatic speech recognition service
“Hello, this is Allan
speaking”
Amazon Transcribe

Amazon Transcribe – Key Features
Channel
Identification
Custom
vocabulary
Speaker
Identification
Word-level
time stamps
Punctuation and
capitalization
Word-level
confidence scores

ringDNA
RingDNA is an end-to-end communications
platform for sales teams.
Hundreds of enterprise organizations use
RingDNA to increase productivity, engage in
smarter sales conversations, gain predictive
sales insights and improve their win rate.
Speech to Text
"A critical component of RingDNA’s Conversation
AI requires best of breed speech-to-text to deliver
transcriptions of every phone call. RingDNA is
excited about Amazon Transcribe since it provides
high-quality speech recognition at scale, helping us
to better transcribe every call to text"
Howard Brown, CEO & Founder, RingDNA
https://www.youtube.com/watch?v=1ZJ_f1bDdog

Natural and accurate language translation

21 Languages
417 Combinations
Key Features
Real-time
< 500ms / sentence on average
< 150ms / conversational / short form
Tag Handling
XML tags placement maintains
styling and formatting through
translation
< / >
Data Security
Data ownership
Encryption
Access Management
Ease of Use
Simple API calls and partner
solutions
$15/1M characters
Or $0.000075 per word;
Pay as you go, 2M characters
monthly free tier
HIPAA Eligible

Amazon Translate
Natural and fluent language translation
“Hello, what’s up? Do
you want to go see a
movie tonight?”
"‫ש‬‫ל‬‫ו‬‫ם‬,‫מ‬‫ה‬‫ק‬‫ו‬‫ר‬‫ה‬?‫ר‬‫ו‬‫צ‬‫ה‬‫ל‬‫ל‬‫כ‬‫ת‬
‫ל‬‫ר‬‫א‬‫ו‬‫ת‬‫ס‬‫ר‬‫ט‬‫ה‬‫ל‬‫י‬‫ל‬‫ה‬?"
Amazon Translate

Translate API example
boazz: ~$ aws translate translate-text
--text "Hello, what’s up? Do you want to go see a movie tonight?"
--source-language-code auto --target-language-code he
{
"TargetLanguageCode": ”he",
"TranslatedText": " ‫ש‬‫ל‬‫ו‬‫ם‬,‫מ‬‫ה‬‫ק‬‫ו‬‫ר‬‫ה‬?,‫ר‬‫ו‬‫צ‬‫ה‬‫ל‬‫ל‬‫כ‬‫ת‬‫ל‬‫ר‬‫א‬‫ו‬‫ת‬‫ס‬‫ר‬‫ט‬‫ה‬‫ל‬‫י‬‫ל‬‫ה‬? ",
"SourceLanguageCode": "en"
}

import boto3
translate = boto3.client("translate")
lang_flag_pairs = [("fr", "!"), ("de", """), ("es", "#"),
("pt", "$"),("zh", "%"), ("ja", "&"),
("ru", "'"),("it", "("), ("zh-TW", ")"),
("tr", "*"), ("cs", "+"), (”he", ",")]
for lang, flag in lang_flag_pairs:
print(flag)
print(translate.translate_text(
Text="Hello, World",
SourceLanguageCode="en",
TargetLanguageCode=lang
)['TranslatedText'])

!
Bonjour, Monde
"
Hallo, Welt
#
Hola, Mundo
$
Olá, Mundo
%
&
'
Привет, Мир
(
Ciao, Mondo
)
,
*
Merhaba, Dünya.
+
Ahoj, světe.
,
‫ש‬‫ל‬‫ו‬‫ם‬,‫ע‬‫ו‬‫ל‬‫ם‬.
https://github.com/ziniman/aws-translate-demo

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scaling
real-time translation
Using Amazon Translate, Lionbridge is able to
scale machine translation in order to localize
content faster and in more languages. Using
Translate, Lionbridge was able to reduce
translation costs by 20 percent.

https://github.com/ziniman/aws-translate-demo

Discover insights and relationships in text

Amazon Comprehend
Di s c o v e r i n s i g h t s a n d r e l a t i o n s h i p s i n t e x t
Entities
Key Phrases
Language
Sentiment
Syntax
Grouping
Amazon Comprehend

Accurately extract health information from patient
notes, clinical trial reports, and other electronic
health records using Amazon Comprehend

Amazon Comprehend Medical
Entities
Medication
Medical condition
Test, treatments, and
procedures anatomy
Protected Health
Information (PHI)
Relationship extraction
Medication
Test, treatments, and procedures
Entity traits
Negation
Diagnosis signs and symptom

Run Amazon Comprehend on S3 Bucket
import boto3
import json
s3 = boto3.resource('s3’)
bucket_name = ‘my_bucket’
region_name = ‘us-east-1’
bucket = s3.Bucket(bucket_name)
comprehend = boto3.client(service_name='comprehend', region_name=region)
for obj in bucket.objects.all():
body = obj.get()['Body'].read()
text = body
sentiment_response = comprehend.detect_sentiment(Text=text, LanguageCode='en’)
print(json.dumps(sentiment_response, sort_keys=True, indent=4))

Conversational interfaces for your applications
powered by the same deep learning technologies as
Alexa

Advent of conversational interactions
2nd gen:
Pointers & sliders
3nd gen:
Conversational
interfaces
1st gen:
Punch cards & memory
registers

Amazon Lex – use cases
CONTACT CENTER BOTS
Customer service IVR
Account inquiries
Bill payments
Service updates
Single Sign On
Users / Roles
Groups
Auditing / Monitoring
Risk & Compliancy
Insights
SECURITY
INFORMATIONAL BOTS
Answer questions
News updates
Weather information
Game scores
APPLICATION BOTS
Conversational interfaces
Book tickets
Order food
Manage bank accounts
Single Sign On
Users / Roles
Groups
Auditing / Monitoring
Risk & Compliancy
Insights
SECURITY
PRODUCTIVITY BOTS
Enterprise efficiencies
Check sales numbers
Inventory status
Expense reports
IoT BOTS
Device interactions
Kiosks
Appliances
Auto
A service for building conversational interfaces into your applications using voice and text

Lex Use Case: Digital Assistant to Book a Hotel
Book hotel
NYC
“Book a hotel in
NYC”
Automatic speech
recognition
Hotel booking
New York City
Natural language
understanding
Intent/slot
model
UtterancesHotel booking
City New York City
Check in November 30
Check out December 2
“Your hotel is booked for
November 30.”
Amazon Polly
Confirmation: “Your hotel is
booked for November 30.”
“Can I go ahead
with the booking?”
a
in

GE Healthcare &
Roche Diagnostics
A C O L L A B O R A T I O N I N A C U T E C A R E M E D I C I N E
*Disclaimer: Technology in development that represents ongoing research and development efforts. These technologies are not products and may
never become products. Not for sale. Not cleared or approved by the U.S. FDA or any other global regulator for commercial availability.
Ruth Bergman, PhD
Director of Engineering
Acute Care

A C R I T I C A L M O M E N T I N H I S T O R Y
The most inspiring discoveries
liberate society in practical and
inspiring ways.

T H E P R O B L E M
THE #1 CAUSE OF
DEATH IN HOSPITALS
$18K PER PATIENT & $27B
IN THE US
20-50% OF SEVERELY
AFFECTED DIE
ESCALATES
UNPREDICTABLY

T H E P R O B L E M
It’s not that patients suddenly deteriorate.
It’s that caregivers suddenly notice.

T H E P R O B L E M
Why is that?

T H E P R O B L E M : W H Y I S S E P S I S G O I N G U N D E T E C T E D ?
Reason 1:
Dark Data
Reason 2:
Trapped Clinician Potential
UNDETECTED
DETERIORATION
e.g. Sepsis / Death

T H E P R O B L E M
Why is DATA not helping like it should?
Dark Data
Data Slower
Than Disease
The data are captured in
snapshots, and the
syndrome or condition
changes between
measurements.
The Data Desert
Must detect patients
before the ICU… before
they’re monitored.
Timeliness
Indicators do not trigger
early enough to respond
effectively.
Sensitivity &
Specificity
The inability to correctly
identify those who have
or don't have the
syndrome.

T H E P R O B L E M
Trapped Clinician Potential
Why is CLINICIAN POTENTIAL trapped?
Care Complexity
How do I ask better
questions, from better
data, faster to inform
diagnosis and care plan?
EMR Time Drain
Reliable system of
record, but frustrating
system of engagement.
Patient Awareness
Staff to patient ratios can
make it difficult to notice
important patient changes.
Collective Awareness
Inability to share a
common medical view across
the care team.

ArchitectureLayers
ServiceData
UI (App & Web Browser)
Ezra-App API Gateway
Bot NLP
HospitalAPIGateway
Log and Audit
Services
Communication Managed Services
Patient Chat
EC2
LEX
NLP
Ingestion
ConfigurationQuery
CloudWatch
S3
Clinical Data
Cloudformation
Route
53
VPC
Kinesis
HAPI FHIR
server
WebSocket
CloudFront
Browser
Cognito
Operational Data

Put AI to work for your business
Modernize your contact center to improve customer service
conversational chat bots | call transcription | intelligent routing | sentiment analysis | VoC analytics
text-to speech | multilingual omni-channel communication
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E X

Amazon Connect
Real time and
historical analytics
Skills-based routing
[Automatic Call Distribution (ACD)]
Call
recording
High-quality
voice capability
Easy to use, cloud-based contact center solution
that scales to support businesses of any size
With tools that grow with your needs

Improving Contact Centers With Artificial Intelligence
A m a z o n
L e x
A m a z o n
T r a n s c r i b e
A m a z o n
C o m p r e h e n d
T R A N S C R I P T
A m a z o n
C o n n e c t
Analytics

Connect Demo
+1-979-335-5593
+1-979-335-5593
Next session for Boaz Ziniman is…
Amazon
Polly
Amazon
Translate
Amazon
DynamoDB
AWS
Lambda
Amazon
Connect

Babel fish
"The Babel fish is small, yellow, leech-like - and probably the oddest thing in the universe. It
feeds on brain wave energy, absorbing all unconscious frequencies and then excreting
telepathically a matrix formed from the conscious frequencies and nerve signals picked up from
the speech centres of the brain, the practical upshot of which is that if you stick one in your ear,
you can instantly understand anything said to you in any form of language: the speech you hear
decodes the brain wave matrix."
The Hitchhiker's Guide to the Galaxy, Douglas Adams

Build your own Babel fish
Person Speaks in
English
Transcribe voice to text in
English
Translate text to
Russian
Speak in
Russian
Amazon
Polly
Amazon
Translate
Amazon
Transcribe

Babel Chat
chat.boaz.cloud
{
"room":"/",
"message":{
"user":"Boaz",
"text":"I'm doing great",
"timestamp":1540978160578
},
"clientId":"07958771059731382"
}
Amazon
Translate
Amazon
DynamoDB
AWS
Lambda
Amazon
IoT Core

Thank you!
http://bit.ly/2SIc2xe
Dr. Ruth Bergman
Director Of Engineering
Acute Care, GE Healthcare
Boaz Ziniman
Technical Evangelist
Amazon Web Services
@ziniman ziniman

Breaking Language Barriers with AI Services

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Breaking Language Barriers with AI Services

Similar to Breaking Language Barriers with AI Services (20)

More from AWS Summits

More from AWS Summits (20)

Breaking Language Barriers with AI Services