Car buyers demand more experiences from their vehicles than ever before. But as our vehicles adopt more advanced features, customers find it difficult to understand how to use them, resulting in low utilization. You can address this challenge using an interactive owner’s manual, providing customers with a visual, accessible way of understanding the features of their vehicles. In this session, we walk through the process of building an augmented reality vehicle owner’s manual application, combining augmented reality with Amazon Sumerian, voice interfaces using Amazon Lex and Amazon Polly, and underlying data provided by AWS IoT. Attendees walk away with an understanding of how to build an augmented reality interface to complement the physical controls of the vehicle cockpit.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building an Augmented Reality
Owner's Manual
Miro Masat
Solutions Architect
AWS
A M T 3 1 5

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Introduction
Architecture
Amazon Sumerian—AR
Supporting services—Amazon Lex, Amazon Polly
Supporting services—Amazon DynamoDB, AWS Lambda,
AWS IoT
Demo and discussion

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
160+ pages
6+ languages
paperback
Typical owner’s manual

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture
iOS
Amazon
Sumerian

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture
iOS
Amazon
Lex
Amazon
Polly
Amazon
Sumerian

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture
iOS
Amazon
Lex
Amazon
Polly
AWS
Lambda
Amazon
DynamoDB
Amazon
Sumerian

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture
AWS IoT
iOS
Amazon
Lex
Amazon
Polly
AWS
Lambda
Amazon
DynamoDB
Amazon
Sumerian

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Sumerian—AR
Sumerian contains a scene with a model of
Audi 2017 Q5
The model is placed into AR scene
with transparent layers of windows to see outside
The cockpit/environment and demonstrate true AR
It demonstrates true size in the real world
and contains both interior and exterior
The scene also contains HTML controls, such as
buttons and labels as well as hidden 3D
elements that illuminate the function that
is being explored at the given moment
Why Sumerian?
• Easy to use—drag & drop visual editor
• Cost-efficient—no license fees and pricing based on consumption
• Connected—dynamic models powered by data from AWS

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Sumerian—How does it work?
Sumerian provides an interface for a question
The Sumerian model listens for voice/tap/click inputs
and utilizes state machines to keep track of the application
<speak>Emergency flashers <s/>, to make other drivers
aware of your vehicle being in danger.</speak>

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Supporting services—Amazon Lex, Amazon Polly
Lex bot contains several intents each formed of
several utterances. Utterances are also being
intelligently extended by Lex engine to accept
synonyms.
Some utterances contain a custom slot, which can
be built according to actions/features needed for
the particular model and spec of the vehicle.
Incomplete utterances (missing a slot or missing
entire construct) have follow up questions to
fulfill correct utterance, slot content, or both.
Polly allows for voice customization based on
gender, region, or local accents.
With Polly, certain pronunciation, volume, and speech rate can be achieved
using SSML tags.

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Lex—How does it work?
Lex understands the request
Lex parses the input from Sumerian
How to use {carManualAction}
How to set {carManualAction}
Where can I set {carManualAction}
{showHide} all functions
{showHide} interface
{showHide} details

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Supporting services—Amazon DynamoDB, AWS Lambda, AWS IoT
DynamoDB table contains information about features
available on the particular vehicle and their status.
The table also contains prices of extras available
for the car. This table can serve both Lex bot and
IoT shadow too.
IoT Core has the vehicle registered as unique IoT thing,
sends, receives, and stores vehicle data in a
shadow. This shadow can be updated both ways
independently of connectivity. Shadows can be
consumed by the Sumerian model in real-time.
Lambda contains functions serving Lex bot
answers. Furthermore, because Lambda is essentially a
general compute engine, it opens up possibilities to tap into
connected Car Reference Architecture (CVRA), IoT device simulator, or any other AWS
service, including ML services (like SageMaker or Rekognition) and managed AI services (like Translate or Kinesis).

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Lambda—How does it work?
Lambda and Polly provide an answer
Lambda answers the intent from Lex logically
if 'condition' in rtrnslot and 'air' in rtrnslot:
rtrnmessage = 'To synchronize airconditioning, press and hold left dial on the centre console.'
rtrn = {
"dialogAction": {
"type": "Close",
"fulfillmentState": "Fulfilled",
"message": {
"contentType": "PlainText",
"content": rtrnmessage
}
}
}
return rtrn

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS IoT—How does it work?
IoT provides metrics of the vehicle
IoT (Shadow) provides additional metrics and state of the interface
{
"desired": {
"steps": {},
"interface": 1,
"details": 1,
"shadow": {
"engine": 1,
"booth_kg": 80,
"booth_percentage": 25
}
}
}
arn:aws:iot:us-east-1:XXXXXXXXXXXX:thing/1N4DL01D8YC138508

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related breakouts
Friday, 30. Nov
AMT305, Building BMW Group's Customer Engagement Platform on AWS
10:00 – 11:00 | MGM, Level 3, Premier Ballroom 316, T1

18. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Miro Masat
mmmasat@amazon.com

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.