Would you like to try hands-on how to create your first Augmented Reality app? In this session, you will see how easy it is to use 3D technologies and automated speech processing (through Amazon Sumerian) to explain healthcare-related topics. The session is defined for health professionals with little or no prior experience in app design, who would love to share their knowledge with new didactic methods. Additionally, we will provide a behind-the-scenes look on the use case “Enlightening Patients with Augmented Reality” within the funded research project “Immersive Media Lab”. Presented at the build.well.being conference on June 14th, 2019: https://buildwellbeing.fhstp.ac.at/ More information about the contents: https://www.andreasjakl.com/

1. St. Pölten University of Applied SciencesSt. Pölten University of Applied Sciences
Platzhalter für möglichen
Bildeinsatz
Create engaging Healthcare Experiences
with Augmented Reality
Andreas Jakl – www.andreasjakl.com
build.well.being 2019
Digital Healthcare
FH St. Pölten
Platzhalter für möglichen
Bildeinsatz

2. Andreas Jakl
▪ Focus areas
▪ AR / VR, mobile apps, sensors, interaction
technology, software architecture, open source
developer (NFC, Bluetooth Beacons)
▪ Microsoft MVP (Most Valuable Professional) &
Amazon AWS Cloud Ambassador
▪ mobility.builders community:
Mobile Developer After-Work Events
▪ Previous Experience
▪ Tieto, Start-up (Mopius), Nokia (Finland),
Siemens Mobile (Munich), FH Hagenberg
(Mobile Computing)
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
https://www.andreasjakl.com/
andreas.jakl@fhstp.ac.at
@andijakl

3. WHAT IS AR / VR / MR?
Overview & Intro
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

4. Virtual Reality?
▪ Dive into another reality
▪ Mainly visual +
acoustic stimulation
▪ Immersion & presence
▪ Feeling to be at
another place
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
Screenshot: Ready Player One, 2018

5. Augmented und Mixed Reality
▪ Virtual Reality
▪ Only virtual, completely hides reality
▪ Augmented Reality
▪ Visible world gets digital overlays
▪ Mixed Reality
▪ Virtual content seamlessly blends into the real world
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
Wired.com

6. Microsoft HoloLens 2
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
https://www.hololens.com/

7. Apple ARKit
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
https://developer.apple.com/augmented-reality/

8. Google DayDream / ARCore
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
Google DayDream
https://vr.google.com/daydream/
Google ARCore
https://developers.google.com/ar/

9. Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
Source: CCS Insight VR and AR forecast (November 2018)
https://www.ccsinsight.com/press/company-news/3726-virtual-reality-device-market-declines-in-2018-but-outlook-remains-positive
Shipments of VR and AR devices, worldwide,
2015-2022

10. AR / VR Estimates: 2025
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
Source: Goldman Sachs Global Investment Research
January 13, 2016

11. Hype &
Reality
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten Source: Gartner
https://www.gartner.com/smarterwithgartner/5-trends-emerge-in-gartner-hype-cycle-for-emerging-technologies-2018/

12. Digital Healthcare Explained
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
App created by Digital Healthcare
@ FH St. Pölten wins global
Amazon AR/VR challenge
in the category: Best AR App!
https://devpost.com/software/digital-healthcare-explained-czflve
https://www.fhstp.ac.at/en/newsroom/news/best-augmented-reality-app-in-amazon-hackathon

13. Immersive Media Lab
▪ Storytelling in AR / VR
▪ Use case: “Enlightening Patients
with Augmented Reality”
▪ Explain corrective eye surgery
▪ Pre-operation, during operation,
afterwards
▪ Study acceptance of different AR
interaction methods
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

14. AMAZON SUMERIAN
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

15. Amazon Sumerian Overview
▪ Interactive 3D scenes
▪ Browser-based 3D editor + viewer (WebGL / WebXR)
▪ Supports VR & AR
▪ Special feature:
▪ Animated avatars (“hosts”)
▪ Speech + gestures
▪ Supports conversations
▪ https://aws.amazon.com/sumerian/
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

16. AWS Account
▪ AWS = Amazon Web Services
▪ Sumerian
▪ Free tier for 12 months
https://aws.amazon.com/free/
▪ 5 GB scene traffic free
▪ Credit card required for sign-up, but not charged!
▪ Pricing after 1 year / exceeding free tier limits
▪ 3D object storage, scene traffic, speech services
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

17. Scene Setup
▪ Log in to Amazon Sumerian Console
▪ https://console.aws.amazon.com/sumerian/home
1. Choose world-wide region for your service
2. Create “Default Lighting” scene
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

18. 3D Editor
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
Canvas / 3D view
Click: select 3D object
Shift + left mouse: move view
Right mouse: rotate view
Entities: everything
in your scene
Assets: currently
available content
Inspector:
properties of
selected part
Top menu: new 3D
models, publish
scene

19. First 3D Object
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
1) Import Assets
2) Select Table Cafe
3) Add

20. Add to Scene
▪ Drag & Drop Model from Assets to Entities
▪ Then wait a few seconds…
▪ Ensure object is visible. Some tips:
▪ Zoom in with mouse wheel
▪ Select object and press “f”-key to center
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

21. Publish!
▪ Link to your scene
▪ Runs in every Chrome + Firefox!
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

22. Custom 3D Models
▪ Free sources
▪ Remix 3D (Microsoft):
https://www.remix3d.com
▪ Poly (Google):
https://poly.google.com/
▪ Sketchfab:
https://sketchfab.com/features/download
▪ Own 3D Models
▪ Paint 3D & 3D Builder (Windows 10)
▪ Blender
▪ Autodesk Maya / 3DS Max
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

23. Import to Sumerian
▪ Human Heart
▪ From Remix3D (Microsoft)*
▪ Download adapted version:
https://www.andreasjakl.com/
human-heart/
▪ Import to Sumerian
▪ Drag from “Assets” to “Entities”
▪ Adapt scale & rotation
▪ Scale: 0.1, Rotate X: -90
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
* https://www.remix3d.com/details/059f2766c027458787256ebb47a4094e
Microsoft, Creative Commons CC-BY-SA

24. Adding Speech
▪ Conversational interfaces
▪ Amazon Lex: natural language understanding
▪ Amazon Polly: text-to-speech
▪ Different Amazon Services!
▪ Secure communication?
▪ Amazon Cognito credentials
▪ One-click setup:
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
Amazon Alexa
Built on these two
services, adds “skills” concept,
skill store, etc.
https://docs.sumerian.amazonaws.com/tutorials/create/beginner/aws-setup/

25. Stack → CognitoIdentyPoolID
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
3
Copy CognitoIdentyPoolID

26. Connect with Sumerian
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
Paste

27. Create Host
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

28. Position Host
▪ Drag host from Assets to
Entities
▪ Then wait a few seconds!
▪ Re-position host
▪ Press Play
▪ Host already moving
naturally
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

29. Let the Host Explain the Heart
▪ Edit the Speech settings in the inspector of your host
▪ (1) Choose voice: “Salli” is US English
▪ (2) Click on “+” in “Speech Files” to create a new file
▪ (3) Enter text
▪ (4) Save (Default name: “Speech”)
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
1
2
3
4

30. Create Logic: When to Speak?
▪ (1) Choose “+ Add Component”
▪ “State Machine”
▪ (2) Add a new behavior (“+”)
▪ (3) Opens editor in bottom part
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
1
2
3

31. Add Actions
▪ Actions define what to do in a situation (“state”) or what to wait for
▪ “Add Action” to default “State 1”
▪ “Click/Tap on entity”
Hint: use search in top right corner
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

32. Second State: Start Speaking!
▪ Add a second state
▪ What should happen after tapping the
host?
▪ (1) + (2) “Add State”
▪ (3) Add action to “State 2”: “Start Speech”
▪ (4) Choose your “Speech” from drop-down
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
1
2
3
4

33. Connect Everything
▪ Logic
▪ Clicking should trigger speech
▪ Speech end should go back to initial state
▪ Drag & drop from action box to new state
▪ Test!
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

34. More Life-Like Host: Gestures
▪ Hand-Movements of the host while speaking
▪ Select host in “Entities”
▪ (1) Inspector: “Speech” → “Gesture Map” → “+”
▪ Triggers gestures based on words in spoken text
▪ (2) Auto-assign gestures to “Speech” file
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten
1
2

35. Publish & Test!
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

36. Augmented Reality & More!
▪ Extended Tutorial
▪ Animation & Timelines
▪ Communicating within the scene
▪ Publish to Android & show as Augmented Reality Scene
▪ https://www.andreasjakl.com/amazon-sumerian-
augmented-reality/
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten

37. THANK YOU!
Andreas Jakl
Lecturer / Digital Healthcare @ FHSTP
https://www.andreasjakl.com/
Create engaging Healthcare Experiences | 2019 | Andreas Jakl | FH St. Pölten