by Yosun Chang / AReality3D

1. © 2017 TWILIO, INC. ALL RIGHTS RESERVED.
AR+AI Interfaces for IoT
Augmented Reality (with AI) Interfaces for the Internet of Things
I. Yosun Chang
yosun@areality3d.com
A Vision of a
AR + ML + Computer Vision enabled world
+ Live Demos Oct 18, 2018

2. © 2018 TWILIO, INC. ALL RIGHTS RESERVED.
IoT is Everywhere
But Access is Painful
• Beyond IoT consumer appliances,
municipal and enterprise
infrastructure are controlled by
connected devices.
• You want to be able to read and
control these connected things at the
speed of thought. But…
• Why isn’t access everywhere, too?
YOUR APP
Webhooks TwiML Response

3. © 2018 TWILIO, INC. ALL RIGHTS RESERVED.
YOU R APP
Webhooks TwiML Response
Problem 1:
Control Fragmentation
Too many different apps, confusing UI,
non-instantaneous entry points

4. © 2018 TWILIO, INC. ALL RIGHTS RESERVED.
YOU R APP
Webhooks TwiML Response
Problem 2:
Connectivity Fragmentation
So many different ways to connect!

5. © 2018 TWILIO, INC. ALL RIGHTS RESERVED.
YOU R APP
Webhooks TwiML Response
Problem 3:
Information Fragmentation
Information should ideally be fluid and accessible - like a uniform melted metal alloy,
and not disjoint, disorganized and silo’d like a pile of junk cars.

6. © 2018 AREALITY3D, INC. ALL RIGHTS RESERVED.
THE SOLUTION!

7. Recognize IoT
objects at the speed
of sight
For things that are less visible
(perhaps represented as
beacons) - have their
tracking be assigned to the
bigger object they’re on for
field of view awareness of
where they are
Augment data and
UI in the real world
where it makes
sense
O B J E C T R E C O G N I T I O N
C O N T E X T UA L “A M B I E N T ”
U N D E R S TA N D I N G
R E A S S I G N M E N T
T R A C K I N G
I N -W O R L D
P L A C E M E N T
Computer Vision for Ubiquitous Access
“If you can see it, you can control it.”
© 2018 AREALITY3D, INC. ALL RIGHTS RESERVED.
Know what’s connected and
which room it is by scanning
a room

8. You can train your model to recognize
whatever you are interested in.
Useful Machine Learning Frameworks
• TensorFlow, Caffe, Dlib, Darknet etc
ML Model Training
• Face tracking
• Inception
• ResNet
• SSD (Single Shot Detector)
• YOLO (You Only Look Once)
Common Models
• BB: Light switches or thermostats
• SO: Specific ports on computers
• SS/IS: Street Traffic Lights, Cars Humans
(Example shown right)
Examples
OBJECT RECOGNITION
• Specific objects
• General classes of objects
• Bounding boxes (BB)
• Sub-object bounding
boxes (SO)
• Semantic segmentation
(SS)
• Instance segmentation
(IS)
© 2018 AREALITY3D, INC. ALL RIGHTS RESERVED.
Credit: Karol Majek

9. TOWARDS ARTIFICIAL GENERAL INTELLIGENCE!
“AMBIENT UNDERSTANDING”
CONTEXT-DEPENDENT RECOGNITION
• Chairs, Computers, Desks, People = Office
• Servers, Servers, Racks = Server Room
• Traffic Light, People, Cars, Bike = Street
© 2018 AREALITY3D, INC. ALL RIGHTS RESERVED.
Scene Identified: Office

10. REASSIGNMENT TRACKING
Assign tracking to a representative object, in addition to the beacon itself.
Beacons can still get lost in clutter or a
local crowd of people. In addition to
beacons: Add in object tracking to face
tracking to avoid this.
D O N ’ T G E T L O S T I N T H E C R O W D !
A beacon can tell you roughly if you’re in
the right area.
P H A S E 1 : B E A C O N L O C A L I T Y
Once you know where to look: Computer
vision analysis of scene to identify what
you’re looking for.
P H A S E 2 : S C E N E D E T E C T I O N
© 2018 AREALITY3D, INC. ALL RIGHTS RESERVED.

11. X-RAY: CABLES
WIRING, ETHERNET, PIPES, ETC
X-RAY: CONTAINERS
KNOW WHAT’S INSIDE
SPECIFIC DATA
READ THAT METER AT A DISTANCE
CONTROL UI
ON DEMAND USER CONTROLS
IN-WORLD PLACEMENT
Data and the relevant UI can be augmented in context.
© 2018 TWILIO, INC. ALL RIGHTS RESERVED.

12. UNIVERSAL ACCESS AT
THE SPEED OF SIGHT
Context-aware control and data HUD
I N P U T : C A M E R A + C O N T E X T
Additional context awareness
I N P U T : G P S , D I R E C T I O N
Context-dependent Control UI
C O N T R O L S T H AT M A K E S E N S E
See behind doors and walls.
ex: wiring, plumbing or what’s in storage
X- R AY V I S I O N
Project maps of where beacons/
automobiles are for intuitive visualization
P R O J E C T I O N
© 2018 AREALITY3D, INC. ALL RIGHTS RESERVED.
Sight for loading the appropriate view
AReality3D IoT

13. Follow along!
http://iot.AReality3D.com
•
LIVE DEMO APPS

14. © 2018 TWILIO, INC. ALL RIGHTS RESERVED.
Turn real world objects into your IoT primary keys!
@ YO S U N O N A R E A L I T Y 3 D I OT

15. © 2018 TWILIO, INC. ALL RIGHTS RESERVED.
IoT at the speed of sight.
@ YO S U N O N A R E A L I T Y 3 D I OT

16. QUESTIONS?

17. THANK YOU!
LET’S TALK!
TWILIOUSER&DEVELOPERCONFERENCE
I. Yosun Chang
yosun@AReality3D.com
i@permute.xyz
yosun@nusoy.com
@yosun
(415)779-6786
http://iot.AReality3D.com