Robotic Spatial AR (로봇 공간증강현실 기술 소개)

한국CAD/CAM학회 2014년 정기 학술발표회
세션:Human-Machine Interaction

프로젝터-카메라 시스템을 활용한 사용자 상호작용 기술
공간증강현실 및 로봇으로의 응용을 중심으로
!

User Interaction Techniques based on

Projector-Camera System

Applications in Spatial AR and Integration with Robotics

이주행
ETRI 융합기술연구부문 책임연구원

Agenda for SCCE 2014
• Projector-Camera System in SAR / RSAR

-

Related works including ETRI FRC

-

RSAR: path tracing, augmentation, IR Pen

-

Kinematics, Projection, Hybrid

• Real Examples

SAR: curve design, FTIR canvas, tracking performance

• Control Issues in RSAR

• Q&A
2

Joo-Haeng Lee (joohaeng at etri.re.kr)

AR, SAR, RSAR
• AR

-

Augmented Reality

• SAR

-

Spatial Augmented Reality

• RSAR

-

Robotic Spatial Augmented Reality

3


AR, SAR, RSAR
AR

SAR

RSAR

User views

Monitor

Real world

Real world

Augmentation over

Camera image

Real world

Real world

HW Components

Camera,

Monitor

Camera,

Projector

Camera,

Projector,
Manipulator

Mobility

Static,

Handheld,

Wearable

Static,

Handheld,
Wearable

Dynamic,

Autonomous

4


AR
• Camera for Real World

• Computer Graphics for Virtual World

• Augmentation of the virtual objects (such as

information, UI, ...) on the captured image of the real
world

• A user mainly watches a monitor screen.

• Need to recognize objects in the camera image

• Need to register the two imageries seamlessly
5


Examples of AR
• Vuforia (Qualcomm AR SDK)

6


Examples of AR
• Vuforia (Qualcomm AR SDK)

7


Examples of AR
• World Lens (Quest Visual)

8


SAR
• Projector + AR

-

mobility through pico projectors

-

A user mainly watches a real world rather than a display
device.

-

Usually, projectors and cameras are not motion-controlled:
static, hand-held, wearable

-

Images can be pre-warped before projection

Similar with AR in recognition and registration

are directly
• Virtual objectsthe surface ofaugmented (or
projected) on
real objects.

9


Examples of SAR
• Early works in SAR by Ramesh Raska (MIT)

-

iLamps: Geometrically Aware and Self-Conﬁguring
Projectors (SIGGRAPH 2003)

-

RFIG Lamps: Interacting with a Self-describing World via
Photosensing Wireless Tags and Projectors (SIGGRAPH
2004)

10


•

Examples of SAR
Osaka University (1984~)

11


Examples of SAR
• SixthSense (2009, MIT)

12


Examples of SAR
• OASIS (2010; U Washington Intel Labs)

13


Examples of SAR

14


Examples of SAR

15


Examples of SAR
• RGB-D (2011; U Washington Intel Labs)

16


Examples of SAR
• Build Your World and Play in It (ISMAR 2010; UIUC)

17


Examples of SAR
• Build Your World and Play in It (ISMAR 2010; UIUC)

18


Examples of SAR
• LightSpace (UIST 2010; Microsoft)

19


Examples of SAR
• LightSpace (UIST 2010; Microsoft)

20


Examples of SAR
• OmniTouch (UIST 2011; CMU + Microsoft)

21


Examples of SAR
• OmniTouch (UIST 2011; CMU + Microsoft)

22


•

Examples of SAR
IllumiShare (ACM CHI 2012; Microsoft)

23


•

Examples of SAR
Augmented Projector (Pervasive 2012; MSR UK)

24


Examples of SAR
• MirageTable (ACM SIGCHI 2012; Microsoft)

25


Examples of SAR
• MirageTable (ACM CHI 2012; Microsoft)

26


Examples of SAR
• MirageTable (ACM CHI 2012; Microsoft)

27


Examples of SAR
• Augmented Interactive Cake (Patent ﬁled; 2012;
Disney Research)

28


Examples of SAR
• HideOut (ACM TEI 2013; Disney Research)

29


Examples of SAR
• HideOut (ACM TEI 2013; Disney Research)

30


Examples of SAR
• SAR in Automotive Manufacturing (Virtual Reality
2012; WCL, U South Australia)

31


Examples of SAR
• “Projected Augmented Reality: Keeping Pace with
Innovation” (Intel; 2012)

32


Examples of SAR
• IllumiRoom (ACM SIGCHI 2013; Microsoft)

33


Examples of SAR
• IllumiRoom (ACM SIGCHI 2013; Microsoft)

34


References
• Book

-

O. Bimber and R. Raskar, Spatial Augmented Reality Merging Real and Virtual Worlds, 2005. A.K. Peters

• Tutorial

-

SIGGRAPH Course 2008

35


RSAR
• Robotics + SAR

-

Projectors and cameras can be mounted on robotic
manipulators and/or mobile robot.

-

More DOFs: Kinematics + Image Warping

New type of end effectors and constraints

•
•

direction, position, shape, illumination intensity, ...

soft collision = visibility and occlusion

• Collaboration of multiple autonomous units.
36


Examples of RSAR
• PixelFlex (IEEE Vis. 2001; MIT)

37


Examples of RSAR
• Ubiquitous Display (HCII 2009; Ritsumeikan U Korea U)

38


Examples of RSAR
• LuminAR (2010; MIT)

40


Examples of RSAR
• Beamatron (UIST 2012; Microsoft)

41


Examples of RSAR
• Beamatron (UIST 2012; Microsoft)

42


Examples of RSAR
PixelFlex

(MIT, 2001)

Ubiquitous Display

(Ritsumeikan Univ. , 2006)

LuminAR

(MIT; 2010)

Beamatron

(Microsoft Research, 2012)

Applications of RSAR
• Interaction Techniques

-

as a component of multi-modal interaction

• Precision Applications with Interactive Performance

-

3D measurement

3D Imaging

•

Stereoscopic Photography

44


RSAR in ETRI
• FRC: Future Robotic Computer

• Multimodal UI (User Interface)

-

Speech, Gesture, TTS, Touch, ...

• Intelligence

-

Situation awareness, Autonomous behavior, ...

• Bridging the real and virtual worlds

-

Spatial AR + Robotics + Sensors
45


ETRI FRC 2012

!46


ETRI FRC 2012

!47


ETRI FRC 2012
• Major components for RSAR

• RSAR = Robotic Spatial Augmented Reality

Robotis
Dynamixel
MX-28

Optoma
PK-320

Logitech
HD Pro Webcam
C920

Pico Projectors
• Microvision SHOWWX

-

Laser-based

No need to control focus

10 lumen

• Optoma PK-301

-

DLP

Manual focusing required

20 lumens; 50 on AC
49


Pico Projectors
• Microvision SHOWWX+

-

Laser-based

No need to control focus

15 lumens

• Optoma PK-320

-

DLP

Manual focusing required

25~50 lumens; 100 on AC
50


Pico Projectors

51


Pico Projectors

52


Pico Projectors

53


Pico Projectors

54


Pico Projectors

55


FRC Kinematic Features
• 5 Motors: (tilt + pan) x 2 + (center pan)

-

Redundancy in Pan Rotation

56


FRC Kinematic Features
• 5 Motors: (tilt + pan) x 2 + (center pan)

-

Redundancy in Pan Rotation

57


RSAR Real Experiments
•

RSAR (with Ritsumeikan Univ.)

- IK path following

- Augmenting details

- Interaction with IR Pen

•

SAR

- Planar curve design (ETRI)

- FTIR Canvas (ETRI)

- 3D Tangram (with Kookmin Univ.)

58


•

RSAR IK Path Following
Jacobian IK to follow a path (with Ritsumeikan U)

59


RSAR Augmenting Details

Image in the world:
Body outline

Image from R Prj:
Skeleton
60

Image from L Prj:
Vessels



•

Highlighting with details (with Ritsumeikan U)

61



•

Highlighting with details (with Ritsumeikan U)

62


•

RSAR + IR Pen
Interaction example (with Ritsumeikan U)

63


•

SAR Simple Game
Planar marker-based design game (ETRI)

64


SAR Simple Game

• Planar curve design (ETRI)

65


SAR Markerless Tracking

66


•

SAR FTIR Canvas
TIR (total internal reﬂection)

67


•

SAR FTIR Canvas
FTIR (frustrated total internal reﬂection)

68


SAR FTIR Canvas

69


SAR FTIR Canvas

70


SAR FTIR Canvas

•

71


Veriﬁcation of Homography
Physical

Projected

Tangram Tangram

Physical

Tangram

Projected

Tangram

Projected

Tangram

Physical

Projected

Physical

Tangram Tangram
Tangram

Physical Tangram with

Projection

Projected

Tangram Piece

ETRI SAR Tangram
Physical Tangram with

Projection

RSAR Control Issues
• Kinematic Control

-

Inverse Kinematics

• Projection Control

-

Inverse Projection + Image Pre-warping

• Hybrid Control

-

Current: Kinematics + Projection

Next: Kinematics + Projection + Object Recognition
82


Forward Control
Kinematics

Projection

K+P

Input

Parameters
- angles

- displacements

Image
+

Parameters
- internal, external

Image
+
Parameters

Process

Transformation
- rigid

Transformation
- perspective

Transformation
- rigid

- perspective

Output

End Effector
in a position

Projected Image
on a surface

Projected Image
on a surface

Exception

Collision
- physical objects

Occlusion
- light

- physical objects

Occlusion
+
Collision

83


Backward Control
Kinematics
Input

Projection

K+P

End Effector
in a specific position

Projected Image
on a specific surface

with a specific
condition

Projected Image
on a specific surface

with a specific
condition

Process

Inverse Kinematics

Inverse Projection

IK
+
IP

Output

Parameters
- angles

- displacements

Image
+

Parameters

Image
+
Parameters

Constraint

Collision
- physical objects

Occlusion
- light

- physical objects

Occlusion
+
Collision

84


More Constraints
• Geometry

-

correct geometry without distortion

-

uniform intensity over the surface

-

clear image with minimum blurring

-

maximum visibility and minimum shadowing

• Photometry

• Image

• Visibility

85


•

Kinematics
Kinematic Setup

87


•

Kinematics
Kinematic Setup

88


Forward Kinematics
•

Input

- k: angles of each joint

•

Output

- q: position / orientation of the end-effector

•

Processing

- A: rigid body transformation: translation + rotation

- q = A(k)

89


•

Forward Kinematics
Formulation: A

90


•

Forward Kinematics
Formulation: A

91


Forward Kinematics

92


Inverse Kinematics
•

Input

- q: position / orientation of the end-effector

•

Output

- k: angles of each joint

•

Processing

- A-1: inverse of A; generally, no analytic form.

- k = A-1(q)

93


•

Inverse Kinematics
Setup

94


•

Inverse Kinematics
Partially Analytic IK

- To satisfy that the ray intersects at the base surface, the zcoordinate of the end-effector should be zero: z = 0.

- The length of the ray can be parameterized with tilt angle

100

80

60

40

20
-1.5

95

-1.0

-0.5


•

Inverse Kinematics

- To satisfy that the ray intersects at the base surface, the zcoordinate of the end-effector should be zero: z = 0.

- The length of the ray can be parameterized with tilt angle

96


•

Inverse Kinematics

-0.8
-1.0
-1.2
-1.4
-3

-2

-1

0

1

2

3

-3

-2

-1

0

1

2

3

-0.8
-1.0
-1.2
-1.4

97


•

Inverse Kinematics

98


•

Inverse Kinematics

99


•

Inverse Kinematics
Incremental Differential IK

- Jacobian of transformation matrix: J = dA/dt

- Inverse of Jacobian: J-1

- Jacobian Transpose: JT

100


•

Inverse Kinematics
Inverse of Jacobian: J-1

101


•

Inverse Kinematics
Inverse of Jacobian: J-1

- unstable near limit angles

102


•

Inverse Kinematics
Jacobian Transpose Method: JT

103


•

Inverse Kinematics
Jacobian Transpose: JT

- a bit slower convergence, but more stable

104


•

Inverse Kinematics
Jacobian Transpose: JT

- a bit slower convergence, but more stable

105


Projection

106


Source Image

Projector

Internal Parameters

Projector

External Parameters

Projector

Target Image

Forward

Projection

World

Geometry

World

Knowns

Unknowns

Target Image

World

Source Image

Projector

External Parameters

Projector

Inverse

Projection

Type 1

Internal Parameters

Projector

Geometry

World

Unknowns

Knowns

Source Image

Projector

Target Image

World

External Parameters

Projector

Internal Parameters

Projector

Unknowns

Inverse

Projection

Geometry

World

Type II

Knowns

Example of
Inverse Projection (1)
• How to project the largest undistorted rectangle
to a viewer with a single projector?

-

Inscribed Rectangle

Anamorphic Illusion


Largest Aligned Inscribing Rectangle

!

!

!

111


!

Anamorphic Illusion

112


Artwork 1

Piet Mondrian, Composition A, 1923
Oil on canvas (Galleria Nazionale d'Arte Moderna e Contemporanea, Rome)

Demo
• Simulation in Mathematica

-

Maximum Inscribing Rectangle inside a Projected
Quadrilateral

123


Example of
Inverse Projection (2)
• Projector Pose Estimation

- Given an arbitrary convex quadrilateral, ﬁnd external
parameters of a projector to project it from a
certain rectangle:

-

Which convex quadrilateral cannot be projected
from a rectangle?

-

How to modify a quadrilateral to be projectable?

Quiz #1
Assume that you have a hand-held projector which can
accept an rectangular source image. Which of the
following convex quadrilaterals is projectable from your
projector?
(a)

(c)

Rhombus

Trapezoid___

(b)

Parallelogram

(d)

Isosceles

Trapezoid

125


Type of Quads

126


Quiz #2
(a)

(b)

(c)

(d)

127


Quad: Configuration
• Ext Params

-

center of projection

• Int Params

-

pc
Qs

filed-of-view

v2

• Source Quad

-

rectangle (cf) aspect ratio

-

convex quadrilateral

v3

ms

v1

m

• Projected Quad

v0

• Length-related coefficients: α , β and γ for each diagonal
i

128

i

i


Quad: Analytic Solution
• Constraint 1: cop

!

d0 =

!

cosθ 0

α0

=

cosθ1

= d1

α1

pc
y1 y0

• Constraint 2: fov

!

tanψ 0 =
!

tanθ 0

β0

=

tanθ1

β1

Qs
ms

v2

y0y1

q1

= tanψ 1

v3

m

q0

v1

• 2 Eqs and 2 Unknowns

-

Existence of solution?

v0

Geometric meaning?
129


Equation

• Constraint 1: cop

!

d0 =

!

cosθ 0

α0

=

cosθ1

cosθ 0 = ±

= d1

α1

A=

• Constraint 2: fov

!

tanθ 0

!

β0

tanψ 0 =

=

tanθ1

β1

B=

= tanψ 1

• 2 Eqs and 2 Unknowns

-

C=

Existence of solution?

C 2 −1
A B −1
2

α1
α0
β1
β0
γ1
γ0

Coefﬁcients

Geometric meaning?
130


Equation

Coefﬁcient

Conditions

cosθ 0 = ±

A ≥ 1,#B ≤ 1, C ≤ 1
OR
2

2

2

A=

A ≤ 1,#B ≥ 1, C ≥ 1
2

2

2

B=
C=

C 2 −1
A B −1
2

α1
α0
β1
β0
γ1
γ0

Coefﬁcients
131


A ≥ 1,#B ≤ 1, C ≤ 1
OR
2

2

2

A 2 ≤ 1,#B 2 ≥ 1, C 2 ≥ 1

Quiz #2

Diagonal Parameters: (d 2 , ρ, t 1 , t 2 )
2
2
2
Solution Coefﬁcients: ! A ,$B , C '

(a)

(b)
(1.3, π / 2, 0.75, 0.3)

(1.3, π / 2, 0.6, 0.2)

(0.3, 1.5, 0.4)

(c)

(12.0, 0.11, 1.33)

(d)
Projectable
(1.3, π / 2, 0.6, 0.3)

(1.3, 0.7, 0.6, 0.2)

(12.0, 0.11, 1.33)

132

(3.1, 0.3, 0.8)


•

Hybrid Control
Inverse Kinematics + Inverse Projection

133


•

Scenario
A big image is augmented with a partial details.

134


•

Scenario

135


•

Scenario

136


•

Scenario

137


•

Scenario

138


•

Scenario

139


•

Scenario

140


•

Scenario

141


Wrap Up: Controls
Inverse

Kinematics

Inverse

Projection

142

Hybrid

Control


Summary
• Projector-Camera System in SAR / RSAR

-

Related works including ETRI FRC

• Real Examples

- RSAR path tracing, augmentation, IR Pen

- curve design, FTIR canvas

• Control Issues in RSAR

-

Kinematics, Projection, Hybrid
143


QA
joohaeng at etri dot re dot kr

Robotic Spatial AR (로봇 공간증강현실 기술 소개)

More Related Content

Similar to Robotic Spatial AR (로봇 공간증강현실 기술 소개)

More from Joo-Haeng Lee

Recently uploaded

Robotic Spatial AR (로봇 공간증강현실 기술 소개)