George Papagiannakis presentation 2014 EVA/Minerva Jerusalem International Conference on Digitisation of Cultural Heritage http://2014.minervaisrael.org.il http://www.digital-heritage.org.il

1. Mobile, cross-platform, life-size animated virtual
characters in indoors and outdoors AR scenes
George Papagiannakis, Greasidou Elissavet, Panos Trahanias, Michalis Tsioumas
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 1

2. Overview
• State-of-the-art in mobile AR
• How many Realities?
• Presence?
• Our previous AR systems for cultural heritage
• Current motivation and research questions
• Methodology
• Results Conclusions
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 2

3. How many Realities? Mixed Reality = AR + VR and AV, CR, IR, HR…
AR AV
VR
MR Reality-Virtuality Continuum
Extent of World Knowledge
Real
Environment
World
Unmodeled
The Mixed Reality (MR) Continuum [Milgram99] [Azuma01]
Virtual
Environment
World Fully
Modeled
CR-IR
HR
AR-VR: Papagiannakis, G., Schertenleib, S., O'Kennedy, B., Poizat, M., Magnenat-Thalmann, N., Stoddart, A., Thalmann, D.,. 2005. Mixing Virtual and
Real scenes in the site of ancient Pompeii. Computer Animation and Virtual Worlds, John Wiley and Sons Ltd 16, 1, 11–24.
Cross-Reality (CR): Davies, C.J., Miller, A., and Allison, C. 2012. Virtual Time Windows: Applying cross reality to cultural heritage. Proceedings of the
Postgraduate Conference on the Convergence of Networking and Telecomunications, ISBN: 978-1-902560-26-7.
Hybrid-Reality (HR): Reda, K., Febretti, A., Knoll, A., et al. 2013. Visualizing Large, Heterogeneous Data in Hybrid-Reality Environments. Computer
Graphics and Applications, IEEE 33, 4, 38–48.
Indirect Reality (IR): Wither, J., Tsai, Y.-T., and Azuma, R. 2011. Indirect augmented reality. Computers Graphics 35, 4, 810–822.
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 3

4. State-of-the-art and recent advances in AR
• Latest research surveys in the field
① Presence and Interaction in Mixed Reality Environments by Egges et al 2007
② A survey of mobile and wireless technologies for AR Papagiannakis et al 2008
③ AR technologies, systems and applications by Carmizziani et al 2011
④ AR in built environment: Classification and implications Wang et al 2013
⑤ Mobile AR survey: a bottom-up approach Huang et al 2013
⑥ Kick-starter camera tracking projects and novel glasses: Meta-View, Occipital,
Atheerlabs, PrimeSense Capri, Google™glasses and recent Samsung, Apple,
Microsoft patents on AR glasses
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 4

5. What is Presence
• “Immersive virtual environments can break the deep, everyday connection between
where our senses tell us we are and where we are actually located and whom we are
with”.
• Typically, the visual fidelity of a VE display is low compared with physical reality.
• The concept of ‘presence’ refers to the phenomenon of behaving and feeling as if we
are in the virtual world created by computer displays [Sanchez-Vives05].
• Another approach is that the sense of ‘being there’ in a VE is grounded on the ability
to ‘do’ there [Sanchez-Vives05] and recently extended to MR [Egges07].
Sanchez-Vives M., and Slater, M., “From presence to consciousness through virtual reality”. Nature Reviews Neuroscience (2005) vol. 6 (4) pp. pp.
332-339
Egges, A., Papagiannakis, G., Magnenat-Thalmann, N., “Presence and Interaction in Mixed Realities”, The Visual Computer, Springer-Verlag, Volume
23, Number 5, May 2007
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 5

6. The importance of Tracking
• Tracking is the basic enabling technology for Augmented Reality
• Without accurate tracking you can’t generate the merged real-virtual
environment
• Tracking is significantly more difficult in AR than in Virtual Environments
track
camera
generate
VR
Papagiannakis, G., Schertenleib, S., O’Kennedy, B., Poizat, M., Magnenat-Thalmann, N., Stoddart, A., Thalmann, D., “Mixing Virtual and Real scenes
in the site of ancient Pompeii”, Journal of Computer Animation and Virtual Worlds, pp. 11-24, vol. 16, issue 1, John Wiley and Sons Ltd,
February 2005
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 6

7. 2. MOBILE AR CULTURAL HERITAGE GUIDE ON A SINGLE
LAPTOP
Papagiannakis, G., Schertenleib, S., O’Kennedy, B., Arevalo-Poizat, M., Magnenat-Thalmann, N.,
Stoddart, A., Thalmann, D., “Mixing Virtual and Real scenes in the site of ancient Pompeii”, Computer
Animation and Virtual Worlds, p 11-24, Volume 16, Issue 1, John Wiley and Sons Ltd, February 2005.
Papagiannakis, G., Magnenat-Thalmann, N., “Mobile Augmented Heritage: Enabling Human Life in
ancient Pompeii”, International Journal of Architectural Computing, Multi-Science Publishing, July
2007, issue 02, volume 05, pp.395-415, 2007.
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 7

8. 3. AR PRESENCE AND INTERACTION BASED ON A SINGLE MOBILE
WORKSTATION
Egges, A., Papagiannakis, G., Magnenat-Thalmann, N., “Presence and Interaction in Mixed Realities“,
The Visual Computer, Springer-Verlag, Volume 23, Number 5, May, 2007.
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 9

9. 4. AR SELF ADAPTIVE ANIMATION ON A UMPC
Chaudhuri, P., Papagiannakis, G., Magnenat-Thalmann, N., “Self adaptive animation based on user
perspective”, The Visual Computer, Springer-Verlag, 24(7-9), pp. 525-533,July 2008
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 11

10. Motivation: Research question I
• Limitations of CG linear algebra
representations
• Points, lines, areas, vector products
• E.g. cross product not defined in 3D
• CG scientists apply several fixes:
• Separate data structures (e.g. quaternions)
• Different combinational programming procedures
(line-sphere-triangle-plane intersections)
• Is there an alternative?
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 15

11. Motivation: Research question II
• In [Kavan et al. 2008] dual quaternions were employed for geometrical
skinning (but not for animation).
• In [Chaudhuri et al. 2008] we employed them for animation blending
• Dual quaternions:
q ˆ= ( q, q, q, q)+ε ( q, q, q, q) = q+εq1234 01020304 0 ε
• where qε and q0 are real quaternions and ε is the dual number where ε2 = 0 . This looks
just like Clifford’s biquaternion q + ωr where ω2 = 0 [Clifford 1873].
• In our recent geometric algebra (GA) [Papagiannakis 2013] framework we
replaced quaternions with GA:
• Character animation blending
• Drop-in, efficient replacement
of existing quaternion spherical interpolation methods
• Can this GA framework be employed in modern ARM architectures for
mobile AR?
[Clifford 1873] CLIFFORD, W.K. 1873. A preliminary sketch of biquaternions. Proc. London Math. Soc. 4:381–395
[Kavan 2008] KAVAN, L., COLLINS, S., ZARA, J., AND CO'SULLIVAN. 2008. Geometric skinning with approximate dual quaternion blending. ACM Transactions on Graphics.
[Chaudhuri 2008] CHAUDHURI, P., PAPAGIANNAKIS, G., AND MAGNENAT-THALMANN, N. 2008. Self Adaptive Animation based on User Perspective. The Visual Computer, Springer-Verlag 24,
7-9, 525–533.
[Papagiannakis 2013] PAPAGIANNAKIS, G. 2013. Geometric algebra rotors for skinned character animation blending. Technical Brief, ACM SIGGRAPH ASIA 2013, Hong Kong, November
2013, 1–6.
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 16

12. Contribution: Main novelties
• Use of GA for encoding
orientations, for mobile AR
devices (ARM architecture)
• The GA rotors e(φ I*α/2n) can
convert any axis α (not only at
the origin) into a rotational
operator.
• Quaternions can be used only on
other quaternions.
• GA rotors are universal operators
capable of rotating other subspaces:
lines, planes, and volumes as first
class operators.
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 19

13. Results on mobile AR
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 20

14. glGA Framework: architecture I
21
Resources
Textures Shader Programs 3D Models
glGA
Utilities
glGAHelper Application
glGARigMesh
glGAMesh
PlatformWrapper
Init
Display
Platforms
Windows Mac Linux IOS
External
Libraries
GLEW
GLFW
AntTweakBar
GLM
Assimp
ImageMagick
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 21

15. Future Work
• Finalize the Roman Forum virtual characters according to
archaeologists
• fully subsume existing different algebras (linear algebra, (dual)
quaternion) under a single representation in a GA algebraic
framework,
• including translation and scaling transformations
• Employ GA rotors for real-time gesture tracking and area light
rotation and rendering
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 24

16. Presence or Realism for digital heritage?
Computer graphics modelling, animation and rendering technology tools demo, Whiskytree.com, ACM
SIGGRAPH 2013
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 25

17. Thank you!
University of Crete Foundation for Research and Technology - Hellas
EVA/MINERVA 2014, Slide 26