Presentation at Salento AVR 2015

1. Creativity Support
In Projection-based Augmented
Environments
Bruno Simões, Federico Prandi, and Raffaele de Amicis (et.al.)
F. Graphitech
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

2. Framework
Problem
Statement
Introduction &
Context
Conclusions
1
2
3
4
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Agenda
Path of Enlightenment

3. Low
Cost
Creativity
Communication
Culture
Content Creation
Collaboration
Crowdsourcing
to develop new creative tools (based on computer vision
techniques and pico-projection) that can transform the
surrounding environment into an user interface
Introduction
c-Space Objective
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

4. Low
Cost
Creativity
Communication
Culture
Content Creation
Collaboration
Crowdsourcing
to develop new creative tools (based on computer vision
techniques and pico-projection) that can transform the
surrounding environment into an user interface
Introduction
c-Space Objective
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

5. What is Creativity?
A. Einstein
“seeing what everyone else has seen, and
thinking what no one else has thought”
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Introduction
What is Creativity?

6. What is Creativity?
Betty Edwards
“creativity is the ability to find new solutions
to a problem or new modes of expression;
thus it brings into existence something new
to the individual and to the culture.”
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Introduction
What is creativity?

7. Presentation
and Interaction
Crowdsourcing and
gamified data collection 4D
Reconstruction
Introduction
c-Space State-of-Art
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)

8. Introduction
c-Space Objective
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
What About Creativity

9. Introduction
c-Space Objective
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
What About Creativity
• Knowledge transfer: How
mechanisms implemented promote
interaction between users
• Germinality: how well does it open
up to new perspectives
• Does it really new ways of
expression

10. Introduction
c-Space State-of-Art Limitations
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Limitations:
- Simple user experience (AR)
- 2DoF (storylines)
- Ergonomics
- Embarrassment factor
- …

11. to develop new creative tools (based on computer vision
techniques and pico-projection) that can transform the
surrounding environment into an user interface
Culture
Low
Cost
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Crowdsourcing
Creativity
Content Creation
Collaboration
Communication
Introduction
c-Space Objective
to develop new creative tools (based on computer vision
techniques and pico-projection) that can transform the
surrounding environment into an user interface
Display limitation ?

12. Projection-Based Augmented Reality
Uniting physical and virtual objects
“combination of physical reality with computer
generated data projected directly within or on the
user's physical space”
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Introduction
Projection-based AR

13. c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Introduction
Projection-based AR
Projection Mapping vs Traditional Projection
Uses the projection surface as
a larger screen
Customizes the appearance
of an object/environment
Spatial AR
uses real physical objects to both represent
and interact with computer-generated
information
Tangible User Interfaces (TUI)
+

14. Introduction
Possible Advantages
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
• Real objects can be physically handled
• Does not require see-through displays or to
wear a head-mounted display
• Multi-user experience
• Facilitates co-located collaboration
• Ergonomics, e.g. might reduce physical stress

15. 1. Visual creation of augmented
environments and automatic recomputation
of projection maps on camera movements
Projective augmented reality using mobile devices raises
calibration issues due to mobility that must be addressed.
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Research Problem
Projection-based AR

16. 2. Automatic Background Color
Compensation
Unfortunately, without accommodating for the different
background colors, the user may be subject to viewing
inaccurate colors and reduced contrast
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Research Problem
Projection-based AR

17. 3. Visual Pollution, Privacy & Human Safety
The invasiveness of projected content can lead to "visual
pollution" or bring annoyance to other people in the
vicinity. Additionally projected content can blind people.
Projective interfaces exposes the end-user to the risk of
projecting sensitive information, e.g. PM popups.
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Research Problem
Projection-based AR

18. 4. Find effective new forms of user
interaction
We want the surrounding environment itself to be
transformed into an interface for the users to interact
through gestures, wearables or simply by moving around.
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Research Problem
Projection-based AR
Collaboration and Information Exchange!

19. c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Framework
Research Problem 1
Solution to visually create new augmented
environments and to automatically re-
compute projection maps on camera
movements

20. c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Framework
PPP Setup
Marker-less tracking,
perspective
correction, and
safety monitoring
computer generated content
real
worldPico-projector
processing unit + WIFI
Camera
projection
Setup(Client)

21. c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Framework
PPP Setup
Server
Client-Server
real world
Normative policies,
collaboration features
descriptors, video samples, 3D
mapping setups, visible objects

22. c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Framework
Mobile Editor
Setupprojectionmapping
Mapping handlersVisual Object

23. c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Framework
PPP Procedure
Client-Server
Device
AR
Output
Query
State
Object
Update
Application
Logic
Render
Graphics
Cloud/Device
Target Database
Camera
Camera
Frame
Pixel Format
Conversion
Converted
Frame
User-
Defined
Targets
Image
Targets
Frame
Markers
Multi-Image
Targets
Cylinder
Targets
Word
Targets
Virtual
Buttons
Detect New Objects
Track Detected Objects
Evaluate Virtual Buttons
Render
Camera
Preview
ServerServer
Projector
calibration
& perspective
correction
collaboration
BCC
Descriptors Stack

24. c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Framework
Background Color Compensation
Rendering
Lights Data
Color
Correction
System
Camera
Frame
Buffer
Color
Correction
Frame
Buffer
Light
Detection
System
Gamma function
Estimation, Homer H.
Chen

25. How can we create an easy-to-use normative
policy to automatically regulate the use and
the power of PPP technology?
e.g. streets, where drivers or passengers could be temporarily
blinded by the projection.
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Framework
Research Problem 3

26. c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Framework
Research Problem 3
imagga
Automated image tagging and keywording

27. During the initial user trials, a small group of
participants reported:
1. No difficulty in learning the techniques that were
proposed to setup a 3D projection mapping
2. They found appealing the ability to combine
information in a shared workspace.
3. Found useful the feature to download content to
their smartphones from projected animated QR
codes
4. Picking sometimes was difficult to archive
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Conclusions
Test Case Results

28. 1. Current pico-projectors have a luminance between 5 and
100 lumens. Hence, these projectors can only be used
indoors or outdoors at night.
2. The use markerless descriptors under certain conditions
can be an issue:
1. Poor illumination of the environment
2. Surfaces with few features, e.g. white halls, etc.
3. Projection without auto-focus can be disruptive
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Conclusions
Biggest Limitations

29. 1. Blending multiple projections into a seamless projection
2. Restrict the projection in such way that it will never hit
people in the face, but while keeping the projecting
around the person.
3. Better way to impose normative policies: executed at a
real time frame rate.
4. Explore interaction concept with drones, e.g. Wristwatch
Drone.
5. Drop the back-server requirement when the use of QR
codes is possible
c-Space
Grant agreement no: 611040
fostering new creative paradigms based on recording and sharing “casual” videos through the internet.
The 2nd International Conference on Augmented and Virtual Reality (SALENTO AVR 2015)
Conclusions
Future Work

30. For your attention bruno.simoes@graphitech.it
@brunogsimoes
linkedin.com/in/brunogsimoes
Grant agreement no: 611040