1. Introduction Design Implementation Conclusion
Correcting Projector Display for
Multi Planar Surfaces
by
Nilesh Heda
under the guidance of
Prof. Sharat Chandran
Computer Science and Engineering Department
Indian Institute of Technology, Bombay
July 7, 2005
Nilesh Heda MTP Final Presentation

2. Introduction Design Implementation Conclusion
Outline
1 Introduction
Motivation
Related Issues
Problem Statement
Our Contribution
2 System Design
Projector-camera System
Display on Irregular Surface
Constraint Homography
3 Implementation and Results
System Requirements
Modules of the system
Results
4 Conclusion
Future Work
Nilesh Heda MTP Final Presentation

3. Introduction Design Implementation Conclusion Motivation Issues Problem Contribution
Outline
1 Introduction
Motivation
Related Issues
Problem Statement
Our Contribution
2 System Design
Projector-camera System
Display on Irregular Surface
Constraint Homography
3 Implementation and Results
System Requirements
Modules of the system
Results
4 Conclusion
Future Work
Nilesh Heda MTP Final Presentation

4. Introduction Design Implementation Conclusion Motivation Issues Problem Contribution
Conceptual Sketch of “Office of the Future”
Office of Future
Projectors creating virtual
office environment
Display anywhere
Teleconferencing
Nilesh Heda MTP Final Presentation

5. Introduction Design Implementation Conclusion Motivation Issues Problem Contribution
Issues Regarding Projector Based System
Geometric Issues
Continuous Display Surface
Projector
Projector Projector
Planar Display Surface Discontinuous Display Surface
Single Surface Display Multi Surface Display
Image Intensity and color
Non-uniform Intensity
Overlapped region
Focussing of projected image
Nilesh Heda MTP Final Presentation

6. Introduction Design Implementation Conclusion Motivation Issues Problem Contribution
Issues Regarding Projector Based System
Geometric Issues
Continuous Display Surface
Projector
Projector Projector
Planar Display Surface Discontinuous Display Surface
Single Surface Display Multi Surface Display
Image Intensity and color
Non-uniform Intensity
Overlapped region
Focussing of projected image
Nilesh Heda MTP Final Presentation

7. Introduction Design Implementation Conclusion Motivation Issues Problem Contribution
Problem Statement
The aim of our project is to create a projector based system for
correctly displaying on irregular surfaces.
Nilesh Heda MTP Final Presentation

8. Introduction Design Implementation Conclusion Motivation Issues Problem Contribution
Problem Statement
The aim of our project is to create a projector based system for
correctly displaying on irregular surfaces.
Irregular Surface
Single Planar
Multi Planar
Curved
Combination of above
Nilesh Heda MTP Final Presentation

9. Introduction Design Implementation Conclusion Motivation Issues Problem Contribution
Problem Statement
The aim of our project is to create a projector based system for
correctly displaying on irregular surfaces.
Problems with projected image
Focussing of projected
image
Image Intensity and color
Geometric issues
Nilesh Heda MTP Final Presentation

10. Introduction Design Implementation Conclusion Motivation Issues Problem Contribution
Problem Statement
The aim of our project is to create a projector based system for
correctly displaying on irregular surfaces.
Problems with projected image
Focussing of projected
image
Image Intensity and color
Geometric issues
Nilesh Heda MTP Final Presentation

11. Introduction Design Implementation Conclusion Motivation Issues Problem Contribution
Overview of work done
Our Contribution
We have been successful in correctly displaying on
irregular surfaces.
Our algorithm doesn’t require any knowledge regarding
surface geometry.
We have proposed an iterative algorithm for satisfying the
constraint between homographies.
We have also sketched out a conceptual design for
shooting range simulator which is based on our method.
Nilesh Heda MTP Final Presentation

12. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Outline
1 Introduction
Motivation
Related Issues
Problem Statement
Our Contribution
2 System Design
Projector-camera System
Display on Irregular Surface
Constraint Homography
3 Implementation and Results
System Requirements
Modules of the system
Results
4 Conclusion
Future Work
Nilesh Heda MTP Final Presentation

13. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Projector-camera Based System
β
Projector(s)
X
Z
Y
Screen
xz
y
α
Projectors
Display surface
Stationary cameras
Projector-screen-camera
calibration.
Nilesh Heda MTP Final Presentation

14. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Projector-camera Based System
Camera
Projector(s)
X
Z
Y
Screen
xz
y
α
β
Projectors
Display surface
Stationary cameras
Projector-screen-camera
calibration.
Camera makes system active by
adding visual feedback.
Nilesh Heda MTP Final Presentation

15. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Projector-camera Based System
Projector(s)
X
Z
Y
xz
y
α
β
pHc
pHs
cHs
Screen
Camera
Projectors
Display surface
Stationary cameras
Projector-screen-camera
calibration.
Projector as a dual of camera.
Nilesh Heda MTP Final Presentation

16. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Projector-camera Homography
Camera and projector optics can be modeled as
projectivities.
We need to find mapping between P(xi, yi) in projector
coordinate system and C(Xi, Yi) in camera coordinate
system.
Homography is a mapping between two perspective
images of scene from two different viewpoints.
Mapping can be expressed in terms of a projective
transform P =p HcC.


xiw
yiw
w

 =


h1 h2 h3
h4 h5 h6
h7 h8 h9




Xi
Yi
1


Nilesh Heda MTP Final Presentation

17. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Finding Projector-camera Homography
Steps for recovering pHc
1 Display a calibration pattern (4 points (P1, P2, P3, P4)).
2 Capture this calibration pattern by camera.
3 Detect and measure Feature points (C1, C2, C3, C4) from
captured camera image using standard computer vision
techniques.
4 Determine the point correspondence between Pi ↔ Ci.
5 Using these four point correspondences, recover pHc.
Nilesh Heda MTP Final Presentation

18. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Camera-Screen Homography
Camera Screen Homography CHS
Mapping between pixels in projected display on the screen
and pixels in the camera image
Can be computed by point correspondence of known point
on screen with those in camera image
Nilesh Heda MTP Final Presentation

19. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Projector-Screen Homography
Projector-Screen Homography pHs
Mapping between pixels in projector input and pixels on
projected output on the screen
pHs = cHs.pHc
Nilesh Heda MTP Final Presentation

20. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Applying Pre-warp to input image
Compute the inverse homography pH−1
s = sHp
pre-warp the input signal to the projector
After Pre−warping
1
P
1
P
2 P
2
P
3P
4
P
4
p
1
p
1
p
2
p
2
P
3
p
4 p
3
p
3
p
4
Projector Image Projector ImageCamera Image Camera Image
Before Pre−warping
P
Nilesh Heda MTP Final Presentation

21. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Irregular Display Surface
Assumptions
A small quadrilateral can be considered as basic unit.
Surface can be reconstructed by tiling these small
quadrilaterals.
Need to determine point correspondence for all grid points
Nilesh Heda MTP Final Presentation

22. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Irregular Display Surface
Assumptions
A small quadrilateral can be considered as basic unit.
Surface can be reconstructed by tiling these small
quadrilaterals.
Need to determine point correspondence for all grid points
Nilesh Heda MTP Final Presentation

23. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Grid Point Detection Algorithm
Previous Approaches
Use different color for each grid points
Not a robust method.
Vertical/horizontal scan
Structured light
O(n) steps for n × n grid points.
Question ?
Can it be done in more faster way i.e. in O(log n) steps?
Nilesh Heda MTP Final Presentation

24. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Binary Encoding based Algorithm
Algorithm
Assign Unique binary code to all Projector feature Points
At each step i, the display of particular projector feature
point depends on ith bit.
Observe them by camera and detect feature points.
Set ith bit of code of these camera feature points.
After completion of logn2 steps, we will have set of n2
camera feature points each having unique code
Remove code with all bits ones as noise.
Find n2 point correspondences between projector and
camera feature points by simply matching the code.
Nilesh Heda MTP Final Presentation

25. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Binary Encoding based Algorithm
Example
For computing point correspondence of 1024 × 1024 grid, we
need steps 20 steps.
Nilesh Heda MTP Final Presentation

26. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Computation of Individual Homography
Compute n2 feature point correspondences.
For each quadrilateral i, Compute homography i
pHs from
the projector to the screen independently.
Problem
These homographies may not be consistent along the
boundaries between quadrilaterals.
Nilesh Heda MTP Final Presentation

27. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Computation of Individual Homography
Compute n2 feature point correspondences.
For each quadrilateral i, Compute homography i
pHs from
the projector to the screen independently.
Problem
These homographies may not be consistent along the
boundaries between quadrilaterals.
Nilesh Heda MTP Final Presentation

28. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
2 × 2 Grid Setup
Assumption
Unit quadrilaterals are extremely small.
Instead of considering all points at the
boundaries between adjacent
quadrilaterals, we consider only the
vertexes of the (tiny) quadrilaterals.
X ↔ xi ≡i
s HpX
x
p s
1
H 4
p
2
3
p s
pHs
Hs
X
H
Nilesh Heda MTP Final Presentation

29. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
2 × 2 Grid Setup
We would like to get
x1 = x2 = x3 = x4.
Example
This may not be the case due
to image processing errors in
the detection of grid points. x
p s
1
H 4
p
2
1
s
2
s
4
s
3
p s
pHs
p
3
s H
Hs
Hp pH
Hp
X
X1
X3 X4
X2
x2x1
x3 x4
H
Nilesh Heda MTP Final Presentation

30. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Approach I - Median
Median Approach
Choosing x as the median (instead of the average) of
x1, x2, x3 and x4.
Gives an acceptable solution even if two homographies are
incorrectly computed.
Nilesh Heda MTP Final Presentation

31. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Approach II - Iterative algorithm
2
P1 P2 P3
P4 P5 P6
C1 C2 C3
C4 C5 C6
Projector Image Camera Image
pHs pHs
1
1
PHS and 2
pHS are homographies mapping between two
surfaces to the projector.
Nilesh Heda MTP Final Presentation

32. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Approach II - Iterative algorithm
p1 p2 p3
p5
c1 c2 c3
c6c5c4p4 p6
Projector Image Camera Image
pHs pHs
1 −1 2 −1
1
PH−1
Sx =2
P H−1
Sx
Nilesh Heda MTP Final Presentation

33. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Approach II - Iterative algorithm
C1
P6
C2 C3
C4 C5 C6
Projector Image Camera Image
pHs
1
P3
P5
P1 P2
P4
Compute 1
PHS using four point correspondence (P1 ↔ C1,
P2 ↔ C2,P4 ↔ C4,P5 ↔ C5).
Nilesh Heda MTP Final Presentation

34. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Approach II - Iterative algorithm
−1
P2
P4 P6
C2 C3
C4 C5 C6
Projector Image Camera Image
pHs
1
P1 P3
P5
C1
c5
c3c1
c4
c2
c6
p1
p4 p5
p2
Compute p1, p2, p4, p5 from c1, c2, c4, c5 using equation
1
SHP
−1
c = p.
Nilesh Heda MTP Final Presentation

35. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Approach II - Iterative algorithm
C6P4 C4
Projector Image Camera Image
P1 C1
c1
c4 c6
p1
p4
2
pHs
P2
p2
p5
P5 P6
P3 C2 C3
c2
c3
c5
C5
Compute 2PHS using six point correspondence (P2 ↔ C2,
P3 ↔ C3, P5 ↔ C5, P6 ↔ C6, 1
SHP
−1
c2 ↔ c2, 1
SHP
−1
c5 ↔ c5)
Nilesh Heda MTP Final Presentation

36. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Approach II - Iterative algorithm
p3
P2
P4 P6
C2 C3
C4 C5 C6
Projector Image Camera Image
P1 P3
P5
C1
c5
c3c1
c4
c2
c6
p1
p4
pHs
−12
p2
p5 p6
Update value of p2 and p5 and compute values of p3, p6 from
c2, c5, c3, c6 using equation 2SHP
−1
c = p.
Nilesh Heda MTP Final Presentation

37. Introduction Design Implementation Conclusion Proj-cam Irregular constraint
Approach II - Iterative algorithm (contd.)
As we have used point correspondence 1
SHP
−1
c2 ↔ c2, we
can say that
1
PHS
−1
c2 =2
P HS
−1
c2 (1)
But, as computation of 2
PHS involves least square
approximation, above equation may not hold true.
Recompute the value of 1
PHS using constraint from 2
PHS.
After some iteration, the solution will converge and above
equation will be satisfied.
Nilesh Heda MTP Final Presentation

38. Introduction Design Implementation Conclusion Requirements Modules Result
Outline
1 Introduction
Motivation
Related Issues
Problem Statement
Our Contribution
2 System Design
Projector-camera System
Display on Irregular Surface
Constraint Homography
3 Implementation and Results
System Requirements
Modules of the system
Results
4 Conclusion
Future Work
Nilesh Heda MTP Final Presentation

39. Introduction Design Implementation Conclusion Requirements Modules Result
Hardware/Software Requirements
Projector - Sharp XR-1X Multimedia Projector, 1100 ANSI
lumen, maximum native resolution of 1024 × 768.
Camera - D-Link DSC 350F camera, USB based,
640 × 480, 320 × 240 and 160 × 120 pixel resolution.
Platform - Windows 9x/NT/XP.
Programming Language and Libraries
VC++ for GUI handelling
DirectShow for video capture and filter graph generation
Direct3D for generation of virtual world.
Intel OpenCV for Computer Vision algorithms
Nilesh Heda MTP Final Presentation

40. Introduction Design Implementation Conclusion Requirements Modules Result
Input/Output of the System
Input
Texture Data
Image files or .x files or Video files
Output
Self adjusted projected image according to
projector-screen-camera position and orientation.
Nilesh Heda MTP Final Presentation

41. Introduction Design Implementation Conclusion Requirements Modules Result
System Diagram
Nilesh Heda MTP Final Presentation

42. Introduction Design Implementation Conclusion Requirements Modules Result
Module details
Capture Module
Captures frame from camera using Microsoft DirectShow at
640 × 480 resolution with 15 fps.
Uses callback functions for further computation.
Camera Calibration Module
Computes lens distortion using calibrated checker board
patterns
Radial Distortion
Tangential distortion
One time process for a particular camera
Nilesh Heda MTP Final Presentation

43. Introduction Design Implementation Conclusion Requirements Modules Result
Modules details
Mapping of Camera Space to Global Space
User specifies 4 fiducial in camera image using mouse
Computes Camera-screen homography CHS
Structured Light Registration
Uses gaussian blob as our feature point.
Determines center of blob with sub-pixel accuracy.
At End, the system has n2
features mapping from projector
pixels to the global coordinate system.
Nilesh Heda MTP Final Presentation

44. Introduction Design Implementation Conclusion Requirements Modules Result
Modules details
Homography Computation
Computes individual homography i
PHS using least square
error SVD method.
Applies iterative recursive algorithm for constraint
homography solution.
Rendering Module
Uses Microsoft Direct3D for rendering.
Uses texture rendering techniques for rendering image data
on surface.
We were able to get 45fps at 1024 × 768 resolution with
5000 polygons on standard PC without any High end
graphics card.
Nilesh Heda MTP Final Presentation

45. Introduction Design Implementation Conclusion Requirements Modules Result
GUI
GUI
Nilesh Heda MTP Final Presentation

46. Introduction Design Implementation Conclusion Requirements Modules Result
2 × 2 grid for a planar surface
Grid patterns
Nilesh Heda MTP Final Presentation

47. Introduction Design Implementation Conclusion Requirements Modules Result
2 × 2 grid for a planar surface
Camera Point correspondence Planar Image after applying
Homograhy
Nilesh Heda MTP Final Presentation

48. Introduction Design Implementation Conclusion Requirements Modules Result
Display on 2-planar surface
Display Surface Distorted image
Corrected Image
Nilesh Heda MTP Final Presentation

49. Introduction Design Implementation Conclusion Requirements Modules Result
Display on 3-planar surface
Display Surface Distorted image
Corrected Image
Nilesh Heda MTP Final Presentation

50. Introduction Design Implementation Conclusion Requirements Modules Result
Keystone Correction
Keystone correction - before and after
Nilesh Heda MTP Final Presentation

51. Introduction Design Implementation Conclusion Requirements Modules Result
Constraint Homography
Constraint homography - Without and with
Nilesh Heda MTP Final Presentation

52. Introduction Design Implementation Conclusion Requirements Modules Result
Video on irregular Surface
Nilesh Heda MTP Final Presentation

53. Introduction Design Implementation Conclusion Requirements Modules Result
Video on irregular Surface
Nilesh Heda MTP Final Presentation

54. Introduction Design Implementation Conclusion Future Work
Outline
1 Introduction
Motivation
Related Issues
Problem Statement
Our Contribution
2 System Design
Projector-camera System
Display on Irregular Surface
Constraint Homography
3 Implementation and Results
System Requirements
Modules of the system
Results
4 Conclusion
Future Work
Nilesh Heda MTP Final Presentation

55. Introduction Design Implementation Conclusion Future Work
Conclusion
Successfully accomplished task of correcting projector
display on irregular surface.
Proposed an iterative refinement algorithm for satisfying
constrained between homography.
Nilesh Heda MTP Final Presentation

56. Introduction Design Implementation Conclusion Future Work
Future work
Improvement in current system
Deciding optimum grid size
Accurate detection of blurred feature points
Improving Iterative algorithm
Implementation of Shooting Range Simulator
Nilesh Heda MTP Final Presentation

57. Introduction Design Implementation Conclusion Future Work
Demo
Nilesh Heda MTP Final Presentation

58. Introduction Design Implementation Conclusion Future Work
Question and Answers
Thank You.
Nilesh Heda MTP Final Presentation

59. Introduction Design Implementation Conclusion Future Work
Shooting Range
Practicing and testing
shooting skills of shooter.
Pre-defined target
Accuracy of shooting
depends on distance
between target and actual
hit.
Important issues
Accurate detection.
Real-time response.
Nilesh Heda MTP Final Presentation

60. Introduction Design Implementation Conclusion Future Work
Our Approach
Screena(x,y) a’(x’,y’)
Camera Projector
Soldier
Projector creates
simulated environment.
Shooter(s) are sitting in a
simulated environment.
Shoots with the laser-gun
on a screen.
The camera(s) captures
the hit screen.
Nilesh Heda MTP Final Presentation

61. Introduction Design Implementation Conclusion Future Work
Important Components
Virtual World
Created by system.
Rendered as First Person View(FPV).
Shooter
Positioned in front of screen.
Shoots with Laser-pointer gun.
Projector-camera system
Projector projects virtual world.
Camera detects the hit.
Nilesh Heda MTP Final Presentation

62. Introduction Design Implementation Conclusion Future Work
Procedure for detection I
1 Compute H and H−1.
2 Create the Virtual world and render the scene from a first
person view (FPV).
3 Pre-warp the render scene by pre-computed H−1 and
project it.
4 Find the background intensity from first frame of each shot.
5 Do the Background Subtraction(Ideally zero).
6 Position the soldier (shooter) in front of display surface,
facing the screen.
7 Ask shooter to shoot the specified target on screen by
laser-gun ( Gun with laser pointer attached).
8 If laser pointer is displayed on screen, we will get change
in the color value in camera image.
Nilesh Heda MTP Final Presentation

63. Introduction Design Implementation Conclusion Future Work
Procedure for detection II
9 Find the region with high value after background
subtraction and mark its centroid as laser pointer position.
10 After detecting the laser pointer position, compute the
distance between actual target coordinate and detected hit
coordinate for finding accuracy of shooting.
Nilesh Heda MTP Final Presentation

64. Introduction Design Implementation Conclusion Future Work
Extension for invisible laser
The hit on screen provided visible feedback for next shot.
Use infra-red laser and additional infra-red camera.
Given a infra-red laser dot hit in infra-red camera image,
we need to determine where it lies in projector world.
Compute infra
c Hs by observing four fiducial points in
infra-red camera.
infra
p Hc =infra
c H−1
s.pHs (2)
Nilesh Heda MTP Final Presentation

65. Introduction Design Implementation Conclusion Future Work
Extension for Multiple Shooter Environment
Factors under consideration
Accurate real time detection of individual laser pointers
Each soldier may have different view point for looking at
screen
We might be required to use multiple projector for
displaying on large area (10 × 3 )
Network based solution
Multiple instances of single shooter system connected to a
centralized rendering system through network.
The centralized rendering system will provide the rendered
view to all single shooter systems from their view point.
Single View solution
n different infra-red cameras configured for n different
frequencies.
Each infra-red camera will capture laser of particular
frequency.
Nilesh Heda MTP Final Presentation

66. Introduction Design Implementation Conclusion Future Work
Multi-Planar Display
Before correction After correction
Nilesh Heda MTP Final Presentation

67. Introduction Design Implementation Conclusion Future Work
Multi-projector Display System
Images from individual projector
Image before and after brightness correction
Nilesh Heda MTP Final Presentation