This document discusses tele-immersion technology, which allows users in different locations to interact in a simulated holographic environment. It provides a history of the concept dating back to 1965, and describes how tele-immersion works using camera clusters to capture 3D environments, reconstruct 3D models, compress the data, transmit it over networks, and allow remote users to interact in the virtual space in real-time. Basic requirements like high-performance displays, computers, tracking sensors and networks are needed to support the technology. Potential applications include remote collaboration and future developments may enable touch interactions through haptic sensors.

1. TELE-IMMERSION

2. FLOW OF PRESENTATION
 Introduction
 History
 Tele-Immersion v/s Tele-Conference and Virtual Reality
 Working
 Basic Requirements
 Applications
 Future Applications
 Problems
 Bibliography

3. INTRODUCTION
• Will be implemented with Internet2.
• Enable users in different geographic locations to interact in a
simulated holographic environment.
• Combines the display and interaction techniques of virtual reality
with latest vision technologies
• Involves the ultimate synthesis of media technologies:
3D environment scanning
Projective and display technologies
Tracking technologies
Audio technologies
Powerful networking
1/32

4. HISTORY
• In 1965, Ivan Sutherland, proposed the ‘ultimate display’
• Allowing the user to experience a completely computer rendered
environment.
• In 1998, Abilene, a backbone research project, was launched and now
serves as a base for Internet2 research
• Internet2 needed tele-immersion to stretch its networks’ capabilities.
• So, the national tele-immersion initiative was formed in May 2000.
• Researchers at the Universities of North Carolina (UNC), the
Universities of Pennsylvania and advanced network and services
reached a milestone in developing this technology.
• In 2000, the first experiment was conducted.
2/32

5. Tele-immersion v/s Tele-
Conference
In tele-immersion
• Differs in user’s view as
• Remote environment changes dynamically as he moves his head.
• Eye contact is maintained
3/32

6. Tele-immersion v/s Virtual Reality
• Virtual reality allows you to move in a computer-generated 3-D
environment.
• Tele immersion can only create a 3-D environment that you can see
but not interact with.
4/32

7. 5/32

8. 2D to 3D…
Image Acquisition
3D Reconstruction
3D Video Transmission
6/32

9. IMAGE ACQUISITION
• Each camera generates an image from its point of view many times
in a second
Depth map Texture map
• Uses cluster of cameras each of this having 4 cameras (three b/w &
one colour).
• Have own dedicated high-end computer to perform the image
rectification, pixel correlation, and triangulation to recover depth
values from the sets of images produced.
7/32

10. - camera cluster
• Each set of the images taken at a given instant is sorted into subsets
of overlapping trios of images.
8/32

11. • From each trio of images, a “disparity map” is calculated,
reflecting the degree of variation among the images at all points in
the Disparity map visual field.
• The disparities are then analyzed to yield depths that would
account for the differences between what each camera sees.
• Depth maps are combined into a single viewpoint independent of
the scene at a given moment.
(a)one of the original video frames, (b) corresponding depth map,
(b)(c) shaded view of the 3D reconstruction, (d) view of the textured 3D
9/32
model.

12. 3D CONSTRUCTION
• The algorithm used for 3D image construction is Trinocular Stereo
Algorithm.
• Here, first the background is subtracted
• The foreground where people are moving and doing their work is
kept as it is.
10/32

13. • A sequence of N (2 or more) background images Bi are acquired in
advance of each session.
• From this set we compute a pixel-wise average background image
B = 1/N Σi Bi
• We then compute the average pixel-wise difference between B and Bi ,
D = 1/N Σi (B - Bi).
• During a tele-immersion session each primary image I is subtracted
from the static mean background ID = B - I, a binary image is formed
via the comparison IB = ID > T × D
• where T is a configurable threshold (generally T = 7).
Background image, foreground image and subtracted result
11/32

14. • The reconstruction algorithm begins by grabbing images from 3
strongly calibrated cameras.
• The system rectifies the images so that their epipolar lines lie along the
same horizontal image rows to simply matches.
• For each pixel (u,v) in the left image, MNCC produces a correlation
profile c(u,v,d) where disparity d ranges over acceptable integer values.
• Selected matches are maxima in this profile, which satisfy various
‘peak’ characteristics.
• The trifocal constraint refines or verify correspondences
• improve the quality of stereo range data
• .gives s hypothesized match [u,v,d] in a pair of images
• A hypothesis is correct if the epipolar lines for the original point [u,v]
and the hypothesized match [u,d,v], intersect in the third camera
image.
• Can be exploited by arranging the camera triple in a right angle (or L-shape),
• allowing matching along the rows and columns of the reference image.
12/32

15. Treat the camera triple (L,C,R) as two independent stereo pairs (L ,CL)
and (CR ,R).
 Use foreground segmentation to consider only one half to one third of
the pixels in the reference image CR. This makes it feasible to
calculate the entire correlation profile for each pixel one at a time.
To calculate the sum of correlation scores precompute a lookup table
of the location (uCL ,vCL) in CL corresponding the current pixel in CR
(based on the right-left rectification relationship).
 Also compute a linear approximation for the disparity dL = M(uCR
,vCR) × dR+b(uCR ,vCR) at [uCL ,vCL] which arises from the same depth
point as [uCR ,vCR ,dR].
Calculate the correlation score corrR(uCR ,vCR ,dR) and look up the
corresponding [uCL ,vCL] and compute dL.
 Then calculate the correlation score corrL(uCL ,vCL ,dL). Select the
disparity dR which optimizes corrT = corrL(uCL ,vCL ,dL) + corrR(uCR
,vCR, dR).
13/32

16. The method can be summarized as follows:
Pixel-wise Trinocular Stereo
Step 1: Pre-compute lookup table for CL locations
corresponding to CR locations, and dL approximation
lookup tables M and b.
Step 2: Acquire image triple (L,C,R).
Step 3: Rectify (L ,CL) and (CR ,R) independently
Step 4: Calculate foreground mask for CR and R
Step 5: For every foreground pixel CRmask(u,v)
Step I: For every disparity dR € Dr
If Rmask(u + dR ,v) € foreground
Step i: compute corrR(uCR ,vCR ,dR)
Step ii: lookup [uCL ,vCL]
Step iii: compute dL = M(uCR ,vCR) ×
dR + b(uCR ,vCR)
Step iv: compute corrL(uCL ,vCL ,dL )
Step v: corrT = corrL + corrR
Step vi: If corrT is a peak
Step 1: Fit parabola to find sub-pixel
correlation peak and disparity
adjustment dadj
Step 2: Update corrbest = corrT ,
dbest = dR + dadj
Step 6: Goto 2 14/32

17. 3D VIDEO TRANSMISSION
• Compress the 3D images constructed
• Reasons to compress: save storage space, conserve bandwidth, and
speed up application software.
• The algorithm used is Modern Driven Compression Algorithm.
• It is a lossy compression algorithm.
• Lossy compression algorithms remove small details that require a
large amount of data to store at full fidelity.
• In lossy compression, impossible to restore the original file due to
the removal of essential data.
• This scheme uses motion JPEG for color compression and the run
length coding plus Huffman coding for depth compression.
15/32

18. Color Compression
Motion JPEG uses JPEG still image compression on each frame
separately.
Depth Compression
•The depth information of an entire frame compressed using Run
Length Encoding (RLE) followed by Huffman coding.
•RLE is used to avoid spatial redundancy in transmitted image of an
object against a blank background.
•Huffman coding is used because the result of RLE is a set of symbols
that have varying frequencies.
16/32

19. Schematic diagram of the compression algorithm
17/32
Schematic diagram of the decompression algorithm

20. • After the compression is done, the images are ready to be
transmitted to the receiver.
• Following the flow of information tele-immersion depends on
intense data processing at each end of a connection, mediated
by high performance network.
• From the sender:
Parallel processors accept visual inputs from the cameras and
reinterpret the scene as a 3-Dimensional computer model.
• To the receiver:
Specific rendering of remote people and places are synthesized
from the model as it is received to match the point of view of
each eye of a user. The whole process repeats many times a
second to keep up with the user head motion.
18/32

21. 19/32

22. WORKING IN BRIEF
20/32

23. BASIC REQUIREMENTS
• Display technologies: stereo immersive displays would have to
present a clear view of the scenes being transmitted.
• Haptic sensors: would allow to touch projections as if they were
real.
• Desktop supercomputers: would perform the trillions of
calculations needed to create a holographic environment. A network
of computers that share power could also possibly support these
environments.
• Ceiling headtracker: used to measure the head position
• 3D stereo glasses: would allow the user to view the remote stereo
displayed 3D scene.
21/32

24. • Tele-cubicle :
1. A tele-cubicle is an office that appear to be one quadrant in a larger
shared virtual space.
2. It consists of a stereo immersive desk surface and two stereo-immersive
wall surfaces.
3. These three display surfaces join to form a virtual conference table
in the centre.
4. Allowing the realistic inclusion of tele-immersion into the work
environment taking up the usual amount of desk space.
5. The main idea behind this work came directly from the Tele-
Immersion meeting on July 21 ,1997 at the Advanced Network
Office.
22/32

25. 23/32

26. 24/32

27. Haptic sensors: Miniaturized force/torque sensors
• Haptic means that relating to or based on the sense of touch.
• External finger forces are measured by placing force sensing pads at
the fingertips.
• An optical sensor mounted on the fingernail detects the force. This
allows the human to touch the environment with bare fingers and
perform fine, delicate tasks using the full range of haptic sense.
• Specifically, the fingernail is instrumented with miniature light
emitting diodes (LEDs) and photo detectors in order to measure
changes in the reflection intensity when the fingertip is pressed
against a surface. The changes in intensity are then used to determine
changes in the blood volume under the fingernail.
25/32

28. • The algorithm used for haptic rendering is God-object or Proxy
paradigm.
• A god object or proxy is a conceptual point restricted to the object
and its position at every single moment is closest to the virtual
representation of the virtual fingers.
• As long as the user is away, no force will be executed.
• Once the user penetrates the virtual object, force will be calculated
proportionally to the distance b/w the proxy and the position of the
user’s virtual representation.
• Force can be calculated using the spring approach b/w the god object
and the representations of every finger. Thus the total force has a
tangential component.
• Applying the method to all contact points, total force and torque to
the virtual object can be calculated.
• Hence the object can be moved and rotated as desired.
26/32

29. Haptic Device
27/32

30. APPLICATIONS
1. Uses in Education:
In education, tele-immersion can be used to bring together
students at remote sites in a single environment relationship
among educational institutions which could improve tremendously
in the future with the use of tele-immersion.
2. Future Office:
Tele-immersion can be used to
provide an office like environment
to business professionals in the
near future.
28/32

31. 3. Medicine:
With the help of tele-immersion, 3D surgical learning for virtual
operation is possible.
Recently, teleconferencing has been used in medicine.
4. Industrial design, 7. Army Training
architectural, evaluation 8. Art and
Entertainment
5. Interactive Scientific 9. Virtual game
Visualization 10. Industrial Design
6. Architectural Review
and Evaluation
29/32

32. FUTURE APPLICATIONS
• Remote Learning & Training
• 3D Motion Capture of Body Segments
• 3D CAD Design
• Entertainment (For foodies – McDonalds’ is trying to use Tele-
Immersion for organizing family dinners and parties for people at
different locations !!)
• Training for complex and hazardous tasks where physical virtual
interactions are the key component
30/32

33. PROBLEMS
• Quality of transmitting multimedia and Tele-immersion data
streams over the internet is affected by high packets loss rates.
• Expensive
• High network bandwidth required .
• High speed internet is required (at least 60 MBPS)
31/32

34. BIBLIOGRAPHY
• Trinocular Stereo: A Real-Time Algorithm and its Evaluation
Jane Mulligan (University of Colorado) , Volkan Isler (University of
Pennsylvania), Kostas Daniilidis (University of Pennsylvania)
http://repository.upenn.edu/cis_papers/79
• Real Time 3D Video Compression for Tele-Immersive Environments
Zhenyu Yang, Yi Cui, Zahid Anwar, Robert Bocchino, Nadir Kiyanclar,
Klara Nahrstedt, Roy H. Campbell (University of Illinois)
https://www.ideals.illinois.edu/bitstream/handle/2142/11085/Real-
Time%203D%20Video%20Compression%20for%20Tele-
Immersive%20Environments.pdf?sequence=2
• Tele-Immersion As a Positive Alternative of The Future
Glasenhardt, S., Cicin-Sain, M., Capko, Z. (University of Rijeka)
http://ieeexplore.ieee.org/ielx5/8683/27508/01225352.pdf?tp=&arnumber
=1225352&isnumber=27508
• www.google.com
• www.wikipedia.com 32/32

35. THANK YOU…!

36. THE EPIPOLAR GEOMETRY
C,C’,x,x’ and X are coplanar

37. EXAMPLES OF EPIPOLAR
LINES
BACK