Allosphere@CNSI: Towards a fully Immersive and Interactive Scientific Experience

in partnership with the California Nanosystems Institute MAT/CNSI allosphere @ CNSI What is a Digital Media Center doing in a Nanosystems institute?

A team of digital media researchers at UCSB has been fostering a cross-disciplinary field that unites science and engineering through the use of new media Allosphere = Integration + availability to a larger community Description and Goals

A team of digital media researchers at UCSB has been fostering a cross-disciplinary field that unites science and engineering through the use of new media

Allosphere = Integration + availability to a larger community

Allosphere Steering Committe JoAnn Kuchera-Morin (Media Arts and Technology Initiatives)‏ Xavier Amatriain (Media Arts and Technology Initiatives)‏ Jim Blascovich (Psychology)‏ Forrest Brewer (Electrical and Computer Engineering)‏ Keith Clarke (Geography)‏ Steve Fisher (Life Sciences)‏ B.S. Manjunath (Electrical and Computer Engineering)‏ Marcos Novak (Media Arts and Technology/Arts)‏ Matthew Turk (Media Arts and Technology/Computer Science)‏ T.B.D. (California Nanosystems Institute)‏

JoAnn Kuchera-Morin (Media Arts and Technology Initiatives)‏

Xavier Amatriain (Media Arts and Technology Initiatives)‏

Jim Blascovich (Psychology)‏

Forrest Brewer (Electrical and Computer Engineering)‏

Keith Clarke (Geography)‏

Steve Fisher (Life Sciences)‏

B.S. Manjunath (Electrical and Computer Engineering)‏

Marcos Novak (Media Arts and Technology/Arts)‏

Matthew Turk (Media Arts and Technology/Computer Science)‏

T.B.D. (California Nanosystems Institute)‏

The Allosphere synthesis, manipulation, exploration and analysis of large-scale data sets .... environment that can simulate virtually real sensorial perception providing multi-user immersive interactive interfaces research into scientific visualization, numerical simulations, data mining, visual/aural abstract data representations, knowledge discovery, systems integration and human perception Description and Goals

The Allosphere

synthesis, manipulation, exploration and analysis of large-scale data sets ....

environment that can simulate virtually real sensorial perception providing multi-user immersive interactive interfaces

research into

scientific visualization, numerical simulations, data mining, visual/aural abstract data representations, knowledge discovery, systems integration and human perception

Allosphere and other labs hosted in UCSB’s California Nanosystems Institute (CNSI)‏ The Building

Allosphere and other labs hosted in UCSB’s California Nanosystems Institute (CNSI)‏

The space itself is already a part of the final instrument: The Space three-story anechoic sphere, ten meters in diameter, containing a built-in spherical screen.

The space itself is already a part of the final instrument:

three-story anechoic sphere, ten meters in diameter, containing a built-in spherical screen.

Once equipped, the CNSI Allosphere will be one of the largest immersive instruments in the world. The Features unique features: true 3D spherical projection of visual and aural data, and sensing and camera tracking for interactivity .

Once equipped, the CNSI Allosphere will be one of the largest immersive instruments in the world.

unique features: true 3D spherical projection of visual and aural data, and sensing and camera tracking for interactivity .

The AlloSphere is situated at one corner of the CNSI building, surrounded by different media labs. Visual Computing Interactive Installation Immersion/Eversion Robotics Plurilabs Other MAT Labs at CNSI

The AlloSphere is situated at one corner of the CNSI building, surrounded by different media labs.

Visual Computing

Interactive Installation

Immersion/Eversion

Robotics

Plurilabs

Research in the Allosphere

Inherent research comprises all of the activities that use the instrument as a research framework for immersive, multimodal environments: Inherent Research

Inherent research comprises all of the activities that use the instrument as a research framework for immersive, multimodal environments:

Sensor and Camera Tracking Systems research related with computer vision as well as innovative interfaces and sensor networks that might be used to capture user interaction Inherent Research. Interactivity

Sensor and Camera Tracking Systems

research related with computer vision as well as innovative interfaces and sensor networks that might be used to capture user interaction

System Design and Integrated Software/Hardware Research integration of the different hardware and software components at play Inherent Research. Systems

System Design and Integrated Software/Hardware Research

integration of the different hardware and software components at play

Immersive Visual Systems Research re-creation of an immersive visual space in a spherical environment Inherent Research. Visual

Immersive Visual Systems Research

re-creation of an immersive visual space in a spherical environment

Immersive Audio Systems Research re-creation of a virtual 3D sound environment in which sources can be placed at arbitrary points in space with convincing synthesis and that allows to simulate the acoustics of real spaces Inherent Research. Audio

Immersive Audio Systems Research

re-creation of a virtual 3D sound environment in which sources can be placed at arbitrary points in space with convincing synthesis and that allows to simulate the acoustics of real spaces

Functional research includes those activities that will use the Allosphere as a tool for scientific exploration: Functional Research

Functional research includes those activities that will use the Allosphere as a tool for scientific exploration:

Multidimensional knowledge discovery deal with issues such as highly dimensional feature descriptors, similarity metrics, and indexing Machine learning , image data mining and understanding... Functional Research. Knowledge

Multidimensional knowledge discovery

deal with issues such as highly dimensional feature descriptors, similarity metrics, and indexing

Machine learning , image data mining and understanding...

Analysis of complex structures and systems Constructing the next generation of engineering paradigms requires a mechanism for rapid simulation, visualization and exploration supporting phenomena at multiple physical and temporal scales Functional Research. Complex Systems

Analysis of complex structures and systems

Constructing the next generation of engineering paradigms requires a mechanism for rapid simulation, visualization and exploration supporting phenomena at multiple physical and temporal scales

Human perception, behavior and cognition valuable instrument for behavioural scientists interested on the impact of virtual environments, large scale visualization, or spatial hearing. Functional Research. Psychology

Human perception, behavior and cognition

valuable instrument for behavioural scientists interested on the impact of virtual environments, large scale visualization, or spatial hearing.

Cartographic display and Information Visualization remote sensing and geographic information science the opportunity to explore the potential of “inside-out” global data displays as tools for collective decision-making Functional Research. Cartography

Cartographic display and Information Visualization

remote sensing and geographic information science the opportunity to explore the potential of “inside-out” global data displays as tools for collective decision-making

Artistic scientific visualization/auralization artistic principles are driving research into real-time interactivity and human manipulation of complex scientific data structures Functional Research. Artistic Visualization

Artistic scientific visualization/auralization

artistic principles are driving research into real-time interactivity and human manipulation of complex scientific data structures

Most of the research in the Allosphere (Functional and Inherent) has a direct mapping into future forms of Entertainment and Edutainment . We envision collaboration from the Entertainment Industry The Future of Entertainment

Most of the research in the Allosphere (Functional and Inherent) has a direct mapping into future forms of Entertainment and Edutainment .

We envision collaboration from the Entertainment Industry

Prototype Projects in the Allosphere

Prototype-driven System State-of-the-art system: still many open research questions need to be addressed. We want content to drive the system design. For that reason we are prototyping the instrument with different projects/requirements.

State-of-the-art system: still many open research questions need to be addressed.

We want content to drive the system design.

For that reason we are prototyping the instrument with different projects/requirements.

The Allobrain In collaboration with UCLA Brain Imaging Institute, Marcos Novak and many MAT/CREATE students (view video)‏

In collaboration with UCLA Brain Imaging Institute, Marcos Novak and many MAT/CREATE students (view video)‏

Quantum Spin Precession In collaboration with Prof. David Awschalom and Spintronics lab. Audiovisual model for coherent electron spin precession in a quantum dot

In collaboration with Prof. David Awschalom and Spintronics lab. Audiovisual model for coherent electron spin precession in a quantum dot

Multicenter Hydrogen Bond With Anderson Genotti – Matrerials Researcher and discoverer of the Hydrogen Bond – and Prof. Van De Walle. Visualization and multi-modal representation of unique atomic bonds for alternative fuel sources (view video).

With Anderson Genotti – Matrerials Researcher and discoverer of the Hydrogen Bond – and Prof. Van De Walle. Visualization and multi-modal representation of unique atomic bonds for alternative fuel sources (view video).

NanoCAD in the Allosphere In collaboration with BinanGroup's NanoCAD

In collaboration with BinanGroup's NanoCAD

Alloproteins In collaboration with the Chemistry/CS department using Chromium and Vmd

In collaboration with the Chemistry/CS department using Chromium and Vmd

An Engineering Challenge

Innovation The Allosphere presents innovative aspects in respect to existing environments such as The Cave Spherical environment with 360 degrees of visual stereophonic information: spherical immersive systems enhance subjective feelings of immersion, naturalness, depth and ``reality''. It is fully multimedia as it combines latest techniques both on virtual audio and visual data spatialization. Combined audio-visual information can help information understanding but most existing immersive environments focus on visual data.

The Allosphere presents innovative aspects in respect to existing environments such as The Cave

Spherical environment with 360 degrees of visual stereophonic information: spherical immersive systems enhance subjective feelings of immersion, naturalness, depth and ``reality''.

It is fully multimedia as it combines latest techniques both on virtual audio and visual data spatialization. Combined audio-visual information can help information understanding but most existing immersive environments focus on visual data.

Innovation Completely interactive and multimodal environment, including camara tracking systems, audio recognition and sensor networks. Pristine scientific instrument - e.g. the containing cube is fully anechoic chamber and details such as room modes or screen reflectivity have been studied. Multiuser : Its size allows for up to 15 people to interact and collaborate on a common research task.

Completely interactive and multimodal environment, including camara tracking systems, audio recognition and sensor networks.

Pristine scientific instrument - e.g. the containing cube is fully anechoic chamber and details such as room modes or screen reflectivity have been studied.

Multiuser : Its size allows for up to 15 people to interact and collaborate on a common research task.

An Engineering Challenge

An Engineering Challenge The Visual subsystem

Allosphere display can only be compared to high-end state of the art planetariums (Gates planetarium at Denver Museum of Nature&Science or Griffith Observatory in LA) Some AlloSphere requirements are considerably more demanding Variety of types of graphics including smaller size text Bright backgrounds and accurate color Stereo projection Excellent system flexibility and expandability Overview

Allosphere display can only be compared to high-end state of the art planetariums (Gates planetarium at Denver Museum of Nature&Science or Griffith Observatory in LA)

Some AlloSphere requirements are considerably more demanding

Variety of types of graphics including smaller size text

Bright backgrounds and accurate color

Stereo projection

Excellent system flexibility and expandability

Overview Key Design Parameters Display quality/performance Mechanical/facilities constraints Overall system architecture, configuration management, automation, calibration Cost Secondary Concerns Aging Maintenance Upgrades Acoustic performance (of video equipment)‏ The Visual subsystem

Key Design Parameters

Display quality/performance

Mechanical/facilities constraints

Overall system architecture, configuration management, automation, calibration

Cost

Secondary Concerns

Aging

Maintenance

Upgrades

Acoustic performance (of video equipment)‏

Display Brightness What is required? Eyestrain-free operation over a decent range of color values Brightness levels at or above photopic threshold for good contrast and color acuity High resolution Stereo/mono operation Given: Screen area: ~320m 2 Projector overlap factor: 1.7 Screen gain, direction averaged: 0.12 14 projectors with a max. 3K lumens/projector

What is required?

Eyestrain-free operation over a decent range of color values

Brightness levels at or above photopic threshold for good contrast and color acuity

High resolution

Stereo/mono operation

Given:

Screen area: ~320m 2

Projector overlap factor: 1.7

Screen gain, direction averaged: 0.12

14 projectors with a max. 3K lumens/projector

Simulation results ~10 cd/m 2 screen luminance per 42,000 lumen of total light input Recommendations 0.7 – 5 cd/m 2 recommended for multimedia domes 50 cd/m 2 for cinema projection (SMPTE)‏ Conclusion 42K lumens is good enough for most applications Display Brightness

Simulation results

~10 cd/m 2 screen luminance per 42,000 lumen of total light input

Recommendations

0.7 – 5 cd/m 2 recommended for multimedia domes

50 cd/m 2 for cinema projection (SMPTE)‏

Conclusion

42K lumens is good enough for most applications

Active stereo introduces more than 50% loss in brightness but ... 5 cd/m 2 is still in the high-end of recommendations for domes Active stereo introduces a dramatic gain in subjective quality perception. On the other hand, we cannot project much more than that because of: Back reflections Cross-reflections Display Brightness

Active stereo introduces more than 50% loss in brightness but ...

5 cd/m 2 is still in the high-end of recommendations for domes

Active stereo introduces a dramatic gain in subjective quality perception.

On the other hand, we cannot project much more than that because of:

Back reflections

Cross-reflections

“ Eye-limiting resolution” is not feasible (right now)‏ Approx. 150M pixels required to achieve in 30lp/deg (1 arc minute) in all directions 11 lp/deg (3 arc minute) is the recommended value for domes 20M pixels, 14 projectors Resolution

“ Eye-limiting resolution” is not feasible (right now)‏

Approx. 150M pixels required to achieve in 30lp/deg (1 arc minute) in all directions

11 lp/deg (3 arc minute) is the recommended value for domes

20M pixels, 14 projectors

Design requirements relate to all aspects of system design Projector side Best image quality, usually combined with color correction. Limited configuration Lower cost and higher flexibility Dedicated hardware Lower latencies DLP projectors are problematic due to the extra frame buffer latency Custom Software Infrastructure? Image Warping and Blending

Design requirements relate to all aspects of system design

Projector side

Best image quality, usually combined with color correction.

Limited configuration

Lower cost and higher flexibility

Dedicated hardware

Lower latencies

DLP projectors are problematic due to the extra frame buffer latency

Custom Software Infrastructure?

An Engineering Challenge The Audio subsystem

Audio Requirements “ Ear-limited” audio rendering Flat freq. response 20 Hz – 22kHz Dynamic Range 120 dB SNR>90 dB T60 < 0.75 sec Spatial Accuracy: 3 ° in horizontal axis and 10° in elevation

“ Ear-limited” audio rendering

Flat freq. response 20 Hz – 22kHz

Dynamic Range 120 dB

SNR>90 dB

T60 < 0.75 sec

Spatial Accuracy: 3 ° in horizontal axis and 10° in elevation

Spatial Audio Examples of Spatial audio: stereo, surround... Geometrical model-based spatialization Mono source + dynamic positioning Three “standard” techniques: Vector-based amplitude panning Ambisonic spatialization Wafefield Synthesis

Examples of Spatial audio: stereo, surround...

Geometrical model-based spatialization

Mono source + dynamic positioning

Three “standard” techniques:

Vector-based amplitude panning

Ambisonic spatialization

Wafefield Synthesis

Wavefield Synthesis Huygens principle of superposition: many closely spaced speakers create a coherent wavefront with an arbitrary source position 3D WFS has still not been attempted because of computational complexity (3D KH integral) : can use Ambisonics on the z axis

Huygens principle of superposition: many closely spaced speakers create a coherent wavefront with an arbitrary source position

3D WFS has still not been attempted because of computational complexity (3D KH integral) : can use Ambisonics on the z axis

Spatial Techniques All these techniques present pros/cons and interesting research problems We already have a framework that can effectively combine them Spatial audio: huge success in the near future Number of speakers depends on the specific technique but in order to have a reasonable spatial resolution we need ~ 500 speakers The technology to use (electrostats, ribbon, tweeter array...) is also still under discussion.

All these techniques present pros/cons and interesting research problems

We already have a framework that can effectively combine them

Spatial audio: huge success in the near future

Number of speakers depends on the specific technique

but in order to have a reasonable spatial resolution we need ~ 500 speakers

The technology to use (electrostats, ribbon, tweeter array...) is also still under discussion.

Input Sensing & Multimodal HCI

Interactivity Dynamic , user-driven environment – how to best give users the ability to interact with data in effective, compelling, and natural ways? Powerful techniques for navigation, selection, manipulation, and signaling Sense and perceive human movement, gesture, and speech via a network of sensors Cameras, microphones, haptic devices, etc. Multimodal interaction!

Dynamic , user-driven environment – how to best give users the ability to interact with data in effective, compelling, and natural ways?

Powerful techniques for navigation, selection, manipulation, and signaling

Sense and perceive human movement, gesture, and speech via a network of sensors

Cameras, microphones, haptic devices, etc.

Multimodal interaction!

Computing Infrastructure

Integration A typical multi-modal AlloSphere application will integrate services running on multiple hosts on the LAN that implement a distributed system composed of: input sensing (camera, sensor, microphone), gesture recognition/control mapping, interface to a remote (scientific, numerical, simulation, data mining) application, back-end processing (data/content accessing), A/V rendering and projection management.

A typical multi-modal AlloSphere application will integrate services running on multiple hosts on the LAN that implement a distributed system composed of:

input sensing (camera, sensor, microphone),

gesture recognition/control mapping,

interface to a remote (scientific, numerical, simulation, data mining) application,

back-end processing (data/content accessing),

A/V rendering and projection management.

Integration Still need software infrastructure to distribute the different graphic pipes from the generation engine to the render farm Develop ad-hoc visual generation software engine and interconnection with data streams. Efforts need to be put forward into building this intermediate integration/coordination layer by combining several specialized packages Cyberinfrastructure grant presented (Hollerer, Wolski and Shea)‏

Still need software infrastructure to distribute the different graphic pipes from the generation engine to the render farm

Develop ad-hoc visual generation software engine and interconnection with data streams.

Efforts need to be put forward into building this intermediate integration/coordination layer by combining several specialized packages

Cyberinfrastructure grant presented (Hollerer, Wolski and Shea)‏

Video Generation Subsystem In order to generate high resolution (1920x1200@120Hz) in active stereo we need high end video cards Sample rendering farm for 14 stereo channels: 7 servers with one Quadro FX5600 in each one. Blending and warping managed mostly at the projector side. Sample video generation unit with a render Linux Box with an NVidia Quadro 5600 (still to appear ) feeding two Christie Mirage S2K+

In order to generate high resolution (1920x1200@120Hz) in active stereo we need high end video cards

Sample rendering farm for 14 stereo channels: 7 servers with one Quadro FX5600 in each one.

Blending and warping managed mostly at the projector side.

Video Distribution

Video Distribution

Audio Generation Subsystem Problem: Distribute 500+ channels of hi-fi audio to speakers Distributed rendering ~1.3 Gbps (at 24bit/96kHz)‏ Multichannel audio streaming over network: Yamaha's mLan, Gibson's Global Information Carrier, Sony's Supermac... Sample synchronous output: Steve Butner's EtherSync Network interface box to be custom built Single Render Point Develop custom DSP hardware Harder signal distribution

Problem: Distribute 500+ channels of hi-fi audio to speakers

Distributed rendering

~1.3 Gbps (at 24bit/96kHz)‏

Multichannel audio streaming over network: Yamaha's mLan, Gibson's Global Information Carrier, Sony's Supermac...

Sample synchronous output: Steve Butner's EtherSync

Network interface box to be custom built

Single Render Point

Develop custom DSP hardware

Harder signal distribution

Audio Generation Subsystem

Audio Generation Subsystem Synthesis/Processing Software: Audio team has extensive experience in developing such software and has ready-to-use frameworks such as CLAM (Amatriain, ACM MM Best Open Source Software 2006); CSL (Pope)‏

Synthesis/Processing Software: Audio team has extensive experience in developing such software and has ready-to-use frameworks such as CLAM (Amatriain, ACM MM Best Open Source Software 2006); CSL (Pope)‏

Open Research Areas and People Graphics (Hollerer)‏ Audio (Amatriain)‏ Auralization (Roads)‏ 3D Audio (Pope)‏ Systems (Hollerer, Brewer, Butner, Pope, Amatriain)‏ Interactivity (Turk, Kuchera-Morin, Amatriain)‏ Experiential Signal Processing (Gibson)‏ HPC, Optimization (Wolski, Krintz)‏ Content Creation Visual (Legrady)‏ Music (Kuchera-Morin)‏ VW (Novak)‏

Graphics (Hollerer)‏

Audio (Amatriain)‏

Auralization (Roads)‏

3D Audio (Pope)‏

Systems (Hollerer, Brewer, Butner, Pope, Amatriain)‏

Interactivity (Turk, Kuchera-Morin, Amatriain)‏

Experiential Signal Processing (Gibson)‏

HPC, Optimization (Wolski, Krintz)‏

Content Creation

Visual (Legrady)‏

Music (Kuchera-Morin)‏

VW (Novak)‏

Open Research Areas and People Nanoscale systems representation (Oster, Garcia-Cervera)‏ Brain Imaging (Grafton)‏ Molecular Dynamics (Shea)‏ GIS (Clarke et al.)‏ Bio-imaging: (Fisher, Manjunath)‏ Perception (Loomis, Beall...)‏

Nanoscale systems representation (Oster, Garcia-Cervera)‏

Brain Imaging (Grafton)‏

Molecular Dynamics (Shea)‏

GIS (Clarke et al.)‏

Bio-imaging: (Fisher, Manjunath)‏

Perception (Loomis, Beall...)‏

http://www.mat.ucsb.edu/allosphere THANKS!