The Allosphere

Allosphere@CNSI: Towards a fully Immersive and Interactive Scientific Experience

in partnership with the California Nanosystems Institute MAT/CNSI allosphere @ CNSI What is a Digital Media Center doing in a Nanosystems institute?

A team of digital media researchers at UCSB has been fostering a cross-disciplinary field that unites science and engineering through the use of new media
Allosphere = Integration + availability to a larger community
Description and Goals

Allosphere Steering Committe
JoAnn Kuchera-Morin (Media Arts and Technology Initiatives)‏
Xavier Amatriain (Media Arts and Technology Initiatives)‏
Jim Blascovich (Psychology)‏
Forrest Brewer (Electrical and Computer Engineering)‏
Keith Clarke (Geography)‏
Steve Fisher (Life Sciences)‏
B.S. Manjunath (Electrical and Computer Engineering)‏
Marcos Novak (Media Arts and Technology/Arts)‏
Matthew Turk (Media Arts and Technology/Computer Science)‏
T.B.D. (California Nanosystems Institute)‏

The Allosphere
synthesis, manipulation, exploration and analysis of large-scale data sets ....
environment that can simulate virtually real sensorial perception providing multi-user immersive interactive interfaces
research into
scientific visualization, numerical simulations, data mining, visual/aural abstract data representations, knowledge discovery, systems integration and human perception
Description and Goals

Allosphere and other labs hosted in UCSB’s California Nanosystems Institute (CNSI)‏
The Building

The space itself is already a part of the final instrument:
The Space
three-story anechoic sphere, ten meters in diameter, containing a built-in spherical screen.

Once equipped, the CNSI Allosphere will be one of the largest immersive instruments in the world.
The Features
unique features: true 3D spherical projection of visual and aural data, and sensing and camera tracking for interactivity .

The AlloSphere is situated at one corner of the CNSI building, surrounded by different media labs.
Visual Computing
Interactive Installation
Immersion/Eversion
Robotics
Plurilabs
Other MAT Labs at CNSI

Research in the Allosphere

Inherent research comprises all of the activities that use the instrument as a research framework for immersive, multimodal environments:
Inherent Research

Sensor and Camera Tracking Systems
research related with computer vision as well as innovative interfaces and sensor networks that might be used to capture user interaction
Inherent Research. Interactivity

System Design and Integrated Software/Hardware Research
integration of the different hardware and software components at play
Inherent Research. Systems

Immersive Visual Systems Research
re-creation of an immersive visual space in a spherical environment
Inherent Research. Visual

Immersive Audio Systems Research
re-creation of a virtual 3D sound environment in which sources can be placed at arbitrary points in space with convincing synthesis and that allows to simulate the acoustics of real spaces
Inherent Research. Audio

Functional research includes those activities that will use the Allosphere as a tool for scientific exploration:
Functional Research

Multidimensional knowledge discovery
deal with issues such as highly dimensional feature descriptors, similarity metrics, and indexing
Machine learning , image data mining and understanding...
Functional Research. Knowledge

Analysis of complex structures and systems
Constructing the next generation of engineering paradigms requires a mechanism for rapid simulation, visualization and exploration supporting phenomena at multiple physical and temporal scales
Functional Research. Complex Systems

Human perception, behavior and cognition
valuable instrument for behavioural scientists interested on the impact of virtual environments, large scale visualization, or spatial hearing.
Functional Research. Psychology

Cartographic display and Information Visualization
remote sensing and geographic information science the opportunity to explore the potential of “inside-out” global data displays as tools for collective decision-making
Functional Research. Cartography

Artistic scientific visualization/auralization
artistic principles are driving research into real-time interactivity and human manipulation of complex scientific data structures
Functional Research. Artistic Visualization

Most of the research in the Allosphere (Functional and Inherent) has a direct mapping into future forms of Entertainment and Edutainment .
We envision collaboration from the Entertainment Industry
The Future of Entertainment

Prototype Projects in the Allosphere

Prototype-driven System
State-of-the-art system: still many open research questions need to be addressed.
We want content to drive the system design.
For that reason we are prototyping the instrument with different projects/requirements.

The Allobrain
In collaboration with UCLA Brain Imaging Institute, Marcos Novak and many MAT/CREATE students (view video)‏

Quantum Spin Precession
In collaboration with Prof. David Awschalom and Spintronics lab. Audiovisual model for coherent electron spin precession in a quantum dot

Multicenter Hydrogen Bond
With Anderson Genotti – Matrerials Researcher and discoverer of the Hydrogen Bond – and Prof. Van De Walle. Visualization and multi-modal representation of unique atomic bonds for alternative fuel sources (view video).

NanoCAD in the Allosphere
In collaboration with BinanGroup's NanoCAD

Alloproteins
In collaboration with the Chemistry/CS department using Chromium and Vmd

An Engineering Challenge

Innovation
The Allosphere presents innovative aspects in respect to existing environments such as The Cave
Spherical environment with 360 degrees of visual stereophonic information: spherical immersive systems enhance subjective feelings of immersion, naturalness, depth and ``reality''.
It is fully multimedia as it combines latest techniques both on virtual audio and visual data spatialization. Combined audio-visual information can help information understanding but most existing immersive environments focus on visual data.

Innovation
Completely interactive and multimodal environment, including camara tracking systems, audio recognition and sensor networks.
Pristine scientific instrument - e.g. the containing cube is fully anechoic chamber and details such as room modes or screen reflectivity have been studied.
Multiuser : Its size allows for up to 15 people to interact and collaborate on a common research task.

An Engineering Challenge

An Engineering Challenge The Visual subsystem

Allosphere display can only be compared to high-end state of the art planetariums (Gates planetarium at Denver Museum of Nature&Science or Griffith Observatory in LA)
Some AlloSphere requirements are considerably more demanding
Variety of types of graphics including smaller size text
Bright backgrounds and accurate color
Stereo projection
Excellent system flexibility and expandability
Overview

Overview
Key Design Parameters
Display quality/performance
Mechanical/facilities constraints
Overall system architecture, configuration management, automation, calibration
Cost
Secondary Concerns
Aging
Maintenance
Upgrades
Acoustic performance (of video equipment)‏
The Visual subsystem

Display Brightness
What is required?
Eyestrain-free operation over a decent range of color values
Brightness levels at or above photopic threshold for good contrast and color acuity
High resolution
Stereo/mono operation
Given:
Screen area: ~320m 2
Projector overlap factor: 1.7
Screen gain, direction averaged: 0.12
14 projectors with a max. 3K lumens/projector

Simulation results
~10 cd/m 2 screen luminance per 42,000 lumen of total light input
Recommendations
0.7 – 5 cd/m 2 recommended for multimedia domes
50 cd/m 2 for cinema projection (SMPTE)‏
Conclusion
42K lumens is good enough for most applications
Display Brightness

Active stereo introduces more than 50% loss in brightness but ...
5 cd/m 2 is still in the high-end of recommendations for domes
Active stereo introduces a dramatic gain in subjective quality perception.
On the other hand, we cannot project much more than that because of:
Back reflections
Cross-reflections
Display Brightness

“ Eye-limiting resolution” is not feasible (right now)‏
Approx. 150M pixels required to achieve in 30lp/deg (1 arc minute) in all directions
11 lp/deg (3 arc minute) is the recommended value for domes
20M pixels, 14 projectors
Resolution

Design requirements relate to all aspects of system design
Projector side
Best image quality, usually combined with color correction.
Limited configuration
Lower cost and higher flexibility
Dedicated hardware
Lower latencies
DLP projectors are problematic due to the extra frame buffer latency
Custom Software Infrastructure?
Image Warping and Blending

An Engineering Challenge The Audio subsystem

Audio Requirements
“ Ear-limited” audio rendering
Flat freq. response 20 Hz – 22kHz
Dynamic Range 120 dB
SNR>90 dB
T60 < 0.75 sec
Spatial Accuracy: 3 ° in horizontal axis and 10° in elevation

Spatial Audio
Examples of Spatial audio: stereo, surround...
Geometrical model-based spatialization
Mono source + dynamic positioning
Three “standard” techniques:
Vector-based amplitude panning
Ambisonic spatialization
Wafefield Synthesis

Wavefield Synthesis
Huygens principle of superposition: many closely spaced speakers create a coherent wavefront with an arbitrary source position
3D WFS has still not been attempted because of computational complexity (3D KH integral) : can use Ambisonics on the z axis

Spatial Techniques
All these techniques present pros/cons and interesting research problems
We already have a framework that can effectively combine them
Spatial audio: huge success in the near future
Number of speakers depends on the specific technique
but in order to have a reasonable spatial resolution we need ~ 500 speakers
The technology to use (electrostats, ribbon, tweeter array...) is also still under discussion.

Input Sensing & Multimodal HCI

Interactivity
Dynamic , user-driven environment – how to best give users the ability to interact with data in effective, compelling, and natural ways?
Powerful techniques for navigation, selection, manipulation, and signaling
Sense and perceive human movement, gesture, and speech via a network of sensors
Cameras, microphones, haptic devices, etc.
Multimodal interaction!

Computing Infrastructure

Integration
A typical multi-modal AlloSphere application will integrate services running on multiple hosts on the LAN that implement a distributed system composed of:
input sensing (camera, sensor, microphone),
gesture recognition/control mapping,
interface to a remote (scientific, numerical, simulation, data mining) application,
back-end processing (data/content accessing),
A/V rendering and projection management.

Integration
Still need software infrastructure to distribute the different graphic pipes from the generation engine to the render farm
Develop ad-hoc visual generation software engine and interconnection with data streams.
Efforts need to be put forward into building this intermediate integration/coordination layer by combining several specialized packages
Cyberinfrastructure grant presented (Hollerer, Wolski and Shea)‏

Video Generation Subsystem
In order to generate high resolution (1920x1200@120Hz) in active stereo we need high end video cards
Sample rendering farm for 14 stereo channels: 7 servers with one Quadro FX5600 in each one.
Blending and warping managed mostly at the projector side.
Sample video generation unit with a render Linux Box with an NVidia Quadro 5600 (still to appear ) feeding two Christie Mirage S2K+

Video Distribution

Audio Generation Subsystem
Problem: Distribute 500+ channels of hi-fi audio to speakers
Distributed rendering
~1.3 Gbps (at 24bit/96kHz)‏
Multichannel audio streaming over network: Yamaha's mLan, Gibson's Global Information Carrier, Sony's Supermac...
Sample synchronous output: Steve Butner's EtherSync
Network interface box to be custom built
Single Render Point
Develop custom DSP hardware
Harder signal distribution

Synthesis/Processing Software: Audio team has extensive experience in developing such software and has ready-to-use frameworks such as CLAM (Amatriain, ACM MM Best Open Source Software 2006); CSL (Pope)‏

Open Research Areas and People
Graphics (Hollerer)‏
Audio (Amatriain)‏
Auralization (Roads)‏
3D Audio (Pope)‏
Systems (Hollerer, Brewer, Butner, Pope, Amatriain)‏
Interactivity (Turk, Kuchera-Morin, Amatriain)‏
Experiential Signal Processing (Gibson)‏
HPC, Optimization (Wolski, Krintz)‏
Content Creation
Visual (Legrady)‏
Music (Kuchera-Morin)‏
VW (Novak)‏

Open Research Areas and People
Nanoscale systems representation (Oster, Garcia-Cervera)‏
Brain Imaging (Grafton)‏
Molecular Dynamics (Shea)‏
GIS (Clarke et al.)‏
Bio-imaging: (Fisher, Manjunath)‏
Perception (Loomis, Beall...)‏

http://www.mat.ucsb.edu/allosphere THANKS!

The Allosphere

by Xavier Amatriain

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 4

Statistics

The Allosphere Presentation Transcript