Ircam forum workshops paris 2018

Suguru Goto
2018/3/8
IRCAM Forum Workshops Paris 2018
Virtual Musical Instruments
- Its Evolution with an application of
Virtual Reality and Augmented Reality -

X Expertise and techniques of VR/AR
O Examples of works which are based on the ideas of VR/AR
X Application to games since 2016, considered the first year of
VR
O Cyber Art since 1990 (Post Cybernetics)
X Recent VR such as an application of Game
O The approach toward VR/AR from the fields of composition of
Contemporary Music, Computer Music, Interactive Music
What you will find in this presentation…

1.  Introduction
2.  Brief personal history
3.  Previous works
4.  Progress, reason, purpose of research of “VR/AR and Music” at the
present
5.  Research on VR/AR
6.  Current projects
7.  Concept
8.  Summary
Overview

1.  Introduction of previous works
2.  Relationship between these works and VR
3.  The way to the current projects that are being conceived
4.  VR/AR and music
5.  Concept
6.  Work in progress
1. Introduction

I.  The period in the United States
a.  New England Conservatory in Boston
b.  The turning point was the study of composition with Earl Brown at
Aspen
2. Brief personal history

II. The period in Germany
a.  Received a scholarship to go to Berlin
b.  The collapse of the Berlin Wall, followed immediately after by the
integration of East and West Germany
c.  The strongest impact in life
d.  Berlin University of the Arts, Berlin Institute of Technical
University in Berlin
e.  Studied composition with Dieter Schnebel, influential mentor
f.  Schnebel is an experimental music composer, who is also known as “
John Cage” in Germany

III. The period at IRCAM in Paris
a.  Since this period, compositions, performances, research and
development are mostly computer-based
b.  Max (interactive environment)
c.  Sensor technologies and robotics
d.  Gestural informatics research team
e.  While living in Paris, worked as an associate professor in digital
music, media art, and creative coding at Bordeaux Art University

III. At present
a.  Associate professor at Department of Musical Creativity and the
Environment, Tokyo University of the Arts since last April
b.  Teaching computer music and media art

I.  Summary
II.  Virtual Musical Instrument
a.  Examples of works: Le SuperPolm, BodySuit, Robotic Music,
Augmented Body and Virtual Body, netBody
b.  Media Art and Music
c.  Duali, and Cymatics
d.  Work in progress of VR/AR
e.  Body in zero G
3. Previous works

II. Definition of Virtual Musical Instruments
a.  Physicality is involved in every case
b.  Relationship between humans and machines
c.  Although the physical controller itself does not change,
programming can alter the sound and the performance method in
countless ways
d.  Physical interface, programming and sound parts can be regarded
as single, unified piece
3. Previous works

e. Historical examples
3. Previous works
Theremin (1919) - Léon Theremin Ondes Martenot (1928) - Maurice Martenot

e. Previous examples
3. Previous works
Max Mathews - Radio Baton

3. Previous works
Joel Chadabe - Musical instruments remodeled from
Theremin

3. Previous works
Tod Machover - Hyperinstruments

3. Previous works
Michel Waisvisz – The Hand

3. Previous works
The previous version of Max, FTS with ISPW on NeXT
Computer at IRCAM

III. The first Virtual Musical Instruments, “Le SuperPolm”
3. Previous works

IV. The second Virtual Musical Instruments, BodySuit
3. Previous works

V. RoboticMusic – Automated Musical Instruments
3. Previous works

V. Augmented Body and Virtual Body
3. Previous works

VII. netBody
3. Previous works

VIII. Progress, reason, purpose of research of “VR/AR and Music” in
the present
a.  Reason and purpose
b.  The problems of sensors and interface
c.  The problems of Interaction
d.  The problems of robots
e.  The problems when these are combined with music
f.  The attempts in order to solve the problems while using video and
other
g.  Fusion with Media Art
3. Previous works

IX. Images
a.  It is different from the traditional image concept
b.  Stage effect or stage design
c.  Projection mapping and augmented reality
d.  Relationship between music and video
e.  Relationship between body and image
f.  System for new cognitive method
3. Previous works

X. Modern Dance and Technology
a.  Motion Capture System
b.  Augmented reality and Projection Mapping seen in Klaus
Obermeyer and Chunky Move
c.  Trigger to sound
3. Previous works

XI. Duali
a.  Duali is a composition where dance and video coexist in a
composition
b.  Media: Motion Capture Systems (BodySuit 1 and BodySuit 2),
Interactive Images and Sound
3. Previous works

XI. Duali
c. Relationship between
humans and machines
d. Performer's
physicality and images
e. Architectural images -
the lighting effects of the
stage
3. Previous works

XI. Duali
h. “Duali” is based on the concept of
dualism
i.  "Body-Spirit" or “Spiritual dualism -
Material", such as dualism of Descartes
j. Eastern philosophy refers to "physical
dualism", for example, Yin and Yang in
Chinese philosophy
k. Space – Architecture / Time – Body
l. Female dancer and male dancer / human
– machine
m. Image simply consists of white and
black
n. Extended body - Virtual space
o. Musical sounds and noise / Expression
and Abstraction
p. Choreography represents logic and
perception / body - machine
3. Previous works

XI. Duali
r. BodySuit is specifically designed for use during performances and is
the name of a system that captures motion
S. Concept of "extended body"
3. Previous works

XII. Cymatics
a.  Fluid and sound installation
3. Previous works

XII. Cymatics
b. Vibration of sound
3. Previous works

XII. Cymatics
c. Conceived when working as a composer in a project designing an
artificial robot mouth (musical instrument robot) at IRCAM
3. Previous works

XII. Cymatics
d. Interactive Sound Installation as Media Art
3. Previous works

CsO
- Body Jack, Suguru Goto and Patrick Defasten
- Continuum Suguru Goto and Lucio Arese
- CsO, Suguru Goto and Antoine Schmitt
Production: Athor Harmonics

XIII. Progress, reason, purpose of research of “VR/AR and Music” in
the present -2
b. Body in zero G as an attempt in Projection Mapping and Anamorphic
Perspective
3. Previous works

-  Body in zero G

Art direction and Composition: Suguru Goto
Dancers: Akiko Suruga, Ayaka Suzuki
Image design: Lucio Arese, Patrick Defasten, Takaaki Uno, Damien
Serban
Production: Athor Harmonics

I.  Binaural
a.  Sounds in VR include binaural and ambisonic
b.  Firstly there is music (not the meaning of sound design), that is
related to binaural and ambisonic
c.  Binaural depending on the direction of the head, while listening on
headphones
d.  The idea that images accompany it
e.  Immersion in the virtual space of 3D images with HMD
f.  The sounds usually accompany the images
g.  However, in this case the opposite is true
h.  Interaction between music and images for create cross modal
perception
i.  The images which accompany the music are abstract and simple
4. Progress, reason, purpose of research of
“VR/AR and music” in the present

Binaural – Oculus Rift + Max

II. Ambisonics
a.  A system of multiple speakers such as 2D is often referred to as
Acousmonium in the music world.
b.  Furthermore, what is precisely controlled by a computer is called
Multi-speakers system
c.  Virtually reproduces the point where sound is emitted in space
d.  Therefore, panning between multiple speakers occurs
e.  For example, the sound moves between the speakers surrounding it
to draw a circle in space
f.  For example, one can simulate a situation where a bird is flying
while drawing a circle around oneself
g.  One can freely alter the parameters in space such as speed, distance
to oneself

II. Ambisonics
h. For example, the situation where an ambulance heads towards you,
and then passes by
i.  Doppler effect
j. Reproducing virtual space by combining Delay, Reverb, various
Filters
k. For example, a church with a lot of reflections, large hall, studio
without any reflections, etc.
l. Buildings which are made of concrete, for example, have long, metalic
reverberations.
m. What makes such a system 3D is called Ambasonics

Ambisonics – 22.2 ch Multi channel sound sound system , NHK

Ambisonics – 22.2 ch Multi channel sound sound system 、Department
of Musical Creativity and the Environment, Tokyo University of the
Arts

Ambisonics – 22.2 ch Multi channel sound sound system 、Department
of Musical Creativity and the Environment, Tokyo University of the
Arts
Marionette1.0 – Hiromu Tanokura
(3D sound panning system for Live Performance)

III. Experiments
a.  Originally, I wanted to recreate the illusion of sound in a three
dimensional acoustic space.
b.  For example, the center point of the sound follows everywhere
(Sweet Spot).
c.  For example, a violinist, who performs in front, follows every where.
d.  If one tilts one’s head, the player's body inclines its angle in the air,
as well.
e.  The project was conceived as the spatial acoustic illusion equivalent
of Escher's eternal staircase or Shepard Tone
f.  For that reason I wanted to be able to move freely anywhere in a
three dimensional space
g.  I aimed to create a cross modal experience by adding the image’s
illusion to the sound,

I.　Motion Capture System, Sensor, Arduino, Wifi
5. Research on VR/AR

II. Mapping Interface, Kalman Filter

VII. Research/Development by Prof. Hiroo Iwata, University of Tsukuba 　
(previous example）

Reference: https://www.youtube.com/watch?v=dAh8uaLqzPk
Large Space

I.  Purpose
a.  To create an illusion of sound based on a three-dimensional acoustic
space
b.  The spatial acoustic illusion equivalent of Escher's eternal staircase
or Shepard Tone
c.  To be able to move freely anywhere in a three dimensional space.
d.  Therefore, a system in which a person is attached to the ceiling and
able to move about the space freely
6. Current projects

II. Overview
a.  As a prototype, a person wearing Oculus Rift is attached to the
gyroscope to reproduce the state of zero gravity.
b.  In the current project in Japan (a work in progress), a harness for
climbing is attached to the ceiling and can freely move about space
thanks to motors
c.  It allows one to freely change the angle in 3D.
d.  Generative Images
e.  To reproduce the images on six surfaces while the floating objects
flutter in the zero gravity state

6. Current projects

III. Work in progress
a.  Even when floating freely in zero gravity space, a reference point is
necessary.
b.  Therefore, there should be always ground on the floor of the image
and something like a horizon in front.
c.  Otherwise one can not feel zero gravity space
d.  This may seem obvious, but when considering only the aesthetics in
art, it is often difficult to express a fundamental effect well
6. Current projects

IV. Future
a.  It will be replaced by an industrial arm robot soon which will move
the human body though space
b.  Arm robot such as Yaskawa Electric
c.  This is still a work in progress.
d.  Style that is heading at the present moment
e.  Also, the concept of personal art project of “VR/AR and music”
6. Current projects

I.  Purpose
a. In both cases, these are not used in a work as the result of research
and development.
a.  The idea of a work is conceived first, and then the research and the
development are conducted in order to realize the art work
b.  The philosophical concept behind the artistic idea
c.  A unique view on the concept
d.  Own original idea
e.  To extend an interpretation of thought and to seek a source of
further inspiration
7. Concept

I. Philosophy
a.  Mille plateaux by Gilles Deleuze and Félix Guattari
b.  Pluralism of Dualism
c.  Transhumanism
d.  Post-humanism
e.  What is “Identity”?
f.  Skepticism about “Identity”
g.  What is the relationship between the physical body and the identity
in the spirit?
7. Concept

I.  When one is confronted with a machine that is like the body, it can
provoke reflection about the reality of the body
II.  Also, when one faces artificial intelligence with enormous abilities,
one think again about the complicated human spirit.
III.  Unless one tries to be conscious about the objective thing, one can
not even be aware of one’s own body.
IV.  A question whether one finds something to be a sign of a new
direction
8. Conclusion

Ircam forum workshops paris 2018

Recommended

Recommended

More Related Content

Similar to Ircam forum workshops paris 2018

Similar to Ircam forum workshops paris 2018 (20)

Recently uploaded

Recently uploaded (20)

Ircam forum workshops paris 2018