This document outlines a study exploring how artists and composers create content for extended reality (XR) technologies like virtual, augmented, and mixed reality. The study will involve 12 artist-composer pairs creating 360-degree videos with 3D soundtracks. Artists will participate in workshops on the technologies and record improvised performances captured with 360 cameras. Composers will create musical mockups and final 3D audio mixes to accompany the videos. Interviews will analyze the creative processes and gauge satisfaction. The goal is to understand the impact of XR tools on creation and propose methodologies for artists new to virtual reality. Results may include spectator testing and comparisons between augmented and 360-degree videos.

1. Collective of Digital Artists and
researchers in Virtual, Mixed and
Augmented Reality

3. Style Transfer
Video/ Audio 360° Neural Networks
Movement extraction3D Modelling

4. Andres Burbano

5. Which is the impact in
the creation process of
scenic artists and
composers when using
XR technologies to
create content?

6. Motivation
●A great convergence between tools and technologies to create immersive content for XR
○ Video 360°-180° / Stereo
○ Sound 3D (Ambisonics/binaural)
●Expensive and complex towards simple and affordable
○ Understand the technology (Stitch, calibration, 360°, stereo)
○ Soundfields sound and ambisonics
●360° is always a “sequence shot”
○ Lighting and other structures
○ Only one shoot - No mistakes

7. Ambisonic codification
360° Video - 3D Sound Pipeline
Sound signal
Video signal
360ª video
capture
Recording Production
W Y
Z
X

8. 360° Image Pipeline
https://blogs.unity3d.com/2018/01/26/stereo-360-image-and-video-capture/
b)Equirectangular stereo scene c)Equirectangular stereo Video
a) Raw Rig images

9. NaturalSoundFieldVirtualcomposition
Stereo
material
Mono source
Reverberation Codification
Ambisonic
sound recording
B format
B format
Conversion
Transformation
B format
Decodification
Restitution
Transmission
storage (multicanal
format)
Introduction to Ambisonics – Lecture notes v0.5 – Daniel Arteaga 2018

10. Study design
●Artist
○ 12 artist - 12 composers
○ Ages 19 – 38
○ First time experience with XR technologies
○ First time as VR content creator
●Methodology
○ 4 different session with actors and 4 interviews
○ 5 steps process with composers and 2 interviews
● Objective
○ To create twelve 360° videos and a thematic analysis from the interviews

11. Dongming Wang
& Gabriela Jaimes
Camille Rau
& Dario Guerrero
Julián García
& Salsangroove
Mateu Tur &
Juan Camilo Guzmán
Ela Matassini
& Ramiro Prada
Effie Gonis
Livia Tepchi
& Nícolás Gómez
Goal: produce 12 videos in 360º and 3D audio with artists
from different backgrounds and cultural environments.

12. Video
Guidelines●Video duration must be less that 5 minutes
●Performance is a free improvisation
●Camera placed in the middle of the scene
●360° sequence shot
b) Dual lens Mountinga) lighting set-up

13. Procedure: Performer
●Project and technology explanations
●Technical workshop to engage with the final user
somatic experiences.
●Shooting the video
●Post-production interview

14. Procedure composer
●The composer receives a list of requirements from the artist to creates
musical mockup.
● The mockup is used to record the video.
●Last adjustments based on the recorded video in 2D format.
●Deliver a final production in stereo
●Deliver a 3D soundtrack in Ambisonic.

15. Expected results from
the interviews:
●Analyses the different artists experiences in
order to find common traits and structures in the
creation process to finally propose an
introductory methodology for artists that
started working with VR technologies.

16. Future work
●Evaluates the accomplishment of aesthetic experience of the 360° videos and 3D
audio with a group of spectators
●Compare the spectator somatic experiences engagement between the “2D
movement augmented” videos and 360° videos

17. Questions
Contact:
Andres Burbano
collectif.anvar@gmail.com

18. My Questions for You
●When to do the interview, just after or give time to process the experience? Both?
●Transcripts of the videos?
●Should I add in the scientific question the research of a introductory methodology for artist to create 360?
●Should I look for directors and choreographers?

19. Interview I
● I ask for a presentation of the artist (name, type of art, experience)
● Have you participated in virtual or augmented reality productions before?
● Before contacting you and accepting to participate in this project, did you know that
virtual reality existed?
● Do you know what is Virtual and Augmented Reality?
● (If applicable) Do you know what the difference is between the two?
● Have you had contact with virtual or augmented reality? When? Which kind of
devices?
● Before you knew that you were going to participate in this project, did you know that
360 videos existed?
● Do you know what is a 360 video?
● What do you expect as a result of your artistic execution in this project?

20. Interview II
●Is there a difference between this experience and the executions that you normally do?
●What was the biggest challenge you encountered for performing to this 360° video?
●Which do you think it is the influence you had in your artistic creation process shooting this 360° video?

21. Interview III
●Between 1 and 10 what is your satisfaction of the result of the video? 0 disappointment - 10 completely
satisfied?
●What would you change?
●What could be improved?
●Do you want to participate in another V/AR project?
●Between 1 and 10 how much do you want to participate in future VR/A project?
●Would you like to participate in a second version of this project?

22. 360° Video Pipeline

23. Video 360° Pipeline
TOWARDS GENERATING AMBISONICS USING AUDIO-VISUAL CUE FOR VIRTUAL REALITY A. Rana, C. Ozcinar, and A. Smolic. ICRA 2019
https://v-sense.scss.tcd.ie/research/towards-generating-ambisonics-using-audio-visual-cue-for-virtual-reality/
(I) Recording II) Production (IV) Ambisonics encoding.

24. Audio Pipeline
Introduction to Ambisonics – Lecture notes v0.5 – Daniel Arteaga 2018