Sam Fisher presents on making audio engineering learning accessible through virtual reality. He discusses creating an authentic virtual recording studio experience using VR to replace textbooks and allow practice anywhere. Current VR and AR technologies are reviewed along with their applications for audio education, development, and spatial audio techniques. The future of VR/AR is predicted to include more powerful yet smaller devices from major brands, enhancing learning and creativity.

1. Making Audio Engineering Learning and
Practice Accessible through Virtual Reality
Presented by: Sam Fisher
Founder & CEO Audio Fusion LLC
Winter NAMM 2017: Education Track

2. Quick bio:
An aspiring audio guru always looking for challenges to
enhance my experience in respect to sound.
- Graduate of Full Sail University, Recording Arts BA program.
- Self taught musician, programmer, entrepreneur.
- Wanting to empower others to learn audio engineering & sciences.

3. When I Was A Student… Not Long Ago...
- I wanted to be a sound guru
- Attended double-time lab sessions
- President of EDM Organization
I knew the good days were ending…
Began conceptualizing a prototype

4. The Mission:
Empower music creators, audio professionals and education with
cutting-edge tools for productivity and training.

5. The First Step:
An AUTHENTIC recreation of the recording studio experience.
“Enable anyone to be instantly immersed into their dream recording studio. To
learn, practice & familiarize themselves with the recording process”
Studio Access Anytime, Anywhere, Anywhen!

6. Virtual Analog Studio 2D

7. Virtual Analog Studio 3D

8. Now, finally... VIRTUAL REALITY

9. So what exactly
do we offer?
VAS 2D (Desktop PC/MAC)
- Streamlined interface
- Easy to learn
- Quick Navigation
VAS 3D & VR (Desktop,Mobile, HMDs)
- A realistic experience
- Developing muscle & memory reflexes
- Replaces the textbook
Student Design Competition

10. AES 137th Convention
Student Design
Competition 2014
Student Design Competition

11. Movies
1. Virtual Studio 3D/VR Teaser - Click Here
2. VAS Speed Demos - Click Here
3. Mixing a Track in VR - Click Here

12. Current Options for Studio Learning
School/Training Program (Campus/Online)
- “Industrialized Revolution” (Textbooks, PDFs)
- Limited Access to technology (if any)
- Cluttered classrooms and lab sessions
Out of pocket Studio Time
- $$$$ adds up real quick
- Liability
- Might as well buy your own hardware

13. The future… Virtual “classrooms”

14. Virtual Reality in the Classroom
Teachers believe that Virtual Reality can have a large impact on how kids learn.
● 85% feeling it would have a positive impact on their students
● 83% believe VR might help improve learning outcome
● 2% of teachers currently use VR in the classroom
● 85% of teachers say in would have a positive effect on students
● %36 of teachers believe their school districts will adopt VR within 5 years
● %28 of teachers think it will NEVER HAPPEN.

15. Music Education Facts
● Music Students do better in school
Music Student avg GPA 3.59
Non-Music Student avg GPA 2.91
● Music Students have better attendance
Why are music education programs being cut?
Unfortunately, it is the easiest budget cut to make.

16. The Phenomena of Music Passion
How many times have you heard people say?
- “Music is my life!”
- “No one loves music like I love music”
- “I’m going to be famous”
Etc etc etc…
The interest of listening to music has NEVER dropped.
The interest to pursue music has NEVER dropped.
The obstacle lies in the reality that it’s a tough business..

17. Bieber FEVER

18. It All Starts With That Dream...

19. Question:
What has enabled both music
access and creation for all in
the world?
Answer….

20. Technology = …

21. “The Devil’s Music” m/,
Technology has seemingly been cruel to the music industry.
Throughout recent history technology has had very strong rejections.
Examples of Strong Opposition in the Music Industry:
- MIDI performance and editing
- The SYNTHESIZER!
- ( File Sharing / Streaming )
Q - Where are they all now?!
A - More relevant in our business than ever!!

22. Technology == our friend!
- More music is composed & played by individuals than ever before using MIDI
- Users could play sampled sounds and invent never before heard sounds
using analog & digital synthesizers
- Music reaches more people than ever before with streaming services.

23. Now we want to make
Audio Engineering learning
and practice accessible!

24. The advantages of the Virtual World
1) TOTAL immersion for deeper comprehension of any space.
2) Instant access. Experience it at your convenience.
3) Building muscle and memory reflexes.
4) No liabilities!!!! Better to mess up in virtuality than reality
`

25. It’s “like a movie”
Virtual Reality is a very broad term. It doesn’t necessarily define the REALISM
of what is being presented in Virtual space.
Non-fiction movie or documentaries will have exaggerated audio/visual
content to entertain and trigger emotions to drive their point home.
These same concepts work in Virtual Reality. Doesn’t always hurt and could
bring out the best of the experience you’re trying to offer.

26. Giving Users More Access
MORE people want MORE access to MORE things music than ever before.
We want to give passionate music fans
a deeper level of engagement with the
artists & music they already love while
opening an entry point for them to
gain a deeper understanding of the
recording process.

27. How Can Both Software Be Used Now?
● Consoles
● Channelstrips
○ Inputs - Mic/Line
○ Equalization
○ Signal Processing
chains
○ Insert Sends/Returns
○ Aux & Cue Sends
○ Track Busses
○ Stereo Busses
● Master Section
○ Master Gains
○ Stereo Returns
○ Dynamics
○ Track
selection/Grouping
○ Metering
● Patchbay
○ Signal Routing
○ Normalled patch points
○ Unorthodox patching
● Application I/O
○ Internal tape player
○ Soundcard
○ Rewire~
○ SPLIT CONSOLE
● Recording Techniques
○ Intro to signal flow
○ Technology
○ Gain staging
○ Pre/Post processing
○ Parallel Compression
○ Session Organization
○ Recall
EMPOWERING AUDIO EDUCATION
EDUCATORS - Offer stronger student
comprehension through interactivity.
STUDENTS - Put classwork to the test at home.
Maximize lab-hours in the studio.
GRADUATES - Stay fresh on studio techniques
when looking for work in the industry.
Lesson Suggestions:
From the I/O sources to the console all the way
through final steps of the final analog mixing
process. Users are introduced to the basics of
recording technologies and workflow.

28. The Perfect Companions
The 2D Edition replaces the textbook
- Learning Environment
The 3D Edition replaces the studio
- Practical Experience

29. The Future of VAS
The Journey Has Just Begun
Some Ideas:
- Microphone Polar Pattern Visualization
- Device Ports - XLR , ¼ inch , optical
- Customizable Studio Setups
- Multiplayer - Local & Global
- Budget Planner
SO MUCH MORE ON THE WAY!!

31. Headset Options
The Present, Future & Beyond

32. Mobile Headsets , using phones as display
Mobile Headsets
(your phone pretty much)
- Google Cardboard $5+
- Samsung Gear VR $99
- Google Daydream
**Other headsets are available that will
utilize the Cardboard SDK**
The lenses are different for each
headset type

33. High End HMDs (Head-Mounted-Displays)
High End HMD (Head Mounted Displays)
- Oculus Rift (Consumer) $699
- HTC Vive $899
- Playstation VR $499

34. What makes the Hi End HMDs “Hi End”
Virtual Input :
- Sensors
- Controllers
- Computers
Computer Requirements for VR:
- PC with GTX 980 Graphics Card
- Enough room space to track user
- PSVR recommends next-gen PS4
Tracking the user in real space!

35. VR Experience Types
Where and How we can
interact with virtual worlds

36. Room Scale VR - “The Coolest ”
The user designates their
playing ground.
Can walk around and interact
with a given environment.
Maneuvering around spaces
larger than the user area is as
simple as “Teleporting”
HTC Vive requires a minimum
5x6 feet and works up to
maximum of 13x13 feet!

37. Standing & Seated Experiences
The user position spends
majority of the experience in
the same position in space.
- Head tracking of position
and rotation.
- Can bend down and move
head around an object.

38. Stationary VR (Mainly for Mobile)
The user position is stationary
and can only rotate.
The user interacts with the
environment via “Gaze
Control” which basically
means staring directly at a
given object
**Google Daydream is the
first to offer a controller**

39. VR Miscellaneous
Other devices that could be
used to deepen the immersion

40. Gloves and Hand Controllers
- Leap Motion Sensor
- VR Gloves
- Haptic Hand Controller
HTC VIVE has made it possible for developers
to make their own customized controllers to
be tracked in VR.

42. Motion in VR
- Treadmill
- Bicycle
- Haptic cockpit/driver seat
**Developers make their own
mechanisms to simulate
experiences in cards,
constructions etc**

43. Notable Mention : Subpack
“Feel immersive experiential sound and music
with your body.”
- Wearable Tech
- Transfers low frequencies directly to your
body
- Fun for music experiences
**Not exactly a haptic feedback vest**

44. Developing for VR
Creating experiences

45. How to be prepared for VR/AR/360?
- Soundscapes, Sound FX, Compositions more relevant than ever.
SOUND IS THE NEW UI
- Understand the recording and production workflow.
- Know how to work with various software.
- Experiment with interactive techniques.
A virtual world is not a convincing world without convincing sound.

46. Some Game Sound Techniques
Seemless Audio Samples
- Soundscapes
- Music Stems Layering
- Interactive Mixing
SFX Pitch Randomization
- Walking
- Gun Shots
- Bird Chirps

47. World Space Interactive VR applications
1 - UNITY 3D
2 - Unreal Engine
Really, that’s all I am going
to recommend for now...

48. Implementing audio in 3D engines
An “Audio Source” placed in 3D space.
Common properties include roll off,
distance, occlusion, etc
3D game-development platforms
typically have their own 3D audio engines
built-in. Can be de-activated if
another engine/library is prefered.

49. Game Audio Middleware
Integrates with popular game engines.
Our tools to interactive game audio.
- F-MOD
- Wwise
Audio events are triggered when called by
the game engine.
This is where collaboration between and
audio and programmers come in.

50. 360 Video Applications
1 - UNITY 3D
2 - Unreal Engine
3 - Video editor with VR
(Adobe Premier)
Really, that’s all I am going
to recommend for now...

51. Spatial Audio
Audio reproduction techniques
that allow the perception of audio
content in space

52. Ambisonics and Binaural
Ambisonics is a full-sphere surround sound technique: in
addition to the horizontal plane, it covers sound sources
above and below the listener.
**Requires encoding/decoding**
Binaural recordings are reproductions of sound the way
human ears hear it. In fact, the word “binaural” literally just
means “using both ears.” When you listen to a binaural
recording through headphones, you perceive distinct and
genuine 360° sound.

53. Recording Sound for VR/360 Video
- Microphone Arrays
- Dedicated Mics
More microphones are starting
to come to market.
Image is VISISONICS spherical
microphone with Heatmap of
incoming sound field

54. Do we all hear sound the same?
A head-related transfer function
(HRTF) is a response that
characterizes how an ear receives a
sound from a point in space; a pair of
HRTFs for two ears can be used to
synthesize a binaural sound that
seems to come from a particular point
in space.

55. What is the future
of AR/VR?
Rival technologies
Enhancements

56. Hi End Augmented Reality
- Microsoft Hololens
- Magic Leap
A visor that projects holograms
over the real world.
Tracks shape of a space
Even remembers where and
what has been left in given space

57. Low End Augmented Reality is Just a Taste
Overlaying a graphic on a video
feed from a live camera
Example: Pokemon Go

58. Remember that old cell phone?
Just like the first mobile phones
were big and bulky…
The next VR/AR devices will be
smaller and more powerful!!

59. What’s Your Imagination of the Future?
The biggest brands in the world
are developing for VR/AR.
The future looks bright!

60. Thanks for listening!
CONTACT ME IF YOU ARE INTERESTED IN GETTING INVOLVED!
Sam Fisher
Email: szfpro@gmail.com
Founder & CEO Audio Fusion LLC
www.audiofusion.com