0
Dynamic Audio
     Synthesis
on the Flash/Flex Platform

         Joe Berkovitz
Overview

• Bit o’ background
• Flash 9 sound capabilities
• The Flash 10 Sound API
• StandingWave audio synthesis library...
About me
Samples ‘n’ frames
   ‘n’ acoustics
Player 9 Sound API

• Sound objects can be loaded from URLs...
• ...or embedded as assets
• Load ‘em, play ‘em, change vol...
Player 9 Audio
// loading a sound over the network
var s:Sound = new Sound();
s.addEventListener(Event.COMPLETE, handleCom...
Player 9 Audio
// loading a sound from a library asset
[Embed(source=”TheBomb.mp3”)]
private static var MySoundAsset:Class...
What we want to do...

• Dynamically compute sounds from scratch
• Process samples on the fly to shift pitch,
  add effects...
FP10 to the rescue

• Synchronization and continuity:
  SAMPLES_CALLBACK event
• Access to codecs: Sound.extract()
• The V...
Dynamic Audio Output
var s:Sound = new Sound();
s.addEventListener(SampleDataEvent.SAMPLE_DATA,
                   handleS...
Codec Access
[Embed(source=”TheBomb.mp3”)]
private static var MySoundAsset:Class;

function getSoundSamples(numFrames:uint...
Some things to know

• Between 2048 and 8192 samples per event
• The fewer samples you provide, the lower
  the latency (t...
Basic Audio Demo
Going beyond the
        FP10 sound API

• Goal: to work at a higher level than samples
• High level concepts: Sources, Fi...
The StandingWave Project


• Open source AS3 library
• Outgrowth of Noteflight
• Requires Player 10
• Enables new class of ...
Audio Processing Concepts

     Sample

          Descriptor: 44.1 kHz Stereo

              left[0]               right[0...
Audio Processing Concepts



                Samples
                  Sample
                   Sample
                  ...
Audio Processing Concepts


         Samples                  Samples
           Sample                   Sample
         ...
Audio Processing Concepts
                                        timed sequence
starts at 0.0 sec                        ...
Audio Processing Concepts
                                   Samples
                                     Sample
         ...
Audio Processing Concepts
starts at 0.0 sec

                                                                  Audio
    S...
IAudioSource
public interface IAudioSource
{
    function get descriptor():AudioDescriptor;
    function get frameCount():...
IAudioFilter
public interface IAudioFilter extends IAudioSource
{
    function get source():IAudioSource;
    function set...
IPerformance
public interface IPerformance
{
  function getElementsInRange
                   (start:Number, end:Number)
 ...
Noteflight:
A Music Notation Editor
        in Flash
What’s missing?


• Direct microphone input
• More codec libraries
• Fully functional and supported audio
  processing in ...
Where to get the
      StandingWave library


http://code.google.com/p/standingwave
Places and Projects

http://joeberkovitz.com
http://www.hobnox.com
http://labs.adobe.com/wiki/index.php/
Alchemy:Libraries...
Upcoming SlideShare
Loading in...5
×

Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform

2,061

Published on

code coverage measurement tools (Flexcover has gotten a lot of attention this year)

visual editor frameworks (I'm just about to release a new open source project called Moccasin that distills a lot of Noteflight's editing infrastructure into a general framework)

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,061
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
29
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "Joe Berkovitz - Dynamic Audio Synthesis on the Flash Flex Platform"

  1. 1. Dynamic Audio Synthesis on the Flash/Flex Platform Joe Berkovitz
  2. 2. Overview • Bit o’ background • Flash 9 sound capabilities • The Flash 10 Sound API • StandingWave audio synthesis library • New avenues for Flash audio and music
  3. 3. About me
  4. 4. Samples ‘n’ frames ‘n’ acoustics
  5. 5. Player 9 Sound API • Sound objects can be loaded from URLs... • ...or embedded as assets • Load ‘em, play ‘em, change volume or stereo pan – that’s about it! • Unpredictable latency, synchronization
  6. 6. Player 9 Audio // loading a sound over the network var s:Sound = new Sound(); s.addEventListener(Event.COMPLETE, handleComplete); s.load(new URLRequest(“TheBomb.mp3”); function handleComplete(e:Event) { Sound(event.target).play(); }
  7. 7. Player 9 Audio // loading a sound from a library asset [Embed(source=”TheBomb.mp3”)] private static var MySoundAsset:Class; function playSound():void { var s:Sound = new MySoundAsset(); var sc:SoundChannel = s.play(); sc.soundTransform = new SoundTransform(0.5 /*volume*/); // set other transform properties... // examine playback position trace(sc.position); }
  8. 8. What we want to do... • Dynamically compute sounds from scratch • Process samples on the fly to shift pitch, add effects, equalization... • Play more than 32 sounds at the same time • Compute continuous sound output
  9. 9. FP10 to the rescue • Synchronization and continuity: SAMPLES_CALLBACK event • Access to codecs: Sound.extract() • The Vector data type • Adobe Pixel Bender runtime • Alchemy
  10. 10. Dynamic Audio Output var s:Sound = new Sound(); s.addEventListener(SampleDataEvent.SAMPLE_DATA, handleSampleData); var sc:SoundChannel = s.play(); function handleSampleData(e:SampleDataEvent) { for (i = 0; i < /*number of frames*/ ; i++) { e.data.writeFloat(/* left channel */); e.data.writeFloat(/* right channel */); } }
  11. 11. Codec Access [Embed(source=”TheBomb.mp3”)] private static var MySoundAsset:Class; function getSoundSamples(numFrames:uint):void { var s:Sound = new MySoundAsset(); var bytes:ByteArray = new ByteArray(); s.extract(bytes, numFrames, _position); bytes.position = 0; for (var i:Number = 0; i < numFrames; i++) { ... = bytes.readFloat(); // Left channel ... = bytes.readFloat(); // Right channel } }
  12. 12. Some things to know • Between 2048 and 8192 samples per event • The fewer samples you provide, the lower the latency (time before a sample is heard) • Initial latency > steady-state latency • More samples/callback = more stability • Don’t take too long to generate samples!
  13. 13. Basic Audio Demo
  14. 14. Going beyond the FP10 sound API • Goal: to work at a higher level than samples • High level concepts: Sources, Filters • Play back timed, overlapping sequences of sonic or musical events: Performances • Create a modular, extensible system
  15. 15. The StandingWave Project • Open source AS3 library • Outgrowth of Noteflight • Requires Player 10 • Enables new class of audio applications
  16. 16. Audio Processing Concepts Sample Descriptor: 44.1 kHz Stereo left[0] right[0] left[1] right[1] left[2] right[2] left[3] right[3] ... ...
  17. 17. Audio Processing Concepts Samples Sample Sample Audio Source Player raw samples, algorithms...
  18. 18. Audio Processing Concepts Samples Samples Sample Sample Sample Sample Audio Source Filter Player transformations
  19. 19. Audio Processing Concepts timed sequence starts at 0.0 sec of sounds Source Filter starts at 0.2 sec Audio Performance Source Filter Player starts at 0.7 sec Source Filter (...continues...)
  20. 20. Audio Processing Concepts Samples Sample Sample starts at 0.0 sec Source Filter Samples Sample Sample starts at 0.2 sec Audio Audio Performance Source Filter Performer Player starts at 0.7 sec Source Filter ...
  21. 21. Audio Processing Concepts starts at 0.0 sec Audio Source Filter Player starts at 0.2 sec Audio Performance Source Filter Filter Performer starts at 0.7 sec Source Filter (...continues...)
  22. 22. IAudioSource public interface IAudioSource { function get descriptor():AudioDescriptor; function get frameCount():Number; function get position():Number; function resetPosition():void; function getSample(numFrames:Number):Sample; function clone():IAudioSource; } implementations: Sample, SineSource, SoundSource...
  23. 23. IAudioFilter public interface IAudioFilter extends IAudioSource { function get source():IAudioSource; function set source(s:IAudioSource):void; } implementations: EnvelopeFilter, EchoFilter, BandpassFilter...
  24. 24. IPerformance public interface IPerformance { function getElementsInRange (start:Number, end:Number) :Vector.<PerformanceElement>; } implementations: ListPerformance...
  25. 25. Noteflight: A Music Notation Editor in Flash
  26. 26. What’s missing? • Direct microphone input • More codec libraries • Fully functional and supported audio processing in PixelBender/Alchemy
  27. 27. Where to get the StandingWave library http://code.google.com/p/standingwave
  28. 28. Places and Projects http://joeberkovitz.com http://www.hobnox.com http://labs.adobe.com/wiki/index.php/ Alchemy:Libraries http://www.getmicrophone.com/ http://void.andre-michelle.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×