Experimental Mobile Music - Pd/iOS Workshop


Published on

Pd (Pure Data) , MobMuPlat, libpd, pd-for-ios, Xcode, iOS SDK

Experimental Mobile Music - Pd/iOS Workshop

  1. 1. Experimental Mobile MusicPd/iOS WorkshopOpen Media LabSchool of Intermedia ArtChina Academy of ArtMay 23-31, 2013Masayuki Akamatsu / IAMAS
  2. 2. Introduction
  3. 3. akaMasayuki Akamatsu, Ph.D● Media Artist● Professor at IAMAS● Fellow at some comapniesGocco., Soneru, Ma-tilde…● Sound, Visual, Network, Mobile…Real-time PerformanceInteractive InstallationMobile/Network Experience● http://akamatsu.org/赤松 正行
  4. 4. akaIAMAS● Institute of Advanced Media and Sciences● Graduate school for new media arts20 students x 2 years = 40 students19 professors24 hours / 364 days facilitiesPersonal desks / Shared studios● http://www.iamas.ac.jp/E/
  5. 5. akaincubator, 1999● Autonomous Distributed Cooperative System
  6. 6. akaTime Machine!, 2002• Image analysis / Image Processing
  7. 7. akaEarthen Bodied Augur, 2007● Image/Audio Analysis and Processing
  8. 8. akaATOMxGALAXIES, 2007-● Image Processing / Reversed Projection Mapping
  9. 9. akaGame Changer in 2007
  10. 10. akaComputer or Mobile ?
  11. 11. akaDinosaurs vs. Rats (Mammals)
  12. 12. akaMobile Information Environment● Bodily Information Environment● Sensory Information Environment● Universal Information Environment● World-wide Information Environment
  13. 13. akaMobile Apps
  14. 14. akaSekai Camera, 2008
  15. 15. akaSnowflakes, 2008
  16. 16. akaOkeanos Buoys, 2010
  17. 17. akaSync for Japan, 2011
  18. 18. akaUroboros Torch, 2012
  19. 19. akaMIDIWriter, 2012
  20. 20. ● Using App Store (on iOS) or iTunes (on OS X)● Search Keywords“Masayuki Akamatsu” for my apps“ARART” for classic paintings“Bloom Card” for flowersakaTry Apps
  21. 21. akaWhat’s your favorite app ?
  22. 22. akaWhy create an app?● Many excellent music apps● Create your own musicwith your own ideawith your own appAnimoog by Moog Music Inc.
  23. 23. MobMuPlat:Basics
  24. 24. akaCreate a Music app with Xcode● Xcode● Objective-C● Core Audio● DSP algorithm● Difficult!!Okeanos by Masayuki Akamatsumini-composer by Karl Bartos and Masayuki Akamatsu
  25. 25. akaCreate a Music app with Pd● MobMuPlatPd + EditorNo programming codeEasy to developLimited UI/GraphicsLimited OS AccessFor Personal Use● libpd / Pd for iOSPd + XcodeObjective-C codeHard to developFull UI/GraphicsFull OS AccessFor App Store Distribution
  26. 26. akaQuestions: Do you have a skill of…● Max● Pd● C/C++● iOS SDK
  27. 27. akaInstall MobMuPlat● http://mobmuplat.com/● Install MobMuPlat on iOS device● Install MobMuPlat distribution on Mac
  28. 28. akaMobMuPlat (iOS app)
  29. 29. akaMobMuPlat Workflow1. Create an audio patch with Pd2. Create a GUI with MobMuPlatEditor3. Test them with PdWrapper.pd4. Transfer them with iTunes5. Play it with MobMuPlat (iOS)Mac iPhone
  30. 30. akaPd tips for Max usersRun/Edit modeAudio On/OffAudio On/Offoption+clickCreate an object
  31. 31. akaApp 1 - DrumsDrums set by Pearl
  32. 32. akaStep 1: Read an Audio File● Open Pd (on Mac)● Open a new patch● Create and connect objects● Save the patch as “Drums.pd”into the same folder of the audio file● Click the buttonYou’ll see the waveformObjectMessageButtonArrayObjectPatch Cord
  33. 33. akaStep 2: Play the Audio Buffer● Create and connect objects● Start DSP● Click the buttonYou’ll hear the sound
  34. 34. akaStep 3: Create a Communication● Create and Connect Objects● What’s “fromGUI” & “/claps”?
  35. 35. ● Open MobMuPlatEditorYou’ll see a new document● Open “Doc” tab● Choose Screen Size● Choose your Pd file● Save the documentas “Drums.mmp”akaStep 4: Create a GUI document122
  36. 36. akaStep 5: Create a Button● Open “Add” tab● Click “button”● Adjust the positionDo not cover “i” button23 Drag1
  37. 37. akaStep 6: Set the Properties● Open “Prop” tab● Set the Addressas “/claps”12
  38. 38. akaStep 7: Test the GUIPd● Open “PdWrapper.pd”MobMuPlatEditor● Open “Lock” tab● Click the buttonYou’ll hear the soundYou’ll see the messages12
  39. 39. akaStep 8: Transfer Files● Connect iPhone to Mac via USB cable● Open iTunes● Choose your iPhone, Apps tab and MobMuPlat● Put files into MobMuPlat Documents1234Drag&Drop
  40. 40. akaStep 9: Hit your Drum● Open MobMuPlat (on iPhone)● Click “i” button● Choose “Drums.mmp”● Play!
  41. 41. akaExercises: Build your own Drums● Add more buttons● Use your own audio files● Use a imageAdd a panel and choose a image fileMake a button transparent● Control the volumePlay softly or loudly
  42. 42. ● Use “Tidy Up” in Edit menu in Pd● .mmp & .pd are text files you can editakaPd Editing Tips
  43. 43. MobMuPlat:Advanced
  44. 44. akaApp 2 - ThereminLéon Theremin and his invention “Theremin”
  45. 45. ● Receive a system message“fromSystem” and “/tilts”● Convert values to pitch and volumeTilt X “-1.0~1.0” -> MIDI Note Number -> FrequencyTilt Y “-1.0~1.0” -> Volume● Smoothingline● Generate a sine waveformosc~*~dac~akaTheremin Patch
  46. 46. akaTheremin GUITilt emulationPanel with a image
  47. 47. akaExercises: Build your own Synth● Subtractive synthesis● Additive synthesis● Wavetable synthesis● Frequency Modulation synthesis● Phase Distortion synthesis● Physical Modeling synthesis● Sample-based synthesisMOOG Synthesizer
  48. 48. akaTutorial & Reference● Programming Electronic Music in Pdhttp://www.pd-tutorial.com/english/index.html● Pure Data Reference Cardhttp://puredata.info/docs/manuals/pdrefcards
  49. 49. akaApp 3 - DelayEchoplex, a tape delay machine by Mike Battle
  50. 50. akaDelay Patch (incomplete)Input Volume ControlDelay FeedbackInput VolumeAudio InputAudio Output
  51. 51. akaDelay GUI
  52. 52. ● Delay● Reverb● Distortion● Overdrive● Flanger● Chorus● Phaser● Tremolo● EQ● Wah● Pitch ShiftakaExercises: Build your own EffectorBOSS Compact Effectors
  53. 53. akaAudio I/O● Headset Mic Earphone● Breakout adapter● Audio Interface● Feedback is also fun!
  54. 54. akaApp 4 - ChorusThe China National Chorus
  55. 55. akaChorus Patch (Standalone)Calculate Frequency Calculate DurationProduce CycleProduce EnvelopGenerate Sine waveModify Amplitude
  56. 56. akaChorus GUI
  57. 57. akaChorus Patch (Network)Send to Network Receive from NetworkSet GUI Valuesset value without output
  58. 58. akaStandalone vs. NetworkStandaloneNetworkYour iPhoneGUI AudiotoNetwork fromNetworkNetworkAnother iPhoneGUI AudiotoNetwork fromNetworkYour iPhoneGUI AudioAnother iPhoneGUI Audio
  59. 59. ● Multicast (Broadcast)● Port NumberakaOSC (Open Sound Control)
  60. 60. ● Unison● Harmonic● Inharmonic● Beating/Chorus● Overtone● Texture● Additive Synthesis● Granular SynthesisakaExercises: Build your own HarmonyHarmonic (from Wikipedia)
  61. 61. akaNetwork(s)● Natural NetworkThrough the AirPhysical Phenomenon● Human NetworkPlay SessionFeelings / Rules● Electronic NetworkLAN / The InternetProtocols
  62. 62. akaLimitations of MobMuPlat● Pd-vanilla (not Pd-extended)● Poor Audio functions● Poor GUI elements● Poor visual expressions● Poor iOS accessibilities● Can’t create a TRUE app● But Fun and Easy to use!SpaceLab by Iglesia Intermedia
  63. 63. libpd/Pd for iOS
  64. 64. akalibpd● http://libpd.cc● Pd as Audio Engine + iOS SDK (Xcode)
  65. 65. akaInstall Xcode● Open App Store (on Mac)● Search “Xcode”● Click “Free”
  66. 66. akaInstall pd-for-ios● https://github.com/libpd/pd-for-ios● Clone RepositoryClick “Clone in Mac”● Update SubmodulesFollow “README.txt”
  67. 67. ● Open PdTest01.xcodeproj● Choose Scheme● Click “Run”akaRun Sample Projects12
  68. 68. akaTrue App ;-) Test
  69. 69. akaTest PatchReceive a messagefrom app (Objective-C code)Send a messageto app (Objective-C code)
  70. 70. akaCreate a new Project● Open Xcode● Choose File > New > Project…
  71. 71. akaChoose a template12
  72. 72. akaChoose options12
  73. 73. akaSave the project12
  74. 74. akaProject Window
  75. 75. akaCreate a User Interface123Interface BuilderDrag&DropDrag&Drop
  76. 76. akaTest the User Interface12
  77. 77. akaAction and Outlet
  78. 78. akaCreate an Actioncontrol-drag123456Code Editor
  79. 79. akaCreate an Outletcontrol-drag1 23
  80. 80. akaGenerated Codes#import <UIKit/UIKit.h>@interface TestViewController : UIViewController- (IBAction)changeFreq:(UISlider *)sender;@property (retain, nonatomic) IBOutlet UILabel *freqLabel;@end#import "TestViewController.h"@interface TestViewController ()@end@implementation TestViewController- (void)viewDidLoad{[super viewDidLoad];! // Do any additional setup after loading the view,typically from a nib.}- (void)didReceiveMemoryWarning{[super didReceiveMemoryWarning];// Dispose of any resources that can be recreated.}- (IBAction)changeFreq:(UISlider *)sender {}- (void)dealloc {[_freqLabel release];[super dealloc];}@endTestViewController.h TestViewController.m
  81. 81. akaAdd Code and Run#import "TestViewController.h"@interface TestViewController ()@end@implementation TestViewController- (void)viewDidLoad{[super viewDidLoad];! // Do any additional setup after loading the view, typically from a nib.}- (void)didReceiveMemoryWarning{[super didReceiveMemoryWarning];// Dispose of any resources that can be recreated.}- (IBAction)changeFreq:(UISlider *)sender {! self.freqLabel.text = [NSString stringWithFormat:@"%f", sender.value];}- (void)dealloc {[_freqLabel release];[super dealloc];}@endTestViewController.m
  82. 82. akaXcode Tips● Snapshot
  83. 83. akaAdd libpd.xcodeprojDrag&DropClose any other projects first because of workspaces problem
  84. 84. akaAdd Target Dependencies1235476
  85. 85. akaAdd Libraries/Frameworks234Add libpd-ios.aAudioToolbox.frameworkand AVFoundation.framework1
  86. 86. akaAdd Header Search Paths14Drag&DropDouble Click3Add libpd/objcand libpd/pure-data/src2
  87. 87. akaAdd the Pd Patch1Drag&Drop23
  88. 88. akaAdd Codes to TestViewController.h#import <UIKit/UIKit.h>#import "PdBase.h"#import "PdAudioController.h"@interface TestViewController : UIViewController <PdReceiverDelegate>- (IBAction)changeFreq:(UISlider *)sender;@property (retain, nonatomic) IBOutlet UILabel *freqLabel;@property (strong, nonatomic) PdAudioController *audioController;@end
  89. 89. - (void)viewDidLoad{[super viewDidLoad];! // Do any additional setup after loading the view, typically from a nib.! // Create Audio Controller! self.audioController = [[[PdAudioController alloc] init] autorelease];! [self.audioController configureAmbientWithSampleRate:44100 numberChannels:2 mixingEnabled:YES];! self.audioController.active = YES;!! // Open Pd Patch! [PdBase openFile:@"Test.pd" path:[[NSBundle mainBundle] bundlePath]];!! // Set PdBase Delegate! PdBase.delegate = self;!! // Subscribe messages from Pd! [PdBase subscribe:@"freqInHz"];}- (IBAction)changeFreq:(UISlider *)sender {! // Send a message to Pd! [PdBase sendFloat:sender.value toReceiver: @"freq" ];}// Handle float messages from Pd- (void)receiveFloat:(float)received fromSource:(NSString *)source {! if ([source isEqualToString:@"freqInHz"]) {! ! self.freqLabel.text = [NSString stringWithFormat:@"%f Hz", received];! }}akaAdd Codes to TestViewController.m
  90. 90. akaRun the True App !
  91. 91. akaUser Interface - Code - Patch- (IBAction)changeFreq:(UISlider *)sender {! [PdBase sendFloat:sender.value toReceiver: @"freq" ];}- (void)receiveFloat:(float)received fromSource:(NSString *)source {! if ([source isEqualToString:@"freqInHz"]) {! ! self.freqLabel.text = [NSString stringWithFormat:@"%f", received];! }}ActionOutletReceive method(Delegate)Send method
  92. 92. akalibpd Project Summary● Add libpd.xcodeproj● Add Target Dependencieslibpd-ios● Add Libraries/Frameworkslibpd-ios.a, AudioToolbox.framework, AVFoundation.framework● Add Header Search Pathslibpd/objc, libpd/pure-data/src● Add Pd Patch● Add CodesCreate PdAudioController, Set it up and Make it activeOpen Pd patchSet PdBase delegate and Subscribe Pd messagesSend messages to Pd patch and Receive messages from Pd patch
  93. 93. akaFurther References● Pd for iOS wikihttps://github.com/libpd/pd-for-ios/wiki/ios● libpd Objective-C APIhttps://github.com/libpd/libpd/wiki/objc● iOSの教科書http://iosbook.net/
  94. 94. Students Project
  95. 95. ● Solo or Group● 3 minutes● Theme/Motif/Feeling● Audio Processing● User Interface● Composition● PerformanceakaCreate your own musicKraftwerk / Pocket Calculator
  96. 96. akaWhy? (again)● Many excellent music apps● Create your own musicwith your own ideawith your own appAnimoog by Moog Music Inc.
  97. 97. akaMake a Documentation● FormatPDFSingle PageLandscape● ContentsTitleNameDescriptionImages (Screen, Photo, etc.)Title: China My LoveName: Masayuki AkamatsuDescription: “China My Love” is a simple app that transmits MIDI notemessages by hitting keys on a keyboard. Playing an input phrase as asequence is also possible. The tempo and the loop can be set. You canuse either the on-screen software keyboard or a Bluetooth/Dock-connectedhardware keyboard for iOS.
  98. 98. One more thing…
  99. 99. akaiDonationUnneeded iPhone, iPad, iPod touch,I am asking for a kind donationhttp://akamatsu.org/aka/ios/idonation/
  100. 100. 謝謝Thanks to Prof. Dajuin Yao and students