Introduction to Motion Tracking to Dance


Published on

Presentation of Robert Wechsler/Palindrome iIntermedia Group at the Geneva Sessions 09 /Made in Lausanne

Published in: Design, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Introduction to Motion Tracking to Dance

    1. 1. Motion Tracking Robert Wechsler Palindrome I nter .media Performance Group Stuttgart, Germany Lausanne, June 2009
    2. 2. workshop concept "dance first, think afterwards." - Estragon (S.B.) "to work with motion tracking as an artist, start and finish with the body, not with the technology." - Robert (Palindrome)
    3. 3. workshop plan 1 Training. non-dancers: 11:30-11:55 ok? 2 Talk / Discussion / video clips 40 min 3 Improv. Chor. programming (patch-writing). 4 Tech. Sessions 5 Project work 6 Stretching, etc.
    4. 4. Talks from me (12-12:40) Talk 1: "What is Palindrome? A very brief history of interactive art. Motion tracking and motion capture." Talk 2: "Sensors: body (physiological) vs. environment-based systems. mapping." Talk 3: "Practical issues: how to get the equipment, etc. A critical discussion of video projections."
    5. 5. Palindrome Performance Group <ul><li>Palindrome is a kind of dance company.... </li></ul><ul><ul><li>new york 82-90, germany 91-09, france?? 09-... </li></ul></ul><ul><ul><li>performances, some installations, touring, workshops </li></ul></ul><ul><ul><li>works with interactive music, video and lights </li></ul></ul><ul><ul><li>develops software hardware applications* </li></ul></ul><ul><ul><li>specialized in input and mapping </li></ul></ul><ul><li>* - I'm not an engineer, but work _with_ engineers, by itself a special skill... </li></ul>
    6. 6. what is motion tracking ? <ul><li>technologies that collect data on human movement (input) </li></ul><ul><li>used to control sounds, music, recorded or projected text, video art, stage lighting (output) </li></ul><ul><li>via performer actions / gestures / movements / bio-data </li></ul><ul><li>used in interactive performances and installations </li></ul><ul><li>other uses.... </li></ul>
    7. 7. what is motion tracking ? <ul><li>other uses: </li></ul><ul><ul><li>animation modeling (motion capture) </li></ul></ul><ul><ul><li>scientific research (musicology, semantics, ergonomics, medicine, sports medicine, architecture) </li></ul></ul><ul><ul><li>therapy for physically and mentally handicapped </li></ul></ul>
    8. 8. motion tracking vs. motion capture <ul><ul><li>motion capture </li></ul></ul><ul><ul><li>tracks location of fixed positions on body </li></ul></ul><ul><ul><li>highly accurate, expensive ($200k-2m) </li></ul></ul><ul><ul><li>generally not realtime </li></ul></ul><ul><ul><li>used for data collection (research) and for making human or animal motion in animations (films, games, etc.) more realistic </li></ul></ul><ul><ul><li>motion tracking </li></ul></ul><ul><ul><li>less equipment, less data, less cost (ca. $1k-2k) </li></ul></ul><ul><ul><li>concerned with motion qualities like dynamic, direction of motion </li></ul></ul><ul><ul><li>real time </li></ul></ul><ul><ul><li>used for live applications: installation art, dance, theater and music </li></ul></ul>
    9. 9. motion capture <ul><ul><li>tracks location of fixed positions on body with reflective markers </li></ul></ul><ul><ul><li>12-24 cameras, each lens is ringed with infrared lights </li></ul></ul>
    10. 10. <ul><li>Vicon is a leading company in motion capture </li></ul>motion capture
    11. 11. motion capture
    12. 12. motion capture
    13. 13. <ul><li>typical uses </li></ul><ul><ul><li>human movement research (sports, musicology, ergonomics, medicine,...) </li></ul></ul><ul><ul><li>Film and Animation -- often used with 3-D animation (modeling) programs to make animations </li></ul></ul><ul><ul><ul><li>maya ( ) </li></ul></ul></ul><ul><ul><ul><li>houdini ( ) </li></ul></ul></ul>motion capture
    14. 14. motion capture
    15. 15. experiments in real-time mo-capture <ul><ul><li>performances of interactive animated characters: &quot; No DNA &quot; ( http://www. nodna .com ) </li></ul></ul><ul><ul><li>experiments with dance performances </li></ul></ul><ul><ul><ul><li>Motione (2003-5) Arizona State U. ( http:// ame . asu . edu / motione ) </li></ul></ul></ul><ul><ul><ul><li>&quot;how long does an object linger on the end of the volume...&quot; (Trisha Brown, Kaiser, </li></ul></ul></ul><ul><ul><ul><li>&quot;22&quot; (Bill T. Jones, Kaiser , </li></ul></ul></ul><ul><ul><ul><li>Luc Vanier et al. at the interactive performance facility U of Wisconsin at Milwaukee ( http://www. isl . uiuc . edu /Publications/final%20dance1. pdf ) </li></ul></ul></ul>
    16. 16. input physical human action motion tracking and interactivity media output sounds, music, text, projections, lighting
    17. 17. input physical human action motion tracking computer sensor (e.g. video camera) output device (e.g. loud speakers) media output sounds, music, text, projections, lighting
    18. 18. input physical human action motion tracking computer sensor (e.g. video camera) output device (e.g. loud speakers) media output sounds, music, text, projections, lighting analogue to digital conversion digital to analogue conversion
    19. 19. digital vs. analogue <ul><li>analogue data </li></ul><ul><li>hard to reproduce </li></ul><ul><li>&quot;rich data&quot; (infinite values) </li></ul><ul><li>very high resolution </li></ul><ul><li>more details </li></ul><ul><li>contaminated data (becomes noisy, but rarely fails completely) </li></ul><ul><li>digital data </li></ul><ul><li>easy to reproduce </li></ul><ul><li>lower resolution, less „human-feel“ </li></ul><ul><li>easy to store </li></ul><ul><li>easy to process </li></ul><ul><li>contaminated data remains clean (errors can be filtered) or signal fails altogether </li></ul>
    20. 20. input physical human action motion tracking sensor (e.g. video camera) output device (e.g. loud speakers) media output sounds, music, text, projections, lighting computer
    21. 21. What does motion tracking have to do with interaction or art?
    22. 22. interaction means human being 1 human being 2 <ul><li>a back and forth of energy, impulse and ideas </li></ul>
    23. 23. Drawings. France and Spain ca. 3000 BC interactive performance is nothing new -- you don't need computers to do it!
    24. 24. interaction <ul><li>who with whom? what with what? </li></ul><ul><li>between performers ? </li></ul><ul><li>between performer and audience ? </li></ul><ul><li>between audience members ? </li></ul><ul><li>between humans and machines ? </li></ul>
    25. 25. interaction vs. automation processing <ul><li>interaction </li></ul><ul><li>is an integral part of human life </li></ul><ul><li>is instinctive, deeply felt, highly communicative </li></ul><ul><li>can and usually does occur on many sensory levels </li></ul><ul><li>is the heart and soul of live performance </li></ul><ul><li>automation </li></ul><ul><li>if new, excites curiosity </li></ul><ul><li>feels like &quot;future vision &quot; </li></ul><ul><li>hugely popular in film, tv... </li></ul>processing action re-action human action human re-action
    26. 26. a very brief history of technology -supported interaction in performance Dionysis Thr. Athens - 3c BC
    27. 27. Ancient greek theater employed a number of mechanical devices to extend the power of the actor to allow them to play gods and goddesses.
    28. 28. Louis XIV, the Sun King, was fascinated by remote-acting mechanical devices
    29. 29. on the battle field. Cannons allowed you to take deadly action from a distance (500 meters). &quot;gesture amplification&quot;. (not an interactive device...)
    30. 30. Electronics, interactive sound and lighting devices were used by New York performance artists in the 60's and 70's Robert Rauschenberg - &quot;Pelican&quot; 1963 Laurie Anderson - &quot;Walking Solo for Tape Bow Violin&quot; 1979
    31. 31. reasons to use motion tracking <ul><li>amplification of gesture </li></ul><ul><li>makes possible new ways of associating and connecting media </li></ul><ul><li>_can_ be interactive </li></ul><ul><li>sense of spontaneity and engagement </li></ul><ul><li>environment becomes lively and responsive </li></ul><ul><li>text and subtext (body can speak, as well as mouth) </li></ul><ul><li>synesthesia </li></ul><ul><li>the technology itself is catchy. marketable. </li></ul>
    32. 32. end of part 1 .
    33. 33. Motion Tracking - part 2 Talk 2: sensors mapping eyecon instruction Talk 3: what you need to track motion different software and hardware available where to buy it cheaply video projection techniques artistic issues robert wechsler
    34. 34. motion tracking sensors <ul><li>body -oriented </li></ul><ul><li>environment -oriented </li></ul>
    35. 35. motion tracking senso rs <ul><li>body -oriented </li></ul><ul><ul><li>body-worn electrodes </li></ul></ul><ul><ul><ul><li>EEG </li></ul></ul></ul><ul><ul><ul><li>ECG </li></ul></ul></ul><ul><ul><ul><li>EMG </li></ul></ul></ul><ul><ul><ul><li>Touching </li></ul></ul></ul><ul><ul><li>joint bend-sensors </li></ul></ul><ul><ul><li>accelerometers, other body-worn devices </li></ul></ul><ul><li>environment -oriented </li></ul><ul><ul><li>video camera-based (eg. eyecon) </li></ul></ul><ul><ul><li>infrared camera-based </li></ul></ul><ul><ul><li>floor sensors, wall sensors, etc </li></ul></ul>
    36. 36. m apping <ul><li>cause and effect. what does what. </li></ul><ul><li>mapping has 3 parts: </li></ul><ul><ul><li>input </li></ul></ul><ul><ul><li>output </li></ul></ul><ul><ul><li>logical relationship </li></ul></ul>
    37. 37. <ul><li>input parameter relationship/processing output parameter </li></ul>components of m apping
    38. 38. relationship/processing <ul><li>logical direction </li></ul><ul><li>compliance </li></ul><ul><li>transparency (clearness) </li></ul><ul><li>diffusion (subtlety) </li></ul>
    39. 39. m apping mapping choices multiply: input parameters x output parameters x logical relationships = number of mappings there are important human perception issues in these choices! some can be perceived, many can not!
    40. 40. <ul><ul><ul><li>video </li></ul></ul></ul><ul><ul><ul><li>body-part locations (presence /absence ), e.g. body part s </li></ul></ul></ul><ul><ul><ul><li>movement dynamic (in defined zones) </li></ul></ul></ul><ul><ul><ul><li>movement direction (left-right, up-down) </li></ul></ul></ul><ul><ul><ul><li>size factors (height, width) </li></ul></ul></ul><ul><ul><ul><li>position in room (or stage) </li></ul></ul></ul><ul><ul><ul><li>bio- sensors </li></ul></ul></ul><ul><ul><ul><li>physiological properties ( muscle contractions, joint bendign, etc. </li></ul></ul></ul><ul><ul><ul><li>contact with objects or other people </li></ul></ul></ul>mapping &quot;useful&quot; inputs
    41. 41. mapping &quot;useful&quot; outputs <ul><ul><li>internal </li></ul></ul><ul><ul><ul><li>audio synthesizer (on/off, volume control, pitch bend, panning) </li></ul></ul></ul><ul><ul><ul><li>audio samples (on/off, volume control, pitch bend, panning) </li></ul></ul></ul><ul><ul><ul><li>images, videos (on/off, scrubbing) </li></ul></ul></ul><ul><ul><ul><li>external (secondary machines) </li></ul></ul></ul><ul><ul><ul><li>sound programs (soundforge, MAX/msp, ...), algorithmic composition </li></ul></ul></ul><ul><ul><ul><li>controlling stage lighting (DMX) </li></ul></ul></ul><ul><ul><ul><li>realtime graphics (Director, Flash, Isadora, Kalypso, Nato*, Jitter*, SoftVNS*, SquishedEyeball*, ...) </li></ul></ul></ul><ul><ul><ul><li>mechanical devices </li></ul></ul></ul><ul><ul><ul><li>* objects that interact with cycling74‘s „Jitter“, the graphics processing environment for MAX/msp. </li></ul></ul></ul>
    42. 42. mapping &quot;useful&quot; relationships Boolean (logic in on/off relationships) •  positive: movt.=media, no movt.=no media •      negative: movt.=no media, no movt.=media •  toggle: movt.=media, 2nd. movt.=no media • iteration: movt.=media, 2nd movt.=2nd media, etc. •   random, sequential or simultaneous effects continuous controllers •  higher/lower – as in a slider (loudness, pitch, filters, etc.) •    thresholds – beyond which a certain change occurs •  
    43. 43. motion tracking systems <ul><ul><ul><li>EyeCon ( ) </li></ul></ul></ul><ul><ul><ul><li>Isadora ( ) </li></ul></ul></ul><ul><ul><ul><li>Eyes Web ( ) </li></ul></ul></ul><ul><ul><ul><li>Very Nervous Syste m ( ) </li></ul></ul></ul><ul><ul><ul><li>Cyclops ( ) * </li></ul></ul></ul><ul><ul><ul><li>CV.jit ( ) * </li></ul></ul></ul><ul><ul><ul><li>MIDI Dancer ( http:// www . troika . org ) </li></ul></ul></ul><ul><ul><ul><li>Big Eye ( ) </li></ul></ul></ul><ul><ul><ul><li>* - programmed in Jitter the graphics processing environment for MAX/msp. </li></ul></ul></ul>
    44. 44. end of part 2 . robert wechsler
    45. 45. Motion Tracking - part 3 Talk 3: what you need to track motion different software and hardware available where to buy it cheaply video projection techniques artistic issues next steps... Eyecon Training how to control images and video how to connection eyecon to MAX/mp Project Work robert wechsler
    46. 46. what you need to track motion Of course there are different ways to do it. Different artists have different styles of working, different needs... Advantages of EyeCon flexibility, sensitivity -- you can use it in many different ways ease of use, quick start-up time, DIY (do it yourself) relatively low cost To use EyeCon in the &quot;best way&quot;, requires 4 things: computer running windows software camera industrial framegrabber*
    47. 47. what you need to track motion <ul><li>Computer needs video input </li></ul><ul><li>Digital video (Firewire, USB2) </li></ul><ul><ul><ul><ul><li>+ digital cameras (camcorder, webcams) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>+ low noise </li></ul></ul></ul></ul><ul><ul><ul><ul><li>+ works with laptops </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- latency issues </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- image resolution issues (smaller chip sizes) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- limited cable length </li></ul></ul></ul></ul><ul><li>Analog video </li></ul><ul><li>+ &quot;unlimited&quot; cable length </li></ul><ul><li>+ lower latency </li></ul><ul><li>+ even digital cams usually have analog output </li></ul><ul><li>- cost more (although many older cameras work quite well) </li></ul><ul><ul><ul><ul><li>- works less well with laptops... (?) i.e. need an external or internal framegrabber </li></ul></ul></ul></ul>
    48. 48. what you need to track motion Framegrabber captures, or digitizes video (turns analog video into digital video) desktop allows for industrial framegrabber: must be: Falcon (LeoLite, et al.)
    49. 49. what you need to track motion Camera almost any camera will work, but... chip size 1/2&quot; allows WIDE angles and high resolution does not need to be color
    50. 50. what you need to track motion Lowest Costs desktop computer get a used one? min. 1 GHz 300 e new, shuttle, barebones, 4 GHz 600 e framegrabber 250 e camera 1/2&quot; chip 350 e zoom lens (4-12 mm) 75 e software licenses 100 – 500 e Total Equipment Cost: 1000 – 2000 e Training (w/ me :) ) includes building equipment 1 week 1000, 3 day intensive 600
    51. 51. motion tracking systems <ul><ul><ul><li>EyeCon ( ) </li></ul></ul></ul><ul><ul><ul><li>Isadora ( ) </li></ul></ul></ul><ul><ul><ul><li>Eyes Web ( ) </li></ul></ul></ul><ul><ul><ul><li>Very Nervous Syste m ( ) </li></ul></ul></ul><ul><ul><ul><li>Cyclops ( ) * </li></ul></ul></ul><ul><ul><ul><li>CV.jit ( ) * </li></ul></ul></ul><ul><ul><ul><li>MIDI Dancer ( http:// www . troika . org ) </li></ul></ul></ul><ul><ul><ul><li>Big Eye ( ) </li></ul></ul></ul><ul><ul><ul><li>* - programmed in Jitter the graphics processing environment for MAX/msp. </li></ul></ul></ul>
    52. 52. motion tracking - issues for artists <ul><li>Artists use technology differently than scientists and engineers! They have different needs, different criteria. </li></ul><ul><ul><li>1 working with engineers </li></ul></ul><ul><ul><li>- higher tech, flashier effects </li></ul></ul><ul><ul><li>- better control, more reliable </li></ul></ul><ul><ul><li>- less problems? probably not. different problems... </li></ul></ul><ul><ul><li>- you will find yourself making work _about_ technology – whether you intended to or not! </li></ul></ul><ul><ul><li>2 working alone </li></ul></ul><ul><ul><ul><li>- easier to focus on artistic issues </li></ul></ul></ul><ul><ul><ul><li>- simpler systems </li></ul></ul></ul><ul><ul><ul><li>- less flashy (= less attention, less money) </li></ul></ul></ul><ul><ul><ul><li>- requires tons of patience </li></ul></ul></ul>
    53. 53. movies, sounds and lights Media can compliment and support the live performer or it can distract from her-him. This means it depends on how you do it. And there is a fundamental issue to think about...
    54. 54. 1. make the image smaller 2. make the image less bright, or black & white 3. lower the resolution of the film, &quot;pixilate&quot; it or use various image abstraction methods 4. use non-human material -- human images and faces distract most of all 5. use semi-transparent screens 6. put the screen up high, down low, in the back, or use thin materials 7. project on the side walls, on the floor or ceiling 8. project on the bodies of the performers 9. have the projection incorporate the performer's live image 10. put the image under the control of the of the performer's movement (ie. make it interactive) movies (moving images) are extremely distracting to the live performer.
    55. 55. APPROACH 2: Use Stills. Low Motion Slow Motion.
    56. 56. <ul><li>Palindrome Intermedia Performance Group </li></ul><ul><li>Krisztina de Chatel </li></ul><ul><li>Igloo </li></ul><ul><li>Ventura Dance (Pablo Ventura) </li></ul><ul><li>Robert Lepage </li></ul><ul><li>André Werner </li></ul><ul><li>Marlon Barrios Solano </li></ul><ul><li>La la la Human Steps </li></ul><ul><li>Georg Hobmeier </li></ul><ul><li>Leine Roebana Dans Kompanie </li></ul><ul><li>Troika Ranch </li></ul><ul><li>Blue Man Group </li></ul><ul><li>you </li></ul>who is using motion tracking?
    57. 57. motion tracking - review Positive amplification of gesture sense of spontaneity and engagement text and subtext (words say one thing, body says another) can make art interactive in new ways generates genuinely new ways of associating and connecting media helps to get grants and invitations Negative substitute art with special effects complex processes -- compromise quality and subtlety of expressive media. extremely time consuming, distracting life-less materials and tools de-humanize art and artists
    58. 58. Motion Tracking End of part 3. &quot;dance first, think afterwards.&quot; - Estragon (S.B.)
    59. 59. e yecon