0
Motion
Sensors
+ Audio Visual
Performance:
Using Kinect to Control
Live Visuals and Sound

by Joey Bargsten, Ph.D.
Florida...
Thanks!

http://film-gate.org

http://sweatrecordsmiami.com
Format/Overview:
• Presentation / Demo (noon – 12:40)
• Questions? (12:40 – 12:50)
• Break (10 min)
• Workshop on PureData...
Background on Kinect
Background on Kinect
• Released November 2010
Background on Kinect
• Released November 2010
• Sold 8M units in 60 days
Background on Kinect
• Released November 2010
• Sold 8M units in 60 days
• SDK Drama!
Kinect (Development)
Community
• Gaming and Virtuality
Kinect (Development)
Community
• Gaming and Virtuality
• Medicine
Kinect (Development)
Community
• Gaming and Virtuality
• Medicine
• Surveillance, Security, Facial Recognition
Kinect (Development)
Community
• Gaming and Virtuality
• Medicine
• Surveillance, Security, Facial Recognition
• Robotics
Kinect (Development)
Community
• Gaming and Virtuality
• Medicine
• Surveillance, Security, Facial Recognition
• Robotics
...
Kinect (Development)
Community
• Gaming and Virtuality
• Medicine
• Surveillance, Security, Facial Recognition
• Robotics
...
Suggestions,
Approaches
Systems & Configurations
My Configuration
• Kinect Sensor (2011); Model 1414, EN/ES US HD
(this is the most ‘open’ one!)
• Synapse Middleware (conv...
Approaches to Performance
• One-Person Band (virtuoso!)
– One person, one controller
– Controls both audio and visual

Vid...
Approaches to Performance
• Division of Labor (conventional ensembles)
Approaches to Performance
• Division of Labor (conventional ensembles)
“For every new technology that comes along, make
an...
Approaches to Performance
• Division of Labor (conventional ensembles)
“For every new technology that comes along, make
an...
What Do You Want to
Control?
SONIC COMPONENTS
Background: ambient • sustained •
slow rate of change • non-pitched
(environ...
How Do You Want To Control It?
• Discontinuous, trigger events
– model: keyboard

• Continuous, gradual events
– model: th...
Kinect Control Parameters
Kinect Kombo™:

Quad Theremin + Sample Player
• Built in PureData

• Free, cc-by-d-nc
Kinect Kombo™:

Skeleton View + Performer Interface
Kinect Kombo™
Kinect Kombo™
Controlling
Visuals

Kinect and
Quartz Composer
The
Augmented
Dance:
Live
Body
—Kinect—
Virtual
Body
Variations
• Figure
– Simple Camera Figure and Skeleton
– Primitives, 3D Objects, Video Mapped to Skeleton

• Paint Brush/...
“My Profundis (2012)”

Video on YouTube

http://www.youtube.com/watch?v=gT1lbhR1Qj8
Resources
• OpenKinect.org

• V002.info (vade & bangnoise)

• OpenNI.org

• Cycling74.com

• PureData.info

• IdeaChannel ...
Upcoming SlideShare
Loading in...5
×

Motion Sensors + Audio Visual Performance

847

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
847
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • ----- Meeting Notes (2/1/14 16:06) -----
    A quick word of thanks to Dileana Alexander and everyone at Filmgate for hosting a really fantastic array of events in this conference,
    Also thanks to Lolo Resnick and eveyone at Sweat Records for hosting this event!
  • ----- Meeting Notes (2/1/14 16:06) -----
    In the domain of sound, I'm trying to use a language that can be applied across a wide spectrum of genres—pop, jazz, world music, classical, electronic, experimental, avant-garde, etc. Kinect has already found its way into the world of dubstep and experimental eletronic performance.
  • Transcript of "Motion Sensors + Audio Visual Performance"

    1. 1. Motion Sensors + Audio Visual Performance: Using Kinect to Control Live Visuals and Sound by Joey Bargsten, Ph.D. Florida Atlantic University • SCMS • MTEn
    2. 2. Thanks! http://film-gate.org http://sweatrecordsmiami.com
    3. 3. Format/Overview: • Presentation / Demo (noon – 12:40) • Questions? (12:40 – 12:50) • Break (10 min) • Workshop on PureData (1pm – 2pm)
    4. 4. Background on Kinect
    5. 5. Background on Kinect • Released November 2010
    6. 6. Background on Kinect • Released November 2010 • Sold 8M units in 60 days
    7. 7. Background on Kinect • Released November 2010 • Sold 8M units in 60 days • SDK Drama!
    8. 8. Kinect (Development) Community • Gaming and Virtuality
    9. 9. Kinect (Development) Community • Gaming and Virtuality • Medicine
    10. 10. Kinect (Development) Community • Gaming and Virtuality • Medicine • Surveillance, Security, Facial Recognition
    11. 11. Kinect (Development) Community • Gaming and Virtuality • Medicine • Surveillance, Security, Facial Recognition • Robotics
    12. 12. Kinect (Development) Community • Gaming and Virtuality • Medicine • Surveillance, Security, Facial Recognition • Robotics • 3D Scanning and Printing
    13. 13. Kinect (Development) Community • Gaming and Virtuality • Medicine • Surveillance, Security, Facial Recognition • Robotics • 3D Scanning and Printing • Avant-Garde Art, Music, and Performance
    14. 14. Suggestions, Approaches Systems & Configurations
    15. 15. My Configuration • Kinect Sensor (2011); Model 1414, EN/ES US HD (this is the most ‘open’ one!) • Synapse Middleware (converts skeleton to OSC) • PureData v. 0.43.4-Extended (incl. MrPeach library) • Quartz Composer v. 4.0 (+ old FW400 SD camcorder) • 2011 MBP, 2.4mHz, running OSX 7.5 “Lion”.
    16. 16. Approaches to Performance • One-Person Band (virtuoso!) – One person, one controller – Controls both audio and visual Video on YouTube http://www.youtube.com/watch?v=1ge7RcIBWsc
    17. 17. Approaches to Performance • Division of Labor (conventional ensembles)
    18. 18. Approaches to Performance • Division of Labor (conventional ensembles) “For every new technology that comes along, make an ensemble out of it!”—JB, EMV (2011)
    19. 19. Approaches to Performance • Division of Labor (conventional ensembles) “For every new technology that comes along, make an ensemble out of it!”—JB, EMV (2011) • Applied to Kinect: –Multiple controllers –Dedicate separate software to audio and visuals –Multiple performers for audio and visuals
    20. 20. What Do You Want to Control? SONIC COMPONENTS Background: ambient • sustained • slow rate of change • non-pitched (environments) or pitched (“harmonic”) Middleground: active • patterned /predictable rates of change • motor rhythm + overlays + articulation points (“rhythmic”) Foreground: solo • draws attention • contrasting ‘change vocabulary’ • contrasting timbre (“melodic”) VISUAL COMPONENTS Background: abstract • blurry, fuzzy, indistinct • textures • slow rate of change Middleground: referential, pictoral images • abstractions of geometry or biology • text / motion graphics • Foreground: interactive visuals (may also be referential, pictoral, live, connected to sound events)
    21. 21. How Do You Want To Control It? • Discontinuous, trigger events – model: keyboard • Continuous, gradual events – model: theremin • Hybrid Control: multiple parameters – model: sample player
    22. 22. Kinect Control Parameters
    23. 23. Kinect Kombo™: Quad Theremin + Sample Player • Built in PureData • Free, cc-by-d-nc
    24. 24. Kinect Kombo™: Skeleton View + Performer Interface
    25. 25. Kinect Kombo™
    26. 26. Kinect Kombo™
    27. 27. Controlling Visuals Kinect and Quartz Composer
    28. 28. The Augmented Dance: Live Body —Kinect— Virtual Body
    29. 29. Variations • Figure – Simple Camera Figure and Skeleton – Primitives, 3D Objects, Video Mapped to Skeleton • Paint Brush/Image Hose (Effects Mapped to Skeleton) • Background (Texture or Stylized Video Controlled by Skeleton)
    30. 30. “My Profundis (2012)” Video on YouTube http://www.youtube.com/watch?v=gT1lbhR1Qj8
    31. 31. Resources • OpenKinect.org • V002.info (vade & bangnoise) • OpenNI.org • Cycling74.com • PureData.info • IdeaChannel ‘Kinect’ Episode • Synapsekinect.tumblr.com • CreateDigitalMusic.com • CreateDigitalMotion.com • OffBook ‘Creative Coding’ Episode • www.fau.edu/scms (MFA - MTEn) • www.badmindtime.com • Melancholalaland™ (my multimedia opera)
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×