Computer Vision For Computer Music

3,405 views

Published on

Presentation given in the Computer Music PhD program at the School of Arts (http://artes.ucp.pt)

Jorge Cardoso - http://jorgecardoso.eu

Published in: Education, Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,405
On SlideShare
0
From Embeds
0
Number of Embeds
94
Actions
Shares
0
Downloads
84
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Computer Vision For Computer Music

  1. 1. Computer Vision Techniques for Musical Interaction Using Processing Computer Music Doctoral Program http://www.artes.ucp.pt/si/doutoramento/index.html 16-05-2009 Jorge Cardoso 1
  2. 2. 2 Topics • Introduction to session – What (is this session about) – How (the session will go through) – Who (am I) • Techniques – Examples and live demos – Brightness/Color detection – Background subtraction – Movement detection – Movement estimation – Face Detection • Techniques – Code Overview – (Depending on time left) – Mixed Initiative (tell me what you want to see in more detail) 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  3. 3. 3 What • Overview of some computer vision techniques – How they work and what are the main limitations – How to use them in Processing • Some –very simple(istic)– examples of how they can be used for musical interaction “Computer vision is the science and technology of machines that see. As a scientific discipline, computer vision is concerned with the theory for building artificial systems that obtain information from images” – Computer vision. (2009, April 24). In Wikipedia, The Free Encyclopedia. Retrieved 11:53, May 9, 2009, from http://en.wikipedia.org/w/index.php?title=Computer_vision&oldid=285848177 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  4. 4. 4 What • Techniques – Color detection (where in the image, does a given color appear – can be used for tracking objects) – Brightness detection (what is the position of the brightest pixels) – Background subtraction for object vectorization – Movement detection (where –coordinates– in the image has movement occurred) – Movement estimation (what direction are the objects moving) – Face detection (is there a face in the image? where?) 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  5. 5. 5 How • General overview of how those CV techniques work – Will try not to dwelve much into code in this phase • Examples of what can be accomplished with the techniques – Vídeos – Live demos • Depending on time left – Tell me if you want to see something in more detail – More detailed description of code (libraries needed, etc) 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  6. 6. 6 Who • Jorge Cardoso (http://jorgecardoso.eu) • Teacher of Interactive Video Art course (School of Arts) – Mainly computer vision techniques for (full) body interaction with screen projections – Students use Processing and CV techniques for interactive art projects 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  7. 7. 7 Color detection • Isolate color regions in an image • Procedure: – Go through the image, pixel by pixel – Calculate the distance between the color of the current pixel and the reference color – If distance is smaller than a given threshold, keep the pixel 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  8. 8. 8 Color Detection • Q: How do we calculate the “distance” between two pixels? • A: Simple version: euclidean distance. – Take a color as a point in 3D space (RGB -> XYZ) – Calculate the distance between the two points • Subtract each component (dR = R1-R2, dG = G1-G2, dB = B1-B2) • Dist = Sqrt(dR*dR + dG*dG + dB*dB) • A1: Slightly (perceptively) better version: weight each component differently: • Dist = Sqrt(2*dR*dR + 4*dG*dG + 3*dB*dB) 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  9. 9. 9 Color Detection • Q: What’s it good for? • A: – Catch of the day [video] – Play-doh as Piano Keyboard [video] – ColorDifferenceInstrument [live demo] – ColorDifferenceDistanceInstrument [live demo] – ColorDifferenceAngleInstrument [live demo] 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  10. 10. 10 Brighteness detection • Slight variation on color detection when we’re only interested in tracking bright pixels – For example, to track a light • Procedure: – Same as color detection but extract the brightness of each pixel and keep only the highest 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  11. 11. 11 Brighteness detection • Q: How do we calculated the “brightness” of a color? • A: – In RGB: • (perceptive) Brightness = (0.2126*R) + (0.7152*G) + (0.0722*B) • (physical/energy) Brightness = (R+G+B)/3 • … – In HSB: • Brightness = B ☺ – Just use the brightness() function in whatever language… 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  12. 12. 12 Brighteness detection • Q: Why not just color detection? • A: – Brightness detection is more robust (i.e., less influenced by lighting conditions) – You don’t care about color, just the brightness (for example to detect a lantern) 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  13. 13. 13 Brighteness detection • Q: What’s it good for? • A: – Drawing With Light [video] – Burning the Sound [video] 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  14. 14. 14 Background subtraction • If we have a fixed camera, and a background image for reference, we can subtract the background to each frame to isolate new objects This - This = This 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  15. 15. 15 Background subtraction • Q: How do we “subtract” two images? • A: Pixel by pixel • Q: How do we “subtract” two pixels? • A: Channel by channel (color component by color component, usually R, G and B) 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  16. 16. 16 Background subtraction • Pixel subtraction alone doesn’t work very well due to noise or shadows in the image • The resulting subtraction must be thresholded to eliminate noise 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  17. 17. 17 Background subtraction • After subtracting and thresholding, we can binarize the image to get a mask 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  18. 18. 18 Background Subtraction • Q: What’s it good for? • A: Isolating objects in a scene. The mask can be vectorized, giving us access to it’s contour (polygon). – Bg Subtraction [Live demo] – Position Through Background [Video] – BouncingBalls [Live demo] – SoundingPolygon [Live demo] 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  19. 19. 19 Movement Detection • If we subtract two consecutive video frames, threshold and binarize it, we get a rough view of the movement in the video • Since we don’t need a reference frame, this technique is more rubost in face of lighting conditions 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  20. 20. 20 Movement Detection • Q: What’s it good for? • A: – WebCam Piano [video] – Air Piano [live demo] 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  21. 21. 21 Motion Estimation (aka Optical Flow) • Estimate the movement vectors of blocks in a sequence of two images • Procedure – Divide both images into blocks – For each block in image one, find the closest (more similar) block in image two Image: http://grauonline.de/wordpress/wp-content/uploads/ovscreenshot1.png 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  22. 22. 22 Motion Estimation • Q: What’s it good for? • A: – BryanChung Example [video] – OpticalFlowMidi [live demo] 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  23. 23. 23 Face detection • Not recognition! • Needs a configuration file that determins what to detect (front faces, side faces, body, etc) • Returns the position and size of detected faces 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  24. 24. 24 Face Detection • Q: What’s it good for? • A: – Face Detection With A Smiley [video] – Funny Eyes [video] [live demo] 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  25. 25. 25 Summary • Color detection – Needs good lighting – Can track (several) objects by color • Brighteness detection – Robust, needs only slightly controled lighting conditions (low light) – High accuracy of position and movement – Can track several objects at the same time (with some restrictions) 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  26. 26. 26 Summary • Background subtraction – Needs initial setup on startup (reference frame) – Needs highly controled lighting conditions. – Isolates objects without any marker (color or light) – Can be used to determine distance to camera (without great accuracy) – Can be used to detect movement 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  27. 27. 27 Summary • Movement Detection/Estimation – No startup configuration – Doesn’t need very controlled lighting conditions – Determins regions where movement occured – Hard to determine number of objects moving 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  28. 28. 28 Summary • Face detection – Needs relatively controlled lighting conditions – Needs relatively controlled positioning of people (near and looking directly at the camera) – Detects position and size (approximate) of faces 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  29. 29. 29 References • Videos – Catch of the day: http://www.youtube.com/watch?v=G8-Nvyx0xtk – Play-Doh as Piano Keyboard!: http://vimeo.com/465726?pg=embed&sec= – Drawing with Light: http://www.youtube.com/watch?v=VDP3e20uYMI – Burning the Sound: http://www.vimeo.com/3096584 – Position through background subtraction: http://www.youtube.com/watch?v=yyc8aionno4 – Webcam piano: http://www.vimeo.com/1219327?pg=embed&sec=1219327 – Bryan Chung optical flow: http://www.youtube.com/watch?v=WCdQ8KQ_Wyo& – Face detection with a smiley: http://www.youtube.com/watch?v=r4XArSFPwPc – Funny Eyes: http://www.youtube.com/watch?v=9WXLs4qQ5nk 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  30. 30. 30 References • Webpages – Computer vision techniques for artists (by Golan Levin): http://www.flong.com/texts/essays/essay_cvad/ – Processing Website: http://processing.org/ – Processing Books: http://processing.org/learning/books/ – Portuguese Processing Forum: http://processing.jorgecardoso.eu – Color metrics: http://www.compuphase.com/cmetric.htm – Bryan Chung (Optical Flow videos): http://www.bryanchung.net/?p=262 – OpenCV Processing library: http://ubaa.net/shared/processing/opencv/ 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  31. 31. Stuff you need to make the examples 31 run • Processing • OpenCV processing library • Webcam 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  32. 32. 32 The End • Code Overview – anything in particular? • Questions? • … 16-05-2009 Jorge Cardoso Computer Music Doctoral Program

×