Your SlideShare is downloading. ×
Computer Vision For Computer Music
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Computer Vision For Computer Music

2,664
views

Published on

Presentation given in the Computer Music PhD program at the School of Arts (http://artes.ucp.pt) …

Presentation given in the Computer Music PhD program at the School of Arts (http://artes.ucp.pt)

Jorge Cardoso - http://jorgecardoso.eu

Published in: Education, Technology, Business

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,664
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
72
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Computer Vision Techniques for Musical Interaction Using Processing Computer Music Doctoral Program http://www.artes.ucp.pt/si/doutoramento/index.html 16-05-2009 Jorge Cardoso 1
  • 2. 2 Topics • Introduction to session – What (is this session about) – How (the session will go through) – Who (am I) • Techniques – Examples and live demos – Brightness/Color detection – Background subtraction – Movement detection – Movement estimation – Face Detection • Techniques – Code Overview – (Depending on time left) – Mixed Initiative (tell me what you want to see in more detail) 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 3. 3 What • Overview of some computer vision techniques – How they work and what are the main limitations – How to use them in Processing • Some –very simple(istic)– examples of how they can be used for musical interaction “Computer vision is the science and technology of machines that see. As a scientific discipline, computer vision is concerned with the theory for building artificial systems that obtain information from images” – Computer vision. (2009, April 24). In Wikipedia, The Free Encyclopedia. Retrieved 11:53, May 9, 2009, from http://en.wikipedia.org/w/index.php?title=Computer_vision&oldid=285848177 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 4. 4 What • Techniques – Color detection (where in the image, does a given color appear – can be used for tracking objects) – Brightness detection (what is the position of the brightest pixels) – Background subtraction for object vectorization – Movement detection (where –coordinates– in the image has movement occurred) – Movement estimation (what direction are the objects moving) – Face detection (is there a face in the image? where?) 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 5. 5 How • General overview of how those CV techniques work – Will try not to dwelve much into code in this phase • Examples of what can be accomplished with the techniques – Vídeos – Live demos • Depending on time left – Tell me if you want to see something in more detail – More detailed description of code (libraries needed, etc) 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 6. 6 Who • Jorge Cardoso (http://jorgecardoso.eu) • Teacher of Interactive Video Art course (School of Arts) – Mainly computer vision techniques for (full) body interaction with screen projections – Students use Processing and CV techniques for interactive art projects 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 7. 7 Color detection • Isolate color regions in an image • Procedure: – Go through the image, pixel by pixel – Calculate the distance between the color of the current pixel and the reference color – If distance is smaller than a given threshold, keep the pixel 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 8. 8 Color Detection • Q: How do we calculate the “distance” between two pixels? • A: Simple version: euclidean distance. – Take a color as a point in 3D space (RGB -> XYZ) – Calculate the distance between the two points • Subtract each component (dR = R1-R2, dG = G1-G2, dB = B1-B2) • Dist = Sqrt(dR*dR + dG*dG + dB*dB) • A1: Slightly (perceptively) better version: weight each component differently: • Dist = Sqrt(2*dR*dR + 4*dG*dG + 3*dB*dB) 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 9. 9 Color Detection • Q: What’s it good for? • A: – Catch of the day [video] – Play-doh as Piano Keyboard [video] – ColorDifferenceInstrument [live demo] – ColorDifferenceDistanceInstrument [live demo] – ColorDifferenceAngleInstrument [live demo] 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 10. 10 Brighteness detection • Slight variation on color detection when we’re only interested in tracking bright pixels – For example, to track a light • Procedure: – Same as color detection but extract the brightness of each pixel and keep only the highest 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 11. 11 Brighteness detection • Q: How do we calculated the “brightness” of a color? • A: – In RGB: • (perceptive) Brightness = (0.2126*R) + (0.7152*G) + (0.0722*B) • (physical/energy) Brightness = (R+G+B)/3 • … – In HSB: • Brightness = B ☺ – Just use the brightness() function in whatever language… 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 12. 12 Brighteness detection • Q: Why not just color detection? • A: – Brightness detection is more robust (i.e., less influenced by lighting conditions) – You don’t care about color, just the brightness (for example to detect a lantern) 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 13. 13 Brighteness detection • Q: What’s it good for? • A: – Drawing With Light [video] – Burning the Sound [video] 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 14. 14 Background subtraction • If we have a fixed camera, and a background image for reference, we can subtract the background to each frame to isolate new objects This - This = This 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 15. 15 Background subtraction • Q: How do we “subtract” two images? • A: Pixel by pixel • Q: How do we “subtract” two pixels? • A: Channel by channel (color component by color component, usually R, G and B) 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 16. 16 Background subtraction • Pixel subtraction alone doesn’t work very well due to noise or shadows in the image • The resulting subtraction must be thresholded to eliminate noise 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 17. 17 Background subtraction • After subtracting and thresholding, we can binarize the image to get a mask 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 18. 18 Background Subtraction • Q: What’s it good for? • A: Isolating objects in a scene. The mask can be vectorized, giving us access to it’s contour (polygon). – Bg Subtraction [Live demo] – Position Through Background [Video] – BouncingBalls [Live demo] – SoundingPolygon [Live demo] 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 19. 19 Movement Detection • If we subtract two consecutive video frames, threshold and binarize it, we get a rough view of the movement in the video • Since we don’t need a reference frame, this technique is more rubost in face of lighting conditions 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 20. 20 Movement Detection • Q: What’s it good for? • A: – WebCam Piano [video] – Air Piano [live demo] 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 21. 21 Motion Estimation (aka Optical Flow) • Estimate the movement vectors of blocks in a sequence of two images • Procedure – Divide both images into blocks – For each block in image one, find the closest (more similar) block in image two Image: http://grauonline.de/wordpress/wp-content/uploads/ovscreenshot1.png 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 22. 22 Motion Estimation • Q: What’s it good for? • A: – BryanChung Example [video] – OpticalFlowMidi [live demo] 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 23. 23 Face detection • Not recognition! • Needs a configuration file that determins what to detect (front faces, side faces, body, etc) • Returns the position and size of detected faces 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 24. 24 Face Detection • Q: What’s it good for? • A: – Face Detection With A Smiley [video] – Funny Eyes [video] [live demo] 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 25. 25 Summary • Color detection – Needs good lighting – Can track (several) objects by color • Brighteness detection – Robust, needs only slightly controled lighting conditions (low light) – High accuracy of position and movement – Can track several objects at the same time (with some restrictions) 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 26. 26 Summary • Background subtraction – Needs initial setup on startup (reference frame) – Needs highly controled lighting conditions. – Isolates objects without any marker (color or light) – Can be used to determine distance to camera (without great accuracy) – Can be used to detect movement 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 27. 27 Summary • Movement Detection/Estimation – No startup configuration – Doesn’t need very controlled lighting conditions – Determins regions where movement occured – Hard to determine number of objects moving 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 28. 28 Summary • Face detection – Needs relatively controlled lighting conditions – Needs relatively controlled positioning of people (near and looking directly at the camera) – Detects position and size (approximate) of faces 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 29. 29 References • Videos – Catch of the day: http://www.youtube.com/watch?v=G8-Nvyx0xtk – Play-Doh as Piano Keyboard!: http://vimeo.com/465726?pg=embed&sec= – Drawing with Light: http://www.youtube.com/watch?v=VDP3e20uYMI – Burning the Sound: http://www.vimeo.com/3096584 – Position through background subtraction: http://www.youtube.com/watch?v=yyc8aionno4 – Webcam piano: http://www.vimeo.com/1219327?pg=embed&sec=1219327 – Bryan Chung optical flow: http://www.youtube.com/watch?v=WCdQ8KQ_Wyo& – Face detection with a smiley: http://www.youtube.com/watch?v=r4XArSFPwPc – Funny Eyes: http://www.youtube.com/watch?v=9WXLs4qQ5nk 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 30. 30 References • Webpages – Computer vision techniques for artists (by Golan Levin): http://www.flong.com/texts/essays/essay_cvad/ – Processing Website: http://processing.org/ – Processing Books: http://processing.org/learning/books/ – Portuguese Processing Forum: http://processing.jorgecardoso.eu – Color metrics: http://www.compuphase.com/cmetric.htm – Bryan Chung (Optical Flow videos): http://www.bryanchung.net/?p=262 – OpenCV Processing library: http://ubaa.net/shared/processing/opencv/ 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 31. Stuff you need to make the examples 31 run • Processing • OpenCV processing library • Webcam 16-05-2009 Jorge Cardoso Computer Music Doctoral Program
  • 32. 32 The End • Code Overview – anything in particular? • Questions? • … 16-05-2009 Jorge Cardoso Computer Music Doctoral Program