Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Computer vision techniques for interactive art


Published on

Presentation about computer vision using Processing and Jitter for the Interactive Art PhD program at the School of Arts (

  • Be the first to comment

Computer vision techniques for interactive art

  1. 1. Computer Vision Techniques forInteractive ArtUsing Processing and Max/Jitter<br />Interactive Art Doctoral Program<br /><br />5/27/11<br />Jorge Cardoso<br />1<br />
  2. 2. Topics<br />Techniques<br />Framedifferencing(for motiondetection)<br />Color detection(whereintheimage, does a given color appear – can beused for trackingobjects)<br />Brightnessdetection(whatisthepositionofthebrightestpixels)<br />Background subtraction(for objectsegmentation) <br />Blobextraction(applied to theprevioustechniques to extractshapeparametersandvectorizingsilhuettes)<br />Movementestimation (whatdirection are theobjectsmoving)<br />Face detection(isthere a face intheimage? where?)<br />5/27/11<br />Jorge Cardoso<br />2<br />
  3. 3. What<br />Overviewof some computervisiontechniques<br />Howtheyworkandwhat are themainlimitations<br />How to use theminProcessingandJitter<br />Some – verysimple(istic) – examplesofhowthey can beused for interactivework<br />5/27/11<br />Jorge Cardoso<br />3<br />
  4. 4. How<br />General overviewofhowthose CV techniqueswork<br />Examplesofwhat can beaccomplishedwiththetechniques<br />Vídeos<br />Live demos<br /><ul><li>CodeexamplesofthesetechniquesinProcessingandJitter</li></ul>5/27/11<br />Jorge Cardoso<br />4<br />
  5. 5. Computer Vision for Interaction<br />Use a computer and camera to extract information about what people are doing and use that as an implicit or explicit form of interaction<br />“Computer vision is the science and technology of machines that see. As a scientific discipline, computer vision is concerned with the theory for building artificial systems that obtain information from images” – Computer vision. (2009, April 24). In Wikipedia, The Free Encyclopedia. Retrieved 11:53, May 9, 2009, from <br />5/27/11<br />Jorge Cardoso<br />5<br />
  6. 6. Image<br />Computer vision are processes applied to sequences of images<br />An image is just <br />5/27/11<br />Jorge Cardoso<br />6<br />
  7. 7. 5/27/11<br />Jorge Cardoso<br />7<br />Image<br />Computer vision are processes applied to sequences of images<br />An image is just a sequence of number (usually arranged in a matrix form)<br />Eachnumberrepresents a colour<br />50 44 23 31 38 52 75 52<br />29 09 15 08 38 98 53 52<br />08 07 12 15 24 30 51 52<br />10 31 14 38 32 36 53 67<br />14 33 38 45 53 70 69 40<br />36 44 58 63 47 53 35 26<br />68 76 74 76 55 47 38 35<br />69 68 63 74 50 42 35 32<br />
  8. 8. Image<br />Each pixel (i.e., number in the matrix) isusuallystoredin RGB or ARGB formatso<br />It can be decomposed:<br />5/27/11<br />Jorge Cardoso<br />8<br />or<br />
  9. 9. Frame Differencing<br />If we subtract two consecutive video frames, threshold and binarize it, we get a rough view of the movement in the video<br />Procedure:<br />Go through the current frame image, pixel by pixel<br />Calculate the distance between the color of the current frame’s pixel and the previous frame’s pixel<br />If distance is smaller than a given threshold, assume no movement (black), otherwise assume movement (white)<br />5/27/11<br />Jorge Cardoso<br />9<br />
  10. 10. Frame Differencing<br />Color difference<br />Q: How do wecalculatethe “distance”betweentwocolors?<br />A: Simpleversion: euclideandistance.<br />Take a color as a pointin 3D space (RGB -> XYZ)<br />Calculatethedistancebetweenthetwopoints<br />Subtracteachcomponent (dR = R1-R2, dG = G1-G2, dB = B1-B2)<br />Dist = Sqrt(dR*dR + dG*dG + dB*dB)<br />A1: Slightly (perceptively) betterversion: weighteachcomponentdifferently:<br />Dist = Sqrt(2*dR*dR + 4*dG*dG + 3*dB*dB)<br />5/27/11<br />Jorge Cardoso<br />10<br />
  11. 11. Frame Differencing<br />Q: What’sitgood for?<br />A:<br />KALOETHOGRAPHIA (2010) <br />Pedro Cascalheira <br /><br />aB (2011) <br />Jorge Coutinho<br /><br />WebCamPiano <br />MemoAkten<br />[video]<br />5/27/11<br />Jorge Cardoso<br />11<br />
  12. 12. Frame Differencing - Code<br />ProcessingCode<br />FrameDifferencing<br />FrameDifferencingWithMotionDetection<br />FrameDifferencingForAirPiano<br />JitterCode<br />FrameDifferencing.maxpat<br />FrameDifferencingWithMotionDetection.maxpat<br />5/27/11<br />Jorge Cardoso<br />12<br />
  13. 13. Color detection<br />Isolate color regions in an image<br />Procedure:<br />Go through the image, pixel by pixel<br />Calculate the distance between the color of the current pixel and the reference color<br />If distance is smaller than a given threshold, keep the pixel<br />5/27/11<br />Jorge Cardoso<br />13<br />
  14. 14. Color Detection<br />Color Difference<br />For a betterapproachthan RGB distance,use HSB<br />Give more weight to theHue<br />Dist = Sqrt(3*dH*dH + dS*dS + 2*dB*dB)<br />5/27/11<br />Jorge Cardoso<br />14<br />
  15. 15. Color Detection<br />Q: What’sitgood for?<br />A: <br />Catchoftheday [video]<br />PlayDohAsInterfaceKeyboard[video] <br />5/27/11<br />Jorge Cardoso<br />15<br />
  16. 16. Color Detection - Code<br />ProcessingCode<br />ColorDifference<br />ColorDifferenceSequencer<br />JitterCode<br />ColorDifferenceWithFindbounds.maxpat<br />ColorDifference.maxpat<br />5/27/11<br />Jorge Cardoso<br />16<br />
  17. 17. Brightenessdetection<br />17<br />Slightvariationon color detectionwhenwe’reonlyinterestedintrackingbrightpixels<br />For example, to track a light<br />Procedure:<br />Same as color detectionbutextractthebrightnessofeach pixel andkeeponlythehighest<br />Jorge Cardoso<br />5/27/11<br />
  18. 18. 5/27/11<br />Jorge Cardoso<br />18<br />Brighteness detection<br /><ul><li>Q: How do wecalculatethe “brightness” of a color?
  19. 19. A:
  20. 20. In RGB:
  21. 21. (perceptive) Brightness = (0.2126*R) + (0.7152*G) + (0.0722*B)
  22. 22. (physical/energy) Brightness = (R+G+B)/3
  23. 23.
  24. 24. In HSB:
  25. 25. Brightness = B 
  26. 26. Just use thebrightness() functioninwhateverlanguage…</li></li></ul><li>Brightnessdetection<br />Q: Whynotjust color detection?<br />A: <br />In some situationsyoudon’tcareabout color, justthebrightness (for example to detect a lantern)<br />Brightnessdetectionis more robust (i.e., lessinfluencedbylightingconditions)<br />5/27/11<br />Jorge Cardoso<br />19<br />
  27. 27. 20<br />Brightenessdetection<br /><ul><li>Q: What’sitgood for?
  28. 28. A:
  29. 29. DrawingWith Light [video]
  30. 30. BurningtheSound [video]</li></ul>Jorge Cardoso<br />5/27/11<br />
  31. 31. Brightenessdetection - Code<br /><ul><li>ProcessingCode</li></ul>BrightnessDetection<br /><ul><li>JitterCode</li></ul>BrightnessDetection.maxpat<br />5/27/11<br />Jorge Cardoso<br />21<br />
  32. 32. Background subtraction<br />22<br /><ul><li>Ifwehave a fixedcamera, and a background image for reference, we can subtractthe background fromeachframe to isolatenewobjects:</li></ul> This - This = This<br />Jorge Cardoso<br />5/27/11<br />
  33. 33. Background subtraction<br />Q: How do we “subtract” two images?<br />A: Pixel by pixel<br />Q: How do we “subtract” two pixels?<br />A: Channel by channel (color component by color component, usually R, G and B)<br />5/27/11<br />Jorge Cardoso<br />23<br />
  34. 34. Background subtraction<br />Pixel subtraction alone doesn’t work very well due to noise or shadows in the image<br />The resulting subtraction must be thresholded to eliminate noise<br />5/27/11<br />Jorge Cardoso<br />24<br />
  35. 35. Background subtraction<br />After subtracting and thresholding, we can binarize the image to get a mask<br />5/27/11<br />Jorge Cardoso<br />25<br />
  36. 36. Background subtraction<br />Q: What’sitgood for?<br />A: Isolatingobjectsin a scenewhenyoudon’thavemuchcontroloverthe background. <br />Background SubtractionProcessingDemo [video]<br />5/27/11<br />Jorge Cardoso<br />26<br />
  37. 37. Background subtraction - Code<br />Processingcode:<br />BackgroundSubtraction<br />Jittercode<br />BackgroundSubraction.maxpat<br />5/27/11<br />Jorge Cardoso<br />27<br />
  38. 38. Blob extraction<br />All the previous techniques result in possibly many white areas distributed in the image<br />5/27/11<br />Jorge Cardoso<br />28<br /><br />
  39. 39. Blob extraction<br />Blob extraction (blob detection, connected component labeling) identifies those regions and and allows the extractions of some features <br />area, <br />perimeter, <br />orientation, <br />bounding box, <br />Silhouette<br />5/27/11<br />Jorge Cardoso<br />29<br />
  40. 40. Blob extraction<br />Procedure:<br />See<br />5/27/11<br />Jorge Cardoso<br />30<br />
  41. 41. Blob extraction<br />Q: What’sitgood for?<br />A:<br />Loliness (2007) <br />Telmo Rocha [video]<br />Células (2009) <br />Ana Lima [video]<br />SilhouetteCollisions [video]<br />5/27/11<br />Jorge Cardoso<br />31<br />
  42. 42. Blob extraction - Code<br />5/27/11<br />Jorge Cardoso<br />32<br />Processing libraries<br />
  43. 43. Blob extraction - Code<br /><ul><li>ProcessingCode (withFloblibrary)</li></ul>BlobExtractionFromFrameDifferencing<br />BlobExtractionAfterColorDifference<br />ProcessingCode(withJmyronlibrary)<br />Em Mac Intel é preciso substituir o ficheiro libjmyron por este:<br />BlobExtractionAfterBackgroundSubtraction<br />5/27/11<br />Jorge Cardoso<br />33<br />
  44. 44. Blob extraction - Code<br />Jitter Code (com external cv.jit)<br />BlobExtractionAfterBrightnessDetection<br />5/27/11<br />Jorge Cardoso<br />34<br />
  45. 45. Motionestimation(akaOpticalFlow)<br />35<br /><ul><li>Estimatethemovementvectorsofblocksin a sequenceoftwoimages
  46. 46. Procedure (naive)
  47. 47. Divide bothimagesintoblocks
  48. 48. For eachblockinimageone, findtheclosest (more similar) blockinimagetwo
  49. 49.</li></ul>Image:<br />Jorge Cardoso<br />5/27/11<br />
  50. 50. Motionestimation<br />36<br /><ul><li>Q: What’sitgood for?
  51. 51. A:
  52. 52. opticalflow-particles [video]
  53. 53. opticalflow-navigation [video]</li></ul>Jorge Cardoso<br />5/27/11<br />
  54. 54. Motionestimation - Code<br /><ul><li>JitterCode</li></ul>OpticalFlow.maxpat<br />5/27/11<br />Jorge Cardoso<br />37<br />
  55. 55. Face detection<br />Notrecognition!<br />Needs a configuration file thatdetermines what to detect (front faces, side faces, body, etc)<br />Returnsthepositionandsizeofdetected faces<br />5/27/11<br />Jorge Cardoso<br />38<br />
  56. 56. Face Detection<br />Q: What’sitgood for?<br />A:<br />Anatomias Urbanas (2008)<br />Sara Henriques [video]<br />Ex.001 (2010)<br />Bernardo Santos <br />[foto]<br />5/27/11<br />Jorge Cardoso<br />39<br />
  57. 57. Face Detection - Code<br />Processing (with OpenCV processing library)<br />FaceDetection<br />FaceDetectionFunnyEyes<br />Jitter (with cv.jit)<br />FaceDetectionWithASmiley.maxpat<br />5/27/11<br />Jorge Cardoso<br />40<br />
  58. 58. Computer Vision Techniques for Interactive Art<br /><ul><li>Not a recent area</li></ul>Myron Krueger experimented with some of these techniques in the 70s [video Myron Krueger]<br />5/27/11<br />Jorge Cardoso<br />41<br />
  59. 59. The End<br />5/27/11<br />Jorge Cardoso<br />42<br /> (tag: ea-phd-ia-2011)<br />