S T U D I O S T U D I OSeeing Like SoftwareAndrew Lovett-Barron@readywater
The machines can see
Computer Vision
CV in the day-to-day
http://techcrunch.com/2010/12/07/videos-the-best-kinect-hacks-and-mods-one-month-in/2011: The Kinect Is Born
Rafael Lozano HemmerComputer Vision and Art
http://www.flickr.com/photos/55705924@N08/5178241193/David Rokeby
http://www.medienkunstnetz.de/works/videoplace/Myron Krueger
http://www.d-p.cc/wp-content/uploads/2013/04/messa_jaapsolo_ars_287015.jpgGolan Levin
Rafael Lozano-HemmerRafael Lozano-Hemmer
http://www.creativeapplications.net/environment/exr3-elli-ot-woods-kyle-mcdonald/Kyle McDonald
http://www.coolhunting.com/culture/funky-forest.phpTheo Watson
http://autoponics.org/?p=147Getting past the Kinect Hack
Cheap distance and shape
Victor Castaneda, Nassir Navab Kinect Programming forComputer Vision Summer Term 2011Kinect and Structured Light
Victor Castaneda, Nassir Navab Kinect Programming forComputer Vision Summer Term 2011Other types of Depth Camera
Sensors live through Software
OpenCV toolkits
Overview of OpenCV toolkitOpenCV 2.3 Cheat Sheet (C++)The OpenCV C++ reference manual is here:http: // opencv. willowgarag...
How to navigate openCV
http://whiteglovetracking.com/ by Evan RothFocus and Regions of Interest
Image credit: KinemeProcessing to Analysis
Haar based detection by Adam HarveyMethods of Analysis
http://golancourses.net/2013/wp-content/up-loads/2013/01/openFrameworks.jpegOpenframeworks
Alternatives to C++ for openCV
Starting off with OF
ofxAddons and community
Kyle McDonald showing off ofxCvOfxCv
HOWTO: Interactive Lighting
Design process
Structuring the program
Ikea, Arduino, etc.
Document and sharehttp://github.com/readywater/seeing-like-softwarehttp://www.andrewlb.com/						2013/06/sls-notes/
Learn to see the invisible
S T U D I O S T U D I OAndrew Lovett-Barron@readywaterhttp://andrewlb.comandrew@relaystudio.com
Upcoming SlideShare
Loading in …5
×

Seeing Like Software

575 views

Published on

A talk from Toronto's FITC Spotlight on Hardware talk. I spoke about using tools like Openframeworks, OpenCV, and the Kinect to create Interactive Installations, and paired it with an interactive lighting installation.

References, citations, and source code can be found here: http://www.andrewlb.com/2013/06/sls-notes/

Published in: Technology
1 Comment
0 Likes
Statistics
Notes
  • Be the first to like this

No Downloads
Views
Total views
575
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
2
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide

Seeing Like Software

  1. 1. S T U D I O S T U D I OSeeing Like SoftwareAndrew Lovett-Barron@readywater
  2. 2. The machines can see
  3. 3. Computer Vision
  4. 4. CV in the day-to-day
  5. 5. http://techcrunch.com/2010/12/07/videos-the-best-kinect-hacks-and-mods-one-month-in/2011: The Kinect Is Born
  6. 6. Rafael Lozano HemmerComputer Vision and Art
  7. 7. http://www.flickr.com/photos/55705924@N08/5178241193/David Rokeby
  8. 8. http://www.medienkunstnetz.de/works/videoplace/Myron Krueger
  9. 9. http://www.d-p.cc/wp-content/uploads/2013/04/messa_jaapsolo_ars_287015.jpgGolan Levin
  10. 10. Rafael Lozano-HemmerRafael Lozano-Hemmer
  11. 11. http://www.creativeapplications.net/environment/exr3-elli-ot-woods-kyle-mcdonald/Kyle McDonald
  12. 12. http://www.coolhunting.com/culture/funky-forest.phpTheo Watson
  13. 13. http://autoponics.org/?p=147Getting past the Kinect Hack
  14. 14. Cheap distance and shape
  15. 15. Victor Castaneda, Nassir Navab Kinect Programming forComputer Vision Summer Term 2011Kinect and Structured Light
  16. 16. Victor Castaneda, Nassir Navab Kinect Programming forComputer Vision Summer Term 2011Other types of Depth Camera
  17. 17. Sensors live through Software
  18. 18. OpenCV toolkits
  19. 19. Overview of OpenCV toolkitOpenCV 2.3 Cheat Sheet (C++)The OpenCV C++ reference manual is here:http: // opencv. willowgarage. com/ documentation/ cpp/ .Use Quick Search to find descriptions of the particularfunctions and classesKey OpenCV ClassesPoint Template 2D point classPoint3 Template 3D point classSize Template size (width, height) classVec Template short vector classMatx Template small matrix classScalar 4-element vectorRect RectangleRange Integer value rangeMat 2D or multi-dimensional dense array(can be used to store matrices, images,histograms, feature descriptors, voxelvolumes etc.)SparseMat Multi-dimensional sparse arrayPtr Template smart pointer classMatrix BasicsCreate a matrixMat image(240, 320, CV 8UC3);[Re]allocate a pre-declared matriximage.create(480, 640, CV 8UC3);Create a matrix initialized with a constantMat A33(3, 3, CV 32F, Scalar(5));Mat B33(3, 3, CV 32F); B33 = Scalar(5);Mat C33 = Mat::ones(3, 3, CV 32F)*5.;Mat D33 = Mat::zeros(3, 3, CV 32F) + 5.;Create a matrix initialized with specified valuesdouble a = CV PI/3;Mat A22 = (Mat <float>(2, 2) <<cos(a), -sin(a), sin(a), cos(a));float B22data[] = {cos(a), -sin(a), sin(a), cos(a)};Mat B22 = Mat(2, 2, CV 32F, B22data).clone();Initialize a random matrixrandu(image, Scalar(0), Scalar(256)); // uniform distrandn(image, Scalar(128), Scalar(10)); // Gaussian distConvert matrix to/from other structures(without copying the data)Mat image alias = image;float* Idata=new float[480*640*3];Mat I(480, 640, CV 32FC3, Idata);vector<Point> iptvec(10);Mat iP(iptvec); // iP – 10x1 CV 32SC2 matrixIplImage* oldC0 = cvCreateImage(cvSize(320,240),16,1);Mat newC = cvarrToMat(oldC0);IplImage oldC1 = newC; CvMat oldC2 = newC;... (with copying the data)Mat newC2 = cvarrToMat(oldC0).clone();vector<Point2f> ptvec = Mat <Point2f>(iP);Access matrix elementsA33.at<float>(i,j) = A33.at<float>(j,i)+1;Mat dyImage(image.size(), image.type());for(int y = 1; y < image.rows-1; y++) {Vec3b* prevRow = image.ptr<Vec3b>(y-1);Vec3b* nextRow = image.ptr<Vec3b>(y+1);for(int x = 0; y < image.cols; x++)for(int c = 0; c < 3; c++)dyImage.at<Vec3b>(y,x)[c] =saturate cast<uchar>(nextRow[x][c] - prevRow[x][c]);}Mat <Vec3b>::iterator it = image.begin<Vec3b>(),itEnd = image.end<Vec3b>();for(; it != itEnd; ++it)(*it)[1] ^= 255;Matrix Manipulations: Copying,Shuffling, Part Accesssrc.copyTo(dst) Copy matrix to another onesrc.convertTo(dst,type,scale,shift) Scale and convert toanother datatypem.clone() Make deep copy of a matrixm.reshape(nch,nrows) Change matrix dimensions and/or num-ber of channels without copying datam.row(i), m.col(i) Take a matrix row/columnm.rowRange(Range(i1,i2))m.colRange(Range(j1,j2))Take a matrix row/column spanm.diag(i) Take a matrix diagonalm(Range(i1,i2),Range(j1,j2)),m(roi)Take a submatrixm.repeat(ny,nx) Make a bigger matrix from a smaller oneflip(src,dst,dir) Reverse the order of matrix rows and/orcolumnssplit(...) Split multi-channel matrix into separatechannelsmerge(...) Make a multi-channel matrix out of theseparate channelsmixChannels(...) Generalized form of split() and merge()randShuffle(...) Randomly shuffle matrix elementsExample 1. Smooth image ROI in-placeMat imgroi = image(Rect(10, 20, 100, 100));GaussianBlur(imgroi, imgroi, Size(5, 5), 1.2, 1.2);Example 2. Somewhere in a linear algebra algorithmm.row(i) += m.row(j)*alpha;Example 3. Copy image ROI to another image with conversionRect r(1, 1, 10, 20);Mat dstroi = dst(Rect(0,10,r.width,r.height));src(r).convertTo(dstroi, dstroi.type(), 1, 0);Simple Matrix OperationsOpenCV implements most common arithmetical, logical andother matrix operations, such as• add(), subtract(), multiply(), divide(), absdiff(),bitwise and(), bitwise or(), bitwise xor(), max(),min(), compare()– correspondingly, addition, subtraction, element-wisemultiplication ... comparison of two matrices or amatrix and a scalar.Example. Alpha compositing function:void alphaCompose(const Mat& rgba1,const Mat& rgba2, Mat& rgba dest){Mat a1(rgba1.size(), rgba1.type()), ra1;Mat a2(rgba2.size(), rgba2.type());int mixch[]={3, 0, 3, 1, 3, 2, 3, 3};mixChannels(&rgba1, 1, &a1, 1, mixch, 4);mixChannels(&rgba2, 1, &a2, 1, mixch, 4);subtract(Scalar::all(255), a1, ra1);bitwise or(a1, Scalar(0,0,0,255), a1);bitwise or(a2, Scalar(0,0,0,255), a2);multiply(a2, ra1, a2, 1./255);multiply(a1, rgba1, a1, 1./255);multiply(a2, rgba2, a2, 1./255);add(a1, a2, rgba dest);}• sum(), mean(), meanStdDev(), norm(), countNonZero(),minMaxLoc(),– various statistics of matrix elements.• exp(), log(), pow(), sqrt(), cartToPolar(),polarToCart()– the classical math functions.• scaleAdd(), transpose(), gemm(), invert(), solve(),determinant(), trace() eigen(), SVD,– the algebraic functions + SVD class.• dft(), idft(), dct(), idct(),– discrete Fourier and cosine transformationsFor some operations a more convenient algebraic notation canbe used, for example:Mat delta = (J.t()*J + lambda*Mat::eye(J.cols, J.cols, J.type())).inv(CV SVD)*(J.t()*err);implements the core of Levenberg-Marquardt optimizationalgorithm.Image ProcesssingFilteringfilter2D() Non-separable linear filtersepFilter2D() Separable linear filterboxFilter(),GaussianBlur(),medianBlur(),bilateralFilter()Smooth the image with one of the linearor non-linear filtersSobel(), Scharr() Compute the spatial image derivativesLaplacian() compute Laplacian: ∆I = ∂2I∂x2 + ∂2I∂y2erode(), dilate() Morphological operations1
  20. 20. How to navigate openCV
  21. 21. http://whiteglovetracking.com/ by Evan RothFocus and Regions of Interest
  22. 22. Image credit: KinemeProcessing to Analysis
  23. 23. Haar based detection by Adam HarveyMethods of Analysis
  24. 24. http://golancourses.net/2013/wp-content/up-loads/2013/01/openFrameworks.jpegOpenframeworks
  25. 25. Alternatives to C++ for openCV
  26. 26. Starting off with OF
  27. 27. ofxAddons and community
  28. 28. Kyle McDonald showing off ofxCvOfxCv
  29. 29. HOWTO: Interactive Lighting
  30. 30. Design process
  31. 31. Structuring the program
  32. 32. Ikea, Arduino, etc.
  33. 33. Document and sharehttp://github.com/readywater/seeing-like-softwarehttp://www.andrewlb.com/ 2013/06/sls-notes/
  34. 34. Learn to see the invisible
  35. 35. S T U D I O S T U D I OAndrew Lovett-Barron@readywaterhttp://andrewlb.comandrew@relaystudio.com

×