Computer Vision For Fun and Profit


Published on

Published in: Technology, Art & Photos
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Computer Vision For Fun and Profit

  1. 1. Computer Vision for Fun and Profit Image Processing: The Lowdown ! ! Steven Mitchell, Ph.D. Componica, LLC
  2. 2. Copyright 2011 - Componica, LLC ( What does Componica have to offer? Decades of Experience and Expertise A large library of code generated internally A community of innovative Computer Engineers and Programmers Access to academic papers and a history of effectively applying archived research libSeal - A long term project to take previously written code and turn it into a single library: SEAL - Steve's Evil (or Eclectic) Algorithms Library
  3. 3. Copyright 2011 - Componica, LLC ( Image Processing Any sort of signal processing done to an image The Acquisition of an image Compression and storage of an image Enhancement and restoration Registration of an image to another. Measurement of data from the image (height of building, speed of car) Interpretation and Recognition of objects.
  4. 4. Copyright 2011 - Componica, LLC ( Getting the Images in the First Place Standard image files: GIF, PNG, JPG: Standard. I tend to always use PNGs TIFF, PDF: More challenging, requires an external library Video files: FFMpeg: Without it, it would be a nightmare to read video files but, if it can't read a video file, I can't read that video file. Web Cam: OpenCV has a simple way of doing it. Outside OpenCV, there are scattered libraries Mobile Devices Scanner MRI and CT scanners: Use a file format called DICOM
  5. 5. Copyright 2011 - Componica, LLC ( A Menu of Tools: Image Enhancement - Computer...uncrop and enhance! Image Segmentation - These pixels belong to this, those pixels belong to that. Image Registration - Line this image up with that image. Object Recognition - This is a image of a frog and that's an image of a cheeseburger. Image Compression - Crush this image and make sure the process is undo- able. (Not Covered...there are plenty of free libraries to do this.)
  6. 6. Copyright 2011 - Componica, LLC ( Image Enhancement Simple Pixel Stuff: Normalizing brightness and contrast. Gamma correction (the non-linear effects of TV) Histogram equalization - maximize the global contrast. Color Correction...color temperature and tint. White balancing.
  7. 7. Copyright 2011 - Componica, LLC ( Geometric Stuff: Interpolation. Make the image bigger or smaller. This is not as easy as it sounds if you want non-aliased results. Trips up web developers whenever they try to roll their own thumbnail generator. Warping an image from one geometry to another. Simple rotation, scale, and translation. You need two or three landmark points. Perspective (aka Projection or Homogeneous) transforms. You need four landmark points. Lens distortions. Images pinch or barrel out in camera lens. You can calibrate and correct for that with enough landmark points. General warping. Typically used for image morphing in special effects. Image Enhancement - Geometry
  8. 8. Copyright 2011 - Componica, LLC ( Image Enhancement - Interpolation AMATEUR PRO UPSAMPLING DOWNSAMPLINGORIGINALS
  9. 9. Copyright 2011 - Componica, LLC ( Image Enhancement - Geometry ORIGINAL IMAGE PERSPECTIVE CORRECTED The original image is warped to a perspective corrected version. The four black dots indicated the landmark points used to normalize the image to this artificial view. The black regions are areas that falls outside the original image. Unknown data.
  10. 10. Copyright 2011 - Componica, LLC ( Noise removing. Median filter, average filter, etc. De-blurring. You estimate the cause of the blurring and then undo it using deconvolution. Motion blur - Estimate the camera movements and undo it. Focus blur - Estimate the lens blur and undo it. Image Enhancement
  11. 11. Copyright 2011 - Componica, LLC ( Divide an image into known parts: It's not quite object recognition because images are typically interpreted on a pixel basis using edge detection or colors. Sometimes it's good enough because you just care about the borders, not content Border of tumor vs. healthy tissue. The bright red ball in the color picture. Is this pixel part of a letter ‘q’ or paper? Often the first step to a bigger solution. Image Segmentation
  12. 12. Copyright 2011 - Componica, LLC ( Make a decision based on a single pixel: Simple thresholding - is this pixel darker than 160? Slightly better - is this pixel red? Even more better - is this pixel statistically more likely to be paper or letter based on it's RGB value. The works - I'm modeling the uneven-ness of the lighting on this paper and made a statistical model of RGB, is this damn pixel ink or paper? The downfall of this is you're looking at single pixels without understand how it's neighbors relate to it, missing the whole picture. Sometimes it works. Fast and easy to do. Image Segmentation - Pixel Classes ORIGINAL SIMPLE THRESHOLD ADAPTIVE BINARIZATION
  13. 13. Copyright 2011 - Componica, LLC ( Minimizing or maximizing a path which borders between elements. This one is a common technique, with many variants: Assign a cost to all the pixels. For example, edge detection - it will cost me a lot to cross an edge. Or color transition, it will cost me to cross to a different color. Use an optimization technique from classic data structures (typically used in graph theory if you still remember) to compute the cheapest path from one location of the image to a different location. Dynamic Programming - Strange name but all it means is compute the cheapest path from one side of an image to another. Works best for paths that tend to be linear. Minimum Graph Cut - Find the cheapest way to split the image in two regions. This works well for circular paths and 3D. Tends to be much more complicated than dynamic programming and slower. Check out this: Image Segmentation - Least Cost Path
  14. 14. Copyright 2011 - Componica, LLC ( Random Decision Forests applied to pixels Apply twenty questions to pixel and it's surrounding neighbors to create decision trees to guess what this pixel is over. Have a number of these decision trees (forest) and aggregate the results. Strangely it tends to works in many cases. Image Segmentation - Bleeding Edge
  15. 15. Copyright 2011 - Componica, LLC ( I have a source image that I can transform (move, resize, rotate, or bend). Make it best fit a target image. Translation Only - Shift the source image until it best fits the target. Similarity Transform - Adjust the scale, rotation, and translation until the source image overlaps the target. Perspective Transform - Move the source image’s four corners until it matches the target in a perspective manner. Non-Rigid Warping - The source image is on a rubber sheet. Warp it onto the target image. Obvious Applications: Augmented Reality Image Stitching Object Detection Image Registration
  16. 16. Copyright 2011 - Componica, LLC ( The most common way to register as image is the following: Find the most interesting points on the two images (usually blobs and corners). Scale-invariant feature transform - SIFT (Patented and slow) Speeded Up Robust Features - SURF (Recently Patented and fast) Features from Accelerated Segment Test - FAST (Very fast but noisy) With SURF and SIFT you get a position, an angle, scale factor and a 64 element vector to compare. With FAST you get a position and maybe a rotation. Compare all the interesting points from one image to the other forming matching pairs of points between images. Naive implementations are SLOW. Use RANSAC to find a consensus (next page). Image Registration - Interesting Points SURF FAST
  17. 17. Copyright 2011 - Componica, LLC ( Matching points locally doesn’t yield global matches. RANdom Sample Consensus - RANSAC - prunes away the mismatches and computes the transform that converts the source image to the target. It works as followed: Most transforms can be described by a minimal number of points. For similarity transforms it’s two points, for projection transforms it’s four points. Pick two (or four) matched pair of points at random. Compute a transform from those two (or four) sets of points. Transform all the source points using this transform and see how many points are close to the target. Repeat this and keep the best transform that matches the most points. Image Registration - Interesting Points Here the green lines indicate pairs of matched points that fit the transform (looks like a similarity transform) and the red lines are matched pairs that failed to fit this transform and therefore rejected.! ! RANSAC may seem ad hoc, but it works surprisingly well.
  18. 18. Copyright 2011 - Componica, LLC ( A very accurate way to line up a template onto an image is to use derivates and linear algebra. Works very well only when the images are very close to each other in the first place. It’s usually the polishing step after using an Interesting Points / RANSAC method. Known as Lucas-Kanade Tracker -or- Baker- Matthews Tracker. It’s the secret sauce to how the “Predator” algorithm works. Image Registration - The Mathy Way
  19. 19. Copyright 2011 - Componica, LLC ( Making computers recognize objects in images Many different algorithms, and each algorithm has appropriate uses depending on the objects being detected. For example: AdaBoost - Awesome at detecting and locating faces, sucks at recognizing who’s face it is. Deep Neural Networks - Works great for highly distorted text, but too slow for generalized OCR. Active Appearance Models - Works great for both recognition and segmentation, but only on generally fixed shapes like faces and hearts. Sucks for livers and cancer. Most recognizers depend on training the algorithm on known objects offline and then testing. Which brings up the topic of data... Object Recognition
  20. 20. Copyright 2011 - Componica, LLC ( The data can be the most valuable part of a trainable system as many algorithms will generally function with somewhat similar hit rates. Often ideas fail to take into account where the data comes from. It’s the killer of many ideas. The basics of training something: Data is normally split into a training and testing set. You train a thingy with the training set and test how well it works with the testing set. Why? Most trainable thingies are prone to overfitting. Splitting the data into two sets prevents this problem because you use the test set to know when to stop training. Disadvantage, you effectively need twice as much data. Sucks. Object Recognition - It’s all about the Data It’s obvious this data is best represented as a line, but! if the model over-fits the data, it may compute a! relationship that’s nonsensical. As your algorithm learns from a training set (blue line),! the error decreases for that set, but in the testing set,! it will hit a point where overfitting is happening and! will increase the overall error in the real world. You stop! training at the point the testing set starts getting worse.
  21. 21. Copyright 2011 - Componica, LLC ( Steve’s crappy breakdown of object recognition algorithms: A ship of fools approach - Armies of stupid algorithms that together become smart. Kinda like democracy sort of...not really. Let’s create a brain, Igor - So what happens if you simulate brain tissue? It’s grown quite a bit since the neural network hype in the late 80s / early 90s. Fun Fact: This will eventually kill us all in a bloody uprising. If the shoe fits... - Well if this template StarBucks logo fits somewhere on my image using object registration as described above, then I’m guessing this StarBucks logo is present in the image. Duh. Find Features to Filter and Fit in a Feed-forward Fashion - You see this pattern all the time and it lacks of creativity. Find interesting features in the image, and feed them into a trainable function like a neural network, a non-linear regression, or a support vector machine. Boring. I’ve stared at a wall for 20 minutes now and I think everything out there is either one or a combination of these general ideas. Hmm...That’s it? Disappointed. Object Recognition - The Menu
  22. 22. End of Part One...