With cheap cameras becoming ubiquitous the camera has become probably the most important sensor for many applications. However extracting usable information from the images produced by cameras is non-trivial. There have been many published successes in recent years using deep learning (multi-layered convolutional neural networks) but it’s not always necessary to apply such techniques to get useful results for many applications. This talk will focus on “classical” machine vision using java and the OpenCV library. We’ll start with a quick refresher on how image data is represented and then cover topics such as determining if an image is blurred (and therefore unusable) and then explore a number of techniques such as shape and face detection.

1. Information from
Pixels
Dave Snowdon
@davesnowdon
https://github.com/davesnowdon/ljc-information-from-pixels
http://www.slideshare.net/DaveSnowdon1/information-from-pixels

2. Summary
• Why? What?
• Range operations and colour spaces
• Kernels & convolution
• Object detection
• Contours
• Conclusion

3. Why me?
• Social robotics developer
• Social robots need to handle unstructured
environments
• Vision is the most versatile way of sensing the
environment

4. Most general purpose
sensor

5. Machine vision
• Tracking movement: Dyson 360, Google Tango
• Recognising people, biometric security
• Recognising medication
• Image search
• …

6. Why this is hard
https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/

7. Why this is hard
• Colour reproduction, lighting & white balance
• Perspective & rotation effects
• Noise
• Different scales

8. Rotations & perspective

9. Open CV

10. The good news
• Open source
• Tried and tested
• Large collections of algorithms
• Language bindings for C, python & java
• Runs on pretty much anything (Linux, Mac,
Windows, android, iOS, RaspberryPi)

11. The less good news
• Native code
• Java API is a bit clunky
• Not much structure
• Not the new shiny

12. Range operations &
colour spaces

13. RGB
https://en.wikipedia.org/wiki/RGB_color_model#/media/File:RGB_color_solid_cube.png

14. HSV
https://upload.wikimedia.org/wikipedia/commons/a/a0/Hsl-hsv_models.svg

15. L*a*b* / CIELAB
https://gurus.pyimagesearch.com/wp-content/uploads/2015/03/color_spaces_lab_axis.jpg

16. Blob detection

17. Get an image
• From a Java image
• From video / webcam
org.opencv.videoio.VideoCapture
• From file
import org.opencv.core.Mat;
Mat image = Imgcodecs.imread(filename);

18. org.opencv.core.Mat
new Mat(numRows, numColumns, CvType.CV_8UC3);
• Dense multi-dimensional matrix
• Variants with int, double, byte values
• Implements basic matrix operations
B G R B G R B G R B G R
B G R B G R B G R B G R
B G R B G R B G R B G R

19. Blur the image
Imgproc.GaussianBlur(image, result,
new Size(kernelSize, kernelSize),
0.0);

20. Convert to HSV
Imgproc.cvtColor(input, hsv,
Imgproc.COLOR_BGR2HSV);

21. Select only pixels in range
Core.inRange(image, low, high, result);

22. Erode & Dilate
final Mat se = Imgproc.getStructuringElement(Imgproc.MORPH
new Size(kernelSize, kernelSize));
Imgproc.erode(image, result, se, new Point(-1, -1), numIteration
Imgproc.dilate(image, result, se, new Point(-1, -1), numIteration

23. Find contours
Imgproc.findContours(image, contours, new Mat(),
Imgproc.RETR_EXTERNAL,
Imgproc.CHAIN_APPROX_SIMPLE);

24. Find largest contour
contours.stream()
.max((c1, c2) ->
(Imgproc.contourArea(c1) > Imgproc.contourArea(c2) ? 1
: -1))
.get();

25. Draw contour (for demo)
Imgproc.circle(image, centre, 5, CENTRE_COLOUR, 2);
Imgproc.drawContours(image,
Arrays.asList(contour), 0, OUTLINE_COLOUR, 2);

26. Output image
• Don’t always need to
• Grab region of interest
Mat roi = mat.submat(Rect)
• Convert to java image
BufferedImage javaImage = Util.matrixToImage(mat);
Util.displayImage(command, javaImage);
• Write to file
Imgcodecs.imwrite(filename, mat);

27. Built-in blob detection
• OpenCV has built-in blob detection:
SimpleBlobDetector
• blob detection by colour may not work
• Blog post: https://www.learnopencv.com/blob-
detection-using-opencv-python-c/

28. Kernels & Convolution

29. Convolution
//developer.apple.com/library/mac/documentation/Performance/Conceptual/vImage/ConvolutionOperations/ConvolutionOperations

30. Example kernels
Gaussian

31. Example kernels
Laplacian

32. Detecting blurred
images

33. Detecting blurred images
• Want to discard images that are unlikely to be of
use
• The more blurred an image is the fewer sharp
edges will be found
• What happens to the laplacian of an image as it’s
blurred…

34. Input image

35. Grayscale + laplacian

36. 3x3 gaussian kernel

37. 5x5 gaussian kernel

38. 7x7 gaussian kernel

39. 13x13 kernel

40. 19x19 kernel

41. Variance of the laplacian

42. Code
// apply laplacian to grayscale copy of image
Imgproc.Laplacian(gray, laplacian, CvType.CV_64F);
// determine variance
MatOfDouble mean = new MatOfDouble();
MatOfDouble stddev = new MatOfDouble();
Core.meanStdDev(laplacian, mean, stddev);
double sd = stddev.toList().get(0);
double var = sd * sd;

43. Line following

44. Detect the line

45. What we want to do

46. Kernel to detect vertical lines
-1 2 -1
Mat kernel = new Mat(1, 3, CvType.CV_64F);
double[] kernel_values = {-1.0, 2.0, -1.0};
kernel.put(0, 0, kernel_values);

47. Convolve image with kernel
Imgproc.filter2D(gray, convolved, -1, kernel);

48. Threshold
Imgproc.threshold(convolved, thresh, 45.0, 255, Imgproc.

49. Result

50. Object detection

51. Sliding window
http://www.pyimagesearch.com/2015/03/23/sliding-windows-for-object-detection-with-python-and-opencv/

52. Haar features

53. Boosting
• Train all features on every training example
• For each feature find the best threshold which
distinguished positive from negative
• Select features with minimum error rate
• Final classifier is weighted sum of these weak
classifiers

54. Cascade
• Hugely expensive to compute all features on every
window location
• Group features into different stages with smaller
number of features
• Only proceed to next stage when previous stage
passes
• In Viola-Jones paper as few as 10 features out of
6000 might be evaluated per window

55. Pre-trained classifiers
• front face
• profile face
• Full body
• Upper body
• Lower body
• Left & right eyes (one classifier each for left & right)
• Smile
• Front cat face
• Russian license plate

56. Using a classifier
// create classifier object from XML definition
final CascadeClassifier faceClassifier =
new CascadeClassifier(classifierFilename);
// apply classifer to get list of matching regions
final MatOfRect mor = new MatOfRect();
clr.detectMultiScale(image, mor);
List<Rect> result = mor.toList();

57. Front face detection

58. Training your own classifier

59. How to train
• Create sample vectors from text files listing +ve & -ve images
• opencv_createsamples -info positives.txt -num 68 -w 60 -h 98 -vec
nao.vec
• Train
• Haar: opencv_traincascade -data classifier -vec samples.vec -bg
negatives.txt -numStages 20 -minHitRate 0.999 -maxFalseAlarmRate 0.5
-numPos 1000 -numNeg 600 -w 60 -h 98 -mode ALL -precalcValBufSize
1024 -precalcIdxBufSize 1024
• LBP : opencv_traincascade -data classifier.lbp -vec samples.vec -bg
negatives.txt -numStages 20 -minHitRate 0.999 -maxFalseAlarmRate 0.5
-numPos 1000 -numNeg 600 -w 60 -h 98 -featureType LBP -
precalcValBufSize 1024 -precalcIdxBufSize 1024

60. Training docs & tutorials
• http://docs.opencv.org/trunk/dc/d88/tutorial_train
cascade.html
• http://coding-robin.de/2013/07/22/train-your-own-
opencv-haar-classifier.html

61. Results

62. More uses for
contours

63. Detecting geometric shapes

64. Find contours
// use Canny edge detector on blurred grayscale image
Imgproc.Canny(blurred, edges, 75, 200);
// find contours
Imgproc.findContours(image, contours, new Mat(),
Imgproc.RETR_EXTERNAL,
Imgproc.CHAIN_APPROX_SIMPLE);

65. Type conversion
// need to convert the contour from a MatOfPoint to
MatOfPoint2f
final MatOfPoint2f m2f = new MatOfPoint2f();
m2f.fromList(contour.toList());

66. Approximate shapes
// approximate contour polygon with 1% or less difference in
perimeter
double perimeter = Imgproc.arcLength(m2f, true);
MatOfPoint2f approx = new MatOfPoint2f();
Imgproc.approxPolyDP(m2f, approx, 0.01 * perimeter,
true);
// check number of line segments
int numSides = approx.toList().size();

67. More information
• OpenCV docs: http://docs.opencv.org/3.1.0/
• Useful blogs:
• http://www.pyimagesearch.com
• https://www.learnopencv.com
• https://opencv-java-tutorials.readthedocs.io/en/latest/
• Code for examples:
https://github.com/davesnowdon/ljc-information-from-pixels

68. Summary
• Colour spaces: RGB, HSV, L*a*b*
• masking images using colour ranges
• Finding outline of objects using contours
• Convolution
• Using cascade classifiers to detect objects