Apps that Can See: Getting Data from CoreImage to Computer Vision You hear about the app that can solve a Rubik's Cube after you take pictures of each side? How about the one that can import Sudoku puzzles by letting you take a picture of a puzzle? Wouldn't it be cool if your apps could get information that way? Come and learn how to integrate image processing in your apps. We'll start with Apple's built-in CoreImage libraries, and then move on to open-source C++ libraries that let you detect colors, shapes, letters and numbers. We'll go through quite a bit of sample code that you can take with you to use in your apps. We'll also talk about practical tips and experiences, like how to debug and troubleshoot code when things aren't working the way you expected.

1. #360iDev@CarlBrwn
Apps That
Can See
Carl Brown
Missing Last Vowel -
I blame Flickr/Tumblr
1Monday, September 9, 13

2. @CarlBrwn #360iDev
Computer Vision?
Turning Photos Into Useable
Data
Image Analysis
Colors
Shapes
Numbers & Letters
2Monday, September 9, 13

3. @CarlBrwn #360iDev
Word Lens
http://www.youtube.com/watch?v=h2OfQdYrHRs
3Monday, September 9, 13

4. @CarlBrwn #360iDev
Word Lens
http://www.youtube.com/watch?v=h2OfQdYrHRs
3Monday, September 9, 13

5. @CarlBrwn #360iDev
Today's Talk
How quick and easy it is to get
actionable info from photos
4Monday, September 9, 13

6. @CarlBrwn #360iDev
Today's Talk
How quick and easy it is to get
actionable info from photos
Demo Code with Source
4Monday, September 9, 13

7. @CarlBrwn #360iDev
Today's Talk
How quick and easy it is to get
actionable info from photos
Demo Code with Source
Not about HOW OpenCV works
4Monday, September 9, 13

8. @CarlBrwn #360iDev
Today's Talk
How quick and easy it is to get
actionable info from photos
Demo Code with Source
Not about HOW OpenCV works
"OpenCV by Example"
4Monday, September 9, 13

9. @CarlBrwn #360iDev
CubeCheater &
Sudoku Grab
http://www.youtube.com/watch?v=HNwx0nbgm7M http://www.youtube.com/watch?v=wt96OomJY9A
5Monday, September 9, 13

10. @CarlBrwn #360iDev
Both have had
Legal Problems
http://cubecheater.efaller.com http://sudokugrab.blogspot.com/2010/12/
sudoku-grab-removed-from-sale-in-japan.html
6Monday, September 9, 13

11. #360iDev@CarlBrwn
Based on
Code
From:
github.com/carlbrown/
OpenCVDemo360iDev2013
7Monday, September 9, 13

12. @CarlBrwn #360iDev
About the Code
Explicit and Redundant to make
it easier to follow/explain
8Monday, September 9, 13

13. @CarlBrwn #360iDev
About the Code
Explicit and Redundant to make
it easier to follow/explain
Not production ready -
for teaching purposes
8Monday, September 9, 13

14. @CarlBrwn #360iDev
About the Code
Explicit and Redundant to make
it easier to follow/explain
Not production ready -
for teaching purposes
Minimum Viable Interface
8Monday, September 9, 13

15. @CarlBrwn #360iDev
About the Code
Explicit and Redundant to make
it easier to follow/explain
Not production ready -
for teaching purposes
Minimum Viable Interface
Don't hate, okay?
8Monday, September 9, 13

16. #360iDev@CarlBrwn
Face
Counter
Demo 1/3
9Monday, September 9, 13

17. #360iDev@CarlBrwn
Core
Image
Option 1
10Monday, September 9, 13

18. @CarlBrwn #360iDev
ViewController.mm
- coreImageButtonPressed:
[CIImage imageWithCGImage:]
Detect
11Monday, September 9, 13

19. @CarlBrwn #360iDev
ViewController.mm
- coreImageButtonPressed:
[CIImage imageWithCGImage:]
[CIDetector detectorOfType:
! ! ! ! CIDetectorTypeFace]
Detect
11Monday, September 9, 13

20. @CarlBrwn #360iDev
ViewController.mm
- coreImageButtonPressed:
[CIImage imageWithCGImage:]
[CIDetector detectorOfType:
! ! ! ! CIDetectorTypeFace]
NSArray *features= [detector
! ! ! ! ! ! featuresInImage:]
Detect
11Monday, September 9, 13

21. @CarlBrwn #360iDev
ViewController.mm
- coreImageButtonPressed:
[CIImage imageWithCGImage:]
[CIDetector detectorOfType:
! ! ! ! CIDetectorTypeFace]
NSArray *features= [detector
! ! ! ! ! ! featuresInImage:]
for(CIFaceFeature* faceFeature
!! ! ! ! ! ! ! in features)
Detect
11Monday, September 9, 13

22. @CarlBrwn #360iDev
ViewController.mm
- coreImageButtonPressed:
for(CIFaceFeature* faceFeature
!! ! ! ! ! ! ! in features)
Debugging
12Monday, September 9, 13

23. @CarlBrwn #360iDev
ViewController.mm
- coreImageButtonPressed:
for(CIFaceFeature* faceFeature
!! ! ! ! ! ! ! in features)
frame = faceFeature.bounds
Debugging
12Monday, September 9, 13

24. @CarlBrwn #360iDev
ViewController.mm
- coreImageButtonPressed:
for(CIFaceFeature* faceFeature
!! ! ! ! ! ! ! in features)
frame = faceFeature.bounds
frame *= contentScaleFactor
Debugging
12Monday, September 9, 13

25. @CarlBrwn #360iDev
ViewController.mm
- coreImageButtonPressed:
for(CIFaceFeature* faceFeature
!! ! ! ! ! ! ! in features)
frame = faceFeature.bounds
frame *= contentScaleFactor
//flip to CIImage Coords
Debugging
12Monday, September 9, 13

26. @CarlBrwn #360iDev
Coordinate
Systems
UIView has (0,0) at Upper Left
Core Image has (0,0) Lower Left
(Like OS X)
13Monday, September 9, 13

27. @CarlBrwn #360iDev
ViewController.mm
- coreImageButtonPressed:
for(CIFaceFeature* faceFeature
!! ! ! ! ! ! ! in features)
frame = faceFeature.bounds
frame *= contentScaleFactor
//flip to CIImage Coords
Debugging
14Monday, September 9, 13

28. @CarlBrwn #360iDev
ViewController.mm
- coreImageButtonPressed:
for(CIFaceFeature* faceFeature
!! ! ! ! ! ! ! in features)
frame = faceFeature.bounds
frame *= contentScaleFactor
//flip to CIImage Coords
faceView.layer.borderWidth = 4;
Debugging
14Monday, September 9, 13

29. @CarlBrwn #360iDev
ViewController.mm
- coreImageButtonPressed:
for(CIFaceFeature* faceFeature
!! ! ! ! ! ! ! in features)
frame = faceFeature.bounds
frame *= contentScaleFactor
//flip to CIImage Coords
faceView.layer.borderWidth = 4;
addSubview:faceView
Debugging
14Monday, September 9, 13

30. #360iDev@CarlBrwn
Core
Image
Option 1
15Monday, September 9, 13

31. #360iDev@CarlBrwn
OpenCV
Option 2
16Monday, September 9, 13

32. @CarlBrwn #360iDev
ViewController.mm
- openCVButtonPressed:
convert to cv::Mat
Detect
17Monday, September 9, 13

33. @CarlBrwn #360iDev
Convert to cv::Mat
CGImageGetColorSpace(image.CGImage)
cv::Mat cvMat()
CGBitmapContextCreate(cvMat.data,...)
CGContextDrawImage(ctx,r,image.CGImage)
CGContextRelease()
18Monday, September 9, 13

34. @CarlBrwn #360iDev
ViewController.mm
- openCVButtonPressed:
convert to cv::Mat
Detect
19Monday, September 9, 13

35. @CarlBrwn #360iDev
ViewController.mm
- openCVButtonPressed:
convert to cv::Mat
cv::cvtColor( CV_BGR2GRAY )
Detect
19Monday, September 9, 13

36. @CarlBrwn #360iDev
ViewController.mm
- openCVButtonPressed:
convert to cv::Mat
cv::cvtColor( CV_BGR2GRAY )
cv::equalizeHist()
Detect
19Monday, September 9, 13

37. @CarlBrwn #360iDev
ViewController.mm
- openCVButtonPressed:
convert to cv::Mat
cv::cvtColor( CV_BGR2GRAY )
cv::equalizeHist()
cv::face_cascade.
! ! ! ! ! ! detectMultiScale()
Detect
19Monday, September 9, 13

38. @CarlBrwn #360iDev
Cascade Classifier
http://docs.opencv.org/doc/tutorials/objdetect/cascade_classifier/cascade_classifier.html
20Monday, September 9, 13

39. @CarlBrwn #360iDev
ViewController.mm
- openCVButtonPressed:
convert to cv::Mat
cv::cvtColor( CV_BGR2GRAY )
cv::equalizeHist()
face_cascade.detectMultiScale()
Detect
21Monday, September 9, 13

40. @CarlBrwn #360iDev
ViewController.mm
- openCVButtonPressed:
convert to cv::Mat
cv::cvtColor( CV_BGR2GRAY )
cv::equalizeHist()
face_cascade.detectMultiScale()
for( int i = 0;
! ! ! ! i < faces.size(); i++ )
Detect
21Monday, September 9, 13

41. @CarlBrwn #360iDev
ViewController.mm
- openCVButtonPressed:
for( int i = 0;
! ! ! ! i < faces.size(); i++ )
Debugging
22Monday, September 9, 13

42. @CarlBrwn #360iDev
ViewController.mm
- openCVButtonPressed:
for( int i = 0;
! ! ! ! i < faces.size(); i++ )
CGRectMake(faces[i].frame)
Debugging
22Monday, September 9, 13

43. @CarlBrwn #360iDev
ViewController.mm
- openCVButtonPressed:
for( int i = 0;
! ! ! ! i < faces.size(); i++ )
CGRectMake(faces[i].frame)
frame *= contentScaleFactor
Debugging
22Monday, September 9, 13

44. @CarlBrwn #360iDev
ViewController.mm
- openCVButtonPressed:
for( int i = 0;
! ! ! ! i < faces.size(); i++ )
CGRectMake(faces[i].frame)
frame *= contentScaleFactor
faceView.layer.borderWidth = 4;
Debugging
22Monday, September 9, 13

45. @CarlBrwn #360iDev
ViewController.mm
- openCVButtonPressed:
for( int i = 0;
! ! ! ! i < faces.size(); i++ )
CGRectMake(faces[i].frame)
frame *= contentScaleFactor
faceView.layer.borderWidth = 4;
addSubview:faceView
Debugging
22Monday, September 9, 13

46. #360iDev@CarlBrwn
OpenCV
Option 2
23Monday, September 9, 13

47. #360iDev@CarlBrwn
Cube
Face
Grabber
Demo 2/3
24Monday, September 9, 13

48. #360iDev@CarlBrwn
Detect
Edges
detectFacesPressed:
25Monday, September 9, 13

49. @CarlBrwn #360iDev
ViewController.mm
- detectFacesPressed:
convert to cv::Mat
Detect
26Monday, September 9, 13

50. @CarlBrwn #360iDev
ViewController.mm
- detectFacesPressed:
convert to cv::Mat
cv::Canny()
Detect
26Monday, September 9, 13

51. @CarlBrwn #360iDev
Canny
27Monday, September 9, 13

52. @CarlBrwn #360iDev
ViewController.mm
- detectFacesPressed:
convert to cv::Mat
cv::Canny()
Detect
28Monday, September 9, 13

53. @CarlBrwn #360iDev
ViewController.mm
- detectFacesPressed:
convert to cv::Mat
cv::Canny()
cv::HoughLinesP()
Detect
28Monday, September 9, 13

54. @CarlBrwn #360iDev
HoughLinesP
29Monday, September 9, 13

55. @CarlBrwn #360iDev
ViewController.mm
- detectFacesPressed:
convert to cv::Mat
cv::Canny()
cv::HoughLinesP()
Detect
30Monday, September 9, 13

56. @CarlBrwn #360iDev
ViewController.mm
- detectFacesPressed:
convert to cv::Mat
cv::Canny()
cv::HoughLinesP()
for( size_t i = 0;
! ! ! i < lines.size(); i++ )
Detect
30Monday, September 9, 13

57. @CarlBrwn #360iDev
ViewController.mm
- detectFacesPressed:
convert to cv::Mat
cv::Canny()
cv::HoughLinesP()
for( size_t i = 0;
! ! ! i < lines.size(); i++ )
if (LineIntersect(l, l2))
Detect
30Monday, September 9, 13

58. @CarlBrwn #360iDev
ViewController.mm
- detectFacesPressed:
convert to cv::Mat
cv::Canny()
cv::HoughLinesP()
for( size_t i = 0;
! ! ! i < lines.size(); i++ )
if (LineIntersect(l, l2))
//Measure Distance to Image corners
Detect
30Monday, September 9, 13

59. @CarlBrwn #360iDev
ViewController.mm
- detectFacesPressed:
for( size_t i = 0;
! ! ! i < lines.size(); i++ )
Debugging
31Monday, September 9, 13

60. @CarlBrwn #360iDev
ViewController.mm
- detectFacesPressed:
for( size_t i = 0;
! ! ! i < lines.size(); i++ )
cv::line() //Draw
Debugging
31Monday, September 9, 13

61. @CarlBrwn #360iDev
ViewController.mm
- detectFacesPressed:
for( size_t i = 0;
! ! ! i < lines.size(); i++ )
cv::line() //Draw
//Check for bounding corners
Debugging
31Monday, September 9, 13

62. @CarlBrwn #360iDev
ViewController.mm
- detectFacesPressed:
for( size_t i = 0;
! ! ! i < lines.size(); i++ )
cv::line() //Draw
//Check for bounding corners
cv::circle() //Draw
Debugging
31Monday, September 9, 13

63. @CarlBrwn #360iDev
ViewController.mm
- detectFacesPressed:
for( size_t i = 0;
! ! ! i < lines.size(); i++ )
cv::line() //Draw
//Check for bounding corners
cv::circle() //Draw
convert to UIImage //Display
Debugging
31Monday, September 9, 13

64. @CarlBrwn #360iDev
Convert to
UIImage
[NSData dataWithBytes:cvMat.data length:]
CGDataProviderCreateWithCFData()
CGImageCreate()
[[UIImage alloc] initWithCGImage:]
CGXXXRelease()
32Monday, September 9, 13

65. @CarlBrwn #360iDev
ViewController.mm
- detectFacesPressed:
for( size_t i = 0;
! ! ! i < lines.size(); i++ )
cv::line() //Draw
//Check for bounding corners
cv::circle() //Draw
convert to UIImage //Display
Debugging
33Monday, September 9, 13

66. #360iDev@CarlBrwn
Detect
Edges
detectFacesPressed:
34Monday, September 9, 13

67. #360iDev@CarlBrwn
Unskew
getAndApplyTransformPressed:
35Monday, September 9, 13

68. @CarlBrwn #360iDev
ViewController.mm
- getAndApplyTransformPressed:
convert to cv::Mat
36Monday, September 9, 13

69. @CarlBrwn #360iDev
ViewController.mm
- getAndApplyTransformPressed:
convert to cv::Mat
getPerspectiveTransform()
36Monday, September 9, 13

70. @CarlBrwn #360iDev
ViewController.mm
- getAndApplyTransformPressed:
convert to cv::Mat
getPerspectiveTransform()
warpPerspective(transform)
36Monday, September 9, 13

71. @CarlBrwn #360iDev
Map the Corners
circled on the Left
To the Corners of
the Images on the
Right
37Monday, September 9, 13

72. @CarlBrwn #360iDev
Map the Corners
circled on the Left
To the Corners of
the Images on the
Right
37Monday, September 9, 13

73. @CarlBrwn #360iDev
Map the Corners
circled on the Left
To the Corners of
the Images on the
Right
37Monday, September 9, 13

74. @CarlBrwn #360iDev
Map the Corners
circled on the Left
To the Corners of
the Images on the
Right
37Monday, September 9, 13

75. @CarlBrwn #360iDev
Map the Corners
circled on the Left
To the Corners of
the Images on the
Right
37Monday, September 9, 13

76. @CarlBrwn #360iDev
ViewController.mm
- getAndApplyTransformPressed:
convert to cv::Mat
getPerspectiveTransform()
warpPerspective(transform)
38Monday, September 9, 13

77. @CarlBrwn #360iDev
ViewController.mm
- getAndApplyTransformPressed:
convert to cv::Mat
getPerspectiveTransform()
warpPerspective(transform)
convert to UIImage //Display
38Monday, September 9, 13

78. #360iDev@CarlBrwn
Unskew
getAndApplyTransformPressed:
39Monday, September 9, 13

79. #360iDev@CarlBrwn
Extract
Colors
extractColorsPressed:
40Monday, September 9, 13

80. @CarlBrwn #360iDev
ViewController.mm
- extractColorsPressed:
convert to cv::Mat
cv::cvtColor(CV_RGB2HSV)
Detect
41Monday, September 9, 13

81. @CarlBrwn #360iDev
HSV Space
Hue/Saturation/Value
Easier to get Color out
http://en.wikipedia.org/wiki/HSV_color_space
42Monday, September 9, 13

82. @CarlBrwn #360iDev
ViewController.mm
- extractColorsPressed:
convert to cv::Mat
cv::cvtColor(CV_RGB2HSV)
Detect
43Monday, September 9, 13

83. @CarlBrwn #360iDev
ViewController.mm
- extractColorsPressed:
convert to cv::Mat
cv::cvtColor(CV_RGB2HSV)
for (int hSlice;;) {
! ! for (int vSlice;;) {
Detect
43Monday, September 9, 13

84. @CarlBrwn #360iDev
ViewController.mm
- extractColorsPressed:
convert to cv::Mat
cv::cvtColor(CV_RGB2HSV)
for (int hSlice;;) {
! ! for (int vSlice;;) {
cv::Rect r()
Detect
43Monday, September 9, 13

85. @CarlBrwn #360iDev
ViewController.mm
- extractColorsPressed:
convert to cv::Mat
cv::cvtColor(CV_RGB2HSV)
for (int hSlice;;) {
! ! for (int vSlice;;) {
cv::Rect r()
Mat cv::roi(r)
Detect
43Monday, September 9, 13

86. @CarlBrwn #360iDev
ViewController.mm
- extractColorsPressed:
convert to cv::Mat
cv::cvtColor(CV_RGB2HSV)
for (int hSlice;;) {
! ! for (int vSlice;;) {
cv::Rect r()
Mat cv::roi(r)
avgColor=cv::mean(roi)
Detect
43Monday, September 9, 13

87. @CarlBrwn #360iDev
ViewController.mm
- extractColorsPressed:
convert to cv::Mat
cv::cvtColor(CV_RGB2HSV)
for (int hSlice;;) {
! ! for (int vSlice;;) {
cv::Rect r()
Mat cv::roi(r)
avgColor=cv::mean(roi)
//Check Saturation and Hue
Detect
43Monday, September 9, 13

88. @CarlBrwn #360iDev
ViewController.mm
- extractColorsPressed:
for (int hSlice;;) {
! ! for (int vSlice;;) {
Debugging
44Monday, September 9, 13

89. @CarlBrwn #360iDev
ViewController.mm
- extractColorsPressed:
for (int hSlice;;) {
! ! for (int vSlice;;) {
roi = avgColor //Fill
Debugging
44Monday, September 9, 13

90. @CarlBrwn #360iDev
ViewController.mm
- extractColorsPressed:
for (int hSlice;;) {
! ! for (int vSlice;;) {
roi = avgColor //Fill
putText(roi, [color cString]
Debugging
44Monday, September 9, 13

91. @CarlBrwn #360iDev
ViewController.mm
- extractColorsPressed:
for (int hSlice;;) {
! ! for (int vSlice;;) {
roi = avgColor //Fill
putText(roi, [color cString]
putText(roi, stringWithFormat:
@"%.0f/%.0f/%.0f")
Debugging
44Monday, September 9, 13

92. @CarlBrwn #360iDev
ViewController.mm
- extractColorsPressed:
for (int hSlice;;) {
! ! for (int vSlice;;) {
roi = avgColor //Fill
putText(roi, [color cString]
putText(roi, stringWithFormat:
@"%.0f/%.0f/%.0f")
convert to UIImage //Display
Debugging
44Monday, September 9, 13

93. #360iDev@CarlBrwn
Extract
Colors
extractColorsPressed:
45Monday, September 9, 13

94. #360iDev@CarlBrwn
Sudoku
Grabber
Demo 3/3
46Monday, September 9, 13

95. #360iDev@CarlBrwn
Find
Edges
findPuzzlePressed:
47Monday, September 9, 13

96. @CarlBrwn #360iDev
Just like Cube
48Monday, September 9, 13

97. #360iDev@CarlBrwn
UnSkew
Puzzle
squarePressed:
49Monday, September 9, 13

98. @CarlBrwn #360iDev
Just like Cube
50Monday, September 9, 13

99. #360iDev@CarlBrwn
Extract
Text
extractPressed:
51Monday, September 9, 13

100. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
Tesseract* tesseract =
Setup
52Monday, September 9, 13

101. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
Tesseract* tesseract =
initWithDataPath:@"tessdata"
language:@"eng"
Setup
52Monday, September 9, 13

102. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
Tesseract* tesseract =
initWithDataPath:@"tessdata"
language:@"eng"
tesseract
setVariableValue:@"0123456789"
forKey:@"tessedit_char_whitelist"
Setup
52Monday, September 9, 13

103. @CarlBrwn #360iDev
tesseract-ocr
http://code.google.com/p/tesseract-ocr/
53Monday, September 9, 13

104. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
convert to cv::Mat
Slice
54Monday, September 9, 13

105. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
convert to cv::Mat
for (int hSlice;;) {
! ! for (int vSlice;;) {
Slice
54Monday, September 9, 13

106. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
convert to cv::Mat
for (int hSlice;;) {
! ! for (int vSlice;;) {
cv::Rect r()
Slice
54Monday, September 9, 13

107. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
convert to cv::Mat
for (int hSlice;;) {
! ! for (int vSlice;;) {
cv::Rect r()
Mat cv::roi(r)
Slice
54Monday, September 9, 13

108. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
convert to cv::Mat
for (int hSlice;;) {
! ! for (int vSlice;;) {
cv::Rect r()
Mat cv::roi(r)
cv::cvtColor(CV_RGB2GRAY);
Slice
54Monday, September 9, 13

109. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
convert to cv::Mat
for (int hSlice;;) {
! ! for (int vSlice;;) {
cv::Rect r()
Mat cv::roi(r)
cv::cvtColor(CV_RGB2GRAY);
cv::threshold(roi_gray)
Slice
54Monday, September 9, 13

110. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
convert slice to UIImage
Detect
55Monday, September 9, 13

111. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
convert slice to UIImage
tesseract setImage:imageToParse
Detect
55Monday, September 9, 13

112. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
convert slice to UIImage
tesseract setImage:imageToParse
tesseract recognize
Detect
55Monday, September 9, 13

113. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
[[UILabel alloc]
initWithFrame:CGRectMake()]
Debugging
56Monday, September 9, 13

114. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
[[UILabel alloc]
initWithFrame:CGRectMake()]
label
setText:recognizedString
Debugging
56Monday, September 9, 13

115. @CarlBrwn #360iDev
ViewController.mm
- extractPressed:
[[UILabel alloc]
initWithFrame:CGRectMake()]
label
setText:recognizedString
self.view
addSubview:recognizedLabel
Debugging
56Monday, September 9, 13

116. #360iDev@CarlBrwn
Extract
Text
extractPressed:
57Monday, September 9, 13

117. #360iDev@CarlBrwn
Based on
Code
From:
github.com/carlbrown/
OpenCVDemo360iDev2013
Questions?
58Monday, September 9, 13