This document provides an overview of image processing and computer vision applications for the iPhone and iPad. It discusses the growing market for mobile apps in this field and the technical capabilities of iPhone devices. The document outlines a mini-course on developing iPhone and iPad apps for image processing and computer vision. It covers fundamentals of iOS development like Xcode, Objective-C, classes and objects, and the model-view-controller design pattern. It also discusses OpenCV and examples of commercial apps and student projects.

1. Image
Processing
and
Computer
Vision
in
iPhone
and
iPad
Oge Marques, PhD
omarques@fau.edu
VII Workshop de Visão Computacional (WVC)
Curitiba, PR - Brazil
May 2011

2. Take-home message
• Mobile image processing and computer vision
applications are coming of age.
• In this mini-course you will learn suggestions,
ideas, and resources for developing iPhone and
iPad apps in this field.

3. Disclaimer
• I’m a teacher, researcher, graduate advisor, author,
…
• … not a developer

5. Background and Motivation
Two
sides
of
the
coin
• The maturity and • The unprecedented success
popularity of image of the iPhone, iPod Touch,
processing and computer and – more recently – the
vision techniques and iPad
algorithms

6. Motivation
• Rich capabilities of iPhone for image and video
processing
• Apple support for image and multimedia:
frameworks, libraries, etc.
• Third-party support for iPhone-based
development: open APIs, OpenCV, etc.
• Success stories and ever-growing market

7. iPhone 4 technical specifications
1
GHz
ARM
Cortex-­‐A8
processor,
PowerVR
SGX535GPU,
Apple
A4
chipset

8. iPhone photo apps
• Huge growth in mobile apps, including image-rich
iPhone apps:
– 200+ photography-related apps available in iTunes
store
– Entire sites for reviews, discussions, etc.
– Subcategories include:
• Camera enhancements
• Image editing and processing (yes, Photoshop on the phone!)
• Image sharing (Instagram: 1 M users in less than 2 months!)
• Image printing, wireless transfer, etc.

9. Smart phone market

10. Smart phone market
Source: http://www.cellular-news.com/story/48647.php?s=h

11. Smart phone market
• According to the American Academy of Cosmetic
Dentistry:
– 5.1 billion people have a cell phone
– 4.2 billion people have a toothbrush
Source: http://www.softcity.com/contribution/AO4UTMwADN/view/more-cell-phone-owners-than-toothbrush-owners

12. Outline
• Part I – Fundamentals of iOS development
– Core concepts, techniques, and tools
• Part II – OpenCV and iOS
– Functionality and portability issues
• Part III – Examples and case studies
– Commercial apps and students’ projects

13. Part
I
Fundamentals of iOS
development

14. Brief survey
• Have worked with Object Oriented Programming?
• Have installed the iOS SDK?
• Have developed apps for iPhone / iPod touch /
iPad?
• Have submitted apps to the App Store?
• Have developed image processing / computer
vision software?
• Have worked with OpenCV?

15. Checklist
• What you’ll need:
– Intel-based Macintosh running Snow Leopard (OS X
10.6.5 or later)
– Sign up to become a registered iOS developer
• Apple requires this step before you’re allowed to download
the iOS SDK and many other goodies from the
iOS Dev Center
– xCode 4 and latest (4.3) version of iOS SDK
– iOS simulator
– iPhone, iPod Touch, or iPad (optional)

16. Welcome to the jungle
• Only one active app
• Only one (fixed-size) window
• Limited access (“sandbox”)
• Limited response time
• Limited screen size
– 640 × 960: retina display devices (iPhone 4, 4th-gen iPod touch)
– 320 × 480: older devices
– 1024 × 768: iPad
• Limited system resources
• No garbage collection
• No (physical) keyboard or mouse

17. App Bundle
• Your code
• Any frameworks you’re linking to
• Nib files (interface builder)
• Resources (images, sound, etc.)
• PLIST files (app settings)

18. Tools and documentation
• Xcode
• Interface Builder
• iOS Simulator
• Instruments
• Apple documentation

19. Xcode 4: what you need to know
• Workspace window
• How to open an existing project
• How to navigate through the project’s contents
• How to organize views, subwindows, etc.
• How to use the debugging options
• How to create a new project
• How to edit targets and build settings
• Taking advantage of code completion and other
sources of immediate help
• How to find help and integrate with Apple
documentation

20. Xcode 4
• Workspace window

21. Interface Builder
• Workspace window
• How to create a new NIB file
• How to add views and objects to the UI
• How to make connections between IB objects and
source code

22. Interface Builder
• Workspace window

23. iOS Simulator
• How it integrates with Xcode
• What it can do
– Simulate different devices and generations (iPhone
before and after retina display, iPad)
– Basic device behavior (home button, etc.)
– Selected built-in apps (Safari, Contacts, etc.)
– Simulate rotation, low memory warnings, etc.
• What it cannot do
– Provide a good measure of app performance on the
actual device
– Simulate camera, accelerometer, etc.

24. Instruments
• Very rich tool (integrated with Xcode) for app
performance and behavior evaluation

25. Example
• Hello, World!

26. iOS technology layers
Our primary interest
Most iOS courses, books, etc.
(2D/3D graphics,
image, audio, video)

27. Object Oriented Programming
• Class vs. Instance
• Methods
• Instance Variables (properties)

28. 3 Pillars of OOP
• Encapsulation - hide the details from the outside
world
• Polymorphism - different objects, same interface
• Inheritance - acquire the features of the parent
and add onto its capabilities

29. Inheritance in Cocoa Touch

30. What is Objective C?
• Superset of C
– Mixing ObjC and C, or even ObjC and C++
– Syntax a little different than C
– Strong or loosely typed (your choice)

31. Additions from C
• Types
– Anonymous object
– Class
– Selectors
• Class definition
• Sending messages

32. File extensions
• Objective-C is a superset of ANSI C and supports
the same basic syntax as C.
• Objective-C files use the extensions below:
Based on “Learning Objective-C: A Primer”

33. #import vs. #include
• When you want to include header files in your
source code, you typically use a #import
directive.
• This is like #include, except that it makes sure
that the same file is never included more than
once.
Based on “Learning Objective-C: A Primer”

34. Classes Instances
• A class is...
– a blueprint to creating an instance
– an object too (type Class )

35. Classes Objects
• Classes declare state and behavior for a type
• An instance of a class maintains its state through
instance variables
• Behavior is implemented through methods
• Instance variables (iVars) are typically hidden
– Usually accessible via getter and setter methods, or
the handy property

36. Classes in ObjC
• Specification of a class in ObjC:
– Interface: class declaration, instance variables (ivars),
and methods associated with the class. Usually a .h file.
– Implementation: actual code for the methods of the
class. Usually a .m file.
Based on “Learning Objective-C: A Primer”

37. Class declaration
Based on “Learning Objective-C: A Primer”

38. Header File
• Example:

39. Implementation File
• All implementation goes between @implementation and @end
• use #pragma mark to block groups of code together

40. Strong and weak types
• ObjC supports both strong and weak typing for
variables containing objects.
– Strongly typed variables include the class name in the
variable type declaration.
– Weakly typed variables use the type id for the object
instead. Typically used for things such as collection
classes, where the exact type of the objects in a
collection may be unknown.
Based on “Learning Objective-C: A Primer”

41. Instances, methods, and messaging
• A class in ObjC can declare two types of
methods:
– instance methods: execution is scoped to a particular
instance of the class.
• Before you call an instance method, you must first create an
instance of the class.
– class methods: do not require you to create an instance.
• These methods must be called on the class.
Based on “Learning Objective-C: A Primer”

42. Instances, methods, and messaging
• Method declaration syntax
Based on “Learning Objective-C: A Primer”

43. Instances, methods, and messaging
• When you want to call a method, you do so by
messaging an object.
• A message is the method signature, along with the
parameter information the method needs.
• All messages you send to an object are dispatched
dynamically, thus facilitating the polymorphic
behavior of Objective-C classes.
Based on “Learning Objective-C: A Primer”

44. Messaging
• When you call a method, its referred to as
sending that message to the receiving object.
• Message Expression
[receiver method:argument];
• Example:
[myCar accelerate];
will send the accelerate message to object myCar

45. Message (cont.)
• Named arguments
[myCar turnOnRadio];
[myCar turnOnRadioWithStation:93.1];
[myCar turnOnRadioWithStation:93.1 volume:0.8];
• Definitions of these methods:
- (void)turnOnRadio;
- (void)turnOnRadioWithStation:(double)station;
- (void)turnOnRadioWithStation:(double)station
volume:(double)volume;

46. Instances, methods, and messaging
• Messaging example:
– To send the insertObject:atIndex: message to an
object in the myArray variable, you would use the
following syntax:
[myArray insertObject:anObject atIndex:0];
Based on “Learning Objective-C: A Primer”

47. Instances, methods, and messaging
• Messages can be nested.
– Example:
• If you had another object called myAppObject that had
methods for accessing the array object and the object to
insert into the array, you could write the preceding example
to look something like the following:
[[myAppObject theArray] insertObject:[myAppObject
objectToInsert] atIndex:0];
Based on “Learning Objective-C: A Primer”

48. Instances, methods, and messaging
• ObjC also provides a dot syntax for invoking
accessor/mutator (getter/setter) methods.
• Using dot syntax, you could rewrite the previous
example as:
[myAppObject.theArray insertObject:
[myAppObject objectToInsert] atIndex:0];
Based on “Learning Objective-C: A Primer”

49. Instances, methods, and messaging
• You can also use dot syntax for assignment.
• Example:
myAppObject.theArray = aNewArray;
is simply a different syntax for writing
[myAppObject setTheArray:aNewArray];
Based on “Learning Objective-C: A Primer”

50. Instances, methods, and messaging
• To use the dot syntax, your getter and setter
methods must have a specific method signature:
- (NSString*)myString;
//getter
- (void)setMyString:(NSString*)aString //setter

51. Instances, methods, and messaging
• Class methods
– Typically used as factory methods to create new
instances of the class or for accessing some piece of
shared information associated with the class.
– The syntax for a class method declaration is identical
to that of an instance method, with one exception.
• Instead of using a minus sign for the method type identifier,
you use a plus (+) sign.
Based on “Learning Objective-C: A Primer”

52. Instances, methods, and messaging
• Class methods
– Example:
• In this case, the array method is a class method on the
NSArray class - and inherited by NSMutableArray - that
allocates and initializes a new instance of the class and
returns it to your code.
NSMutableArray *myArray = nil;
// nil is essentially the same as NULL
// Create a new array and assign it to the myArray variable.
myArray = [NSMutableArray array];
Based on “Learning Objective-C: A Primer”

53. Properties
• Properties are a shortcut for creating getter/
setter methods
• Can still use dot notation because getter/setter
methods are being used under the hood
double speed = myCar.speed;
• Attributes of the property are declared with the
property itself:
– read-only/read-write access
– Memory management policy

54. Properties (.h file)
@interface Car : NSObject {
double speed_;
}
@property (nonatomic, assign) double speed;
@end

55. Properties (.m file)
• @synthesize - creates the getter/setter
methods for you based on declaration
• @dynamic - allows you to create the getter/
setter methods

56. Properties (.m file)
• Example:
@implementation Car
@synthesize speed = speed_;
//rest of your class implementation
@end

57. read only/read-write
@property double speed;
// read-write by default
@property (readonly) double speed;

58. Initializing Instances
• 2 steps:
– Allocate
– Initialize
• Example: Car *myCar = [[Car alloc] init];
– the myCar var is a pointer, but that s all you need to
know about pointers
• Some classes have additional initialization methods
Car *myCar = [[Car alloc] initWithPassengers:2];

59. Types
• id - a reference to any object
– Car *myCar; (static typing)
– id myCar; (notice no *)(dynamic typing)
• NSObject vs. id
– id is a placeholder for anything
– NSObject is the root class
– No compile-time warning when sending messages to
id
– Messages sent to an NSObject will be checked

60. nil
• nil represents nothing or null
if (myCar == nil) { do something }
if (!myCar) { do something }
Car *myCar = nil;
• Messages CAN be sent to nil:
Car *myCar = nil;
[myCar accelerate];

61. BOOL
• Assigned values of YES or NO
BOOL isCarOn = NO;
if (isCarOn == YES)
if (isCarOn)
if (!isCarOn)
isCarOn = 1;

62. The Class object
• An object that gives details about a given class (i.e.
name, inheritance, etc.)
Class *mustangClass = [Mustang class];
[mustang isSubclassOfClass:[Car class]];
• Introspection - asking a class about itself
if ([myObject isKindOfClass:[Car class]]) {
// do something, since we know it’s a Car
Car *myCar = (Car*)myObject;
}

63. Identity Equality
• Identity - if two pointers point to the same
memory
if (objectA == objectB) {
//objectA and objectB are the same instance
}
• Equality - if contents are semantically equal
if ([objectA isEqual:objectB]) {
//different instances, but conceptually equal
}

64. NSObject
• Root class
• Built-in implementation that all sub-classes
inherit
– Memory management
– Introspection
– Object equality

65. NSObject’s description method
• NSObject defines the description method:
- (NSString*)description;
– By default, returns memory location and type
– You can override this to show more meaningful info

66. NSString
• Abstracts unicode characters
• Always use this instead of char* (unless you
absolutely need to)
– If you re invoking a C method that requires a char*,
you can get char* from an NSString through the
UTF8String method
NSString *myString = @”Hello, World!”;

67. NSString (cont.)
• Formatting strings
NSString *name = @”Bob”
NSString *hello = [NSString
stringWithFormat:@”Hello, %@”, name];
– uses standard printf formatting
• Logging strings
NSLog([myObject description]);
NSLog(@”Hello, %@”, name);
• Lots of other useful methods on NSString (see
documentation)

68. NSString (cont.)
• NSString is immutable
• @”hello” is an NSString
– ex, if a method exists on Logger class:
- (void)logMessage:(NSString*)message;
– calling it with a string:
Logger *logger = [[Logger alloc] init];
[logger logMessage:@”Hello, World!”];
– No need to instantiate an NSString object, just pass
the string in directly

69. Collections
• NSArray - an ordered collection of objects
• NSDictionary - collection of key-value pairs
• NSSet - unordered collection of unique objects
– These are all immutable collections.
• Mutable versions: NSMutableArray,
NSMutableDictionary, NSMutableSet
– Careful with exposing these as properties

70. NSNumber
• A wrapper class that abstracts the concept of a
number (int, double, float, etc).
NSNumber *myInt = [NSNumber numberWithInt:10];
NSNumber *myDouble = [NSNumber numberWithDouble:
5.2];
int a = [myInt intValue];
double b = [myDouble doubleValue];
• Useful when you need to store a number in a
dictionary (because dictionaries only accept
NSObjects).

71. Example
• Car

72. Memory management in ObjC
• Referencing Counting
– Every object has a retain count (how many other
objects are interested in it)
• As long as retain count 0, object is alive
• When retain count drops to 0, memory is freed

73. Referencing Counting
• [myObject retain] will increment retain
count
• [myObject release] will decrement retain
count
• Once retain count = 0, object is no longer
available, no turning back.

74. Referencing Counting
• Calling alloc sets retain count to 1
Car *myCar = [[Car alloc] init];
– Retain count on myCar = 1

75. Dealloc
• The dealloc method deallocates memory for
an object
• dealloc is never explicitly called by your
code!!! NSObject handles this call for you.
• In your dealloc method, you release any
objects you have a reference to

76. Dealloc
You override dealloc when you need to clean up
(release) your own iVars
- (void)dealloc {
//clean up any objects referenced
[super dealloc];
}
- (void)dealloc {
[name_ release];
//clean up any objects referenced
[super dealloc];
}

77. Ownership
You will typically use properties, which generate getter/
setter methods automatically.
Internally, this is what they look like (or how you would
code it if you did it yourself).
- (void)setCar:(Car*)car { - (void)setName:(NSString*)name {
[car_ autorelease]; [name_ autorelease];
car_ = [car retain]; name_ = [name copy];
} }

78. Autorelease
• Solves the problem where you need to return a
newly created object before releasing it.
• Example:
+ (NSString*)fullNameWithFirstName:(NSString*)first lastName:(NSString*)last
{
NSString *full = [[NSString alloc] initWithFormat:@”%@ %@”, first,
last];
[full autorelease];
return full;
}

79. Autorelease
• Use the same release rules as before
• Even if retain count goes to 0, object will stick
around for a while longer
• System will automatically get rid of it
– Q: What manages auto-released objects?
– A: Auto release pools

80. Autorelease Pools
• You typically don t create them yourself (except
for threading cases)
• When an autorelease pool is drained,” any
object that was sent the message “autorelease”
will get released
• On the main thread, this is done automatically
– At the end of each run loop execution
• Avoid autoreleasing if possible. Better to alloc/
release (frees memory right away).

81. Rules Conventions
• When do you release an object?
• Methods that contain the words alloc or
copy return a retained object - you must
release it
• Any other method is returning an autoreleased
object.
• You should follow this convention when defining
your own methods!

82. Design Patterns
• A design pattern is usually defined as “a template
for a design that solves a general, recurring
problem in a particular context”
• Use them! (don’t go against them)
• Design patterns used often in iOS development:
– Model-View-Controller (MVC)
– Delegate
– Target-Action

83. MVC
• Model - The entity that models the
data
• View - The user interface
• Controller - Manages the connection
between view and model

84. MVC - Model
• The model should only be concerned with the
data.
• Nothing to do with visual representation
• Can be persisted
• Models should be reusable

85. MVC - View
• The presentation of the data (or model)
• Allows user to interact with the data
• Does not store data, except for caching
• Easily reusable

86. MVC - Controller
• Brings the Model and View together
• Receives events from View, and acts on the
Model
• Updates the View when the Model changes
• This is where app logic is typically written

87. Delegation
• Delegation is a design pattern in which one object in a
program acts on behalf of, or in coordination with,
another object.
• The delegating object keeps a reference to the other
object (the delegate) and at the appropriate time sends a
message to it.
• The message informs the delegate of an event that the
delegating object is about to handle or has just handled.
• The delegate may respond to the message by updating
the appearance or state of itself or other objects in the
application, and in some cases it can return a value that
affects how an impending event is handled.

88. Delegation
• The main value of delegation: it allows you to easily
customize the behavior of several objects in one central
object.
• This concept is closely related to the concept of a
callback function, used, for example, in Ajax-based web
development.
– A callback is a function that is triggered when an event
occurs.
– Since we usually don’t know when the event will occur, we
set a callback that acts as a listener for that event.
– In Cocoa Touch, callbacks are implemented using delegation.
• Most iOS applications will include delegates for some of
their classes.

89. Delegation
• The concept of delegation is closely related to the
Objective-C construct known as protocol.
• A protocol is simply a list of method declarations,
unattached to a class definition.
• Protocols define an interface that other objects are
responsible for implementing.
• When you implement the methods of a protocol in
one of your classes, your class is said to conform to
that protocol.
• Protocols are used to specify the interface for
delegate objects.

90. Target-Action
• Allows you to handle actions or events by a
UI control
• Some events:
– touchDown
– touchUpInside
– valueChanged

91. Target-Action
• 3 things are defined:
– Target myObject
– Action @selector(convertFtoC:)
– Event UIControlEventTouchUpInside

92. Application event loop

93. MVC in Cocoa Touch
• Create your View components using Interface Builder
– Sometimes you ll create / modify your interface from code.
– You might also subclass existing views and controls.
• Model = Objective-C classes designed to hold your
application s data or data model using Core Data
• Controller = typically composed of classes that you
create and that are specific to your app.
– Controllers can be completely custom classes (NSObject
subclasses), but more often, they will be subclasses of one of
several existing generic controller classes from the UIKit
framework, such as UIViewController.
– By subclassing one of these existing classes, you will get a lot of
functionality for free and won t need to spend time recoding
the wheel, so to speak.

94. Outlets
• Outlets
– Our controller class can refer to objects in the nib file
by using a special kind of instance variable called an
outlet.
– Think of an outlet as a pointer that points to an object
within the nib.
• Example: suppose you created a text label in Interface
Builder and wanted to change the label s text from within
your code. By declaring an outlet and connecting that outlet
to the label object, you could use the outlet from within
your code to change the text displayed by the label.

95. Actions
• Actions
– Going in the opposite direction, interface objects in our
nib file can be set up to trigger special methods in our
controller class.
– These special methods are known as action methods.
• Example: you can tell Interface Builder that when the user
touches up (pulls a finger off the screen) within a button, a
specific action method within your code should be called.

96. Outlets
• More on Outlets
– Outlets are instance variables that are declared using the
keyword IBOutlet.
• A declaration of an outlet in your controller s header file might
look like this:
@property (nonatomic, retain) IBOutlet
UIButton *myButton;

97. Outlets
• More on Outlets
– IBOutletdoes absolutely nothing as far as the compiler is
concerned.
• Its sole purpose is to act as a hint to tell Interface Builder that this
is an instance variable that we re going to connect to an object in
a nib file.
• Any instance variable that you create and want to connect to an
object in a nib file must be preceded by the IBOutlet keyword

98. Outlets
• Recent changes
– Before iOS 4:
IBOutlet UIButton *myButton;
– Since then, Apple s sample code has been moving
toward placing the IBOutlet keyword in the
property declaration:
@property (nonatomic, retain) IBOutlet UIButton
*myButton;
– Both mechanisms are supported.

99. Actions
• Actions are methods that are part of your
controller class.
– They are also declared with a special keyword,
IBAction, which tells Interface Builder that this
method is an action and can be triggered by a control.
– Typically, the declaration for an action method will
look like this:
- (IBAction)doSomething:(id)sender;
– The actual name of the method can be anything you
want, but it must have a return type of IBAction,
which is the same as declaring a return type of void.

100. Actions
• If you don t need to know which control called
your method, you can also define action methods
without a sender parameter:
- (IBAction)doSomething;

101. Example
• Button Fun
– Creating the View Controller
• ButtonFun_ViewController.h

102. Example
ButtonFun_ViewController.m

103. Example
ButtonFun_ViewController.m

104. Example
• Button Fun
– Creating the App Delegate
Button_FunAppDelegate.h

105. Example
• Button Fun
– The UIApplicationDelegate protocol

106. Example
Button_FunAppDelegate.m

107. Example
Button_FunAppDelegate.m

108. Example
• Button Fun
– Editing MainWindow.xib

109. Example
• Button Fun
– Editing ButtonFun_ViewController.xib

110. Example
• Button Fun
– Screenshots

111. Another Example
• SayMyName
– Screenshots

112. UI design aspects
• The iOS development environment adheres to human
interface guidelines, strategies and best practices.
– See http://tinyurl.com/3yj7b5y
• Each iPhone or iPad app consists of familiar UI
elements, organized in a somewhat predictable way,
and associated with equally predictable actions.
• UI elements can be selected, configured, and
customized interactively using Interface Builder.
– They can also be created and manipulated
programmatically.

113. A guided tour of UI elements

114. Example
• Control Fun

115. Camera, photo library, image manipulation
• The iPhone, iPod touch, and second-generation
iPad each have one or two built-in cameras and a
built-in application called Photos that allows you to
manage the device’s photo and video library.
• Due to the “sandboxed” nature of iOS, an
application cannot have direct access to
photographs or any other data that live outside of
their own sandboxes.
– This limitation has been circumvented by exposing the
camera and the media library to other applications by
way of an image picker mechanism.

116. Camera, photo library, image manipulation
• The main classes that you need to understand in
order to develop basic applications involving
images, camera, and photo library for the iPhone
are:
– UIImageView
– UIImagePickerController
• (and its associated protocol, UIImagePickerControllerDelegate).

117. UIImageView
• An image view object provides a view-based container for
displaying either a single image or for animating a series of
images.
• New image view objects are configured to disregard user events
by default.
– If you want to handle events in a custom subclass of UIImageView, you
must explicitly change the value of the userInteractionEnabled property
to YES after initializing the object.
• When a UIImageView object displays one of its images, the actual
behavior is based on the properties of the image and the view.
– If either of the image’s leftCapWidth or topCapHeight properties are
non-zero, then the image is stretched according to the values in those
properties.
– Otherwise, the image is scaled, sized to fit, or positioned in the image
view according to the contentMode property of the view.

118. UIImageView
• Apple recommends that you use images that are
all the same size.
– If the images are different sizes, each will be adjusted
to fit separately based on that mode.
• All images associated with a UIImageView object
should use the same scale.
– If your application uses images with different scales,
they may render incorrectly.

119. UIImagePickerController
• The UIImagePickerController class provides basic,
customizable user interfaces for taking pictures and movies
and for giving the user some simple editing capability for
newly-captured media.
• The role and appearance of an image picker controller
depend on the source type you assign to it before you
present it:
– A sourceType of UIImagePickerControllerSourceTypeCamera
provides a user interface for taking a new picture or movie (on
devices that support media capture).
– A sourceType of UIImagePickerControllerSourceTypePhotoLibrary
or UIImagePickerControllerSourceTypeSavedPhotosAlbum provides a
user interface for choosing among saved pictures and movies.

120. UIImagePickerControllerDelegate
• The UIImagePickerControllerDelegate protocol
defines methods that your delegate object must
implement to interact with the image picker
interface.
• The methods of this protocol notify your delegate
when the user either picks an image or movie, or
cancels the picker operation.
• The delegate methods are responsible for
dismissing the picker when the operation
completes.

121. The UIImage class
• A UIImage object is a high-level way to display
image data.
• Images can be created from files, from Quartz
image objects, or from raw image data received by
the application.
• The UIImage class also offers several options for
drawing images to the current graphics context
using different blend modes and opacity values.
• The UIImage class supports the most popular
image file formats.

122. The UIImage class
• Recommendations and warnings:
– Image objects are immutable, i.e., you cannot change their
properties after creation.
• This means that you generally specify an image’s properties at
initialization time or rely on the image’s metadata to provide the
property value.
• Because image objects are immutable, they also do not provide
direct access to their underlying image data.
– In low-memory situations, image data may be purged from
a UIImage object to free up memory on the system.
• This purging behavior affects only the image data stored internally
by the UIImage object and not the object itself.
– Avoid creating UIImage objects that are greater than 1024
× 1024 in size.

123. Example
• Camera
– Illustrates the image picker mechanism

124. Example
• Camera

125. The Media Layer
• The Media Layer of the iOS architecture contains
the graphics, audio, and video technologies that
provide multimedia support on a mobile device.
• The simplest (and most efficient) way to create an
application is to use pre-rendered images together
with the standard views and controls of the UIKit
framework and let the system do the drawing.
– However, there may be situations where additional
technologies are needed to manage the application’s
graphical content.

126. The Media Layer
• Core Graphics framework: contains the interfaces
for the Quartz 2D drawing API, which handles
native 2D vector- and image-based rendering.
• Core Animation (part of the Quartz Core
framework): provides advanced support for
animating views and other content.
• OpenGL ES framework: provides support for 2D
and 3D rendering using hardware-accelerated
interfaces.

127. The Media Layer
• Core Text framework: contains a set of simple, high-
performance C-based interfaces for laying out text
and handling fonts.
• Image I/O: provides interfaces for reading and writing
most image formats and associated image metadata.
– This framework makes use of the Core Graphics data
types and functions and supports all of the standard image
types available in iOS.
• Assets Library framework: provides a query-based
interface for retrieving photos and videos from the
user’s device.

128. Quartz 2D
• Advanced, two-dimensional drawing engine available for iOS
application development.
• Provides low-level, lightweight 2D rendering with
unmatched output fidelity regardless of display or printing
device.
• Resolution- and device-independent.
• The Quartz 2D application programming interface (API) is
easy to use and provides access to powerful features.
– The Quartz 2D API is part of the Core Graphics framework, so
you may see Quartz referred to as Core Graphics or, simply,
CG.
• In iOS, Quartz 2D works with all available graphics and
animation technologies, such as Core Animation, OpenGL
ES, and the UIKit classes.

129. Quartz 2D
• The painter’s model

130. Quartz 2D
• Graphics context
– A graphics context represents a drawing destination.
• It contains drawing parameters and all device-specific information
that the drawing system needs to perform any subsequent
drawing commands.
– A graphics context defines basic drawing attributes such as
the colors to use when drawing, the clipping area, line
width and style information, font information, compositing
options, etc.
– A graphics context is represented in code by the data type
CGContextRef.
• After you obtain a graphics context, you can use Quartz 2D
functions to draw to the context, perform operations (such as
translations) on the context, and change graphics state
parameters, such as line width and fill color.

131. Quartz 2D
• Bitmap graphics context
– Allows you to paint RGB colors, CMYK colors, or
grayscale into a bitmap.
• A bitmap is a rectangular array (or raster) of pixels, each
pixel representing a point in an image.
• Bitmap images are also called sampled images.
– A bitmap graphics context accepts a pointer to a
memory buffer that contains storage space for the
bitmap.
• When you paint into the bitmap graphics context, the buffer
is updated.
• After you release the graphics context, you have a fully
updated bitmap in the pixel format you specify.

132. Creating a bitmap graphics context

133. Drawing to a bitmap graphics context

134. Bitmap images and image masks
• A bitmap image is an array of pixels, where each pixel
represents a single point in the image.
• Bitmap images are restricted to rectangular shapes,
but with the use of the alpha component, they can
appear to take on a variety of shapes and can be
rotated and clipped.
• Each sample in a bitmap contains one or more color
components in a specified color space, plus one
additional component that specifies the alpha value to
indicate transparency.
– Each component can be from 1 to as many as 32 bits.

135. Bitmap images and image masks
• When you create and work with Quartz images
(which use the CGImageRef data type), you will
notice that some Quartz image-creation functions
require that you specify all this information, while
other functions require a subset of this
information.
– What you provide depends on the encoding used for
the bitmap data, and whether the bitmap represents an
image or an image mask.

136. Functions for creating images

137. Image masks
• An image mask is a bitmap that specifies an area to
paint, but not the color.
• A Quartz bitmap image mask is used the same way an
artist uses a silkscreen.
• A bitmap image mask determines how color is
transferred, not which colors are used.
• Each sample value in the image mask specifies the
amount that the current fill color is masked at a
specific location.
• The sample value specifies the opacity of the mask.
– Larger values represent greater opacity and specify
locations where Quartz paints less color.

138. Image masks
• The function CGImageMaskCreate creates a
Quartz image mask from bitmap image
information that you supply.

139. Image I/O
• The Image I/O programming interface allows applications to
read and write most image file formats.
– Originally part of the Core Graphics framework, Image I/O now
resides in its own framework to allow developers to use it
independently of Core Graphics (Quartz 2D).
• Image I/O provides the definitive way to access image data
because it is highly efficient, allows easy access to metadata,
and provides color management.
• The Image I/O framework provides opaque data types for
reading image data from a source (CGImageSourceRef) and
writing image data to a destination (CGImageDestinationRef).
• The Image I/O framework understands most of the
common image file formats.

140. iPad
• Different considerations must be made when
building an iPad app
• Because of the larger screen, users expect a more
immersive experience
• You do NOT want to simply auto-size your
iPhone app to stretch out to the new dimensions
– This is a horrible waste of space, and makes for a poor
UX

141. iPad
• Some new controls
– UISplitViewController
– Popovers
• Other differences from iPhone
– Due to the physical design of the iPad, it is much easier
to turn the device to a different orientation.
Therefore, it is assumed that iPad apps support all
orientations
• This is not the case with iPhone

142. UISplitViewController

143. Popovers

144. iPad
• Two ways to build for iPad:
– Create an iPad-only project
• All code will be specific for iPad
• It is more difficult to reuse code this way
• Good if you only plan on releasing an iPad app
– Create a universal app
• Will support both iPhone and iPad in one download
• Allows you to easily share code

145. Building an iPad-only project
• Use the same concepts as you would with an
iPhone app
– MVC
– Delegation
– Etc.
• The only real difference is how views are laid out
on the screen (be sure to use the screen real
estate appropriately).

146. Building a universal app
• Still use the same patterns from iPhone
development
• The project will consist of 2 different versions of
each of your views
– If your views are built in NIB files, then you would have
2 NIB files for each view: one for iPhone, and another
for iPad
– If you are subclassing UIView for any of your views,
you will have 2 subclasses: one for iPhone, and another
for iPad

147. Building a universal app
• Your project will consist of 2 versions of each of your
view controllers
– The problem here is that the view controllers typically
contain business logic (business rules, logical flow of the
app, etc.)
– You do not want to duplicate business logic
– The answer:
• Create a base class for each view controller
• This base class contains shared code (anything that is reusable
between iPhone/iPad)
• You then create 2 subclasses of this base class: one for iPhone, and
another for iPad
• These subclasses contain the details that only pertain to that
form.

148. Building a universal app
• Creating subclasses is fine, but how does the app
know which one to create at runtime?
– It all starts from the main NIB file
– In a universal app, there are 2 main NIB files: one for
iPhone, and another for iPad
– The PLIST configuration file settings indicate which
NIB gets loaded for a specific form factor
– Your job: Customize these NIB files so that it starts
loading the proper classes
– This creates a chain reaction

149. Building a universal app
• What about just having an IF statement
that checks for the type of device?
if
(device
==
iPad)
do
something
for
iPad
else
do
something
for
iPhone
• NO!!! This is bad!
– Conceptually this may work (and it would),
however the logic in your view controllers
becomes cluttered with IF-ELSE statements
– Very difficult to maintain
• Apple’s recommended method: create
subclasses instead of IF-ELSE logic

150. Part II
OpenCV and iOS

151. OpenCV
• OpenCV (Open Source Computer Vision) is
a library of programming functions for real-
time computer vision.
• OpenCV is released under a BSD license; it is
free for both academic and commercial use.
• Goal: to provide a simple-to-use computer
vision infrastructure that helps people build
fairly sophisticated vision applications quickly.
• The library has 2000+ optimized algorithms.
– It is used around the world, has 2M downloads
and 40K people in the user group.

152. OpenCV
• 5 main components:
1. CV: basic image processing and higher-level computer
vision algorithms
2. ML: machine learning algorithms
3. HighGUI: I/O routines and functions for storing and
loading video and images
4. CXCore: basic data structures and content upon
which the three components above rely
5. CvAux: defunct areas + experimental algorithms; not
well-documented.

153. OpenCV
• Image representation: IplImage
– Basic structure used to encode [grayscale, color, or
four-channel (RGB + alpha)] images.
• Moreover, each channel may contain any of several types of
integer or floating-point numbers, which makes for a very
flexible representation.
• For practical purposes, we can say that an IplImage is
derived from another class, CvMat, which can be though of
as inherited from a third, abstract base class, CvArr.

154. OpenCV
• Image processing and computer vision
• Machine learning
– Filters
– K-means clustering algorithm
– Morphological operators
– Naive Bayes classifier
– Geometric transforms
– Binary decision trees
– Edge detection
– Boosting algorithms, e.g., AdaBoost
– Hough transform
– Viola-Jones classifier and its application
– DFT, DCT
to face detection
– Histogram-based matching
– Expectation maximization (EM)
– Template matching
clustering
– Freeman codes
– K-nearest neighbors (KNN) classifier
– Shape matching
– Support vector machine (SVM)
– Foreground-background segmentation
– etc.
– Inpainting
– Corner and interest point detection
– Optical flow
– Object tracking in video sequences
– Extensive support for stereo imaging
– etc.

155. OpenCV and iOS
• Since OpenCV is not available as a bundled library in
Apple’s iOS framework, we must compile and bring in
the OpenCV libraries and headers ourselves.
• Step-by-step setup described in detail at
http://niw.at/articles/2009/03/14/using-opencv-on-iphone/en
– Follow the instructions and look for possible updates.
• Once you have completed the setup process, you
should have access to both compiled versions of
OpenCV:
– one used when building our applications to test on the iOS
Simulator;
– one needed for final deployment to the actual device.

156. OpenCV and iOS
• When working with OpenCV in the iOS SDK:
1. Convert the UIImage object to an IplImage
structure;
2. Apply OpenCV library function(s);
3. Convert IplImage back to a UIImage before
attempting to display it on the screen.

157. OpenCV and iOS
• Code to convert from UIImage to IplImage
– Note that IplImage also needs to be released by
using cvReleaseImage.

158. OpenCV and iOS
• Code to convert from IplImage to UIImage

159. OpenCV and iOS
• Edge detection code snippet

160. OpenCV and iOS
• Example: OpenCV sample app
– Based on code from Yoshimasa Niwa
– Demonstrates two functions:
• Edge detection
• Face detection

161. OpenCV and iOS
• Main methods
– opencvFaceDetect: iOS-compatible wrapper
around the OpenCV library function that implements
the Haar classifier (cvHaarDetectObjects).
– opencvEdgeDetect: iOS-compatible wrapper
around the OpenCV library function that implements
the Canny edge detector (cvCanny).

162. OpenCV and iOS
DEMO

163. Part III
Examples and case studies

164. OpenGL ES
• OpenGL is “the most widely adopted
graphics standard in the computer graphics
and gaming industry”.
• The OpenGL ES framework
(OpenGLES.framework) provides tools for
drawing 2D and 3D content.
• It’s a C-based framework that works closely
with the device hardware to provide high
frame rates for full-screen game-style
applications.

165. OpenGL ES
• Example: GLImageProcessing sample app
– Demonstrates how to implement simple image processing
filters (Brightness, Contrast, Saturation, Hue rotation,
Sharpness) using OpenGL ES1.1.
– Also shows how to create simple procedural button icons
using CoreGraphics.
– By looking at the code you'll see how to set up an
OpenGL ES view and use it for applying a filter to a
texture.
• The application creates a texture from an image loaded from
disk.
– Use the slider to control the current filter.
• Only a single filter is applied at a time.
• Result cannot be saved.

166. OpenGL ES
• Screenshots

167. OpenGL ES
DEMO

168. Image tagging using the IQEngines API
• IQEngines is the brain behind the oMoby
iPhone app)

169. Image tagging using the IQEngines API

170. The IQEngines API
• Query API: allows the user to send an image to IQ Engines’
server to be processed by IQ Engines’ image labeling
engine.
– It expects an HTTP callback as one of its arguments; as soon as
the images are labeled, the results will be posted to that URL.
• Update API: a long polling request that returns the results
for one or more images as soon as they have been labeled.
• Training API: allows you to upload images to the IQ Engines’
object search database (alpha trials, by request only).
• Result API: returns the labels for a specific image if it has
been successfully processed, or informs that the image is
still being processed.
• Crowdsource API: “coming soon”.

171. The IQEngines API demo app
• Screenshots

172. The IQEngines API demo app
• XML-formatted response

173. The IQEngines API demo app
DEMO

174. Other third-party APIs
• Successful photo-based iPhone app developers are
starting to release APIs for third-party developers.
• Recent examples:
– PicPlz has made their API (+ examples) available earlier
this year
– Instagram appears to have followed suit
a few days later

175. iTranslatAR
• By Julie Carmigniani [Florida Atlantic University]
• An app for translating pictures of text into any
language supported by Google Translate.
• The picture’s text is converted into a string using
the Tesseract OCR software (now available at
Google Code)

176. iTranslatAR
• The app currently has three views in a tab
controller:
– a view for translating text through pictures;
– one for translating text through typing; and
– one for changing the target language.

177. iTranslatAR
• Screenshots

178. iTranslatAR
DEMO

179. iTranslatAR
• Will be extended to include (near) real-time
camera support inspired by Word Lens and
augmented reality (the AR in iTranslatAR)
capabilities.

180. An integrated app: Photastic
• By Asif Rahman
[Florida Atlantic University]

181. An integrated app: Photastic
DEMO

182. Concluding thoughts
• Mobile image processing, image search, and computer
vision-based apps have a promising future.
• There is a great need for good solutions to specific
problems.
• I hope this mini-course has provided a good starting
point and many useful pointers.
• I look forward to working with some of you!

183. Thank You
• Questions?
• For additional information: omarques@fau.edu