Implementing Computational Camera

Implementing
Computational Camera
Michael Mangan
Product Manager, Senior
Qualcomm Technologies, Inc.

Computational camera:
What is it?

Computational camera
Utilizes both the ISP HW and Qualcomm® Snapdragon™ processing to create new and exciting
camera effects!
• Example: Image segmentation and filtering an image allow for the digital re-creation of a
sophisticated portraiture effect, called the “Bokeh” effect.
Sensor1Sensor2
ISP 1
ISP 2
Qualcomm®
KRAIT CPU
Qualcomm®
ADRENO™ GPU
Qualcomm®
HEXAGON™ DSP
Qualcomm Snapdragon, Qualcomm Adreno, Qualcomm Krait and Qualcomm Hexagon are products of Qualcomm Technologies, Inc.

Depth calculation designed to enable shoot once, focus later
Sensor1Sensor2
ISP 1
ISP 2
KRAIT CPU
ADRENO GPU
HEXAGON DSP

Depth calculation designed to enable range finding for instant auto-focus
Sensor1Sensor2
ISP 1
ISP 2

Computational
photography examples

Qualcomm® UbiFocus™
camera feature
Qualcomm UbiFocus is a product of Qualcomm Technologies, Inc.

UbiFocus
UbiFocus
Description
UbiFocus is a camera feature that generates an all in focus image from a mobile camera with a limited depth of field. Additionally, this feature allows
users to selectively refocus their image in post-processing.
KPI
1. No fewer than 5 images.
2. For 13M images
1. Take no longer than 1.0s to capture images. 0.7s with further optimization
2. Take no longer than 0.5s after the capture to process input images generate an all in focus image.
OEM feature
deliverable
1. UbiFocus available to licensees provided as part of the MSM8974 LA software release as optional software.
2. UbiFocus provided in binary form, and called by OEM enabled camera applications via an API interface defined by Qualcomm Technologies.
3. Use-Case
a) All-In-Focus Mode:
a) When ‘All-in-Focus’ mode for UbiFocus is selected, the snapshot operation will result in an ‘All-in-Focus’ JPEG image from the
mobile camera.
b) Dynamic Focus Mode:
a) User will need to select ‘Dynamic Focus Mode’ option within Camera App.
b) UbiFocus will allow the end-user to touch an object in a scene and bring it in focus.
c) The user can either share the refocus-able object on the web, or;
d) The user can select the desired focus point and save the image as a final JPEG image.

UbiFocus
• Mobile camera’s have a limited Depth-Of-Field
Out of focus
In focus Out of focus
Input image 1 Input image 2
In focus

UbiFocus
• From a Limited Depth-of-field Camera
• UbiFocus generates an all-in-focus image
• Refocus at a desired location as post processing
Out of focus Out of focus
Input image 1 Input image 2 UbiFocus
In focus
In focus
In focus

Video: UbiFocus

UbiFocus
Use case
• Refocus at a user-desired location
• Share images that can be re-focused (the
recipient does not need UbiFocus to perform
dynamic re-focus)
• Generates an all-in-focus image
Benefits
• Eliminates the need for auto-focus
• Focus-on-demand “live” images
• Same effect as Lytro and Pelican Imaging, but with
much higher resolution and no additional hardware
• Preserves resolution (unlike Lytro)
• Integrated with Snapdragon camera pipeline

OptiZoom camera
feature
OptiZoom is a product of Qualcomm Technologies, Inc.

OptiZoom
OptiZoom
Description
OptiZoom is a multi-frame super-resolution feature that enhances the resolution by 2x on each side (i.e., 4x number of pixels)
that mimics an optical zoom. OptiZoom uses a Qualcomm Technologies-designed multi-frame super-resolution technique to
allow an end-user to zoom into an image beyond normal digital zoom levels. OptiZoom designed to increase image resolution
and also reduce noise.
OptiZoom, can be used to super-resolve only a region of interest or it can be used to super-resolve the entire image.
KPI
1. Eight images of 12MP shall be taken in less than 0.6s.
2. Processing time for converting a 4MP image into a 16MP image is ~700ms on 8974. Processing time varies almost linearly
wrt the number of pixels.
OEM feature
deliverable
1. OptiZoom, licensed to OEMs as optional software, as part of the MSM 8974 software release.
2. OptiZoom provided in binary form, and called by OEM enabled camera applications via an API interface defined by
Qualcomm Technologies.
3. Use Case:
a) The end-user zooms into the area of interest.
b) The end-user initiates OptiZoom on the area of interest.
c) The OptiZoom algorithm enhances the resolution of the area of interest.

OptiZoom use cases
Option 1
When user presses the “take image” button, the
device takes N full-size images.
OptiZoom is invoked with the full
frame
The user can zoom in to any area of
the image without invoking
OptiZoom again
Option 2 User first zooms-in in the preview and specifies
RoI and then N images are taken.
OptiZoom is invoked with the RoI The user gets a 4x image of the RoI.
Option 3
When user presses the “take image” button, the
device takes N full-size images. Then user zooms
in and selects an RoI
OptiZoom is invoked with the RoI The user gets a 4x image of the RoI.
If the user want another region of
the image to be super-resolved,
OptiZoom can be invoked again.

OptiZoom - Disabled

OptiZoom - Enabled

OptiZoom
Disabled Enabled

OptiZoom
Use case
• Increasing the image resolution by 4x number of pixels
• See previous slide for more details
• Useful for allowing users to zoom in on key regions
within an image.
- Read fine text
- See faces clearly
Benefits
• Increases photo resolution and reduces image noise.
• Fast and scalable
• Utilizes Snapdragon Hardware.

Qualcomm® ChromaFlash™
camera feature
Qualcomm Cromaflash is a product of Qualcomm Technologies, Inc.

ChromaFlash
ChromaFlash
Description
ChromaFlash takes two images in quick succession: one with flash enabled and one without flash enabled to produce a beautiful
composite image. Flash images usually suffer from over-exposedness and bluish tint and dark background. Our ChromaFlash
technology overcomes these problems algorithmically and produces and image that preserves color, texture, and brightness.
KPI
1. ChromaFlash Images shall be taken in less than 0.135s on top of flash image capture time to capture 8/13MP input images.
2. ChromaFlash algorithm processing shall take no more than 700ms for two 13MP images on 8974. Processing time varies
almost linearly with the number of pixels.
OEM feature
deliverable
1. ChromaFlash, licensed to OEMs as optional software, as part of the MSM8974 LA software release.
2. ChromaFlash provided in binary form, and called by OEM enabled camera applications via an API interface defined by
Qualcomm Technologies.
3. Use-Case/Default Operation
a) ChromaFlash will engage whenever normal Flash is used, effectively replacing ‘normal’ flash operation.
b) ChromaFlash 2.0 can handle dynamic scenes (i.e., scenes that contain motion) as well as camera motion due to
handshake

No-flash image Flash image ChromaFlash image
Another ChromaFlash example
Note: Projector quality may limit ability to see enhancements visible on laptop monitor.

ChromaFlash
Use case
• Flash that preserves color
• Enhances Low-light imaging
• Enhances Dynamic Range
Benefits
• No extra hardware is required
• Reduces the noise observed in no-flash images taken
in low-light.
• Maintains color correctness lost using the flash.
• Increases the dynamic range

Low-light video
enhancement

Low light vision enhancements
Description
Low-light video enhancement performs two main functions:
- Adaptively brightens regions of the video frame that are too dark. If the scene is sufficiently bright already, Low-light video
enhancement does not interfere.
- Reduces noise
KPI
Processing time is ~7ms on a 1080p frame on 8974. Processing time varies almost linearly wrt the number of pixels.
Tier 1 OEM feature
deliverable
1. Low-light video enhancement, licensed to OEMs as optional software, as part of the MSM8974 LA software release.
2. Low-light video enhancement shall be provided in binary form, and called by OEM enabled camera applications via an API
interface defined by Qualcomm Technologies.
3. Use-Case: Low-light video
Competition
1. Arcsoft NightHawk
2. Morpho

What is Low-light video enhancement?
Low-light video enhancement is an algorithm for adaptively brightening areas of video/images that are
too dark
Attributes
• It works for dark scenes and scenes that are generally bright, but include dark regions
• It is a low-power and low-complexity algorithm. As a result, it can run comfortably on even low-tier chips
@ 30fps for 1080p resolution
• It also includes a fast and efficient noise-reduction to attenuate noise in the areas that were under-exposed
in the original frame

Video: Low light enhancement

Low-light video enhancement comparison
Original Revision-LTM

Low-light video enhancement comparison
Original Low-light video enhancement

Noise reduction
No Temporal NR With Temporal NR

Video: Preserving sharpness

Low-light video enhancement enhancing low light video
Original Low-light video enhancement

Portrait enhancement

Description
A a camera feature that takes a regular portrait image taken with a camera phone and synthesizes shallow depth-of-field (aka Bokeh effect) as if the
image is produced by an DSLR camera.
This feature also brings a static image to life by automatically generating 3D parallax and zoom effect (see the videos in upcoming slides)
The input image does not even have to be taken by a camera equipped with portrait enhancement.
KPI
1. Requires an image with 1-3 people in the scene (the max number of people is adjustable if needed)
2. Processing time on 8974:
a) Initial segmentation: 1.2s
b) Re-segmentation after user edit: 200ms
c) Bokeh effect: 1.1s for 13MP output
d) Parallax effect: 250ms
e) Dolly Zoom effect: 250ms
f) Horizontal motion blur effect: 1.1s for 13MP output
g) Sketch effect: 550ms for 13MP output
h) Fusion effect: 450ms for 13MP output
OEM feature
deliverable
1. Portrait enhancement licensed as optional software as part of the MSM8974 LA software release.
2. Portrait enhancement shall be provided in binary form, and called by OEM enabled camera applications via an API interface defined by Qualcomm
Technologies.
3. Use-Case:
- Synthesize a shallow-depth-of-field automatically even on images that have been taken previously without portrait enhancement
- Generates 3D Parallax and Dolly Zoom effects

• What is it?
− It takes a single image and automatically segments out humans in the image. Then one of several effects
(see following slides) can be applied applied to the background
• Attributes
− portrait enhancement can be applied even to images that are taken previously using an ordinary camera
− If needed, it allows the user to edit the foreground and background by providing small hints

Portrait enhancement vs. DSLR
Camera phone (input image)
Portrait enhancement DSLR w/ expensive lens

Input image

Shallow depth-of-field
(mimics portrait mode of DSLR cameras)

Parallax

Dolly-zoom

Horizontal motion blur

Sketch

Still camera image
enhancement

Still camera image enhancement
Description
Takes a single image and enhances the dynamic range of the still image by automatically brightening regions of the image that
are too dark and remove haze from the image.
It allows the user to move a slider and change the amount of adaptive enhancement applied to the image.
The input image does not even have to be taken with a camera that is equipped with Still camera image enhancement.
KPI
Processing time for 13MP image on 8974 is ~80ms. Processing time varies almost linearly wrt the number of pixels.
OEM feature
deliverable
1. Still camera image enhancement, licensed as optional software, provided as part of the MSM8974 LA software release.
2. Still camera image enhancement shall be provided in binary form, and called by OEM enabled camera applications via an API
interface defined by Qualcomm Technologies. A sample app is included as a reference to guide OEM’s implementation of the
feature
3. Use-Case:
- Enhance the image quality dramatically by adaptively and automatically removing haze and brightening regions of the
image that are too dark

Video: Still camera image
enhancement

General image
segmentation

Image segmentation
• Accurate image segmentation
with user interactive
• Applications include
– Photo editing for re-focusing
– Multi-layer color editing
– Multi-layer image filtering
• Available – October 2014
SegmentationMatting

Video: Image segmentation

Touch to Track
A general object tracker

Touch to Track (T2T) v2.0
Track, detect, and learn
• Lucas-Kanade optical flow
tracking
• Decision Forest and NCC
detection
• Online learning bootstrapping
binary classifier
• Leverages Motion Vector
Information from Venus Video
Core.
Applications
• Camera and video auto focus,
exposure, zoom, etc.
• Robot control with T2T
• Advanced video recording /
snapshot

Video: Touch to Track (T2T) v2.0

Touch to Track – Power optimized performance
• Below are the power consumption in performance-mode* and power-optimized-mode** of Touch to Track on 8974
using the same test sequences
• In performance mode, Touch to Track power consumption is about 200-300% of those in the power-optimized-mode.
The ratio could be higher in full frame detection
382
367
210
395
109
123 113
183
0
50
100
150
200
250
300
350
400
450
video1 video2 video3 video4
mW
performance mode power-optimized mode
* Performance mode (CPU Only) and power-optimized (DSP & CPU) mode selectable
** Power saving achieved by utilizing multiple processors including DSP on 8974
Note: No tracking performance differences between performance and power-optimized modes.

Touch to Track
Use case
• Tracker enhanced 3A algorithms
• Single Camera Picture in Picture in preview and
recording
• Artistic special effects in camera preview and video
recording
Benefits
• Multiple object tracking
- Track 2, up to 4 objects
• Robust tracking algorithm tightly integrated with
camera 3A and auto-zoom algorithms, enhances:
- Auto Exposure
- Auto Focus
- Auto Zoom.
• Allow selected region of image to be displayed in
single-camera PIP (Picture in Picture)
• Tracker is Snapdragon HW accelerated
• Tracking on 1080p @30fps

Object removal &
cloning

Object removal and cloning 2.0 – single frame
This feature is different from multiframe motion based object removal
• Content based on-device image editing
• Object Removal allows the user to select and remove and object
• Cloning allows users to replicate and insert an image
Object removal Object cloning

Object removal and cloning
Use case
Object Removal
• Allows on-device, user selectable, removal of
blemishes within the image.
• Uses advanced background matching algorithm to
match the background.
Object Cloning
• Allows on-device, user selectable, cloning of new
images onto existing photographs.
Benefits
• Low power, high quality, image processing features
• Easily integrated into existing Camera Application to
fit existing User Interface.
• Written using OpenCL to fully utilize Adreno GPU

For more information on Qualcomm, visit us at:
www.qualcomm.com & www.qualcomm.com/blog
©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Qualcomm, Snapdragon, Adreno, and Hexagon are trademarks of Qualcomm Incorporated,
registered in the United States and other countries. Krait, Uplinq, UbiFocus and ChromaFlash are
trademarks of Qualcomm Incorporated. All Qualcomm Incorporated trademarks are used with
permission. Other products and brand names may be trademarks or registered trademarks of
their respective owners.
References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm
Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate
structure, as applicable.
Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of
its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm
Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s engineering,
research and development functions, and substantially all of its product and services businesses,
including its semiconductor business, QCT.
Thank you FOLLOW US ON:

Implementing Computational Camera

Recommended

Recommended

More Related Content

What's hot

What's hot (7)

Similar to Implementing Computational Camera

Similar to Implementing Computational Camera (20)

More from Qualcomm Developer Network

More from Qualcomm Developer Network (20)

Recently uploaded

Recently uploaded (20)

Implementing Computational Camera