Rock steady video stabilization on your on-the-move video devices.

Copyright © 2016 FotoNation 1
DIGITAL GIMBAL
Rock-Steady Video Stabilization Without Extra Weight!
Dr. Petronel Bigioi
May 3, 2016

IMAGING
Wide Expertise
Strong Innovation
Top Quality
SOFTWARE
Always Real Time
Platform Agnostic
Full Flexibility
HARDWARE
Proven
Low Power
High Performance
QA
Advanced Data Acquisition
Multi-million Image Database
Testing Automation Tools
FotoNation in a Nutshell

FotoNation in Numbers
550+ 1000+ 60% 2.5+ 450+
imaging patents man years of experience high end smartphone penetration billionunits shipped milliondevices / year
understand ● enhance ● accelerate

DIGITAL GIMBAL
A state-of-the-art, energy-efficient image stabilization solution for
high frequency vibration by FotoNation

Problem Statement
High quality videos 4K @ 60 FPS are accepted as
norm in cameras on the fly and move
Low frequency motion is not the only problem
anymore
HIGH and VERY HIGH frequency motion due to
motors and activities need to be addressed including
motions with horizon lock

Mechanical Gimbal Solution — Current State- of-the-Art
to Attenuate Motors Vibration Effects
ADVANTAGES
DISADVANTAGES
WHAT IS IT?
• Mechanical assembly built to counteract vibration effects
• Uses gyros in close loop to control motors
• Have large range of movement
• Cost!!!
• Large & inconvenient
• Heavy
• Consume power
• Reduce flight time due to extra weight
• Slow reaction time

Digital Gimbal Solution — FotoNation’s technology to
deal with vibrations (low and high frequency)
ADVANTAGES
DISADVANTAGES
WHAT IS IT?
• Algorithms removing the need for mechanical assembly; uses gyro to measure high frequency
vibration and digital image processing for frame to frame registration to reverse the effects in the
image/frames
• Low cost
• No extra weight
• Low power
• Ultra fast reaction time
• Advanced image stabilisation
• Rolling shutter correction
• Limited range of movement

Image Stabilization Requirements
Image stabilization has always been a subject of great interest especially when
associated with camera modules on the move; today’s solutions are influenced by
system requirements (cost) as well as jitter specific problems (usage).
PerformanceSystem requirements
good sampling quality
stable frame rate
small error estimations
low power consumption
future proof (2K, 4K, 8K)
roll (x-y rotation)
panning
out of plane rotations
rolling shutter distortions
large moving objects

The rolling Shutter Problem
Hand jitter vibration patterns Drone/motorbike jitter vibration patterns
Rolling shutter effects — caused by slow camera
motion during line by line exposure and frame
readout
Rolling shutter effects — caused by rapid camera
motion during line by line exposure and frame
readout
Have low frequency (up to 15Hz) Have high frequency (hundredsof Hz)

ISP
Inertial
Sensor
Motion
Filtering
Camera
projection
model
Motion
Correction
Image
Corrected
Image
Correction
Grid
Motion
Estimation
Image
Sensor
Legend:
Memory Buffer
Digital Gimbal Components
HW Module
App Processor & DDR
256 independent motion
vectors;
IMU samples
acquisition and sync;
sensor fusion
(IMU + frame to frame)
Corrects lens distortion,
roll, pan & complex
rolling shutter in 1 pass
Adaptive motion filtering
& rolling
shutter effects estimation
System Architecture and Workflow

Motion Estimation — Synchronization
• Accurate synchronization of inertial and video data is essential
• Hardware core used to read inertial data (IMU) and assign precise timestamps, with same
time source for frames and IMU samples
• Two possible implementations
• Pure HW implementation — ensures minimum latency and highest timing
accuracy, low power consumption
• Hybrid implementation — more flexible, but higher power. Requires high
sampling frequency to minimize motion measurement errors

SOF
Interrupt
SOF
InterruptSample with
FSYNC marker
Sample with
FSYNC marker
unknown time
periods
Maximum unknown period is equal to IMU sample interval. Very inaccurate for low sampling frequencies
Hybrid Synchronization Timing Accuracy Limit

Bias — a constant value added to the measured signal
Motion Filtering — IMU calibration

Sampled IMU data needs to be filtered (4x rule) in order to avoid aliasing
Motion Filtering — Data Filtering

Use sensor fusion to track camera
orientation over time
Using gyroscope alone to track camera
orientation leads to error accumulation
Acceleration and magnetic field
lack short-term accuracy but
provide good long-term reference
frame
Gyroscope provides short-term
accurate changes in the
camera orientation
Motion Filtering — Sensor Fusion

• The real camera orientation is tracked using quaternion arithmetic
• The required correction is the difference between the real and the estimated trajectory
• Our motion filters adapts to the camera motion in order to produce naturally smooth
experience and react fast to the intentional motion
Motion Filtering

• Stabilisation margin is limited by the sensor size. Sudden camera movements can cause optimal
correction to exceed available correction margin. Excessive motion needs to be limited to the
available correction margin
• Limits imposed on 2D projection must control 3D rotation represented as 4D quaternion — difficult
task. Problem gets even more complicated for highly distorted rectilinear lenses (left) or fisheye
lenses (right)
Motion Filtering — Motion Limitation

time
reference line
• Orientation of the camera in the first line is the
reference point
• Each consecutive line is transformed to counteract
camera movement up to the given line
• Inertial samples and the video frames must be precisely
synchronized
• Camera projection must be accurately modeled
Motion Correction — Rolling Shutter Correction

Motion Correction — Rolling Shutter Correction

Motion Correction – Rolling Shutter Correction

Camera frame
reference vector
Gravity vector
from
accelerometer
Correction
amount
After correction
vectors are aligned
The horizon locking feature keeps the horizon line in the middle of the
frame and horizontal regardless of the drone orientation.
Motion Correction — Horizon Locking Option

Motion Correction With “Follow Me” Feature / Owner
Tracking

FUTURE
PROOF
FAST TO
DEPLOY
EFFICIENTLY
DESIGNED
In a seamless way that is simultaneously
A DEDICATED
HARDWARE — IPU
Low gate count implementation
to offload heavy image and video
computation. Most difficult tasks
completed locally and then passing
processed data to host CPU
How Do We Deliver This?

OBJECT DETECTION ENGINE
~ 1M gates, 240 kB Sram, 54 mW
Multi-Core CPU
GPU DSP ISP
3G/4G Baseband
Memory & I/OsIPU
Video Encoder/Decoder
MOTION PROCESSING ENGINE
~ 125K gates, 9.8 kB Sram, 2 mW
DISTORTION CORRECTION ENGINE
~ 390K gates, 48 kB Sram, 18 mW
4K@60FPS, 28nm technology
Typical AP
IPU components overview

• Precise lens modeling allows for lens distortion correction and re-projection.
• Any type of correction is possible (rectilinear to perspective, fisheye to perspective,
distorted fisheye to perfect fisheye, fisheye to cylindrical, arbitrary input projection to
arbitrary output projection and freeform image warping).
• All distortion corrections are done at the same time with stabilisation and high frequency
rolling shutter correction (single pass).
• Adding distortion correction has no impact on system performance.
• Frames are resampled only once to ensure maximum image quality.
• Lowest possible power consumption (18mW for 4k@60fps correction in 28n technology).
Motion Correction — Uses FotoNation’s DCE
(Distortion Correction Engine)

DCE GPU
Designed to work with
rectangular texture mapping for
2D image transformations
Native support for
bicubic resampling
Scalable with increasing input
size and frame rata
Small gate count
Low power consumption
Low bandwidth thanks to specific
cache design
Designed to support generic
triangular texture mapping for
3D scenes rendering
Bi-cubic resampling requires
extra processing power & time
Designed to work at display
resolution
Large gate count
High power consumption
High bandwith
Motion Correction — Done DCE (DistortionCorrection Engine)
Part of FotoNation’s IPU (Image Processing Unit)

INPUT OUTPUT
Correction — DCE example
Corrected by FotoNation DCE

Correction — DCE example
INPUT OUTPUT
Corrected by FotoNation DCE

FotoNation EIS / High Frequency Stabilization

• Digital Gimbal components (synchronization,motionfilteringand correction)
are part of FotoNation’s IPU (Image Processing Unit) for optimum low power
high performance implementation.
• IPU incorporates additional units to enable more features such as owner
recognitionand trackingcombined with image stabilization(e.g.high
performance object detectionand tracking,high performance face detectionand
trackingand face recognition).
Conclusion

Thank You

FotoNation EIS / a no trade-off user experience

Rock steady video stabilization on your on-the-move video devices.

Recommended

Recommended

More Related Content

Similar to Rock steady video stabilization on your on-the-move video devices.

Similar to Rock steady video stabilization on your on-the-move video devices. (20)

Recently uploaded

Recently uploaded (20)

Rock steady video stabilization on your on-the-move video devices.