Tomer kimhi mobmodcon-nov2015-integrating new camera hardware

MANTIS VISION PROPRIETARY & CONFIDENTIALMANTIS VISION PROPRIETARY & CONFIDENTIAL
Integrating new Camera* hardware
with mobile OEM

How does Android look like in 3D ?
Mantis and Android
What’s so special about 3D cameras?
What about the future to come?

So how does
Android look
like in 3D ?

Mantis Vision developed state-of-the-art 3D enabling technology (MV4DTM), with unprecedented
accuracy (static and dynamic scenes), for the use in mobile devices and 3D scanners.
Our Mission: To become the platform of choice for accurate capturing of depth and motion in a
way that best portrays reality.
Company Introduction

MANTIS VISION PROPRIETARY & CONFIDENTIAL
TIME
PERFORMACE
High Quality 3D Depth
Sensing is a NEW Technology
S-Curve
Imaging Disruptive Technologies
Real High Quality 3D is a disruptive technology!
1963
2015
1996
2006
3DHTC One M8

Our technology - Triangulation
Triangulation is the process of determining the location of a point by measuring angle to it from known points at
either end of a fixed baseline, rather than measuring distances to the point directly.
The point can then be fixed as the third point of a triangle with one known side and two known angles.
PRINCIPLES OF OPERATION
 Baseline
 Correspondence
 Localization

MANTIS VISION PROPRIETARY & CONFIDENTIALMANTIS VISION PROPRIETARY & CONFIDENTIALMANTIS VISION PROPRIETARY & CONFIDENTIAL
Our Technology – Epipolar Coding
Our unique code (patented-US8208719) allows a smaller footprint to
uniquely identify many more points than in standard methods (i.e.: any
desired point-resolution could be provided by using a more robust
coded-pattern due to a “smaller footprint”)
This unique technology provide a higher resolution & accuracy for lower
MOS (minimal object size)

Endless Integration Options

Instant User Generated
High-Quality 3D Content

Parametric 3D Imaging & Dynamic registration

Enhanced 3D Communication

Mantis and Android

Android Dev in MV

So what’s so special
about 3D camera?
(That we haven’t work with till now…)

• Structured light flash projector
• Hardware projector driver circuit
• Camera modules
• Processing algorithms
NIR Camera
(C)
REAR COLOR
CAMERA
STROBE
LIGHT
FLASH
PROJECTOR (F)
System Overview
System Diagram

System Overview
System Diagram
• Structured light flash projector
• Hardware projector driver circuit
• Camera modules
• Processing algorithms

New controlled target – “Depth Flash Projector”
Raw data (monochrome, no ISP, full bit-depth)
Unique sensors – RGB-IR
Simultaneous Dual Camera (2 surfaces at once)
Low level sensors tweaking (e.g. Manual exposure, PLL, unique FPS, etc…)
Timing!
“Unordinary” Tasks

Device Abstraction
MV4D SDK API
DataFlow
OSDepth Camera Driver
Camera Control
AGCDecoderRecorder
Projector Control
Mantis Vision (MV) SDK
Projector DriverColor Camera Driver

MV SDK requires for the Projector Driver operating system software support.
Driver interface should provide access to the laser driver hardware through I2C protocol
and should comply to the standard operating system permissions mechanism.
Design shall include pre implemented I2C projector communication service.
 In order to control the operating system need to control the Laser Drive via I2C.
 This can* be done by adding I2C kernel module and a HAL service that will provide access via
the standard android permissions mechanism for applications.
 Service should be formed in order to obtain connection by the SDK.
Projector Driver Interface

Experience with different OEMs:
OEM System Service
 In most cases it is the OEM that implement System Service according to given request.
 Sometimes might come in a different forms of APIs – effects the generic code.
 OEM/Vendor might not expose everything…
Projector Device Driver
 Might be good only as root adb
 Might be changed under different permissions
Projector Driver Interface – Cont.

Connecting to Color and IR camera is implemented through Android Standard HAL.
For the IR camera, RDI callback being used*.
CameraListener will be the interface to connect to both cameras (connect(x) will connect
to the IR camera, connect(y) will connect to the Color camera). Both camera instances
must be functional at the same time.
The callback need to expose real frame timestamp. (This is currently unsolved…)
Camera S/W Interface

Experience with different OEMs:
Different platforms required different interfaces:
 CameraListner, CpuConsumerListner, ProCamera
Some requires very specific headers to be
included in compilation, and might be vulnerable
to OEM changes
Camera S/W Interface – Cont.

APPMV SDK
Camera HAL v3 posed new challenges
On one hand new advanced HAL included extended features never (or partially)
supported before (e.g. RAW), but simple callback connection were changed or dropped.
One of our current solutions -
Camera S/W Interface – Cont.
JAVA SDK (.jar)
NATIVE SDK (.so)
JAVA
c/c++
jni

Android HAL bridges the gap between hardware and software.
It allows the Android application/framework to communicate with the hardware specific
device drivers.
For Android application, HAL provides API’s through which service can place a request to
device.
And HAL uses functions provided by the lower layer Linux system to service the request
from the android framework.
HAL is a c/c++ layer which is a vendor specific implementation.
About HAL

There are two different types of Android HAL architecture:
 First one is the legacy Android HAL which looks similar to library implementation.
 Second one gives the complete abstraction and control over the device vendor.
Android HAL implementation looks similar to a Linux device driver.
Most of the Vendor specific implementations can be done in Android HAL rather than the
driver. So that license difference between the driver (Open source license GPL) and the
HAL (Apache License) will give more level of abstraction to vendor.
HAL has structures which specify the HAL type, Module type, version detail, and a set of
function callbacks (methods) which are registered to android framework layer.
 For example if a service from android framework needs data from a sensor device, the service
will make a call to corresponding function in sensor HAL which in turn will talks directly with
Linux device driver.
 The driver will trigger the sensor driver and deliver the data back to the HAL so it can be
passed back to Android application.
The HAL module is compiled into a shared object that is dynamically loaded at runtime
when needed.
HAL Architecture

Device Driver
 Linux has predefined system calls which can be used by the user space application to perform
the operation on hardware.
Android kernel Patches
 There are limitation in the Linux kernel, that required patches to make it more powerful for
android.
 Some patches can be found in security, networking, IPC, Power management.
Android Device Drivers
 Device drivers written for Android hardware platforms can not get merged into the main kernel
tree because they have dependencies on code that only lives in Google’s kernel tree, causing it
to fail to build in the kernel.org tree.
 Linux device drivers will also work on android but it will lack some functionality which is specific
to android.
Android Specific Device Drivers
 Device drivers related to android which are integrated with Linux staging folder (e.g. Low
Memory Killer)
 Some of the device drivers which are specific to android are not merged with linux (e.g. Binder)
Device Drivers

And what about the
future to come?

developing a unified service that should run on any targeted Arm platform (Projector
driver service)
Allow RAW camera processing without being constrained to use color ISP (RDI).
Been able to access the hardware correctly without being limited by OEM open API
(Google Tango API not allowing us to trigger our laser and get RAW data).
Ability to secure CPUGPU affinity as will be required when other developers will run their
code as well in parallel.
Expose some configuration parameters of the depth decoding process without "opening"
the complete parameters to everybody (our use of encrypted config + the additional cfg
parameters that are not encrypted).
How to synchronize 3D camera with other host sensors (e.g. IMU & Temperature
sensors)
Challenges

And if you are having a good
understanding how to cope with these
kind of challenges –
we are looking for you...

Thank You
Tomer Kimhi, VP R&D
TomerK@mantis-vision.com
(Time for Demos…)

Tomer kimhi mobmodcon-nov2015-integrating new camera hardware

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Tomer kimhi mobmodcon-nov2015-integrating new camera hardware

Similar to Tomer kimhi mobmodcon-nov2015-integrating new camera hardware (20)

Recently uploaded

Recently uploaded (20)

Tomer kimhi mobmodcon-nov2015-integrating new camera hardware