Building a Native Camera Access Library - Part I - Transcript.pdf

Building Native Camera Access - Part I
This module covers low level API's and some iOS/Android specific details. I don't aim to teach Objetive-C or some core concepts of Android development. You don't
necessarily need to know these things but they would help.

It’s important to notice that the code in this module is fresh and incomplete. It's here as a porting reference. Since this is pre-alpha level code it should be treated as
such!

Introduction
© Codename One 2017 all rights reserved
✦Low level camera access has been a long time “wish
list” item
✦First Step: Survey Existing Solutions
✦Existing platforms don’t have something close
✦Camera Kit Android stood out as the best/simplest
option 
https://github.com/CameraKit/camerakit-android/
I've discussed low level native interfaces & 3rd party library integrations before. But I never did it for an API as complex as the camera API. Arguably the maps support is
harder so developers could have used it as a reference.  
My chief motivation in doing this module is building the cn1lib itself. Low level camera access has been a wish list of ours since we introduced z-ordering of components.
As such the tutorial is an "afterthought".

The first step of implementing a cn1lib is looking for available API's on both sides and sometimes surveying the work done by other cross platform tools for inspiration.

In this case other tools don't oﬀer something as comprehensive so I didn't spend too much time on that. Android's native camera API is a mess and at some point
Google just threw the whole thing out and replaced it with Camera2. If you want to support older devices you need to support both API's and there's a lot of nuance with
permissions etc.

I didn't want to use the native Android API "as is" and started checking for options. Two open source camera API's stood out andCamera Kit Android looked like the best
option.

Process
✦I went with the Android API “As Is”
✦Camera Kit Android already did the hard work of “design”
✦I started with an app that uses native interfaces, not a
cn1lib
✦API is mostly based on CameraView: 
https://github.com/CameraKit/camerakit-android/blob/
master/camerakit-core/src/main/java/com/wonderkiln/
camerakit/CameraView.java
Normally, this is the place where I would go over the Android API then go over the iOS API and try to find common ground. Instead I chose to skip that and just write the
Android implementation then think about the iOS implementation after the fact.

My reasoning was that Camera Kit already spent a lot of time designing an abstract API. It makes sense to follow their lead. I'm not a camera expert but they already
dealt with user requirements and design issues. This would make the Android implementation trivial and if something doesn't work on iOS (or elsewhere) we can create
special cases for that.

I didn't build a cn1lib. I started with an app that uses native interfaces. Apps I can debug unlike a cn1lib. Once the app was done I just copied the sources into a new
cn1lib project and created that.

So without further ado lets dive into the portable portion of the code. As I mentioned before I decided to model the API based on the Android Camera Kit API. Which is
mostly exposed in the CameraView class you can see in the camera view project

public interface CameraNativeAccess extends NativeInterface {
PeerComponent getView();
boolean isStarted();
void start();
void stop();
float getVerticalViewingAngle();
float getHorizontalViewingAngle();
int getFacing();
boolean isFacingFront();
boolean isFacingBack();
void setFacing(int facing);
void setFlash(int flash);
int toggleFacing();
int toggleFlash();
int getFlash();
void setFocus(int focus);
void setMethod(int method);
void setPinchToZoom(boolean zoom);
void setZoom(float zoom);
void setPermissions(int permissions);
void setVideoQuality(int videoQuality);
void setVideoBitRate(int videoBitRate);
void setLockVideoAspectRatio(boolean lockVideoAspectRatio);
CameraNativeAccess
CameraView is a big class with a lot of UI code so I wanted to distill the API calls within. I searched within the class for public methods and isolated them. I then created
a new native interface within com.codename1.camerakit.impl. The one method that's a special case that didn't exist in the original is PeerComponent getView(). This is a
method that returns the preview for viewing the camera, in the native code it's the actual CameraView class but in our case the native interface hides that.

This method returns the CameraView in the native code

void start();
void stop();
int getFacing();
int toggleFacing();
int toggleFlash();
int getFlash();
CameraNativeAccess
CameraView had a method getCameraProperties() which returned an object called CameraProperties. That object contained two fields horizontal/vertical viewing angles
which I just mapped directly

void start();
void stop();
int getFacing();
int toggleFacing();
int toggleFlash();
int getFlash();
CameraNativeAccess
In retrospect some of these API's like toggleFacing/Flash & isFacingBack/Front should have been implemented in the Java side

void start();
void stop();
int getFacing();
int toggleFacing();
int toggleFlash();
int getFlash();
CameraNativeAccess
The API already used ints for constants so this was pretty easy to translate

int toggleFacing();
int toggleFlash();
int getFlash();
void setJpegQuality(int jpegQuality);
void setCropOutput(boolean cropOutput);
void captureImage();
void captureVideo();
void captureVideoFile(String videoFile);
void stopVideo();
int getPreviewWidth();
int getPreviewHeight();
int getCaptureWidth();
int getCaptureHeight();
}
CameraNativeAccess
These two rely on event listener code, I'll elaborate more on that soon

int toggleFacing();
int toggleFlash();
int getFlash();
void setJpegQuality(int jpegQuality);
void setCropOutput(boolean cropOutput);
void captureImage();
void captureVideo();
void captureVideoFile(String videoFile);
void stopVideo();
int getPreviewWidth();
int getPreviewHeight();
int getCaptureWidth();
int getCaptureHeight();
}
CameraNativeAccess
These were originally preview & capture size. They returned a size object which was just width & height in a class. As you can see the translation was mostly easy and
other than a few methods that returned objects everything was really smooth.

public boolean setTextDetectionListener(
CameraKitEventCallback<CameraKitTextDetect> callback);
public void captureImage(CameraKitEventCallback<CameraKitImage> callback);
public void captureVideo(CameraKitEventCallback<CameraKitVideo> callback);
public void captureVideo(File videoFile,
CameraKitEventCallback<CameraKitVideo> callback);
public void addCameraKitListener(
CameraKitEventListener CameraKitEventListener);
public void bindCameraKitListener(Object object);
Missing from the Native Interface
I did gloss over some API's I chose not to implement within the native interface. You will notice all of these methods have one thing in common: callbacks. We can't pass
a callback into the native interface so all of these methods can't be implemented in the native interface.

We can implement them in the high level abstraction though and I'll get to that soon.

public interface Constants {
public static final int PERMISSION_REQUEST_CAMERA = 16;
public static final int FACING_BACK = 0;
public static final int FACING_FRONT = 1;
public static final int FLASH_OFF = 0;
public static final int FLASH_ON = 1;
public static final int FLASH_AUTO = 2;
public static final int FLASH_TORCH = 3;
public static final int FOCUS_OFF = 0;
public static final int FOCUS_CONTINUOUS = 1;
public static final int FOCUS_TAP = 2;
public static final int FOCUS_TAP_WITH_MARKER = 3;
public static final int METHOD_STANDARD = 0;
public static final int METHOD_STILL = 1;
public static final int PERMISSIONS_STRICT = 0;
public static final int PERMISSIONS_LAZY = 1;
public static final int PERMISSIONS_PICTURE = 2;
public static final int VIDEO_QUALITY_480P = 0;
public static final int VIDEO_QUALITY_HIGHEST = 4;
public static final int VIDEO_QUALITY_LOWEST = 5;
public static final int VIDEO_QUALITY_QVGA = 6;
}
Constants
But before we go there I did mention the constants from the native API. They are implemented in the CameraKit class in the native project. I had to implement our own
constants which I did using an interface.

You'll notice that this is copied from the Android version and so values returned from there should automatically work for us on Android which is pretty cool.

High Level Abstraction
✦Never expose a native interface to a user
✦Always wrap the native interface in a regular class
✦This lets you change the native API
Never expose a native interface to a user. It's a recipe for disaster as things might not work as expected.

Always wrap the native interface in a regular class as that allows for fine grained control.

It lets you change the native API and move forward while keeping compatibility and simplicity. So the next order of business was the decision of how to encapsulate this
functionality.

public class CameraKit implements Constants {
private CameraNativeAccess cma;
private static CameraKit instance;
private ArrayList<CameraListener> listeners = new ArrayList<>();
static boolean thisIsForIOS;
private static void initBuildHint() {
Map<String, String> buildHints = Display.getInstance().
getProjectBuildHints();
if(!buildHints.containsKey("ios.NSCameraUsageDescription")) {
Display.getInstance().setProjectBuildHint(
"ios.NSCameraUsageDescription",
"We need camera access to grab pictures and videos");
}
}
public static CameraKit create() {
if(instance == null) {
if(isSimulator()) {
initBuildHint();
CameraKit
For that I created the CameraKit class which abstracts this native interface.

We implement the Constants interface so we'll have easy access to them

}
}
if(isSimulator()) {
initBuildHint();
CameraKit
The native interface instance is shared between the methods of the class

}
}
if(isSimulator()) {
initBuildHint();
CameraKit
This is a singleton. It's not how the native API works but it's much simpler and we don't need anything better

}
}
if(isSimulator()) {
initBuildHint();
CameraKit
We use this list to send camera events to the user of the API, I'll discuss that further soon

}
}
if(isSimulator()) {
initBuildHint();
CameraKit
iOS strips out unused code which can cause callbacks to fail, we need to trick the iOS VM into thinking that some code will execute

}
}
if(isSimulator()) {
initBuildHint();
CameraKit
iOS needs a build hint, we can just inject it into the build hint list if the user didn't declare it

}
}
if(isSimulator()) {
initBuildHint();
CameraKit
We review the build hints, if the ios.NSCameraUsageDescription isn't declared we add it. Notice that this will work only on the simulator

}
}
if(isSimulator()) {
initBuildHint();
CameraKit
I called this create() even thought it's more like a getInstance() method

if(isSimulator()) {
initBuildHint();
if(thisIsForIOS) {
CameraCallbacks.onError(null, null, null);
CameraCallbacks.onImage(null);
CameraCallbacks.onVideo(null);
}
return null;
}
instance = new CameraKit();
instance.cma =
NativeLookup.create(CameraNativeAccess.class);
if(instance.cma != null && instance.cma.isSupported()) {
return instance;
}
instance = null;
CameraKit
While the camera kit won't work on the simulator I can still bind functionality to it. In this case we do two things in the simulator

if(isSimulator()) {
initBuildHint();
if(thisIsForIOS) {
}
return null;
}
instance.cma =
return instance;
}
instance = null;
CameraKit
We register the required build hint if it isn't there already

if(isSimulator()) {
initBuildHint();
if(thisIsForIOS) {
}
return null;
}
instance.cma =
return instance;
}
instance = null;
CameraKit
We invoke all the callback methods. Notice that this code will never execute but our iOS VM can't detect that because thisIsForIOS isn't private. Without these calls the
iOS VM would strip out these methods and the C code that invokes them wouldn't compile

initBuildHint();
if(thisIsForIOS) {
}
return null;
}
instance.cma =
return instance;
}
instance = null;
}
return instance;
}
CameraKit
If the native interface is supported on this device then we can return the instance of this class otherwise we return null. There is a lot to digest here. Luckily this was the
hardest part. The rest is verbose but it's far simpler than this…

public PeerComponent getView() {
return cma.getView();
}
public boolean isStarted() {
return cma.isStarted();
}
public void start() {
cma.start();
}
public void stop() {
cma.stop();
}
CameraKit
Most of the class looks like this we delegate the calls to the native interface to keep that code hidden. Nothing interesting. There are a few interesting methods within the
class so I'll address them here and skip these boring calls.

public int toggleFacing() {
if(getFacing() == FACING_BACK) {
setFacing(FACING_FRONT);
return FACING_FRONT;
}
setFacing(FACING_BACK);
return FACING_BACK;
}
public int toggleFlash() {
switch (getFlash()) {
case FLASH_OFF:
setFlash(FLASH_ON);
break;
case FLASH_ON:
setFlash(FLASH_AUTO);
break;
case FLASH_AUTO:
case FLASH_TORCH:
setFlash(FLASH_OFF);
break;
}
return getFlash();
}
CameraKit
toggleFacing is in the native interface but instead of implementing it in iOS I chose to implement it in the CameraKit and ignore the native interface

public int toggleFacing() {
if(getFacing() == FACING_BACK) {
setFacing(FACING_FRONT);
return FACING_FRONT;
}
setFacing(FACING_BACK);
return FACING_BACK;
}
public int toggleFlash() {
switch (getFlash()) {
case FLASH_OFF:
setFlash(FLASH_ON);
break;
case FLASH_ON:
setFlash(FLASH_AUTO);
break;
case FLASH_AUTO:
case FLASH_TORCH:
setFlash(FLASH_OFF);
break;
}
return getFlash();
}
CameraKit
The same is true for toggleFlash. I'm sure I could do this for several other methods and simplify the native interface layer

public void addCameraListener(CameraListener listener) { listeners.add(listener); }
public void removeCameraListener(CameraListener listener) { listeners.remove(listener); }
public void fireCameraErrorEvent(final CameraEvent ev) {
if(!isEdt()) {
callSerially(() -> fireCameraErrorEvent(ev));
return;
}
for(CameraListener l : listeners) {
l.onError(ev);
}
}
public void fireCameraImageEvent(final CameraEvent ev) {
if(!isEdt()) {
callSerially(() -> fireCameraImageEvent(ev));
return;
}
for(CameraListener l : listeners) { l.onImage(ev); }
}
public void fireCameraVideoEvent(final CameraEvent ev) {
if(!isEdt()) {
callSerially(() -> fireCameraVideoEvent(ev));
return;
}
for(CameraListener l : listeners) { l.onVideo(ev); }
}
CameraKit
The add/remove listener code just modify the list of listeners. Notice I trimmed the code a bit so it would all fit in one slide…

if(!isEdt()) {
return;
}
l.onError(ev);
}
}
if(!isEdt()) {
return;
}
}
if(!isEdt()) {
return;
}
}
CameraKit
The fire methods are invoked internally to send events to listeners

if(!isEdt()) {
return;
}
l.onError(ev);
}
}
if(!isEdt()) {
return;
}
}
if(!isEdt()) {
return;
}
}
CameraKit
Since the native code invokes them they might not be on the EDT and so they recurse via callSerially to move the context into the EDT

if(!isEdt()) {
return;
}
l.onError(ev);
}
}
if(!isEdt()) {
return;
}
}
if(!isEdt()) {
return;
}
}
CameraKit
The event dispatch logic is identical for all the fire methods it's just a loop over the listeners & a method invocation

if(!isEdt()) {
return;
}
l.onError(ev);
}
}
if(!isEdt()) {
return;
}
}
if(!isEdt()) {
return;
}
}
CameraKit
The rest of the listeners follow the same semantic for updating a video result or an image result. So who invokes the fire methods?

public class CameraCallbacks {
public static void onError(String type, String message,
String exceptionMessage) {
CameraKit ck = CameraKit.create();
if(ck != null) {
ck.fireCameraErrorEvent(
new CameraEvent(type, message, exceptionMessage));
}
}
public static void onImage(byte[] jpeg) {
if(ck != null) {
ck.fireCameraImageEvent(new CameraEvent(jpeg));
}
}
public static void onVideo(String file) {
if(ck != null) {
ck.fireCameraVideoEvent(new CameraEvent(file));
}
}
}
CameraCallbacks
I deprecated this and the native interface so developers won't use them by mistake. This class is for internal usage only

if(ck != null) {
}
}
if(ck != null) {
}
}
if(ck != null) {
}
}
}
CameraCallbacks
All 3 events are handled in the same way through the fire methods

if(ck != null) {
}
}
if(ck != null) {
}
}
if(ck != null) {
}
}
}
CameraCallbacks
Notice the diﬀerent types of arguments to the constructors, we'll discuss the complexities of them in the iOS port. The final two missing pieces are the listener interface
and event class. Both should be pretty obvious so I won't list them here. You can check out the project source code if you are curious.

Building a Native Camera Access Library - Part I - Transcript.pdf

Recommended

Recommended

More Related Content

Similar to Building a Native Camera Access Library - Part I - Transcript.pdf

Similar to Building a Native Camera Access Library - Part I - Transcript.pdf (20)

More from ShaiAlmog1

More from ShaiAlmog1 (20)

Recently uploaded

Recently uploaded (20)

Building a Native Camera Access Library - Part I - Transcript.pdf