Building a Native Camera Access Library - Part V - Transcript.pdf

Building Native Camera Access - Part V
In this final part we’ll finish the iOS portion implementation

-(void*)getView{
return container;
}
-(int)getFlash{
return flash;
}
-(int)getFacing{
return direction;
}
-(BOOL)isFacingFront{
return direction == FACING_FRONT;
}
-(BOOL)isFacingBack{
return direction == FACING_BACK;
}
-(int)getPreviewWidth{
return (int) previewLayer.frame.size.width;
}
-(int)getPreviewHeight{
return (int) previewLayer.frame.size.height;
}
-(BOOL)isSupported{
return YES;
}
com_codename1_camerakit_impl_CameraNativeAccessImpl.m
There are some methods that I've skipped entirely so I won't discuss them here but there are a lot of simple methods like we had in the Android port which just delegate
onwards. The getters are the simplest as they don't even need to enter the iOS thread.

This returns the camera view which should be initialized once start() is invoked

-(void*)getView{
return container;
}
-(int)getFlash{
return flash;
}
-(int)getFacing{
return direction;
}
-(BOOL)isFacingFront{
return direction == FACING_FRONT;
}
-(BOOL)isFacingBack{
return direction == FACING_BACK;
}
-(int)getPreviewWidth{
return (int) previewLayer.frame.size.width;
}
-(int)getPreviewHeight{
return (int) previewLayer.frame.size.height;
}
-(BOOL)isSupported{
return YES;
}
This is a standard method in every native interface. Everything in between is trivial.

-(void)setZoom:(float)param{
if(zoom != param) {
zoom = param;
dispatch_async(dispatch_get_main_queue(), ^{
[self updateZoom];
});
}
}
-(void)setFocus:(int)param{
if(focus != param) {
focus = param;
[self updateFocus];
});
}
}
-(void)setFlash:(int)param{
// same code...
}
-(void)setFacing:(int)param{
// same code...
}
-(void)setVideoQuality:(int)param{
// same code...
}
The setters are a bit more verbose but not by much, they are all practically identical so I'll only focus on the first.

I don't want to call update if the value didn't actually change

-(void)setZoom:(float)param{
if(zoom != param) {
zoom = param;
[self updateZoom];
});
}
}
-(void)setFocus:(int)param{
if(focus != param) {
focus = param;
[self updateFocus];
});
}
}
-(void)setFlash:(int)param{
// same code...
}
-(void)setFacing:(int)param{
// same code...
}
-(void)setVideoQuality:(int)param{
// same code...
}
We update on the main thread by invoking the update methods from before. Notice I use the async call. As a rule of thumb always use the async call unless you MUST
use the sync call. It's faster and has a lower chance of a deadlock. In this case I don't need the action to happen within this millisecond so async will work just fine.

-(float)getVerticalViewingAngle{
__block float fov = 0;
dispatch_sync(dispatch_get_main_queue(), ^{
fov = device.activeFormat.videoFieldOfView / 16.0 * 9;
});
return fov;
}
-(float)getHorizontalViewingAngle{
fov = device.activeFormat.videoFieldOfView;
});
return fov;
}
I also have two special getters that need to run on the event dispatch thread. Normally this wouldn't be a problem but when we need to to return a value that's a bit
challenging.

The __block keyword allows us to mark the field as a field we can change from the lambda expression.

-(float)getVerticalViewingAngle{
fov = device.activeFormat.videoFieldOfView / 16.0 * 9;
});
return fov;
}
-(float)getHorizontalViewingAngle{
fov = device.activeFormat.videoFieldOfView;
});
return fov;
}
Here is a case where we MUST use dispatch_sync. If we used async the return statement would have happened before we assign the value.

Now all that is left is the actual capture methods!

-(void)captureImage{
capturingVideo = NO;
if([AVCapturePhotoOutput class]) {
if(photoOutput == nil) {
photoOutput = [[AVCapturePhotoOutput alloc] init];
}
AVCapturePhotoSettings* settings =
[AVCapturePhotoSettings photoSettings];
[photoOutput capturePhotoWithSettings:settings
delegate:self];
} else {
// ... Code for iOS 9 compatibility
}
});
}
The capture methods are a bit more complicated. Surprisingly capturing a still image is a bit harder than capturing a video. Since the captureImage method is a bit long
I've split it into two like before. This is a similar case of iOS 10 requiring a new API.

All capture methods must run on the native UI thread as they deal with the native UI. Failing to do this leads to weird crashes.

}
delegate:self];
} else {
}
});
}
This is the new iOS 10+ API, if it isn't here we'll execute the iOS 9 compatible code

}
delegate:self];
} else {
}
});
}
Up to this point everything is pretty trivial. Delegate is a special concept in iOS similar to Java's interfaces this means the current class implements the delegate

@interface com_codename1_camerakit_impl_CameraNativeAccessImpl :
NSObject<
AVCapturePhotoCaptureDelegate,
AVCaptureFileOutputRecordingDelegate> {
// everything here is unchanged
}
// everything here is unchanged
@end
com_codename1_camerakit_impl_CameraNativeAccessImpl.h
Before I go to the iOS 9 code within this method lets look at the delegate... In order to implement the delegate we need to make a small change to the header file. The
delegates are declared in a syntax that's reminiscent of the Java generics syntax. I also added the delegate needed for video recording while I'm here already so that's
two delegates.

-(void)captureOutput:(AVCapturePhotoOutput *)captureOutput
didFinishProcessingPhotoSampleBuffer:(CMSampleBufferRef)
photoSampleBuffer
previewPhotoSampleBuffer:(CMSampleBufferRef)previewPhotoSampleBuffer
resolvedSettings:(AVCaptureResolvedPhotoSettings *)resolvedSettings
bracketSettings:(AVCaptureBracketedStillImageSettings *)bracketSettings
error:(NSError *)error {
if(error == nil && !capturingVideo) {
NSData *d = [AVCapturePhotoOutput
JPEGPhotoDataRepresentationForJPEGSampleBuffer:photoSampleBuffer
previewPhotoSampleBuffer:previewPhotoSampleBuffer];
JAVA_OBJECT byteArray = nsDataToByteArr(d);
com_codename1_camerakit_impl_CameraCallbacks_onImage___byte_1ARRAY(
getThreadLocalData(), byteArray);
return;
}
if(error) {
struct ThreadLocalData* d = getThreadLocalData();
com_codename1_camerakit_impl_CameraCallbacks_onError___java_lang_String_java_lang_String_java_lang_String(
d, nil, nil, nil);
return;
}
}
Now that these are in place lets look at the delegate code.  
Delegate method declarations are huge! This goes all the way to the curly brackets, this makes Java's verbosity seem quaint. This is all just one delegate callback
method declaration.

photoSampleBuffer
return;
}
if(error) {
d, nil, nil, nil);
return;
}
}
If this is an error or the delegate was invoked because of a video event we don't want to step into the image processing code

photoSampleBuffer
return;
}
if(error) {
d, nil, nil, nil);
return;
}
}
NSData handles almost all IO in iOS. It's similar to ByteBuﬀer in Java SE and allows us to map files from/into storage.

photoSampleBuffer
return;
}
if(error) {
d, nil, nil, nil);
return;
}
}
I use nsDataToByteArr to convert the NSData object to a Java byte array, notice that in the native code all Java objects are eﬀectively JAVA_OBJECT

photoSampleBuffer
return;
}
if(error) {
d, nil, nil, nil);
return;
}
}
I can now invoke the callback method with the given byte array, notice the callback method includes the full name and argument types

photoSampleBuffer
return;
}
if(error) {
d, nil, nil, nil);
return;
}
}
The VM passes thread local context on the stack for every method, this makes things like stack traces work

#include "com_codename1_camerakit_impl_CameraCallbacks.h"
extern JAVA_OBJECT nsDataToByteArr(NSData *data);
For the code to compile we need to add these declarations below the #import statements. The VM generates C code so we import it with include.

The nsDataToByteArr method is a method from the iOS port. I could have just included the whole header but there is no need in this case. It makes the process of
converting an NSData much simpler

if(stillImageOutput == nil) {
stillImageOutput = [[AVCaptureStillImageOutput alloc] init];
NSDictionary *outputSettings = [[NSDictionary alloc]
initWithObjectsAndKeys: AVVideoCodecJPEG, AVVideoCodecKey, nil];
[stillImageOutput setOutputSettings:outputSettings];
}
[captureSession addOutput:stillImageOutput];
AVCaptureConnection *videoConnection = nil;
for (AVCaptureConnection *connection in stillImageOutput.connections) {
for (AVCaptureInputPort *port in [connection inputPorts]) {
if ([[port mediaType] isEqual:AVMediaTypeVideo] ) {
videoConnection = connection;
break;
}
}
if (videoConnection){
break;
}
}
[stillImageOutput
captureStillImageAsynchronouslyFromConnection:videoConnection
Now that all this is out of the way lets go back to the captureImage class and the iOS 9 compatibility block. This is eﬀectively code I got from stackoverflow

I needed this since most samples are for iOS 10+ now and ignore the legacy but I still have quite a few iOS 9 devices that can't upgrade to 10. So I'm assuming there is
still some market for compatibility:

Most this code is boilerplate code for detecting the camera, no wonder it was reworked

break;
}
}
break;
}
}
[stillImageOutput
completionHandler:^(CMSampleBufferRef imageSampleBuffer,
NSError *error) {
NSData *d = [AVCaptureStillImageOutput
jpegStillImageNSDataRepresentation:imageSampleBuffer];
[captureSession removeOutput:stillImageOutput];
}];
The capture image process is asynchronous and invokes the lambda expression below to process the resulting image

break;
}
}
break;
}
}
[stillImageOutput
completionHandler:^(CMSampleBufferRef imageSampleBuffer,
NSError *error) {
NSData *d = [AVCaptureStillImageOutput
jpegStillImageNSDataRepresentation:imageSampleBuffer];
[captureSession removeOutput:stillImageOutput];
}];
The capture image process is asynchronous and invokes the lambda expression below to process the resulting image. The rest of the code should be very familiar as it's
almost identical to the one we used in the iOS 10 version. We just callback into Java.

With this image capture should now work for both old and new devices. All the basics should work with the exception of video!

-(void)captureVideo{
NSURL *furl = [NSURL fileURLWithPath:[NSTemporaryDirectory()
stringByAppendingPathComponent:@"temp.mov"]];
[self captureVideoFile:[furl absoluteString]];
}
Capturing video is surprisingly easier than capturing still images. First the simplest method is captureVideo. The no-arguments version of this method saves to a
temporary file name. We invoke the version that accepts the file with a “temp.mov” file path.

-(void)captureVideoFile:(NSString*)param{
if(movieOutput != nil) {
[captureSession removeOutput:movieOutput];
[movieOutput stopRecording];
[movieOutput release];
}
capturingVideo = YES;
movieOutput = [[AVCaptureMovieFileOutput alloc] init];
[captureSession addOutput:movieOutput];
[movieOutput startRecordingToOutputFileURL:
[NSURL URLWithString:param] recordingDelegate:self];
});
}
The version that accepts a path isn't much harder.

The AVCaptureMovieFileOutput class represents the recording process, we lazily initialize it

}
});
}
We bind the output value to the capture session

}
});
}
Now we can provide a URL from the file argument and start recording, again we use `self` as the delegate

- (void)captureOutput:(AVCaptureFileOutput *)output
didFinishRecordingToOutputFileAtURL:(NSURL *)outputFileURL
fromConnections:(NSArray<AVCaptureConnection *> *)connections
if(capturingVideo && outputFileURL != nil) {
NSString* url = [outputFileURL absoluteString];
com_codename1_camerakit_impl_CameraCallbacks_onVideo___java_lang_String
(d, fromNSString(d, url));
return;
}
if(error != nil) {
com_codename1_camerakit_impl_CameraCallbacks_onError___java_lang_String_java_lang_String_java_lang_String
(d, nil, nil, nil);
return;
}
}
Which obviously leads us to the delegate method. Most of this should be very familiar now that we went through the image capture code.

- (void)captureOutput:(AVCaptureFileOutput *)output
didFinishRecordingToOutputFileAtURL:(NSURL *)outputFileURL
fromConnections:(NSArray<AVCaptureConnection *> *)connections
if(capturingVideo && outputFileURL != nil) {
NSString* url = [outputFileURL absoluteString];
com_codename1_camerakit_impl_CameraCallbacks_onVideo___java_lang_String
(d, fromNSString(d, url));
return;
}
if(error != nil) {
com_codename1_camerakit_impl_CameraCallbacks_onError___java_lang_String_java_lang_String_java_lang_String
(d, nil, nil, nil);
return;
}
}
This callback accepts a path String which we can translate from NSString using the fromNSString API call

-(void)stopVideo{
movieOutput = nil;
});
}
The last remaining piece is the stop video method. This is a trivial method if you've been keeping up. We just stop the recording and cleanup. And with that last bit of
code we are DONE!

I hope you found this useful, there is a lot of code to go through but most of it is damn trivial once you look it over. You can create native interfaces that do just about
anything if you have the patience to debug and google the native API's.

Building a Native Camera Access Library - Part V - Transcript.pdf

Recommended

Recommended

More Related Content

Similar to Building a Native Camera Access Library - Part V - Transcript.pdf

Similar to Building a Native Camera Access Library - Part V - Transcript.pdf (20)

More from ShaiAlmog1

More from ShaiAlmog1 (20)

Recently uploaded

Recently uploaded (20)

Building a Native Camera Access Library - Part V - Transcript.pdf