OpenCV Image Processing on Android Devices Wilson Wingston Sharonwingston.email@example.com
Le me• Day job python developer.• Hobby technologies o Machine Learning o Hardware Hacking o Image processing o Graphics & Video design
Introduction• Open Computer Vision (OpenCV) developed & donated to FOSS by Intel.• Open Source Image Processing library in C/C++• Optimized for real time image / video processing applications• OS/hardware independent with ports to Java, Python with wrappers to numerous other languages and platforms.
Why OpenCV?• No need to reinvent the wheel to deal with image processing solutions.• Has both high level and low level components.• Lets you treat images as a 2D Matrix.• Image processing comes down to simple mathematical operations on this Matrix.• Also provides a framework for machine learning, 3D scene information, depth profile, object recognition, motion tracking etc..
Suiting up• Sun JDK 6 o Very important you DO NOT use Sun JDK 7. unless you want to deal with intermittent weird run time issues that make no sense. In that case go right ahead.• Android SDK + NDK o Keep it updated once in a while. o Remember:You need “ia32-lib” if you’re running a 64 bit ubuntu/debian o NDK is required only if native code is erquired to be compiled.• Android SDK tools rev14+ o fun fact: using older versions can cause heartburn.• SDK Platform API 8 is minimum required o But usually a device running 11 would have the processing power you need to play with image processing application without lag.• Eclipse with ADT + CDT o Duh.
OpenCV4Android SDK• Obtain CV-2.4.2-android-sdk.zip from sourceforge.• SDK o Contains the openCV libraries and Java classes for Android. o /java, /native and /etc• APK o Opencv manager apk that must be installed to make the openCV libaries on the android device available. o Android market installation can also be done.• Samples o Contains a decent amount of openCV samples for android. o These samples are written as documentation. o See the openCV wiki for setting up Eclipse with OpenCV.
org.opencv.android• Contains the Async loader that you’ll need to connect your application to the openCV libraries installed on your device.• Utility function to convert between openCV data types and android ones.• Some other loader classes for your viewing pleasure.
org.opencv.core•Contains the core building blocks of openCV•Matrix, point, rect and Bitmap data types defined here.org.opencv.highgui•Allows you to draw some GUI components on the screen.•Also has the openCV videcapture class as an alternative to the android camera interface.
org.opencv.imgproc• The main image processing functions exist here, like blurring edge detection, colour conversion.• This file also contains documentation and usage of each function.org.openCV.ml• Contains classes to assist in developing almost any machine learning algorithm in images.• Trees, statistical models, Bayes, Markov models, K nearest all provided here with in-code commented documentation
The application structure• Best way (as suggested in samples) is as follows.• MainActivity o Extends the android activity class & is the Launcher Activity. o Initializes the openCV libraries. o Handles the android activity lifecycle functions.• opencvView o Extends opencvViewBase o Running openCV code goes here.• opencvViewBase o Extends surfaceView as a holder for the video stream o Abstract class to simplify usage of OpenCV o Sets up and handles camera / preview & drawing
MainActivity.java• onCreate() o This is called when the activity is first created. o OpenCVLoader.initAsync(…) attempts to start the openCV library and on completion calls mOpenCVCallBack.• mOpenCVCallBack() o Inside there is a function called onManagerConnected() which checks to see if no errors were there o If openCV loaded sucessfully, then an instance of Sample1View can be created called say mView• Also this class has the onPause(), onResume() functions which start / stop camera streaming as per the android activity lifecycle.
opencvView.java• This class contains code relevant for your application, this directly extends the absract opencvViewBase class.• processFrame(byte data) o Input comes in as byte data as raw camera feed o Using the mat.put() this picture can be stored in a matrix o Any openCV functions from imgproc etc.. can be performed on this mat object o Display must be in bitmap format, so the mat object is converted to bmp o This bmp object is returned to run() which displays it on the screen.
• Camera mCamera; o Camera object to open/close/ retrieve video information• SurfaceHolder mHolder; o Holder to the surfaceview to enable a live streaming of video• Mat mRgba; o Matrix Object that openCV uses for image manipulation• Bitmap mBitmap; o Bitmap object that can be displayed on the surface view• mRgba.put(0, 0, data); o Returns the raw data feed in an openCV matrix format.• Imgproc.cvtColor(mGraySubmat, mRgba, Imgproc.COLOR_GRAY2RGBA, 4); o How to convert an RGB matrix to a greyscale• Utils.matToBitmap(mRgba, bmp); o Converting a matrix to a bitmap
Real time Face detection• fdActivity example is an openCV sample that shows how to implement face detection algorithm on a real time video stream.• It is also an example on how to efficiently implement an algorithm in C and call it via the JNI interface.• Also implements an FPS meter class to judge the usability of the algorithm used.• A Haar Classifier from the ml library with the openCV trained xml file is used on the image stream to return if the image contains a face or not.• It is the standard openCV way of face detetcion.
Possible Applications on android devices• Obtain features from any image and send it over network for matching against an online database for object recognition.• Use the front camera to monitor head movement and provide a hands free interface for some games.• Auto-tagging of friends in photos taken and associating them with facebook/android contacts.