International Journal of Engineering Research and Development
android_project
1. DEVELOPMENT OF APPS FOR ANDROID WITH NDK INCLUDED FOR
PHOTO, VIDEO AND AUDIO RECORDING & YUV CONVERSIONS
496 PROJECT, FALL 2014
ADIT GHOSH
JUNIOR, COMPUTER ENGINEERING
PURDUE UNIVERSITY
2. INTRODUCTION
Software development for android operating systems is usually done in Java using the
popular Android Studio IDE. Eclipse SDK provides Java and CVS support and includes Java
development tools (JDT). Android consists of a Linux kernel 3.4 or later; above the kernel are
the libraries written in C/C++ (SSL, SGL, Surface Manager, Runtime Core Libraries, etc). Then
there are application software (APIs) used by core applications and includes includes a set of
visual objects, a resource manager, a activity manager and package manager (Wikipedia
Commons 2014). Applications are at the top level and are generally in Java.
Android high level architecture (Wikipedia Commons 2014)
Google Android’s Native Development Kit (NDK), allows developers to extending this to
native code languages like C or C++ and can be useful for integrating existing C, C++ code
libraries into the Android platform (Android 2014). Java Native Interface (JNI) allows programs
written in Java language to interact with native code (written in C/C++). An important utility of
the NDK toolset is to overcome the limitations of Java, such as memory management and
performance by programming directly into Android native interface.
Android applications that can incorporate native code using the NDK toolset would allow
developers to reuse legacy code, code to low-level hardware, or differentiate their apps by taking
3. advantage of features otherwise not optimal or possible. It enables support for different
architectures by including the libraries x86, mips, arm64 etc. Media apps (for photography, video
and audio recording) that include native functionality would have wider cross platform
applications.
Photos taken with this android app are saved as png or jpg files. Whereas png or tiff have
lossless image compression, jpeg is considered lossy. The baseline operation mode
in jpeg is
lossy compression with a quality/compression factor from 1 to 100. Jpeg operates on 8x8 or
16x16 pixels macro blocks which are compressed independently. Jpeg compression requires
color space conversion (to YUV). YUV encodes a color image or video taking human perception
into account, allowing reduced bandwidth for chrominance components, thereby typically
enabling transmission errors or compression artifacts to be more efficiently masked by the
human perception than using a "direct" RGB-representation (Wikipedia 2014). Other color
spaces have similar properties, and the main reason to implement or investigate properties of
YUV would be for interfacing with analog or digital television or photographic equipment.
Conversion from RGB to YUV color space uses the relation
Y = 0.299*R + 0.587*G + 0.114*B
U = 128 – 0.1687*R – 0.3312*G + 0.5*B
V = 128 + 0.5*R – 0.4186*G – 0.0813*B
RGB files are typically encoded in 8, 12, 16 or 24 bits per pixel. YUV files are encoded in
12, 16 or 24 bits per pixel. The common formats are YUV444, YUV411, YUV422 and YUV420
(Wikipedia 2014). Of these file formats, YUV420 is the most common, as the data is more easily
compressed (file extension .yuv).
The objective of this work was to develop apps with native support for capturing
photographs, videos and audios. These recordings would be converted to yuv format that will
enable easier data compression for analog or digital interfacing.
4. APPLICATION DEVELOPMENT
1. Software installation :
a. Android SDK bundle : This installation was done on a Mac running Yosemite 10.0.1 with
XCode 6.1.1. Since this version of XCode uses Objective C Mac Developer options were used to
enable C within XCode. JDK was also installed with the latest packages. Android SDK bundle
was downloaded from the official Android site (Android 2014). Then additional SDK packages
were installed by opening the executable android at SDK à tools à android and selecting all
sdk build tools upto Android SDK 19, Google API and system images for ARM EABI v7a, Intel
x86 Atom Intel x86 and Atom_64 for Android 5.0, 4.4W2 and 4.4.2. From Extras the following
were installed : Android Support Repository, Library, Google Play Services & Play Services for
Froyo, Intel x86 Emulator Accelarator (HAXM installer) and Android API Auto Simulators.
b. Android NDK : NDK was downloaded and installed with the command chmod a+x
android-ndk-r10c-darwin-x86_64.bin and ./android-ndk-r10c-darwin-x86_64.bin. To configure
Eclipse for use with NDK the following was done : In Window->Preferences->Android->NDK
the NDK installation path was put in.
c. Homebrew : Homebrew is a useful package manager that allows installation of software
packages not available in Mac with the command
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
(Homebrew 2014)
Brew install commands are then used to install missing dependencies that are essential for
installation of other software. In particular latest gcc versions were installed with brew and so
was cmake. Brew update and brew doctor were also particularly useful for plugging in loopholes.
d. Bash.rc and bash_profile : Unlike pure Linux systems, Mac does not come with bash.rc
and bash_profile for the User and these files are located in /etc and are not available for
changing. Since Iit is necessary to add library paths to bash.rc and bash_profile, these were
created under /User/Adit with the following library exports
if [ -f ~/.bashrc ]; then
./.bashrc
fi
PATH=$PATH:
PS1='h:W u$ '
export CDPATH=:$HOME
5. # general path munging
PATH=/usr/local/sbin:/usr/local/bin:/:/usr/local/Cellar//Users/adit/Downloads/android-ndk-r10c:$PATH
# set editor variable for general editing
export EDITOR=vi
# sdk-ndk path
export NDKROOT=/Users/adit/Downloads/android-ndk-r10c
export ANDROID_HOME=/Users/adit/Downloads/adt-bundle-mac-x86_64-20140702/sdk
export NDK_HOME=$NDKROOT
#path to arch
export SYSROOT=$NDKROOT/platforms/android-19/arch-arm
export PATH=$PATH:$ANDROID_HOME/tools
export PATH=$PATH:$ANDROID_HOME/platform-tools
export PATH=$PATH:$ANDROID_HOME/build-tools
export ANDROID_STANDALONE_TOOLCHAIN=$PATH:/Users/adit/Downloads/android-ndk-
r10c/toolchains
export NDK_PROJECT_PATH=$PATH:/Users/agtec/Downloads/android-ndk-r10c/build/core
export CXX=/usr/local/bin/g++-4.9
export CC=/usr/local/bin/gcc-4.9
e. Torch : Torch7 (Torch 7 installation 2014) and torch-android were installed from github
using curl -sk https://raw.github.com/torch/ezinstall/master/install-deps | bash and curl -sk
https://raw.github.com/torch/ezinstall/master/install-luajit+torch | bash. In this version of Mac
OS, gcc errors and cmake errors are observed so all of these libraries had to be installed through
Homebrew. I also downloaded and installed Node.js and gfx.js. Luajit and luarocks seemed to
have installed as seen by the output to the command luajit –v and luarocks --version but there
were numerous errors when running the exercises. Working with Torch in a Mac Yosemite OS
seems to create serious issues. In particular it required uninstalling of gcc which created other
problems. In view of the serious problems encountered, I installed the package for yuv
transformation individually. For this purpose, ffmpeg was best suited and was therefore installed.
f. FFMPEG : Ffmpeg version for Snow Leopard and above was installed from
FFMPEGMAC (2014) . Ffmpeg installed without any problems.
7. app (I used the ECE icon). The directory src/com/example/android/cameravideo contains the
subactivity .java files CameraVideo.java, BaseAlbumDirFactory.java,
AlbumStorageDirFactory.java and FroyoAlbumDirFactory.java. The CameraVideo.java file
contains the imports, public class Activity and Private files (private to the application). Other
.java files contain various public and private classes for storage.
For the audio recorder a min SDK version of 10.0 was used. Permissions to record audio
and to write to external storage were included as
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
The main.xml file included a LinerLayout for buttons and images. The /res/values directory
contained the Strings.xml description for strings used. The drawable directories contained an
icon for the app from Androids own icon library. The folder src/com.adit/MyRecordings contains
the file MyRecordingsActivity.java where the public and private classes are defined. There were
choices of two formats for the recordings – the popular mp4 format and also 3GPP format. The
3GPP file format is a multimedia container format defined by the Third Generation Partnership
Project (3GPP) for 3G UMTS multimedia services. It is used on 3G mobile phones but can also
be played on some 2G and 4G phones (Wikipedia 2014).
3GPP is a required file format for video
and associated audio files and timed text in ETSI 3GPP technical specifications for IP
Multimedia Subsystem (IMS), Multimedia Messaging Service (MMS), Multimedia
Broadcast/Multicast Service (MBMS) and Transparent end-to-end Packet-switched Streaming
Service (PSS).
The apps developed were first tested on an Emulator. To start an Emulator, first the Java
option tab was clicked followed by Windows à Android Virtual Device à Create. The
Emulator I used, was Galaxy Nexus Android 4.4.2 with API 19 with emulated cameras, RAM
1024, VM heap 64 and SD card 256MB. After the apps were run on the Emulator, they were
installed on a Samsung S4 phone. But before this can be done, developer options had to be
enabled on the phone. To do this the android phone has to go into debugging mode. In phone
settings I clicked on 'Build Number' 7 times. It said 'Developer Mode enabled'. Then the phone
was connected to the Mac. I right clicked on the project folder listed in package explorer and
clicked on run asà Android Application. This installed the app on the phone.
To build native support with NDK in Eclipse I righted click the project then -> Android
Tools -> Add Native Support -> Finish. Project configuration is updated with NDK build-related
8. settings. On rebuilding the project, it first compiles and builds the NDK code, copies the built
libs to appropriate project folders, and then continues with the regular APK build process. In
order for the app to use x86 and ABI, further processing is required. I right clicked the jni folder
in the project, then New -> File -> Enter “Application.mk” as the filename, and clicked Finish.
In the Application.mk file I added these lines :
APP_ABI := all
APP_ABI :=x86
When this app is rebuilt, it will build for all ABIs in NDK. Right clicking the project
in
Eclipse
-‐>
Run
As
-‐>
Android
Application
automatically
invokes
the
NDK
build
and
generates
all
native
libraries.
This
is
seen
in
the
libs
folder.
The
jni
layer
can
be
represented
as
(Adapted from Aptina Imaging Corporation 2012)
File conversion was done using ffmpeg. To convert jpg files to yuv format I used the
command
ffmpeg -i test-1080x1140.jpg -s 1080x1140 -pix_fmt yuv420p test-yuv420p.yuv
where the left yellow highlight is the size of the file to convert and the right highlight is the
size of the output file. The command converts it to a yuv420 file. To reconvert to jpg I used the
command
ffmpeg -s 1080x1140 -pix_fmt yuv420p -i test-yuv420p.yuv test-1080x1140.jpg
Since the video I took had a very large memory, I first took a 3 sec clip of it using
Application
Code
Android
Framework
Code
Core
Libraries
DVM
JNI
Android_hardware_
camera.cpp
Camera
Service
Libcameraservice.so
Camera
HAL
Implementation
9. ffmpeg -i input.mp4 -ss 00:00:50.0 -codec copy -t 3 output.mp4
then converted it to a yuv format using
ffmpeg -i testvideo2.mp4 -s 284x160 -pix_fmt yuv420p testvideo2.yuv
The file was reconverted to mp4 with the command
ffmpeg -i testvideo_1.mov -ss 00:00:50.0 -codec copy -t 3 testvideo2.mp4
RESULTS
I made two sets of apps. The first set were two wholly Android apps for photo and Video
recording and storage (CameraVideo) and for audio recording and storage (MusicRecorder).
Subsequently I included native functionality with jni and necessary library files for native
applications.
After many trials I could get to run successfully on the emulator. Fig1a shows the
CameraVideo app (with ECE logo) installed in the emulator (close up in Fig1b). I also installed
the MusicRecorder on the emulator. This can be seen in Fig 1c &d with a music note as logo.
On entering the CameraVideo app in the emulator the visual is as shown in Fig 2a. When
the photo or video options buttons are pressed we get a default video as in Fig 2c. The
MusicRecorder app when entered appears as in Fig2b &c.
Finally, when all seemed to work well, I installed the apps on my phone. Fig 3a is a
screenshot of the app on my phone. This screenshot was also taken with Eclipse. The process
was as follows : First connect the phone to the Mac, then in Eclipse go to Windows à Open
Perspective à DDMS and click on the phone that is shown on the left. Then the first tab shows
the option ‘screen capture’. The screenshot is saved.
The screenshot in Fig3a shows the CameraVideo app. Fig 3b is a photo of me taken using
my app. The photo was saved to my images photo (public). I also installed the MusicRecorder
app on my phone. (Fig 3c last one). Two pieces of music were recorded and got saved in /My
Files/Music Recorder folder (Fig 3d). The recorded files were in that folder in mp4 format (Fig
3e).
The attachment folder ‘test photo’ has the yuv and re-transformed yuv files of the photo
taken with the photo app. The input photo had dimensions 1080 x 1440. The transformed file
was in yuv420 format. This was reverted to the jpg form (testphoto_1-R.jpg) of the same
dimension as the original. The attachment folder ‘test video’ has a test video file testvideo2.mp4
10. (568 x 320) which is converted to its yuv420. This was reconverted to a file with half the
dimensions (284 x 160). Two mp4 audio files recorded with my app are also attached.
SUMMARY
In this work two different apps were developed – one for capturing and storing photos and
videos and another for recording audio. The apps were successfully tested on an Emulator and a
Samsung S4 mobile phone. All photo, video and audio captures were stored in the phone in
public locations. The apps run well and produce good quality pictures, videos and audio
recordings. The apps were also enabled for native applications and thereby increase their
versatility of use. Finally, the jpg files from the app were converted to the yuv format and
checked for conversion success by reconverting again to the jpg form.
REFERENCES
1. Android (2014) Retrieved from http://developer.android.com/index.html
2. Aptina Imaging Corporation (2012) Exposing the Android Camera Stack. Retrieved from
https://thenewcircle.com/s/post/1268/Exposing_the_Android_Camera_Stack.pdf
3. Creating Android Project (2014) Retrieved from
http://developer.android.com/training/basics/firstapp/creating-‐project.html
4. FFMPEGMAC
(2014)
FFMPEG
Binary
for
Mac.
Retrieved
from
http://ffmpegmac.net/
5. Torch7
Installation
Guide
(2014)
elab
Purdue
University.
Retrieved
from
https://engineering.purdue.edu/elab/html/wiki-‐torch7-‐installation.html
6. Homebrew (2014) Retrieved from http://brew.sh/
7. Wikipedia (5th
November 2014) 3GP and 3G2 Retrieved from
http://en.wikipedia.org/wiki/3GP_and_3G2
8. Wikipedia (29 November 2014) YUV Retrieved from http://en.wikipedia.org/wiki/YUV
9. Wikipedia Commons (2014) Retrieved from
http://commons.wikimedia.org/wiki/File:Android-System-Architecture.svg
11. APPS
INSTALLED
ON
A
SAMSUNG
GALAXY
EMULATOR
Fig 1 : CameraVideo and MusicRecorder apps installed on Galaxy Nexus Emulator
(a) Photo-cum-video app installed on emulator (b) Closeup of the photo-video app
(c) Photo-video and audio apps on emulator
12.
(a) CameraVideo app on entry
(note the ECE logo used)
(b) MusicRecorder on entering
(c) Video recording from emulator
(inbuilt video)
(d) Audio recorder app
Fig 2 : CameraVideo and MusicRecorder apps installed on Galaxy Nexus Emulator
13. APPS
INSTALLED
ON
A
SAMSUNG
S4
(a) The installed app on my cell (b) A photo of me taken with my app
(c) The Music Recorder on my cell (d)The music saved in a folder in my cell (e)The music saved in files in my cell
Fig 3 : CameraVideo and MusicRecorder apps installed on a Samsung S4 cell & the captured
files
