Container based android

Copyright 2016 ITRI 工業技術研究院 0
Container-based Android
(Android Containerization)
姓名：吳天健、許維德、鄭博文
單位：資通所

1Copyright 2016 ITRI 工業技術研究院
Outline
❖Preliminaries
❖Android Containerization
➢Isolate/share services among Android instances
❖Linux Kernel, Drivers and Android Binder
❖Window System and Graphics
❖Android X86 Containerization
❖TODOs
❖ICL’s Big Picture - Android as a Service

Preliminaries
❖ Android Platforms
➢ Nexus 7 with Android 6.X/Kernel 3.4.0
➢ Pixel XL with Android 7.1.X/Kernel 3.18.31
❖ LXC and Other Tools
➢ Built with Android NDK: android-toolchain-arm_64-4.9-android-24
➢ LXC
➢ libcap (https://github.com/abstrakraft/lxc-android-libcap)
• Only libcap is required, built in static (libcap.a)
➢ SELinux
• Using essential Android code base: external/selinux
• libselinux: external/selinux/libselinux
– NOT external/libselinux which is a small port of libselinux for Android framework
• libsepol: external/selinux/libsepol

Preliminaries (2)
❖ LXC ❖ Packaging LXC and tools
/odm
├── bin
│ ├── ....
│ ├── lxc-start
│ ├── lxc-stop
│ ├── ....
│ ├── setfilecon
│ ├── setfiles
│ └── ...
├── lib
│ ├── liblxc.la
│ ├── liblxc.so
├── libexec
│ └── lxc
│ ├── hooks
│ │ └── unmount-namespace
│ ├── lxc-apparmor-load
│ ├── lxc-containers
│ ├── lxc-monitord
│ ├── lxc-net
│ └── lxc-user-nic
export CBA_HOME=/odm
export PREFIX=$CBA_HOME
export PIXEL_LIBS=/home/sting/cba/lxc/pixel-libs/usr
CFLAGS="--sysroot=$SYSROOT -I$PIXEL_LIBS/include -fPIE"
CPPFLAGS="$CPPFLAGS $CFLAGS"
LDFLAGS="$LDFLAGS -fPIE -pie -L$PIXEL_LIBS/lib"
./configure --host=aarch64-linux-android
--enable-shared=yes --enable-static=yes
--disable-api-docs
--enable-selinux
--enable-capabilities
--disable-examples
--disable-lua
--disable-python
--disable-bash
--enable-configpath-log

--prefix=$PREFIX
--with-systemdsystemunitdir=$PREFIX/lib/systemd/system
--with-config-path=$CBA_HOME/containers
--with-global-conf=$CBA_HOME/.config
--with-runtime-path=/cache

Android Containerization

Containerization of Android
❖ Android was essentially designed for single instance architecture
❖ In order to better containerized Android, we need to
➢ Refactor essential Android key services architecture design to make it namespace-
aware and sharable among Android containers
➢ Extend the current LXC to support namespace-aware device model and namespace-
aware deriver development
➢ Extend Android virtual devices and drivers to support namespace-awareness
Unified Linux Kernel & Drivers
LXC Container (Cgroups, Namespace)
SELinux
Virtual Device
Drivers
Host Android
Shared
Framework
Services
Service Manager Surface Flinger
Camera
LXCTools
Virtual
Devices
binder RIL
alarm
….
Netfilter,
Virtual Ethernet Bridging
iptables
Power Mgmt. ….
Android 1 Android 2
Audio Sensors

Linux Kernel, Drivers and
Android Binder
Victor

Isolate/Share Services among
Android Containers
❖ Refactoring essential Android binder architecture design to
enable isolate/share services among Android instances
➢ Namespace-aware IPC binder (Inspired by Container Virtualization
Adapted to Android @浙江大学计算机系统结构实验室)
➢ Reduce the total # of Linux process
➢ Can be configured which services to be shared
➢ Caller is aware of the container ID of callee
Linux Kernel + LXC + Drivers
Virtual Devicesbinder 1 binder n
Virtual Devices namespace-awarebinder
Shared service invocation
Unshared service invocation
Before After

Android Service
Registration & Discovery
ServiceManagerService App
1. addService(name, service)
2. getService(name)
3. Binder IPC Transactions
service

Virtual Binder Driver
Lei Xu, Wenzhi Chen, Zonghui Wang, Guoxi Li, Chuan Li, Weijie Sun, Condroid:A Container-Based
Virtualization SolutionAdapted forAndroid Devices , Proceedings of IEEE International Conference onMobile
Cloud Computing, Services, and Engineering, 2015
Real
binder
driver
Host
Service
Manager
SHARED

Android Virtual Binder Architecture
Service
Manager
Service App
Host
Service App
Container1
Service App
Container2
/dev/binder /dev/virtualbinder1 /dev/virtualbinder2
Virtual Binder Driver
Real Binder Deriver
Linux Kernel

Virtual Devices and their Drivers
❖ Device Namespaces : namespace aware device driver (inspired by
Cells proj. @ Columbia University)
➢ Stateful/Statelessdevice driver, stateful devices, like alarm device, the
state of different namespaces must be maintained
❖ AlarmManagerService
➢ alarm-dev driver has been removed in Android Nougat, the functionality
provided by the Android alarm-dev driver should now be present in the
timerfd interface
➢ com_android_server_AlarmManagerService.cpp
static const clockid_t android_alarm_to_clockid[N_ANDROID_TIMERFDS]
• CLOCK_REALTIME_ALARM (since Linux 3.11)
• CLOCK_REALTIME
• CLOCK_BOOTTIME_ALARM (since Linux 3.11)
• CLOCK_BOOTTIME (since Linux 3.15)
• CLOCK_MONOTONIC
• CLOCK_REALTIME
Virtual Devices alarm- dev
Android 1 Android n
State1 Staten

Window System and Graphics
Sting

Share or Not to Share
SurfaceFlinger/WindowManager
❖浙江大学计算机系统结构实验室
➢Share both SurfaceFlinger and Window
Manager
➢Maintains different z-order range for different
containers
Surface Flinger buffer
Host Android
Shared
Framework
Services
Surface Flinger
Window Manager

Share or Not to Share
SurfaceFlinger/WindowManager (2)
❖ICL Approach
➢Window Manager in each container
➢Single SurfaceFlinger instance on host
• Maintains list of Layers of different Android containers
• Groups Layers according to their container ID
Surface Flinger buffer
Host Android
Shared Framework
Services Surface Flinger
Window Manager
Android n
Window Manager

Display Switching between Containers
• Layers of different Android
containers in SurfaceFlinger
• SurfaceFlinger groups layers
according to their container
ID
Assign layers of host container to
layersSortedByZ of primary display
primary display
compose
Assign layers of container 1 to
layersSortedByZ of primary display
primary display
compose

Android X86 Containerization
TC

Who is maintaining Android-x86 ?

Android-as-a-Service on PC
Container1
App App App
Android Framework
Surface
-
flinger
Windows
-
Manager
Service-
Manager
Input-
Manager
Container2
App App App
Android Framework
Surface
-
flinger
Windows
-
Manager
Service-
Manager
Input-
Manager
Container3
App App App
Android Framework
Surface
-
flinger
Windows
-
Manager
Service-
Manager
Input-
Manager
PC
LXCTools
namespaces
Graphic Driver Binder Driver
Keyboard Driver
Mouse Driver
Touch-Panel
Other
Device Drivers
Display
Virtualization
Binder (IPC)
Virtualization
Input
Virtualization
…... …... …... …... …... …...

Multiple instances of SurfaceFlinger
Can there be multiple surfaceflinger(s) ?
This implies to a second question : is gpu / driver
(drm) shareable for gui(s) ? void SurfaceFlinger::doComposition()
{
….pseudo code…
FOR EACH DISPLAY i
IF DISPLAY_ON( i )
IF (i) == MAIN_DISPLAY AND byte 1 of
“/data/surfaceflinger/control” is Zero
GOTO END
r = GET_DIRTY_REGION
DO_DISPLAY_COMPOSITION(r)
DIRTY_REGION_CLEAR()
HARDWARE_FLIP()
ENDIF
NOTITY_HARDWARE_COMPOSITION_
COMPLETE()
ENDFOR
POST_FRAMEBUFFER()
LABEL END
}
Kernel
Display
Driver
GPU
Driver
Graphic
Driver
Container
... ...
OpenGL
ES
Surface-
flinger
Windows-
Manager
Hw Comp
oser
Container
... ...
OpenGL
ES
Surface-
flinger
Windows-
Manager
Hw Comp
oser
Container
... ...
OpenGL
ES
Surface-
flinger
Windows-
Manager
Hw Comp
oser

Static /dev
I want in each container servicemanager connect (open)
/dev/binder (not /dev/binde1, /dev/binder2 , etc)
/dev/ was mounted tmpfs, how about make it static?
Container1
Servic
e
Manag
er
... APP
/dev/binder
Kernel
/dev/binder1
Container2
Servic
e
Manag
er
... APP
/dev/binder
/dev/binder2
mknod /data/con1/rootfs/dev/binder 57 0
mknod /data/con2/rootfs/dev/binder 58 0
##host /dev/binder major number 56
##host /dev/binder1 major number 57
##host /dev/binder2 major number 58
int main(int argc, char **argv)
{
int fd_count= 0;
struct pollfdufds[4];
char *tmpdev;
char* debuggable;
char tmp[32];
int property_set_fd_init = 0;
int signal_fd_init = 0;
int keychord_fd_init = 0;
bool is_charger = false;
char initrc_path[PROP_VALUE_MAX];
/* If we are calledas'modprobe' command, we run asa
* standaloneexecutable andreuse ueventd'slogic to do
the job.
*/
if (strstr(argv[0], "modprobe"))
return modprobe_main(argc, argv);
if (!strcmp(basename(argv[0]), "ueventd"))
return ueventd_main(argc,argv);
if (!strcmp(basename(argv[0]), "watchdogd"))
return watchdogd_main(argc, argv);
/* clear the umask*/
umask(0);
/* Get the basic filesystem setup we needput
* together in theinitramdiskon / and then we'll
* let the rc filefigureout therest.
*/
mkdir("/dev",0755);
mkdir("/proc", 0755);
mkdir("/sys", 0755);
#if STATIC_DEV
printf(">>>>>skip mount/devn");
#else
mount("tmpfs", "/dev", "tmpfs", MS_NOSUID,
"mode=0755");
#endif
mkdir("/dev/pts", 0755);
mkdir("/dev/socket", 0755);
#if STATIC_DEV
printf(">>>>>skip mount/dev/ptsn"); //use lxc's
/dev/ptsinstead
#else
mount("devpts", "/dev/pts", "devpts", 0, NULL);
#endif
mount("proc", "/proc","proc", 0, NULL);
mount("sysfs", "/sys", "sysfs", 0, NULL);

Exploit (kernel) memory
Android-x86_5.1 is defaultly 32bits build
As a server, install 16g RAM and enable kernel option
‘HIMEM_64G’...
The OOM killer be called even when there is still plenty of
memory available <0>[ 331.441013]
lowmem_reserve[]: 0 0 0 0” and then
Out-of-memory invoked
<3>[ 2839.557148] binder4:
binder_mmap: 6382 adf02000-
ae000000 get_vm_area failed -12
<4>[ 2859.865885] vmap allocation
for size 1044480 failed
Recipe - In bootloader’s booting
parameter, add vmalloc=512MB
<6>[ 0.000000] vmalloc :
0xdfbfe000 - 0xffbfe000 ( 512 MB)

Camera HAL
Besides virtualizing native linux device , it’s necessary to
virtualize android HAL
I want take a error on arm device as an example
02-18 02:52:31.744 24 24 I CameraService: Loaded"QCamera Module" camera module^M
02-18 02:52:31.744 24 24 E mm-camera-intf:mm_camera_open: dev name= /dev/video1, cam_idx = 1^M
02-18 02:52:31.941 24 24 E mm-camera-intf:mm_camera_open: opened, breakout while loop^M
02-18 02:52:31.941 24 24 E mm-camera-intf:mm_camera_socket_create: socket_fd=-1 No such fileor directory
^M
^M
02-18 02:52:31.970 24 24 E mm-camera-intf:mm_camera_open: cannot opendomain socket fd of
'/dev/video1'(No such fileor directory)^M
02-18 02:52:32.029 24 24 E mm-camera-intf:camera_open: mm_camera_open err = -1^M
02-18 02:52:32.029 24 24 E QCamera3HWI:static int
qcamera::QCamera3HardwareInterface::initCapabilities(int): camera_open failed^M
02-18 02:52:32.033 24 24 E CameraFlashlight: createFlashlightControl: failed to get camerainfo for camera0^M
02-18 02:52:32.033 24 24 E CameraFlashlight: findFlashUnits: failedto createflash control for 0^M
02-18 02:52:32.033 24 24 E mm-camera-intf:mm_camera_open: dev name= /dev/video2, cam_idx = 2^M
02-18 02:52:32.597 24 24 E mm-camera-intf:mm_camera_open: opened, breakout while loop^M
^M
^M
02-18 02:52:32.628 24 24 E mm-camera-intf:mm_camera_open: cannot opendomain socket fd of
'/dev/video2'(No such fileor directory)^M
02-18 02:52:32.874 24 24 E mm-camera-intf:camera_open: mm_camera_open err = -1^M
02-18 02:52:32.874 24 24 E QCamera3HWI:static int
qcamera::QCamera3HardwareInterface::initCapabilities(int): camera_open failed^M
02-18 02:52:32.874 24 24 E CameraFlashlight: createFlashlightControl: failed to get camerainfo for camera1^M
02-18 02:52:32.874 24 24 E CameraFlashlight: findFlashUnits: failedto createflash control for 1^M

TODOs

TODOs
❖Power Management
➢To share or not to share PowerManagerService
❖Activity Manager Service
❖Low Memory Killer
❖SELinux
➢To further isolate host and containers
❖SEE (Secure Execution Environment)
Integration
➢vold and Qualcomm SEE issues
➢DRM

Android-X86
ICL is constructing the first cost-effective
APP streaming cloud infrastructure,
consisting of
❖ Low latency audio/video streaming services
❖ Scalable & cost-effective virtual/physical
smartphone pool
➢Android-X86 virtual smartphones
➢Cost-effective smartphone SOC cluster server
➢Containerized Android doubles/triples the
Android instances density of per U chassis
❖ BMC management software
➢H/W & S/W deployment automation
❖ APP Streaming Service Management
CPU CPU
CPU CPU
CPU CPU
CPU CPU
4 GB
8GB
4 GB
8GB
ICL’s Big Picture - Android as a Service

Google group: Android-x86
http://groups.google.com/group/android-x86
IRC channel at irc.freenode.net
#android-x86

Thank You!
Questions and Comments?

Container based android

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Container based android

Similar to Container based android (20)

Recently uploaded

Recently uploaded (20)

Container based android