DEVIEW2013: Automating Performance Tests for Android Applications

안드로이드 앱의 성능 테스트,
자동화가 가능할까?
이경민 / LG전자

About Me

Android Platform
SDK

About Me

Android Platform

SDET

SDK

About Me

webOS

Android Platform

SDET

SDK

About Me

webOS

Android Platform

SDET
Test Automation

SDK

About Me

webOS
kandroid.org

Android Platform

SDET

Test Automation

SDK

About Me

webOS
kandroid.org

Android Platform

SDET

SDK

Test Automation
snailee

CONTENTS
Why Automated Performance Testing?
How to Automate Performance Testing?
- Historical Analysis w/ kmemtracer
- Measuring Performance
- Controlling UI

If you can’t measure it,
you can’t manage it.
Peter F. Drucker
(1909~2005)

If you can’t measure it,
you can’t manage it.
Peter F. Drucker
(1909~2005)

If you can’t automate
the measurement of it,
it is difficult to improve it.

Diversity or Fragmentation
Platform
Versions

http://developer.android.com/about/dashboards/index.html

Diversity or Fragmentation
Platform
Versions

Screen Sizes
and Densities

http://developer.android.com/about/dashboards/index.html

http://www.mactrast.com/2012/05/visualized-the-truth-about-android-fragmentation/

- Controlling UI

Why Historical Analysis?

Source: Memory and Performance at the 11th Korea Android Conference

Why Historical Analysis?

Activity
Navigation
& Lifecycle
Source: Memory and Performance at the 11th Korea Android Conference

How to Track Activity Navigation & Lifecycle?


Override onCreate, onStart, …
of your Activity classes


Modify android.app.Activity
in the framework


Is there a way that does not
require the modification of
an application or framework?


Is there a way that does not
require the modification of
an application or framework?

Instrumentation

Android instrumentation is a set of control methods
or "hooks" in the Android system.
These hooks control an Android component
independently of its normal lifecycle.
They also control how Android loads applications.
(http://developer.android.com/tools/testing/testing_android.html)

Historical Analysis w/ kmemtracer
Application
Package

Control
android.test.
Instrumentation
TestRunner

Test
Package

Historical Analysis w/ kmemtracer
https://github.com/snailee/kmemtracer-libs

Application
Package

Control

Track

android.test.
Instrumentation
TestRunner

org.kandroid.
memtracer.Memory
Instrumentation

Test
Package

Trace
Package

android.app.Instrumentation
void
void
void
void
void
void
void
void
void
void
void
void
void

callActivityOnCreate(Activity, Bundle)
callActivityOnDestroy(Activity)
callActivityOnNewIntent(Activity, Intent)
callActivityOnPause(Activity)
callActivityOnPostCreate(Activity, Bundle)
callActivityOnRestart(Activity)
callActivityOnRestoreInstanceState(Activity, Bundle)
callActivityOnResume(Activity)
callActivityOnSaveInstanceState(Activity, Bundle)
callActivityOnStart(Activity)
callActivityOnStop(Activity)
callActivityOnUserLeaving(Activity)
callApplicationOnCreate(Application)

void
void
void
void
void
void
void
void
void
void
void
void
void


Call for
Control

void
void
void
void
void
void
void
void
void
void
void
void
void


Call for
Control

Override for
Track

org.kandroid.memtracer.MemoryInstrumentation
public class MemoryInstrumentation extends Instrumentation {
private static final String ARGUMENT_INSTRUMENT_CLASS = "class";
private MemoryTracer mMemoryTracer;
private String mMainActivityName;
...
@Override
public void onCreate(Bundle arguments) {
mMainActivityName = arguments.getString(ARGUMENT_INSTRUMENT_CLASS);
mMemoryTracer = createMemoryTracer();
mMemoryTracer.startTracing(getTargetContext().getPackageName());
start();
}
@Override
public void callActivityOnCreate(Activity activity, Bundle icicle) {
String tag = activity.getLocalClassName()+"-OnCreate";
Bundle snapshot = mMemoryTracer.addSnapshot(tag);
super.callActivityOnCreate(activity, icicle);
}

How to Use kmemtracer?
1. Create an Android Test Project for the trace package.

2. Add ‘kmemtracer-libs.jar’ in the ‘libs’ directory.

3. Edit the <instrumentation> element in the Manifest file.
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.example.android.apis.test"
android:versionCode="1"
android:versionName="1.0">
<instrumentation
android:name="org.kandroid.memtracer.MemoryInstrumentation"
android:targetPackage="com.example.android.apis" />
<uses-permission
android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
<application android:label="@string/app_name"
android:icon="@drawable/ic_launcher“>
<uses-library android:theme="android.text.runner“>
</application>
</manifest>

4. Install the trace package as well as the application package.
$ adb install –r ApiDemos.apk
$ adb install –r ApiDemosTest.apk


5. Start the instrumentation with ‘am instrument’ in the shell.
$
>
>
>

adb shell am instrument -e class
com.example.android.apis.ApiDemos
com.example.android.apis.tests/


$
>
>
>


6. Do interact with the application.


$
>
>
>


6. Do interact with the application.
7. Pull the trace file ‘kmemtrace.csv’ from /sdcard/kmemtracer.
$ adb pull /sdcard/kmemtracer/kmemtrace.csv .

8. Open the trace file with Excel and create charts.

What to Measure?

Rendering (or GUI) Performance
Frame Rate (FPS), Frame Time , Jankiness, Input Latency, …

Resource Consumption
Memory, CPU, I/O (Disk, Network), Battery, …

Network Performance
Response time, Throughput, …

…

How to Measure Resource Consumption?


API

cmds


cmds

API

Public

Internal


cmds

API

Public

Internal

Android

Linux


cmds

API

Public

Easy

Internal

Android

Linux

Flexible

Measuring Memory Usage
Public
android.app.ActivityManager.MemoryInfo
android.os.Debug
android.os.Debug.MemoryInfo

Public

Internal

android.os.Debug
dalvik.system.VMDebug

Public

Internal

Android

android.os.Debug
adb shell dumpsys meminfo [pid|pname]
adb shell procrank

Public

Internal

Android

Linux

android.os.Debug
adb shell dumpsys meminfo [pid|pname]
adb shell procrank

adb shell cat /proc/meminfo
adb shell cat /proc/<pid>/[s]maps
adb shell cat /proc/vmstat
adb shell top
adb shell ps

Measuring CPU Usage
Public
android.os.Process
android.os.Debug.InstructionCount

Measuring CPU Usage
Public

Internal

android.os.Process

com.android.internal.os.ProcessStats

Measuring CPU Usage
Public

Internal

Android

android.os.Process

adb shell dumpsys cpuinfo

Measuring CPU Usage
Public

Internal

Android

android.os.Process

adb shell dumpsys cpuinfo
adb shell cat /proc/stat
adb shell top

Linux

Measuring I/O Usage
Public
android.net.TrafficStats

Measuring I/O Usage
Public

Internal

android.net.NetworkStats
android.net.NetworkStatsHistory

Measuring I/O Usage
Public

Internal

Android

adb shell dumpsys netstats
adb shell dumpsys diskstats

Measuring I/O Usage
Public

Internal

Android

adb shell dumpsys netstats
adb shell dumpsys diskstats
/proc/net/*

Linux

Measuring Battery Usage
Public
android.os.BatteryManager

Public

Internal


android.os.BatteryStats
com.android.internal.os.PowerProfile

Public

Internal

Android


adb shell dumpsys battery
adb shell dumpsys batteryinfo

Public

Internal

Android


adb shell dumpsys battery
adb shell dumpsys batteryinfo
None (?)

Linux

Tracing Performance Snapshots in kmemtracer

...
...
@Override
...
...
}
...

Create an instance
of MemoryTracer

...
...
@Override
...
...
}
...
@Override
public void callActivityOnCreate(Activity activity, Bundle icicle) {
String tag = activity.getLocalClassName()+"-OnCreate";
Bundle snapshot = mMemoryTracer.addSnapshot(tag);
...
}
...
}

Create an instance
of MemoryTracer

Add performance snapshot

MemoryTracer$addSnapshot
public Bundle addSnapshot(String label) {

...
long nativeMax = Debug.getNativeHeapSize() / 1024;
long nativeAllocated = Debug.getNativeHeapAllocatedSize() / 1024;
long nativeFree = Debug.getNativeHeapFreeSize() / 1024;
Debug.MemoryInfo memInfo = new Debug.MemoryInfo();
Debug.getMemoryInfo(memInfo);
...

Get usage data

...
...
Bundle snapshot = new Bundle();
snapshot.putString(METRIC_KEY_LABEL, label);
...
snapshot.putLong(METRIC_KEY_NATIVE_SIZE, nativeMax);
snapshot.putLong(METRIC_KEY_NATIVE_ALLOCATED, nativeAllocated);
snapshot.putLong(METRIC_KEY_NATIVE_FREE, nativeFree);
...

Get usage data

Save data to
a bundle

...
...
Bundle snapshot = new Bundle();
snapshot.putString(METRIC_KEY_LABEL, label);
...
snapshot.putLong(METRIC_KEY_NATIVE_SIZE, nativeMax);
snapshot.putLong(METRIC_KEY_NATIVE_ALLOCATED, nativeAllocated);
snapshot.putLong(METRIC_KEY_NATIVE_FREE, nativeFree);
...
if (mResultsWriter != null) {
mResultsWriter.writeTraceSnapshot(snapshot);
}
return snapshot;
}

Get usage data

Save data to
a bundle

Write date using
ResultsWriter

MemoryTracerCsvWriter$writeTraceSnapshot
public class MemoryTraceCsvWriter implements MemoryTracer.ResultsWriter {

...
public MemoryTraceCsvWriter(String[] metricKeys) {
mMetricKeys = metricKeys;
}
...

Create an instance
with metrics to print

...
}
...
private void openTraceFile(String filename) {
...
mTraceOut = new PrintStream(new FileOutputStream(traceFile));
...
}
...

Create an instance
Open a trace file in
the sdcard at init

...
}
...
private void openTraceFile(String filename) {
...
mTraceOut = new PrintStream(new FileOutputStream(traceFile));
...
}
...
@Override
public void writeTraceSnapshot(Bundle snapshot) {
PrintStream out = mTraceOut;
for (String key : mMetricKeys) {
out.print(snapshot.get(key));
out.print(',');
}
out.println();
out.flush();
}
...
}

Create an instance
Open a trace file in
the sdcard at init
Write snapshot to
a line of the file

To Do’s

Add more tracers
CpuTracer
IOTracer
BatteryTracer

To Do’s

Add more tracers
CpuTracer
IOTracer
BatteryTracer

Make metrics to gather configurable

To Do’s

Add more tracers
CpuTracer
IOTracer
BatteryTracer

Make trace output destination configurable
Trace out to internal file (not on sdcard)
Trace out to an analytics server

To Do’s

Add more tracers
CpuTracer
IOTracer
BatteryTracer

Make trace output destination configurable
Trace out to internal file (not on sdcard)
Trace out to an analytics server

Welcome contributions
https://github.com/snailee/kmemtracer-libs

Automate Controlling UI
Application
Package

Control

Track

android.test.
Instrumentation
TestRunner

org.kandroid.
memtracer.Memory
Instrumentation

Test
Package

Trace
Package

Three Alternatives

Instrumentation
API Level 1 ~
Single activity
Framework APIs
White-box

http://developer.android.com/tools/testing/testing_android.html

Three Alternatives

Instrumentation
API Level 1 ~
Single activity
Framework APIs
White-box

Robotium
2010/01 ~
Multiple activities
High-level APIs
White-box & black-box

https://code.google.com/p/robotium/

Three Alternatives

Instrumentation
API Level 1 ~
Single activity
Framework APIs
White-box

Robotium
uiautomator
2010/01 ~
Multiple activities
API Level 17 ~
High-level APIs
White-box & black-box Multiple processes
Simple APIs
Black-box

https://code.google.com/p/robotium/
http://developer.android.com/tools/testing/testing_ui.html

Case Study: ApiDemos
https://github.com/snailee/deview-2013-samples


Click


Click

Scroll
down


Click

Scroll
down
Click


Click

Scroll
down
Click

Click

Controlling UI Using Instrumentation

1. Create an Android Test Project for the test package.
$ android create test-project --main ../ApiDemos
> –-name ApiDemosInstrumentTest --path ApiDemosInstrumentTest


2. Add a test class inheriting ActivityInstrumentationTestCase2.
public class ApiDemosInstrumentTest extends
ActivityInstrumentationTestCase2<ApiDemos> {
...
public ApiDemosInstrumentTest() {
super(ApiDemos.class);
}


public class ApiDemosInstrumentTest extends
...
public ApiDemosInstrumentTest() {
}

3. Add a test method and launch the activity.
public void testNavigate() {
final ApiDemos apiDemos = getActivity();
...
}

4. Find and act on a control.


Find a view
to control

...
final ListView demoList = apiDemos.getListView();
final View graphicsDemo = getChildViewByText(demoList, "Graphics");
assertNotNull("Graphics demo should exist", graphicsDemo);


Find a view
to control

...
final ListView demoList = apiDemos.getListView();
final View graphicsDemo = getChildViewByText(demoList, "Graphics");
assertNotNull("Graphics demo should exist", graphicsDemo);
apiDemos.runOnUiThread(new Runnable() {
public void run() {
int pos = demoList.getPositionForView(graphicsDemo);
demoList.smoothScrollToPosition(pos);
demoList.setSelection(pos);
demoList.performItemClick(graphicsDemo, pos,
graphicsDemo.getId());
}
});
...

Act on
the view

4. Find and act on a control: getChildViewByText.
private View getChildViewByText(ListView listView, String text) {

View view = null;
int count = listView.getCount();
Log.e(TAG, “no. children = " + count);
for (int i = 0; i < count; i++) {
TextView textView = (TextView) listView.getChildAt(i);
Log.e(TAG, i + "-th text view is " + textView);

Iterate through
child views

View view = null;
if (textView != null && textView.getText().equals(text)) {
view = textView;
break;
}

Iterate through
child views

Compare
texts

View view = null;
view = textView;
break;
}
}
return view;
}

Iterate through
child views

Compare
texts

View view = null;
view = textView;
break;
}
}
return view;
}
com.example.android.apis.instrument.test.ApiDemosInstrumentTest:
INSTRUMENTATION_RESULT: shortMsg=junit.framework.AssertionFailedError
INSTRUMENTATION_RESULT: longMsg=
junit.framework.AssertionFailedError: OpenGL ES demo should exist
INSTRUMENTATION_CODE: 0

Iterate through
child views

Compare
texts

4. Find and act on a control: Revised getChildViewByText.
private View getChildViewByText(final ListView listView, String text) {
...
view = textView;
break;
}
if (textView == null) {
final int lastPos = listView.getLastVisiblePosition();
getInstrumentation().runOnMainSync(new Runnable() {
public void run() {
listView.smoothScrollToPosition(lastPos + 5, lastPos);
}
});
getInstrumentation().waitForIdleSync();
i = listView.getFirstVisiblePosition();
}
}
return view;
}

4. Find and act on a control: Revised getChildViewByText.
private View getChildViewByText(final ListView listView, String text) {
...
view = textView;
break;
}
if (textView == null) {
final int lastPos = listView.getLastVisiblePosition();
getInstrumentation().runOnMainSync(new Runnable() {
public void run() {
listView.smoothScrollToPosition(lastPos + 5, lastPos);
}
});
getInstrumentation().waitForIdleSync();
i = listView.getFirstVisiblePosition();
}
}
return view;
}

Scroll down

5. Monitor a new Activity to launch.
...
Instrumentation.ActivityMonitor graphicsDemoActivityMonitor =
getInstrumentation().addMonitor(
ApiDemos.class.getName(), null, false);
final ApiDemos graphicsDemoActivity =
(ApiDemos) graphicsDemoActivityMonitor.waitForActivity();
assertNotNull("Graphics Demo (ApiDemos) activity should not be null",
graphicsDemoActivity);
...

5. Monitor a new Activity to launch.
...
Instrumentation.ActivityMonitor graphicsDemoActivityMonitor =
getInstrumentation().addMonitor(
ApiDemos.class.getName(), null, false);
final ApiDemos graphicsDemoActivity =
(ApiDemos) graphicsDemoActivityMonitor.waitForActivity();
assertNotNull("Graphics Demo (ApiDemos) activity should not be null",
graphicsDemoActivity);
...

6. Iterate step 4 & 5.

7. Build and Install the test package.
$ cd ApiDemosInstrumentTest
$ ant debug
$ adb install –r bin/ApiDemosInstrumentTest-debug.apk

$ ant debug

8. Run the test and see the result.
$ adb shell am instrument -w
> com.example.android.apis.instrument.test/
> android.test.InstrumentationTestRunner

$ ant debug


$ ant test

Controlling UI Using Robotium
> –-name ApiDemosRobotiumTest --path ApiDemosRobotiumTest


2. Add ‘robotium-solo-<ver>.jar’ in the ‘libs’ directory.


2. Add ‘robotium-solo-<ver>.jar’ in the ‘libs’ directory.
public class ApiDemosRobotiumTest extends
...
public ApiDemosRobotiumTest() {
}
...

4. Create an instance of Solo in the setUp method and
close open activities in the tearDown method.
...
private Solo solo;
...
@Override
public void setUp() throws Exception {
solo = new Solo(getInstrumentation(), getActivity());
}

@Override
public void tearDown() throws Exception {
solo.finishOpenedActivities();
}
...

solo.assertCurrentActivity("Should be on ApiDemos",
ApiDemos.class);
solo.clickOnText("^Graphics$", 1, true);
solo.assertCurrentActivity("Should be on a new ApiDemos",
ApiDemos.class, true);
solo.clickOnText("^OpenGL ES$", 1, true);
solo.assertCurrentActivity("Should be on a new ApiDemos",
ApiDemos.class, true);
solo.clickOnText("^Kube$", 1, true);
solo.assertCurrentActivity("Should be on a Kube",
Kube.class, true);
solo.sleep(3000);
}

$ cd ApiDemosRobotiumTest
$ ant debug
$ adb install –r bin/ApiDemosRobotiumTest-debug.apk

$ ant debug

> com.example.android.apis.robotium.test/

$ ant debug

> com.example.android.apis.robotium.test/

$ ant test

Controlling UI Using uiautomator
1. Create an Java Project for the test jar.

2. Add ‘android.jar’ and ‘uiautomator.jar’ in the build path.

2. Add ‘android.jar’ and ‘uiautomator.jar’ in the build path.
3. Add a test class inheriting UiAutomatorTestCase.
public class ApiDemosUiAutoTest extends UiAutomatorTestCase {
...
}

4. Launch ApiDemos.
public void testNavigate() throws UiObjectNotFoundException {

4. Launch ApiDemos.
getUiDevice().pressHome();

Press home button

4. Launch ApiDemos.

UiObject allAppsButton = new UiObject(
new UiSelector().description("Apps"));
allAppsButton.clickAndWaitForNewWindow();

Press home button
Click ‘All Apps’
button

4. Launch ApiDemos.


Press home button
button

UiObject appsTab = new UiObject(new UiSelector().text("Apps"));
appsTab.click();

Click ‘Apps’ tab

4. Launch ApiDemos.


Press home button
button

appsTab.click();


UiScrollable appList = new UiScrollable(
new UiSelector().scrollable(true));
appList.setAsHorizontalList();
UiObject apiDemos = appList.getChildByText(
new UiSelector().className(TextView.class.getName()), "API Demos");
apiDemos.clickAndWaitForNewWindow();

Click ‘ApiDemos’
icon

4. Launch ApiDemos.


Press home button
button

appsTab.click();


UiScrollable appList = new UiScrollable(
new UiSelector().scrollable(true));
appList.setAsHorizontalList();
UiObject apiDemos = appList.getChildByText(
new UiSelector().className(TextView.class.getName()), "API Demos");
apiDemos.clickAndWaitForNewWindow();

Click ‘ApiDemos’
icon

Validate the launch
of ‘ApiDemos’

UiObject apiDemosPackage = new UiObject(
new UiSelector().packageName("com.example.android.apis"));
assertTrue(“Should be on ApiDemos", apiDemosPackage.exists());
...

...
UiScrollable demoList = new UiScrollable(
new UiSelector().className("android.widget.ListView"));

...
UiObject graphicsDemo = demoList.getChildByText(
new UiSelector().className(TextView.class.getName()), “Graphics");

...
graphicsDemo.clickAndWaitForNewWindow();

...
UiObject alphaBitmapDemo = new UiObject(
new UiSelector().text(“AlphaBitmap"));
assertTrue(“AlphaBitmap should be visible", alpahBitmapDemo.exists());
...

...
UiObject alphaBitmapDemo = new UiObject(
new UiSelector().text(“AlphaBitmap"));
assertTrue(“AlphaBitmap should be visible", alpahBitmapDemo.exists());
...

6. Iterate step 5.

7. Compile your test case into a JAR file.
$
>
$
$

android create uitest-project --name ApiDemosUiAutoTest
--target 1 --path ApiDemosUiAutoTest
cd ApiDemosUiAutoTest
ant build

$
>
$
$

ant build

8. Install the jar on your test device.
$ adb push bin/ApiDemosUiAutoTest.jar /data/local/tmp/

$
>
$
$

ant build


9. Run the test and view the test result.
$ adb shell uiautomator runtest ApiDemosUiAutoTest.jar
> –c com.example.android.apis.uiauto.test.ApiDemosUiAutoTest

$
>
$
$

ant build


9. Run the test and view the test result.
$ adb shell uiautomator runtest ApiDemosUiAutoTest.jar
> –c com.example.android.apis.uiauto.test.ApiDemosUiAutoTest

$ ant test

Doing both Controlling and Tracking

1. Modify org.kandroid.memtracer.MemoryInstrumentation.
public class MemoryInstrumentation extends InstrumentationTestRunner {
...
@Override
...
start();
super.onCreate(arguments);
}
@Override
public void onStart() {
String mainActivityName = getMainActivityName();
if (mainActivityName != null && mainActivityName.length() > 0) {
launchMainActivity(getTargetContext().getPackageName(),
getMainActivityName());
} else {
super.onStart();
}
}
...

3. Edit the <instrumentation> element in the manifest file.

$ ant debug

$ ant debug

> org.kandroid.memtracer.MemoryInstrumentation

$ ant debug

> org.kandroid.memtracer.MemoryInstrumentation

6. Pull the trace file ‘kmemtrace.csv’ from /sdcard/kmemtracer.
$ adb pull /sdcard/kmemtracer/kmemtrace.csv .

SUMMARY
Measure to improve / Fragmentation

SUMMARY

Instrumentation for track

SUMMARY

API (Public / Internal) / cmds (Android / Linux)

SUMMARY

API (Public / Internal) / cmds (Android / Linux)
- Controlling UI
Instrumentation / Robotium / uiautomator

DEVIEW2013: Automating Performance Tests for Android Applications

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to DEVIEW2013: Automating Performance Tests for Android Applications

Similar to DEVIEW2013: Automating Performance Tests for Android Applications (20)

Recently uploaded

Recently uploaded (20)

DEVIEW2013: Automating Performance Tests for Android Applications