This is the slides for the talk of ICSE 2019 paper "PIVOT: Learning API-Device Correlations to Facilitate Android Compatibility Issue Detection". A link to the paper is available at: http://sccpu2.cse.ust.hk/liliwei/ICSE-main-350-CR.pdf

1. PIVOT: Learning API-Device Correlations to
Facilitate Android Compatibility Issue Detection
Lili Wei1 Yepang Liu2 Shing-Chi Cheung1
1Hong Kong University of Science and Technology, Hong Kong, China
2Southern University of Science and Technology, Shenzhen, China
May 31st, 2019

2. Many Device Manufacturers Build Devices Based on Android
2

3. Android Ecosystem is Heavily Fragmented
➤ Over 24,000 distinct Android device models
3

4. 4
Source: https://vine.co/v/MgWLMmmwUQQ
Fragmentation-Induced Compatibility Issues (Compatibility Issues)
Compatibility issues are commonly caused by inconsistent behaviour of the same
API across different device models [Wei et al. ASE 2016][Wei et al. TSE 2019]
An App Can Behave Inconsistently on Different Devices

5. Identifying Compatibility Issues is Challenging
➤ Limited documentation available for customized systems and devices
➤ App developers mainly identify compatibility issues through testing on different
device models
5Source: https://techcrunch.com/2012/06/02/android-qa-testing-quality-assurance/

6. Identifying Compatibility Issues is Challenging
➤ The search space for compatibility issue testing is combinatorially large
6
Provide remote access to real Android devices
200+ distinct Android device models
Amazon Device Farm
An Android App
[Lu et al. CCS 2012]
50+ entry methods on average
The search space is also evolving!
50 × 200 = 10,000 combinations!

7. !7
“… mostly because of this issue I am focusing more on
iOS devices, where fragmentation is minimal.”

8. Existing Work
➤ Differential Testing [Fazzini et al. ASE 2017][Ki et al. ICSE 2019]
➤ Focus more on test oracle problem but not reducing the search space
➤ Test device prioritization [Khalid et al. FSE 2014][Lu et al. ICSE 2016]
➤ Not effective in reducing the search space since they did not relate
device models with compatibility issues
➤ FicFinder [Wei et al. ASE 2016][Wei et al. TSE 2019]
➤ Detect compatibility issues via static analysis
➤ Rely on manually-extracted knowledge of compatibility issues
8
Our goal: Automatically learn knowledge of compatibility
issues to facilitate compatibility issue detection

9. Knowledge of Compatibility Issues Can Be Learned From Mature Apps
➤ Mature Android apps contain code snippets handling compatibility issues
➤ Such code snippets demonstrate common patterns
9
Camera.setRecordingHint
Can we automatically extract knowledge of compatibility
issues from such issue-handling code snippets?
“Nexus 4”

10. API-Device Correlation & Its Application Scenario
➤ PIVOT: API-DeVice cOrrelaTor
➤ Learn knowledge of compatibility issues from mature Android apps in the
form of API-device correlations
10
<Android API, Device Identifier>
<Activity.openOptionsMenu(), “LGE”>
Identify potential locations of
compatibility issues in apps
Test on potential issue-
triggering devices

11. Illustrative Example
➤ Apps can crash when pressing options menu hardware button on LG devices
11

12. Illustrative Example
➤ Apps can crash when pressing options menu hardware button on LG devices
12
A code snippet handling the compatibility issue
API
Device Identifier
?

13. Correlating APIs and Device Identifiers
13
Device condition:
A conditional statement that uses
android.os.Build
<Activity.openOptionsMenu(), “LGE”>
<Log.i(String, String), “LGE”>
<Activity.getCurrentFocus(), “LGE”>
Invalid API-device correlations:
Irrelevant to compatibility issues

14. Invalid API-Device Correlations Are Massive
➤ In our evaluation, 97% of our randomly sampled API-device correlations are
invalid
14
97%
3%
Valid Invalid
How to identify valid API-device correlations and
filter out such massive noises?

15. Mining API Preconditions & Learning API-Device Correlations
15
?
Can we apply existing API precondition mining techniques to identify
valid API-device correlations?
Yes!
No!
API usages in large app corpora can help
Existing API precondition mining techniques cannot be directly applied

16. Mining API Preconditions & Learning API-Device Correlations
16
No! Existing API precondition mining techniques cannot be directly applied
Assume that the APIs are mostly guarded by correct API preconditions across
different projects
This assumption does not hold for API-device correlations!
Compatibility issues are commonly left unhandled in practice due to the large and
evolving search space
In our evaluation: An existing API precondition mining technique identified
ONLY ONE valid API-device correlation among the top 50 in its ranked list

17. Observation: Similar Assumptions Can Hold Within the Same App
➤ Within the same app
➤ APIs related to the same compatibility issues are often guarded by the
same device conditions
➤ APIs irrelevant to compatibility issues can be invoked at other places
without device conditions
17
<Activity.openOptionsMenu(), “LGE”>
<Log.i(String, String), “LGE”>
<Activity.getCurrentFocus, “LGE”>
<Log.i(String, String), “LGE”>

18. In-App Confidence
➤ Confidence for an API-device correlation within an app
18
# callsites of the API that are guarded by the device condition
Total # callsites of the API
Input Apps
In-App
Confidence
In-App
Confidence
In-App
Confidence
In-App
Confidence
In-App
Confidence
+ + + +
In-App Confidence for an API-device correlation

19. Libraries & Cloned Methods Can Mess Up the Results
➤ Libraries and cloned methods are common among android apps
➤ Invalid API-device correlations can be cloned in different apps
➤ Difficult to distinguish valid and invalid API-device correlations recurring in
libraries and cloned methods
19
In a method cloned from VLC (a famous open-source app)
➤ Recurred in cloned methods in many different apps
➤ Is rarely used at other places
➤ High in-app confidence
<Activity.openOptionsMenu(), “LGE”>
<Log.i(String, String), “LGE”>
<Activity.getCurrentFocus, “LGE”><Activity.getCurrentFocus, “LGE”>

20. Occurrence Diversity
➤ Occurrence Diversity: Quantify the diversity of the occurrences of each API-
device correlation
➤ App-level diversity
➤ App identifier
➤ App company
➤ Method-level diversity
➤ Method package name
➤ Control-flow information: Centroid [Chen et al. ICSE’14]
➤ Metric: Shannon-Index
➤ A diversity measurement commonly leveraged in ecology to measure the
diversity of species
20

21. PIVOT - Summary
21
API-device correlation
Extractor
API-device correlation
Prioritizer
1. ————
2. ————
3. ————
Input App
Corpus
In-app confidence ✕ occurrence diversityInter-procedural static analysis
Ranked
API-device
correlations

22. Experiment Setup
➤ Evaluation subjects
➤ Top apps in each app category on Google Play (collected in 2017 & 2018)
➤ Each app corpus contains millions of classes & methods
22

23. Evaluation
23
➤ RQ1 (Effectiveness) Can PIVOT effectively identify valid API-device correlations
from large-scale Android app corpora?
➤ Baseline: Adapted from a state-of-the-art API precondition mining
technique [Ngyen et al. FSE’ 14]
➤ Rank the API-device correlations based on overall confidence
➤ Evaluation Metric: Precision@N
➤ The percentage of valid correlations among the top N (1, 5, 10. . . )
correlations in each ranked list.
➤ Ground Truth: manually-constructed ground truth
➤ An API-device correlation is considered as a true positive if it either can
cause compatibility issues or can be used to fix compatibility issues

24. RQ1 - Effectiveness
24

25. RQ1 - Effectiveness
➤ Among the top 50 API-device correlations of the two corpora
➤ 49 distinct valid API-device correlations
➤ 39 of them are not related to any compatibility issues in FicFinder’s
empirical study dataset published in 2016 [Wei et al. ASE 2016]
25

26. Evaluation
➤ RQ2 (Usefulness) Can the API-device correlations learned by PIVOT be
applied to facilitate compatibility issue detection?
➤ Case study with FicFinder
➤ Manually encode valid API-device correlations of 5 issues learned by
PIVOT to the format of the input of FicFinder
➤ Leverage FicFinder to detect compatibility issues in the latest version
of open-source Android apps
➤ Reproduce the detected compatibility issues using online device
platforms (Amazon Device Farm, WeTest)
➤ Report the detected issues their original app developers
!26

27. RQ2 - Usefulness
➤ We detected 10 previously-unknown issues in 10 open-source apps
➤ 7 of the issues have been acknowledged
➤ 4 of the issues have been fixed
➤ All of these issues cannot be detected by FicFinder 2016 version
!27
Knowledge learned by PIVOT can be leveraged to facilitate
compatibility issue detecion
Demo Issues

28. Conclusion
➤ Fragmentation-induced compatibility issues
➤ Big headache for app developers
➤ Difficult to identify: The search space is large and evolving
➤ PIVOT
➤ Effectively identify API-device correlations by learning from code snippets
handling compatibility issues in mature Android apps
➤ Identified API-device correlations can be used to detect potential
compatibility issues in Android apps
➤ All the valid API-device correlations and their corresponding compatibility
issues identified by PIVOT have been released on GitHub
28

29. Thank you!
More interesting details (including limitations and future work)
are in our paper!
Our paper, tool, and identified issues are available on our project homepage
https://ficissuepivot.github.io/Pivot/
Scan
Me!
29