This document presents a case study analyzing factors that contribute to high and low ratings of free Android apps. It examines 28 factors across dimensions like app size, code complexity, library dependence, UI complexity, user requirements, and marketing efforts. The study finds that high-rated apps differ significantly from low-rated apps in 17 factors. The top 3 most influential factors for predicting ratings are app size, number of promotional images, and target SDK version. While some prior studies identified other factors like library quality and UI complexity, this multi-factor analysis shows those are less important than size and marketing efforts. Future work could explore additional factors and conduct more fine-grained analysis by app category.

1. What Are the Characteristics of High-rated Apps?
A Case Study on Free Android Applications
David LoYuan Tian Meiyappan
Nagappan
Ahmed E.
Hassan

2. Dramatic mobile app market & mobile app usage
growth in recent years.
“
”
In 2012 the app
economy was worth
$53Bn and is expected
to expand at a 28%
CAGR up to 2016,
reaching $143Bn.
2

3. Mobile app market is attractive, however building
a successful app is hard.
Highly Competitive:
With other 50,000 apps in the
app store
3
Limited Chance:
Smartphone users stick to
limited mobile apps.

4. How to make a successful app?
- Let’s contrast high- and low-rated apps!

5. Prior Studies
• Bavota et al. find that low-rated apps have method calls
to APIs that are more change or fault prone.
(Bavota et al., TSE 2015)
• Taba et al. find that low-rated apps have more complex
user interface. (Taba et al., ICWE 2014)
• Chia et al. find that number of required permissions
impacts app ratings (Chia et al. WWW 2012)
• Many other studies.
5

6. This Study
• Consider a comprehensive set of factors that may
impact app ratings instead of only a few.
‒ Consider existing factors
‒ Plus additional factors not considered before
• Compare the relative importance of each of the factors
on app ratings
‒ In predicting high-rated and low-rated apps
‒ On a reasonably large dataset of more than 1000 apps
6

7. Agenda
7
• Motivation
• Factors
• Case study
• Discussion
• Conclusion

8. We analyze 28 factors in 8 dimensions
Cate-
gory
App
Size
Code.
Comp.
Library
Depend.
Library
Quality
UI
Comp.
User
Req.
Market-
ing
8

9. App Size Dimension
Why App Size?
+ Large code => richer
functionality
- Larger code => higher
chance for bugs
(Zimmermann, 2007)
9
Cate-
gory
App
Size
Code.
Comp.
Library
Depend.
Library
Quality
UI
Comp.
User
Req.
Market-
ing

10. 10
Why Code
Complexity?
+ More complex code
=> more advanced
functionality
- More complex code
=> more bugs
(Subramanyam, 2003)
Cate-
gory
App
Size
Code.
Comp.
Library
Depend.
Library
Quality
UI
Comp.
User
Req.
Market-
ing
Code Complexity Dimension

11. 11
Why Library
Dependence?
+ Higher dependence =>
richer functionality built
upon third party code
- Higher dependence =>
difficulty to keep up with
library evolution, which
results in bugs.
(Syer, 2014)
Cate-
gory
App
Size
Code.
Comp.
Library
Depend.
Library
Quality
UI
Comp.
User
Req.
Market-
ing
Library Dependence Dimension

12. 12
Why Library Quality?
- Buggy library code =>
buggy apps built on
top of them.
- Frequently changed
libraries => bugs if
apps are not properly
maintained
Cate-
gory
App
Size
Code.
Comp.
Library
Depend.
Library
Quality
UI
Comp.
User
Req.
Market-
ing
Library Quality Dimension

13. 13
Why UI Complexity?
- More complex UI => app
is harder to use.
+ More complex UI =>
more functionality.
Cate-
gory
App
Size
Code.
Comp.
Library
Depend.
Library
Quality
UI
Comp.
User
Req.
Market-
ing
UI Complexity Dimension

14. 14
Why User
Requirements?
+ Larger target SDK
version => incorporation
of latest feature, active
maintenance effort.
+ Number of permission
request => more
features.
- Number of permission
request => privacy risk.
User Requirement Dimension
Cate-
gory
App
Size
Code.
Comp.
Library
Depend.
Library
Quality
UI
Comp.
User
Req.
Market-
ing

15. 15
Why Marketing Effort?
+ More marketing effort
=> better first impression,
more functionality.
Marketing Dimension
Cate-
gory
App
Size
Code.
Comp.
Library
Depend.
Library
Quality
UI
Comp.
User
Req.
Market-
ing

16. 16
Why Category?
Different category
=> different user
expectation.
Category Dimension
Cate-
gory
App
Size
Code.
Comp.
Library
Depend.
Library
Quality
UI
Comp.
User
Req.
Market-
ing

17. Factors in App Size Dimension
• Binary size of the APK file (measured in KB)
Install Size
• Total number of classes (including library code).
Total classes
• Total number of app specific classes.
App classes
• Total number of activities defined in the
AndroidManifest.xml file.
# Activities
• Total number of services defined in the
AndroidManifest.xml file.
# Services
17

18. Factors in Code Complexity Dimension
• Chidamber and Kemerer’s object oriented complexity metrics,
e.g., the number of methods in each class.
• Note that we compute the mean over all classes in each app.
Six CK metrics
• Mean of the number of other classes that depend upon each
class.
Afferent coupling
• Mean of the number of public methods in each class.
Number of public methods
18

19. Factors in Library Dependence Dimension
• Total number of (percentage of) calls to libraries that start
with “android.”.
Absolute (percentage) dependence on Android
• Total number of (percentage of) calls to third party libraries.
19
Absolute (percentage) dependence on third party libraries

20. Factors in Library Quality Dimension
• Mean number of methods changed in used Android APIs
(Bavota et al. 2015)
Change of used Android APIs
• Mean number of bugs in the used Android APIs
(Bavota et al. 2015)
Faultiness of used Android API
20

21. Factors in UI Complexity Dimension
• Mean number of input elements per layout.
Input elements per layout
• Mean number of output elements per layout.
Output elements per layout
21

22. 22
Factors in User Requirements Dimension
• The minimum SDK version required for the app to run.
Minimum SDK version
• The SDK version that the app targets. If not set, the default
value equals to minSDK.
Target SDK version
• Number of required features from user’s device (e.g., camera).
Required device features
• Number of permissions needed from user.
Required user permission

23. Factors in Marketing Effort Dimension
• Number of words appearing in the description of the app in its
Play Store page.
Length of description
• Number of images shown on the app’s store page.
Promotional images
23

24. 24
All the factors could be calculated by using tools including:
ApkTool, dex2jar, BCEL based on app apk and info on app store.

25. 25
Meta Data
Extract: Category, Size
Rating, Rating Count
Google Play
Extract:
Marketing
Extract: Size
APKs
ApkTool
AndroidManifest
Files, Resource
Extract:
Requirements on Users, UI
BCEL
dex2jar
Extract: Code Complexity, Library
Dependence, quality of library Code
Jars
Android API Change
and Bug Logs
Step 1
Step 2 Step
3

26. 26
Meta Data
Extract :
Marketing
Extract :
Category, Size
Rating, Rating Count
Google Play
Step 1

27. 27
Extract: Size
Step 2
APKsApkTool
AndroidManifest
Files, Resource
Extract:
Requirements on Users, UI

28. 28
BCEL
Step 3
APKs dex2jar
Extract: Code Complexity,
Library Dependence, quality of library Code
JarsExtract: Size
Android API Change
and Bug Logs

29. Our case study is done on 1,492 android apps:
29
Step 1: Randomly selected and
crawled 10,000 apps.
Step 2: Filter out apps that:
‒ Have less than 10 ratings
‒ Could not be processed by
our tools
Step 3: Sort apps in each category by
their ratings. Select the top
10% (high-rated) and bottom
10% (low-rated) apps.

30. Research Questions
RQ2: What are the important factors that could be
used to predict app ratings?
RQ1: Is there a relationship between each factor
an app rating?
30

31. 31
RQ1: Relation between factors and rating
• Compare the values of each factor between
high-rated and low rated apps.
• Analyze the statistical significance and effect
size of the difference between the two groups of
apps.
‒ Use Mann-Whitney U test at p-value of 0.01.
‒ Compute Cliff’s Delta (or d).

32. 32
-ve
+ve
+ve
RQ1: Relation between factors and rating

33. RQ1: Summary of Findings
High-rated apps are statistically significantly
different from low-rated apps in 17 out of the 28
factors.
Generally, high-rated apps are larger with more
complex code, more preconditions, more
marketing efforts, more dependence on
libraries, and they make use of higher quality
Android libraries.
33

34. 34
RQ2: Important factors for prediction
• Remove highly correlated factors
• Remove redundant factors
• Use random forest
‒ Ten fold cross validation
‒ Measure performance in terms of F1 and AUC
‒ Repeat the process with different factors omitted
• Employ Scott-Knott test
‒ Identify groups of factors that are statistically
significantly different from one another

35. The RF model
could achieve:
F-measure of
0.74 + AUC
of 0.81
RQ2: Important factors for prediction
35
Top-3
Size of the app
# Promotional
Images
Target SDK
RQ2: Important factors for prediction

36. RQ2: Summary of Findings
The size of an app, the number of promotional
images on its store page, and the target SDK are
the three most influential factors in determining the
likelihood of an app being a high rated app.
36

37. Discussion: Comparison with Past Findings
• Reinforce findings by Bavota et al. (TSE’15)
‒ API quality influence rating
‒ However, it is not the most important factors (#5)
• Refute findings by Taba et al. (ICWE’14)
‒ UI complexity is not statistically significantly related
with app ratings.
• Reinforce findings by Chia et a. et al. (WWW’12)
‒ # required permission is weakly associated with rating
• Highlight additional factors: app size, promotion
effort, target sdk.
37

38. Discussion: Power of Multi-Factor Analysis
0
0.2
0.4
0.6
0.8
Precision Recall F-measure
All Size
Code Complexity Dependence
Library Quality UI Complexity
Requirement on User Marketing
Category
38
Single dimension factors are not enough to successfully
differentiate high-rated from low-rated apps.
0.7

39. 39

40. 40
Future Work
• Explore additional factors
• Do a more fine-grained study (individual category)
• Employ a causality analysis to get a deeper
understanding
• Interview Android app developers to get their
insight

41. Questions? Comments? Advice?
{yuan.tian.2012,davidlo}@smu.edu.sg
mei@se.rit.edu, ahmed@cs.queensu.ca
41
Thank You !