1. AppFunnel: A Framework for Usage-centric
Evaluation of Recommender Systems
that Suggest Mobile Applications
Matthias Böhmer
Lyubomir Ganev
Antonio Krüger

2. Introduction
‣ Number of mobile applications steadily increasing
‣ Finding good apps can become a difficult task
‣ Recommender systems can help
- Mobile apps special type of items
- Context is important
‣ „Dead apps“ on smartphones
- Less than 1/2 of apps are actively used
- Installation counts good for evaluation?

3. Our paper...
(1) Introduces a usage-centric evaluation framework
(2) Presents results of a case study
3

4. ‣ AppFunnel
4

5. Concept of AppFunnel
‣ Idea: evaluate recommendation based on app engagement
‣ Concept of funnels adopted from advertising domain
‣ Apps can reach different stages after recommendation
LONG-TERM
VIEW INSTALLATION DIRECT USAGE
USAGE
RECOMMENDATION
‣ Tracking engagement with apps along stages

6. ‣ Case Study
6

7. Testbed: appazaar
‣ Recommender system for Android applications
‣ Available for end-users on Google Play Market for free
‣ 6,680+ users of appazaar, worldwide distribution
‣ 45 users contributed to case study over three months

8. Recommender Engines under Test
Non-personalized Personalized
Context-less - App popularity - Usage-based CF
Context-aware - App-aware filtering - Location-aware CF
- Time-aware CF

9. Conversion Stages
AppFunnel stage
view view market installation direct usage long−term usage
Average number of occurrences
per recommendation list
1.0
0.8
0.6
0.4
0.2
0.0
rity
App−popularity CF
Usage−based
a
App−awarere CF CF
Time−aware Location−aware
a ed aw e e
ul
Collaborative Collaborative Collaborative
p- ar ar
asFiltering
Filtering Filtering
op b p aw Filtering -aw Filtering
p -p e- A e- n
p g m io
A sa Ti at
U
L oc

10. Conversion Rates
Conversion
view to installation installation to direct usage installation to long−term usage
100
Conversion rates in percent
90
80
70
60
50
40
30
20
10
0
ity F
Usage−based
C re CF
Time−aware Location−aware
CF
r wa
App−popularity App−aware
u la d
Collaborative
e a re re
Collaborative Collaborative
as Filtering pp- waFiltering waFiltering
p Filtering Filtering
o -a -a
-p -b A e n
pp ge m io
A sa Ti at
U
L oc

11. Results of Case Study
‣ AppFunnel shows difference in engines‘ performances, e.g.
- Context-less engines: more views
- Context-aware engines: better from installation to direct use
‣ AppFunnel enables validation of engines‘ design goals, e.g.
- App-aware engine: high installation to direct-usage rate
- Context-less engines: higher rates for long-term usage
‣ AppFunnel helped us to trace other issues of engines, e.g.
- Location-aware engine: many views but no installations

12. Limitations
‣ Recommender engines under test are rather simplistic
‣ Updates and removal not taken into account
- Update does not tell much about app engagement
- Removal events needs further refinement of AppFunnel
‣ Future work:
- Improve recommender engines by incorporating
results of case study

13. Discussion
‣ Choosing from AppFunnel‘s metrics
- Metrics might counteract (e.g., direct vs. long-term usage)
- Metrics should reflect design goal (context -> direct usage)
‣ Usage-centric evaluation valuable for other domains
- When engagement with item can be traced
- Changing view from „selling items“ to UX of items

14. ‣ Conclusion
14

15. Conclusion
‣ We presented AppFunnel
- Usage-centric evaluation of recommender engines
- Recommendation of mobile applications
‣ We presented a case study of AppFunnel in the wild
- Traces down performance beyond installations
- Shows differences between engines
LONG-TERM
VIEW INSTALLATION DIRECT USAGE
USAGE
RECOMMENDATION