AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications
AppFunnel: A Framework for Usage-centricEvaluation of Recommender Systemsthat Suggest Mobile ApplicationsMatthias BöhmerLyubomir GanevAntonio Krüger
Introduction‣ Number of mobile applications steadily increasing‣ Finding good apps can become a diﬃcult task‣ Recommender systems can help - Mobile apps special type of items - Context is important‣ „Dead apps“ on smartphones - Less than 1/2 of apps are actively used - Installation counts good for evaluation?
Our paper...(1) Introduces a usage-centric evaluation framework(2) Presents results of a case study 3
Concept of AppFunnel ‣ Idea: evaluate recommendation based on app engagement ‣ Concept of funnels adopted from advertising domain ‣ Apps can reach diﬀerent stages after recommendation LONG-TERM VIEW INSTALLATION DIRECT USAGE USAGERECOMMENDATION ‣ Tracking engagement with apps along stages
Testbed: appazaar‣ Recommender system for Android applications‣ Available for end-users on Google Play Market for free‣ 6,680+ users of appazaar, worldwide distribution‣ 45 users contributed to case study over three months
Conversion Stages AppFunnel stage view view market installation direct usage long−term usageAverage number of occurrences per recommendation list 1.0 0.8 0.6 0.4 0.2 0.0 rity App−popularity CF Usage−based a App−awarere CF CF Time−aware Location−aware a ed aw e e ul Collaborative Collaborative Collaborative p- ar ar asFiltering Filtering Filtering op b p aw Filtering -aw Filtering p -p e- A e- n p g m io A sa Ti at U L oc
Conversion Rates Conversion view to installation installation to direct usage installation to long−term usage 100Conversion rates in percent 90 80 70 60 50 40 30 20 10 0 ity F Usage−based C re CF Time−aware Location−aware CF r wa App−popularity App−aware u la d Collaborative e a re re Collaborative Collaborative as Filtering pp- waFiltering waFiltering p Filtering Filtering o -a -a -p -b A e n pp ge m io A sa Ti at U L oc
Results of Case Study‣ AppFunnel shows diﬀerence in engines‘ performances, e.g. - Context-less engines: more views - Context-aware engines: better from installation to direct use‣ AppFunnel enables validation of engines‘ design goals, e.g. - App-aware engine: high installation to direct-usage rate - Context-less engines: higher rates for long-term usage‣ AppFunnel helped us to trace other issues of engines, e.g. - Location-aware engine: many views but no installations
Limitations‣ Recommender engines under test are rather simplistic‣ Updates and removal not taken into account - Update does not tell much about app engagement - Removal events needs further reﬁnement of AppFunnel‣ Future work: - Improve recommender engines by incorporating results of case study
Discussion‣ Choosing from AppFunnel‘s metrics - Metrics might counteract (e.g., direct vs. long-term usage) - Metrics should reﬂect design goal (context -> direct usage)‣ Usage-centric evaluation valuable for other domains - When engagement with item can be traced - Changing view from „selling items“ to UX of items
Conclusion‣ We presented AppFunnel - Usage-centric evaluation of recommender engines - Recommendation of mobile applications‣ We presented a case study of AppFunnel in the wild - Traces down performance beyond installations - Shows diﬀerences between engines LONG-TERM VIEW INSTALLATION DIRECT USAGE USAGERECOMMENDATION
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.