Your SlideShare is downloading. ×
AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. AppFunnel: A Framework for Usage-centricEvaluation of Recommender Systemsthat Suggest Mobile ApplicationsMatthias BöhmerLyubomir GanevAntonio Krüger
  • 2. Introduction‣ Number of mobile applications steadily increasing‣ Finding good apps can become a difficult task‣ Recommender systems can help - Mobile apps special type of items - Context is important‣ „Dead apps“ on smartphones - Less than 1/2 of apps are actively used - Installation counts good for evaluation?
  • 3. Our paper...(1) Introduces a usage-centric evaluation framework(2) Presents results of a case study 3
  • 4. ‣ AppFunnel 4
  • 5. Concept of AppFunnel ‣ Idea: evaluate recommendation based on app engagement ‣ Concept of funnels adopted from advertising domain ‣ Apps can reach different stages after recommendation LONG-TERM VIEW INSTALLATION DIRECT USAGE USAGERECOMMENDATION ‣ Tracking engagement with apps along stages
  • 6. ‣ Case Study 6
  • 7. Testbed: appazaar‣ Recommender system for Android applications‣ Available for end-users on Google Play Market for free‣ 6,680+ users of appazaar, worldwide distribution‣ 45 users contributed to case study over three months
  • 8. Recommender Engines under Test Non-personalized Personalized Context-less - App popularity - Usage-based CF Context-aware - App-aware filtering - Location-aware CF - Time-aware CF
  • 9. Conversion Stages AppFunnel stage view view market installation direct usage long−term usageAverage number of occurrences per recommendation list 1.0 0.8 0.6 0.4 0.2 0.0 rity App−popularity CF Usage−based a App−awarere CF CF Time−aware Location−aware a ed aw e e ul Collaborative Collaborative Collaborative p- ar ar asFiltering Filtering Filtering op b p aw Filtering -aw Filtering p -p e- A e- n p g m io A sa Ti at U L oc
  • 10. Conversion Rates Conversion view to installation installation to direct usage installation to long−term usage 100Conversion rates in percent 90 80 70 60 50 40 30 20 10 0 ity F Usage−based C re CF Time−aware Location−aware CF r wa App−popularity App−aware u la d Collaborative e a re re Collaborative Collaborative as Filtering pp- waFiltering waFiltering p Filtering Filtering o -a -a -p -b A e n pp ge m io A sa Ti at U L oc
  • 11. Results of Case Study‣ AppFunnel shows difference in engines‘ performances, e.g. - Context-less engines: more views - Context-aware engines: better from installation to direct use‣ AppFunnel enables validation of engines‘ design goals, e.g. - App-aware engine: high installation to direct-usage rate - Context-less engines: higher rates for long-term usage‣ AppFunnel helped us to trace other issues of engines, e.g. - Location-aware engine: many views but no installations
  • 12. Limitations‣ Recommender engines under test are rather simplistic‣ Updates and removal not taken into account - Update does not tell much about app engagement - Removal events needs further refinement of AppFunnel‣ Future work: - Improve recommender engines by incorporating results of case study
  • 13. Discussion‣ Choosing from AppFunnel‘s metrics - Metrics might counteract (e.g., direct vs. long-term usage) - Metrics should reflect design goal (context -> direct usage)‣ Usage-centric evaluation valuable for other domains - When engagement with item can be traced - Changing view from „selling items“ to UX of items
  • 14. ‣ Conclusion 14
  • 15. Conclusion‣ We presented AppFunnel - Usage-centric evaluation of recommender engines - Recommendation of mobile applications‣ We presented a case study of AppFunnel in the wild - Traces down performance beyond installations - Shows differences between engines LONG-TERM VIEW INSTALLATION DIRECT USAGE USAGERECOMMENDATION