Your SlideShare is downloading. ×
AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications

1,346
views

Published on


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,346
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. AppFunnel: A Framework for Usage-centricEvaluation of Recommender Systemsthat Suggest Mobile ApplicationsMatthias BöhmerLyubomir GanevAntonio Krüger
  • 2. Introduction‣ Number of mobile applications steadily increasing‣ Finding good apps can become a difficult task‣ Recommender systems can help - Mobile apps special type of items - Context is important‣ „Dead apps“ on smartphones - Less than 1/2 of apps are actively used - Installation counts good for evaluation?
  • 3. Our paper...(1) Introduces a usage-centric evaluation framework(2) Presents results of a case study 3
  • 4. ‣ AppFunnel 4
  • 5. Concept of AppFunnel ‣ Idea: evaluate recommendation based on app engagement ‣ Concept of funnels adopted from advertising domain ‣ Apps can reach different stages after recommendation LONG-TERM VIEW INSTALLATION DIRECT USAGE USAGERECOMMENDATION ‣ Tracking engagement with apps along stages
  • 6. ‣ Case Study 6
  • 7. Testbed: appazaar‣ Recommender system for Android applications‣ Available for end-users on Google Play Market for free‣ 6,680+ users of appazaar, worldwide distribution‣ 45 users contributed to case study over three months
  • 8. Recommender Engines under Test Non-personalized Personalized Context-less - App popularity - Usage-based CF Context-aware - App-aware filtering - Location-aware CF - Time-aware CF
  • 9. Conversion Stages AppFunnel stage view view market installation direct usage long−term usageAverage number of occurrences per recommendation list 1.0 0.8 0.6 0.4 0.2 0.0 rity App−popularity CF Usage−based a App−awarere CF CF Time−aware Location−aware a ed aw e e ul Collaborative Collaborative Collaborative p- ar ar asFiltering Filtering Filtering op b p aw Filtering -aw Filtering p -p e- A e- n p g m io A sa Ti at U L oc
  • 10. Conversion Rates Conversion view to installation installation to direct usage installation to long−term usage 100Conversion rates in percent 90 80 70 60 50 40 30 20 10 0 ity F Usage−based C re CF Time−aware Location−aware CF r wa App−popularity App−aware u la d Collaborative e a re re Collaborative Collaborative as Filtering pp- waFiltering waFiltering p Filtering Filtering o -a -a -p -b A e n pp ge m io A sa Ti at U L oc
  • 11. Results of Case Study‣ AppFunnel shows difference in engines‘ performances, e.g. - Context-less engines: more views - Context-aware engines: better from installation to direct use‣ AppFunnel enables validation of engines‘ design goals, e.g. - App-aware engine: high installation to direct-usage rate - Context-less engines: higher rates for long-term usage‣ AppFunnel helped us to trace other issues of engines, e.g. - Location-aware engine: many views but no installations
  • 12. Limitations‣ Recommender engines under test are rather simplistic‣ Updates and removal not taken into account - Update does not tell much about app engagement - Removal events needs further refinement of AppFunnel‣ Future work: - Improve recommender engines by incorporating results of case study
  • 13. Discussion‣ Choosing from AppFunnel‘s metrics - Metrics might counteract (e.g., direct vs. long-term usage) - Metrics should reflect design goal (context -> direct usage)‣ Usage-centric evaluation valuable for other domains - When engagement with item can be traced - Changing view from „selling items“ to UX of items
  • 14. ‣ Conclusion 14
  • 15. Conclusion‣ We presented AppFunnel - Usage-centric evaluation of recommender engines - Recommendation of mobile applications‣ We presented a case study of AppFunnel in the wild - Traces down performance beyond installations - Shows differences between engines LONG-TERM VIEW INSTALLATION DIRECT USAGE USAGERECOMMENDATION