AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications

  • 1,017 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,017
On Slideshare
865
From Embeds
152
Number of Embeds
2

Actions

Shares
Downloads
8
Comments
0
Likes
0

Embeds 152

http://matthiasboehmer.de 151
http://translate.googleusercontent.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. AppFunnel: A Framework for Usage-centricEvaluation of Recommender Systemsthat Suggest Mobile ApplicationsMatthias BöhmerLyubomir GanevAntonio Krüger
  • 2. Introduction‣ Number of mobile applications steadily increasing‣ Finding good apps can become a difficult task‣ Recommender systems can help - Mobile apps special type of items - Context is important‣ „Dead apps“ on smartphones - Less than 1/2 of apps are actively used - Installation counts good for evaluation?
  • 3. Our paper...(1) Introduces a usage-centric evaluation framework(2) Presents results of a case study 3
  • 4. ‣ AppFunnel 4
  • 5. Concept of AppFunnel ‣ Idea: evaluate recommendation based on app engagement ‣ Concept of funnels adopted from advertising domain ‣ Apps can reach different stages after recommendation LONG-TERM VIEW INSTALLATION DIRECT USAGE USAGERECOMMENDATION ‣ Tracking engagement with apps along stages
  • 6. ‣ Case Study 6
  • 7. Testbed: appazaar‣ Recommender system for Android applications‣ Available for end-users on Google Play Market for free‣ 6,680+ users of appazaar, worldwide distribution‣ 45 users contributed to case study over three months
  • 8. Recommender Engines under Test Non-personalized Personalized Context-less - App popularity - Usage-based CF Context-aware - App-aware filtering - Location-aware CF - Time-aware CF
  • 9. Conversion Stages AppFunnel stage view view market installation direct usage long−term usageAverage number of occurrences per recommendation list 1.0 0.8 0.6 0.4 0.2 0.0 rity App−popularity CF Usage−based a App−awarere CF CF Time−aware Location−aware a ed aw e e ul Collaborative Collaborative Collaborative p- ar ar asFiltering Filtering Filtering op b p aw Filtering -aw Filtering p -p e- A e- n p g m io A sa Ti at U L oc
  • 10. Conversion Rates Conversion view to installation installation to direct usage installation to long−term usage 100Conversion rates in percent 90 80 70 60 50 40 30 20 10 0 ity F Usage−based C re CF Time−aware Location−aware CF r wa App−popularity App−aware u la d Collaborative e a re re Collaborative Collaborative as Filtering pp- waFiltering waFiltering p Filtering Filtering o -a -a -p -b A e n pp ge m io A sa Ti at U L oc
  • 11. Results of Case Study‣ AppFunnel shows difference in engines‘ performances, e.g. - Context-less engines: more views - Context-aware engines: better from installation to direct use‣ AppFunnel enables validation of engines‘ design goals, e.g. - App-aware engine: high installation to direct-usage rate - Context-less engines: higher rates for long-term usage‣ AppFunnel helped us to trace other issues of engines, e.g. - Location-aware engine: many views but no installations
  • 12. Limitations‣ Recommender engines under test are rather simplistic‣ Updates and removal not taken into account - Update does not tell much about app engagement - Removal events needs further refinement of AppFunnel‣ Future work: - Improve recommender engines by incorporating results of case study
  • 13. Discussion‣ Choosing from AppFunnel‘s metrics - Metrics might counteract (e.g., direct vs. long-term usage) - Metrics should reflect design goal (context -> direct usage)‣ Usage-centric evaluation valuable for other domains - When engagement with item can be traced - Changing view from „selling items“ to UX of items
  • 14. ‣ Conclusion 14
  • 15. Conclusion‣ We presented AppFunnel - Usage-centric evaluation of recommender engines - Recommendation of mobile applications‣ We presented a case study of AppFunnel in the wild - Traces down performance beyond installations - Shows differences between engines LONG-TERM VIEW INSTALLATION DIRECT USAGE USAGERECOMMENDATION