Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The MovieLens Datasets: History and Context

786 views

Published on

Presented at IUI 2016. The MovieLens datasets are widely used in education, research, and industry. They are downloaded hundreds
of thousands of times each year, reflecting their use in popular press programming books, traditional and online courses, and software. These datasets are a product of member activity in the MovieLens movie recommendation system, an active research platform that has hosted many experiments since its launch in
1997. This article documents the history of MovieLens and the MovieLens datasets. We include a discussion of lessons learned from running a long-standing, live research platform from the perspective of a research organization. We document best practices and limitations of using the MovieLens datasets in new research.

Published in: Science
  • Be the first to comment

  • Be the first to like this

The MovieLens Datasets: History and Context

  1. 1. The MovieLens Datasets: History and Context Max Harper (presenter) Joe Konstan
  2. 2. 2 http://tiis.acm.org/iui16/
  3. 3. MovieLens: 5 star movie ratings userId,movieId,rating,timestamp 1,2,3.5,1112486027 1,29,3.5,1112484676 1,32,3.5,1112484819 1,47,3.5,1112484727 1,50,3.5,1112484580 1,112,3.5,1094785740 1,151,4.0,1094785734 1,223,4.0,1112485573 1,253,4.0,1112484940 ... 138493,69644,3.0,1260209457 138493,70286,5.0,1258126944 138493,71619,2.5,1255811136 3 web site: dataset:
  4. 4. ratings data is interesting, intuitive, and pervasive 4
  5. 5. dataset impact » 140,000 downloads in 2014 » a search for “movielens” yields • 6,020 results in Google Books • 8,920 results in Google Scholar 5
  6. 6. dataset uses » research » technical: programming books + blogs » educational (including a MOOC) » industrial R&D, demos 6
  7. 7. overview » MovieLens datasets overview » dataset stability, system change 7
  8. 8. 8 <user, movie, rating, timestamp>
  9. 9. 9 <user, movie, rating, timestamp> <Max, Toy Story, 4.0, 2010-12-01 12:00:00>
  10. 10. MovieLens benchmark datasets 10 Name Dates Users Movies Ratings Density ML 100K ‘97 – ‘98 943 1,682 100,000 6.30% ML 1M ‘00 – ‘03 6,040 3,706 1,000,209 4.47% ML 10M ‘95 – ‘09 69,878 10,681 10,000,054 1.34% ML 20M ‘95 – ‘15 138,493 27,278 20,000,263 0.54% designed for replicability
  11. 11. MovieLens latest datasets 11 Name Dates Users Movies Ratings Density ML Latest ‘95 – ‘16 247,753 34,208 22,884,377 0.003% ML Latest Small ‘96 – ‘16 668 10,329 105,339 0.015% designed for recency
  12. 12. overview » MovieLens datasets overview » dataset stability, system change 12
  13. 13. tension: datasets vs. system » ideal (pure) vs. actual (it’s complex) » systems want to change • stay current, constant improvements • A/B tests, beta testing, and other experiments » context changes • devices, competing sites, changing user base 13
  14. 14. 14
  15. 15. 15
  16. 16. 16
  17. 17. 17
  18. 18. 18
  19. 19. some key changes » core flow of browse/search » rating widget » recommender » new user experience » … 19
  20. 20. history of experiments » both online field experiments and online lab experiments » created temporary and permanent changes, changed pattern of use 20
  21. 21. 21
  22. 22. in the paper » the story of MovieLens (1997 origins) • lessons learned from running a “real” system in a research lab • lots of fun descriptive stats/charts » best practices for dataset researchers • limitations • alternatives 22
  23. 23. people who made this possible » John Riedl » Istvan Albert, Al Borchers, Dan Cosley, Brent J. Dahlen, Rich Davies, Michael Ekstrand, Dan Frankowski, Nathaniel Good, Jon Herlocker, Daniel Kluver, Shyong (Tony) Lam, Michael Ludwig, Sean McNee, Chad Salvatore, Shilad Sen, and Loren Terveen » MovieLens users 23
  24. 24. in ACM Transactions on Interactive Intelligent Systems, Dec. 2015 » feedback? contact us: grouplens-info@cs.umn.edu presented by Max Harper, Research Scientist, University of Minnesota, harper@cs.umn.edu written with Joe Konstan, Distinguished McKnight University Professor, University of Minnesota, konstan@cs.umn.edu This material is based on work supported by the National Science Foundation under grants DGE-9554517, IIS-9613960, IIS-9734442, IIS-9978717, EIA-9986042, IIS-0102229, IIS- 0324851, IIS-0534420, IIS-0808692, IIS-0964695, IIS-0968483, IIS-1017697, IIS-1210863. This project was also supported by the University of Minnesota’s Undergraduate Research Opportunities Program and by grants and/or gifts from Net Perceptions, Inc., CFK Productions, and Google. 24 The MovieLens Datasets: History and Context
  25. 25. 25
  26. 26. 26 version 0 (1997) version 4 (2014)
  27. 27. one solution » document change, include with datasets 27
  28. 28. key dataset limitations (1/2) » system UI and recommender changes » bias towards “successful” users » possible bias towards users with tolerance for “research quality” design » timestamps do not reflect time of consumption 28
  29. 29. key dataset limitations (2/2) » recommender systems research community attitudes • implicit behaviors > ratings? • dataset-only research increasingly discouraged 29
  30. 30. 30
  31. 31. MovieLens system evolution key changes and experiments 31
  32. 32. alternative datasets 32 Name Domain Rating Scale Ratings Density Book- Crossing books 0 - 10 1.1m 0.003% EachMovie movies 0 - 14 2.7m 2.872% Jester (dataset1) jokes -10 - 10 4.1m 57.463% Amazon many 1 - 5 82.8m < 0.001% Netflix Prize movies 1 - 5 100.5m 1.178% Yahoo Music (C15) music (various) 0 - 100 262.8m 0.042%
  33. 33. 33 EachMovie
  34. 34. lessons from running MovieLens » lessons from startups apply (it’s hard, fail fast) » continual work, not one-time effort » encourage code quality through good social coding conventions » invest in tools that allow users to help 34
  35. 35. dataset uses » recommender systems research » recommender systems MOOC • http://coursera.org/learn/recommender-systems » code examples (popular press, blogs) » higher education » commercial – internal testing 35
  36. 36. 36

×