Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Avant-garde product development in the Age of Data – Ed Cooke, Memrise

41 views

Published on

In his talk, Ed will highlight three key features of Memrise's experimental product development process: their use of prototyping and user-testing in product exploration; their commitment to radical diversity in team and technique; and the exploration of bold ideas to develop fresh content and ideas, such as their tour round Europe in the double-decker 'Membus'.

Ed will explain how these e orts tie together in helping Memrise invent and quickly deploy unorthodox learning experiences at scale.

Published in: Design
  • Be the first to comment

  • Be the first to like this

Avant-garde product development in the Age of Data – Ed Cooke, Memrise

  1. 1. Avant-garde product management in the age of data Ed Cooke @tedcooke
  2. 2. Memrise is 25 million people’s favourite Language learning app ● 200 language pairs ● Billions of things learned monthly ● Team of 60 in London ● iOS, Android, Web devs / PMs / Designers, content creators, language specialists
  3. 3. Holistic language learning with a multitude of learning modes
  4. 4. Help anyone to fall in love with and learn a language in a few months OUR PRODUCT VISION
  5. 5. Help anyone to fall in love with and learn a language in a few months OUR PRODUCT VISION Enchantment
  6. 6. Help anyone to fall in love with and learn a language in a few months OUR PRODUCT VISION Enchantment Effectiveness
  7. 7. Help anyone to fall in love with and learn a language in a few months OUR PRODUCT VISION Enchantment Broad appealEffectiveness
  8. 8. The adventures in product management: 1) Enchantment 3) Broad appeal 2) Effectiveness
  9. 9. Enchantment
  10. 10. Building the magic of travel and adventure into the product-development process
  11. 11. ● 20k beautiful videos ● Top premium feature ● Good expression of product vision
  12. 12. Reflections and learnings on this kind of experiential product development ● It is is deliberately upside-down ● Challenging to get buy-in from more technically minded team-members ● By betting big, you create an energy that’s difficult creatively to resist
  13. 13. Effectiveness
  14. 14. The challenges of enhancing learning effectiveness ● Cognitive science largely hasn’t studied learning effectiveness ● Difficult to generalise from (sometimes conflicting) lab findings ● Optimisation only relevant to tuning what you have
  15. 15. a) External efforts Effectiveness
  16. 16. “The people outside your organisation are always going to be infinitely smarter than those within.” Matt Mullenweg
  17. 17. Memrise Prize: an applied science competition ● Create the best learning method to teach 100 foreign words in an hour, tested a week later ● Compete by providing data from an empirical trial ● Final round is a test on 10s of 1000s of Memrise users
  18. 18. ● 15 different labs tested experiments ● 5 made it through to the final round ● Finalists entries tested on 10s of 1000s of Memrise users
  19. 19. ● 4 showed >2x of flashcard control ● Methodologies varied widely
  20. 20. Reflections and learnings ● State of expert art + where we stand ● Litany of actionable micro-improvements + techniques ● New way of thinking about modelling complicated mental capacities REFLECTIONS AND LEARNING: ● It’s a big long term bet, collaborations take years not quarters. ● Danger creating a fourth code-base:
  21. 21. b) Internal efforts Effectiveness
  22. 22. Learning Research Team
  23. 23. Prototype > user test > analyse (RINSE AND REPEAT)
  24. 24. User testing ● 3 days per week ● Full experiments ● Weekly iteration cycle ● Empowers language creators
  25. 25. Example experiment Training Evaluation Results 21% 75% 86% 62% Condition A Condition B B-tests A-tests B-testsA-tests
  26. 26. Prototyping as a tool for research (and product development) Dynamics of product experiment ● 1 experiment ● 3 developers ● 2-3 product / data ● 1 month ● 1-2 iterations Dynamics of prototype-experiment ● 12 experiments ● 1 developer ● 4 learning researchers ● 1 user tester ● 1 month Cost per experiment: $50k Cost per experiment: $4k
  27. 27. Hybrid approach to cognitive engineering Effectiveness EXTERNALLY ● Stimulate original scientific research ● Integrate the discoveries ● Collaborate with that community Challenge: SLOW !!! INTERNALLY ● Prototype and user test for accelerated discovery ● Build tools to increase accessibility and speed Challenge: 4th CODE BASE
  28. 28. Broad appeal
  29. 29. Build diversity into team
  30. 30. Opening up ideation as widely as possible Prototyping Hackathons
  31. 31. Opening up ideation as widely as possible
  32. 32. Hackathons ● Every 6 weeks ● Engineers set free ● Aim to have working prototype in 2 days
  33. 33. Challenges and reflections ● Ideas that you don’t personally like can thrive ● You kill ideas you build (which non-battle hardened product managers can find upsetting) ● Can over-complicate the product
  34. 34. Help anyone to fall in love with and learn a language in a few months OUR PRODUCT VISION Enchantment Broad appealEffectiveness
  35. 35. memrise.com/jobs

×