Abstracting the Individual Sys at Scale by Expedia Prod. Analyst
•
0 likes•109 views
- Design has shifted from creating positive user experiences to creating positive (societal) outcomes and our systems need machine intelligence and the ability to scale to achieve that - To capture positive outcomes we need to abstract the design of our machine intelligence systems at the highest level possible, while maintaining the ability to get it right for each and every individual - With the advent of more complex systems, assessing system performance, experimenting and measuring impact is also becoming an increasingly complex problem. What new developments should we be aware of and why should we care about them?
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Abstracting the Individual Sys at Scale by Expedia Prod. Analyst
Similar to Abstracting the Individual Sys at Scale by Expedia Prod. Analyst (20)
More from Product School
More from Product School (20)
Recently uploaded
Recently uploaded (20)
Abstracting the Individual Sys at Scale by Expedia Prod. Analyst
- 1. www.productschool.com Abstracting the Individual Sys at Scale by Expedia Prod. Analyst
- 2. Join 40,000+ Product Managers on Free Resources Discover great job opportunities Job Portal prdct.school/PSJobPortalprdct.school/events-slack
- 3. CERTIFICATES Your Product Management Certificate Path Product Leadership Certificate™ Full Stack Product Management Certificate™ Product Management Certificate™ 20 HOURS40 HOURS40 HOURS
- 4. Corporate Training Level up your team’s Product Management skills
- 5. Achilleas Fragkoulis T O N I G H T ’ S S P E A K E R
- 6. Abstracting the individual Systems at scale Achilleas Athanasiou Fragkoulis achilleasatha@gmail.com
- 7. Personalisation at scale A very real and hard problem ● Why it matters ● Process: Exploration - Discovery - Testing - Implementation ● Common pitfalls (eg. ML debt) ● Thoughts on tackling those ● How is our toolkit evolving? ● Key takeaways Achilleas Athanasiou Fragkoulis - tiny.cc/ps-talk
- 8. Why personalisation matters - a retrospective ● We have a product - Design 1.0 ● We optimise our product - Design 2.0 ● ??? - Design 3.0 ○ The system behind the “things” ○ Common factor ○ The outcome(s) our customers are after and we are optimising for (hopefully) ○ We need machine intelligence and systems that scale! Achilleas Athanasiou Fragkoulis - tiny.cc/ps-talk
- 9. Process Exploration Discovery Testing Implementation Achilleas Athanasiou Fragkoulis - tiny.cc/ps-talk
- 10. Common pitfalls ● Fast prototyping - MVPs ● Iterations ● Legal requirements ● Whitelisting / Blacklisting logic ● Dependencies ● and many more… Not a product, software, data-science or management problem - A complex systems problem Achilleas Athanasiou Fragkoulis - tiny.cc/ps-talk
- 11. My thoughts on tackling those ● Move a step up ● Then a step down ● Then climb at the top of the ladder "How can we design systems when we don't know what we're doing? The most exciting engineering challenges lie on the boundary of theory and the unknown. Not so unknown that they're hopeless, but not enough theory to predict the results of our decisions. Systems at this boundary often rely on emergent behavior — high-level effects that arise indirectly from low-level interactions." Ben Victor Achilleas Athanasiou Fragkoulis - tiny.cc/ps-talk
- 12. My thoughts on tackling those ● Move a step up ● Then a step down ● Then climb at the top of the ladder Abstract "How can we design systems when we don't know what we're doing? The most exciting engineering challenges lie on the boundary of theory and the unknown. Not so unknown that they're hopeless, but not enough theory to predict the results of our decisions. Systems at this boundary often rely on emergent behavior — high-level effects that arise indirectly from low-level interactions." Ben Victor Exploration Discovery Testing Implementation Achilleas Athanasiou Fragkoulis - tiny.cc/ps-talk
- 13. My thoughts on tackling those ● Rethink metrics (single vs multi-objective) ● Abstract your data ● Rethink what your “system” is comprised of (is something vital missing) ● Consider implications and downstream effects ● Unintended consequences ● Is the problem you are trying to solve misguided? Achilleas Athanasiou Fragkoulis - tiny.cc/ps-talk
- 14. Assessing performance ● Systems become more complex → evaluation becomes a complex problem itself ● Honourable mentions - experimentation toolkit: ○ Multi armed bandits ○ Interleaving ○ Sequential testing ○ Variance reduction techniques ○ Sensitive metric combinations Multi Armed Bandit Experiments Adam Lofting Achilleas Athanasiou Fragkoulis - tiny.cc/ps-talk
- 15. Key takeaways Rethink appropriate metrics / assessment methods EVALUATION Take a step back and reassess between iterations ABSTRACT Design for comlex, adaptive systems inc. implications and downstream effects OUTCOMES Achilleas Athanasiou Fragkoulis - tiny.cc/ps-talk
- 16. Interesting and useful references 1. Future of design: https://medium.com/@stephenanderson/the-future-of-design-computation-complexity-a434d2da3cd5 2. Intelligent things: https://www.linkedin.com/pulse/intelligent-things-its-all-machine-learning-roger-attick/ 3. ML debt - https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf 4. Ladder of abstraction: http://worrydream.com/LadderOfAbstraction/ 5. Multi Armed Bandits: https://medium.com/@SYM0920/the-multi-armed-bandits-problem-bba9ea35a1e4 Achilleas Athanasiou Fragkoulis - tiny.cc/ps-talk
- 17. www.productschool.com Part-time Product Management Training Courses and Corporate Training
Editor's Notes
- If you’re interested to connect with other Product Managers, aspiring PMs, or those within tech, join our Slack community of over 40,000 professionals. It’s a great place to network and to find interesting content. We host a weekly AMA through our Slack channel on Tuesdays from 11:15am - 12pm PST. We have also recently launched the Job Portal where you can find the latest Product Management opportunities! As members of the Product School community, we'd like to provide you with these resources at your disposal.
- Product School’s Product Management Certificate Path comprises of 3 part-time courses for professionals with strong technical or business background who want to further explore Product Management at software-based companies. During Product Management Training you will first learn Product Management fundamentals to understand the software product lifecycle and what it takes to successfully transition into a product management role. You’ll then be trained to retrieve data, understand its value and make impactful decisions with SQL, data visualization and Tableau. Learn to understand your users to deliver exceptional UX/UI design and develop a robust digital marketing plan. During the Full Stack Product Management Training, you will deep dive into the technical knowledge to enhance your ability to work with agile teams. Finally, Product Leadership Training will elevate your product knowledge to become an effective Product Leader. You'll do an in-depth analysis on how to implement best PM practices on a strategic level to significantly impact your company’s portfolio and revenue. Learn the soft skills to manage product teams and manage stakeholders to deliver performing products.
- As well as individual courses we provide corporate training across the world! If you’d like to upskill your product team this is the best option for you. We have trained employees from multiple companies such as Deloitte, Salesforce, JP Morgan, Bank of America amongst many other companies across all industries.
- Tonight's talk is “ [TITLE] ” with [NAME]. Welcome, [NAME].
- Talk title Sharing problems and struggles Giving a talk is a good way to clarify your ideas - digestible content Your own opinion crystallizes on the matter Structure
- How did we get here and why? What are we trying to make out of all this? Why should we even care? The process (just the usual) Common conundrums How to tackle them Evolution of tools Wrap up (short) Class activity - (share: MeSo revenue - Expedia vs Amazon, when you do things right no one notices) Before thinking about process - how did we find ourselves here?
- We have a product and customers (Design 1.0) [product focused - get the design right] We want our customer to be happy, deliver value (hopefully) not only to beat competition and grow but also because we care about the positive outcome our product/service has! So we optimise our product (Design 2.0) [experience focused - get the experience right] across multiple touchpoints/devices, browsers, screen sizes - seamless experiences between them Introduce dynamic and UGC, better data management (store & leverage session and cross-session, cross-device data) other examples? What comes next? What is design 3.0? - we have great and intuitive design, great experiences, what is lacking? System behind all the "things" How are all the things we did connected between themselves? What is the outcome(s), essentially our customers were after (and we optimise for) Need to design with machine intelligence and systems that scale Amazon or Netflix or any other Bring a product/service to market, there is demand and room for growth Optimise - bring in additional revenue, more satisfied users etc Finding common patterns Hold on we were optimising say for devices, so kind of optimising for a specific user "attribute", what other user attributes can we think of? How can we scale our optimisation efforts across all of them? Or continuous optimisation vs static at a point in time? Touch base on structure - we understand the benefits let's see what the usual development cycle looks like and where problems arise Now let's talk about process (bread and butter product development - in whatever shape or form)
- Is everyone familiar with these 2? (same thing) Example here to demonstrate how complexity is built into the system Take for example Netflix's recommendation engine (hypothetical and made up examples) Start with a vanilla algo We do some analysis around greater regions, say NA, EU etc and optimise accordingly, say NA users prefer comedies more than EU users that prefer romance so we boost accordingly Next up we find that we can introduce better cross-genre recommendations with some algo enhancements, so we test and it works in half our markets, we roll out and override this in the other half - product parity is already an issue Then we get hit with a piece of legislation in 5 specific countries, so we have to integrate this, say filtering out particular thumbnail artwork in the carousels - further aggravate product parity And so on so forth, soon this "behemoth" becomes unmanageable - complexity is built into the system rather than simplicity Why does this happen?
- We like prototyping and iterating fast No one ever got a complex problem right the first time Deployment / improvements as an "afterthought" It's a software problem as much as it is a DS or management problem etc (IT'S JUST A COMPLEX SYSTEM PROBLEM) Touch base on structure - entering uncharted territory Thoughts on addressing this
- Ben Victor quote - eg. Neural Nets Joan Robinson - "A model which took account of all the variation of reality would be of no more use than a map at the scale of one to one." Leverage the least amount of info with max impact - abstraction ladder The challenge lies not in constructing the system, but in understanding it. We must develop an intuition to guide our decisions. The design process is thus one of exploration and discovery Slide part 2 - Introduce an abstraction step in your development cycle
- Slide part 2 - Introduce an abstraction step in your development cycle Some tips to help abstractions
- Rethink metrics! (single vs multi-objective) Abstract your data! Rethink your system Comprised of what? In-Out? Something missing? Enhance/enrich? Evolving? Design for complex, adaptive systems Thinking about broader implications, and future, downstream effects Articulate broader concerns, unintended consequences, double-loop learning… Perhaps the problem you were trying to solve in the first place was misguided As always iterate!!! Understanding Ideas Tools (touch base on structure) - Complex system need good ways to assess system performance
- (touch base on structure) - Complex system need good ways to assess system performance what does our toolbox look like and how is it evolving? (touch base on structure) - Summarize key learnings and take-away message(s)
- (touch base on structure) - Summarize key learnings and take-away message(s) class activity - volunteer to say and explain why aaand you're free, with an ice breaker at hand and something interesting to chat about all night! Slides and refs available
- Feel free to speak with me and I can point you in the right direction (explain where to apply). Or you can visit www.productschool.com Have a good night!