Product innovation AI 2020
•Download as PPTX, PDF•
0 likes•120 views
This document discusses skills, attitudes, and responsibilities for product managers of AI-driven applications. It outlines four key points: 1) understand the technology, 2) communicate the value proposition, 3) define the minimum viable product (MVP), and 4) don't forget non-AI aspects of product management. Product managers must assess opportunities, define the product to be built, and improve performance with experience.
Recommended
Recommended
More Related Content
Similar to Product innovation AI 2020
Similar to Product innovation AI 2020 (20)
Recently uploaded
Recently uploaded (20)
Product innovation AI 2020
- 1. Magic Dust for Artificial Intelligence Product Management Mark Cramer Applied AI Product Management PARC Specific Skills, Attitudes and Responsibilities for PMs when it comes to AI-driven applications Product Innovation: Artificial Intelligence 2020 February 7th, 2020
- 6. © 2019 Xerox Corporation. All rights reserved. Xerox® and Xerox and Design® are trademarks of Xerox Corporation in the United States and/or other countries. BRXXXXX
- 7. “The product manager has two key responsibilities: 1. assessing product opportunities and 2. defining the product to be built.” - Marty Cagan, Inspired
- 8. Defining artificial intelligence isn’t just difficult; it’s impossible… “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.” – Tom Mitchell
- 10. 1. Understand the technology 2. Communicate the value proposition 3. Define the MVP 4. Don’t forget the non-AI stuff
- 16. Define the MVP
- 18. Diagram courtesy of Jussi Pasanen, with acknowledgements to Aarron Walter, Ben Tollady, Ben Rowe, Lexi Thorn and Senthil Kugalur.
- 19. source: Andrej Karpathy, Train AI 2018 Conference
- 24. Don’t Forget the non-AI Stuff
- 26. “Tell ’em what you’re gonna tell ’em. Tell ’em. Tell ’em what you told ’em. - Some salesperson somewhere
- 27. Thank you! 1. Understand the technology 2. Communicate the value proposition 3. Define the MVP 4. Don’t forget the non-AI stuff https://twitter.com/markdcramer fun poll! https://www.linkedin.com/in/mcramer articles
Editor's Notes
- Not so long ago, a Product Manager, and we’re talking about software applications, could go out into the field and study the market, look at competitors, speak to potential customers, do ethnographic studies, etc., and then come back and define a feature set that, with reasonable certainty, could be built and delivered, as specified. Sure, if you’re Agile, which of course you are, you’ll be quickly iterating on that functionality, but in general the software did what it was spec’ed to do.
- Today I’m hear to talk about two of the most exciting things in tech, and thus the world: peanut butter and chocolate. Actually, product management and AI. Two great tastes that taste great together. This entire conference is about AI and I’m told that 40% of you are PMs. Raise you hand if you’re a PM. Raise you hand if you work in AI. Who does both? So the question is, are we still in ‘once upon a time,’ or have things changed? Is it business as usual just applied to a new technology or are there new tools that PMs need to navigate the AI landscape, assuming that’s where they want to be. Deborah Leff, CTO for data science and AI at IBM, recently pointed out that 87% of data science projects never make it into production (https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production). That’s a failure rate that sounds well outside of business as usual. As PMs, are there things we can do to buck this trend?
- People often refer to “sprinkling a little magic AI dust” on a product to effortless transform it into something superior. I tried to find the origin of that phrase but couldn’t pinpoint it. Regardless, it’s often used derisively to describe the over-optimism or hype surrounding AI, along with perhaps the lack of understanding, among business executives and perhaps customers, regarding what AI can and cannot do. I’m going to discuss some of the issues around AI, but this notion of ‘magic dust’ is pretty fundamental to the challenge. But it’s also fundamental to the excitement. People love and hype AI because, primarily, of all the amazing things it can do. Many of us also love it because it is so cutting edge. We’re doing things today that were inconceivable not even 5 years ago, and stuff from last year is already obsolete. As a PM working in AI, however, you need to be out in front of this stuff. You need to understand the technology and you need to be able to communicate the right message around AI in your products, and this take some special skills.
- As a very quick introduction to myself, I went to school here. I also studied electrical engineering, and this is one of my school projects, which was a pretty amazing experience. Even though I was EE, however, I took software classes and studied AI with one of the pioneers of AI, Patrick Henry Winston.
- I’m still hacking away at programming – this is me developing a game in Vegas – although my career for the past couple decades has been focused on brining new products to market in the role of Product Manager. I’ve worked for a number of different startups and more recently founded a company that developed AI to enhance search results, which I ended up running for about a decade.
- Now I work at PARC running Product Management for Applied AI. We’re developing AI-powered SaaS application to assist knowledge workers with document creation and management. Here are some of the things that have been invented at PARC, include the personal computer, laser printing, the GUI, object-oriented programming, as well as natural language processing. There are people who have been working on AI at PARC since the 70s and 80s, when it was just cool as opposed to being insanely hyped.
- Let’s get back to Product Management. I’m sure everyone’s familiar with Product Management, but the role of the Product Manager is, and has been, for whatever reason, somewhat amorphous. To frame our conversation of what a product manager does, I like using this definition from Marty Cagan, which is the best I’ve ever seen: assessing product opportunities and defining the product to be built. No matter your field of endeavor, there are going to be particular skills you’re going to need to accomplish both of these tasks. Assessing opportunities and defining products are horizontal disciplines, and certainly there are horizontal skills and techniques that go along with them, but if your product is a toothbrush, there are special things you need to know to successfully PM that product. If you product is AI-based, what are the things you need to know?
- Here they are. {Read them off} Let’s go through them and provide some motivating examples and perhaps a little advice.
- In another meeting with a different development team, and a different product, it was explained me that we’re going to use 4000-dimensional sentence embeddings, as opposed to 300-dimensional word embeddings, for matching and retrieval. I wondered to myself, “how are we going to train that,” before I realized we’d be using a pre-trained model.
- This famous Tweet is funny because it accurately characterizes the gulf between developers, who write in Python, and business executives, who deal more in PowerPoint. As a Product Manager, you’re going to be in the middle. Marty Cagan, again in his book Inspired, talks about how Product Managers can come from many different disciples, including those that are non-technical, such as Marketing. With respect to AI product, given the complexity and ‘newness’ of the technology (and I put newness in quotes because, even though the principals have been around for decades, the extent to which AI is being embraced is somewhat new), if you’re going to bridge this gap you’re going to have to understand the fundamentals of how it works. With respect to communicating with executive stakeholders, your challenge is going to be to explain to them, in as non-condescending a way as possible, of course, the limitation of the magic dust. Your knowledge of the subject will give you credibility and the good news here is that in the land of the blind the one-eyed man is king. With respect to developers, credibility is also critical. They all have two eyes but you’re going to need to have at least one. The idea here is clearly not to know what they know, or anywhere near that, but to understand the fundamentals well enough to be able to have credible conversations about the products you’re building. You don’t want to be casually brushed off.
- You don’t necessarily need to understand this.
- But the news is good. There are tons and tons of online resources you can sign up for and learn about how this AI stuff works. I’m presently taking Masters classes in AI at Stanford, but that is mega-overkill. I’m doing it because I’m a bit obsessed with it, but if you’re a Product Manager and want to manage some AI product, I’d highly encourage you to learn a little Python, if you don’t know it already, and take a class if you don’t, and then bang out of few of these classes. In additional to everything else, it’ll also demonstrate your passion for AI, which will go a long way. What this (mostly) won’t get you, however, is an understanding of the gap between academic and real-world implementations. Typically in academic of coursework settings you’re going to have cleaner and more structured data than you’re going to have in the wild, whether that’s consumer or enterprise settings. Nevertheless, by understanding the academic work, you’ll be much better equipped to foresee and react to challenges in the real-world.
- I’m presently pursuing a Graduate Certificate in AI from Stanford. It’s awesome. It’s a ton of work. It’s also overkill. Last week for homework we had to design an AI to play Pacman. There was also a competition and in the end I placed 7th in a class of 325. It’s a lot of fun and if you’re truly passionate about learning AI you could go for it, but really a few online go-at-your-own-pace courses is all you need.
- I was in a meeting with the development team one day when the development manager expressed to me that “we’ll add the AI in later.”
- It is my personal opinion that defining an MVP, and getting buy-in from all the stakeholders, is the single most difficult task of any product manager. The incentive to add features and functionality, often driven purely by fear, is often uncontrollable, but good Product Managers need to figure that out how to control that. Everyone’s definition of what constitutes ‘minimum’ or ‘viable’ is going to vary, and in general people are going to be conservative. It’s human nature. If beyond that you take a lowest common denominator approach to everyone’s perception of MVP, you could potentially end up with feature creep and a bloated, delayed product upon which you don’t iterate fast enough.
- Raise your hand if you have not seen this graphic. It’s a brilliant way to depict what an MVP should be, and this hold true whether your product is using AI or not. You want to find a way to satisfy a wide range of need in a small segment of the market. But what if your product isn’t ‘usable’ until a large portion of the market is actively involved with the product and providing data for learning?
- AI does not make this easier. In fact, it makes it much harder. {Talk about slide.} The reason is, it’s difficult to define what ‘works’ and, as you know from studying the technology, AI gets better with time and more data. Addressing the first point, stakeholders are asking me, pretty much all the time, “Does it work?” The answer is quite complicated. When you’re dealing with probabilistic predictive models, what’s the threshold for ‘working’? It’s clearly not 100%, because even the best, most highly trained models in the world are not 100%. “A really interesting thing happens when you go from developing a Software 1.0 (i.e., traditional software) to a Software 2.0 system. In Software 1.0 we spend the majority of our effort on writing code, expressing how the system achieves its goals. Our whole tool chains are geared around the creation and validation of that logic. But in Software 2.0 the majority of our effort goes into curating training data, i.e., specification-by-example of what the system should do. We need a whole new tool chain geared around the creation/curation and validation of that data.” – Morning Paper, Migrating a privacy-safe information extraction system to a software 2.0 design
- The fundamental question is, therefore, “Does it work?” The answer is you have to figure out what threshold of ‘accuracy,’ or whatever you want to call it, delivers sufficient value to your customers, which not overly inconveniencing them with the misses. And these leads us to the 2nd point, which is that even though your AI is going to get better with use, it needs to clear a certain minimum threshold of ‘goodness’ right out of the gate. Otherwise people won’t use it and it’ll never have the opportunity to get better. So your MVP needs to achieve some sort of minimally acceptable accuracy right off the bat, and this is something you need to learn from customers and then communication to engineering and management. That being said, from day 1 it’s imperative to implement AI feedback loops into the product. Whatever data you need to improve your models or accuracy, you need to make sure that you’re not only collecting that, but you have a mechanism whereby you can continuously improve the models. This is, unfortunately, probably not something that β customers or initial users we feel or experience very quickly, but over time it’ll make the difference between a good product and a great one.
- In another meeting, this time with executive stakeholders, it was clear that were interested in taking a run-of-the-mill application and super-charging it with AI. In other words, we were going to sprinkle on some magic AI dust and transport our company to some magical realm.
- So, now that you understand how this stuff works, you’re going to have to tools to communicate the value proposition to all the necessary stakeholders, principally management, development and customers. With respect to management and customers, you’re going to have to battle the ‘magic dust’ myth. I had a β customer say to me, “well, the AI is just going to learn how to do this and then pretty soon it’ll be perfect.” Setting expectations properly while not dampening enthusiasm is a tricky balancing act. I had to explain to this person that she was still going to have to do work, but Product Managers, and people in sales, know how important it is to set expectations properly.
- You’re also going to be presenting, at some point, in from of executive stakeholders. I mentioned this a little bit at the beginning. No one wants a PM who doesn’t ‘believe’ in the product, so you’re going to have to find a way, here, too, to set expectations properly. A good trick, from the beginning, is to clearly establish what the user (i.e. the human) is going to have to do. If you leave the human out of the equation, it’s going to be assumed that the AI is going to completely fill that gap. Every good PM focuses on the user – solving his or her pain points – integrating with his or her workflow – conducting ethnographic user studies and empathy maps to fully understand the user – but in the end it’s key to not forget to focus on what the human is going to be expected to do, and this will help to set the bounds for what the AI is anticipated to do. You can often accomplish this through a well-defined MVP.
- Story about not being able to change the profile picture.
- This last one is probably obvious, but it’s a trap that super-easy to get caught in. Don’t fall so in love with your AI that you forget the rest of the product. Your self-driving car need door handles and cup holders. That’s perhaps not super-sexy, and maybe it’s not the thing you signed up to do, but your product is going to fail if you don’t put in the time and effort to create great features that are non-AI. My current product at Xerox is, more or less, split into two parts – the AI part and the non-AI part, and the non-AI part is getting a majority of the development resources. This is because everyone has correctly realized that the magic dust isn’t going to solve everything. We’ve had the conversations internally and with our β customers and have identified where the AI is going to leave off and where the humans are going to need to take it from there.
- OK, I told you what I was going to tell you, then I told you, so now I’m going to tell you what I just told you.
- Here they are. By the way, this is culled from my personal professional experiences, as well as conversations that I’ve had with people in similar roles, but it’s certainly not meant to be comprehensive. If you have others, or additional details to lend to these 4, please let me know. Call to action. Go to LinkedIn. Meet after talk.