5 Lessons Learned in Product Management by Twitch Senior PM
•
1 like•315 views
Main takeaways: - How to take a non traditional path to product management - How to leverage your unique background to differentiate yourself as a Product Manager - Steps you can take to build your product management skills/portfolio while in other fields
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to 5 Lessons Learned in Product Management by Twitch Senior PM
Similar to 5 Lessons Learned in Product Management by Twitch Senior PM (20)
More from Product School
More from Product School (20)
Recently uploaded
Recently uploaded (20)
5 Lessons Learned in Product Management by Twitch Senior PM
- 1. www.productschool.com 5 Lessons Learned in Product Management by Twitch Senior PM
- 2. Join 40,000+ Product Managers on Free Resources Discover great job opportunities Job Portal prdct.school/PSJobPortalprdct.school/events-slack
- 3. CERTIFICATES Your Product Management Certificate Path Product Leadership Certificate™ Full Stack Product Management Certificate™ Product Management Certificate™ 20 HOURS40 HOURS40 HOURS
- 4. Corporate Training Level up your team’s Product Management skills
- 5. Greg Smith T O N I G H T ’ S S P E A K E R
- 7. What to Expect from Today’s Talk ● Overview of the Product Development Process in the Extensions org at Twitch ● A deep dive into the individual segments framed around a product we launched and iterated from early 2018 to early 2019. ● Stories about what went well (some), what went poorly (more of those), and what I wish I knew ahead of time. ● How YOU can leverage these for your own product work
- 8. Product Development Process Working Backwards Customer Research, Product Vision & Strategy Iterate, Iterate, Iterate Analyze usage data, Address customer issues, improve product to deliver outcomes MVP + GTM Scope the Product and Deliver Initial Version to Market Product Discovery Test Product Hypotheses with Customers This is at a high level and will definitely differ from product to product.
- 9. Context: Extensions and the Rig in 2018
- 10. Phase 1: Working Backwards Desired Outcome: Create the product vision centered around the customer, and then the strategy of how you’ll go after solving their problems.
- 11. What Does That Mean? Five Questions to Ask. How do we know this?03 What are their problems/opportunities?02 Who is the customer?01 What would be the biggest benefit?04 What does the customer experience look like?05
- 12. How this Worked in Practice How do we know this?03 ● Interviews with devs and analysis of our developer funnel/lifecycle. What are their problems/opportunities?02 ● Extensions are much more technically challenging to build than traditional apps. Who is the customer?01 ● Developers across multiple cohorts of size, capability and motivation. Also broadcasters and viewers secondarily. What would be the biggest benefit?04 ● Driving down the development/operational cost and iteration time enables developers to focus on building great experiences for broadcasters and their audiences What does the customer experience look like?05 ● Integrated experiences across tools and services removing the need for developers to build or scale across commons patterns.
- 13. Working Backwards: What I Wish I Knew Beforehand 1. This takes a long time to do it right - budget the time for this. 2. Share share share. Getting folks aligned would have taken less time if they could have weighed in earlier (particularly other functions)
- 14. Phase 2: Product Discovery Desired Outcome: Get as much feedback as you can from customers to test your hypotheses before you actually build the product. “Software projects can be thought of as having two distinct stages: figuring out what to build (build the right product), and building it (building the product right). The first stage is dominated by product discovery, and the second stage is all about execution.” ― Marty Cagan, Inspired: How To Create Products Customers Love
- 15. Product Discovery: What We Did
- 16. Product Discovery: What I Wish I Knew Beforehand 1. Should have done UX Prototype testing if we had the option 2. Getting some reference customers is critical, although ours weren’t chosen as well as I would have liked - they were the one who simply responded to our things. We had community champions, but their data could be misleading.
- 17. Phase 3: MVP and Scoping Desired Outcome: Scope down your release so you can get something into customers hands ASAP. Be ruthless. Many key decisions happen here and this is where the details are especially important Give yourself room to cut things. The thought exercise of writing additional scope into the spec will give context on what really needs to be in there. If your dev team hasn’t been involved earlier, it’s really important to work closely with them, as technical details will have consequences short and long term.
- 18. MVP + Scoping: What We Did
- 19. MVP + Scoping: What I Wish I Knew Beforehand 1. Could have been more ruthless - added a logger feature that was superfluous and took time we could have spent elsewhere 2. Wish I had known how the choice of Yarn and the other technical dependencies would screw us for most of the year. Technical choices really really really matter. 3. Set tripwires if you’re working with new partner teams. Our sample got delivered at the last minute, all wrong, and our devs had to fix it all.
- 20. Phase 4: Initial Go to Market Desired Outcome: Get the MVP in Customers hands with an ability to measure the impact.
- 21. GTM: What We Did
- 22. GTM: What I Wish I Knew Beforehand 1. Date driven shipping sucks. Never going to try to live announce and launch a product again (or at least until I forget how crappy this was). 2. The data tradeoff for open source was a bad call and one that would haunt us. Understand the implications of this before you go in that direction. 3. Test test test - and really make extra room for this, more than you think. It’s the first thing that’s usually cut, but the amount of polish it provides cannot be understated.
- 23. Phase 5: First Iteration - Local Mode Desired Outcome: Get the data, look at how it impacts your metrics, talk to customers and iterate. Are you meeting your goals? What’s the qualitative and quant feedback. Think about where you are in the life cycle. Adoption, engagement, retention are the stats you want to look at. With this data - make your next decision. New features? Bug fixes? Need to pivot the initial value prop of the product? Double down on something?
- 24. Iteration 1: What We Did (Wrong)
- 25. Iteration 1: What I Wish I Knew Beforehand 1. The data indicated great engagement, but I wish I had a better sense of the UX and technical problems tolerated by the early adopter audience. 2. The amount of testing we needed as we went to a broader audience. Developers as a customer have some wild configurations on their machines.
- 26. Phase 6: Second Iteration - Shit We Need to Fix This Desired Outcome: Quickly address the symptoms and solve the underlying issue. Structure and triage. What are the biggest fires? What can you fix now or later? Be responsive and accountable - make users understand that you are working to solve the problem. Define metrics around your biggest issues so you can measure whether you fixed them.
- 27. Iteration 2: What We Did
- 28. Iteration 2: What I Wish I Knew Beforehand 1. Not addressing the core problems in some way exacerbated the problems the product was facing, even if the experience was improved. 2. Action isn’t always the best approach - sometimes doing something yourself is worse than not doing it at all. 3. Make sure you don’t neglect your internal stakeholders when managing expectations
- 29. Phase 7: Iteration 3 - Do the Right Thing and Rebuild Trust Desired Outcome: Reset the narrative of your product and rebuild customer trust. Start by rebuilding internal trust and find your champions. Look for a way to dramatically pivot customer expectations. Be accountable to your customers. Empathy is super important. Be especially careful about what you bring to market. Look to iterate in small groups first.
- 30. Iteration 3: What We Did
- 31. Iteration 3: What I Wish I Knew Beforehand 1. How much faster we could iterate on Electron vs the open source model. I would have pivoted to it much sooner, even as early as the first iteration. 2. How effective the limited beta group would be for this product for feedback and promotion. Should have been pushing versions to them a lot sooner.
- 32. Final Takeaways ● Always start with your customer and work backwards ● Validate as much as you can - the features, the experience before coding starts ● Be relentless in your pursuit of truth (quantitative and qualitative data). ● Shipping is about learning - and applying those learnings to improve the outcomes for customers. ● This is not a one PM show - get everybody bought in ● Be stubborn!
- 33. www.productschool.com Part-time Product Management Training Courses and Corporate Training
Editor's Notes
- If you’re interested to connect with other Product Managers, aspiring PMs, or those within tech, join our Slack community of over 40,000 professionals. It’s a great place to network and to find interesting content. We host a weekly AMA through our Slack channel on Tuesdays from 11:15am - 12pm PST. We have also recently launched the Job Portal where you can find the latest Product Management opportunities! As members of the Product School community, we'd like to provide you with these resources at your disposal.
- Product School’s Product Management Certificate Path comprises of 3 part-time courses for professionals with strong technical or business background who want to further explore Product Management at software-based companies. During Product Management Training you will first learn Product Management fundamentals to understand the software product lifecycle and what it takes to successfully transition into a product management role. You’ll then be trained to retrieve data, understand its value and make impactful decisions with SQL, data visualization and Tableau. Learn to understand your users to deliver exceptional UX/UI design and develop a robust digital marketing plan. During the Full Stack Product Management Training, you will deep dive into the technical knowledge to enhance your ability to work with agile teams. Finally, Product Leadership Training will elevate your product knowledge to become an effective Product Leader. You'll do an in-depth analysis on how to implement best PM practices on a strategic level to significantly impact your company’s portfolio and revenue. Learn the soft skills to manage product teams and manage stakeholders to deliver performing products.
- As well as individual courses we provide corporate training across the world! If you’d like to upskill your product team this is the best option for you. We have trained employees from multiple companies such as Deloitte, Salesforce, JP Morgan, Bank of America amongst many other companies across all industries.
- Tonight's talk is “ [TITLE] ” with [NAME]. Welcome, [NAME].
- We’ll take a look at how the Extensions organization at Twitch builds product, which has a large influence from Amazon’s process. We’ll also dive into the individual parts of the process, from inception to launch, followed up by several iterations from one of the products my team launched. This product is interesting, particularly in that we had to deal with very different challenges from iteration to iteration. I find the best lessons and stories come from when things don’t go well, so we’ll get plenty of that. I’ll dig into what I wish I had known and how I would have changed my approach. Most importantly, I hope these lessons that I learned, will be useful for your own product careers.
- To level set everyone, let’s take a high level look at the product process. This may be mostly familiar to folks, although each company/org may approach things differently. There are 4 main sections, although the 4th is really just a variation of steps 2 and 3 after the product is in market. It starts with a process we use at Twitch called working backwards, where we set the product vision and strategy, looking several years out, and developing an initial set of hypotheses. The next phase, is product discovery, which is where we work to test our hypotheses with customers before the team writes significant code. The next is getting your MVP built and in the hands of customers as part of your Go to Market. Once the product is in market, hopefully you’ve properly instrumented it and are able to analyze the product’s performance. From there, it’s about making the appropriate adjustments so you are able to deliver the outcomes for your customers.
- One other thing I’d like to do is set some context for Twitch, extensions and the product we’ll be talking about today. First, if you’re not familiar with Twitch, the power of its platform comes from the interaction between broadcasters and their audiences. While much of the content is based around users streaming video games, the success of other types of content (IRL streams, creative, cooking etc) indicate that the content is the frame, but it’s the community and interaction taking place that’s special. Originally, this core interaction was users being able to live chat with the broadcaster during the stream. Since then, Twitch has added abilities to subscribe to the channel, cheer, etc to create new interactions. Given the diverse content on Twitch, it would be near impossible for Twitch to build specific features to satisfy all the different types of content and communities. This is where extensions comes in. Extensions is an app platform on top of Twitch that lets 3rd parties build web apps that broadcasters install on their channel page to create new audience interactions. Much like the content on Twitch, there is a huge variety of extensions, ranging from apps like Twitch Prime to remind people of their subscriptions, to ones that provide detailed information on the content on the channel page when you mouseover, to ones that actually all you to influence the underlying content as an audience member. Extensions is important strategically to Twitch because it allows 3rd parties to multiply the value of the broadcast experience, without Twitch needing to do it itself. Extensions live on top of the channel page over the video, or in panels below it. It’s a new kind of application because it’s shared by potentially hundreds of thousands of viewers on a channel at the same time. The channel page is dynamic with many things happening and all kinds of state for a viewer to be in. Developers building these extensions need to account for all of this to deliver a great user experience. Additionally, building extensions is technically complex. We don’t need to go into all the details, but developers have to build a number of things that interact with Twitch systems and handle Twitch scale. There’s a lot to do to build a good experience. With all this in mind the product we’ll be talking about today is the “Developer Rig”, a standalone tool that helps developers build better extensions. I decided to go with this product for a few reasons. First it’s a standalone product and we can look at it’s development somewhat in isolation. Second the development process had some interesting twists and turns that I think can be really useful as examples for your own work. With that, let’s get started.
- The first phase of the product development process for our group is called working backwards. Essentially, it’s a framework how you develop the product vision around the customer and build the strategy to go after solving their problems. This is a process that comes from Amazon, and it’s a great way to work, particularly when you’re looking to invest in a new area or product. At this phase, you want to immerse yourself is as much data as possible, both quantitative and qualitative. Interviewing customers in the area is critical, to build empathy and understand of their problems. Additionally, any sources of data, particularly if you’re extending an existing product area is important to help prioritization. Once you have this data, you can start answering the five questions, which I’ll get into on the next slide. The outcome of this whole phase is a vision document called a PRFAQ - which can encompass multiple products. You may have heard about this online before, and I’ll dig into this in more detail later.
- Once you’ve done your initial research, you want to bring your team together to brainstorm and answer five core questions. Definitely don’t do this in isolation - it’s a great way to get more stakeholders (particularly engineers) involved at the outset and bought into the vision. Go through the questions
- Now I’ll go into the process at a high level of what we answered (which ultimately resulted in the Developer Rig (and other products) Go through the answers and discuss what we got from it. There were a number of customer problems, which is why we involved other members of the product team in this brainstorm. Resulted in a 2 pronged product strategy around scaling solutions and tools/experiences. Essentially the rig would be Unity for extensions + Firebase. Another benefit of this process is it helped us clarify the output metrics were were trying to drive and what the input metrics would be to help support that.
- It takes a long time to set up and getting multiple days to go through the questions is key. I think it was still rushed. I would have loved a broader data set of other developer cohorts, even just so their information was on the record. I brought in PMs and the eng team. I wish I had brought in marketing, design and other functions in this one (I fixed that in other ones I did in the future).
- Lots of ways to do this: conversations, experiments, design prototypes, RFCs (both internal and external validation). The Working backwards process is an early part of this as well. Twitch value: Experiment to decide. One developer in the ecosystem (muxy) had already build their own “Rig”, but given that this was to be a big part of the developer experience, ceding it to a 3rd party didn’t make sense strategically. Get the spec together of what you think you’re doing to be delivering as a product Also evaluating the technical parts of this too depending on the products. Developers are your most precious resource. Don’t waste their time having them build things that aren’t important.
- Initially tested concepts in user interviews across multiple devs, including ones that had failed. Making sure we distilled down the most important problems and solutions. Wrote a spec for a bunch of the features (3 waves), worked with devs to create an RFC which is both technical and product focused. Given this was a development tool - the technical design as well as the UX was going to be important. Eg. the technical dependencies, how it would play with the existing set of tools. Web devs are really opinionated. It was a good opportunity to get feedback quickly. We looked at potential different options for how the app was structured- scripts? In the Twitch channel itself? Did it satisfy security requirements for unreleased content?
- I would have done UX prototyping absolutely if I had known. Even not having the design resources to do it, it would have made a huge different. If you have a UX to your product, do this absolutely. I would have landed on a specific set of customers in well defined cohorts and worked directly with them to get more rounds of feedback. It’s often the follow up questions that really help you get the data you need. Takeaway: Be really thoughtful about where you’re getting your info- so it’s not just a bunch of noise. Your counterparts here are critical - particular the technical info you’re getting This should be continuous if possible
- This is where you work with the team to actually build the product. Have your hypotheses in place that have been partially validated by product discovery. Not all of it can be validated there, and it’s a call basically with how much effort you want to expend to them. Work closely with your dev team so you have as much context on their decision making, particularly if it will impact the customer experience. Think through the whole customer journey - not just how they use it, how do they get it? How do they learn about it? How do you support it? Know what’s coming up next in your backlog. Maybe talk about open sourcing
- 2 key scenarios: Getting Started Easily, and Testing the Different Views, Roles and Contexts of Different Viewers. Talk about the problems we wanted to solve. Think about the customer journey and all the things they need - eg samples. Build a backlog of 3 releases worth of features and took a machete to it. Cut down to 2 main things: testinging views and roles, Worked with design on the UX (very limited) - basically the stuff from the RFC to work from Elected to ship Hello World samples with product - worked with partner teams to produce the content Open Sourcing Decision and it’s implications Technical Dependencies based on the approach we took
- When you have a limited time to get something to market, you need to be crisp about the value. The logger we built duplicated some functionality the browser could have used, and while it didn’t take long, we didn’t need to do it. I wouldn’t have used Yarn - I would have pushed for something self contained like Electron and shipped it earlier. We would have avoided sooo many problems I would have escalated to leadership a lot earlier when they weren’t delivering. It’s good to build trust but you can be burnt.
- Work with marketing, determine your release vehicle. Ideally marketing has been involved much earlier, but it really depends on the culture of your team. Get your metrics in order (although you should already have these in place before you start building). Ideally get several rounds of testing/beta before going to a broader audience. Make your customers feel like owners. SEnd out an email to the org celebrating your team, talk about what you shipped, why it’s important, and what you’re going to measure (ideally with a follow up for the report). You need to manage internal expectations. This is not about you, it’s about the product and team.
- Had decided in advance to launch at GDC when there was more marketing support . Also had an event at Twitch HQ that week with a number of customers. Compressed testing period due to date driven shipping. Was working the night before QA’ing bugs with the devs before pushing it out the door. Getting Riggy with It - turning people in to owners on our live stream. Talk about our decision to open source - flexibility, getting the community to address their own problems Limited ability to collect data based on the open sourcing decision. Basically one main event had to proxy for a lot of stuff.
- Dates suck. I’m not positive how much lift we got directly from GDC, and shipping outside that window would have given us more polish. It would have made it better for our dev team as well. I wouldn’t have open sourced this (and had better instrumentation) I scheduled bug bashes and time for testing, but dev work of the MVP ran up against it and we werent’ realy E2E testing until days before we were supposed to ship. We could have had a ton of more polish at that point. Would have pushed out. Talk about the iron triangle - resources (I appled more), time, didn’t push out, or quality, it was shittier.
- This is where the fun really starts as a product manager - you have way more data sources now and you can figure out how to iterate on the product to create a great experience for your users. Realy want to make sure you ahve th eright metrics your measureing - product health is also helpful. At this point, you could have this be a multivariant test and you’ll want to get back the data on teh way to move forward - could just be shipping to 1% of your population and have to decide to ramp If the product is integrated, you need to look at metrics so you aren’t cannibalizing
- The data looked great - immediately 60% of our active developers were using the product and using it a ton per day! Still to early to see if it’s helping more people ship, but we at least have them using something daily in their dev process. Ok so what’s our decision to make? Fix bugs, add new features, try and open up the user base? Desire is to pour fuel on the fire and get more people to use it. Added some new features, and got signed up for a new date to ship this by. Decision: Worked to improve data - feeling the pain of the open source decision. Add a feature to circumvent friction higher in the funnel and get more people building. Added a feature called Local Mode. Discovery process with another RFC. Shipped a feature called Edit Context - more on that later. Big problems when it shipped - due to dependency of installing shit. Went to outside hacks and helped people and realized it was much more problematic.
- I wish I had gotten more direct customer feedback on the ux beforehand and tried to get some devs into the lab for testing. It would have revealed the usability problems. I knew onboarding was painful, but I didn’t know how bad it was. I would have either initially done some scripting, or moved to electron right away (also gives you file system access). Get a set of integration tests, or at least VMs in different configurations so we could regression test our changes, or at least work with a beta audience before going broader.
- Customer need to know you’re on it. The first step is triage. What needs to be tackled immediately, and focus on that. Then it’s the medium and long term plan to address the fundamental problem. There’s the symptoms and the actual problem. Sometimes the symptoms need to be fixed first if the underlying problem will take longer.
- Onboarding was the issue (3 Parts) Tech dependencies Configuring your project Getting all the processes kicked off Time to Hello World is the metric I used. Needed to drive that down and prove it Scripted the onboarding and try to be the troubleshooter for them. Overcorrected on this and had to back off when we found new problems that the scripts buried. Did the design, (dealt with tech debt), and pushed out the change, even without design. The fixes and adjustments bumped up our new user growth again (nice!) But the rig still looked crufty, and even if we had addressed some of the issues, the perception was still a problem we had to get around (internally and externally). For people who had been burned, it still looked like the old rig.
- Onboarding was an issue, but perception/look and feel was a big one that wasn’t addressed. In some ways, not doing the medium term release and looking for a longer term solution would have been better. Would not have designed it myself, or at least tried to go to other designers at Twitch for feedback to see if I could have gotten help. The scripting stuff could have used more bake time. I was so focused on my external customers with this, I didn’t build enough confidence with my leadership. There were others in the org who wanted a different approach, and I created an opening for them that I could have avoided.
- All about going back to fundamentals. In this case, we’ve lost a bunch of customers that haven’t used the product, even if it’s been improved, and we still have the perception problem. We need to make some big moves or we should abandon
- Got a lifeline on the product for a month so I wouldn’t lose the dev (we were fixing “bugs”). I knew that if we could solve these core problems, we could be successful. Developer converted the Rig to Electron over Thanksgiving after we had talked about it. Killed open source version for this. Worked with awesome designer to do a heavy refresh on the UX. Iterated with a set of NDA customers - many of whom were big detractors of the Rig. Shipped multiple releases a week. Announced Native rig beta on stream live with a live demo - drove huge signups.
- Huge success. Live demo drove significant sign ups to the beta. Qualitative data from customers has been very positive from hobbyists to AAA game developers. Re-design + electron went a super long way to pivot internal expectations and get us the runway we needed. This is a team activity - and it wouldn’t have worked without the amazing work from Camille and Chris. Rapid iteration with our detractors, turned them into promoters for when we went broader in the community. I would have leveraged them a lot sooner. Be stubborn! I wish I had been more stubborn earlier in the process but it helped now.
- - talk about PM as the director of a play.
- Feel free to speak with me and I can point you in the right direction (explain where to apply). Or you can visit www.productschool.com Have a good night!