This document discusses A/B testing strategies for new product launches. It begins by explaining what A/B testing is and why companies use it. For new products, qualitative data is more important than quantitative data in the early stages. A minimum viable product (MVP) should be launched to create a foundation for A/B testing. Iterative testing can introduce other features to determine the winning variant, and holdouts can measure long-term success. Other validation methods like focus groups and beta testing are also discussed. The key is to qualify feedback before extensive A/B testing and measure performance over the long run.
How to Focus On the Problem, Not the Solution by Spotify PMProduct School
Main takeaways:
-The five why’s - a tried-and-true method to effectively uncover user needs.
-Leveraging the JTBD framework - what are your customers trying to accomplish?
-How do you know you’ve solved the problem? Defining success metrics with your customers from the beginning.
A talk I gave at Google on Strategy and Product Discovery
We discussed:
Discovering Features and Products (Product Strategy)
Discovering Products and Product Lines (Product Line / Company Strategy)
Marty Cagan: Using High Fidelity Prototypes for Product Discovery
Prioritization Method for Every Case by fmr Atlassian Principal PMProduct School
Main Takeaways:
- Prioritization is about understanding each other and making alignment.
- Choose an appropriate prioritization method depending on your project situation and stakeholder group.
- Find a way to scale up or down your prioritization method and balance between quantitative and qualitative approaches.
Customer to Product Idea Iteration by Amazon's Product ManagerProduct School
In this talk, Akshay Kerur from Amazon explored working backwards from the concept/customer to an initial product idea.
Main takeaways:
1. Why it's so important to put your product on paper.
2. Questions you need to know and answer about a product prior to any engineering commitment.
3. Ratifying your product idea through proper customer and internal stakeholder identification.
Behind every great product is a great team doing work in a way that guarantees results. They are following a roadmap from the starting point to the end product. But a product roadmap can be elusive. This talk addresses why it is important and presents an approach to make one.
Mental Models to Guide Product Decisions by Google Product ManagerProduct School
Main takeaways:
-What are mental models?
-Understand the different types of mental models that are relevant for product roles
-How to incorporate mental models to help with problem solving and make better/ faster decisions
How to Focus On the Problem, Not the Solution by Spotify PMProduct School
Main takeaways:
-The five why’s - a tried-and-true method to effectively uncover user needs.
-Leveraging the JTBD framework - what are your customers trying to accomplish?
-How do you know you’ve solved the problem? Defining success metrics with your customers from the beginning.
A talk I gave at Google on Strategy and Product Discovery
We discussed:
Discovering Features and Products (Product Strategy)
Discovering Products and Product Lines (Product Line / Company Strategy)
Marty Cagan: Using High Fidelity Prototypes for Product Discovery
Prioritization Method for Every Case by fmr Atlassian Principal PMProduct School
Main Takeaways:
- Prioritization is about understanding each other and making alignment.
- Choose an appropriate prioritization method depending on your project situation and stakeholder group.
- Find a way to scale up or down your prioritization method and balance between quantitative and qualitative approaches.
Customer to Product Idea Iteration by Amazon's Product ManagerProduct School
In this talk, Akshay Kerur from Amazon explored working backwards from the concept/customer to an initial product idea.
Main takeaways:
1. Why it's so important to put your product on paper.
2. Questions you need to know and answer about a product prior to any engineering commitment.
3. Ratifying your product idea through proper customer and internal stakeholder identification.
Behind every great product is a great team doing work in a way that guarantees results. They are following a roadmap from the starting point to the end product. But a product roadmap can be elusive. This talk addresses why it is important and presents an approach to make one.
Mental Models to Guide Product Decisions by Google Product ManagerProduct School
Main takeaways:
-What are mental models?
-Understand the different types of mental models that are relevant for product roles
-How to incorporate mental models to help with problem solving and make better/ faster decisions
How to Increase Your Product Sense by ServiceNow Senior PMProduct School
Main takeaways:
- Framework of learning and improving your product sense
- Learn how to do your skill gap analysis and ideas to level up
- How to build it as a muscle and create successful products
Feature Prioritization Frameworks by Spotify Sr PMProduct School
Main takeaways:
-Why Feature Prioritization is Important?
-Overview of the popular prioritization frameworks: Rice, Value vs. ---Effort, The MoSCoW Method, Kano, Opportunities
-How to use frameworks?
-Tips and Tricks
Product-Led Growth by Amazon Senior Product ManagerProduct School
Main takeaways:
- How to build a product that sells itself
- Product-led growth is when your product sits in the center of the growth strategies to attain new customer acquisition, retention, and expansion
- Product-led means creating a champion product which serves as a primary driver for business growth
Managing Conflicting Stakeholders by Deliveroo Sr Product Manager (1).pdfProduct School
Main Takeaways:
- Alignment on overall goals and outcomes is key
- Build empathy for diverse points of view
- If in doubt escalate in the spirit of seeking input on "hotly contested" topics
How to Manage a Platform Product by Yelp Product ManagerProduct School
In this presentation you will learn what a platform product manager does, how to build platforms that delight your customers and learn more about the rewards and challenges of platform product management.
How to Drive Prioritization and Planning by Microsoft PMProduct School
In this talk, Salah talked about how to be deliberate about the investments your team makes and the features you build. He discussed the importance of defining goals and getting buy-in from key stakeholders, how to use those goals to prioritize your features, how to keep the team motivated to deliver on those features and how to track success. Roadmap planning is a key aspect of a Product Managers job and becomes more important as you advance in your career.
How to Run A/B Tests Successfully by Vice Media Product ManagerProduct School
This was an informal chat about what it's like running A/B Tests for a media company. George covered the different ways A/B Tests can be implemented within an organization and also touched on some of the misconceptions of A/B Testing.
How to Break Into Product Management by Uber Product ManagerProduct School
Darrell focused on how he broke into product. He walked through his career and some of the lessons that he learned on the road to becoming a Product Manager at Uber.
How to Prioritize as a PM by Google's Product ManagerProduct School
In order to build the best possible solution, a product manager needs to understand why they are building. Once the problem and stakeholders have been identified, how does a PM tackle solving it? Jeff Betts, a Product Manager at Google, covered this problem-solving framework and share insights and anecdotes from his user-first approach to product development and prioritization strategy.
Customer Centricity and Product Led Growth by Airbnb Product & Growth Product School
Product Management Event at #ProductCon San Francisco about Customer Centricity and Product Led Growth by Product & Growth Manager at Airbnb, Pratik Shah.
Feature Prioritization Techniques for an Agile PMs by Microsoft PMProduct School
Main takeaways:
-PMs don't need a lot of data points to prioritize the features for the upcoming sprint. They just need to identify the relevant one's.
-PMs should be skilled to strike the balance between agility in making decisions and accuracy of perceived outcomes
-PMs should be able to prioritize the feature requests with minimum data points available and optimum techniques
How to Use Customer Feedback on Your Product by UserVoice CEOProduct School
Increased competition and higher customer expectations are forcing modern product management teams to up their game to keep pace. In this talk Richard White, CEO of UserVoice, talked about how product leaders are leveraging customer feedback, collected across the organization, to inform how they build the best product possible in the shortest amount of time.
Intro to Data Analytics with EA's Director of ProductProduct School
Building an analytics strategy is crucial for every product manager. How do you build and implement an effective product analytics strategy that quantifies and drives product success and iteration? Director of Product at Electronic Arts, Bertram Chan, talked about how data analytics is no longer a luxury, its a competitive and integral need for Product Managers. He also discussed ways to understand not only who your users are but how they behave.
Cracking the Product Manager Interview with Gayle McDowellProduct School
In this talk, Gayle McDowell, author of the book "Cracking the PM Interview", taught people how to prepare for Product Manager interviews, what top companies like Google, Amazon, and Microsoft really look for, and how to tackle the toughest problems.
She talked about how the role of a Product Manager varies across companies, what experience you need, how to make your existing experience translate, what a great Product Manager resume and cover letter look like, and finally, how to master the Product Manager interview questions.
Managing an Experimentation Platform by LinkedIn Product LeaderProduct School
Main Takeaways:
-Establishing a culture of experimentation at scale
-Developing the product vision and strategy
-Backlog prioritization based on Impact Score formula
A/B Testing for New Product Launches by Booking.com Sr PMProduct School
Main takeaways:
-There is no one right way of validating a product, A/B testing is just one of them
-Get your product 'qualitatively' validated before 'quantitatively' validating
-Use holdouts to measure long term success of your new products, while running A/B test in parallel
How to Increase Your Product Sense by ServiceNow Senior PMProduct School
Main takeaways:
- Framework of learning and improving your product sense
- Learn how to do your skill gap analysis and ideas to level up
- How to build it as a muscle and create successful products
Feature Prioritization Frameworks by Spotify Sr PMProduct School
Main takeaways:
-Why Feature Prioritization is Important?
-Overview of the popular prioritization frameworks: Rice, Value vs. ---Effort, The MoSCoW Method, Kano, Opportunities
-How to use frameworks?
-Tips and Tricks
Product-Led Growth by Amazon Senior Product ManagerProduct School
Main takeaways:
- How to build a product that sells itself
- Product-led growth is when your product sits in the center of the growth strategies to attain new customer acquisition, retention, and expansion
- Product-led means creating a champion product which serves as a primary driver for business growth
Managing Conflicting Stakeholders by Deliveroo Sr Product Manager (1).pdfProduct School
Main Takeaways:
- Alignment on overall goals and outcomes is key
- Build empathy for diverse points of view
- If in doubt escalate in the spirit of seeking input on "hotly contested" topics
How to Manage a Platform Product by Yelp Product ManagerProduct School
In this presentation you will learn what a platform product manager does, how to build platforms that delight your customers and learn more about the rewards and challenges of platform product management.
How to Drive Prioritization and Planning by Microsoft PMProduct School
In this talk, Salah talked about how to be deliberate about the investments your team makes and the features you build. He discussed the importance of defining goals and getting buy-in from key stakeholders, how to use those goals to prioritize your features, how to keep the team motivated to deliver on those features and how to track success. Roadmap planning is a key aspect of a Product Managers job and becomes more important as you advance in your career.
How to Run A/B Tests Successfully by Vice Media Product ManagerProduct School
This was an informal chat about what it's like running A/B Tests for a media company. George covered the different ways A/B Tests can be implemented within an organization and also touched on some of the misconceptions of A/B Testing.
How to Break Into Product Management by Uber Product ManagerProduct School
Darrell focused on how he broke into product. He walked through his career and some of the lessons that he learned on the road to becoming a Product Manager at Uber.
How to Prioritize as a PM by Google's Product ManagerProduct School
In order to build the best possible solution, a product manager needs to understand why they are building. Once the problem and stakeholders have been identified, how does a PM tackle solving it? Jeff Betts, a Product Manager at Google, covered this problem-solving framework and share insights and anecdotes from his user-first approach to product development and prioritization strategy.
Customer Centricity and Product Led Growth by Airbnb Product & Growth Product School
Product Management Event at #ProductCon San Francisco about Customer Centricity and Product Led Growth by Product & Growth Manager at Airbnb, Pratik Shah.
Feature Prioritization Techniques for an Agile PMs by Microsoft PMProduct School
Main takeaways:
-PMs don't need a lot of data points to prioritize the features for the upcoming sprint. They just need to identify the relevant one's.
-PMs should be skilled to strike the balance between agility in making decisions and accuracy of perceived outcomes
-PMs should be able to prioritize the feature requests with minimum data points available and optimum techniques
How to Use Customer Feedback on Your Product by UserVoice CEOProduct School
Increased competition and higher customer expectations are forcing modern product management teams to up their game to keep pace. In this talk Richard White, CEO of UserVoice, talked about how product leaders are leveraging customer feedback, collected across the organization, to inform how they build the best product possible in the shortest amount of time.
Intro to Data Analytics with EA's Director of ProductProduct School
Building an analytics strategy is crucial for every product manager. How do you build and implement an effective product analytics strategy that quantifies and drives product success and iteration? Director of Product at Electronic Arts, Bertram Chan, talked about how data analytics is no longer a luxury, its a competitive and integral need for Product Managers. He also discussed ways to understand not only who your users are but how they behave.
Cracking the Product Manager Interview with Gayle McDowellProduct School
In this talk, Gayle McDowell, author of the book "Cracking the PM Interview", taught people how to prepare for Product Manager interviews, what top companies like Google, Amazon, and Microsoft really look for, and how to tackle the toughest problems.
She talked about how the role of a Product Manager varies across companies, what experience you need, how to make your existing experience translate, what a great Product Manager resume and cover letter look like, and finally, how to master the Product Manager interview questions.
Managing an Experimentation Platform by LinkedIn Product LeaderProduct School
Main Takeaways:
-Establishing a culture of experimentation at scale
-Developing the product vision and strategy
-Backlog prioritization based on Impact Score formula
A/B Testing for New Product Launches by Booking.com Sr PMProduct School
Main takeaways:
-There is no one right way of validating a product, A/B testing is just one of them
-Get your product 'qualitatively' validated before 'quantitatively' validating
-Use holdouts to measure long term success of your new products, while running A/B test in parallel
Getting Customer Validation of Your Product Before Release | Emily Hossellman...UCICove
About UCI Applied Innovation:
UCI Applied Innovation is a dynamic, innovative central platform for the UCI campus, entrepreneurs, inventors, the business community and investors to collaborate and move UCI research from lab to market.
About the Cove @ UCI:
To accelerate collaboration by better connecting innovation partners in Orange County, UCI Applied Innovation created the Cove, a physical, state-of-the-art hub for entrepreneurs to gather and navigate the resources available both on and off campus. The Cove is headquarters for UCI Applied Innovation, as well as houses several ecosystem partners including incubators, accelerators, angel investors, venture capitalists, mentors and legal experts.
Follow us on social media:
Facebook: @UCICove
Twitter: @UCICove
Instagram: @UCICove
LinkedIn: @UCIAppliedInnovation
For more information:
cove@uci.edu
http://innovation.uci.edu/
MAU Vegas 2016 — Lessons Learned From A/B Testing Wins and LossesGrow.co
Mobile Apps Unlocked Vegas 2016
Thursday, May 5
Retention Breakout Session — 3:30pm - 3:50pm
By now, we hope you have notepad’s full of ideas and new thoughts. If for some reason you have not hit your new things to try quota, don’t worry, our closing session will take you over the top featuring first-hand learnings from Life360's non-stop product experimentation engine.
Andreas Gross, Founding Team / Product @ Life360
Building a Product Strategy by Microsoft Sr PMProduct School
Main takeaways:
-Creating an elevator pitch for your product strategy
-3 main pillars for your strategy
-Go to market plan for minimum lovable product
This Event is Sponsored by:
Flatfile is the leading data onboarding platform for product teams. Stop spending time wrangling messy and unstructured customer data into your product. With Flatfile, your customers can seamlessly import their own data resulting in faster time to value.
[Webinar] Visa's Journey to a Culture of ExperimentationOptimizely
Join us as we hear Ramkumar Ravichandran, the Director of A/B Testing at Visa Checkout, explain how he created a high impact experimentation program. Ram will take us through the growth of Visa’s program: from selling the value, to laying down the vision, the roadmap and success criteria, to creating the right team and driving engagement with the program.
Attend this webinar to learn:
-How an experimentation program drives business impact.
-A model to drive continuous stakeholder engagement with the program.
-How to build a roadmap that goes above and beyond simple UX optimization.
Candid Conversations With Product People: Using Continuous Customer Testing f...Aggregage
After weathering recessions with a wide range of iconic customers, CEO and Product Manager Luke Freiler has seen first hand the impact the Voice of the Customer has had in making or breaking tech companies during hard times. He's going to walk you through an agile process for continuous customer testing that saves you time and gives you full confidence in your products — no matter how many you're sending out the door this year.
PM for Enterprise Software by Google Product LeaderProduct School
Main takeaways:
-Developing a strong relationship with sales
-Experimentation for enterprise products
-Roadmap development - Crafting your product vision
Product Development Demystified: Launching Faster with Confidence through Hum...Aggregage
Kandice Durden is here to discuss how leading organizations build, validate, and embed customer feedback into their development cycles. It’s time to take the guesswork out of product development by putting your customers at the center of your decisions!
Exploring how to take a product that been built for internal use by a company to market. Starting with why it could / should be done, discussing the stages in which this can be done, specific tasks for a Product Manager and finally various tasks for various team.
How to Achieve Customer Satisfaction Through Beta TestingCentercode
As the core stage of Customer Validation, Beta Testing offers a unique approach to product feedback completely different than any other testing strategy because you're able to evaluate the customer satisfaction of a product with your target market before it launches. Here, we'll teach you how to run an effective Beta Test, discuss the different types of feedback you'll collect, and show you what to do with the data you've collected.
In this webinar, we share our expert advice on how to run a successful B2B beta tests that will help you save time, money, and resources so you can hit the ground running with happy customers. Use this link to access the on-demand webinar: https://www.centercode.com/webinar/2016/august/
A/B Testing Best Practices by Hulu Product LeaderProduct School
Main Takeaways:
- Test Lifecycle: From Ideation to production and post-analysis
- Experiments & Features: Benefits of testing large features
- Top Challenges of A/B Testing: Main challenges product managers face as it pertains to testing
Similar to A/B Testing for New Product Launches by Booking.com Sr PM (20)
Webinar How PMs Use AI to 10X Their Productivity by Product School EiR.pdfProduct School
Explore AI tools hands-on and smoothly integrate them into your work routine. This practical experience is here to empower you, offering insights into the mindset of successful Product Managers. Learn the skills to become a more effective Product Manager.
Main Takeaways:
Hands-On AI Integration:
Learn practical strategies for integrating AI tools into your workflow effectively.
Mindset Insights for Success:
Gain valuable insights into the mindset of successful Product Managers, unlocking the secrets to their achievements.
Skill Empowerment for Growth:
Acquire essential skills that empower your evolution toward becoming a more effective and impactful Product Manager.
Webinar: Using GenAI for Increasing Productivity in PM by Amazon PM LeaderProduct School
In this webinar, you will learn how AI can take work off your plate, allowing you to focus on deep thinking or critical work. Cut out the drudge work in Product Management and get more out of your day.
Learnings:
Improve workflows that are high frequency - "manual tasks"
Increase the quality of output that has high importance - "brainy tasks"
Put GenAI to work today
Unlocking High-Performance Product Teams by former Meta Global PMMProduct School
Main Takeaways:
- High-Performing Team Dynamics: You’ll gain insights into fostering high-performance teamwork.
- Unveiling Team Personas: You’ll learn about different personas in the team and how to foster these differences.
- Decoding the Team Needs x Productivity Equation: You’ll learn about different team needs and how they correlate with engagement and productivity.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
4. 🗣 Do you enjoy Product
School events?
We want to hear from you!
Scan the code to fill out this quick questionnaire
so we can provide you with more value.
6. A/B testing new products
BY : Namrata Panwar
Sr. Product Manager, Booking.com [Dec 2020]
LinkedIn : https://www.linkedin.com/in/namratapanwar/
7. About me
● Namrata Panwar, Sr PM @ Booking.com, Amsterdam
● 8 years of experience
● Current : Working on partner facing products - onboarding,
performance, pricing for Home (Airbnb - like) partners
● Previously : with customer facing products in - E-commerce
startups
● Started career with Reliance Industries, as PM for Jio Mobile Apps
● Hobbies : Travelling, adventure sports, decorating my new house
8. Existing vs New product development cycle
A/B testing : What and Why?
01
02
Qualitative data vs quantitative data
03
Steps to NPD testing via A/B test
04
Recap
Agenda
9. ● Know basics about A/B testing
● Overview of a product development lifecycle
● Know about A/B testing techniques for new products
● Other ways for validating new products- apart from A/B testing
By the end of the
presentation....
10. A/B testing : What?
Base version Another version
In this example : Adding a buffer icon
before loading search results of a flight in
order to increase page views
Objective : To choose winning variant
based on a hypothesis
Showing 2 (or more ) versions to different
set of similar users, chosen randomly
Control A Variant B
V/S
Variant B Winning variant B!
50% 50%
11. Examples
Different landing cards allows the platform to tap into the psychology
of its viewers
Asking repeat users to ‘Continue’ with registration lead to more users
completing it
12. Making low risk modifications
(that can be easily undone)
Understanding users and solving user
painpoints
A/B testing : Why?
Introducing features and feature changes
13. Typical Product development cycle
Ideate
Design &
develop
Launch and
Iterate
Maturity
01
02
03
04
Lesser ambiguity, More experience
14. Existing product development
For such products, iterations are mostly
trials- regular A/B or multivariate testing
Fail fast : optimizing the user experience
to improve an existing product
Existing product : A product base exists
with reasonable user adoption
Tiny optimizations can have a big
impact -example changing the way
Microsoft’s search engine displayed ad
headlines increased revenue by an
astonishing 12%
Approach 1:
15. New product development
Example : Polaroids or Google extending
its product lines such as Google flights or
Google Health
Most amount of time is spent between
ideation and launch, focus on product
definition and qualitative feedback
New product : Establishing a new base
product from scratch -solving new
problems
No base for traditional A/B
testing
Approach 2:
16. No prior definition of success
Limited set of users/ testers
No base to iterate on
Traditional A/B testing :
Challenges with new
products
18. NPD testing
Step 1 : Qualitative data over quantitative data
Start with extensive qualitative research
Invest in getting design and research right upfront
Use Qualitative metrics that validates your product
: e.g. User satisfaction, Upvotes, Likes etc
Objective : Maintain high level of product quality
BEFORE you start A/ B testing and lower risk
Design
and build
Get Qualitative
feedback
19. NPD testing
Step 2 : Establish foundations
Launch this MVP to create a foundation-
using A/B tests with Non inferiority
success metrics
Decide on your MVP - Minimum Version of your
Product
MVP BASE
MVP
FINAL PRODUCT
W/o hurting metrics
20. NPD testing
Step 3 : Iterate, Measure success
Short term success: Running A/B tests to
introduce other modules -decide winning
variants
Long term success : Setting up a holdout
: excluding a small % of your users from
all tests for a long time (typically 6 mo- 1
yr) to compare performance
BASE
A/B test
V1
V2
10%
90%
21. NPD testing
Outcome : Winning variant ready to be ‘launched’ in market
V1 FINAL PRODUCT
Ready for launch
Winning variant
=
22. Some other ways of validating new products
FOCUS GROUPS
Invite a group of similar users and
giving them a trial/ demo
SUPPORT STAFF
Validate the product by those that
interact most with your users :
customer support, local offices, etc
BETA testing
Launching your product to a limited
set of users and getting extensive
feedback, before release to all public
High effort/ $$$
Low effort/ $
23. Depending on new vs existing product you will focus on
different lifecycle phases
A/B testing : Effective and easy way to learn and remove
biases 01
02
For NPD, define your MVP and introduce your base
(if it passes Non inferiority condition) 03
Measure long term performance of your new
product via holdouts 04
Recap
05
Focus groups and Beta testing are other ways of
validating new products
24. CREDITS: This presentation template was created by Slidesgo,
including icons by Flaticon, and infographics & images by Freepik
THANKS!
Do you have any feedback or questions?
Email : namrata16.it@gmail.com
Sr. Product Manager, Booking.com
LinkedIn : https://www.linkedin.com/in/namratapanwar/