Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
What	to	test	next?	
Prioritizing	your	tests
Pauline	Marol
Senior	Product	Manager
pmarol@hotwire.com
Josephine	Foucher
Opti...
Agenda • Where	to	start
• Defining	Prioritization	Rules
• The	Hotwire	framework
• Monitoring	the	Health	of	your	Optimizati...
Where	to	start?
Why	you	need	a	prioritization	process
• Managing	stakeholders
• Bring	transparency	to	optimization
• Build	trust	from	the	...
Know	your	destination
• Testing	profile
• Product	strategy
• Optimization	goals
5
Know	what	you	want
Know	yourself
• Get	your	analytics	right
• Listen	to	your	customers
• Look	at	your	competitors
6
Identify	your	opportuniti...
Structure	your	ideas
• Translate	ideas	into	hypotheses
• Centralize
• Organize	and	tag
7
Build	your	backlog	of	hypotheses
Defining	prioritization	rules
Bryan	Eisenberg	rules Monetate	modelPIE	framework Points	model
There	are	many	prioritization	frameworks
It	all	comes	down	to	Value	and	Effort
If	you	get	value	and	effort	right,	the	rest	is	easy
10
High
Low
High Low
Value
Effor...
Reach Lift Strategic	fit
Identify	the	drivers	of	Value	and	Effort
11
Value Effort
How	many	
people	will	be	
impacted?
Reas...
In	the	PIE	framework
12
Value
Reach Lift Strategic	fit
Effort
Creative Development Coordination
Potential Importance Ease
...
In	the	Monetate	model
13
Value
Reach Lift Strategic	fit
Effort
Creative Development Coordination
Potential Creative	LOE Te...
In	the	Bryan	Eisenberg	rules
14
Value
Reach Lift Strategic	fit
Effort
Creative Development Coordination
Impact Resources
T...
Define	binary	value	rules	that:
• Are	objective (yes/no)
• Discriminate ideas
• Leverage	your	knowledge
Taking	it	to	the	n...
Hotwire	Car	Framework
1. Primary	metric	is	
conversion
2. Adding	or	
removing	info
3. Research/analytics
4. Benchmarking
In	the	Points	model
17
...
Hotwire	Car	prioritization
If we	add	a	Free	cancellation	flag	on	the	checkout	page	
Then our	conversion	rate	will	increase...
Hotwire	Car	prioritization
19
Fold
Test 1
Hotwire	Car	prioritization
Free cancellation	flag
20
Fold
100%
Test 1 0
Fold
100%
Main	metric
Test 1 0 1
Fold
...
Hotwire	Car	prioritization
21
Fold
100%
Main	metric
Addingor	
removing
Analytics	or	
research
Benchmarking
$$	product
Canc...
Monitoring	the	Health	of	your	Optimization	Program
Use	KPIs	to	monitor	and	communicate	performance
Choose	metrics	based	on	your	risk	profile	and	
testing	strategy.
We	recomm...
Benchmark	your	prioritization	rules	to	stay	relevant
Validate	the	correlation	between	score	and	reality
• Value	score	vs.	...
Share	and	Educate
• Share	your	prioritization	rules	and	be	transparent
• Get	feedback	to	make	the	rules	better
• Educate	a...
Build	knowledge
• Aggregate	results	
• Translate	CR	lift	into	learning	and	customer	
insights
• Adjust	your	testing	profil...
Your	turn!
27
1 2 3
Do	your	research Build	prioritization	rules Keep	improving	them
We’re	Hiring!
Pauline	Marol
pmarol@hotwire.com
Josephine	Foucher
jfoucher@hotwire.com
Upcoming SlideShare
Loading in …5
×

of

[CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 1 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 2 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 3 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 4 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 5 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 6 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 7 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 8 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 9 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 10 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 11 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 12 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 13 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 14 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 15 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 16 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 17 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 18 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 19 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 20 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 21 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 22 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 23 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 24 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 25 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 26 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 27 [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol Slide 28
Upcoming SlideShare
[CXL Live 16] SaaS Optimization - Effective Metrics, Process and Hacks by Stephen Pavlovich
Next
Download to read offline and view in fullscreen.

2 Likes

Share

Download to read offline

[CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol

Download to read offline

Ideas are great - no doubt - but what do you do once you have an entire backlog of ideas? Prioritization is a critical part when building an efficient, impactful testing program. Creating a framework to make smarter choices and thinking deeply about the key factors will make your program successful in the long term.

In this session we will talk about how to create a strong prioritization process, how to keep it running, and how to constantly benchmark and optimize your process.

Related Books

Free with a 30 day trial from Scribd

See all

[CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol

  1. 1. What to test next? Prioritizing your tests Pauline Marol Senior Product Manager pmarol@hotwire.com Josephine Foucher Optimization Product Manager jfoucher@hotwire.com ConversionXL Live 2016
  2. 2. Agenda • Where to start • Defining Prioritization Rules • The Hotwire framework • Monitoring the Health of your Optimization Program 2
  3. 3. Where to start?
  4. 4. Why you need a prioritization process • Managing stakeholders • Bring transparency to optimization • Build trust from the company • Empower employees 4 Make your life easier
  5. 5. Know your destination • Testing profile • Product strategy • Optimization goals 5 Know what you want
  6. 6. Know yourself • Get your analytics right • Listen to your customers • Look at your competitors 6 Identify your opportunities
  7. 7. Structure your ideas • Translate ideas into hypotheses • Centralize • Organize and tag 7 Build your backlog of hypotheses
  8. 8. Defining prioritization rules
  9. 9. Bryan Eisenberg rules Monetate modelPIE framework Points model There are many prioritization frameworks
  10. 10. It all comes down to Value and Effort If you get value and effort right, the rest is easy 10 High Low High Low Value Effort Validate Hypothesis Do it! Forget it If you have the resources
  11. 11. Reach Lift Strategic fit Identify the drivers of Value and Effort 11 Value Effort How many people will be impacted? Reasons to expect a strong reaction Aligned with strategy All the frameworks break down value and/or effort into sub-elements, here are the most common. Creative Development Coordination
  12. 12. In the PIE framework 12 Value Reach Lift Strategic fit Effort Creative Development Coordination Potential Importance Ease Breaking down Value into Potential and Importance
  13. 13. In the Monetate model 13 Value Reach Lift Strategic fit Effort Creative Development Coordination Potential Creative LOE Technical LOE Breaking down Effort into Creative and Development
  14. 14. In the Bryan Eisenberg rules 14 Value Reach Lift Strategic fit Effort Creative Development Coordination Impact Resources Time Breaking down Effort into Resources and Time
  15. 15. Define binary value rules that: • Are objective (yes/no) • Discriminate ideas • Leverage your knowledge Taking it to the next level: The Points model 15 Value Reach Lift Strategic fit Effort Creative Development Coordination Define effort rules that: • Represent your bottlenecks • Are aligned with existing sizing Break down each sub-element into specific rules, to remove personal bias
  16. 16. Hotwire Car Framework
  17. 17. 1. Primary metric is conversion 2. Adding or removing info 3. Research/analytics 4. Benchmarking In the Points model 17 Value 1. Above the fold 2. All customers will see it 1. $$ product 2. Cancellation 3. Choice 4. Multiple themes Effort Creative Development Coordination UX/UI points Copywriting needs Engineering points Reach Lift Strategic fit
  18. 18. Hotwire Car prioritization If we add a Free cancellation flag on the checkout page Then our conversion rate will increase Because customers will feel more confident about reserving a car knowing that they can change their mind 18
  19. 19. Hotwire Car prioritization 19
  20. 20. Fold Test 1 Hotwire Car prioritization Free cancellation flag 20 Fold 100% Test 1 0 Fold 100% Main metric Test 1 0 1 Fold 100% Main metric Addingor removing Test 1 0 1 1 Fold 100% Main metric Addingor removing Analytics or research Test 1 0 1 1 0 Fold 100% Main metric Addingor removing Analytics or research Benchmarking Test 1 0 1 1 0 1 Fold 100% Main metric Addingor removing Analytics or research Benchmarking $$ product Test 1 0 1 1 0 1 0 Fold 100% Main metric Addingor removing Analytics or research Benchmarking $$ product Cancellation Test 1 0 1 1 0 1 0 0 Fold 100% Main metric Addingor removing Analytics or research Benchmarking $$ product Cancellation Choice Test 1 0 1 1 0 1 0 0 0 Fold 100% Main metric Addingor removing Analytics or research Benchmarking $$ product Cancellation Choice 2 themes Test 1 0 1 1 0 1 0 0 0 1 Fold 100% Main metric Addingor removing Analytics or research Benchmarking $$ product Cancellation Choice 2 themes Total Test 1 0 1 1 0 1 0 0 0 1 5 Fold 100% Main metric Addingor removing Analytics or research Benchmarking $$ product Cancellation Choice 2 themes Total UX Dev Test 1 0 1 1 0 1 0 0 0 1 5 S S
  21. 21. Hotwire Car prioritization 21 Fold 100% Main metric Addingor removing Analytics or research Benchmarking $$ product Cancellation Choice 2 themes Total UX Dev Test1 1 0 1 1 0 1 0 0 0 1 5 S S Test2 1 1 1 1 1 0 1 0 0 0 6 M L Test 150 0 0 1 1 1 1 1 1 1 1 8 L S
  22. 22. Monitoring the Health of your Optimization Program
  23. 23. Use KPIs to monitor and communicate performance Choose metrics based on your risk profile and testing strategy. We recommend: • Velocity • Win Rate • Conversion lift If your KPIs go down, revisit your rules.
  24. 24. Benchmark your prioritization rules to stay relevant Validate the correlation between score and reality • Value score vs. Win rate If your rules efficiency flattens, revisit your rules. 0% 25% 50% 75% 100% 0 2 4 6 8 10 Win Rate Value Score Benchmarking Value Points
  25. 25. Share and Educate • Share your prioritization rules and be transparent • Get feedback to make the rules better • Educate and scale outside your optimization team When you aggregate feedback, revisit your rules.
  26. 26. Build knowledge • Aggregate results • Translate CR lift into learning and customer insights • Adjust your testing profile, your optimization goals As you accumulate learning, revisit your rules.
  27. 27. Your turn! 27 1 2 3 Do your research Build prioritization rules Keep improving them
  28. 28. We’re Hiring! Pauline Marol pmarol@hotwire.com Josephine Foucher jfoucher@hotwire.com
  • YourVisibility

    Mar. 16, 2018
  • lkraav

    Apr. 4, 2017

Ideas are great - no doubt - but what do you do once you have an entire backlog of ideas? Prioritization is a critical part when building an efficient, impactful testing program. Creating a framework to make smarter choices and thinking deeply about the key factors will make your program successful in the long term. In this session we will talk about how to create a strong prioritization process, how to keep it running, and how to constantly benchmark and optimize your process.

Views

Total views

5,150

On Slideshare

0

From embeds

0

Number of embeds

3,874

Actions

Downloads

61

Shares

0

Comments

0

Likes

2

×