“The Evolution of All 4 Product: A story told by User Testing” User Testing forms the fundamentals of any product design or development process. In her talk Divya shares how the All 4 product team incorporate user testing and data into their experimentation strategy.

1. A story told by User Testing

3. • On >20 different platforms
• 16M registered users
• ½ of all 16-24s
• ~2 million views per day
• ~10% of Channel 4 revenues
About All 4

4. Who we were...

5. Who we are.

6. Who am I?
Divya Isaiah
User Testing Manager – All 4
All 4 Product
Qualitative Testing
Quantitative Testing Contact Info:
divyaisaiah
disaiah@channel4.co.uk

7. How did we get here?
• When & How
• Where
• ‘Some’ challenges we face

8. Problem Concept Research Hypothesise Prioritise Test Learn Build
Problem statements
Performance analytics
Viewer enquiries
App store reviews
Exploratory research
User observations
Idea generation
Concept backlog
Observe & Deduce
Feature analytics
User research
Usability testing
Competitor research
Core 4 Panel
Product backlog
Testing backlog
Designs
Prototypes
Planning
Potential
Impact to KPIs
Effort
Validate designs
Test prototypes
Test builds
AB/MVT testing
Induce learnings
Communicate findings
Refinement
Concept development
Build & Release
Feature performance
User Validation User Validation User Validation User Validation
When & How: Tool box of methods
Learn & Iterate
Test live app
Evolve
Use all the tools in your tool box

9. How: Hypothesis Capture
We believe that by doing [A] for [B – people/segment] we
will make [C - outcome] happen. We will know this when
we observe [D - data] and / or obtain [E – feedback].
Don’t throw away hypothesis

10. How: Prioritisation methodology
HIGH IMPACT (Quantifiable) SOFT IMPACT (Meaningful)
Higher Conversion rates:
• VoD views
Metrics we want to maximise:
• Unique visitors
Metrics we want to reduce:
• Errors
Changing perceptions
• Catch up > Browse
Internal buy in
• Informs teams
Data driven decision making
• Informs roadmaps
Technical Teams
• HTML
• CSS
• Java script
• Backend changes
• Advanced audience
tracking
• Time to run
• Agency
• Designers
• Homepage Editors
• Picture team
• Developers
IMPACT FACTORS EFFORT FACTORS
Also think about your ‘Meaningful’ metrics

11. active viewers
Access & Connect Find Watch Continue
visitors to viewers views views per viewer
Where: Customer Experience Map

12. Access & Connect Find Watch Continue
REGISTRATION
SIGN IN
GLOBAL NAV
HOMEPAGE
SEARCH
PLAYBACK
WATCH LIVE
BRAND PAGES
MY 4
ONWARD JOURNEY
CHANNELS
CATCH UP
CATEGORIES
COLLECTIONS
ADVERTS
BOX SETS
EMAIL MKT
TASTE SEG
Where: Opportunities & Challenges
84
Look at the whole experience
to identify opportunities

13. Tests: Watch Live

14. Tests: Watch Live results
• ↑ Watch Live stream starts
• ↑ Minutes watched per unique
visitor
• ↑ Non-Watch Live VoD starts per
unique visitor

15. Tests: Homepage hero
↓ in click through rate on #1
item on hero

16. Tests: Homepage hero
↓ in click through rate on #1
item on hero
↑ in click through rate on #1
item on hero

17. The challenges we face
1. How to test?
2. Dynamic content
3. Keep learning

18. Challenges: How to test?
1. How
o What is it we are trying to do?
o New Feature
o Optimise
o What do we want to get out of it?
o Insights
o Incremental Improvements to KPIs
2. Where & Who
o Journeys
o Technology Stack
o Devices
Again,
Use all the tools in your tool box

19. Challenges: How to test on the Homepage
Dotcom Big Screens Tablet Mobile
Know your devices

20. Challenges: Dynamic content
Hero
Most Popular
My ListWatch Live Recommendations
Continue watching
Test your assumptions & rules

21. Challenges: Sharing Learnings

22. Summary
Test Everywhere
Look at the whole experience to identify opportunities
Think about your ‘Meaningful’ metrics
Test your assumptions & rules
Use all the tools in your tool box to test
Don’t throw away hypothesis
Analytics is important but listen to your customers
Know your devices

23. Can you create your own story?
Questions?
Thank you!!