Jon Warden and Shruti Ahuja share their story how they established a Testing Culture at Haymarket during the Optimizely Experience London 2014.

1. HOW HAYMARKET IS
ESTABLISHING !
A TESTING CULTURE
Jon Warden
Head of User Experience
!
Shruti Ahuja
Head of Analytics and Optimisation

2. Jon Warden
Head of User Experience!
Haymarket Media Group
!
uk.linkedin.com/in/jonwarden/!
@jonwarden
My background!
10 years at News UK in various design & UX roles
5 years as a UX consultant, freelancing at various organisations
Haymarket to date: Initial set up, and 2 years management of internal UX team

3. Shruti Ahuja
Head of Analytics and Optimisation!
Haymarket Media Group
!
uk.linkedin.com/in/shrutiahuja/
My background!
8 years experience in Analytics and Optimisation across Media, eCommerce, CE, Automotive, Travel and
Financial services
Haymarket to date: Develop analytical infrastructure across 20 B2C brands
Haymarket this year: Expand Analytics to B2B brands and strengthen Optimisation structure for B2C brands

4. “One accurate measurement is worth a thousand expert
opinions”
– Grace Murray Hopper, Computer scientist and US Navy Admiral

5. Who are Haymarket Media Group?

6. AN INTERNATIONAL PUBLISHING HOUSE
We create award winning specialist content for international audiences
• 72 market leading brands
• 20 offices
• Across six countries
• 1500 + staff
• Business Media division
• Consumer Media division
• For Print, Digital, Live Media and events

7. AN INTERNATIONAL PUBLISHING HOUSE
Some of our digital brands

8. AN INTERNATIONAL PUBLISHING HOUSE
Some of our digital brands

9. AN INTERNATIONAL PUBLISHING HOUSE
Some of our digital brands

10. Our journey with Optimizely

11. WITH OPTIMIZELY SINCE MAY 2013
• We started off on a month-by-month Gold
package
!
• We completed very basic experiments on:
Whatcar.com
PistonHeads.com
CaravanSiteFinder.com
!
• We quickly found we were regularly hitting
our traffic allocation
• In April 2014 we upgraded to Platinum

12. WHY WE UPGRADED
• We needed dedicated support
• We needed unlimited projects, user accounts,
and more features to use on all our brands
• Other areas of the business were interested in
using Optimizely, including international divisions
• We now have dedicated accounts for:
Haymarket Business Media!
Haymarket Consumer Media !
PistonHeads.com

13. The Optimisation process

14. OPTIMISATION & BUSINESS PRACTICE
Each company has their own process for A/B testing
Test report
created
Success/fail
Assumption/
hypothesis
reported to
Product team
Publisher/feature
request
Business criteria UX / UI / FE
create experiment
Analytics reports
Test runs for 2
weeks minimum
Product / UX /
Analytics
define test
No
Yes
Test report
created
Sent to business for
approval
Budget agreed to create dev
Pushed to live work for sprint
Test fail

15. OPTIMISATION & BUSINESS PRACTICE
Each test goes on the testing road map

16. OPTIMISATION & BUSINESS PRACTICE
Each test has a reporting document and test ID

17. OPTIMISATION & BUSINESS PRACTICE
• The reports

18. OPTIMISATION & BUSINESS PRACTICE
• The reports

19. OPTIMISATION & BUSINESS PRACTICE
• The reports

20. IN AN IDEAL WORLD…
• Over all our process is a good one, but…
• We’ve found that we’ve had a few issues
• Maintaining focus, and learning is difficult, we’re a busy group
• We don’t have a dedicated Optimisation team, yet!
• Everyone has their own day jobs, business, analytics, product, ux, editorial
• Resourcing - we have small analytics and UX teams
• Business see the validity, but it isn’t business critical in certain disciplines
• Getting budget released or time for iteration is difficult due to dev roadmap

21. SOLUTIONS
• We’re trying to change the mindset and the culture
• We need to create an Optimisation process ‘all’ buy into
• We’ve enlisted the help of Optimizely experts to help us
• Getting onsite training for the team
• Discussing testing strategies with senior management
• In the past we’ve hired a third party CRO agency to get more focus
• We need to create an in house dedicated team to focus efforts

22. Experiments - what we’ve learnt

23. WHAT WE’VE LEARNT
1. Do - ‘Start simple’, until you know what you’re doing
2. Do - Create as many tests as possible, and run them consecutively
3. Do - Measure the numbers, and then learn about the results
4. Do - Leave experiments to run as long as needed
5. Do - Embed a testing culture into your business
6. Do - Ask for help to learn more

24. WHAT WE’VE LEARNT
1. Don’t - run before you can walk, but learn as you go
2. Don’t - try experiments with too many goals
3. Don’t - create A/B and MVT tests with too many variants at first,
without having the proper knowledge
4. Don’t - launch multiple experiments on the same site, at the same
time, they have a tendency to clash with each other, causing
experiments to fail
5. Don’t - forget to inform stakeholders you’re running tests

25. WE’VE SIMPLIFIED OUR TESTS DOWN, !
AND HAD SUCCESS
• We recently created an experiment on one of our sites
• Using two styles and four variants
Variant A Variant B
Variant C Variant D

26. WE’VE SIMPLIFIED OUR TESTS DOWN, !
AND HAD SUCCESS
• The results were astounding!
• Variant D had an uplift of +197.6%

27. PISTONHEAD: ANALYSIS-LED
OPTIMISATION
Shruti Ahuja
Head of Analytics and Optimisation

28. ABOUT PISTONHEADS.COM
• Connects car enthusiasts to
dealers and one another
• An automotive classifieds
platform
• One-stop car shop

29. HOW DID WE CREATE OUR
OPTIMISATION ROADMAP?

30. WHAT IS OUR OPTIMISATION PROCESS?
Insights
Generation
Roadmap
Creation
Test
Learning
Results Implementation
Analysis

31. WHAT DETERMINES OUR TESTING ROADMAP?
• Identify KPI’s that support the
Business Model
• Investigate issues that hinder
KPI conversion
• Generate ideas to test and
improve performance on an
on-going basis
!!!!!! Visitors
!!!!!! Inquiries
!!!!!! Leads
!!!!!! Sales

32. WHAT INSPIRES OUR TEST IDEAS?
• Monthly dashboard reviews
• Ad-hoc analysis
• Product related issues

33. OUR MOST SUCCESSFUL TEST
Objective:!
Determine if introducing a new layout with an
inbuilt Contact Seller form and a large image
increases site conversion
!
Hypothesis:!
Currently users have a 2-step process to
contacting sellers. The simplification of this
and introduction of larger images will improve
interaction with the ad details pages
!
Test Description:!
A/B test. Control (Existing content: 2 step form
& small images), Alternate B (In-page form, in-page
images)
!
Success Metrics:!
• Traffic
• Retention
• Contact Seller Conversion
• Revenue

34. OUR MOST SUCCESSFUL TEST

35. WHEN TESTS FAIL…
Objective:!
Understand how user behaviour is affected by
using a car like VRM on the parts and plates
landing page.
!
Hypothesis:!
The current parts and plates landing page
design mirrors the old style SSI landing page.
The new VRM form style has been seen to
perform better for cars. This test will confirm
that the design works for parts and plates too.
!
Test Description:!
A/B test. Control, variation B - car-like VRM
!
Success Metrics:!
Proceed to next step
Ad creation conversion rate

36. WHEN TESTS FAIL…

37. WHAT WE LEARNT
• Optimisation is a learning
process
• Never assume / preempt
• You win some, you lose some
• In either instance, test further!!

38. THANK YOU!
QUESTIONS?