Shaun Rowland -- Director, Insights and Personalisation for AKQA, and Luke Corkin -- Corporate Account Executive for Optimizely with our latest webinar from the Experimentation Insights Tour. View the webinar here: https://optimizely.wistia.com/medias/f49pqtsy98 DESCRIPTION: Why the “Test and Learn” culture is critical to your CRO success. Go beyond just knowing the need for experimentation and build a strategy for ongoing success. Learn the top challenges involved in developing and ensuring a long-term experimentation vision. You'll learn: * What roles and personalities you need to recruit for the team * How to establish test and learn guardrails that keep your program on track with your business strategy * Prioritisation and valuation of experimentation pipeline * The tools and process used to track the health of your programme

1. August 2017

4. A L I T T L E B I T A B O U T M E …
Shaun Rowland
Director, Insights and Personalisation
AKQA Melbourne
• 15 years digital marketing experience in
Australia and the UK
• Worked for brands Bupa, NAB,
Officeworks, Kmart, adidas and Suncorp
• At AKQA for 2.5 years managing its I&P
team in Melbourne
• Optimizely partner for 2.5years +

5. This is a story of an experimentation program gone wrong

6. And how it was rescued
It’s loosely based on a true story. Names have been changed to protect sources.

7. Meet Amelia…
• Digital Marketer at a national retailer
• Good relationship with IT and Commerce teams
• Responsible for establishing and managing testing
program

8. Amelia at the start of her testing journey

9. Amelia after twelve months…

10. R O A D B L O C K S
3. No Clear Direction
No sense of how testing laddered up to overarching
business goals and objectives
4. Competing Interests
Website had multiple owners with different priorities and
objectives.
1. Ideas couldn’t be executed
Agency wasn’t fully across technical constraints so many
tests were not possible to implement
2. Lack of practical experience
Amelia had too many marketers in her experimentation
team and not enough people who had hands on testing
experience

11. R O A D B L O C K S
7. Business momentum slowed
Tests focused on easy to change but prepherial issues to
the customer experience and delivered only marginal
improvements
8. Management of tests was cumbersome and slow
Communication and production was managed through
email and spreadsheets.
5. Implementation of custom tags
Custom event tagging had to be implemented by
developer in IT who had other tasks and priorities
6. Test plans were weak
Traffic volumes were too low for many proposed tests
which resulted in tests being scrapped or left running too
long

12. Amelia was defeated by the three Ps:
People, Politics and Process

13. In twelve months Amelia had launched just three tests.
The program was on life support

14. 5 F ATA L A S S U M P T I O N S
4. The client / agency relationship could continue as is
She assumed this type of work could be run along similar lines to
campaign work
5. The team was aligned around a common purpose
She assumed everyone on the team was aligned with her marketing and
business objectives and that the testing program was the strategy that
would help her meet these
1. Technology was the key to success
She assumed getting the technology implemented meant much of the
job was done
2. The business was excited and motivated
She assumed other people in the business had as much of an interest in
her program as her
3. The right team was in place
She assumed the team she had created contained the skills she needed
for the creation and execution of great test ideas

15. How to avoid Amelia’s mistakes and set yourself up for long
term experimentation success
H O W W E F I X E D I T

16. Technology is only one piece of the puzzle

17. Strategy
•Testing guardrails
•Alignment to business objectives
•Accountability
T H R E E P I L L A R S O F S U C C E S S
S T R A T E G Y

18. Strategy
•Testing guardrails
•Alignment to business objectives
•Accountability
People
•The right mix of skills and aptitude
•Diversity of views and backgrounds
•Thinkers and doers
PEOPLE
T H R E E P I L L A R S O F S U C C E S S
P E O P L ES T R A T E G Y

19. Strategy
•Testing guardrails
•Alignment to business objectives
•Accountability
People
•The right mix of skills and aptitude
•Diversity of views and backgrounds
•Thinkers and doers
Process
•Effective ideation
•End to end production
•Tools
T H R E E P I L L A R S O F S U C C E S S
P R O C E S S
PEOPLE
P E O P L ES T R A T E G Y

20. T H R E E P I L L A R S O F S U C C E S S
V A L U EStrategy
•Testing guardrails
•Alignment to business objectives
•Accountability
People
•The right mix of skills and aptitude
•Diversity of views and backgrounds
•Thinkers and doers
Process
•Effective ideation
•End to end production
•Tools
P R O C E S S
PEOPLE
P E O P L ES T R A T E G Y

21. Align and Galvanise
S T R AT E G Y

22. Test and Learn is not strategy
Establish your strategic guardrails

23. Ease of use can lead to misuse
Prepare the team before diving in

24. Without strategic guardrails you can very quickly get off track

25. O B A M A 2 0 1 2
S T R AT E G Y
Get many small donations from many people

26. O B A M A 2 0 1 2
S T R AT E G Y
Get many small donations from many people
E N A B L E R
Use A/B testing to understand what content and
user experience was most effective in driving small
donations

27. O B A M A 2 0 1 2
S T R AT E G Y
Get many small donations from many people
E N A B L E R
Use A/B testing to understand what content and
user experience was most effective in driving small
donations
R E S U LT
Over 500 tests delivered 49% increase in
donations, 1.6x improvement in email sign ups

28. S T R AT E G I C F R A M E W O R K
OBJECTIVE OBJECTIVE OBJECTIVE
T E S T I N G P R O G R A M
PRINCIPLES PROCESS PEOPLE
O N L I N E M A R K E T I N G G O A L S
OBJECTIVE OBJECTIVE OBJECTIVE
OBJECTIVE OBJECTIVE OBJECTIVE
B U S I N E S S G O A L S

29. M A P T O Y O U R B U S I N E S S ’ S T R AT E G Y
B U S I N E S S G O A L S
Increase RPU by $10
Increase online acc
registrations by 10%
Increase NPS Score by 20pts
O N L I N E M A R K E T I N G G O A L S
Increase revenue by 5% Reduce cost of service by 20%
Increase retention of customers
by 10%
T E S T I N G P R O G R A M
Principles Process People
Increase attachment sales
Reduce online account
abandonment
Increase ease of use and
personalisation effectiveness
E N A B L E R S

30. Galvanise your team around this common vision and strategy
Where are we headed and how are we getting there?

31. Get the right people involved
P E O P L E

32. People are attracted to people like them

33. Strive for diversity

34. Get the right balance of thinkers and doers
40% 60%

35. Think about the people and skills you need end to end
Ideation Planning Management Production Execution Analysis
Programming
Optimizely Set up
HTML
Design
Copywriting
Team Management
Resourcing
Prioritisation
Time Management
Statistical Analysis
Business Analysis
Commercial
Modelling
Resourcing
Prioritisation
Presentation Skills
Creative thinking
Analytical thinking

36. T
Hire
Shaped Talent

37. O P T I M I Z E LY
W E B T E S T I N G
O P T I M I Z E LY
P E R S O N A L I S AT I O N
O P T I M I Z E LY
F U L L S TA C K
W E B A N A LY T I C S
F R O N T E N D
D E V E L O P M E N T
B U S I N E S S
A N A LY S I S
C R E AT I V E
I D E AT I O N

38. Look beyond your team
Democratise your testing program

39. Find enthusiastic sceptics

40. Dry but vital to long term success
P R O C E S S

41. Establish your testing principles
What are your dos and don’ts?

42. Architect your process

43. C R E AT E A P R O C E S S T H AT W O R K S F O R Y O U
E X P L O R E P R I O R I T I S E T E S T
B A C K L O G
T E S T
P L A N S P R O D U C T I O N A N A LY S E A C T
Divergentthinking
Convergentthinking
Divergentthinking
Convergentthinking

44. F I N D T O O L S T H AT H E L P Y O U M A N A G E T H E P R O C E S S

45. Aggressively focus on showing returns to your business
V A L U E

46. Find quick wins and ruthlessly prioritise them
Don’t lose momentum and business energy

47. Give the right people the right information at the right time
Transparency helps you build relationships and gain traction

48. People, process and politics dealt with…
T H E R E S U LT S

49. Within three months Amelia had delivered:
• 8 tests to market
• Prioritisation of 28 tests in backlog
• Incremental uplifts across all three marketing objectives

50. K E Y TA K E O U T S
1. Don’t focus on the technology
2. Consider the three pillars of success
•Strategy
•People
• Process
3. Deliver value fast
•Quick Wins
•Share success