This document discusses building a culture of testing and optimization. It outlines key steps to create an effective test and learn organization, including defining roles like sponsors, ideators, executors and analyzers. It emphasizes the importance of developing a testing plan, conducting regular testing meetings, and standardizing the testing process. The overall message is that organizations should start with basic testing to establish processes and not panic if early tests show no significant results, as the goal is to learn through testing.

1. © 2016 Tealium Inc. All rights reserved. | 1
Building a Culture of Testing and
Optimization
Joel Ring, HUB International

2. 2000 2006 20152009
Joel Ring
Director of Analytics, HUB International
15+ Years in digital analytics
Created and
managed the Web
Analytics
department at
bestbuy.com
Sr. Digital
Marketing
Consultant
working with
some of the
biggest brands
on the web
Created the
Digital Strategy &
Solutions team
helping clients
with their
analytics and
digital marketing
Manage the
Analytics team
focusing on
digital analytics,
data sciences,
and testing &
optimization

3. A lot has changed
2000 2006 20152009

4. The great thing
about the internet
is you can
measure anything!
We’re ‘gonna fail-
fast and break
things!
Like Facebook!
The only way to
know what will work
best is to test it!

5. With all the data
generated through
testing we will be
able to understand
our customers like
never before and
learn new ways to
meet their needs.

6. Every minute…
• Facebook users share nearly 2.5 million pieces of content.
• Twitter users tweet nearly 300,000 times.
• Instagram users post nearly 220,000 new photos.
• YouTube users upload 72 hours of new video content.
• Email users send over 200 million messages.
• Amazon generates over $80,000 in online sales.

7. “Two-thirds of business are
unable to extract value from their
information.”
Source: 2015 Information Value Index from PwC and Iron Mountain

8. If we have access
to all this data,
how do we create
an environment
that enables us to
derive value from
that data?

9. Definition of
an effective
test and learn
organization
look like?

10. Implemented roles and procedures
to create, execute, and evaluate
tests
AND
are actively communicating results
to drive change.
“The key to great analysis
is not math,
it’s communication.”

11. Keys to
creating a
test and learn
organization
Sponsors Ideators
Executors Analyzers
The sponsor helps evangelize
the testing organization and
helps drive changes based on
the results.
The ideators help develop testing
ideas. These can be part of the
testing team or stakeholders.
This is the group that executes
the tests. This will generally
encompass multiple departments
and can include creative and
technical resources.
The analyzers tabulate and
communicate the results of the
tests. They will be the team
members who declare the test
“winners”.
• Individuals can, and will, fall into multiple roles
• Include people from multiple departments
• Make sure to include stakeholders
• How many people in these roles?
• Forrester estimates 5-15

12. How do I get the initial buy in?

13. Demonstrate value using
existing channels that
already allow for testing
Email – Display Advertising – Paid Search
How do I get the initial buy in?

14. Conduct a kick off meeting

15. Conduct a kick off meeting
• Bring together people who will be either involved in or interested in the
execution and effectiveness of your testing program
• Goal of meeting is to educate generate excitement through
collaboration
• Kick off agenda
o Testing 101
• Types of test (A/B, multivariate, etc.)
• What makes a good test
• What is a testing plan
o Intro to our testing platform
o Intro to our testing process
o Ideation workshop
• Used the ideation ideas to start our testing prioritization

16. Develop a testing plan

17. Develop a testing plan
• Guides your testing prioritization, status and results
• What is in a testing plan?
o List each test idea
o Conversion
o Status (Live, scheduled, complete, removed, on-hold)
o Date launched
o Date completed
o Results (winner)
o Test type (A/B, A/B/C, multivariate)
o Test location (site, email, SEM, display advertising)
• Keep in a central location accessible by the team

18. Test Name
Test
Cell Test Description Conversion Test Type Test Location Status
Launch
Date
E
D
Control: Current quote first position
Test:
The IRS is expecting “many filings” to be in error at some level. No one can just blow off the law, but there will be something
of a pass on reporting penalties as long as it can be demonstrated that effort went into compliance. Walter Fagan
Control:
What’s dominating the Employee Benefits agenda in 2016?
Get the answers to the big questions →
Test A:
What’s dominating the Employee Benefits agenda in 2016?
Get expert insights and tools you can use in the year ahead →
Control: current template design
Test: Test: plain text format that will appear personalized including 1. Dear [name] 2. Sincerely [regional EB president]
Control: no nurture email
Test: CMnurture series to those who completed ACA nurture Ems
Control: always on
Test A: second panel of ACA site
Test B: new contact form to be designed for campaign
Control: no button/SMbuttons
Test: remove SMbuttons and add contact us button that goes to contact a rep form
Control: https://www.hubinternational.com/employee-benefits/contact-us/
Test: modal pop-up
Control: Current copy
Test:
Headline: Optimize your Employee Benefits in 2016
Box 1: ACA Reporting & Verification Audits
Box 2: Cost Management Strategies
Q1 Campaign Site
email
Live 9-Feb T
Complete 4-Feb 1EB Always On HP Interstitial Link clicks A/B Q1 ACA Campaign Site
ACAPanel Jack Quote 1 Share button clicks A/B
TBD T
CM nurture to ACAlist
Increase lead
pipeline
A/B email
Live 9-Feb T
CM nurture email #2 template Email CTR A/B On-Hold
Contact Us button location Button clicks A/B Q1 ACA Campaign Site
ACAAd words Landing page test
Form Completion
Rate
A/B/C Ad words
Live 7-Mar
Complete 22-Feb 11
T
Contact Rep FormDesign - sub
pages
Form Completion
Rate
A/B Always On Site Live 9-Mar
L1 page CTA Link clicks A/B Q1 L1 page Complete 9-Mar 6

19. Conduct regular testing/analytics meetings

20. Conduct regular testing/analytics meetings
• Include core team and stakeholders
• Use the testing plan as a guideline for the agenda
o Review tests in progress
• Determine any status changes
o Review tests scheduled to launch
o Discuss any new tests
o Reprioritize based on new tests (if necessary)
o Present deep dives from previous meeting questions
• Use the meeting to generate new test and analysis ideas

21. Standardize the test process and embed it in business as
usual

22. DK Cool CrayonTestapproved BuildtestAsset(s)
needed?
Add to testplan N
Create
asset(s)
Y
Review test Launchtest
Wait 7days

23. DK Cool CrayonTestapproved BuildtestAsset(s)
needed?
Review test
Create
asset(s)
LaunchtestAdd to testplan
Y
N
Wait 7days
Why wait 7 days?
• Early results show greater variance.
• Behavior often varies by day of week,
want to ensure all days are
represented.

24. DK Cool CrayonTestapproved BuildtestAsset(s)
needed?
Review test
Create
asset(s)
LaunchtestAdd to testplan
Y
N
Wait 7days
Declare
winner?
Y
Review meeting
N
Determinenext
steps
Updatetest plan End
Suspend test
DeclareWinner
Continue testing
Updatetest plan
Updatetest plan
End
Revisetest?
N
Y
Review results

25. Final
thoughts

26. Start Slow

27. Start Slow
Use basic tests to
establish your process,
then grow into complexity

28. Don’t panic if your tests don’t have significant results

29. You still learn from tests
with no differences
Don’t panic if your tests don’t have significant results

30. Implemented roles and procedures
to create, execute, and evaluate
tests
AND
are actively communicating results
to drive change.

31. Questions?

32. THANK
YOU