James Klassen conducted several A/B tests of different page designs on an insurance company's website to better understand customers and improve conversions. Initially, changing to a design with two boxes led to decreased conversions. Further A/B tests found that removing one box significantly increased quotes obtained. However, the grey box design still performed well overall. The tests showed that assumed improvements don't always work as expected and bias must be set aside to truly understand user preferences through iterative testing.

1. Getting Strategic with Digital
Intro to Analytics
Research & Product Management:
How I Learned to Stop Worrying and
Love the Customer

2. Who am !?
• ME = James Klassen
• TALKING ABOUT = Research and putting the
customer 1st!
• If you have a question, please just yell!
• Background in market research, online marketing,
SEO, social media, analytics… blah blah blah
• Digital Product Manager @ AMA Insurance
• In addition to product management, also lead a
small, elite team of marketing hackers

3. Questions
• Who regularly uses analytics/research to
learn about their customers? Online
specifically?
• Who works in digital/ directly with people
in digital?

4. I have a case study!

5. First, some results

6. Which one converts better?

7. Which one converts better?
50% increase

8. Which one converts better?

9. Which one converts better?
75% decrease

10. Some context…

11. What are we trying to learn?
• Big question: who are the people using our
website?
– What do they want, how do they want it, what do
they already know?
– How can we organize our site to help them find
what they’re looking for?
– Because it’s insurance, are they looking for the
right things?
• We want to help educate about insurance
policies/coverage/etc.
– What functionality should we add/delete?
– How can we design our site better?

12. Our path
• Market research: understand broad strokes
• Online survey 1: User intent
– We had no idea, clean slate
• Traditional market research gave us direction, but we wanted to
validate online: were those people already talking to us?
– Tried to keep in mind that the state of our current site was
going to influence responses
• Online survey 2: Service preference
– Offline vs. online – many services not available online
– How do people want to interact with us?
– What type of insurance are they looking for?
• Online survey 3: Current customer?
– Already insured? Already a member?

13. Our path
• So, we decided to change our main-page
templates to reflect what we learned
• For /insurance/ and central products:
1. Make it easy to accomplish high-priority
tasks
2. Make it more obvious that you’re on an
insurance-related page
3. (/insurance/) make it easier to access
second-level self-service

14. Thus…

15. And…

16. Successes are easy… so about
that grey box...
• Initially:
– Made the change without testing… OBVIOUSLY
THE NEW DESIGN IS BETTER
– Saw big increase in bounce rate, decrease in
conversion rate
– This was basically a before/after test
• Ok, we need to learn more:
– Run it A/B style in a more controlled environment
(maybe external factors were impacting the
results… this is what I was hoping for)

17. 2nd test: A/B

18. 2nd test: A/B
70% decrease
in quotes

19. 2nd test: A/B
194% increase in
bounce rate

20. The horrible grey box
• Still really hate the grey box
• There must be something wrong inside of
the 2-box setup
– Let’s try different headings

21. 3rd test: A/B 3 treatments

22. 3rd test: A/B 3 treatments
58% increase
in quotes
133% increase
in quotes

23. The horrible, terrible grey box
• Simply removing 1 of the boxes, led to 133%
increase in quotes…
• Unfortunately, it looks like 2 boxes are just a
losing combination
• SIGH
• So we switched back to the original grey
box…

24. The horrible, terrible, ugly grey box
• BUT I still think we can make it better!
• So let’s try making the single box better!
– More visible text
– More visible box colour
– Remove overall page header

25. 4th test: A/B 4 treatments

26. 4th test: A/B 4 treatments
INCONCLUSIVE!!!

27. So what’s the outcome of
James vs. the grey box?

28. So where are we now with the grey
box?
• Slowly coming to terms with it
• We’re still learning and looking for ways to
improve
– “best practice” doesn’t work
• The fact is: it performs well!

29. So where are we now with the grey
box?
• What we should all learn:
– This stuff doesn’t always go according to plan
– Users don’t always want what you want them to
want
– Testing isn’t magical, it’s actually more work
– Check your bias – even THIS IS OBVIOUSLY
better doesn’t always work
– The same approach in 2 different contexts might
not work… and you might not realize there are
different contexts!
• Can’t set it and forget it... Once you start down the dark
path...

30. Some common mistakes
• Not using the appropriate research tool for the
question
• Making bias or stats-related errors in A/B tests
– Chasing 100% confidence
– Not determining the parameters of your experiments
– Not accounting for environmental factors like
seasonality
• Putting too much weight into results
• Testing too much, too big, too small
• Testing things you can’t change

31. Research tips
• If you don’t have research/ stats experience, get help
– Ask your internal business intelligence and market
research team(s)
– There are loads of great resources online
• Test a lot, but know you can test too much
– Easy to test ”too big” or “too small”
• Get to know your analytics platform
• Research isn’t just a/b testing
– Don’t forget about surveys, focus groups, analytics,
internal interviews (retail, product experts)
• Think about using Agile
– Puts the customer at the centre and that’s really what
we’re talking about here

32. Questions?

33. Thanks!
James Klassen
Digital Product Manager
AMA Insurance
AMAInsurance.ca
ca.linkedin.com/in/jrklassen
@forsoothed