This document summarizes a presentation on experimentation and product development. The presentation covered: - 8 lessons for running effective experiments, including choosing the right tools, spending enough but not too much, and learning from results - Examples of experiments run at 99designs on growth, activation, and conversion goals across email, search, and cross-selling - The importance of making decisions based on experimental results, including potentially retiring features that did not perform as expected

1. Leading The Product 2017
Speaker Slides
Sydney, Australia
Susan Teschner
99designs
For more information go to
www.leadingtheproduct.com

2. © 99designs, Inc.© 99designs, Inc.
Experimenting
with
ExperimentationPro tips for doing it well
by Susan Teschner

3. © 99designs, Inc. 3
Who am I?
some days I’m not sure

4. 4
design
Retail
catalogues &
ad networks
Accounting
software
VoIP
Shipping for
eCommerce
Classifieds
99
iNC
MYOB
Delta 3
Australia Post
Fairfax
SEEK

5. © 99designs, Inc. 5
Manage products.
Manage people
managing
products.
by Evilltimmby LanaD.

6. 6
World’s largest marketplace
for graphic design
We’re very busy
—
450,000happy customers
We’re famous but with
hidden talents
—
36
Design categories
by Vladson
We’re home grown and
a bit grown up
—
2008
Founded in Melbourne
115
Fabulous people
3
Offices (BER, OAK, MEL)
$200mpaid to designers

7. © 99designs, Inc. 7
Experimentation?

8. © 99designs, Inc. 8
“doing a thing to discover something
unknown or to test a principle”
— poorly paraphrased from dictionary.com

9. 9
The universe of experiments
—
by Gorcha
by huhi

10. © 99designs, Inc. 10
So this is an agile thing?

11. © 99designs, Inc. 11
Where are
they now?
by kazoe

12. © 99designs, Inc. 12
How does it help?

13. © 99designs, Inc. 13
8 lessons

14. © 99designs, Inc.
DO pay attention
—
1

15. © 99designs, Inc. 15
I don’t have all the
ideas
—
by Thedaydreamer

16. © 99designs, Inc. 16
How 99designs was
born…
—
by xeniatm

17. © 99designs, Inc. 17
Why we created 1:1
Projects
—
by ktoons

18. © 99designs, Inc.
DON’T underestimate the
importance of experiment design
—
2

19. © 99designs, Inc.Commercial in confidence 19
Start off on the right foot
—
▪ clearly stated hypothesis
▪ definition of success
▪ specific metrics
by ktoons

20. © 99designs, Inc. 20
Beware the dragon of
statistical significance
—
by ilustreishon

21. © 99designs, Inc. 21
Here’s how
—
https://unbounce.com/ab-test-duration-calculator/
Sample Size & Test Duration Calculator
Current conversion rate (%)
Desired lift (%)
Number of variations
Average daily visitors
To achieve valid test results*, you need:
A total sample of (users):
Test duration of (days**):
10800
5
10
20
3
2000

22. © 99designs, Inc.
Sample Size & Test Duration Calculator
Current conversion rate (%)
Desired lift (%)
Number of variations
Average daily visitors
To achieve valid test results*, you need:
A total sample of (users):
Test duration of (days**):
10800
54
10
20
3
200
Sample Size & Test Duration Calculator
Current conversion rate (%)
Desired lift (%)
Number of variations
Average daily visitors
To achieve valid test results*, you need:
A total sample of (users):
Test duration of (days**):
10800
5
10
20
3
2000
22https://unbounce.com/ab-test-duration-calculator/
Here’s how
—

23. © 99designs, Inc. 23
This was a bit too
long to wait
—
How long will your experiment take?
How many eligible visits do you have per day?
What percentage of visits will see the change?
What is your current conversion rate?
How will you change conversion?
How confident do you want to be of this?
How likely should you be to detect the change?
1,400
50%
2%
5%
95%
90%
You should run this experiment for 1 year, 7 months and 28 days.

24. © 99designs, Inc.
DO choose the
right tools
—
3

25. © 99designs, Inc. 25

26. 26
other data
sets
Performance
UI / UX
Resourcing
Maturity
$
Things to think about
—

27. © 99designs, Inc. 27
So, who’s your favorite?
—
by glasshopperart

28. © 99designs, Inc.
DON’T spend too much
—
4

29. © 99designs, Inc. 29
Beware the gold plating
—
by MattDyckStudios
https://www.quora.com/How-is-pretotyping-different-from-prototyping

30. © 99designs, Inc. 30
Baby steps towards a
Gift Shop
i1
i2

31. © 99designs, Inc.
…but DO spend enough
—
5

32. © 99designs, Inc.© 99designs, Inc. 32
“We increased
sales by 37%
and all we had
to do was
change the
color of a
button!
by Pierre d’Arterie

33. © 99designs, Inc. 33
This was never
going to wow.

34. © 99designs, Inc. 34
by Hildebrand

35. © 99designs, Inc.
1
Testing tools
—
2
Variations
—
3
Opportunity cost
—
by RaidenD

36. © 99designs, Inc.
DON’T ignore the results
—
6

37. © 99designs, Inc. 37
If it seems too good to be
true…
—
by Frankie Cadillac

38. © 99designs, Inc. 38
Share the results.
Publicly.
—
by Sam!

39. © 99designs, Inc.
DO make decisions
& follow through
—
7

40. © 99designs, Inc. 40
R.I.P.
—
by RVST

41. © 99designs, Inc. 41
… or put your money
where your experiment is
—

42. © 99designs, Inc. 42
… or take your chances
with the people who look
after it
—
by Thedaydreamer

43. © 99designs, Inc. 43
A story about Social Sharing
by clouds940

44. © 99designs, Inc. 44
by clouds940
A story about Social Sharing

45. © 99designs, Inc. 45
A story about Social Sharing

46. © 99designs, Inc. 46
A story about Social Sharing
by miljanristic by FRITO-DESIGNstudio
by Joedsign

47. © 99designs, Inc. 47
by clouds940
A story about Social Sharing

48. © 99designs, Inc. 48
WALK AWAY
extremeuncertainty.com

49. © 99designs, Inc.
Until one
day…
—
Ashish
Yep, I’ve also flagged this feature with @Miranda Burford as
something we should critically evaluate as I’m concerned the
benefits have not outweighed the costs.
Susan
Good call @Miranda Burford. And it didn’t go unnoticed that we’re
the only ones who ticked off our To-Do’s.
Miranda
Just to circle back on this, we reviewed Social Sharing and made
the decision to retire it. If you’re interested in how we came to that
decision, you can check out the slide deck.

50. © 99designs, Inc.
DO learn from how
you’re learning
—
8

51. © 99designs, Inc. 51
How’s it going?
by stormyfuego

52. © 99designs, Inc.
2017-05-17
2017-07-05
2017-05-12
2017-04-18
Start End Team
2017-04-29
Growth
Activation
Growth
-2% click through
-11% reduction of click through to
Other design category
50%
2017-07-17 -10% clicks
0% conversion impact
Growth
Acquisition
/categories
Email
forward in
brief
Improved
search on
categories
Spruced up
cross sell
Experiment Page
Conversion
goal
/work
Click to
product
pages
Click
repeat
purchase
Visual brief Increase
email
capture
Minimum /
Aimed
improvement
5%
65%
33%
Observed
improvement
52
At a glance
—

53. © 99designs, Inc. 53
Facing the
music?
—
by jestyr37

54. © 99designs, Inc. 54
Treat it like
an
experiment
—
by dante banuelos

55. © 99designs, Inc.
Bottom
line?
by Mad pepper

56. © 99designs, Inc. 56
Thank you
by Thedaydreamer