JEREMY YOCHES, SENIOR PRODUCT MANAGER, STELLA & DOT
MARK HALVORSON, HEAD OF INTERACTIVE MARKETING, ATLASSIAN
CHARLOTTE WEISKITTEL, SENIOR DIRECTOR, DONORSCHOOSE.ORG
Trying to get more traction for testing at your company? Wondering how to build and scale your team most effectively? Looking for your next big testing win? In this session, we hear directly from testers who have tackled the same challenges.
This session brings together Optimizely users from Atlassian, Stella & Dot, and DonorsChoose.org for a behind--the--scenes look at testing at their organizations. Coming from a variety of industries and company sizes, each speaker presents a unique perspective on how they've built and scaled testing programs successfully.
You'll walk away armed with interesting testing ideas and actionable steps you can take to build a data-driven testing culture at any organization.
1. Mark
Halvorson
Head
of
Interac2ve
Marke2ng,
Atlassian
@halv0012
Charlo=e
Weiski=el
Sr
Director,
Product
DonorsChoose.org
@noDornaught
Jeremy
Yoches
Sr
Product
Manager,
Stella
&
Dot
Customer
Showcase
7. Remove
$500
and
$1000
• Removing
$500
and
$1000
increased
conversion
by
58%.
8. Photo
and
tagline
(mul0variate
test)
• Female
teacher
performed
best
with
3
of
4
taglines
• “Join
3,500
people…”
did
slightly
be=er
than
the
original,
but
it
wasn’t
significant.
1. A
teacher
spends
$40
each
month
on
classroom
essen2als.
(original)
2. Make
giving
a
monthly
habit.
3. Join
the
3,500
people
who
support
a
classroom
every
month.
4. Help
a
classroom
every
month.
9. Charge
your
card
now
or
in
a
month?
• Made
the
change
to
wait
a
month
before
we
had
Op2mizely
• Removing
men2on
of
the
next
month
increased
conversion
+217%.
10. $50,
$25
or
no
incen0ve
• Conversion
rate
– $50
incen2ve:
1.8%
– No
incen2ve:
1.6%
– $25
incen2ve:
1.3%
• The
no
incen0ve
group
was
the
least
likely
to
cancel
a`er
two
months
(8%)
• The
no
incen2ve
group
had
the
highest
average
monthly
signup
amount
of
$26.
11. Tips
from
a
lean
non-‐profit
• Just
start
tes2ng
• Clear
the
path
for
the
tester
• Be
a
broken
record
• Look
at
the
results
holis2cally,
not
just
A
vs.
B
• Don’t
overfill
tes2ng
road
map
13. About
Atlassian
We
help
great
teams
build
be=er
so`ware,
together.
• Products
include:
JIRA,
Confluence,
Bitbucket,
HipChat…
• Founded
in
2002
• Over
35,000
Customers
• Over
700
employees
globally
• No
sales
people
14. A/B
Tes0ng
at
Atlassian
is
cross
func0onal
Central
Growth
Hacking
Team
to
support:
• Email
• Product
Management
• Customer
PlaDorm
• Customer
Advocates
• Internal
Systems
• Leads
• Product
Marke2ng
• Support
• Bitbucket
• HipChat
21. Lather,
rinse,
repeat
Our
Process
• Collect
great
ideas
• Triage
based
on
poten2al
impact
• Build-‐out
good
candidates
• Schedule
to
avoid
conflicts
• Run
experiments
&
analyze
results.
24. Our
Test
Philosophy
• Ideas
are
everywhere
• Everything
affec2ng
conversion
is
tested
• No
traffic
wasted
• Be
aggressive
• Regular
cadence
• Never
run
a
test
that
you
wouldn’t
want
to
win
25. Types
of
tests
• Pebbles
– Change
in
copy
or
CTA
presenta2on
• Rocks
– New
page
(e.g.
no
Dev
Tools
family
page)
• Boulders
– Simplified
order
form.
32. About
Stella
&
Dot
•
Stella
&
Dot
helps
women
become
successful
entrepreneurs
with
a
thriving
business
selling
bou2que-‐style
jewelry
and
bags
through
in-‐
home
trunk
shows,
online
and
person-‐to-‐person.
• Today
there
are
over
17,000
Independent
Stylists
who
sell
the
accessories
in
the
US,
Canada,
Puerto
Rico,
UK,
France
and
Germany.
• In
2009,
retail
sales
were
$33
million.
In
2010,
$103
million.
In
2011,
over
$175
million.
In
2012,
sales
hit
$200
million.
And
in
2013
sales
were
$220
million.
33. The
Stella
&
Dot
Tes0ng
Team
Jeremy
Yoches
–
Full
0me
product
manager,
and
also
full
0me
conversion
op0mizer
• Own
the
test
plan/roadmap
• Develop
and
set
up
tests
• Analyze,
compile,
share
and
evangelize
results
• Collaborate
with
teams
as
needed
– Crea2ve
to
create
new
assets
– Dev
help
for
more
complicated
javascript/CSS
34. The
Stella
&
Dot
Collabora0on
Model
Share
the
plan,
the
status,
and
the
history
Make
it
easy
for
anyone
in
the
company
to:
• see
what
tests
are
live
where
• inves0gate
live
and
past
tests
in
more
detail
• Propose
new
test
ideas
36. The
Stella
&
Dot
Tes0ng
Success
Story
Many
small
successes
and
important
insights,
A
few
big
wins:
The
Super
Nav
Tests
-‐
Op0mize
our
“triple
funnel”
by
tes0ng
the
primary
CTAs
in
our
header
38. Super
Nav
Test
Results
3
CTA’s
=
3
compe0ng
goals
Increased
clicks
on
all
3
by
more
than
30%
And
we
are
just
gejng
started…
• Color,
size,
shape,
etc…
39. Mark
Halvorson
Head
of
Interac2ve
Marke2ng,
Atlassian
@halv0012
Charlo=e
Weiski=el
Sr
Director,
Product
DonorsChoose.org
@noDornaught
Jeremy
Yoches
Sr
Product
Manager,
Stella
&
Dot
Q&A
and
thank
you