Discusses techniques I've used to do Lean Startup style pivots in an agile team and web application. Presented at LAST Conference 2015. We use impact mapping, geckoboard, datahero to show how having a hypothsis backed by data has helped us make decisions to improve bidcontender.com.

1. CONFIDENTIAL | 1

2. CONFIDENTIAL | 2
Featured Project:
The Wroclaw West Bypass, Poland
Trusted by the world’s largest projects
Pivots for real: how we've
hacked up some data to
know we've hit the mark
Mark Smith @markjdsmith

3. CONFIDENTIAL | 3
Obligatory Who am I

4. CONFIDENTIAL | 4
BidContender
• Online Tender Management platform that has connected builders and subbies
on $48 billion in projects this year

5. CONFIDENTIAL | 5
Stories -> Hypothesis
120 Points =
done
http://www.agileproductdesign.com/

6. CONFIDENTIAL | 6
How to know what’s of value?
• Experiments, validated learning, testing things
Read it, loved it, but… when it
came to it. How?

7. CONFIDENTIAL | 7
Torres format hypothesis
http://www.producttalk.org/2014/11/the-5-components-of-a-good-hypothesis/

8. CONFIDENTIAL | 8
Hypothesis Wall

9. CONFIDENTIAL | 9

10. CONFIDENTIAL | 10

11. CONFIDENTIAL | 11
Data Measures Outcomes

12. CONFIDENTIAL | 12
Share your data using online apps, not IT, and not Excel
Geckoboard
DataHero

13. CONFIDENTIAL | 13
Then build until the number moves…
4 Weeks

14. CONFIDENTIAL | 14
or doesn’t.
2 weeks

15. CONFIDENTIAL | 15
Data can be sourced from anything…
• Don’t feel you need to build amazing infrastructure to get it

16. CONFIDENTIAL | 16
Fake it `til you make it.

17. CONFIDENTIAL | 17
TV’s are essential

18. CONFIDENTIAL | 18
The motivation is palpable

19. CONFIDENTIAL | 19
The journey from spec to outcome

20. CONFIDENTIAL | 20
An example
Hypothesis
2 X Transactions
FeaturesOutcomes
$
Say, 20/ month 40/ month &
growing
Many
iterations of
price,
duration
offer, app,
marcomms

21. CONFIDENTIAL | 21
Waterfront City, Australia | US $788M Value
Goals drive hypotheses

22. CONFIDENTIAL | 22
Goal Hypothesis* Measure
* Many of these,
in a hierarchy

23. CONFIDENTIAL | 23
Discover the drivers for your goals (Impact Mapping)
The Goal
Who needs to
change
Ways we can
drive change
Down here –
you’re just
guessing
(hypothesis!)
http://www.impactmapping.org/

24. CONFIDENTIAL | 24
The ‘guesses’ down the tree can be stated as
hypothesis
.
We believe that if we
implement semi
automated invites we
will see the number of
invited friends increase
x% within 1 month

25. CONFIDENTIAL | 25
Goals are the outcome at the end of a funnel
Step 1
Step 2
Step
3
% that get to
this step is the
goal
100 users
50 users
10 users
(10%)

26. CONFIDENTIAL | 26
Funnel maps steps through a product workflow
Step 1
Step 2
Step 3
Step 1
Step 2
Step
3

27. CONFIDENTIAL | 27
We took a flow from our app and created
hypothesis for change
STEPS
Registered
Profile Updated
Subscribed
Subbie connected
to builder
Labelled by
builder
Invited to quote
quoted

28. CONFIDENTIAL | 28
Hypothesis for changes to a step
STEPS
Registered
Profile Updated
Subscribed
Subbie connected
to builder
Labelled by
builder
Invited to quote
quoted
Adding a list of ‘builders in
their state’ will increase
connections to builders for
subbies by 50% within 1
month of introduction
Hypothesis

29. CONFIDENTIAL | 29
Hypothesis for changes to a step
STEPS
Registered
Profile Updated
Subscribed
Subbie connected
to builder
Labelled by
builder
Invited to quote
quoted
Adding a list of ‘builders in
their state’ will increase
connections to builders for
subbies by 50% within 1
month of introduction
Hypothesis
Search filter for builders in
Network directory
Send URL to builders list in
email
Static list (A-Z) of builders
Options to solve

30. CONFIDENTIAL | 30
Inference Tree

31. CONFIDENTIAL | 31
Inference Tree

32. CONFIDENTIAL | 32
Increasing Activation to 90% will
increase retention by 20% for pro
subscribers within a quarter
Ensuring Subbies are being added to
builders networks will increase
invitations to quote by 30% for subbies
within 1 month of introduction
Adding a list of builders in their
state will increase connections to
builders for subbies by 50% within
1 month of introduction
If this holds
Then this
may hold
And this will
hold
(invitations create activation)

33. CONFIDENTIAL | 33
And then build it…
Adding a list of builders in their
state will increase connections to
builders for subbies by 50% within
1 month of introduction
As a subbie I want to
see a list of
contractors in my
state so that I can
connect to them
As a builder I want an
indication that subbies are
trying to connect with me
so that I will go and
complete the connection
As a product manager I
want to know how many
subbies use the ‘find
contractor feature’ so I can
measure the outcome
Etc…

34. CONFIDENTIAL | 34
There’s the measure:

35. CONFIDENTIAL | 35
Indira Gandhi International Airport, Terminal 3, India | US $2B Value
Data informs hypotheses.
Passing Hypotheses deliver
your goals.
Failing Hypotheses don’t.
Find out which is which!

36. CONFIDENTIAL | 36
Further Reading
http://www.businessmodelgeneration.com/book
http://practicetrumpstheory.com/http://www.amazon.com/Value-Proposition-Design-Customers-Strategyzer/dp/1118968050
http://shop.oreilly.com/product/0636920033851.do

37. CONFIDENTIAL | 37
The Center to Grand Forks (CGF) Project, U.S.A. | US $350M Value
We’re Hiring!
http://Aconex.com/careers

38. CONFIDENTIAL | 38
Seattle Tunnel State Route 99, U.S.A | US $1.4B Value
Thank You

39. CONFIDENTIAL | 39
Preparation (for 2nd Session – if you’re coming along!)
• Bring your laptop (or use one of the computers in the lab)
• Ensure you know your Google Analytics credentials.
• Have a google apps account so you can use google sheets
• Bring a list of 5 epics from your current backlog

40. CONFIDENTIAL | 40

41. CONFIDENTIAL | 41
Dubai International Airport, U.A.E | US $4.5B Value
Session 2: Using
Geckoboard and
DataHero to measure
your hypothesis

42. CONFIDENTIAL | 42
Yas Island, U.A.E. | US $40B Value
How do I turn my backlog
of stories into
hypotheses?

43. CONFIDENTIAL | 43
Some techniques
• Extract the assumptions/intention from the story
• As a <who>
• I want to <the change>
• So that <the measure>
As a player
I want to have
semi automated
invites sent
So that my list of
friends increases

44. CONFIDENTIAL | 44
But Why?
• The 5 why’s exercise on a stories ass:
– Q: Why do I want to get semi automated invites?
– A: so I increase the list of my friends
– Q: why do you want to increase the list of my friends?
– A: so that we get more players
– Q: Why do I want more players?
– A: so that we get to 1M players
– Q: why 1M players
– A: Cos then I get a Ferrari….
As a player
I want to have
semi automated
invites sent
So that my list of
friends increases

45. CONFIDENTIAL | 45
Check the (HIPPO’s) bullshit
Bullshit! Will semi
automated REALLY
drive more friends?
• The 5 why’s exercise on a stories ass:
– Q: Why do I want to get semi automated invites?
– A: so I increase the list of my friends
– Q: why do you want to increase the list of my friends?
– A: so that we get more players
– Q: Why do I want more players?
– A: so that we get to 1M players
– Q: why 1M players
– A: Cos then I get a Ferrari….
Bullshit! Will more
friends in the list really
get more players?
The implied hierarchy of the 5
Why’s and the bullshit detector
drives a natural inference tree

46. CONFIDENTIAL | 46
Let’s try it…. Volunteer?
• Let’s take someone’s backlog and have a go at converting it to an inference
tree:
– Volunteer – bring out your top 5 cards
– Let’s do it together.

47. CONFIDENTIAL | 47
Measures are easy in geckoboard
• Pick any piece of data you need for your inference tree
• Sign in to Geckoboard.
• Create a dashboard
• Put the data there.

48. CONFIDENTIAL | 48
Dubai International Airport, U.A.E | US $4.5B Value
Demo: Geckoboard

49. CONFIDENTIAL | 49
Dubai International Airport, U.A.E | US $4.5B Value
Demo: DataHero

50. CONFIDENTIAL | 50
Dubai International Airport, U.A.E | US $4.5B Value
OK – your turn

51. CONFIDENTIAL | 51
Get going!
Your turn – in groups of 4:
– Take your backlog, and find the real goal
– Find the assumptions
– Write down the hypothesis as a tree of inference
– Work out the data you need
– Connect it to your geckoboard

52. CONFIDENTIAL | 52
Objections to Dashboards
• Data Objections:
– Don’t have access. (you don’t have permissions to that data)
– Can’t get that yet (integration problem)
– It’s in an email.
• Answer:
– Google Sheets.
• IT objection
– We’re not paying till you do a business case.
– OFFS – buy it on your credit card and then show the improvement, then get the
corporate card on it…..
• Priority Objection
– We don’t have time to build measures, we need to build feature X
• (if nobody uses it, if it doesn’t convert, what’s the point!)
• A good product manager knows to kill features as well as make them.

53. CONFIDENTIAL | 53
Seattle Tunnel State Route 99, U.S.A | US $1.4B Value
Visualisation

54. CONFIDENTIAL | 54
Static Numbers tell us NOTHING!
That’s nice. Are
we better or
worse than last
month/ pre
change?

55. CONFIDENTIAL | 55
Stacked time series is better
Oh, up and to
the right….
But why did it
slow?

56. CONFIDENTIAL | 56
Sparklines show some history – puts the number
in context!

57. CONFIDENTIAL | 57
Set a goal – so motivating
This goal is
probably too far!
Just to 5 would
be great right
now!

58. CONFIDENTIAL | 58
The % change – when you want to see increments

59. CONFIDENTIAL | 59
Seattle Tunnel State Route 99, U.S.A | US $1.4B Value
Never forget you are
telling a story

60. CONFIDENTIAL | 60
Pro Activation Funnel has improved from our work….
• We’ve moved from 25% (28-May-2015) to 41% (3-Jun-2015) to 47% (17-Jun-
2015)
• We did these changes:
– Introduced auto-labelling
– Pending network connections show as icon on network directory
28-may-2015 3-Jun-2015 17-Jun-2015

61. CONFIDENTIAL | 61
And look at starter!
• This has been sitting
on 7% for some
weeks (haven’t
captured the
progression).
• Look at the numbers
on it now!

62. CONFIDENTIAL | 62
Still a problem, but not enough to finish?
• Drop off on connection
requests
– so we’re doing
reminder email tonight.)
• Drop off on invite to
quoted
– Will cover this as part
of builder onboarding
(they want to get
coverage).

63. CONFIDENTIAL | 63
Seattle Tunnel State Route 99, U.S.A | US $1.4B Value
Thanks – questions?

64. CONFIDENTIAL | 64
The Center to Grand Forks (CGF) Project, U.S.A. | US $350M Value
We’re Hiring!
http://Aconex.com/careers