Testing Roadmaps and Experiment Design

TEST ROADMAPS
How to win sales and influence success

Hi I am Tim Stewart from trsdigital
You might know starring roles such as:
Commercial Manager for the largest English language motorcycle
site in the world
Consultant with the highest success rate, longest emails AND
undefeated go-kart champion during Maxymiser’s highest growth
years
Web Optimization Consultant for SiteSpect with client’s such as
well, er Connor you just met, Confused.com, Hotels.com and Guess
Jeans EU
Some may simply know me as that bloke who talked for 5 hours
about Excel on the #MeasureFest training course yesterday
The internet mostly knows me as @pjeedai on most social networks
The p stands for Padawan as I am always learning
INTRODUCTION
@pjeedai

Effective testing starts with the boarding process
Vendor or Agency bringing on a client, an Internal
team rolling out a new initiative or flying solo
Its not as simple as adding the tool and taking off
You can’t just go up the ramp, turn left to First
Class and wait for the complementary peanuts
BOARDING

The energy when people talk about CRO can be
breathtaking
It may feel like Boarding is simply a question of
hanging on and letting the wave push you
There’s nothing more to do except enjoy the ride.
HIGH ENERGY BOARDING

But some stakeholders don’t have as much energy
as others
You need to understand the level of buy-in and
competence of all the stakeholders
It might mean your plans to roll out your testing
program have to be adapted so they don’t
overwhelm some parties.
LOWER ENERGY BOARDING

And some are just along for the ride
MINIMAL ENGAGEMENT BOARDING

BOARDING
For initial boarding you should establish:
• Who needs to be involved in the process
• What responsibilities each person has
• How you will roll out your testing plan
Otherwise your life-changing trip might end with a bang
… and even worse a Celine Dion song

DISCOVERY
Arguably the most important part of any roadmap process
You find out what you are working with
And what you need to work around.

DISCOVERY: UNDERSTAND YOUR LAUNCH CONDITION
Try understand the inertia you have to overcome
What is currently weighing your business down?
• How much weight do you need to shift?
What is the shape of the business?
• Where is money being made?
• Which objectives have highest importance
• What outcomes have the most volumes?
• Which generate the most revenue?
Can you quantify this?
• How do web metrics report these values?
• How closely do web metrics relate to outcomes?

DISCOVERY: NOT JUST NUMBERS, ALSO RESOURCE
Before looking at any website issues
Be clear on the following:
• What the business needs for take-off
• What needs to be handled to achieve this
Who looks after this?
Who needs to be involved?
What is required in terms of:
• Development resources
• Design resources
• Sign-offs

DISCOVERY: TRAJECTORY IS IMPORTANT
Testing is often pitched as a way to
• Bypass IT
• Fast track past release schedules.
It can be used for this. But care is needed.
Marketers don’t want to hear it, but:
• Checks
• Balances
• Sign-Off
• Pre-flight checks
are important!
Too much push
Too little direction
Will mean a short, sharp trip
A spectacular end to your plans

Binary choices:
• Leave a page or move forward
• Add or Not Add
• Checkout Now or Continue Shopping
• Click to Read More
• Sign up for our mailing list
Where do the best opportunities lie?
• What makes up the total?
• Where are the biggest holes?
• Where is the most money spent on traffic?
Where is the signal loudest?
• Like - Will continue
• Don’t Like - Will go no further?
Are there channel or device differences?
DATA

Data about how those choices happen
• Click patterns
• Heat maps
• User Journey
• Detailed step conversion tracking
Data about users and their motivation
• Never will Buys
• Might Buys
• Could be Persuadeds
• Plan to Buy
• Will Buy no matter what
Do you have customer feedback?
• User testing
• Customer survey
• Live Chat sessions
• Personas
SERIOUS DATA

Quantitative and qualitative
• Cohort and behaviour profiles
• NPS
• Customer Buying Patterns
• Typical Basket composition
• CRM and userID
• BI: Margin per product, AOV, LTV, RPU
Can you stitch it together?
• Get an idea of themes
• Areas to explore
• Key templates
• Decision points?
• Customer Persona
• Non-Customer Persona
It is a term that is often misused
Its not Big Data in terms of machine learning
REALLY SERIOUS DATA

But it is a hopefully large collection of data sets and useful intelligence
And importantly it is not a fixed set of Data
It will be added to over time
• Through tests results
• Through additional research
• Through improved analytics
Once your testing shows more detail around what drivers user choices you
SHOULD be tweaking your measurement on site to get a clearer picture
And obviously this new data should be fed back into the process and used
to create and refine testing ideas and roadmap areas.
NOT REALLY “BIG DATA” (BUT IT MIGHT BE LOTS)
(This is actually Lore)

It can be overwhelming
But this is why Boarding and Discovery play such an important role.
You aren’t on your own in this.
You brought in the Business Intelligence team, the Department stakeholders, the Board, the HiPPOs
When faced with a mountain of Data to look at it’s generally best to look for what makes the biggest
difference to the company
And look at the customer and analytics data to see what makes the biggest difference for the users.
LOTS OF DATA CAN OVERWHELMING

So if your stakeholders and customers have given you some clues… Start there.
Be methodical. Drill down and work outwards to eventually the periphery of “other interesting stuff”
when you come to focus on an area.
Initially at least you are looking for a few places to start and some holes which need investigating
INVESTIGATE YOUR DATA TO FIND SOME CLUES

At this point you haven’t looked in any depth at any specific pages or processes
You’ve looked at the business, the people in the business, the audience you can work with.
IDEAS
So you’ve established

Who you need involved from the business
What level they need to be involved
The best way to communicate the strategy and the results
IDEAS: WHO IS BEING INCLUDED?
Stakeholders

What they need for the business
What they consider to be the critical results to meet the business strategy
Do they have a timeline in mind?
Is it reasonable?
Fundamentally what you need to achieve from testing to ensure its continued support and growth.
HiPPOs are not stupid. They are looking out for the best for the business.
If you can show value and align your testing program with what the business needs to grow and
succeed then HiPPOs will back you
Which is how they became the Highest Paid People.
And delivering a testing roadmap that delivers success for the business will make you more Highly
Paid too
IDEAS: THE HIPPO’S IDEAS DO MATTER
The HiPPO

The Audience is your sample. It is what you have to test against
It is their behaviour that you want to influence.
• How much you can test?
• How it can be segmented?
What is the composition?
New, Returning, Regular Purchasers, Irregular, Distress Purchase, Subscription.
They all have different motivations and will react differently
What is the shape?
Frequency, Recency, Device preference
Are users doing the same thing on every visit?
Do different devices get used for different purposes?
Are there obvious places where mobile outperforms or underperforms Desktop? WHY?
IDEAS: WHO AND HOW MANY TO TEST AGAINST
Audience

How many of these have audience?
Sometimes area of great internal debate have much lower engagement or user volumes to test
against.
Sometimes you want to measure increase to Final Sale
But there simply aren’t enough volumes
Or the Final Sale is 4 visits, 2 devices and 18 pages after the area you want to test.
So is Final Sale the most closely related metric to what you are changing?
You will, eventually, influence it. You may want to track it every test for a benchmark
But unless you are testing the page directly before and have large volumes of commonly distributed
purchases every day.. It will limit your testing ability
It should not be your sole focus or your primary decision metric on every test.
If you focus on just the cash register but don’t stock the shelves, promote the offers, price match
competitors, see how users navigate the aisles…
Your Cash Register is going to remain too quiet and will provide you very little intelligence on why.
IDEAS: WHERE AND WHAT NEEDS FOCUS
Places

The potential for Win in those places and with the Audience identified means you can start to think
on priorities
You also start to understand the size of the challenge you have to convince the Audience to change
their behaviour
You will NOT go from 0.3% conversion rate to 3% easily
An idea of the value? It won’t be perfect but you need an idea of the value, in revenue, happy
customers, business objectives (acquisition, AoV, RPU etc)
Possibly from the data alone you will already have a clear idea on some of the questions you need to
answer and options you can test
Which of these are traffic wins? (more people into a key area)
Which of these are business wins? (more sales, greater value, the “right” product being
seen/chosen)
Do these align or are they always separate tracks?
IDEAS: WHICH ARE BIG ENOUGH TO MATTER
Scale of Opportunity & Challenges

What are the targets you have?
What budgets? What current actuals?
Want != Need
I want more sales now, for less spend
Translates to
I need to find a way to increase conversion from the same audience
I need to identify where this is being impacted and reduce this
This NEEDS time, experimentation and commitment.
If one test gave all the answers, halved the CPA .. Then there wouldn’t be a need for a Roadmap.
IDEAS: WHAT TARGETS NEED TO BE MET
What the Business wants
Hopefully what the business actually needs (these two rarely align)

Look at the pages with fresh eyes.
Think – What do WE want here?
Think – What does the CUSTOMER want here?
Look – What does the Customer ACTUALLY do?
Why is this different to what we want/intend?
(it will be, customers are usually a LOT less keen to give you all their money immediately)
IDEAS: THINK LIKE A CUSTOMER
What the Customer DOESN’T want

Go to the places your data has shown need attention
Go to the places the business and stakeholders want to improve
TRY to break the user journey.
Screen grab every error, every misstep, every button/link/option that is NOT move forward
Look for friction, miscommunication, ways where Where Next is not clearly answered
Blur your eyes, see if you can still see a CTA
Try different screen sizes, is that important information above fold (spoiler: it still matters, even if you
can encourage people to scroll)
The UX book is called Don’t Make Me Think. Do you? Do you make the customer have to *shudder*
think? And Read?
Can they do it in < 3 seconds from landing?
Then think about how you can change that, some alternative ways to handle those choice points
IDEAS: GET CREATIVE

And you will end up with ideas, lots and lots of ideas.
It’s probably a good idea to write them down somewhere.
LOTS AND LOTS AND LOTS OF IDEAS

You’ve sat down
• Got the business objective(s)
• Got the web stats which relate to that objective
• Got a budget
• Got a timeline
• Got lots and lots and lots of ideas
Just a question of pointing it in the right direction and heading to Success-ville?
A ROADMAP TO SUCCESS

If that is the conclusion of your boarding and discovery – Get Ideas, Test Ideas, Epic Win!
I have news for you.
You Shipment of Fail has arrived.
Because you may have created the biggest shopping list of issues ever and “testing will fix that”
But its at this point a lot of businesses choke.
And I bet at this stage some of your boarders have run out of energy and you won’t have the traffic
or resource to test everything.
EXPECTATION VS. DELIVERY

Too many, dare I say it, Ideas
Because we can’t test Ideas. Ideas are formed into hypotheses. Hypotheses are
formed into tests.
Because the road is never that smooth. The choke points of resource and traffic are
always too small.
If your boarding and discovery showed that you have everything you need, no
impedance, no need for alternatives or expectation that you will deviate
You are going to have a bad time
EXPECTATION VS. DELIVERY

Set a destination that looks more achievable for your circumstances
Plan a more realistic way to get there with the resources you have
PROCESS TO RELIABLY REACH SUCCESSA ROADMAP TO SUCCESS?

Now we all work in digital. This isn’t anything new, or at least it shouldn’t be.
The Board & hungry HiPPOs may think this is what they will get – a direct line to Success
The reality is that you won’t fly first class to success even with the best Boarding & Discovery process

The reality of release schedules, sign off delays, legal
& brand policy departments getting involved will be
specific to your situation and will always be evolving.
Your tests won’t always work out the way you expect;
you might need to test again and check results, stop
testing in one area to work in another, account for
changes in market or tactical need etc.
It is very likely there will be congestion on the ideal
route, diversions, unplanned stop-overs & lost luggage
If you only have a single focus plan, only test when
conditions are perfect then you will step your way
there. But slowly, through adverse conditions.
You will back step regularly, roadblocks can & will stop
you completely unless you have something prepared.

PROCESS TO RELIABLY REACH SUCCESS
Now obviously you don’t know, what you don’t know.
You can’t draw a plan to account for every possible
outcome upfront, no matter what Data you have
But you can and should have a few alternative paths to
explore that will also get you to your destination
A successful Roadmap will combine multiple areas of
investigation, several paths forward to your objectives
The idea is to have a Process that keeps you testing
somewhere & keeps momentum in your Roadmap

What we need are tests. What a test needs is a reason to exist; a Hypothesis
You need to have a think, cogitate on your next move. Think a few moves ahead
COGITATE: PLAN YOUR MOVES
So I said earlier that we don’t really want Ideas.
But initially we do need them, we just need to craft them into hypotheses
We need to put them together. Chew them over. Have a think. Make a list

IDEAS WILL COME IN MANY FORMS, SOME EASIER TO COMPILE THAN OTHERS
Ideas will come in from all stakeholders and Persons with Opinions (Highly Paid or not)
They will come in many different formats and need to be consolidated into a big pile of “stuff we
could look at”
And you will welcome these (because understanding who and what was part of the process and
involvement helps you get traction)
COLLECT & COLLATE
An unordered list of business
problems and design wishlists
A set of ideas, grouped into tracks
Some simple time & sequence planning
A fully quantified spreadsheet, with tracks matching business
areas and priorities accounting for release schedules
An email with only basic ideas

These ideas need to be refined into something you can actually
test with:
Hypotheses
Structured Test Concepts not just “make it better and more
punchy”
What parts do you have?
What parts are missing?
CONSOLIDATE: ORGANISE THE PUZZLE PIECES
These ideas are likely to be pieces of the puzzle, parts of a hypothesis, not a complete test
The next phase therefore is to pull these together into a pile of parts you can work with

You won’t get this right on the first pass. Or the second.
Like every part of testing this is an ongoing optimisation. Permanently re-adjusting and tweaking
until you have something that works
But initially at least… good enough will do, even if there are some bits that just don’t look quite
right.
Split Ideas into groups; themes, site sections, areas of business interest
Divide sometimes (often?) conflated and unviable ideas into discrete achievable tests
Match available hours, ability and traffic to achieve the most appropriate mix
CONSOLIDATE: SEE WHAT FITS

Just because you can, doesn’t mean you should
EXPERIMENTAL DESIGN
TIME FOR A QUICK DETOUR INTO EXPERIMENT DESIGN
Seeing as we are talking about turning ideas into tests.
So a quick refresher on Experimental design

THE PROBLEM WITH MOST EXPERIMENT DESIGNS
When given a tool that allows you to change almost anything
And a list of business problems which, if fixed, could make LOTS more money
The typical reaction is to dive in and try to do it all in one big test…
The typical result is #epicfail

Hypothesis Kit 3
Craig Sullivan @OptimiseOrDie
Colin Mcfarland, Lukas Vermeer, Rik Higham, Doug Hall, Michael Aagard and many others who helped shape this
https://medium.com/@optimiseordie/hypothesis-kit-2-eff0446e09fc
HYPOTHESES
Simple Kit
1. Because we saw (data/feedback)
2. We expect that (change) will cause (impact)
3. We’ll measure this using (data metric)
Advanced Kit
1. Because we saw (qual & quant data)
2. We expect that (change) for (population) will cause (impact(s))
3. We expect to see (data metric(s) change) over a period of (x business cycles)
“We don’t like this page” is NOT a valid Hypothesis
There are several ways to frame this but here is one I like for it’s clarity and simplicity

A lot of stakeholders don’t think in terms of hypotheses
They’ve focused on the business problems and who actually has to do this
So a Scope, Brief, Qualifying Criteria for a test may be better explained if you can answer the 5W1H
questions
QUALIFYING CRITERIA
What
• Do we want to achieve?
Where
• Are the changes to be made?
Why
• Based on what data?
• Based on what need?
Who
• Is this aimed at?
• Is responsible for sign off?
When
• Can we do this?
• Is this running?
How
• Is that measured? How will that show in the metrics we have defined
• Much difference could be made?

DECISION METRICS: WHAT DOES SUCCESS LOOK LIKE?
Decision Metrics
• Appropriate to the question you are asking?
• Directly relate to Test Objective?
What are you changing?
• Expected effect
• What should users do more of, more often?
Unexpected effect
• What if they do less?
• Or worse… No Significance?
• What if it is substitution?
Primary Call To Action or Outcome
• Ancillary Outcomes
• Value or Meta Metrics

The most effective metric is the closest Next Action for a user who has seen
your change
• What are you changing?
• Why?
• What should the user do if it works?
• What could a user do if it doesn’t?
Measure that as your main signal
Sometimes users do less of your main metric but substitute another
If you can, measure that too
Sometime the main metric increases but an alternate path to Next Action
decreases
This can mean no NET gain, what you have is substitution not incremental
increase
So measure that too.
You want a clear result
It doesn’t need to be Up, sometimes quantifying the Risk of Down is critical
The only failed test is one where you don’t learn something
UNCLEAR METRIC DEFINITIONS: UNCLEAR RESULTS

• Weighting
• Priority
• Potential
• Importance
• Ease
• Reusability
• Dependencies
• Restrictions
TRIAGE
Many methods to prioritise YOUR tests to suit YOUR situation, limitations and objectives
You know what they are, right?
After all we established this in Boarding and Discovery… didn’t we?

Emotion and tactical need will enter into it
But if your whole roadmap is led by whim and gut feel it is easy to go off course
PRIORITISATION
Many people will think Priority is decided by which HiPPO is the most loudy shouty
Whilst this can be accurate, it is not the most reliable way to plan an effective test roadmap

PRIORITISATION
It is the HiPPOs job to focus on
• The end result
• Big difference for the whole company
• Technically difficult site-wide changes
They will want BIG RESULTS quickly
They need to quantify risk or reward
But if you wait for only those tests…
Your roadmap will stall
Test for intelligence on what wins
You can’t test to only get wins
Explain, carefully, sometimes slowly:
• Some tests are small
• Some are far away

• Keep a central record of what is agreed
• Keep a revision history of that over time
Check it from time to time to remind yourself how you have had to re-plan
Remind yourself and stakeholders that changing plans is expected and required
PRIORITISATION
This is where a framework and Discovery are very useful. Don’t get into emotional debate
Quantify what happens when, why and what is needed to make later tests happen sooner.

CRO IS UNDERSOLD. IT CAN DO MORE THAN “JUST” GET WINS
But the ease with which consistent victories are achieved is massively oversold
Technology will not solve all the issues
What makes most difference is the people and the teamwork
SUPERHEROES OF CRO
We like to portray CRO as heroes of the business, fixing bad UX and saving the world
But, a confession.
In reality it is not the company-saving Alpha-male magic beans we were sold

DIFFERENT BREED OF HERO
In fact it needs a different kind of hero.
One that is much more … office-based & organised
And wears a lot less lycra

Working through what is needed, not just what looks good requires a methodical approach
You can run around smashing heads and breaking hearts but it is easy to make more noise than value
The biggest gains are made from long-term cultural shift in behaviour
It doesn’t mean you have to go slow.
With a good repeatable process you can go faster
Because you already have a list of targets that add up to a clear goal
You waste less time than looking around for only the headline grabbing big battles after you recover from the last
DIFFERENT BREED OF HERO
That’s not to say that this different breed of heroes can’t kick ass
With the right approach and the right tools they are very much in the world-saving business
They just do so with a little more of a measured approach and less collateral damage.

PROJECT MANAGEMENT
In fact we aren’t talking about a revolutionary approach
When it comes to Roadmapping for success an essential tool is… Project Management
I recommend and use www.liquidplanner.com but there are many options out there

Excel, Smartsheet, Whiteboard, Piece of Paper. Ultimately its what fits and what you can use.
But something you can share with the team and easily update is recommended
PROJECT MANAGEMENT
A good basic (and cheap) option is www.smartsheet.com which offers several templates
ProTip: You can also link sheets so the Projects in the Gantt links to items from your Priority/Ideas sheet
It also has good Google integration and allows creation of Forms for adding new items from Stakeholders

Task lists don’t give you an oversight of how all tests fit together
Basic lists of Projects won’t show you what Task is needed next from each person
Simple Gantts or Card Views don’t let you move Tasks independently of Projects
It is entirely likely (and recommended) to be in planning phase of Test 4 and design phase of Test 3 when Test 1 is live and Test 2 is
in Final QA
PROJECT MANAGEMENT
Task based tools like Asana or Basecamp can also work, as can card style tools like Trello.
These type of tools are simple to use but lack oversight on scheduling resource over multiple tests
(The image is actually Card view from Liquid Planner – simple view, underpinned by a proper scheduling system)

PROJECT MANAGEMENT
Liquid Planner works well for CRO because it is flexible & any plan needs to be flexible
• Several ways to look at, reprioritise and reschedule multiple overlapping projects
• Move tasks up the priority list & work on them out of sequence when you need to work ahead
• Every time you update priority it re-calculates the likely delivery for each task & project

COMMON SENSE: SO RARE IT’S A SUPERPOWER
Prioritise and plan for delivery of Tests, not just ideas.
How MUCH you do depends on your setup & situation:
Resource Capacity
• Traffic and sample size for metrics
• Developer availability
• Analyst & Planning availability
• Design availability
Resource Capability
• Technical ability within the team
• Technical function of the Testing tool
• Experiment planning ability
• Analytics and Statistical ability
Want to go fast and run small tests. Plan for that
Want to plan more and test for impact? Do that
Want BIG TESTS and quickly. Be big. Pay big (still fail lots)
Ultimately use Common Sense.
It’s still so rare it’s a super power

Multiple Paths
Varying Timelines
Changing Complexity
Plans will change
Embrace that.
Simply planning Tests will breed new test ideas
Results will create more need to explore in some areas than originally expected.
Results will kill whole planned sets of tests if they are shown to have less value than anticipated.
EVOLVE
Test 1
Mobile 1
Mobile 1a Mobile 2
Mobile 3
Mobile LP
Landing
Pages
LP 1
LP 2
LP 3
LP 4
LP 2a
LP 2b
Checkout
Checkout 1
Checkout 1a
Checkout 1b
Checkout 2
Checkout 3
Checkout 4
Checkout 5
Test 1a

Nothing is set in Stone
• Revise those plans and documents after each update
• Document each revision, track what takes time
• The aim is to improve the process as well as the site
It takes an investment – a commitment on time
You can’t just cut out fat to make efficiencies and optimize
You have to measure and invest where your system is lacking
Anyone says otherwise is telling Porkies
EVOLVE

For every test on the Working List
Create Test Documentation
• Document Your Hypotheses
• Confirm decision metrics
• Record what you intended to happen
• Scope how you want that to be built
• Specify URLs to include/exclude
• Specify Segments to include/exclude
If you came back to this in a year could you explain to a
new starter what was intended? Memory is fallible.
If you had to come back to your developer in a week and
explain why it failed QA.. Is it in here?
If you’re “just trying something” without data, objectives,
a Hypothesis and Metrics…It’s not a test.
You can build it and run it.
But it’s not a test
EXECUTION: DOCUMENT EVERY TEST
Buy Today
Buy Today
Buy Today
click
Why is it bigger?
How much bigger?
What should users
do more of?
Is this on all pages?
All devices?
Image or CSS?
What does this do to
nearby elements?

Wireframes
• Blueprint your changes
• Show what it should look like
Take a closer look at the page
• Look at what else could happen
• Is the change clear?
• Could something else be clicked on?
• Are you measuring that too?
This is your blueprint, the concept to get approved
before you spend time and money on design.
This is the idea your stakeholders must buy into, your
designer must understand, your developer must build.
Have you shown all the states the page can be in
for the test? Does it work in all cases?
EXECUTION: BLUEPRINT THE CHANGES
clickclick
click
clickclick
click
clickclick
click

Issue Resolution
• Plan for Use cases
• Try to break the page
• Try it logged in, not logged in
• Checkout with nothing in basket
• Submit forms with fields empty
Build and try to find omissions
• QA
• QA again
Hold a pre-mortem
Blame no-one, open mind and think:
• If in n weeks you are reporting a failure
• Why did it fail? What broke? What is the cause of death you have to pronounce?
• Could it have been avoided?
Produce a conclusion report for EVERY test. What worked, what didn’t
EXECUTION: PRE AND POST MORTEM EVERY TEST

Test Results should change your plans
• Regular reviews
• Is this still in line with strategy?
• Could we go faster? Should we go slower?
• Test Results
• Did the result show further potential?
• Did the result show greater risk?
• Test non-results
• Why is it not clear?
• Did the user not care enough?
• Did one segment “outvote” another?
Don’t underestimate Tactical Issues
• Releases
• Team Changes
• Market conditions
• Seasonality
ADAPT
Review
Results
Non-
Results
Releases
Team
Changes
Tactical
issues

CONCLUSION
Bet you thought this would never end, nearly time for the coffee break
We’ll finish with a quick recap

BUILD FROM GOOD BOARDING
Good Boarding process No Boarding process
GOOD BOARDING = UNDERSTAND THE PEOPLE

DISCOVERY PHASE ENSURES SUCCESSFUL TAKE-OFF
DISCOVERY = UNDERSTAND THE ENVIRONMENT AND EQUIPMENT

BE SERIOUS ABOUT YOUR DATA
DATA DRIVEN DECISIONS,
OTHERWISE IT’S GUESSWORK

All too easy to think data and a plan has to be perfect to start
All you need is Good Enough
The process is to refine all of this as you go along
TOO MUCH CAN BE INTOXICATING
DECIDE – DON’T JUST COLLECT DATA AND TAKE NO ACTION

BUT GET IT RIGHT, AND YOU’RE LAUGHING
STRIKE A PRAGMATIC BALANCE THAT SUITS YOUR SITUATION

IDEAS FROM BUSINESS, USERS AND DATA
BE CREATIVE
TRY TO BREAK YOUR USER JOURNEYS
IT IS NEVER THE FLOW YOU INTENDED

BALANCE YOUR PRIORITIES
BALANCE TO SUIT YOUR CAPACITY

QUALIFY WHAT MAKES AN IDEA A TEST CANDIDATE
MAKE EVERY IDEA EARN ITS PLACE ON THE LIST

QUANTIFY YOUR PRIORITIES TO THE HIPPOS
Calmly explain; facts and data back up strong arguments
Anger, fear, aggression lead to the dark side
(turning off the lights has a similar effect)
OPEN COMMUNICATION WITH STAKEHOLDERS
USE DATA TO BACK UP YOUR POSITION

EXPLAIN HIGH PRIORITY DOESN’T MEAN “NEXT TEST”
BY DEFINITION EVERYTHING IS ASAP
Because Soon and Possible are subjective variables. Use Common Sense

BE A NEW BREED OF CREDIBLE HERO
DON’T GET CAUGHT UP IN YOUR OWN LEGEND
Be “Get things done” rather than “Look at me”

PROJECT MANAGEMENT IS NOT A DIRTY WORD
A TESTING PLAN REQUIRES PLANNING
Don’t forget Common Sense
Common Sense and pragmatism go a long way

RECORD ALL PLANS (BUT NOTHING IS FROZEN)
WRITE IT DOWN, BUT BE PREPARED AND HAPPY TO CHANGE IT

ADAPT: LEARN FROM BAD RESULTS
BAD RESULTS ARE STILL RESULTS, LEARN FROM THEM

EVOLVE: TEST FOR INTELLIGENCE NOT JUST RESULTS
BE AGILE: SEE AN UNEXPECTED OPPORTUNITY THEN EXPLOIT NEW AREAS

EFFECTIVE ROADMAPS
@pjeedai
www.trsdigital.com

Testing Roadmaps and Experiment Design

More Related Content

What's hot

Similar to Testing Roadmaps and Experiment Design

Recently uploaded

Testing Roadmaps and Experiment Design

Editor's Notes