Optimizely Webinar: Accelerate Agile Delivery with Experimentation Platforms, Featuring Forrester

Accelerate Time To Value With
Experimentation Platforms
Christopher Condo, Senior Analyst
February, 2019

We work with business and
technology leaders to develop
customer-obsessed strategies
that drive growth.
3© 2019 FORRESTER. REPRODUCTION PROHIBITED.

Leading organizations make SW development their
core competency!
Digital matters.. Software Makes
Digital..
Excellence in
Developing
SW IS A Must
!!!..

Digital
Transformation
Customer
Experience
Feature
Release
Cycle

Release Velocity Is Stagnant
Base: 546 to 3,228 global developers (base sizes vary by year)
Source: Forrester Analytics Business Technographics® Global Developer Survey, 2014 to 2015 and Forrester Analytics Global Business Technographics Developer
Survey, 2016 to 2018

End to end Agility comes with both
Agile + DevOps
Agile
DevOps
Speed, Quality and CX
Common Language
AGILE
Infrastructure
Automation
Infrastructure: Grip, Security
Scaled
Infrastructure
AGILE
Development

8© 2016 Forrester Research, Inc. Reproduction Prohibited
Yes you want speed…
Delivering Faster is not Enough To Please Customers

9© 2016 Forrester Research, Inc. Reproduction Prohibited
Speed With Quality Is Needed To Deliver True Value !

What Hinders Delivering Value With Velocity?
Replicating production
environments is
complicated
Lack of Insight:
No news is good
news
AD&D and the
business are not in
synch

38%
Of developers
use indirect
measurements
(like defects) to
gauge success
Oops!
Something
went wrong.
Source: Forrester Analytics’ Business Technographics Global Developer Survey, 2018

“Press and Pray”
deployments are a
surefire way to
disappoint customers
and endanger revenue.
• High drop off rates
• Unintuitive designs
• Increased customer
service call volume

So What’s Missing?
›A methodology and process for trying new ideas, and safely testing
them in production
›A way to connect product performance directly with AD&D teams
›A system that is inclusive of business and engineering
›What’s missing is Experimentation!

“We want to make runtime
decisions in the field and
control a feature launch
without having to deploy a
new client-app.”
What Customers Look To Achieve By Experimenting
“We want to use this
methodology to figure out
if improvements are
delivering value.”
“We want to gain a more
objective view that will
allow us to settle internal
disputes.”
“We want more
information to understand
why one version won over
another.”
“Our dev group wanted to
encourage single-trunk
development and get
work-in-progress out of
branches.”
“We wanted to be able to
target features by region.”

Experimentation Platforms for AD&D
Direct User
Testing
A/B Testing Feature
Flags

A/B Multivariate Testing
A/B Multivariate
Testing

Characteristics Of A/B Multivariate Testing
›A/B Multivariate Testing enables in-production testing of different
versions of an application (work flow, algorithms, search engines,
etc.) to determine a winning strategy within hours or days.
›Components:
• A/B testing logic embedded in code
• A system that controls which populations see Experience A vs. Experience B
• A statistical engine to determine the “winning experience”
A/B Multivariate
Testing

A/B Multivariate Testing
Source: “AD&D Pros: Stop Guessing And Start Experimenting”
A/B Multivariate
Testing

Experimenting with different patterns is simpliﬁed and
becomes an organic part of development. Statistical engines
can immediately inform devs which strategy is performing
better.
Switching logic needs to be maintained. After experiments
conclude that switching logic (along with the “old” code)
needs to be removed to prevent unused code from
continuing to be deployed.
A/B Multivariate Testing Pros and Risks
A/B Multivariate
Testing

Feature Flagging
Feature Flagging

Characteristics Of Feature Flagging
›Feature Flagging allows application code to be released in “off”
mode and managed with a dashboard enabling single-trunk
development that enables stabile releases.
›Components:
• If-Then-Else logic embedded in app code
• App or service that communicates value of the feature flag
• Administrative console for managing flags
• Ability to configure which user populations see each feature when switched
“on”
Feature Flagging

Feature Flagging
Users Feature Flag Code
Feature Flagging

Simpliﬁes release, enables continuous delivery, enables
business teams to better coordinate release dates with
AD&D, and enables user demographic splits.
Flags need to be maintained. Forward thinking needs to
happen (when and where to place these ﬂags) to get the
most value, and re-engineering is needed to place them into
legacy code.
Feature Flagging Pros and Risks
Feature Flagging

Direct User Testing
Direct User
Testing

Characteristics Of Direct User Testing
›Direct user testing provided the input from real people encouraged
to speak their thoughts while using your application to help you
understand “Why” they prefer one version over another or uncover
issues.
›Components:
• Real people invited to participate in testing
• A lab or platform that allows users to be recorded as they test your app
• A well-thought-out experiment or hypothesis for users to test
• Note: A finished app is not required for subjects to provide meaningful results
Direct User
Testing

Direct User Testing (DUT)
Direct User
Testing

Glean insight directly from people and overhear their
assumptions, allowing teams to better meet or exceed
expectations.
You need to recruit and reward people for their time. Finding
suitable test subjects for testing skill-based applications
might be difﬁcult.
Direct User Testing Pros and Risks
Direct User
Testing

Case Studies

Continuous Innovation Requires Continuous Discovery
Direct User
Testing With
MVP Apps
A/B Testing
with MVP
Application In
Production
Focus Group
Using Simple
Mocks

EdX: Back-end Application Server Optimization
Discover
Optimal Design
A/B Testing In
Production
Create
Competing
Logic
Competing Logic Operating
In Production

Digital
Transformation
Customer ExperienceExperimentation
Cycle
Feature Release Cycle

“This allowed us to deploy
code and feel confident
that customers are seeing
the right features.”
So What Did Customers See After Experimenting?
“We get senior executives
engaged on the release
process because they can
see the data from the
experiments.”
“Now our product manager
can coordinate with
marketing, sales etc. until
they are ready to release.”
“Using feature flags does
helps the team release
faster by helping us push
continuous integration.”
“These tools force us to
align around the customer
experience.”
“We reduced the number
of dependencies, people
involved, and the stress of
making changes. It’s much
easier to iterate and
improve.”

Recommendations

Experimenting In Prod Is The New Normal
›Create a premise to narrow down choices
›Get the pros involved for high-impact changes
›Know that building your own experimentation platform can quickly
become impractical
›Don’t use experimentation as a substitute for tried and true testing
›Understand that experimentation creates technical debt

FORRESTER.COM
Thank you
Christopher Condo
Senior Analyst, Forrester Research
+1 671-613-6289
ccondo@forrester.com
@ChrisCondo2017

Optimizely Webinar: Accelerate Agile Delivery with Experimentation Platforms, Featuring Forrester

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Optimizely Webinar: Accelerate Agile Delivery with Experimentation Platforms, Featuring Forrester

Similar to Optimizely Webinar: Accelerate Agile Delivery with Experimentation Platforms, Featuring Forrester (20)

More from Optimizely

More from Optimizely (20)

Recently uploaded

Recently uploaded (20)

Optimizely Webinar: Accelerate Agile Delivery with Experimentation Platforms, Featuring Forrester