Leveraging Cloud data to optimize your product decisions and Agile processes - Ronen Aseo & Efrat Egozi Levi

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Leveraging cloud data to
optimize your Agile
processes
Ronen Aseo, Product Manager
Efrat Egozi-Levi, Senior Data Scientist, CTO office

HP Agile Manager defined
Agile project management
Release & User Story mgmt.
Defect management
ALM/PPM synchronization
ALI integrations
Key Capabilities
A SaaS solution for Agile
Instant-on service
Regular, incremental
releases driven by
user feedback
Web-based, intuitive UI
Complements HP ALM/QC
Key
Characteristics

Ø  3x number of users within the last year
Ø  Monthly drops to production
Ø  Vast internal adoption throughout HP
Our Story
http://www.tonycooke.org/

HP Agile Manager Development Teams
Agile Manager Product
characteristics
•  3 applications
•  Teams are co-located
•  On premise & SaaS product
Products
ALM Synchronizer
Application
Lifecycle
Intelligence
Agile Manager
Sunnyvale,
CAProduct
Management
Prague,
CZDevelopment
Yehud,
ILDevelopment
Ho-chi minh, Vi
Development

AgM Delivery Model
SaaS Production Farms (External)
Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6
SaaS Production Farm (Internal – early adapters)
Test Farm (trunk)
Nightly
deployment
trunk
Deploy after two weeks
on internal farm, including
required bug fixes
RELEASE/PSI
Internal branch
Bug fixes
Test Farm – internal branch
Bug fixes
Deploy after two weeks
on internal farm, including
required bug fixes
Drop 1 Drop 2
Drop 1 Drop 2
Sprint length: 2 weeks

Source: http://www.redeemthecommute.com/

Challenges
Roadmap
Features
Coverag
e
Resource
s
Customer
Requests

BIG DATA -
TO THE RESCUE
Source: http://coffeebreakwithgamechangers.tumblr.com/

Data Science Office @ HP Software
Add data-
science and
analytic
capabilities in
HP products
machine
learning
graph
analysi
s
static /
dynamic
code
analysis
other
technique
s

•  We want to make informed product/dev decision based on real
users behavior
•  We have real user monitoring today e.g. Google Analytics
•  Information is event based not flow based
•  Flow based is hard to understand
•  There is code coverage but this doesn’t reflect real user usage
•  What we need is real user coverage and insights
Problem scope
Event-based analysis in Google analytics

Problem scope
External Flows
Ope
n
Chat
Write
Msg
Attac
h
Send
SendLogi
n
Test Flows
Feature:
Use Chat
Login
Open Chat
Write Msg
Send
Ope
n
Chat
Write
Msg
Attac
h
Send
SendLogi
n Ope
n
Chat
Write
Msg
Attac
h
Send
SendLogi
n

Real user test coverage algorithm
•  Difference between two flows based on their similarity score
definition
•  The similarity score combines 2 criteria (Law & Order):
–  the identical steps, which exist in both flows
–  the similarity in the step order in each flow
•  Match flows based on the difference function with an adaptive
sensitivity threshold
•  Compute the weighted similarity for a set of matched production
flows per test
Production Flows Internal / Test Flows
1. Login > 2. Open Chat > 3. Write Msg > 4. Send
1. Login > 2. Open Chat > 3. Attach > 4. Write Msg > 5.
Send
1. Login > 2. Open Chat > 3. Send
1. Login > 2. Open Chat > 3. Attach > 4. Write Msg > 5. Write Msg > 6.
Write Msg > 7. Attach > 8. Write Msg > 9. Send
1. Login > 2. Open Chat > 3. Write Msg > 4. Send > 5.
Attach
Usage
: 50%
Coverag
e: 78%
Feature X
84
78
70
75

Key Findings
Ø  The automated tests cover
35% of production flows
Ø  50% of production events are
not covered by any automatic
test
Ø  Interesting insights from
comparing internal vs.
external user flows
From the motion picture “Moneyball”,2011

Significant increase in real user test coverage
after gathering insights
0
10
20
30
40
50
60
70
80
100 150 200 250 300 350 400 450
Drop 1
stabilization
Drop 2
New
Content
#tests
Realusercoverageintests

Example: Dev Summary
External Flows Internal Flows
Release mngmnt > Taskboard > Zoom > ali summary > all > builds
Release mngmnt > Taskboard > Zoom > ali summary > all > retrospective
Release mngmnt > Taskboard > Zoom > Kanban > Kanban: Configure Team Storyboard> project
configuration teams > Taskboard > Zoom > ali summary > all > shared items
•  Release mngmnt > Taskboard > Zoom > ali summary > all > retrospective > Taskboard >
Zoom > Kanban > Kanban: Configure Team Storyboard> project configuration teams > tab:
Storyboard
•  Release mngmnt > Taskboard > Zoom > Kanban > Kanban: Configure Team Storyboard>
project configuration teams > Taskboard > Zoom > ali summary > all > shared items
•  Release mngmnt > Taskboard > Zoom > ali summary > all > builds
•  Release mngmnt > Taskboard > Zoom > ali summary > all > retrospective
•  Compared: external
customers to internal
customers
•  Surprise: page is used
similarly, regardless of
dev configuration
•  => Cancel “prompt to
config” plans
Feature:
Dev
Summary
Same usage
in external
and Internal

Example: Planning board
•  Planning board
event is rarely
used internally –
Should we
invest?
•  Surprise:
Planning board
is used as the
1st step in many
flows
Planning board Kanban
Release
backlog
[?]
Sprint Backlog …

•  Surprise: “Sprint Closure” flows have high usage (>20%) but low coverage
(3%)
•  Sprint closure should have high priority and additional test automation
Example: Sprint closure

Benefits - Recap
Data driven decision making
Better understanding & evaluation of customer needs
Improved real user test coverage
Optimize resource allocations

Thank You
Contact:
ronen.aseo@hp.com
eegozi@hp.com

Release Planning
•  Feature Ranking
•  Feature Scoping - MMF
•  Initial Feature Sizing
•  Define Feature Lead
•  Define Release Scope

Definition Of Done - DoD
ü  Unit tests are written and green
ü  Acceptance tests are defined by QA and passed on top of a build
ü  Main functionality of the user story is covered by automated acceptance tests
ü  Sanity and Regression tests are passed green after the user story is checked in
0 C/H open defects
ü  Majority of Medium defects are fixed
ü  All fixed defects are validated
ü  The user story is pushed to production, monitored and green

Delivery Model to Production
Release Criteria
Progress Status Target Current Status
Test Coverage On Track 100% •  100%
Quality On Track No Critical defects
No C/H New features defects
No C/H customer encountered defects
No Functional Regression defects
<10 High defects
70% M defects fixed
•  0 Critical defects
•  0 C/H New features defects
•  1 C/H customer encountered defects
•  3 Functional regression
•  10 open High defects
•  61% of the M defects were fixed
Performance On Track No C/H issues No C/H issues
Security On Track No C/H issues No C/H issues

Product Owner Challenges
•  How to prioritize features and enhancement requests?
•  Can I cut a story from a feature and it will still have value for the user?
(MMF)
•  How will a new feature impact current usage?
•  Does my acceptance criteria cover the end user’s flow?
•  Are features being used the way they are “supposed” to?
•  Is there a difference between how features are used internally vs.
externally?

Team Challenges
•  How much to invest in automation?
•  What areas to automate and how to automate?
•  What to include in our Regression testing?

Leveraging Cloud data to optimize your product decisions and Agile processes - Ronen Aseo & Efrat Egozi Levi

More Related Content

What's hot

Viewers also liked

Similar to Leveraging Cloud data to optimize your product decisions and Agile processes - Ronen Aseo & Efrat Egozi Levi

More from AgileSparks

Leveraging Cloud data to optimize your product decisions and Agile processes - Ronen Aseo & Efrat Egozi Levi