In this session we will share innovative directions in which the Hewlett Packard Agile Manager development team, leverages big data analytics to optimize its agile processes and align with customer feedback. We will focus on - What are challenges of developing a an Enterprise product in continuous delivery - Personas involved in the process and their related challenges - Importance of Customer feedback and incorporating it in the development process We will discuss how production data is utilized to support decision making, prioritization and continuous improvement in development ,quality and product usability.

1. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Leveraging cloud data to
optimize your Agile
processes
Ronen Aseo, Product Manager
Efrat Egozi-Levi, Senior Data Scientist, CTO office

2. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2

3. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3
HP Agile Manager defined
Agile project management
Release & User Story mgmt.
Defect management
ALM/PPM synchronization
ALI integrations
Key Capabilities
A SaaS solution for Agile
Instant-on service
Regular, incremental
releases driven by
user feedback
Web-based, intuitive UI
Complements HP ALM/QC
Key
Characteristics

4. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4
Ø 3x number of users within the last year
Ø Monthly drops to production
Ø Vast internal adoption throughout HP
Our Story
http://www.tonycooke.org/

5. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5
HP Agile Manager Development Teams
Agile Manager Product
characteristics
• 3 applications
• Teams are co-located
• On premise & SaaS product
Products
ALM Synchronizer
Application
Lifecycle
Intelligence
Agile Manager
Sunnyvale,
CAProduct
Management
Prague,
CZDevelopment
Yehud,
ILDevelopment
Ho-chi minh, Vi
Development

6. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6
AgM Delivery Model
SaaS Production Farms (External)
Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6
SaaS Production Farm (Internal – early adapters)
Test Farm (trunk)
Nightly
deployment
trunk
Deploy after two weeks
on internal farm, including
required bug fixes
RELEASE/PSI
Internal branch
Bug fixes
Test Farm – internal branch
Bug fixes
Deploy after two weeks
on internal farm, including
required bug fixes
Drop 1 Drop 2
Drop 1 Drop 2
Sprint length: 2 weeks

7. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
Source: http://www.redeemthecommute.com/

8. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8
Challenges
Roadmap
Features
Coverag
e
Resource
s
Customer
Requests

9. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
BIG DATA -
TO THE RESCUE
Source: http://coffeebreakwithgamechangers.tumblr.com/

10. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10
Data Science Office @ HP Software
Add data-
science and
analytic
capabilities in
HP products
machine
learning
graph
analysi
s
static /
dynamic
code
analysis
other
technique
s

11. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11
• We want to make informed product/dev decision based on real
users behavior
• We have real user monitoring today e.g. Google Analytics
• Information is event based not flow based
• Flow based is hard to understand
• There is code coverage but this doesn’t reflect real user usage
• What we need is real user coverage and insights
Problem scope
Event-based analysis in Google analytics

12. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12
Problem scope
External Flows
Ope
n
Chat
Write
Msg
Attac
h
Send
SendLogi
n
Test Flows
Feature:
Use Chat
Login
Open Chat
Write Msg
Send
Ope
n
Chat
Write
Msg
Attac
h
Send
SendLogi
n Ope
n
Chat
Write
Msg
Attac
h
Send
SendLogi
n

13. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13
Real user test coverage algorithm
• Difference between two flows based on their similarity score
definition
• The similarity score combines 2 criteria (Law & Order):
– the identical steps, which exist in both flows
– the similarity in the step order in each flow
• Match flows based on the difference function with an adaptive
sensitivity threshold
• Compute the weighted similarity for a set of matched production
flows per test
Production Flows Internal / Test Flows
1. Login > 2. Open Chat > 3. Write Msg > 4. Send
1. Login > 2. Open Chat > 3. Attach > 4. Write Msg > 5.
Send
1. Login > 2. Open Chat > 3. Send
1. Login > 2. Open Chat > 3. Attach > 4. Write Msg > 5. Write Msg > 6.
Write Msg > 7. Attach > 8. Write Msg > 9. Send
1. Login > 2. Open Chat > 3. Write Msg > 4. Send > 5.
Attach
Usage
: 50%
Coverag
e: 78%
Feature X
84
78
70
75

14. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14
Key Findings
Ø The automated tests cover
35% of production flows
Ø 50% of production events are
not covered by any automatic
test
Ø Interesting insights from
comparing internal vs.
external user flows
From the motion picture “Moneyball”,2011

15. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15
Significant increase in real user test coverage
after gathering insights
0
10
20
30
40
50
60
70
80
100 150 200 250 300 350 400 450
Drop 1
stabilization
Drop 2
New
Content
#tests
Realusercoverageintests

16. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16
Example: Dev Summary
External Flows Internal Flows
Release mngmnt > Taskboard > Zoom > ali summary > all > builds
Release mngmnt > Taskboard > Zoom > ali summary > all > retrospective
Release mngmnt > Taskboard > Zoom > Kanban > Kanban: Configure Team Storyboard> project
configuration teams > Taskboard > Zoom > ali summary > all > shared items
• Release mngmnt > Taskboard > Zoom > ali summary > all > retrospective > Taskboard >
Zoom > Kanban > Kanban: Configure Team Storyboard> project configuration teams > tab:
Storyboard
• Release mngmnt > Taskboard > Zoom > Kanban > Kanban: Configure Team Storyboard>
project configuration teams > Taskboard > Zoom > ali summary > all > shared items
• Release mngmnt > Taskboard > Zoom > ali summary > all > builds
• Release mngmnt > Taskboard > Zoom > ali summary > all > retrospective
• Compared: external
customers to internal
customers
• Surprise: page is used
similarly, regardless of
dev configuration
• => Cancel “prompt to
config” plans
Feature:
Dev
Summary
Same usage
in external
and Internal

17. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.17
Example: Planning board
• Planning board
event is rarely
used internally –
Should we
invest?
• Surprise:
Planning board
is used as the
1st step in many
flows
Planning board Kanban
Release
backlog
[?]
Sprint Backlog …

18. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.18
• Surprise: “Sprint Closure” flows have high usage (>20%) but low coverage
(3%)
• Sprint closure should have high priority and additional test automation
Example: Sprint closure

19. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.19
Benefits - Recap
Data driven decision making
Better understanding & evaluation of customer needs
Improved real user test coverage
Optimize resource allocations

20. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank You
Contact:
ronen.aseo@hp.com
eegozi@hp.com

21. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21
Release Planning
• Feature Ranking
• Feature Scoping - MMF
• Initial Feature Sizing
• Define Feature Lead
• Define Release Scope

22. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.22
Definition Of Done - DoD
ü Unit tests are written and green
ü Acceptance tests are defined by QA and passed on top of a build
ü Main functionality of the user story is covered by automated acceptance tests
ü Sanity and Regression tests are passed green after the user story is checked in
0 C/H open defects
ü Majority of Medium defects are fixed
ü All fixed defects are validated
ü The user story is pushed to production, monitored and green

23. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23
Delivery Model to Production
Release Criteria
Progress Status Target Current Status
Test Coverage On Track 100% • 100%
Quality On Track No Critical defects
No C/H New features defects
No C/H customer encountered defects
No Functional Regression defects
<10 High defects
70% M defects fixed
• 0 Critical defects
• 0 C/H New features defects
• 1 C/H customer encountered defects
• 3 Functional regression
• 10 open High defects
• 61% of the M defects were fixed
Performance On Track No C/H issues No C/H issues
Security On Track No C/H issues No C/H issues

24. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.24
Product Owner Challenges
• How to prioritize features and enhancement requests?
• Can I cut a story from a feature and it will still have value for the user?
(MMF)
• How will a new feature impact current usage?
• Does my acceptance criteria cover the end user’s flow?
• Are features being used the way they are “supposed” to?
• Is there a difference between how features are used internally vs.
externally?

25. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.25
Team Challenges
• How much to invest in automation?
• What areas to automate and how to automate?
• What to include in our Regression testing?