This document discusses the transformation of HP's firmware development process from 2008 to 2011 through adopting agile practices at a large scale. In 2008, the development process was struggling with long build cycles, high costs, an inability to add resources or new products. Through adopting integrated tools, consistent environments, agile development with mini milestones, automated testing, and organizational change management, the development process was transformed. Cycle times were greatly reduced, costs dropped by 70%, capacity for innovation increased from 5% to 40%, and headcount was reduced by 50% while developing more capabilities. The transformation provided a breakthrough in development capacity through establishing a large scale agile development engine.
2. 4+ Year Large-Scale Agile Journey
400+ engineers
Complex features
Over 10M lines of codec
around the world
High-end
LaserJet
printers and
MFPs
Embedded SW
& FW
Digital Sending
and HP open
Extensibility
Platform
3. State of the Development Process: 2008
•
Lengthy Build
Integration &
Testing Cycles
6 weeks + to get through a complete testing
cycle (mainly manual)
•
Build integration taking 15-20% of resources a
week to get fixes to main
•
Manual testing a key driver and constraint for
adding products
•
Products
Lagging the
Competition
Ongoing customer issues with consistency
and lack of features
•
Marketing had essentially given up asking
for FW innovations
4. State of the Development Process: 2008
•
•
Up to 10 different branches (driven by each
product release window) in MFP
•
Costs out of
control
Development costs growing 2.5X from 20042008 and the business was
still constrained
CPE driving millions/year in CPE investments
•
Couldn’t Add
Enough
Resources
80-90% of resources just porting existing
FW to new products and qualifying
•
Unable to add new products to the plans
due to lack of FW resources
•
20% of resources developing plans that
quickly became obsolete
5. Firmware Development Transformation
Integrated
Tools
Consistent Dev
Environment
Agile
Development
with Mini
Milestones
(Sprints)
Fully automated
unit and
system test
Organizational
Change
Management
Continuous
integration and
test system
Architected for
product
variability
One branch for
all products
including CPE
6. Breakthrough Capacity Transformation
Firmware Development for Development
New Customer
Capabilities
FutureSmart
FW Large Scale
Agile
Development
Engine
Defect
Fixes
•
•
•
•
•
•
400+ developers
10+M LOC
75,000-100,000 LOC turmoil
100-150 Commits
10-15 builds /day
15,000 hours/day of testing
(90% pass rate)
7. Cycletime Driver Improvements
2008
2011
Build Bosses 1 Week
Continuous Integration 3hrs
Number of Builds 1-2
Continuous Integration 10-15/Day
Feedback on Main 1 Commit/Day
Autorevert ~100 Commits/Day
Full Manual Registration 6 Weeks
Auto Regression Testing 24 Hrs
8. Development Cost Driver Improvements
2008
2011
Code Integration 10%
Continuous Integration 2%
Detailed Planning 20%
Agile Planning 5%
Porting Code 25%
One Main Branch 15%
Current Product Support 25%
One Branch CPE 5%
Manual Testing 15%
Most Testing Automated 5%
Capacity for Innovation ~5%
Capacity for Innovation ~40%
9. State of the art FW development model
2008
2011
Costs out of control
~70% reduction in FW
development cost per program
Couldn’t add resources fast enough
50% reduction in FW headcount
Lengthy build, integration
and testing cycles
Cont. integration, daily
automated regression
Products lagging the competition
Vintage chart unleashed
and capacity for innovation
13. Improvements Best Driven at the
Enterprise Level
✔
Business
Objectives/
Priorities
✔
Enterprise
Level
Continuous
Improvement
✔
✔
CI/CD and test
automation
infrastructure
Planning
Process
14. Business Objectives (Don’t “Do Agile”)
Define
your value
proposition
Understand your
cost & cycle-time
drivers
Either automate, eliminate, or engineer out the
drivers that aren’t key to the value prop
15. Interative Approach to Agile Management
Mini-milestone
Objectives
Agile Adjustments
Cascading Objectives
to Track Progress
Having real-time
metrics is essential for
the speed of agile &
aligning the org.
But don’t manage by
metrics.
Use the metrics to
understand where to
have conversations
about what is not
getting done.
Learnings
Conversations
18. CI/CD and Test Infrastructure
How much of the system
do you put together how
often and in what order?
Where do you create test
harnesses/simulators/emulator
s?
How do you build up the
system?
Automated testing is as hard or
harder than writing good code.
How do you create
frameworks that improve
stability and productivity?
Where do you turns builds red
and stop the train versus logging
defects and tracking passing
rates?
19. Building up a Large SW System
Agile
Comp 4
Agile
Comp 2
Agile
Comp 5
Agile
Comp 3
Agile
Comp 6
Legacy
Waterfall
IT 1
Interface Test Simulator
Agile
Comp 1
Legacy
Waterfall
IT 2
Legacy
Waterfall
IT 3
20. One of the biggest challenges with
Agile Planning at the enterprise
level is getting the organization to
accept the uncertainty in SW
development and appreciate the
flexibility and opportunity.
21. Long Term Predictability for SW Schedules
Do we really need the predictability of our current planning processes?
Are our current planning processes really that accurate?
Accuracy
100%
Planning Investment
23. FutureSmart Firmware per Sprint
FutureSmart Firmware User Stories User Stories per
Sprint
Input Q ueue of user stories (not in W IP yet; being evaluated)
N ow on quarterly releases (2 “ feature sprints plus 1 “ final qual” sprint),
”
so Input Q ueue is ~2 releases of backlog (80 - 00 User Stories per release)
1
2 nd year
throughput (big
drive to complete
architecture; not
sustainable)
1 st year
throughput
(25 user
stories/ sprint)
F
inal qual for
1 st release
L
atest
throughput
(need more
data to know
new steadystate; probably
40 -50/ sprint)
F
inal qual F
inal qual
for 2 ndt
for 3 rd
release
release
24. Getting Mgmt/Mktg Buy-in to Agile Planning
“FW will still commit to basic
new product support one
year ahead”
Means prioritizing “product turn-on
and delivery/qualification” ahead of
new features
“You will get 20% more
features this way”
Easy to explain the 20% of
resources previously used to
estimate
“You get to decide what we
work on first”
Establish a “1-N feature request list”
and the combined marketing teams
decide the order
“We’ll actually listen to your
last-minute requests”
Just put it at the top of the list,
ahead of all the other “input
queue” features
25. A Practical Approach to
Large Scale Agile Development
E-mail: gbgruver@gmail.com
Blog: largescaleagile.com
Twitter: @GRUVERGary
Editor's Notes
This content should fit the slide set layout and kill the build outStart with architected, go clockwise, then organizational change management
This content should fit the slide set layout and kill the build out
Fix arrow build
I cleaned up some wording
Fix arrow build
These next two slides got dropped for some reason. I am okay with better slides but I need to keep the content
1Understand your value proposition2-Define your cost driversEither automate or engineer out the drivers that aren’t focused on value prop
I added some text
I need to accuracyvs planning investment graph in here somehow.
Katie, can we put a slide here that has a bunch graphic type people where we are trying to find who committed the offending code. I have an idea that might work where we build with colors focusing it down to a smaller group of people and then one person. Start with a hole page of people colored when you are doing qrtly releases. Then focus CI and daily builds down to 5 and then 1 with the build process.
I think this would make a good puzzle example.
A judo graphic would be nice if it work but I would understand if it does not work with the theme
I need to accuracyvs planning investment graph in here somehow.
Any way we can focus this slide to have just this concept and this graphic. Limit long term commitments to 50-60% of capacity is a key point here. Not sure how to do this but I am just putting out the key context for your design flexibility
Any way we can focus this slide to have just this concept and this graphic.