Susan Hanson from IBM discusses the transformation of WebSphere Application Server towards agile and continuous delivery methodologies since 2009. The document outlines challenges faced, such as a large geographically dispersed team, and how improvements in tooling, team structure, and communication have led to measurable gains in quality and efficiency. Additionally, it highlights ongoing initiatives and best practices for sustaining agile practices in a large organizational context.

1. AW9
Agile Development Concurrent Session
11/12/2014 2:45 PM
"Agility at Scale: WebSphere’s
Agile Transformation"
Presented by:
Susan Hanson
IBM Software Group
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Susan Hanson has twenty-five years of experience in developing
and delivering IBM software products across the WebSphere and
Tivoli brands. Susan has held various positions spanning testing,
development, release management, and people management.
Currently the process architect for the WebSphere Application
Server team, she is responsible for development transformation,
including processes and tooling. Susan was part of the
transformation team as the product embraced agile technologies.
She is one of the leaders in the move to a continuous
delivery/DevOps model, and the DevOps focal point for the
application integration and middleware portfolio of products.

3. 1
Agility at Scale: WebSphere
Application Server's Agile
Transformation
Susan M Hanson
IBM Corporation
hansons@us.ibm.com

4. 2
Who am I?
 A member of the IBM WebSphere
Application Server development team
 Based in Research Triangle Park, NC
 Development Transformation Lead and
Continuous Delivery Architect
 Responsible for process transformation,
including tooling, for the development team

5. 3
Who are we?
 Global development and support team
550+ employees
 Major labs in United States, United Kingdom,
Canada, China, Israel, India, and Mexico
 Individuals scattered in a few other countries
 Development and support for:
Multiple releases, editions
Multiple platforms
Multiple DBs

6. 4
Why Agile and CD?
 Adapt more quickly and easily to rapidly-
changing requirements in the marketplace
 Full lifecycle including customer feedback
feeding directly into the development plans
 Continue to improve the quality of our
deliverables to customers
 Deliver add-on features for existing
releases on a more rapid/consistent pace

7. 5
Our Major Challenges
 LARGE team (around 550 people total)
 Geographically disperse (timezone spread ~15 hours in some
cases)
 Historically silo’d around components
 Older technology with a multitude of home-grown tools and
processes built around it
 Multiple tools, some linked together, some not so much
 10 years of waterfall culture, org structure, process, and
mindset before Agile transformation

8. 6
WAS and Liberty v8.5.5.x
and beyond
Continuous Delivery Era
2014 and beyond
Waterfall Era
1998-2008
WAS v1-v7
Agile Era
2009-2013
WAS v8, v8.5, v8.5.5
Liberty Profile/Core v8.5, v8.5.5
Our Journey

9. 7
Our Journey
 Multiple major “Era’s” in our journey
 Waterfall – bulk of our releases
 Traditional waterfall (design, then dev, then test, then release)
 Agile – major transition in multiple phases
 Tooling
 Automation
 Team Structure
 Communication
 Continuous Delivery
 Improved tooling and automation as well as product architecture
changes that better enable CD (modular, zero migration,etc..)
 Changes are yielding measureable improvements in
quality, efficiency, flexibility, predictability, cost, time to
market, and ultimately competitiveness.

10. 8
What we did
Unleash the power of Rational Tools
• RTC code, builds, tests, and project management
• Clear visibility to status
• Customer requirements bridged from DevWorks into
RTC and linked to the implementing Feature
Code ready to ship at
the end of each
iteration
Team Structure
• Multi-discipline, co-located teams (where possible)
• Single release backlog
• Pull vs Push methodology
• Joint team demos, sizing, design issues, rotating
team leadership
Ownership & Accountability
• No individual code ownership, shared responsibility
• Functional Expertise areas that cross teams
• Value leadership and expertise over title and
ownership

11. 9
Tooling and Processes
 Shifted to the Rational CLM Suite
 Rational Team Concert (v2 to start, now on v4.0.5)
 Rational Quality Manager (now integrated with RTC in single
JTS)
 Single, highly-customized project area for development
linked to a separate project area for requirements
 Closed-loop process with customer feedback loop for
requirements (via IBM DeveloperWorks RFE)
 Heavy use of automation
 Shift towards joint team ownership and accountability

12. 10
Push vs Pull
 Work based on Priority only
 Previously, work was
“pushed” to a team based on
a component ownership
 Always had an owner
(component lead)
 We removed ownership but
retain a general “functional
area” in each work item
 Teams “pull” work into their
team backlog based on
priority, availability, and team
skills
 Mindset shift:
 Works items don’t have an
Owner until it floats up in the
priority queue
 Work items may not have an
owner until the priority brings it to
the “top”
 Multiple teams can contribute to
a single functional area
 Teams may be asked to work
outside of their area of expertise
to assist in other areas (we are
smart people!)
 Teams don’t sit and wait for
people to give them work ... they
go GET it
 Constantly watch a single,
prioritized list of work for work
that is higher priority than what
they have currently.

13. 11
Communication
 Agile has a focus on communications, including
stand-up scrums
 By aligning our teams to be as co-located as
possible, we are able to have more non-virtual
communication
 Individual team scrums, and then a team rep
(Iteration Lead) participates in a Scrum of
Scrums
 Geo-handover (AP -> EMEA -> US)
 Incorporated more IRC rooms and mailing lists
for cross-team collaboration
 Constant feedback via retrospectives, process
and tools continuously refined

14. 12
Ownership
 Moved many items to a “whole team ownership”
model
 Builds are monitored by a rotating set of build
monitors, so each team spends time monitoring
builds. Provides better understanding of the impacts
of build issues and infrastructure challenges.
 Each team has a technical Iteration Lead (rotates)
and each iteration, a team has Lead of Leads
responsibility
 Runs LoL scrum
 Coordinates Iteration Demos
 Facilitates cross-team collaboration
 Teams “manage” the teams

15. 13
Cycle Time Improvements
Lifecycle
Measurements
2009 2010-2011
2012-
2013
Total
Improvement
Final Regression Test
Period
28 days 14 days
N/A, done
each
iteration
28 days
Iteration Length 6 weeks 4 weeks 2 weeks 4 weeks
Build + Full Automated
Regression
N/A 1 week 6 hours 1 week
(compared to 2010)
Time Between
Releases
24 Months 24 Months 12 Months 12 Months
 Automation increased 70% (to 1.3 million test cases weekly) initially under Agile,
currently between 3 and 6 million spread across our supported platforms
 Increased test coverage across all test phases
 First beta drop for latest release delivered 7 months earlier in the cycle than under
Waterfall process
 Increased product quality

16. 14
14
Quality Improvement
 Quantifiable quality improvements by moving
from Waterfall to Agile
 Comparison of average customer reported problems
for 3 releases under Waterfall to the 2 releases under
Agile, each month from when publically available
 29.5% improvement at 12 months
 40% improvement at 24 months
 Improved customer satisfaction
 Reduced service resources allows us to put more
resources on satisfying more customer
requirements.

17. 15
Average customer reported issues by
methodology
0
200
400
600
800
1,000
1,200
GA+0 GA+3 GA+6 GA+9 GA+12 GA+15 GA+18 GA+21 GA+24
Waterfall
Agile
Quality Improvement with Agile

18. 16
16
Best Practices
 Automation, Automation, Automation
 Have your tools do everything they can and leave your engineers
to do what they do best (code, test, whatever)
 Communicate, Communicate, Communicate
 Over-communicate, be transparent with intent and direction
 Can’t see the forest for the trees - be open to suggestions for
improvements from the team ... they are living it every day in the
trenches, you may not see everything they are going through
 Don’t kill the forest to keep a couple trees – not every suggestion
from the team is going to be best for the overall organization.
 90/10 rule
 Make things as easy as possible for the 90% of cases, deal with
the 10% as exceptions
 Work items, workflows, build parameters, etc

19. 17
Best Practices
 Ensure good Agile practices
Don’t allow Technical Debt to accrue
Fix defects early, this ensures a defect is not
“masking” other defects that you find later
Regressions cannot be tolerated
Test fixes are just as important as product
fixes (again, could mask a potential product
failure)
Done actually does mean Done (everything,
not “just” the development piece of it)

20. 18
18
Best Practices
 Small teams are much easier to transform than large
ones
 For very large team organizational transformation:
 Invest in upfront education for everyone (including managers &
execs)
 Form transformation team with reps from major areas of org
(dev, test, support, docs, etc) to plot the transformation
 Prototype process changes at a smaller scale
 Find enthusiastic change agents in the trenches to help lead
the change and add credibility
 Stage transformation actions over time to avoid too much
overwhelming change at once
 Be pragmatic, don’t give up, don’t plateau,
 Keep transforming, improving, getting better
 Improvise .. Adapt ... Overcome

21. 19
What is next for us?
 In the process of a shift into a Continuous
Delivery model for a portion of our deliverables
 Monthly betas
 Monthly cloud offering
 Quarterly new feature updates
 Shift from automated-tests-with-story to
automated-tests-with-every-task
 Auto-provisioning of environments and
supporting test artifacts (like an LDAP server or
database)
 Continuous (loop) test environment with in-place
upgrades instead of single execution test buckets