This document summarizes a presentation given by Bianca Jiang and Ginny Ghezzo at the O'Reilly Software Architecture Conference on March 18, 2015 about re-architecting maintenance for continuous delivery. The presentation discussed the challenges faced with traditional maintenance approaches and how adopting DevOps practices like automation, continuous delivery and testing can help improve maintenance by providing quicker, higher quality updates on a more predictable schedule with lower costs. Key aspects covered included implementing maintenance as incremental patches in an automated pipeline, generating documentation, and ensuring quality through continuous testing. The goal is to make maintenance a seamless process for customers.

1. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Chaos to DevOps
Re-architecting Maintenance for Continuous Delivery
Bianca Jiang, Senior Software Engineer
Ginny C Ghezzo, Program Director
March 18, 2015
O’Reilly Software Architecture Conference

2. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
What we do - Collaborative Lifecycle Management
2

3. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
What is DevOps
“The agile, "devops" culture will move from the software development lab to the front office. That will
be required of sales and marketing, customer service, HR, product development… they (customers)
will expect you to behave differently. Rapid response, iterative, fast with quality. It is a rare front office
today that is ready for this.”
– Ginni Rometty, CEO IBM
3

4. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Strength in the Delivery Pipeline
4

5. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
CLM Improvements through DevOps
5
Continuous Delivery exposed some challenges in our implementation of
the IBM Software Support LifeCycle Policy

6. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Maintenance Solution in a DevOps Journey
Challenge: How do we deliver high quality, timely resolution
to defects that meets IBM’s Support Lifecycle agreement
1.Limited content through governance
2.Detailed description of content
3.Consistent for all Customers and Cumulative
4.Clarity on Upgrade Path
5.In-place installation
7.Easy Rollback
8.Right-sized
6

7. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Challenges exposed by CD
7

8. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Challenges exposed by CD
100+ product releases in Support Lifecyle in 2014
8

9. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Challenges exposed by CD
100+ product releases in Support Lifecyle in 2014
Half a million downloads from jazz.net
9

10. Traditional Approaches: Hot Fix & Full Release
Hot Fix A
Hot Fix B
Hot Fix B
Full
Upgrade
10

11. Cost of Traditional Maintenance - One Defect
4.0.6 Maintenance Stream
Joe’s 4.0.3 workspace
for Customer A
Mark’s 4.0.3 workspace
for Customer B
Lisa’s 3.0.1.6 workspace
for Customer B
Hot Fix
Hot Fix
Hot Fix
3.0.1.6 Maintenance Stream Full Releases
Customer A
Customer B
All
Full Releases
11

12. “The Matrix of Chaos”
APAR 1 APAR 2 APAR 3 … ...
… ...
12

13. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Mission
Quality , predictable and timely maintenance
that is a no-brainer to deploy and manage for
customers, with lower cost.
13

14. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Mission
Quality , predictable and timely maintenance
that is a no-brainer to deploy and manage for
customers, with lower cost.
L3 Subtitle:
Make customers happier, faster, with less resources
14

15. Maintenance Stream
WI 1
GA
WI 2 WI 3 WI 4 WI 5 WI 6
What is required of CLM Maintenance?
Delta
Scope:
● Corrective + Proactive Fixes
● Changes on Maintenance Stream after GA
Goal:
● Deliver the “delta”: timely, frequently, and iteratively with high quality
● Deploy the “delta”: easily, quickly, and continuously with low risk
15

16. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Mission
Quality , predictable and timely maintenance
that is a no-brainer to deploy and manage for
customers, with lower cost.
L3 Subtitle:
Make customers happier, faster, with less resources
16

17. Maintenance Stream
WI 1
GA
WI 2 WI 3 WI 4 WI 5 WI 6
Delta
Deploy
The Deploy & Build problem
Traditional Installer
1. directly modify existing
installed bits
2. can not rollback
3. big footprint
4. can’t do *iterative update*
17

18. Maintenance Stream
WI 1
Base
WI 2 WI 3 WI 4 WI 5 WI 6
Delta
Patch Service
Drop in one file to patch the entire server.
1. does not modify existing bits, run-time
2. fast, easy install, easy rollback
3. very lean
4. *iterative update* possible
Traditional Installer
1. directly modify existing
installed bits
2. can not rollback
3. big footprint
4. can’t do *iterative update*
Deploy
New Approach
The Deploy & Build problem
18

19. Maintenance Stream
WI 1
Base
WI 2 WI 3 WI 4 WI 5 WI 6
Delta
Deploy
The Deploy & Build problem
Traditional Installer
1. directly modify existing
installed bits
2. can not rollback
3. big footprint
4. can’t do *iterative update*
Patch Service
Drop in one file to patch the entire server.
1. does not modify existing bits, run-time
2. fast, easy install, easy rollback
3. very lean
4. *iterative update* possible
Make it Reusable!
19
New Approach

20. Maintenance Stream
Patch
WI 1 WI 2 WI 3 WI 4 WI 5 WI 6GA
Generate
The Deploy & Build problem
?
20

21. Maintenance Stream
Patch
WI 1 WI 2 WI 3 WI 4 WI 5 WI 6GA
Automation!
Generate
The Deploy & Build problem
21

22. Maintenance Stream
Patch
WI 1 WI 2 WI 3 WI 4 WI 5 WI 6GA
Automation! Generate
The Deploy & Build problem
22

23. Maintenance Stream
Patch
WI 1 WI 2 WI 3 WI 4 WI 5 WI 6GA
Automation! Generate
The Deploy & Build problem
23

24. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Mission
Quality , predictable and timely maintenance
that is a no-brainer to deploy and manage for
customers, with lower cost.
24

25. Maintenance Stream
Patch
WI 1 WI 2 WI 3 WI 4 WI 5 WI 6
Easy to manage: Traceability
GA
Automation! Generate Manage
?
25

26. Manage
Automation!
Generate
Maintenance Stream
WI 1GA WI 2 WI 3 WI 4 WI 5 WI 6
Patch
Easy to manage: Traceability
26

27. Manage
Automation!
Generate
Maintenance Stream
WI 1GA WI 2 WI 3 WI 4 WI 5 WI 6
Patch
readme.txt
Easy to manage: Documentation
Traceability
27

28. Deploy, Build and Documentation
Manage
Automation!
Generate
Maintenance Stream
WI 1GA WI 2 WI 3 WI 4 WI 5 WI 6
Patch
readme.txt
28

29. Manage
Automation!
Generate
Maintenance Stream
WI 1GA WI 2 WI 3 WI 4 WI 5 WI 6
Patch
readme.txt
Automation!
Deploy, Build and Documentation
29

30. Manage
Automation!
Generate
Maintenance Stream
WI 1GA WI 2 WI 3 WI 4 WI 5 WI 6
Patch
readme.txt
Automation!
Deploy, Build and Documentation
30

31. Manage
Automation!
Generate
Maintenance Stream
WI 1GA WI 2 WI 3 WI 4 WI 5 WI 6
Patch
readme.txt
Automation!
Deploy, Build and Documentation
change
build
31

32. Manage
Automation!
Generate
Maintenance Stream
WI 1GA WI 2 WI 3 WI 4 WI 5 WI 6
Patch
readme.txt
Automation!
Deploy, Build and Documentation
change
build
result
32

33. Manage
Automation!
Generate
Maintenance Stream
WI 1GA WI 2 WI 3 WI 4 WI 5 WI 6
Patch
readme.txt
Automation!
Deploy, Build and Documentation
resultchange
build
change
build
result
33

34. Manage
Automation!
Generate
Maintenance Stream
WI 1GA WI 2 WI 3 WI 4 WI 5 WI 6
Patch
readme.txt
Automation!
Deploy, Build and Documentation
resultchange
build
resultchange
build
change
build
result
34

35. Manage
Automation!
Generate
Maintenance Stream
WI 1GA WI 2 WI 3 WI 4 WI 5 WI 6
Patch
readme.txt
Automation!
Deploy, Build and Documentation
Towards Continuous Delivery
resultchange
build
resultchange
build
change
build
result
35

36. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Mission
Quality , predictable and timely maintenance
that is a no-brainer to deploy and manage for
customers, with lower cost.
36

37. Quality: Continuous Testing
PipelineBuild
Unit Test
Acceptance Test
Scenario Test
Integration Test
Patch Content
37

38. Quality: Continuous Testing
PipelineBuild
Unit Test
Deploy
Acceptance Test
Scenario Test
Integration Test
Patch Content
Generate
38

39. Quality: Continuous Testing
PipelineBuild
Unit Test
L3 DevDeploy
Acceptance Test
Scenario Test
Integration Test
Patch Content
Generate Notify
39
Analyze
Failure & Fix

40. Quality: Continuous Testing
PipelineBuild
Unit Test
L3 DevDeploy
Acceptance Test
Scenario Test
Integration Test
Patch Content
Generate Notify
Auto Auto Auto
40
Analyze
Failure & Fix

41. Quality: Continuous Testing
PipelineBuild
Unit Test
L3 DevDeploy
Acceptance Test
Scenario Test
Integration Test
Patch Content
Generate Notify
Auto Auto Auto
41
Analyze
Failure & Fix

42. Quality: Continuous Testing
PipelineBuild
Unit Test
L3 DevDeploy
Acceptance Test
Scenario Test
Integration Test
Patch Content
Generate Notify
Change
Auto Auto Auto
Analyze
Failure & Fix
42

43. Quality: Continuous Testing
PipelineBuild
Unit Test
L3 Dev
Acceptance Test
Scenario Test
Integration Test
Patch Content
Additional
Manual Testing
Customer
Scenerio
Verification
change
build
feedback
deploy test
change
build
feedback
test
change
build
feedback
deploy test
change
build
result
43

44. Quality: Continuous Testing
PipelineBuild
Unit Test
L3 Dev
Acceptance Test
Scenario Test
Integration Test
Patch Content
Additional
Manual Testing
Customer
Scenerio
Verification
change
build
feedback
deploy test
change
build
feedback
test
change
build
feedback
deploy test
change
build
deploy
44

45. Quality: Continuous Testing
PipelineBuild
Unit Test
L3 Dev
Acceptance Test
Scenario Test
Integration Test
Patch Content
Additional
Manual Testing
Customer
Scenerio
Verification
change
build
feedback
deploy test
change
build
feedback
test
change
build
feedback
deploy test
change
build
deploy test
45

46. Quality: Continuous Testing
PipelineBuild
Unit Test
L3 Dev
Acceptance Test
Scenario Test
Integration Test
Patch Content
Additional
Manual Testing
Customer
Scenerio
Verification
change
build
feedback
deploy test
change
build
feedback
test
change
build
feedback
deploy test
change
build
feedback
deploy test
46

47. Quality: Continuous Testing
PipelineBuild
Unit Test
L3 Dev
Acceptance Test
Scenario Test
Integration Test
Patch Content
Additional
Manual Testing
Customer
Scenerio
Verification
change
build
feedback
deploy test
change
build
feedback
test
change
build
feedback
deploy test
change
build
feedback
deploy test
change
build
feedback
deploy test
change
build
feedback
deploy test
Quality Continuous Delivery
47

48. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Mission
Quality , predictable and timely maintenance
that is a no-brainer to deploy and manage for
customers, with lower cost.
48

49. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Process: Development & Delivery
• 1 Code Review, 1 Team Lead
Approval
•1 Code Review, 2 Team Lead
Approvals
•1 Code Review, 2 Team Lead
Approvals, 1 PMC Approval
49

50. Maintenance via iFix
Maintenance Stream
GA
Patch File 1
Non-Patch Files
Patch File 2
Non-Patch Files
Patch File 3
Non-Patch Files
iFix001
iFix002
iFix003 Latest
Customers
WI 1 WI 2 WI 3 WI 4 WI 5 WI 6
● Cumulative
● Continuous deploy
● Predictable
● Tracibility
● Upgrade Path
● Consistent
● Customer Feedback
● Agile
50

51. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Mission
Quality , predictable and timely maintenance
that is a no-brainer to deploy and manage for
customers, with lower cost.
51

52. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
CLM Maintenance Improvements
52
“The process to get the iFixes for CLM works perfect. I’m up and running with V.5.0.1 ifix004 within 30min
after announcement on first server. A big compliment to IBM Rational for this ifix concept and the
implementation of it. It’s a huge improvement for us as customers and it gives a lot of trust into the product.”

53. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
1. Smaller improvement, faster feedback loop
2. Short term investment, long term gain
3. Make it reusable, make it repeatable
4. [leadership] lead by example, lead by influence
5. [leadership] More on the why, less on the how
Feedback
Continuous Architecturing
Improve FeedbackImprove FeedbackImprove
Take away: DevOps Architecture
53

54. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Resources
● Jazz.net - https://jazz.net/
● IBM DevOps - http://www.ibm.com/ibm/devops/us/en/
● Bianca Blog - https://jazz.net/blog/index.php/2014/06/27/devops-for-clm-
maintenance-1/
● Working Like a Startup - http://radar.oreilly.com/2014/09/working-like-a-
startup-at-ibm.html
● IBM Support Lifecycle - http://www-01.ibm.
com/software/support/lifecycle/lc-policy.html
● Enterprise DevOps Best Practices for Executives - https://ibm.
biz/BestPracticesExec
● Inspirational Leadership TED talk -
http://www.ted.com/talks/simon_sinek_how_great_leaders_inspire_action
54

55. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
55
Contact Us:
bjiang@us.ibm.com
ginnyb@us.ibm.com

56. @BiancaJiang @GinnyGhezzo #oreillysacon #devops
Disclaimer
Copyright © 2015 by International Business Machines Corporation (IBM).
Information in these presentations (including information relating to products that have not yet been announced by IBM)
has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or
typographical errors. IBM shall have no responsibility to update this information. THIS document is distributed "AS IS"
without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this
information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM
products and services are warranted according to the terms and conditions of the agreements under which they are
provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal
without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples
are presented as illustrations of how those customers have used IBM products and the results they may have achieved.
Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such
products, programs or services available in all countries in which IBM operates or does business.
56