1) The document discusses the lessons learned from adopting DevOps practices at a large scale for IBM z Systems software development. 2) It describes the journey of transforming over 20,000 developers through practices like continuous integration, automated testing, and collaboration tools. 3) Challenges included supporting mainframe environments, dispersed teams, legal requirements, and integrating many products; successes included improved quality, speed, and job satisfaction.

1. Lessons Learned from Large-Scale Adoption
of DevOps for IBM z Systems Software
Per Kroll
Director Offerings Mgmt, Development & Support
DevOps for Enterprise Systems

2. 1
My journey…
Creator and Chief Architect
 Extended Core Team
 Transforming +20,000 developers
 Practice library for agility@scale
 Deployment assets
 … Initiator and Project lead
Product Manager and Development Manager
Rational Unified Process
Development and Support Executive
DevOps for Enterprise Systems

3. Discussion Topics
• Our context
• A journey – all things agile
• Running an agile business: Offerings Management and IBM Design Thinking
• Adopting DevOps
• Driving large-scale organizational change
• Parting thoughts
2

4. Why transform
• Innovation – exceed
customer expectations and
beat competition
• Delight customers through an
iconic user experience
• Quality
• Speed
• Cost
• Job satisfaction
3

5. Complexities we face as a product organization
Common in the industry
• Extremely distributed
• Varying need for business agility and investment levels
• Mixture of tools, including homegrown
• Dispersed cultures, many acquisitions
Specific for our product development organization
• Multi platform (multiple OSs, DBs, App Server, browsers, ….) – 100s of permutations to test
• Translations to dozens of languages
• Legal and packaging constraints
Specific for mainframe development
• Many commercial and open source tools do not support mainframes
• Attitude: ‘Mainframe is different’, average age is higher 4

6. Discussion Topics
• Our context
• A journey – all things agile
• Running an agile business: Offerings Management and IBM Design Thinking
• Adopting DevOps
• Driving large-scale organizational change
• Parting thoughts
5

7. Continuous Improvement: An Agile Non-negotiable
6
• There is no such thing as “100% Agile”
• Stop living with pain - leverage known practices to
alleviate your pain
• Automated build and deployment
• Shift left testing: test early, test often
• Loosely coupled architecture
• ….
“If you can’t tell me you’re better this month than you
were last month, don’t tell me you’re ‘Agile!’”

8. All things agile
User
Experience
Development
Business

9. All things agile
User
Experience
Development
Business
Design Thinking Offerings Management
DevOps
Innovation

10. Discussion Topics
• Our context
• A journey – all things agile
• Running an agile business: Offerings Management and IBM Design Thinking
• Adopting DevOps
• Driving large-scale organizational change
• Parting thoughts
9

11. Problem: Business Process
• Our business process, IPD, was too heavy
– A lot of overhead
– Encouraged over specification
– Discouraged early experimentation
– Forced too early commitment to a specific solution
– Expensive business change process
• Solution: Introduce Offerings Management Discipline
– A lighter-weight business process
– Centered around the principles for the LeanStartup

12. Offerings Management Discipline: Inspired by the Lean StartUp
Light-weight artifacts, with focus on learning and pivoting
11
Learn and Pivot
Light-weight artifacts
Focus on markets and
business outcomes

13. Design Thinking

14. Design Thinking
Understand
Explore
Prototype
Evaluate
Hills
Invest for market
outcomes
Sponsor Users
Understand the user
experience
Playbacks
Collaborate. Align.
Engage!

15. IBM is increasingly recognized for design leadership

16. Discussion Topics
• Our context
• A journey – all things agile
• Running an agile business: Offerings Management and IBM Design Thinking
• Adopting DevOps
• Driving large-scale organizational change
• Parting thoughts
15

17. Adopting DevOps: 2015 Focus Areas
• Build and deployment automation: Fast and extremely high ROI
– Weeks or months, not quarters or years
– Key products: Rational Team Concert Enterprise Edition, Jenkins, and Urban Code Deploy
• Test automation: Crucial, but labor intensive
– Incremental adoption, targeted investments: default priority order
1. Build Verification Test – Automate most frequently run tests
2. New features – Stop digging a deeper hole…
3. Components with many defects – Immediate quality benefits
4. Components with low test coverage – Incrementally bite the bullet
– If tests never fail and component not updated => Do you need to automate?
– Key products: RDz zUnit, Rational Test Workbench, Application Delivery Intelligence
• Not a focus: Already mature areas
– Collaborative Development: Rational Team Concert (RTC)
– Modern Development: Modern IDEs / Problem Determination tools (RDz / PD Tools) 16

18. • Cross discipline feature teams
• Establishing rhythm, continuous
integration, definition of done
• Shift left testing
• Complex testing run continuously
• Automation part of development
• Anything done manually analyzed
for elimination or automation
• Retrospectives, eliminate friction
points
17
Plan Develop Develop Develop Test/Fix/Test
4 weeks 4 weeks 4 weeks
Warmup Sprint 1 Sprint 2 End Game…
Develop Develop Test/Fix
Test
Plan
retrospective
Sprint 1 Sprint 2 End Game
4 weeks4 weeks
 Quarterly Releases
 Ship ready with each Sprint
 Create a culture committed to automating
everything…
Continuously Test Using Automation and Virtualization
Example: Collaborative Lifecycle Management Team

19. Example: Increasing Test Automation in IBM Rational Developer for z
Problems
• Long release cycles for both new releases and service
• Development process more waterfall than agile
• Too much effort spent on manual testing
Progress made in 2015H2
• Educate team (JUnit and Rational Functional Tester)
• Ensure JUnits are added for all new capabilities
• Optimize BVT suite to run in 15 min or less, triggered by a build. Publish results.
• Analyze components to understand where to invest: Quality issues, test coverage, churn
• Apply swat team for rapid increase in test automation
• Adopt strategic automation tooling: UrbanCode Deploy
• Incorporate data-driven frameworks to automate complex areas
18

20. Example: Increasing Test Automation in IBM Rational Developer for z
Results so far (work in progress)
• Automated test coverage increased from 20% to 50%
• Daily BVT uncovered defects earlier in the cycle, resulting in quality improvement
• Reduced regression testing for each deliverable from 3 weeks to 2 weeks
• Shortened time to test while growing number of test cases and improving quality
• Test team able to handle more deliverables and cover more integrations
19

21. Executes 10000s of Junits & security scans
Develop
Test (continuous) Interactive Testing
Usability Test
Security Pen testing
Unit
Test
Production
Jazz.net
System
Test
Performance
Test
Integration Test
Function
Test
Build
Production-Like Environment Production-Like
Environment
 Global dev/test/ops team ~100 daily commits/deploys
using our tools to build our tools
 191,000 jazz.net community members, 50K sandbox
instances, >105K page views
DeploymentsDeploy Golden Topology
IBM Bluemix
DevOps Services
20
Developers launch
test from their
workspace Rapid install and smoke Tests
Functional and systems tests on Golden Topologies
B
V
T
“Simply put, things always had to be in a
production-ready state: if you wrote it, you darn
well had to be there to get it running!”
- Mike Miller,
Co-founder and Chief Scientist at Cloudant,
an IBM Company
Continuous Integration and Test
Build a Delivery Pipeline Leveraging Tools-As-a-Service
Example: IBM Collaborative Lifecycle Management Team

22. Continuous Delivery Pipeline Dashboard
Example: IBM Collaborative Lifecycle Management Team
21
• Automated quality gates
• Single view of quality
• Ability to quickly drill down on problems
• View the live dashboard at: jazz.net

23. IBM CICS Tools (Ant / RTC Build)
IBM CICS Explorer (Ant / RTC Build)
Example: IBM z Systems DevOps Transformation
Key Challenges (2015, Q2):
• Products get built, tested & delivered
separately
• Inconsistent build technologies &
infrastructure
• Lack of synergy in driving product
compatibility
• Lots of manual interventions (e.g.
testing, deployment)
Impact to customers:
• Difficult for customers to figure out what
version of which products work together
• Late discovery of defects, especially
related to cross-product integrations
• Some products may not work well
together 22
DevIESCEX
DB2IES Dev SCM Build Test
Feedback
IBM Rational Developer for z Systems (Eclipse PDE Build)
IBM Problem Determination Tools (Maven / Jenkins Build)
IBM z/OS Explorer (Ant / RTC Build)
IM Download site
(RDz)
IM Download site
(PD Tools)
Eclipse p2 Update Site
(CICS Explorer)
Eclipse p2 Update Site
(z/OS Explorer)
IM Download site
(CICS Tools)

24. “Internal” DevOps (Phase 1) delivers IBM ADF for z Systems
23
Eclipse 4.4 Luna
z/OS Explorer V3.0 Aqua
Rational Developer for z Systems
Application Delivery Foundation for z Systems
App Performance
Analyzer
Debug
Tool
File
Manager
Fault
Analyzer
CICS
Explorer
z/OS
Connect EE
RTC
CICS Tools
Single IM
Composite
Site
Single Eclipse
p2 Update
Site
IBM Explorer for z/OS Aqua Release Highlights
 A single “Release Train” delivery pipeline
(more than a dozen products on z/OS Explorer)
 Common build technology & infrastructure
 Common software stack & dependencies
 Improved cumulative build time
 Improved integrated testing across products
 Improved overall product compatibility & quality
 A single download site for IM and Eclipse p2 installs
 Improved user experience with a single “Mainframe
Dev” site on IBM DeveloperWorks for all products

25. IBM Application Delivery Foundation for z Systems V1.2
24
• A comprehensive set of z/OS tools with a modern, integrated & consistent user experience
• Best of breed capabilities in a single bundle enables the delivery of high quality z/OS applications quickly & efficiently
– Accelerates development, test & maintenance of z/OS applications
– Increases debugging efficiencies & reduces application development lifecycle
– Helps isolate system constraints quickly & maximize application response time
– Expedites the identification & analysis of system and application failures
– Simplifies manipulation of data in data stores such as DB2, IMS, CICS, MQ, data sets & HFS files
• A single & attractive pricing model for the complete bundle
• Accelerates the DevOps transformation of z/OS application delivery
RDz
DT
FA
APA
FM

26. IBM z Systems DevOps Transformation – Q2 2016 (Phase 2)
Phase 1 – Dec 2015 Phase 2 – Q2 2016
Theme – “Extend DevOps through automation”
• Extend existing Continuous Integration pipeline to support Test & Deploy
• On-board & streamline product delivery to be on the common pipeline
• Increase focus & investment on test case automation (e.g. BVT, smoke test, IVT)
• Integrate with IBM UrbanCode Deploy to orchestrate test automation & deployment
Expected Outcome & ROI
• Reduce TCO (total cost of ownership) by replacing manual work with automation
• Improve speed of product delivery across our portfolio
• Further improve product quality & compatibility through continuous/automated testing
Single Eclipse
p2 Update Site
Single IM
Composite Site

27. ADF: Business Value of our DevOps Transformation
26
Our adoption of DevOps improves
speed, reduces waste & enhances quality
Improvement Area Q1, 2015 Q2, 2016 Improvement Benefits
Release frequency 12‐18 months 1 month 90‐95%
‐ Accelerates innovation
‐ Improves Time‐To‐Market (TTM)
‐ Continually drive higher product quality
‐ Enables continuous delivery
Cumulative build time ‐‐ ‐‐ 30%
‐ Earlier detection of issues
‐ Shortens development lifecycle
WASTE
REDUCTION Build & delivery process
5 different
processes
1 consolidated
process 80%
‐ Eliminates duplications of work
‐ Deflects customer reported issues
Automation: BVT & Integration Test 0% 100% N/A
‐ Earlier detection of issues
‐ Enables continuous testing & delivery
Automation: Deployment 0% 100% N/A
‐ Reduced labour
‐ Enables continuous testing & delivery
Dependency Management Ad‐hoc
Common
repository N/A
‐ Earlier detection of issues
‐ Enhances product compatibility
SPEED
QUALITY

28. Discussion Topics
• Our context
• A journey – all things agile
• Running an agile business: Offerings Management and IBM Design Thinking
• Adopting DevOps
• Driving large-scale organizational change
• Parting thoughts
27

29. Driving Large-Scale Organizational Change
Key factors
• Provide clarity in objectives
• Drive the right culture
• Gain executive buy-in
• Establish bottoms-up communities
• Encourage self-assessments
• Drive tools standardization while allowing appropriate levels of freedom
• Automate everything: Build, Test, Deploy
• Govern through transparency: Aid, remove roadblocks, and establish accountability
28
Becoming agile is a journey. Enjoy your journey.

30. Drive the right culture: Values and Principles
Help install values through social contracts
29
Principles

31. Make self-organization work: Clarity of Outcome
30

32. Establish bottom-up communities
31
+10,000
members
Several events
every week
Easy access
to experts
Learn from, teach and interact with
people in your area of interest

33. Line of
Business
Rational Doors Next Generation (RDNG)
Rational Team Concert (RTC)
Request for Enhancement (RFE)
Rational Quality Manager (RQM)
Rational Test Workbench (RTW)
Rational Test Virtualization Server (RTVS)
jUnit Automation Tool (JAT), JazzSM-Dash
Jenkins
RTC EE, RTC, ANT, Jenkins
Rational Team Concert (RTC)
Rational Developer for the Enterprise
(RDz, RDi, RDp, RD A&L)
Git
Pipeline automation
Rational Team Concert (RTC)
Jenkins
UrbanCode Deploy (UCD)
zService, SPoRT
Golden Topology: Driving tools standardization
while allowing appropriate levels of freedom
15

34. Self Assessments and Roadmaps to Address Gaps
Project
Management
Project Tracking Design & Design
Thinking
Builds Technical Debt Automated
Testing
Whole
Teams
Provisioning Automated
Deployments
Monitoring of
Environments
Collaboration Customer Focus Monitoring
Customer Usage
1 1 1.5 1.25 1.5 3 0.5 5 6 2 0 1.75 0.5
PLANNING BASIC SOFTWARE ENGINEERING PRACTICES DEPLOYMENTS TO TEST AND
PRODUCTION (as applicable)
DEV & OPS
WORKING
TOGETHER
CUSTOMERS
3.5 11.25 8 0 2.25
Area: XXX
Contact: XXX
Model assessment results – as of 04/09/2015
No actions Planned
Future actions TBD
Areas being addressed
Area Summary of actions Target date Target
Metric
Automated Testing • Building test center of competency
• Adopting test driven development
• Created detailed action deck already being executed on
• 06/30/15
• 11/01/15
1
Automated Deployments • Investigate UCD usage
• Utilize for new offerings
• Adopt for existing products and include provisioning
• 07/01/15
• 12/1/15
• 06/30/16
3
Dependency • Requires ability to staff Mexico team
Risks • Continued escalations and test fixes taking resource from these efforts
Issues • None

35. Effective governance through transparency:
Aid, remove roadblocks, and establish accountability
Central dashboards providing visibility into
• Improvement actions
• Metrics
• Inhibitors
View by organization
• Product
• Area
• Executive
Used to
• Identify common problems so we can address them strategically
• Help teams to structure their improvement efforts
• Provide accountability
34

36. 35
The wrong metrics can incent negative behavior

37. Leverage Outcome and In-Process Metrics
Outcome Metrics
• Determines whether you achieve your
end goals or not
• Responds slowly to change
• Examples from my organization:
– Delivery time of defect fix release
– Delivery time of major release
– Customer-reported quality issues
– Customer satisfaction (survey)
– Employee satisfaction (survey)
In-Process Metrics
• Determines whether you are effectively
adopting the right practices
• Responds rapidly to change
• Examples from my organization:
– Code quality at sprint end
– Point of customer feedback for release
(e.g. 20% into a project)
– % of test automation
– % of build verification test automation
– % of regression test automation
– Duration of full regression testing (manual
as well as automated)
36

38. Discussion Topics
• Our context
• A journey – all things agile
• Running an agile business: Offerings Management and IBM Design Thinking
• Adopting DevOps
• Driving large-scale organizational change
• Parting thoughts
37

39. Driving Large-Scale Organizational Change
We are here to help!
• DevOps requires an end-to-end integrated tool set
– We are the only vendor providing an end-to-end, cross platform DevOps solution
• Driving real change requires a cultural transformation
– We have many experts and assets to help you in your transformation
• Start with our Deployment Project Office
– Work with your IBM account team, and contact Jon Sayles (jsayles@us.ibm.com) if
uncertain who to talk to
– Example: We can offer ongoing advisory sessions between our and your executives and
senior leaders
38
Becoming agile is a journey. Enjoy your journey.

40. How to Get Started  IBM DevOps Workshop
Wasted effortLegend:
Think Product-based
Agile
Automated
Collaborative
Optimizing
More
predictable
More
transparent
More
continuous
Process-based
Process-heavy
Manual
Silo-ed
Code/
Test
Run
Manage
Inefficient Leaner
Leaner and
Smarter
Productive effort
Workshop Objectives
– Define business drivers for DevOps, and see what is
currently possible for your organization
– Identify existing or planned DevOps initiatives, and
map them to the business drivers
– Determine the top inhibitors within the current
software lifecycle
– Create an adoption roadmap for these DevOps
practices, including milestones, time frames and
known adoption challenges.
Overview
– No charge!
– Led by IBM DevOps Solution Architects
– Audience : For senior IT and LOB executives and
managers in application development and IT
Operations
Sign up for a free
DevOps Innovation & Optimzation
workshop
39

41. Notices and Disclaimers
40
Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission
from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of
initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS
DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE
USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY.
IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers
have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in
which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials
and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or
their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and
interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such
laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law

42. Notices and Disclaimers Con’t.
41
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not
tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.
Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the
ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual
property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®,
FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG,
Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,
PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®,
StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business
Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

43. Thank You
Your Feedback is Important!
Access the InterConnect 2016 Conference Attendee
Portal to complete your session surveys from your
smartphone,
laptop or conference kiosk.