Is your delivery and testing approach able to keep pace with today’s business demands? Poor or low levels of collaboration between application owners, developers and testers using a variety of tools & practices and long cycles of provisioning of dev/test environments can impede quality, increase waste and cost. Join this workshop discussion and share your opinion. Collaborate with peers on best practices to eliminate testing bottlenecks through virtualized dependent services and how to stand up realistic, production-like test labs that can be easily deployed, shared, and updated as systems change. The workshop will conclude with Q&A among the participants.

1. © 2015 IBM Corporation
DevOps Technical Workshop -
Addressing Quality Challenges
of Highly Complex and
Integrated Applications
6204 -DevOps Workshop
Bill Hudacek (IBM)
Vinit Kutty (IBM)
Andrew Williams (IBM)

2. Workshop Objectives
• Interact with other clients and explore case studies on how to
use new approaches to testing to reduce waste and increase
speed and agility
• Share and capture best practices in new testing processes,
tools and capabilities
• Walk away with an action plan developed with insights from
other clients and subject matter experts
1

3. Agenda
• DevOps and Quality
• Concepts & Terminology
• Scenario Discussion
• Practices & Challenges
• Solutions
• Best Practices Discussion
2

4. DevOps and Quality
3

5. The reality our customers face every day…
Testing Labor Test Lab Complexity
Heterogeneous Environments
Public Cloud
Private Cloud
Data Warehouse Mainframe
Enterprise
Service Bus
Directory
Identity
File
systems
Collaboration
Web/
Internet
Routing
Service
Third-party
Services Portals
Content
Providers EJB
Shared
ServicesArchives
Business
Partners
Messaging
Services
Traditional
Testing
Other costs
Testing costs
of customers
experience
production delays
>45%
to deliver validated test
environments*
~2-8weeks
of large organizations
spend approximately
25%–30% of project
budget on testing **
>60%
Concepts & Terminology
4
Development Velocity

6. Clients are experiencing increasing complexity
…while demanding cycle reductions
Concepts & Terminology
5

7. Continuous
Customer
Feedback &
Optimization
Collaborative
Development
Continuous Release
and Deployment
Continuous
Monitoring
Continuous
Business Planning
Continuous
Testing
Operate Develop/
Test
Deploy
Steer
DevOps
Continuous
Feedback
Introducing IBM DevOps
6
Concepts & Terminology
Accelerate
software delivery –
for faster time to value
Balance speed, cost,
quality and risk –
for increased capacity
to innovate
Reduce time to
customer feedback –
for improved customer
experience

8. Eliminate testing
bottlenecks through
virtualized dependent
services
Reduces the cost of testing
by increasing levels of
automation
Accelerates time to
delivery through shorter test
cycle times
r
Remove defects earlier
through advanced defect
management
Interconnected
Technology
Operate
DataDefects
Reports
BUILD
EXECUTE / VIRTUALIZE
PREPARE
ANALYZE &
REPORT
Automated
Process
DevOps
Continuous
Testing
Environment
Application
Test
Execution
Enterprise
Dashboard
DevOps delivers an accelerated test approach
Interconnected and seamless
7
Concepts & Terminology

9. Continuous Testing features
Continuous Testing
Test Automation
• Run unattended
• Validate business
flows across channels
• Durable, maintainable,
resilient to change
Orchestration from build to production deployment
• Reliable • Repeatable • Traceable
Test Environments
• Virtual services
• Realistic & complete
- (production like)
• Available on demand
• Inexpensive to setup
Manage Quality across the lifecycle
• Collaborate • Govern • Report
8
DevOps Foundation

10. … and interlocked with other DevOps solutions
At least 1/3 of The
Continuous Delivery Pipeline
is testing!
Everything
between development &
production is a test stage!
Concepts & Terminology
9

11. Scenario Discussion
The world before adopting
DevOps and Continuous
Testing…
Introduction to some
Practices and Solutions
10

12. Thursday: Phil finds a test
environment but it’s not an
exact match; he identifies
back-ends from other test
environments that can be
used but now tests will
need to be scheduled
carefully to avoid conflicts.
Monday: Mike discovers
that the requested test
environment and a few Core
Banking Solution back-end
services are not available yet.
Mike calls Phil (Chuck’s boss)
because Chuck (Env guy) is
on leave and it’s been a week.
An urgent message is left.
!
Test Environment
NOT delivered
MIKE
Dev Lead
Cost
Productivity
An every day scenario
Testing a Core Banking Solution
11
Practices and Solutions
!
PHIL
Env Lead
Tuesday: Phil gets the
message and estimates it will
take 2 weeks to set up the test
environment and make all
required Core Banking
Solution back-ends available.
Phil drops everything to find
an alternative…
BTW Test team is
ready
SUE
Test Lead
Friday: Test
Execution is
rescheduled!
Friday: The Test Team starts
test environment validation but
identifies significant issues –
some infrastructure-related,
and some from conflicts due to
another team’s testing hitting
the same back-end services
The following Monday,
There’s no test automation so testing
is manual. After three days
of testing, 450 defects are reported –
most turn out to be invalid.
1 week later: Sue
gives the green light
one week
late!
Wednesday: Test Team is still
ready to test but stops work on
preparation until an alternative
environment is identified
THE Test Team

13. Let’s analyse the scenario
12
!
IMPACT
CAUSE
SOLUTION
• Test delays and disruptions
• Higher risk of component failure
• Manual, inefficient and inaccurate
• More unplanned IT work
• High level of re-work
• T&M costs increased
• Delays to implementation
of new functionality
• Manual provision of test environments
• Manual gaps between automated processes
• Manual install/configure
• Manual test setup (back end providers, data setup &
resource dependency)
?
Practices and Solutions
Automated
Integration Testing
Service
Virtualization
Deployment
Automation
Continuous
Testing

14. User
Interface
Integration
Component
Unit Tests
13
Testing Pyramid – Automation is Key, but where ?
Integration Testing and Virtualization
Practices and Solutions

15. Using virtual components
Continuously test what you want, when you want, how you want
Practices and Solutions
Progressively replace Virtual Services with ‘live’ ones…Progressively replace Virtual Services with ‘live’ ones…
Actual Service/App
Virtual Service/App
• Service Virtualization is an enabler for continuous testing
• Services, applications, and systems are introduced into the
continuous integration cycle in a prioritized, controlled fashion.
• Controlled integration testing helps isolate defects for faster
resolution.
14

16. Automated Integration Testing & Virtualization
An Example
Authorization
system
IFX over
TCPIP
ISO8583 over TCPIP
SOAP over HTTPS
Rational Test Workbench /
Rational Test Virtualization Server
ATM
Virtual authorization
system
ATMATMATM
15
VISA
Practices and Solutions
System
Enterprise Payment
System
WebSphere MQ
ATM device driver (Java)
ISO8583 over TCPIP
SOAP over HTTPS
Enterprise Payment
System
WebSphere MQ
ATM device driver (Java)
+
Legend:

17. • Automated, consistent deployments
• With rollbacks of application
deployments
• Orchestration of changes across
servers, tiers and components
• Configuration and security differences
between environments
Deployment by using UrbanCode Deploy
• Multi-Tier Application Models
• Easy Process Designer
• Track what is where:
Inventory
• Scalable Distributed
Automation
• Quality Gates and Approvals
• Integrations Replace Custom
Scripting
Key Features
16
Deployment Automation
Practices and Solutions
Automated
Integration Testing
Service
Virtualization
Deployment
Automation
Continuous
Testing

18. ;;
Service Virtualization with IBM UrbanCode Deploy
Facilitates rapid creation of environment / applications from patterns
Mobile App
Middleware
Backend Systems
of Record
Automate your multi-platform
deployments
Practices and Solutions
QA
SIT
UAT
PROD
IBM UrbanCode Deploy
17

19. Best Practices
18

20. Do not wait for the UI to
automate your Tests
Test early and often
Leverage single functional
tests in (regression) test
suites and in performance
tests
Create Virtual Services to
enable full functional
testing even without all
back-end Service Providers
running ‘live’
Automated Integration Testing Best Practices
19
Best Practices
Automated
Integration Testing
Service
Virtualization
Deployment
Automation
Continuous
Testing

21. Service Virtualization Best Practices
Consumer-centric rather
than Producer-centric
Governance: choose your
Organizational Model
• Access to all
• Tools Maker/specialist
• CoE
• Federated CoE
Virtual Services at tier N
• Service providers at deeper
tiers tend to be smaller,
simpler, more stable
• Virtual Services with more
“behaviors”, change more
often
Make Virtual Services
small/self-contained and
data-driven when possible
Less Logic is more
• The more logic in a Virtual
Service, the more it starts to
resemble an application itself
• Remember Rule #1 of
software development
• “All Software Has Bugs”
You define reality: more
powerful negative testing
• The mainframe looks like it’s
down!
• Services that respond in 15
minutes…an hour… a day
• Return invalid output – or test
using invalid inputs
20
Best Practices
Automated
Integration Testing
Service
Virtualization
Deployment
Automation
Continuous
Testing

22. Automate everything
Tie deployments to the rest
of the lifecycle
Provide visibility into the
deployment process
Consider enabling
technologies like “cloud”
Deployment Automation Best Practices
21
Best Practices
Automated
Integration Testing
Service
Virtualization
Deployment
Automation
Continuous
Testing

23. Continuous
Testing
Test
Automation
Security
Mobile
IBM’s continuous testing solution
Continuous Testing requires a strong platform
22
IBM Rational
Test Virtualization
Server
IBM Rational
Test Virtualization
Server
IBM InfoSphere
Optim Test Data Management
IBM Rational
Quality Manager
IBM Rational
Quality Manager
IBM Rational
Performance
Test Server
IBM Rational
Performance
Test Server
IBM Mobile
Quality Assurance
Perfecto Mobile
IBM Mobile
Quality Assurance
Perfecto Mobile
IBM Security
AppScan
IBM Security
AppScan
IBM Rational
Test Workbench
IBM BPM
IBM Rational
Test Workbench
IBM BPM
IBM UrbanCode DeployIBM UrbanCode Deploy
Test Data
Management
Quality
Management
Environment
Provisioning
Service
Virtualization
Performance
Best Practices

24. A CT workflow engine and enterprise dashboard
Significant reduction opportunities in cycle time and effort
23
Environment
Verification
Test Execution
& Service
Virtualization
Environment
Provisioning
Defect
Management
Intelligent testing
Notify
Data
setup
Create
defect
enterprise
dashboard
workflow
engine
Best Practices

25. Intelligent Testing within CT
On a scheduled basis, CT identifies modified code within a test environment by using a client’s
version control tool combined with structured content searches of design documentation– with
the objective of down selecting regression test cases for execution. This allows daily verification
of modified code against a clients test automation suite.
VALUE
• Reduce time to analyze
changed components
• Reduce cycle time
• Automatic application
validation
Source Code tool
ACTIVITIES CONNECTS TO
.
Design
documents
Environment
Data
Execution
Report
Defect
Application
PREPARE
ANALYZE
• Analyze design documents corresponding to changes
• Identify associated System test case
• Down select key Regression Test Cases using tags
text match
EXECUTE
Environment
Data
Execution
Report
Defect
Application
• Initiate test of the selected Regression test cases
• Create defect on failed tests
• Schedule Intelligent Test Execution
• Identify changed code from source control tools
24
Test
Management
tool
Create
defect
Best Practices

26. Defect Management with CT
Defect triage will be guided by CT and change defect priority for blocking
defects, send timed reports based on status change and kick off a retest of any
fixed defect with an associated automated test case
REPORT
EXECUTE
• Send a timed report for aged defects
• Send a timed report for Deferred, Rejected or
duplicate defect
• Initiate Retest of any defect
in “Retest Ready”
ACTIVITIES
.
Environment
Data
Execution
Report
Defect
Application
CONNECTS TO
Test
Automation
tool
Test Management
Tool
Test Management
tool
ANALYZE
Environment
Data
Execution
Report
Defect
Application
1. Regularly identify defect status change
2. Change priority value for blocking defects
Best Practices
25

27. Business objectives
• Accelerate test delivery
• Increase efficiency
• Reduce delivery risk
Automation
Virtualization
Analytics
• Online service catalogue for automated test suites
• Dashboard and analytics with automated production
of test reports
• End-to-end regression test suite fully automated and
interlocked with environment, reporting and defect
management workflows
• Environment provisioning and virtualization services
to produce test labs on time and improve productivity
IBM solution
Delivered outcomes
0
10
20
30
40
50
60
70
80
90
Baseline Effort in
hrs
HATc Effort in hrs
40% effort reduction
in test execution times
50% effort reduction
in report preparation times
50% reduction*
in test environment provisioning
Provisioning is one simple step vs.
40+ hours
Continuous Testing Case Study
In 2014, a Global Insurance firm was provided with Continuous Testing service from IBM
26
Best Practices

28. LOB LOBLOB
Dev & Test Enterprise
Location BLocation B
LOB LOBLOB
Test Delivery Manager
Test Agile Manager
Test Architect
Location ALocation A
Test Enterprise
Core Test Delivery
Flex
Input Output
Test Delivery Manager
Test Planner
Test Infra Environment
Test Architects / Performance
Test Analysts
Test Management
Test Architects
Test SMEs
Test Analysts
Test Delivery Manager
Test Architect
Location BLocation BLocation ALocation A
Transform
Organization Transformation:
A DevOps approach requires a realignment of skills and resources
Specialist Teams
* 2015 European Bank
FunctionalStrategicOperational
Test Management
Flex
Test Architects
Test SMEs
Test Analysts
Core Test Delivery
Test Manager
Test Planner
Test Analysts
API Developer
Input Output
Virtualization / Automation
Virtualization Lead
Virtualization Specialist
Workflow specialist
Performance / Automation
Test Environment
Cloud Architect
Cloud Specialist
Test Infra Environment
TDM Specialist
Mobile Test
Mobile Test Architect / Leader
Mobile Test Specialist
Mobile Test Automator
FunctionalStrategicOperational
Best Practices
27

29. Scenario Discussion
After adopting DevOps and
Continuous Testing
Workshop free discussion period
28

30. DevOps
Continuous
Testing
Monday: Mike requests
CT to build Test
Environment and deploy
the application. 1 hour
later it is ready and
automatically validated
by CT. Virtual Services
are deployed (for
unavailable back-ends).
Wednesday: CT schedules re-
tests related to the
fixed code from the
previous day’s defects.
Test
Environment
delivered
Tuesday - Thursday :
CT automatically
issues reports to core
stakeholders and the
Defect Manager who
works proactively via
CT with Mike’s Dev
Team.
Tuesday: Susan monitors
CT which is running
automated Tests against
the Core Banking Solution.
CT automatically raises 20
defects against the Release
Costs
Productivity
A solution with CT with real benefits
Friday: Test
Execution is
completes
ahead of time
Friday: Stakeholder review based on CT
automatically issued reports
Monday: Core
Banking Release is
ready for
deployment with
zero High-severity
defects
29
The Benefits….
Avoidance of
business impact
More reliable
applications
Proactive problem
prevention
More effective and
quicker testing
Less re-work
More efficient and
happier workers
Lower operational
costs and improved
top-line

31. Open Discussion
What Quality and Complexity issues are you facing?
Are they worse?
Not as severe?
Describe the issues you are facing not touched upon today?
We would love to hear about new aspects
where DevOps/CT might help.
So…. what is working today for you?
30
Scenario discussion

32. Actions today
31
Identify area of significant benefit and engage IBM
Let’s meet one on one for a more focused discussion
Talk to Andrew about our Continuous Testing Service
Arrange IBM to deliver a Proof of Technology
Contact Info:
Bill Hudacek hudacek@us.ibm.com
Vinit Kutty vinit@us.ibm.com
Andrew Williams Andrew.Williams@au1.ibm.com

33. Notices and Disclaimers
Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or
transmitted in any form without written permission from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been
reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM
shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY,
EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF
THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT
OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the
agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without
notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are
presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual
performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products,
programs or services available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not
necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither
intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal
counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s
business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or
represent or warrant that its services or products will ensure that the customer is in compliance with any law.
32

34. Notices and Disclaimers (con’t)
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products in connection with this
publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM
products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to
interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED,
INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any
IBM patents, copyrights, trademarks or other intellectual property right.
• IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live, CICS, Clearcase, DOORS®, Enterprise Document
Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand,
ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™,
PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®,
pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®,
urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of
International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and
service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on
the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
33

35. Thank You
Your Feedback is
Important!
Access the InterConnect 2015
Conference CONNECT Attendee
Portal to complete your session
surveys from your smartphone,
laptop or conference kiosk.
34