How to Test the New Fashioned Way

How to Test the New-Fashioned Way
Chris Kraus
DevOps: Continuous Delivery
CA Technologies
Product Manager
DO4X80E
@ChrisKraus3
#CAWorld

2 © 2015 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
For Informational Purposes Only
Terms of this Presentation
© 2015 CA. All rights reserved. All trademarks referenced herein belong to their respective companies. The presentation provided at CA
World 2015 is intended for information purposes only and does not form any type of warranty. Some of the specific slides with customer
references relate to customer's specific use and experience of CA products and solutions so actual results may vary.
Certain information in this presentation may outline CA’s general product direction. This presentation shall not serve to (i) affect the rights
and/or obligations of CA or its licensees under any existing or future license agreement or services agreement relating to any CA software
product; or (ii) amend any product documentation or specifications for any CA software product. This presentation is based on current
information and resource allocations as of November 18, 2015, and is subject to change or withdrawal by CA at any time without notice. The
development, release and timing of any features or functionality described in this presentation remain at CA’s sole discretion.
Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product release referenced in
this presentation, CA may make such release available to new licensees in the form of a regularly scheduled major product release. Such
release may be made available to licensees of the product who are active subscribers to CA maintenance and support, on a when and if-
available basis. The information in this presentation is not deemed to be incorporated into any contract.

Abstract
Manual testing got you down? It's time to step into the
application economy and get rid of slow, sloppy and
ineffective testing methods. Learn from this introductory
presentation what's new and how CA Application Test can
expand test scenarios and allow reuse of test assets to
catch defects earlier in the SDLC. Learn how to build
portable, executable test cases that are easy to extend,
modify and maintain.
Chris Kraus
CA Technologies
How to Test the New
Fashioned Way

Agenda
HOW HAVE APPLICATIONS AND TESTING APPROACHES CHANGED?
WHY MOVE THE TESTS CLOSER TO THE APPLICATIONS
DON’T FORGET TO INVOKE AND VERIFY
SHIFT TESTING TO THE LEFT
1
2
3
4

3270/CICS Data
Streaming Testing
User-interface
Oriented Testing
Presentation Testing
vs. Backend Testing
Device Testing
vs. API Testing
Mainframe Client Server Apps 3 Tier/N Tier
SOA Apps
Mobile Apps
Customer Facing
Apps
Testing has Evolved With Development
1980 1985 1990 1995 2000 2005 2010 2015 2020

Remember When…
 Mainframe testing was simple
– UI, calculations, and persistence
all in 1 layer
– All errors reported back to the
3270 Screen
– Trained employees were the
users of application

Then…
 Client server testing got more complicated,
we moved CUI to GUI
– UI and calculations inside the GUI
– RDA (Remote Data Access) means the
persistence was external
– Most business logic was in GUI so testing
GUI ran logic (sans. Stored procedures)
– Employees were trained on how to click
and select
– Errors were still presented in the UI –
better business errors presented

Now... We Start with the Layers – 3-Tier, N-Tier, to SOA
 Originally HTML web pages provided simplistic markup of static
and dynamic data
 The logic of the application was decoupled from the presentation layer
 Presentation testing was look and feel – users struggled to test
the business logic
 SOAP and SOA services meant we had to change to testing data
on the wire – no UI to test
 Introduction of hiding error messages – a clever response
“sorry something went wrong”
 Load testing of applications became an exercise in putting
data on the wire

Now… Mobile Applications and the Consumer Rule
 Mobile applications have created the shift of the user –
users are no longer trained on the application
 Mobile applications are similar to client servers there
is business logic within the mobile app
 RDA was replaced with services and API’s, so business logic
in backend systems must be tested separate of the device
 The layers of the application are blurring - Enterprise Service Buss
 A new approach is needed to test API’s to ensure function
and performance
 The application running on the device needs to be treated
as a SUT (System Under Test)
 100’s of testing options may have to be created to test
mobile devices
ESB

Why Move the Tests Closer to the Applications?
 Moving the test closer to the code it is validating
enables you to:
– Distribute the tests across the architecture
– Remove layers of complexity by testing as close to
logic as possible
– Start functional testing before development is complete
– Decompose big response times into little pieces

Key Considerations When Looking at the Layer
 What business rules and validation
is at the Mobile layer?
 Are the API/Service end points
changing data?
 ESB all the data on the wire?
 Backend systems –
what is old is new or is it new?
ESB

Considerations for Mobile and HTML5 Layers
 Need to validate the data on the screen for:
– Formatting
– Alignment – visual representation
– Input validation
– Enabling functions - business rules X and Y
but not Z
 Performance testing – how is the code
on the device performing
 Traditionally brittle small changes in UI’s
cause test case maintenance
ESB

Considerations for API and Service Delivery Layers
 Where’s the UI? We need a new approach!
– SOAP XML
– REST JSON
– HTTP & Message Queue’s
 Testing orchestration of multiple service calls
to accomplish a requirement
 API calls are less brittle than UI testing
 Nonfunctional load testing to understand
composite response times
ESB

Enterprise Service Buss – it is all the Wires in the House
 Always the simple central point of blame
 Hardest system to test
– Data on the wire is needed to test – stimulate the system
– Need specific dependent data to test the transformations
– simulations to match required transformations
– Thousands of combinations of data
 Small changes to contracts consistently requested
 Shared services mean small changes can break multiple
consumers
 Changes in performance break everyone
ESB

Systems of Record – What’s Old Is New and More New
 Business functionality turns into composite services or API –
service decomposition becomes important
 Legacy systems behind the ESB have there own set
of testing challenges
– Data on the wire is needed to test – it may not be pretty
– Cobol Copybooks, CSV and EJB’s custom layouts
– Non human readable data – hello java object or CORBA
– Business logic and persistence are combined functionality
 Building out regress for undocumented system
 Failures this low in the stack can be gobbled up
by a asynchronous process
ESB

So Many Layers – Where Do I Automate?
 Everywhere you can!
And YES there are some exceptions:
– Look and feel is just hard –
you need eyes on the glass
– Graphical diff’s of images is not great
but okay
 CA Application Test
– Applications on mobile devices
– Browser based UI’s on PC’s and services
– Modern API’s and services
– Legacy technologies
ESB

CA Application Test for Mobile and UI’s
 Write your test once and run anywhere
– Locally attached devices
– Cloud attached devices
– Local and remote simulators
 Focus on testing the specific code
in the mobile program
– Enable/disable buttons, sliders, input
– Formatting of data
– Navigation
– Application to application dependencies
ESB

CA Application Test Testing the Services
 Service testing is all about the data on the wire
– SOAP XML
– REST JSON
– HTTP POST and GETS
– Message Queue’s with XML
 There is no UI so you need CA Application Test
to put data on the wire and get it off
 One test can be used for functional,
regression and non-functional
ESB

CA Application Test Supporting the Complexity of ESB
 CA Application Test support industry standards
and custom protocols for ESB
– webMethods Broker
– webMethods IS
– Tibco Activities
– Tibco EMS
– MQ Series and JMS providers
 The secret sauce is CA Service Virtualization
to stabilize the environment
 Regression is a huge problem – test generation
via CA Continuous Application Insight creates
the volume of tests needed with velocity
ESB

CA Application Test – Good Thing it was Born in 1999
 System behind the ESB may be modern stateless
services or not
 Or the backend systems are the museum
of technology
– Databases
– Java objects, Corba, EJB’s
– MQ Series
– FTP files, read files, check directories
– Non XML data formats like Swift, FIX, Fin, EDI, CSV,
Cobol Copybook
ESB

Invoke and Verify With a Test
 Don’t trust the response
– Transaction path can traverse
multiple boxes
– Asynchronous processes that happens
fast enough looks synchronous
– Never trust the System of Record
is updated until you verify that the
update happened
– CA Continuous Application Insight
automatically generates Database
Validation to ensure SOR has
been updated
ESB

Why “shift left”? Discover Defects Earlier in SDLC
Find
defects
here
Not
here
UNIT SYSTEM PRODUCTIONUATINTEGRATION PERFORMANCE STAGING
$977
$2,517
$4,057
$5,596
$7,136
$14,272
$-
$2,000
$4,000
$6,000
$8,000
$10,000
$12,000
$14,000
$16,000
1 2 3 4 5 6
Source: Lyon, Dan; Systems Engineering: An Essential Role for Cost-Effective Development of Secure Products.

How Does the Secret Sauce Help - “Shift left” the SDLC
Without SV
WITH SV
Dev1
Dev2
Dev3
System Test
Integration
Performance Test
UAT
SystemDev
1
Dev
2
Dev
1
Dev
3
Dev
2
Dev
3 Integration Performance UAT…
Uncertain delivery schedule – defects persist until UAT
Typical composite app
today waits for whole
assembly to beginWith CA Service Virtualization quality
effort moved earlier in the lifecycle
Reduction in SDLC
Faster Delivery
“Went from finding 40% of our defects in the 3rd
and final stage of testing to often finding ZERO.
Service Virtualization turned this final test phase
into a dress rehearsal for production.”
- AutoTrader http://bit.ly/AutoTrader-SV

Q & A

Recommended Sessions
SESSION # TITLE DATE/TIME
DO3T06S
Vision & Roadmap: The CA Roadmap for Service
Virtualization and the Dev and Test Portfolio
11/18/2015 at 2:00 pm
DO3X118S
The Why, Where and How of Service Virtualization
Adoption
11/19/2015 at 2:00 pm
DO3X114S
Freeing the World from Slow: How Service Virtualization
and the Concept of S.P.E.E.D.
11/19/2015 at 3:45 pm

Must See Demos
Application
Test Demo
Create Better
Application Tests
Theater 4
DevOps Sim
DevOps Simulation
Experience
Theater 3
Continuous
App Insight
Automate Tests for
Better Apps
Theater 4
Service
Virtualization
Deliver Better Apps
Faster with SV
Theater 3

Follow On Conversations At…
Smart Bar
Theaters 3 & 4
Tech Talks
Theaters 3 & 4

For More Information
To learn more, please visit:
http://cainc.to/Nv2VOe
CA World ’15

How to Test the New Fashioned Way

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How to Test the New Fashioned Way

Similar to How to Test the New Fashioned Way (20)

More from CA Technologies

More from CA Technologies (20)

Recently uploaded

Recently uploaded (20)

How to Test the New Fashioned Way