Organizations on their journey to Digital Transformation increasingly adopt Agile and DevOps to accelerate software development and time to market. But, how can testing fit in faster software development cycles without becoming a bottleneck?

1. Georg Thurner
© 2017 by .
DevOps Requires Continuous Testing
It’s Not Negotiable
Georg Thurner & Hemalatha Murugesan& Henrik Rexed

2. Housekeeping RULES
1. Type into the Question Panel of GoToWebinar
2. Questions will be answered at the end of the presentation
All audience members are muted during the webinar
The recordings of our webinars can be found here:
www.tricentis.com/resources

3. The Presenters
Speaker
Henrik Rexed
Performance Engineer
Tricentis
Speaker
Hemalatha Murugesan
Head of Performance Testing
& Engineering Services
Infosys
Speaker
Georg Thurner
Product Management
Tricentis

4. Agenda
Infosys will guide you through the importance of
DevOps and how to master it
Tricentis speaks about how to succeed in Continuous Testing
Neotys demonstrates the importance of Continuous
Performance Testing and Monitoring

5. Global Partnership
Innovation
center
360 degree relationship leveraging mutual synergies to deliver higher
value to our customers
Building
Accelerators
Scaling
capabilities
8
Joint Marketing
24 x 7 technical
support
Professional
Services

6. Functional & Non-Functional Testing at an Inflection point –
Continuous Testing
Cultural Shift Tectonic shift in processes Mind-set change
Organizational change management
Continuous testing is all about Outcomes
Continuous Testing
Traditional Testing

7. Fast Forward
Cultural Shift
• Customer sentiment
• End user experience
• Multi-generation end user experience – Thrust in Quality of Service
Tectonic shift
in processes
• Run time development, automate, test- deploy
• Lean process with minimal documentation
• Performance Driven Development – high digitization, high availability
• Multi-dimensioned skillset
• “T” shaped skills
• Adoption of open-source tools
Mind-set
change
• Instant gratification and appreciation
• Reachability
• Short shelf life – at times less than 24 hours day
• Continuous Performance Testing culture promotion
• Creation of blameless culture – failures and success are taken in stride

8. *Bottom Line. Test automation has not kept its promises.
80%
Manual Testing
20%
Automated Testing
UI
API
Today
The reliance on
manual testing is the
top technical challenge
in app development.
World Quality Report, 2015/16
Test automation
requires developers.
Testing Present

9. *Bottom Line. Continuous testing requires automation.
UI
API
Today Tomorrow
Exploratory Tests
Automated UI Tests
API Tests
Orchestrated Service Virtualization
80%
Manual Testing
+85%
20%
Automated Testing - key enabler for high automation rates -
Testing Future

10. 100%
0%
Sprint
Start
Sprint
End
API
UI
Start testing early.
Shift left test automation.
API Testing
Degree of
Completion
Time
Efficiency
Gain
+4x
Creation
+6x
Maintenance
+20x
Execution Future test automation must focus on API testing.

11. *voke, marketsnapshotreport on service virtualization - 2012
On average, organizations require access to 33systems for development or testing.
18Average # of systems with
unrestricted access
96% of testers have
restricted test lab access
Service Virtualization
Testing
Present
Enterprise system landscapes are alike disease gene networks.
Welcome to the tester’s hell.

12. Orchestrated Service Virtualization reduces defect costs by 38%
Unit Integration System Acceptance Production
Defect
Costs 1 3 4 6 30
Relative Defect Frequency [%]
10%
20%
30%
40%
0%
OSV
w/o OSV
Shift
Left
Identify 60% to 90% more defects at least
one step earlier in the development process.
Service Virtualization
18Average # of systems with
unrestricted access
96% of testers have
restricted test lab access
Testing
Present
*Service Virtualization – WindowintoAdvanced DevOps - 2014

13. Automation Approaches
Record & Replay
Linear Framework
Sophisticated
Framework
Model-Based
Automation
1st Generation 2nd Generation 3rd Generation
Script-Based
10%
Automation
30%
Automation
90%
Automation
Tosca Recorder

14. System
SUT
Model Test Cases
Technical Information Business Information
Extract System Model
Create Test Cases
Feed Data
Automate System
The Big Picture

15. Technical Information
Next >>
Enter Vehicle Data
Make Audi
Engine Performance [kW] 200
Number of Seats 5
Fuel Petrol
Year of Construction 2016
List Price [$] 35.000
License Plate Number B-CD 123
Annual Mileage [mi] 10.000
Usage Private
sampleapp.tricentis.com
Technical Name Value Range
-
-
1;2;3;4;5;6;7;8;9
Petrol;Diesel;Gas;Other
-
-
{CLICK};{RIGHTCLICK}
Private;Commercial
2016;2015;2014;2013
-
Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Next
Usage
Year of Construction
License Plate Number
Insurant Data
Quote Details
Product Data
Next {CLICK}
Start Date 01/03/2016;01/04/2016
Insurance Sum [$] 3Mio;7Mio;10Mio;15Mio
Payment Option Yearly;Quarterly;Monthly
Scan Application….

16. Logical Name
Enter Insurant Data
Enter Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Next
Usage
Year of Construction
License Plate Number
Create Vehicle Insurance
Value
Audi
200
5
Petrol
35.000
10.000
{CLICK}
Private
{Year}
B-CD 123
Action
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Enter Product Data
Next
Start Date
Insurance Sum [$]
Payment Option
{CLICK}
01/03/2016
7Mio
Yearly
Input
Input
Input
Input
Verify Quote Details
Price List
Insurance Sum
Gross Premium [$] 1.535,22 Verify
Select
Select
Business Information
Technical Name Value Range
Technical Information
-
-
1;2;3;4;5;6;7;8;9
Petrol;Diesel;Gas;Other
-
-
{CLICK};{RIGHTCLICK}
Private;Commercial
2015;2014;2013;2012
-
Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Next
Usage
Year of Construction
License Plate Number
Insurant Data
Quote Details
Product Data
Next {CLICK}
Start Date 01/03/2016;01/04/2016
Insurance Sum [$] 3Mio;7Mio;10Mio;15Mio
Payment Option Yearly;Quarterly;Monthly
It’s like playing Lego.

17. Next >>
Enter Vehicle Data
Make Audi
Engine Performance [kW] 200
Number of Seats 5
Fuel Petrol
Year of Construction 2016
List Price [$] 35.000
License Plate Number B-CD 123
Annual Mileage [mi] 10.000
Usage Private
sampleapp.tricentis.com
Logical Name
Enter Product Data
Enter Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Next
Usage
Year of Construction
License Plate Number
Enter Insurant Data
Verify Quote Details
Next
Start Date
Insurance Sum [$]
Payment Option
Create Vehicle Insurance
Price List
Insurance Sum
Gross Premium [$]
Value
Audi
200
Petrol
35.000
10.000
{CLICK}
Private
{Year}
B-CD 123
{CLICK}
01/03/2016
7Mio
Yearly
1.535,22
Action
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Verify
Select
Select
Business Information
5

18. Next >>
Enter Vehicle Data
Make Audi
Engine Performance [kW] 200
Number of Seats 5
Fuel Petrol
Year of Construction 2016
List Price [$] 35.000
License Plate Number B-CD 123
Annual Mileage [mi] 10.000
Usage Private
sampleapp.tricentis.com
...
<section class="step">
<div class="field">
<label class="main">Make</label>
<select id="make" name="Make">
<option value="Audi">Audi</option>
<option value="BMW">BMW</option>
<option value="Ford">Ford</option>
...
</select>
<span class="error"></span>
</div>
<div class="field">
<label class="main">Engine Performance[kW]</label>
<input id="performance" name="[kW]" type="text">
<span class="error"></span>
</div>
...
<div class="field">
<label class="main">Number of Seats</label>
<select id="nofseats" name="Number of Seats">
<option value="1">1</option>
<option value="2">2</option>
...
</select>
<span class="error"></span>
</div>
...
</section>
...
sampleapp.tricentis.com
Non Visual Interface
HTML

19. Next >>
Enter Vehicle Data
Make Audi
Engine Performance [kW] 200
Number of Seats 5
Fuel Petrol
Year of Construction 2016
List Price [$] 35.000
License Plate Number B-CD 123
Annual Mileage [mi] 10.000
Usage Private
sampleapp.tricentis.com
<Envelope xmlns = 'http://…'>
<Body>
<CalculatePrice xmlns = 'http://…'>
<Request>
<VehicleData xmlns = 'http://…'>
<Make>Audi</Make>
<Performance>200</Performance>
<Year>2016</Year>
<Seats>5</Seats>
<Fuel>Petrol</Fuel>
<ListPrice>35000</ListPrice>
<PlateNumber>B-CD 123</PlateNumber>
<Usage>Private</Usage>
<AnnualMileage>10000</AnnualMileage>
</VehicleData>
<InsurantData xmlns = 'http://…'>
</InsurantData>
<ProductData xmlns = 'http://…'>
<StartDate>01/03/2016</StartDate>
<InsuranceSum>7000000</InsuranceSum>
<PaymentOption>Yearly</PaymentOption>
</ProductData>
</Request>
</CalculatePrice>
</Body>
</Envelope>
sampleapp.tricentis.com
Non Visual Interface
Webservice

20. <Envelope xmlns = 'http://…'>
<Body>
<CalculatePrice xmlns = 'http://…'>
<Request>
<VehicleData xmlns = 'http://…'>
<Make>Audi</Make>
<Performance>200</Performance>
<Year>2016</Year>
<Seats>5</Seats>
<Fuel>Petrol</Fuel>
<ListPrice>35000</ListPrice>
<PlateNumber>B-CD 123</PlateNumber>
<Usage>Private</Usage>
<AnnualMileage>10000</AnnualMileage>
</VehicleData>
<InsurantData xmlns = 'http://…'>
</InsurantData>
<ProductData xmlns = 'http://…'>
<StartDate>01/03/2016</StartDate>
<InsuranceSum>7000000</InsuranceSum>
<PaymentOption>Yearly</PaymentOption>
</ProductData>
</Request>
</CalculatePrice>
</Body>
</Envelope>
sampleapp.tricentis.com
Non Visual Interface
Webservice
Audi
200
5
Petrol
35.000
10.000
Private
{Year}
B-CD 123
01/03/2016
7Mio
Yearly
1.535,22
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Verify
Select
Select
Business Information
API Test Case
Webservice
Enter Product Data
Enter Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Usage
Year of Construction
License Plate Number
Enter Insurant Data
Verify Quote Details
Start Date
Insurance Sum [$]
Payment Option
Price List
Insurance Sum
Gross Premium [$]
Webservice
Request
Webservice
Response

21. Logical Name
Enter Product Data
Enter Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Next
Usage
Year of Construction
License Plate Number
Enter Insurant Data
Verify Quote Details
Next
Start Date
Insurance Sum [$]
Payment Option
Create Vehicle Insurance
Price List
Insurance Sum
Gross Premium [$]
Value
Audi
200
5
Petrol
35.000
10.000
{CLICK}
Private
{Year}
B-CD 123
{CLICK}
01/03/2016
7Mio
Yearly
1.535,22
Action
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Verify
Select
Select
Business Information
UI Test Case
Browser
Audi
200
5
Petrol
35.000
10.000
Private
{Year}
B-CD 123
01/03/2016
7Mio
Yearly
1.535,22
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Verify
Select
Select
Business Information
API Test Case
Webservice
Enter Product Data
Enter Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Usage
Year of Construction
License Plate Number
Enter Insurant Data
Verify Quote Details
Start Date
Insurance Sum [$]
Payment Option
Price List
Insurance Sum
Gross Premium [$]

22. Logical Name
Enter Product Data
Enter Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Next
Usage
Year of Construction
License Plate Number
Enter Insurant Data
Verify Quote Details
Next
Start Date
Insurance Sum [$]
Payment Option
Create Vehicle Insurance
Price List
Insurance Sum
Gross Premium [$]
Value
Audi
200
5
Petrol
35.000
10.000
{CLICK}
Private
{Year}
B-CD 123
{CLICK}
01/03/2016
7Mio
Yearly
1.535,22
Action
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Verify
Select
Select
Business Information
UI Test Case
Browser
Audi
200
5
Petrol
35.000
10.000
Private
{Year}
B-CD 123
01/03/2016
7Mio
Yearly
1.535,22
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
Verify
Select
Select
Business Information
API Test Case
Webservice
Enter Product Data
Enter Vehicle Data
Make
Engine Performance [kW]
Number of Seats
Fuel
List Price [$]
Annual Mileage
Usage
Year of Construction
License Plate Number
Enter Insurant Data
Verify Quote Details
Start Date
Insurance Sum [$]
Payment Option
Price List
Insurance Sum
Gross Premium [$]
Manual Testers become Automation Specialists

23. Low
Risk
High
Risk
Medium
Risk
80%20%
Business Risk
Coverage
Test
Cases
The time needed for
testing is infinitely larger
than the time available.
Critical Limit
Risk-Based Testing
Have the right test cases.
Do the right things.

24. 100%
Manual
Testing
0%
Automated
Testing
Testing
Present
Customer Case Study
There’s a way to do it
better. Find it!
Project Charter
11
Manual
Testers
4755
Manual
Test Cases
?
Unknown
Coverage
10
Weeks
Execution

25. 48
Hours
Execution
Single Agent
8
Hours
Execution
Multiple Agents
Risk Coverage Optimization Test Data Management
Customer Case Study
11
Manual
Testers
4755
Manual
Test Cases
?
Unknown
Coverage
10
Weeks
Execution
89%
Business Risk
Coverage
1193
Manual
Test Cases
37%
Business Risk
5
Weeks
Execution
2.5
Weeks
Execution
92%
Automated
Test Cases
53%
Business Risk
Coverage
Smoke Testing
34
Minutes
Execution
Smoke Testing
Test Case Count 7%
Automation (UI & API) & OSV Continuous Integration & Distributed Execution
75% Redundancy 50% Effort Test Data
64% API Tests

26. 8
Hours
Execution
Multiple Agents
Customer Case Study
72%
Critical Defect Reduction
ProductionDefects
89%
Business Risk
Coverage
1193
Manual
Test Cases
5
Weeks
Execution
2.5
Weeks
Execution
92%
Automated
Test Cases
34
Minutes
Execution
Smoke Testing
53%
Business Risk
Coverage
Smoke Testing
Test Case Count 7%
Test Case Design & Exploratory Testing
Risk Coverage Optimization Test Data Management Automation (UI & API) & OSV Continuous Integration & Distributed Execution
64% API Tests

27. It doesn‘t end with Functional Testing…
#Load TestVictims make excuses, leaders deliver results.

28. Test Early
Early Risk
Analysis • Understand the application
• Performance needs to be included in the PRA
• ATAM ( Architecture Tradeoff Analysis Method)
Automation Smarter, Faster
Validation Based on your KPIs
Monitor Detect problems before
the end users
Loadtest

29. Understand the Application
 Performance is driven by :
 The endusers
 Third party system
 …etc
 We need to ask the right questions :
 How the users work on the system?
 What are their habits
 When ? and how often
 Are we going to expand in different geos?
 Is the marketing plan to market/promote
the application? If yes what is the type audience?
…etc
Loadtest

30. Unit Performance Testing
Any part of the system
Not a standard practice
Do not wait until the system is assembled
Test cases are simpler, fewer variables
• Test-Driven Development may be an answer
Many systems are monolithic
Third-party components
Loadtest

31. Workflow
3) Development life cycle
Component
Testing
Business
Test Case
1) Specification
2) Risk Analysis
Project Lifecycle

32. Continuous Performance Validation
Component
testing
Requirements
Daily Cycle
Performance
End-to-End testing
Sprint Assemble and Deployment
Feedback
Production
Monitoring
Performance
Qualification
Continuous Integration

33. Re-Using Functional Testing Assets

34. Using APIs To Connect To Your Toolchain

35. DevOps Toolchain
Planning
•Provide transparency to the stakeholders
Source Control
•Tracking Change of the code ( infrasctructure, application..Etc)
Configuration management
•Infrasctructure is treated exactly like code
Continuous integration
•Tools to automate the Build and the deployment
Deployment tools
•Application can be released to production any time you want in order to improve time to market
Testing and validation
•Validate the qualiity of the release
APM
•applications are commonly tested and monitored with APM tools to ensure high availability, low response time, and quality of service

36. Stage 1: Build
Build
request
Prepare
Build
Compile
Packaging

37. Stage 2: Deploy
Deploy
DEV QA UAT

38. Stage 3: Test
receive task
integration
test
DEV QA UAT
Test
Integration Functional Performance Security

39. Stage 4 : Release
Release
PROD

40. receive task
integration
test
Devops toolchain
Build
request
Deploy Release
DEV QA UAT
PROD
Test
Prepare
Build
Compile
Packaging
Integration Functional Performance Security

41. 15 60 15 90 30 60 15 60 45
15 30 15 60 15 20 15 60 30
Test Tuning
Test Tunning
What Our Customers Experience
Same Test Case with

42. Test Faster

43. Automate Test Scripts Maintenance

44. Summary
Infosys guided you through the importance of
DevOps and how to master it
Tricentis spoke about how to succeed in Continuous Testing
Neotys demonstrated the importance of Continuous
Performance Testing and Monitoring

45. Contact Us
Name: Georg Thurner
Email: g.thurner@tricentis.com
Reach Out!
© 2017 Tricentis GmbH. All rights reserved.
Name: Henrik Rexed
Email: henrik.rexed@neotys.com
Name: Hemalatha Murugesan
Email:
Hemalatha_Murugesan@infosys.com

46. Questions?
© 2017 Tricentis GmbH. All rights reserved.