PDQ, SOA, SLA SOA Performance is a Critical Success Factor Ralph Decker Head of Performance Serviceline 7 th May 2008

Agenda What is SOA Service Level Agreements (SLA’s) for SOA Key to SOA Performance Testing SOA Performance Testing Challenges Overcoming Challenges Methodology and Approach Discovery Test Planning Automation and Tools Test Execution Monitoring Measurement and Analysis Summary

What is SOA

Service Level Agreements (SLA’s) for SOA

Key to SOA Performance Testing

SOA Performance Testing Challenges

Overcoming Challenges

Methodology and Approach

Discovery

Test Planning

Automation and Tools

Test Execution

Monitoring

Measurement and Analysis

Summary

What is SOA Service Oriented Architecture Architectural paradigm (pattern/model) Variety of heterogeneous systems (dissimilar) Different locations and owners Web Services? SOA provides benefits in four basic categories: Reduces expensive integration Allows for more asset reuse Increased business agility Reduces business risk

Service Oriented Architecture

Architectural paradigm (pattern/model)

Variety of heterogeneous systems (dissimilar)

Different locations and owners

Web Services?

SOA provides benefits in four basic categories:

Reduces expensive integration

Allows for more asset reuse

Increased business agility

Reduces business risk

SLAs for SOA A service-level agreement (SLA) is a formal contract between a service provider and a consumer Service availability Performance Traffic levels Messages / queries per hour / minute / second Response time Rejected transactions Errors … Poor Response Time Missed SLA Noncompliance with industry and government regulations

A service-level agreement (SLA) is a formal contract between a service provider and a consumer

Service availability

Performance

Traffic levels

Messages / queries per hour / minute / second

Response time

Rejected transactions

Errors

…

Key to SOA Performance Testing The key to successful performance testing in general require: Understanding the application and the infrastructure Understanding the user/stakeholders of the application Generate accurate anticipated volumes of traffic Investigate the impact of the traffic on the application and systems under test SOA Adds Complexity to Performance Testing Wide range of technologies Many different applications and usages Different hardware / infrastructure Knowledge of the application and the technologies Replicating traffic patterns

The key to successful performance testing in general require:

Understanding the application and the infrastructure

Understanding the user/stakeholders of the application

Generate accurate anticipated volumes of traffic

Investigate the impact of the traffic on the application and systems under test

SOA Adds Complexity to Performance Testing

Wide range of technologies

Many different applications and usages

Different hardware / infrastructure

Knowledge of the application and the technologies

Replicating traffic patterns

SOA Performance Testing Challenges SOA being a distributed environment, finding right skill sets who possess in-depth knowledge of involved platforms, applications, databases and any middleware. Assign appropriately skilled team of performance engineers to the test effort with knowledge of the systems to monitor and analyze the impact of testing An in-depth understanding of the service is required for adequate testing and evaluation A significant increase in testing activities and test assets (performance testing suites that include sophisticated harnesses and stubs) will be required at a service level Predicting the future usage of services to assist with performance, load, stress, scalability Test strategy differs from traditional testing and generally has to encompass many internal political factors e.g. ownership and responsibility

SOA being a distributed environment, finding right skill sets who possess in-depth knowledge of involved platforms, applications, databases and any middleware.

Assign appropriately skilled team of performance engineers to the test effort with knowledge of the systems to monitor and analyze the impact of testing

An in-depth understanding of the service is required for adequate testing and evaluation

A significant increase in testing activities and test assets (performance testing suites that include sophisticated harnesses and stubs) will be required at a service level

Predicting the future usage of services to assist with performance, load, stress, scalability

Test strategy differs from traditional testing and generally has to encompass many internal political factors e.g. ownership and responsibility

Overcoming Challenges To simplify performance testing for SOA applications break them down into the smallest components possible: Individual Service Systems Databases Technology Protocols Messaging Functionality Evaluate and analyze the performance of individual services based on components

To simplify performance testing for SOA applications break them down into the smallest components possible:

Individual Service

Systems

Databases

Technology

Protocols

Messaging

Functionality

Evaluate and analyze the performance of individual services based on components

Methodology and Approach Discovery: Narrow the testing event to the smallest element/service and understand the transaction, application service and the environment/systems Test Plan: Document the testing approach and the expected deliverables Automation: Develop automation to replicate transactions Test Execution: Conduct testing generating traffic increasing the traffic to pre-defined levels Monitoring: Monitor the response time for the requests sent under varying traffic levels and the impact of the traffic on the application and infrastructure under varying traffic levels Measurement and Analysis: Analyze the traffic patters to with the traffic/load

Discovery: Narrow the testing event to the smallest element/service and understand the transaction, application service and the environment/systems

Test Plan: Document the testing approach and the expected deliverables

Automation: Develop automation to replicate transactions

Test Execution: Conduct testing generating traffic increasing the traffic to pre-defined levels

Monitoring: Monitor the response time for the requests sent under varying traffic levels and the impact of the traffic on the application and infrastructure under varying traffic levels

Measurement and Analysis: Analyze the traffic patters to with the traffic/load

Discovery Identify the services and components to be included as part of the testing efforts Identify lower level and/or external services called by the services under test Identify the infrastructure hosting the services to be tested Identify the messages sent to and received from the service that will be automated for generating load Narrow the “functionality” to a subset that will be used for the testing Define data that will be used and how it will be validated Define pass/fail criteria (usually response time or transactions per second) Identify skills required for the testing effort Determine SLA or load/transactions for the testing Prioritize testing

Identify the services and components to be included as part of the testing efforts

Identify lower level and/or external services called by the services under test

Identify the infrastructure hosting the services to be tested

Identify the messages sent to and received from the service that will be automated for generating load

Narrow the “functionality” to a subset that will be used for the testing

Define data that will be used and how it will be validated

Define pass/fail criteria (usually response time or transactions per second)

Identify skills required for the testing effort

Determine SLA or load/transactions for the testing

Prioritize testing

Test Planning Test Plan - Document the Following: Services and sub services Infrastructure Transaction Automation Data SLA requirements (load/transactions) Pass / fail criteria

Test Plan - Document the Following:

Services and sub services

Infrastructure

Transaction

Automation

Data

SLA requirements (load/transactions)

Pass / fail criteria

Automation and Tools Automation Subset of the functionality of the service Prepare data and mechanism to validate Tools to generate the transaction Proprietary SOA Custom Test Harness Enterprise Tools Loadrunner 9.1 Parasoft SOAtest Green Hat GH Tester Borland SilkPerformer SOA OpenSource Tools

Automation

Subset of the functionality of the service

Prepare data and mechanism to validate

Tools to generate the transaction

Proprietary SOA

Custom Test Harness

Enterprise Tools

Loadrunner 9.1

Parasoft SOAtest

Green Hat GH Tester

Borland SilkPerformer SOA

OpenSource Tools

Test Execution Load Testing (up to defined SLA transactions per second or other) Stress Testing (up to service failure) Volume Testing (introducing large amounts of data into system) Reliability Testing (high levels of load over long periods of time)

Load Testing (up to defined SLA transactions per second or other)

Stress Testing (up to service failure)

Volume Testing (introducing large amounts of data into system)

Reliability Testing (high levels of load over long periods of time)

Monitoring Load Size (transactions per second/other) Throughput Response Time Hardware OS Disk Web (for Web Services) Application specific counters Database

Load Size (transactions per second/other)

Throughput

Response Time

Hardware

OS

Disk

Web (for Web Services)

Application specific counters

Database

Measurement and Analysis Size of message Response time Throughput under load CPU memory disk Application specific counters

Size of message

Response time

Throughput under load

CPU memory disk

Application specific counters

Summary NEED 6 BULLET SUMMARY HERE

NEED 6 BULLET SUMMARY HERE

Questions Thank You

Thank You