Software performance engineering is becoming increasingly important to businesses as they look to improve the non-functional performance of applications and get more out of IT investments. By leveraging performance engineering techniques, IT professionals can be indispensable in building and optimizing scalable systems. This
introductory course will teach you the essentials of software
performance engineering including :
• The performance challenges faced by Enterprise IT today
• What is software performance engineering (SPE)?
• Best practices for building scalable software systems
• The approaches to integrating SPE into IT project lifecycles
• Common frameworks for measuring application performance and service levels
• The impact of SPE on software developers, testers, capacity planes,
and other IT professionals
• Case studies from the finance, retail, and insurance industries
Instructor: Walter Kuketz, SVP and CTO, Collaborative Consulting
This training is sponsored by Correlsense, Collaborative Consulting,
and New Horizons
2. Housekeeping
Presentation will last 45 Submit questions via the Slides will be made
minutes chat window available
2
2012 Collaborative Consulting
3. Purpose of today’s webinar
Review the current challenges of being a performance engineer
Overview of the five knowledge areas of Software performance engineering body of knowldge
Where to inject PE tasks and activities into the SDLC to build more performant applications
Overview of application performance management (end-to-end user experience)
Roles and career path overview
Case study – 10X event
3
2012 Collaborative Consulting
4. Performance engineering challenges today
Highly • Complex and tangled system of systems
distributed • Distributed Infrastructure
apps • Distributed production operations
• Performance engineering and testing
Distributed
• Architecture and development
teams
• Business
Evolving & • Mobile devices
changing • Third party services
application • SaaS products
architectures • Cloud
Changing • Driven by mobile anywhere
usage
• Flash traffic events
patterns
4
2012 Collaborative Consulting
5. Performance engineering challenges
User
Enterprise Budget Team skills
experience
• Release • User populations • Constant cost • Performance
schedules, mont are exploding and budget organization;
hly, quarterly pressure skillsets, multi
• Web site disciplined,
• Project based performance • Maintaining a communication,
directly impacts meaningful split team
revenue performance lab
(production-like) • Training and
career paths
5
2012 Collaborative Consulting
6. No clear standards in the market for PE
Test automation technician
Capacity planner
?
Performance tester
?
? Performance Engineer
Performance analyst
?
Performance Architect
?
Benchmarks
?
Troubleshooting
Performance Data engineer
?
6
2012 Collaborative Consulting
7. Goal of the SPE Body of Knowledge
Define the profession of performance
Define profession engineering
Establish Body of Establish a body of knowledge for
Knowldge performance engineering
Establish Establish a common language for roles,
common responsibilities, and practice areas using
language the form of Knowledge Areas
Define Define the competencies shared across
competencies the knowledge areas
Capture common Capture common techniques for sharing
techniques across the industry
7
2012 Collaborative Consulting
8. The SPE Body of Knowledge
Enterprise Software Performance Engineering
Planning, coordination, information sharing, and control
Knowledge areas
Performance
validation and testing Problem
Application
SDLC & (PVT) detection
performance
architecture and
management
(SA) Capacity planning (APM)
resolution
(PDR)
(CP)
Underlying competencies and roles for delivering
Techniques supporting the knowledge areas
8
2012 Collaborative Consulting
9. The Enterprise PE team: There must be a practice leader
A shared service and clearing house for all performance, scalability and
capacity planning information. Transforms performance information to
business knowledge and advantage.
Communicates across the teams to share information
For instance, the production workload and the
Operations PT performance test workload – are they in sync?
Architecture and Make sure the architecture team knows how the
production new release, or newest build behave in production.
The business risks are clearly identified and
Business direction communicated.
Can provide performance guidelines to the
Development teams developers, must influence the unit testing process.
9
2012 Collaborative Consulting
10. Information sharing across the knowledge areas
Enterprise PE
Capacity
planning and planning Triage
monitoring
SPE impacts Architects,
Results,
Developers, testing, and
Forecast Workload operations
Workload scalability
NFR’s Performance User Exp
Volumetrics, validation, Workload
Utilization
workload testing
SDLC, Results
Architecture Observations APM
, NFR
Growth plans
Workload
NFR’s
10
2012 Collaborative Consulting
11. PE: Knowledge areas
SA PVT CP APM PDR
Software development Performance testing and Capacity planning Application performance Problem detection and
lifecycle and architecture validation management resolution
11
2012 Collaborative Consulting
12. PE: Software development lifecycle and architecture
SA PVT CP APM PDR
Software development Performance testing and Capacity planning Application performance Problem detection and
lifecycle and architecture validation management resolution
12
2012 Collaborative Consulting
13. PE: SA (software development lifecycle)
Waterfall methodologies
Functional Performance
Define Design Develop Deploy
test test
Need to plan for Build code to performance
performance testing requirements
Define
Deploy Agile Design
methods
Test Build
Can a system be designed to support 10 TPS and 1 second response time work @ 100 TPS and 1 second?
13
2012 Collaborative Consulting
14. SDLC: Define and design
Define Design
1. Project risk scorecard 1. Technology Architecture validated
- Key business transactions to the performance Usage
identified Scenarios. How will it achieve the
- Workflow constraints key performance requirements?
Sub-second
- SLA’s and penalties response time. Define the scalability approach.
Flash marketing
2. Workload model and usage event. 2. Scalability to support flash events
scenarios
3. Environments: include any
3. Business volumes with peak
required changes to develop and
factors
test the performance usage
4. Business performance scenarios.
requirements
Communication to your downstream processes and people
Prepare to deal with flash events
Design to avoid flash events
(they are expensive. E.g. Work with business why does the outbound email
have to send email to all xM customers at the same time
14
2012 Collaborative Consulting
15. SDLC: Develop and functional testing
Functions are tested. Performance needs to be analyzed.
Develop Functional test
• The functions are mapped to software • Verify business functions and key
components. The performance req’s are performance functions.
assigned to the components.
• Verify the key performance workflows
• Unit testing for early validation of the and functions.
performance req’s for the software
components. Response time • Define and execute micro-performance
measurements. testing in the QA process.
• Integration testing include micro- • Produce report on for each key
performance testing of key functions. performance Usage Scenario with
Code review of the key performance response time and throughput.
functions.
• Enhance unit and integration test for
early validation of key performance
functions
Greatest impact on performance and scalability, but you must know the goals.
Communication to your downstream processes and people
15
2012 Collaborative Consulting
16. Performance budgets
Do your developers know how much time they have for their piece of the transaction?
Code runs
everywhere.
SERVER
WEB APP SVR
PROXY
MQ/ESB
LDAP CORBA
DCOM
Respond to the user
in 2 seconds. Web
Services
Datacenter
16
2012 Collaborative Consulting
17. PE: Performance testing and validation
SA PVT CP APM PDR
Software development Performance testing and Capacity planning Application performance Problem detection and
lifecycle and architecture validation management resolution
17
2012 Collaborative Consulting
18. PE: PTV (performance testing and validation)
What is changing? Where does it need to be validated? What (all) is being measured?
The project: Why are we testing and what are the success criteria?
Prerequisites:
Business goals and non-functional requirements are defined
Workload and scenarios defined
Environment defined (where are we testing)
Application in a stable state
Configuration management and release management ready
Core performance team identified
Supporting team committed
18
2012 Collaborative Consulting
19. Organization: Performance team value
You still need to explain to the rest of the organization what you do
Business and Early involvement
application area with the applications Test execution Environment
knowledge team
• Deep understanding of • Performance engineers • Well-understood • Testing environment is
the application being involved in design workload model and a close approximation
tested sessions characterization of production, including
database size
• Deep understanding of • Understand the nature • All components in the
the technical of the data required for test are monitored and • Differences between
architecture of the testing reported on test and production are
application clearly known
• Able to understand • A defined triage process
• Understanding of the technical gaps to meet for troubleshooting • Release and
business area requirements not configuration
• A process to manage
supported by the supported by the tool management in place
the inevitable
applications unplanned tests and and procedures
enforced
test archive exists
Performance team value
19
2012 Collaborative Consulting
20. Performance team reference model
Primary test, design, and execution team Performance engineering support skills
Planning and
Application
project
Performance architect
management
engineer
Technology
SME
Test
Results
automation
analyst Data
technician
engineering
Must determine the core team members vs. on-demand team
20
2012 Collaborative Consulting
21. Enterprise performance team
PT problem Strategy and
PT technology
resolution planning
PMO
board (architecture)**
Infrastructure**
Performance
Performance test engineering and
execution team optimization
Application
Vendor Application
support
management deployment
Vendor IBM Oracle EMC Informatica Vendor Vendor
Vendors
21
2012 Collaborative Consulting
22. Model test case execution
In the end, the test has to be run.
Prepare test
Prepare test Confirm
Select test harness online Initiate test
0 case or batch
monitoring
components
environment
readiness
case 1
messages
No
Problem
analysis and Evaluate
triage team impact of
Select test Normal test
1 case completion?
change
Yes
Yes Gather test
Analysis for Create test
results
specific report; publish
Txn RT, CPU,
DB
change? to repository 2
No
Are Categorize Can you
Implement
changes change – enter retest with Rerun the test
2 required for
Yes
change to CM
configuration
change
the
Yes
case
retest? tracking tool change?
No No
Move to next Move to next
test case 0 test scenario
22
2012 Collaborative Consulting
23. PE: Capacity planning
SA PVT CP APM PDR
Software development Performance testing and Capacity planning Application performance Problem detection and
lifecycle and architecture validation management resolution
23
2012 Collaborative Consulting
24. PE: CP (capacity planning)
The right system at the right price!
Plan for • Forecast workloads
future • Plan for usage
Analyze
• Track utilization of
existing systems current
capacity
• Track workload
Determine • Define system to
capacity support workloads
requirements • Test
• Define service
levels
Workloads drive system transactions, which drive resource utilization
24
2012 Collaborative Consulting
25. Capacity: Use of resources
The overall system has a capacity; each component has a capacity.
SERVER
WEB APP SVR
PROXY
MQ/ESB
LDAP CORBA
DCOM
Web
Services
Datacenter
25
2012 Collaborative Consulting
26. Workload model validation
Actual workload Synthetic workload
Model calibration
System System
Measured response times, Measured response times,
throughputs, utilizations, etc. throughputs, utilizations, etc.
Acceptable No
Yes
26
2012 Collaborative Consulting
27. PE: Application performance management
SA PVT CP APM PDR
Software development Performance testing and Capacity planning Application performance Problem detection and
lifecycle and architecture validation management resolution
27
2012 Collaborative Consulting
28. Five elements of APM (Gartner)
Monitoring the performance of complex distributed applications
End user experience measurement – browser/mobile
1 device
2 Create a model of the run-time environment (discovery)
Profile the performance and behavior of user-defined
3 transactions
Performance metrics from each of the
4 applications/systems technical components (Webserver,
App server, Database, etc.)
5 Application performance management database
28
2012 Collaborative Consulting
30. Track key user experience metrics
Mobile Total end-user response time
Browser rendering time
Network latency
Mobile Real User Monitoring
Rendering
30
2012 Collaborative Consulting
31. And see every hop of the transaction
Mobile
Proxy
server
Web server
App server
Data warehouse gateway
Mainframe
Database
Dynamic Transaction
Path Detection
Rendering
31
2012 Collaborative Consulting
32. PE: Problem detection and resolution
SA PVT CP APM PDR
Software development Performance testing and Capacity planning Application performance Problem detection and
lifecycle and architecture validation management resolution
32
2012 Collaborative Consulting
33. Performance issues and root-cause analysis
This leverages several knowledge areas
A solution must be found quickly (under pressure)
Performance issues are found in:
• Preproduction environment
• Production and operations environments
Key business transactions are not meeting their service level objectives:
• End user experience is degrading (acute or chronic)
• Overnight processing is taking longer
• Stability and performance issues
33
2012 Collaborative Consulting
34. When problems occur
Flash web site events (all at once)
New code released causes problems
Infrastructure upgrade
Consolidation of applications and servers
New workload model (batch and online)
34
2012 Collaborative Consulting
35. Underlying competencies
Analytical
Project Communication
thinking
management skills
problem solver
Business Monitoring and
Statistics
Knowledge tuning
Workload
Behavioral Application and
model
characteristics tool knowledge
Underlying development
competencies
Test data Test execution
Forecasting
management process
35
2012 Collaborative Consulting
36. Technical competencies
Code Deep dive Resource
Database profiling tools
utilization
monitoring tools (HP Diagnostics,
tools
CA Wily, etc, system
(Static analysis, Jprobe, etc)
diagnostics) level
End to end Comprehensive
Statistics and
performance results collection
Distributions
tools and reporting
Technical
competencies Test design Queuing
and and Test data
Little’s law
execution modeling creation
tools tools
36
2012 Collaborative Consulting
37. PE Techniques
Brainstorming Data modeling
Creating a workload model Performance testing
Triage approach Static code analysis
Logging and
System metrics collection
instrumentation
37
2012 Collaborative Consulting
38. PE roles: The team – many possible roles
Managing and developing the core team members
1 Technical Project manager
2 Performance Architect
3 Sr. Performance Engineer
4 Performance Engineer
5 Sr. Test automation technician
6 Test automation technician
7 Performance data engineer
8 Sr. Capacity planner
9 Capacity planner
10 Modeler
38
2012 Collaborative Consulting
39. Career path: Performance engineer foundation
Performance
Engineer
Grow your Performance
testing
leadership and Lead
communication technical Workload
role models
Technical Production
architecture triage
Core Additional
technical tech
competency Communication
competency
Grow your breadth of capabilities
39
2012 Collaborative Consulting
40. Case study: Preparing for a flash marketing event
Planning a large event to introduce a new product
• Send out email to all registered customers (500,000 currently), there is a link in the email.
• Run multiple TV ad’s encouraging people to register and then come to the event, results in
100,000 new registrations
• No new functionality required to support the event
• Potential 600,000 users of the web site
The business is forecasting at least a 10X increase during the
event, in terms of product sales
• How does this translate into system workload and utilization?
• Do we need more servers?
• The current system was not really designed for such a flash event
40
2012 Collaborative Consulting
41. PE: Process to prepare for the event (10X)
CP CP PVT SA
Forecast the
workload Tune application
based on results
APM PDR
Measure the current Plan and execute
workload and PT for forecast
resource utilization workload
41
2012 Collaborative Consulting
42. CP: Workload model
CP
The CP team uses this information to forecast the
new workload 10X.
Step Transaction – by product Percentage
1 Login 100
2 Search product 100
3 View product detail 100
4 Add to shopping cart 50
5 Confirm purchase method 50
6 Shipping information 50
7 Send confirmation email 50
42
2012 Collaborative Consulting
43. CP: Analyze current capacity
CP Review current Production workload, and the under lying system
utilization.
• Average workload
• Current peak workload
Response
time
Service level
100%
System utilization
43
2012 Collaborative Consulting
44. PVT: planning, executing, and analysis
PVT
• Use the workload model from CP
• Enable APM during testing
• Work with the Application Architect
Methodology is iterative and tactical
Define business
Review infrastructure
activity profiles & Design & build tests
& architecture
service levels
• Identify risk areas • Types & numbers of • Test data generation
• Review configuration users • Create test scripts
settings, topology, & sizing • Business activities & • User & transaction profiles
• Define points of frequencies • Infrastructure configuration
measurement
Iterate testing & tuning
44
2012 Collaborative Consulting
45. Questions?
Thank you!
Get your free copy
SharePath RUM today!
More information:
www.collaborative.com
info@correlsense.com
www.correlsense.com/demo
www.real-user-monitoring.com
45
2012 Collaborative Consulting
Editor's Notes
Performance is often at odds with other key quality attributes, such as flexibility, portablility, maintainability, etc.
The performance team fits inside a large context model. How does it interact with other groups?
The performance database is required to support performance analysis and analytics. This is a resource for the CP team.
You have the ability to watch/monitor every transaction for every user. What Is A Transaction?Request issued by an end-userGoes through multiple componentsEach component may be activated multiple timesNote that:Different transaction types take different paths and flowsSometimes instances of the same transaction type execute across different flows
What Is A Transaction?Request issued by an end-userGoes through multiple componentsEach component may be activated multiple timesNote that:Different transaction types take different paths and flowsSometimes instances of the same transaction type execute across different flows
The process for PDR is need for production and preproduction environments. This is required in Performance testing, critical for a key POC that demonstrates business advantage. Most POC’s experience problems along the way. Having a well defined detection and resolution process is key to success. Problems will happen and you must be ready for them, and move past them quickly.They can occur gradually or suddenly. A small change can cause big problems or disruptions. How do they manifest themselves to the end-user, or business workflow?
Flash events can wreak havoc on a system that was not designed for scaling. Database contention can come into play at much higher transaction levels. Slow memory leaks can suddenly become exposed at 10x level.Consolidation of servers or applications is a difficult tasks. Combining workloads onto one physical server that is hosting many Virtual Servers always has few unintended consequences.
How do we support the delivery of the five knowledge areas?Analytic skills are required; Decision making, Creative thinking, Learning, Problem solving, Systems thinkingBusiness knowledge – awareness of your organization and how it makes decisions (especially on budgeting)Interaction skills - Negotiation and facilitation, Leadership, Consultative/Influencing, Teamwork. Test data management – a very critical part of the Body of Knowledge.
You should correlate the TV ad to the system utilization. Did the ad drive traffic to your web site?
Each step is multiple system transactions. For instance, I have seen some login processes display too much information. In a low volume site, that might be ok. But under stress, the information for the login screen could crush the system.
Understand when the current system violates the service level. How many transactions and how far away from the Flash event goal are we? This relies on the APM tools and procedures in place. Collecting the system CPU metrics and database usage information.
Test planning, results review.Stress testing approach for the application. Most application break well before the 10X level. During the testing process, the development team will be making coding changes to help scale the application.Results reporting, detailed metrics on utilization are critical, also, often times deep diagnostics are required to find the problems.