This document discusses an introduction to software performance engineering (SPE) presented by Steve Feldman, Tim Moore, and Ryan O'Neil from Blackboard's Performance Center of Excellence. It provides an overview of SPE methodology, which takes a systematic, quantitative approach to constructing software to meet performance objectives. The 7-step SPE process involves assessing performance risk, identifying critical use cases, selecting key performance scenarios, establishing performance objectives, constructing performance models, determining software resource requirements, and adding system resource requirements. The presentation also demonstrates applying SPE to develop a sample Blackboard building block called "Santa's Little Helper" for monitoring performance metrics.

1. 15-Nov-07
Introduction to Software
Performance Engineering
Blackboard Performance Center of Excellence
Steve Feldman, Tim Moore and Ryan O’Neil
perfeng@blackboard.com
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

2. Any statements in this presentation about future
expectations, plans and prospects for the Company, including
statements about the Company, the Building Blocks Program
and other statements containing the words “believes,”
“anticipates,” “plans,” “expects,” “will,” and similar
expressions, constitute forward-looking statements within
the meaning of The Private Securities Litigation Reform Act
of 1995. Actual results may differ materially from those
indicated by such forward-looking statements as a result of
various important factors, including: product development,
and other factors discussed in our Registration Statement
filed on Form S-1 with the SEC. In addition, the forward-
looking statements included in this press release represent
the Company’s views as of July 26, 2004. The Company
anticipates that subsequent events and developments will
cause the Company’s views to change. However, while the
Company may elect to update these forward-looking
statements at some point in the future, the Company
specifically disclaims any obligation to do so. These forward-
looking statements should not be relied upon as representing
the Company’s views as of any date subsequent to July 26,
2004.
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

3. Read the Fine Print
Any statements in this presentation about future expectations, plans and prospects for the Company, including statements about the Company, the Building Blocks Program and other
Any statements in this presentation about future expectations, plans and prospects for the Company, including statements about the Company, the Building Blocks Program and other
statements containing the words “believes,” “anticipates,” “plans,” “expects,” “will,” and similar expressions, constitute forward-looking statements within the meaning of The Private Securities
statements containing the words “believes,” “anticipates,” “plans,” “expects,” “will,” and similar expressions, constitute forward-looking statements within the meaning of The Private Securities
Litigation Reform Act of 1995. Actual results may differ materially from those indicated by such forward-looking statements as a result of various important factors, including: product
Litigation Reform Act of 1995. Actual results may differ materially from those indicated by such forward-looking statements as a result of various important factors, including: product
development, and other factors discussed in our Registration Statement filed on Form S-1 with the SEC. In addition, the forward-looking statements included in this press release represent the
development, and other factors discussed in our Registration Statement filed on Form S-1 with the SEC. In addition, the forward-looking statements included in this press release represent the
Company’s views as of July 26, 2004. The Company anticipates that subsequent events and developments will cause the Company’s views to change. However, while the Company may elect
Company’s views as of July 26, 2004. The Company anticipates that subsequent events and developments will cause the Company’s views to change. However, while the Company may elect
to update these forward-looking statements at some point in the future, the Company specifically disclaims any obligation to do so. These forward-looking statements should not be relied upon
to update these forward-looking statements at some point in the future, the Company specifically disclaims any obligation to do so. These forward-looking statements should not be relied upon
as representing the Company’s views as of any date subsequent to July 26, 2004.
as representing the Company’s views as of any date subsequent to July 26, 2004.
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

4. Purpose of Today’s Session
Introduction to SPE (5 Minutes)
SPE Methodology (15 Minutes)
Applying SPE to Building Blocks (5 Minutes)
Performance Analysis Tools and Techniques (5
Minutes)
Demonstration of Building Block (10 minutes)
Answer some questions… (10 minutes)
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

5. Introduction to SPE
Q: What is software performance?
A: Any characteristic of a software product that you
could quantifiably measure?
Q: Why is software performance important?
A: The world we live in today is becoming more digitally
sophisticated. We expect our digital transactions to be
faster then our paper transactions.
Q: How do we manage performance in the SDLC?
A: Plan, Predict, Prove, Improve performance
throughout the SDLC?
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

6. Introduction to SPE
Performance is at the top of everyone’s mind.
If Performance is poor, software adoption will decline
and potentially cause usage attrition or adoption of
alternative products/processes.
Performance Failures cost institutions a large amount
of unplanned money.
Performance problems can create a huge trust
barrier between user and provider of technology.
So then what is SPE?
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

7. Introduction to SPE
SPE is a methodology that…
Provides a systematic, quantitative approach to
constructing software systems that meet performance
objectives.
Provides a software-oriented approach to
architecture, design and implementation choices.
Prescribes principles and performance patterns for
creating responsive software.
Performance antipatterns for recognizing and correcting common problems,
the data required for evaluation, procedures for obtaining performance
specifications, and guidelines for the types of evaluation to be conducted at
each development stage.
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

8. SPE Methodology
SPE is a 7 Step Cyclical Process…
Assess Performance Risk
Identify Critical Use Cases
Select Key Performance Scenarios
Establish Performance Objectives
Construct Performance Models
Determine Software Resource Requirements
Add System Resource Requirements
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

9. SPE Methodology
Assess Performance Risk
Distinguish between new development and refactor.
Work with requirements specialists to understand
the problem/domain issues for the work.
Understand potential inter- versus intra-system
integration.
Apply common sense and basic business logic from
past experiences and similar development projects.
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

10. SPE Methodology
Identify Critical Use Cases
Most important operations/actions in the feature or
system.
Responsiveness driven.
Risk driven.
You look for use cases where there is a risk that, if performance goals are
not met, the system will fail or be less than successful.
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

11. SPE Methodology
Select Key Performance Scenarios
Unlikely that all critical use cases will be important to
performance.
The key performance scenarios are those that are
executed frequently, or those that are critical to the
perceived performance of the system.
Each performance scenario corresponds to a
workload.
Performance scenarios represented through sequence
diagrams augmented with some useful extensions.
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

12. SPE Methodology
Establish Performance Objectives
Identify and define performance objectives.
Specify the quantitative criteria for evaluating the performance
characteristics of the system under development.
Response time, throughput, or constraints on resource usage
Identify and define workload objectives.
Specify the level of usage for the scenario.
They are specified as an arrival rate (e.g., number of Web site hits per
hour), number of concurrent users or number of parallel transactions.
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

13. SPE Methodology
Construct Performance Models
Use of Execution Graphs to represent software
processing steps in the performance model.
The sequence diagram representations of the key
performance scenarios are translated into execution
graphs.
Reverse Engineer Reverse Engineer Reverse
Engineer
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

14. SPE Methodology
Determine Software Resource Requirements
Understand at an atomic level the types of software resource
requirements.
Method Calls
SQL Executions
Data Loading
Caching
Understand the effects of this software to inter- and intra-
system components.
Understand at best- and worst-case perspective.
90% of the time
Peak time.
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

15. SPE Methodology
Add System Resource Requirements
Apply Workload Understanding
Based on findings in the software resource
requirements phase you should be able to understand at
an atomic level...
CPU Cycles consumed by the method call and/or sql operation.
Network overhead (packet transfer)
Memory requirements
Processing Thread/DB Connection Requirements
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

16. Applying SPE to Blocks
We chose to create our own building block for this demo…
Santa’s Little Helper
B2 that provides performance/capacity metrics of sub-system
usage within Blackboard.
Parses an Apache access.log file
Generate a graphical display of the following metrics
– Sub-System Breakdown (Primary Graphical Requirement)
– Hits Per Day in terms of Visits (Secondary Graphical Requirement)
– Average Length Per Visit (Nice to Have Graphical Requirement)
Generate a textual summary of the following metrics
– Sub-System Breakdown (Primary Graphical Requirement)
– Top 50 Click Paths (Primary Graphical Requirement)
– Hits Per Day in terms of Visits (Secondary Graphical Requirement)
– Average Length Per Visit (Nice to Have Graphical Requirement)
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

17. Applying SPE to Blocks
SPE Phase Application of B2 to SPE
Assess Performance Risk Relatively low use case, maybe executed by 1 person on a system a
handful of times per day or week. No major database activity,
performed in isolation on the application server.
Identify Critical Use Cases 1. Textual Report (Default Report) of Sub-System Breakdown.
2. Textual Report of Hits Per Day.
Select Key Performance Scenarios Limited only to administrator role access to this sub-system. Expected
to be executed daily.
Establish Performance Objectives 1. Ability to Parse up to 10mb in size. (Manage via a log rotation
parameter)
2. 1mb file processed in 90 seconds.
3. Conversion of text into graphic 10 seconds or less.
Construct Performance Models Sequence Diagram Attached
Determine Software Resource Requirements Review Profiling Metrics
Add System Resource Requirements Report from Grinder and Study System Resource Usage
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

18. Applying SPE to Blocks
Example
Sequence/Workflow
Diagram
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

19. SPE Tools and Techniques
Load Induced Tools for Simulation
Java Profiling Interface to plug into the JVMPI API.
SQL Profiler
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

20. SPE Tools and Techniques
Load Induced Tools for Simulation
Commercially Available Tools
Mercury LoadRunner (Primary Tool of BB Perfeng)
Seque Silk Performer
Freeware Tools
The Grinder (Primary Tool of BB Dev)
Volano
Countless others on Sourceforge
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

21. SPE Tools and Techniques
Java Profilers
Commercially Available Tools
JProbe (Used by BB Perfeng)
Performasure
OptimizeIt
Wily Enterprise Application Analyzer
HyPerformix
Freeware Tools
HPJmeter
JProbe Light (Used by BB Dev)
Eclipse Profiler (Used by BB Dev)
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

22. SPE Tools and Techniques
SQL Profilers
Commercially Available Tools
Hotsos (Used by BB Perfeng)
IronEye SQL
Freeware Tools
Oracle SQL Analyze (Used by BB Perfeng and Dev)
SQL Profiler for MS SQL Server (Used by BB Perfeng and Dev)
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

23. Demonstration…
Here is our new
Building Block
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

24. Demonstration…
Click Here to
Study Usage
Statistics
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

25. Demonstration…
Sub-Systems are Categorized and
Distributed by Hits
Total System Hits for the Day
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

26. Demonstration…
Contents of the Grinder.Properties File
grinder.processes=1
grinder.threads=10
grinder.runs=5
grinder.duration=0
grinder.script=tests/usage_analysis.py
grinder.logDirectory=logs
grinder.useConsole=false
grinder.initialSleepTime=5000
grinder.sleepTimeFactor=1
grinder.sleepTimeVariation=1.0
grinder.logProcessStreams=true
grinder.recordTime=true
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

27. Demonstration…
Example Grinder .py file
for Load Testing
Santa’s Little Helper
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

28. Demonstration…
Test Iterations Errors Mean Mean Response Response Transaction
Transaction Response Bytes Per Errors
Time (ms) Length Second
Test 1 50 0 494.32 36565.00 NA 0 "My
Institution"
Test 2 50 0 2547.30 43182.00 NA 0 "Admin
Panel"
Test 3 50 0 106592.2 16566.94 NA 0 "Usage
Stats"
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

29. Demonstration…
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

30. Demonstration…
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

31. Demonstration…
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

32. Demonstration…
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

33. Question and Answer…
©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.