B2conference performance 2004

797 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
797
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

B2conference performance 2004

  1. 1. 15-Nov-07 Introduction to Software Performance Engineering Blackboard Performance Center of Excellence Steve Feldman, Tim Moore and Ryan O’Neil perfeng@blackboard.com ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  2. 2. Any statements in this presentation about futureexpectations, plans and prospects for the Company, includingstatements about the Company, the Building Blocks Programand other statements containing the words “believes,”“anticipates,” “plans,” “expects,” “will,” and similarexpressions, constitute forward-looking statements withinthe meaning of The Private Securities Litigation Reform Actof 1995. Actual results may differ materially from thoseindicated by such forward-looking statements as a result ofvarious important factors, including: product development,and other factors discussed in our Registration Statementfiled on Form S-1 with the SEC. In addition, the forward-looking statements included in this press release representthe Company’s views as of July 26, 2004. The Companyanticipates that subsequent events and developments willcause the Company’s views to change. However, while theCompany may elect to update these forward-lookingstatements at some point in the future, the Companyspecifically disclaims any obligation to do so. These forward-looking statements should not be relied upon as representingthe Company’s views as of any date subsequent to July 26,2004. ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  3. 3. Read the Fine PrintAny statements in this presentation about future expectations, plans and prospects for the Company, including statements about the Company, the Building Blocks Program and otherAny statements in this presentation about future expectations, plans and prospects for the Company, including statements about the Company, the Building Blocks Program and otherstatements containing the words “believes,” “anticipates,” “plans,” “expects,” “will,” and similar expressions, constitute forward-looking statements within the meaning of The Private Securitiesstatements containing the words “believes,” “anticipates,” “plans,” “expects,” “will,” and similar expressions, constitute forward-looking statements within the meaning of The Private SecuritiesLitigation Reform Act of 1995. Actual results may differ materially from those indicated by such forward-looking statements as a result of various important factors, including: productLitigation Reform Act of 1995. Actual results may differ materially from those indicated by such forward-looking statements as a result of various important factors, including: productdevelopment, and other factors discussed in our Registration Statement filed on Form S-1 with the SEC. In addition, the forward-looking statements included in this press release represent thedevelopment, and other factors discussed in our Registration Statement filed on Form S-1 with the SEC. In addition, the forward-looking statements included in this press release represent theCompany’s views as of July 26, 2004. The Company anticipates that subsequent events and developments will cause the Company’s views to change. However, while the Company may electCompany’s views as of July 26, 2004. The Company anticipates that subsequent events and developments will cause the Company’s views to change. However, while the Company may electto update these forward-looking statements at some point in the future, the Company specifically disclaims any obligation to do so. These forward-looking statements should not be relied uponto update these forward-looking statements at some point in the future, the Company specifically disclaims any obligation to do so. These forward-looking statements should not be relied uponas representing the Company’s views as of any date subsequent to July 26, 2004.as representing the Company’s views as of any date subsequent to July 26, 2004. ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  4. 4. Purpose of Today’s Session Introduction to SPE (5 Minutes) SPE Methodology (15 Minutes) Applying SPE to Building Blocks (5 Minutes) Performance Analysis Tools and Techniques (5Minutes) Demonstration of Building Block (10 minutes) Answer some questions… (10 minutes) ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  5. 5. Introduction to SPEQ: What is software performance?A: Any characteristic of a software product that youcould quantifiably measure?Q: Why is software performance important?A: The world we live in today is becoming more digitallysophisticated. We expect our digital transactions to befaster then our paper transactions.Q: How do we manage performance in the SDLC?A: Plan, Predict, Prove, Improve performancethroughout the SDLC? ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  6. 6. Introduction to SPE Performance is at the top of everyone’s mind. If Performance is poor, software adoption will declineand potentially cause usage attrition or adoption ofalternative products/processes. Performance Failures cost institutions a large amountof unplanned money. Performance problems can create a huge trustbarrier between user and provider of technology. So then what is SPE? ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  7. 7. Introduction to SPESPE is a methodology that… Provides a systematic, quantitative approach toconstructing software systems that meet performanceobjectives. Provides a software-oriented approach toarchitecture, design and implementation choices. Prescribes principles and performance patterns forcreating responsive software. Performance antipatterns for recognizing and correcting common problems, the data required for evaluation, procedures for obtaining performance specifications, and guidelines for the types of evaluation to be conducted at each development stage. ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  8. 8. SPE MethodologySPE is a 7 Step Cyclical Process… Assess Performance Risk Identify Critical Use Cases Select Key Performance Scenarios Establish Performance Objectives Construct Performance Models Determine Software Resource Requirements Add System Resource Requirements ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  9. 9. SPE MethodologyAssess Performance Risk Distinguish between new development and refactor. Work with requirements specialists to understandthe problem/domain issues for the work. Understand potential inter- versus intra-systemintegration. Apply common sense and basic business logic frompast experiences and similar development projects. ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  10. 10. SPE MethodologyIdentify Critical Use Cases Most important operations/actions in the feature orsystem. Responsiveness driven. Risk driven. You look for use cases where there is a risk that, if performance goals are not met, the system will fail or be less than successful. ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  11. 11. SPE MethodologySelect Key Performance Scenarios Unlikely that all critical use cases will be important toperformance. The key performance scenarios are those that areexecuted frequently, or those that are critical to theperceived performance of the system. Each performance scenario corresponds to aworkload. Performance scenarios represented through sequencediagrams augmented with some useful extensions. ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  12. 12. SPE MethodologyEstablish Performance Objectives Identify and define performance objectives. Specify the quantitative criteria for evaluating the performance characteristics of the system under development. Response time, throughput, or constraints on resource usage Identify and define workload objectives. Specify the level of usage for the scenario. They are specified as an arrival rate (e.g., number of Web site hits per hour), number of concurrent users or number of parallel transactions. ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  13. 13. SPE MethodologyConstruct Performance Models Use of Execution Graphs to represent softwareprocessing steps in the performance model. The sequence diagram representations of the keyperformance scenarios are translated into executiongraphs. Reverse Engineer Reverse Engineer ReverseEngineer ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  14. 14. SPE MethodologyDetermine Software Resource Requirements Understand at an atomic level the types of software resourcerequirements. Method Calls SQL Executions Data Loading Caching Understand the effects of this software to inter- and intra-system components. Understand at best- and worst-case perspective. 90% of the time Peak time. ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  15. 15. SPE MethodologyAdd System Resource Requirements Apply Workload Understanding Based on findings in the software resourcerequirements phase you should be able to understand atan atomic level... CPU Cycles consumed by the method call and/or sql operation. Network overhead (packet transfer) Memory requirements Processing Thread/DB Connection Requirements ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  16. 16. Applying SPE to BlocksWe chose to create our own building block for this demo…Santa’s Little Helper B2 that provides performance/capacity metrics of sub-systemusage within Blackboard. Parses an Apache access.log file Generate a graphical display of the following metrics – Sub-System Breakdown (Primary Graphical Requirement) – Hits Per Day in terms of Visits (Secondary Graphical Requirement) – Average Length Per Visit (Nice to Have Graphical Requirement) Generate a textual summary of the following metrics – Sub-System Breakdown (Primary Graphical Requirement) – Top 50 Click Paths (Primary Graphical Requirement) – Hits Per Day in terms of Visits (Secondary Graphical Requirement) – Average Length Per Visit (Nice to Have Graphical Requirement) ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  17. 17. Applying SPE to Blocks SPE Phase Application of B2 to SPEAssess Performance Risk Relatively low use case, maybe executed by 1 person on a system a handful of times per day or week. No major database activity, performed in isolation on the application server.Identify Critical Use Cases 1. Textual Report (Default Report) of Sub-System Breakdown. 2. Textual Report of Hits Per Day.Select Key Performance Scenarios Limited only to administrator role access to this sub-system. Expected to be executed daily.Establish Performance Objectives 1. Ability to Parse up to 10mb in size. (Manage via a log rotation parameter) 2. 1mb file processed in 90 seconds. 3. Conversion of text into graphic 10 seconds or less.Construct Performance Models Sequence Diagram AttachedDetermine Software Resource Requirements Review Profiling MetricsAdd System Resource Requirements Report from Grinder and Study System Resource Usage ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  18. 18. Applying SPE to Blocks Example Sequence/Workflow Diagram ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  19. 19. SPE Tools and Techniques Load Induced Tools for Simulation Java Profiling Interface to plug into the JVMPI API. SQL Profiler ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  20. 20. SPE Tools and TechniquesLoad Induced Tools for Simulation Commercially Available Tools Mercury LoadRunner (Primary Tool of BB Perfeng) Seque Silk Performer Freeware Tools The Grinder (Primary Tool of BB Dev) Volano Countless others on Sourceforge ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  21. 21. SPE Tools and TechniquesJava Profilers Commercially Available Tools JProbe (Used by BB Perfeng) Performasure OptimizeIt Wily Enterprise Application Analyzer HyPerformix Freeware Tools HPJmeter JProbe Light (Used by BB Dev) Eclipse Profiler (Used by BB Dev) ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  22. 22. SPE Tools and TechniquesSQL Profilers Commercially Available Tools Hotsos (Used by BB Perfeng) IronEye SQL Freeware Tools Oracle SQL Analyze (Used by BB Perfeng and Dev) SQL Profiler for MS SQL Server (Used by BB Perfeng and Dev) ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  23. 23. Demonstration… Here is our new Building Block ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  24. 24. Demonstration… Click Here to Study Usage Statistics ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  25. 25. Demonstration… Sub-Systems are Categorized and Distributed by Hits Total System Hits for the Day ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  26. 26. Demonstration…Contents of the Grinder.Properties Filegrinder.processes=1grinder.threads=10grinder.runs=5grinder.duration=0grinder.script=tests/usage_analysis.pygrinder.logDirectory=logsgrinder.useConsole=falsegrinder.initialSleepTime=5000grinder.sleepTimeFactor=1grinder.sleepTimeVariation=1.0grinder.logProcessStreams=truegrinder.recordTime=true ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  27. 27. Demonstration… Example Grinder .py file for Load Testing Santa’s Little Helper ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  28. 28. Demonstration… Test Iterations Errors Mean Mean Response Response Transaction Transaction Response Bytes Per Errors Time (ms) Length SecondTest 1 50 0 494.32 36565.00 NA 0 "My Institution"Test 2 50 0 2547.30 43182.00 NA 0 "Admin Panel"Test 3 50 0 106592.2 16566.94 NA 0 "Usage Stats" ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  29. 29. Demonstration… ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  30. 30. Demonstration… ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  31. 31. Demonstration… ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  32. 32. Demonstration… ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.
  33. 33. Question and Answer… ©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

×