Principles of Performance Engineering Steve Feldman, Director Performance Engineering and Architecture [email_address]
Agenda <ul><li>Questions of the Mind… </li></ul><ul><li>PE @ Bb </li></ul><ul><li>Process and Methodology </li></ul><ul><l...
Part 1: Questions of the Mind
Questions of the Mind… <ul><li>What is Performance? </li></ul><ul><li>What is Scalability? </li></ul><ul><li>What is Perfo...
What is Performance? <ul><li>Performance = Response Times </li></ul><ul><li>Response Times affect the User Experience. </l...
What is Scalability? <ul><li>Scalability = Heavy Adoption/Usage  </li></ul><ul><ul><li>If Heavy Adoption = Positive User E...
What is Performance Engineering? <ul><li>Engineering discipline with a primary focus on performance and scalability. </li>...
Why does Blackboard invest in Performance Engineering? <ul><li>Some of the world’s largest application deployments outside...
Part 2: PE @ Bb
PerfEng at Bb… <ul><li>Triangle of Priorities </li></ul><ul><ul><li>Performance: Response Times (Apdex) </li></ul></ul><ul...
Triangle of Priorities
PerfEng at Bb…
Performance: Response Times (Apdex) <ul><li>Every transaction in the application is defined as a state, an action state or...
State/Action Modeling <ul><li>State: A condition or point of reference within a sub-system in which an actor has the optio...
State/Action Modeling <ul><li>Action State: The navigation to an action.  </li></ul><ul><ul><li>For example, an instructor...
State/Action Modeling <ul><li>Action: An actor driven process that occurs within a state.  </li></ul><ul><ul><li>Actions o...
What is Apdex(Application Performance Index)? <ul><li>What is Apdex </li></ul><ul><ul><li>Apdex is an open standard develo...
Apdex and User Abandonment <ul><li>Bb defines (4) Apdex Dimensions:  </li></ul><ul><ul><li>Low (2s-8s) </li></ul></ul><ul>...
Apdex and Workload Variations <ul><li>Apdex scores are taken across each data model variation. </li></ul><ul><li>Expect sc...
Scalability: High Session Activity with Low Abandonment (PAR) <ul><li>What is a PAR? </li></ul><ul><ul><li>P erformance  A...
Scalability: High Session Activity with Low Abandonment (PAR) X-Axis: Iterations Y-Axis: Resource Utilization Resource Uti...
PAR Process: Step 1 Calibration <ul><li>Calibrate Workloads with User Abandonment </li></ul><ul><ul><li>Peak of Concurrenc...
PAR Process: Step 2 App. Saturation <ul><li>Take workloads from abandonment run and disable abandonment. </li></ul><ul><ul...
PAR Process: Step 3 DB Saturation <ul><li>Multiply the workload from Step 2 across identical application servers. </li></u...
PAR Process: Additional Steps <ul><li>Hypothesis and Proof </li></ul><ul><ul><li>Essential part in this process is determi...
User Experience: High Performance and Heavy Integration (Reference Architecture) <ul><li>Insert Visio Here </li></ul>
Investment and Relationships <ul><li>PerfEng Team (10 Team Members) </li></ul><ul><ul><li>Combined both teams as part of t...
Investment and Relationships <ul><li>Performance Lab Sponsors </li></ul><ul><ul><li>Dell: Servers and Remote Lab </li></ul...
Part 3: Process and Methodology
Process and Methodology
SPE Overview… Assess  Performance Risk <ul><li>Assessing the performance risk at the outset of the project (During Require...
Method-R: Requirements of a Good Methodology (Milsap, Carey) <ul><li>Predictive Capacity: A method must enable the analyst...
Method-R: Requirements of a Good Methodology (Milsap, Carey) <ul><li>Finiteness: A method must have a well-defined termina...
Method-R: Response Time Performance Improvement <ul><li>Practical way of thinking. </li></ul><ul><li>Often asking someone ...
Part 4: Performance Lab ‘07
What projects are coming out of the PerfEng Lab in ‘07 <ul><li>Blackboard Performance Sizing and Certification Program. </...
What is the BPSC? <ul><li>The BPSC is a benchmarking program designed to showcase the enterprise architecture, performance...
Special Projects <ul><li>Virtualization </li></ul><ul><ul><li>VMWare, Zen and LDOMs </li></ul></ul><ul><li>Management </li...
Questions?
Links/References <ul><li>Blackboard Academic Suite Hardware Sizing Guide  (Behind the Blackboard) </li></ul><ul><li>Perfor...
Merci
Upcoming SlideShare
Loading in …5
×

Sfeldman performance bb_worldemea07

1,230 views

Published on

Published in: Technology
  • An amazing tourist destination for its ancient wonders, Peru is the best tourist destination in South America Car Rental Peru: http://goo.gl/Nebq1s
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Sfeldman performance bb_worldemea07

  1. 1. Principles of Performance Engineering Steve Feldman, Director Performance Engineering and Architecture [email_address]
  2. 2. Agenda <ul><li>Questions of the Mind… </li></ul><ul><li>PE @ Bb </li></ul><ul><li>Process and Methodology </li></ul><ul><li>What projects are coming out of the PerfEng Lab in ’07 </li></ul><ul><li>Links </li></ul>
  3. 3. Part 1: Questions of the Mind
  4. 4. Questions of the Mind… <ul><li>What is Performance? </li></ul><ul><li>What is Scalability? </li></ul><ul><li>What is Performance Engineering? </li></ul><ul><li>Why does Blackboard invest in Performance Engineering? </li></ul>
  5. 5. What is Performance? <ul><li>Performance = Response Times </li></ul><ul><li>Response Times affect the User Experience. </li></ul><ul><li>If the User Experience is acceptable Abandonment becomes less likely which can positively affect Adoption. </li></ul><ul><li>When Adoption increases, greater the need for Scalability. </li></ul>
  6. 6. What is Scalability? <ul><li>Scalability = Heavy Adoption/Usage </li></ul><ul><ul><li>If Heavy Adoption = Positive User Experience </li></ul></ul><ul><ul><ul><li>If Positive User Experience = Acceptable Response Times </li></ul></ul></ul><ul><li>Patterns of predictable usage </li></ul><ul><ul><li>Present Population </li></ul></ul><ul><ul><li>User Behavior </li></ul></ul><ul><li>Unpredictable Behavior and New Populations. </li></ul><ul><ul><li>Make the unpredictable predictable. </li></ul></ul><ul><li>Changes in Adoption  Growth  Behavior </li></ul>
  7. 7. What is Performance Engineering? <ul><li>Engineering discipline with a primary focus on performance and scalability. </li></ul><ul><li>Combination of Software and System activities with intent of improving Performance and Scalability. </li></ul><ul><li>Performance Design  Performance Development  Performance Verification  Performance Benchmarking </li></ul>
  8. 8. Why does Blackboard invest in Performance Engineering? <ul><li>Some of the world’s largest application deployments outside of commercial portals and e-commerce sites are Blackboard. </li></ul><ul><li>Desire to add more applications, sub-systems and features to drive adoption and increase growth. </li></ul><ul><li>Differentiate Blackboard from other LMS/CMS ISVs and the Open Source Community. </li></ul>
  9. 9. Part 2: PE @ Bb
  10. 10. PerfEng at Bb… <ul><li>Triangle of Priorities </li></ul><ul><ul><li>Performance: Response Times (Apdex) </li></ul></ul><ul><ul><li>Scalability: High Session Activity with Low Abandonment (PAR) </li></ul></ul><ul><ul><li>User Experience: High Performance and Heavy Integration (Reference Architecture) </li></ul></ul><ul><li>Investment and Relationships </li></ul><ul><ul><li>Software Tool Set </li></ul></ul><ul><ul><li>Lab Sponsors </li></ul></ul>
  11. 11. Triangle of Priorities
  12. 12. PerfEng at Bb…
  13. 13. Performance: Response Times (Apdex) <ul><li>Every transaction in the application is defined as a state, an action state or an action. </li></ul><ul><li>Each transaction is assigned a ranking: critical, essential or trivial. </li></ul><ul><li>Each transaction is assigned an abandonment policy: low, medium, high and very high. </li></ul><ul><ul><li>Abandonment represents (4) dimensions of Apdex. </li></ul></ul>
  14. 14. State/Action Modeling <ul><li>State: A condition or point of reference within a sub-system in which an actor has the option to move to a sub-state, perform an action or move to a super-state. </li></ul><ul><ul><li>States are often considered navigation items easily identified by a bread crumb. </li></ul></ul><ul><ul><li>States can also be considered pages if and only if a sub-state or action is branched from the state. </li></ul></ul><ul><ul><ul><li>Example: Discussion Board Forum List Widget (State of the Discussion Board sub-system) </li></ul></ul></ul>
  15. 15. State/Action Modeling <ul><li>Action State: The navigation to an action. </li></ul><ul><ul><li>For example, an instructor wants to create a message for a topic. The instructor selects Add Message and is brought to a page that requires the user to input information followed by a submit to actually create the message. </li></ul></ul>
  16. 16. State/Action Modeling <ul><li>Action: An actor driven process that occurs within a state. </li></ul><ul><ul><li>Actions occur when an actor can not move into a sub-states. </li></ul></ul><ul><ul><li>Most often associated with a use case. </li></ul></ul><ul><ul><ul><li>Example: Replying to a thread within a Discussion Board Message. </li></ul></ul></ul>
  17. 17. What is Apdex(Application Performance Index)? <ul><li>What is Apdex </li></ul><ul><ul><li>Apdex is an open standard developed by an alliance of companies that defines a standardized method to report, benchmark, and track application performance. </li></ul></ul><ul><li>http://www.apdex.org </li></ul>
  18. 18. Apdex and User Abandonment <ul><li>Bb defines (4) Apdex Dimensions: </li></ul><ul><ul><li>Low (2s-8s) </li></ul></ul><ul><ul><li>Medium (5s-20s) </li></ul></ul><ul><ul><li>High (8s-32s) </li></ul></ul><ul><ul><li>Very High (12s-48s) </li></ul></ul>
  19. 19. Apdex and Workload Variations <ul><li>Apdex scores are taken across each data model variation. </li></ul><ul><li>Expect scores of 85% or higher for under-loaded systems. </li></ul><ul><li>During Verification Testing and Benchmarks, expect scores of 75% or higher. </li></ul><ul><li>If scores return below accepting, move to instrumentation and profiling (Method-R) </li></ul>
  20. 20. Scalability: High Session Activity with Low Abandonment (PAR) <ul><li>What is a PAR? </li></ul><ul><ul><li>P erformance A rchetype R atio </li></ul></ul><ul><ul><li>Scoring method to determine resource requirements of a deployment based on given system workload. </li></ul></ul><ul><li>Any component of the deployment can have a PAR score. </li></ul>
  21. 21. Scalability: High Session Activity with Low Abandonment (PAR) X-Axis: Iterations Y-Axis: Resource Utilization Resource Utilization Threshold Line Optimal Workload CPU
  22. 22. PAR Process: Step 1 Calibration <ul><li>Calibrate Workloads with User Abandonment </li></ul><ul><ul><li>Peak of Concurrency (POC): The virtual user workload in which response times are acceptable and the highest volume of virtual users are park of the scenario. </li></ul></ul><ul><ul><li>Level of Concurrency (LOC): The virtual user workload in which response times are acceptable the steadiest volume of virtual users are participating in the scenario. </li></ul></ul><ul><ul><li>Average Concurrency: The average workloads of the POC and LOC measurements combined. </li></ul></ul>
  23. 23. PAR Process: Step 2 App. Saturation <ul><li>Take workloads from abandonment run and disable abandonment. </li></ul><ul><ul><li>Run based on Peak of Concurrency workload (Abandonment Disabled) </li></ul></ul><ul><ul><li>Run based on Level of Concurrency workload (Abandonment Disabled) </li></ul></ul><ul><ul><li>Run based on Average of Concurrency workload (Abandonment Disabled) </li></ul></ul><ul><li>Example Metrics </li></ul><ul><ul><li>Response times consistently lower then ~5 seconds </li></ul></ul><ul><ul><li>Application CPU saturation close to X > 90% where X = CPU utilization + (1) Standard Deviation of the CPU Utilization </li></ul></ul><ul><ul><li>Total Sessions </li></ul></ul><ul><ul><li>Total Transactions </li></ul></ul><ul><ul><li>Application Server Hits Per Second </li></ul></ul><ul><ul><li>Database CPU saturation </li></ul></ul><ul><li>Strategies </li></ul><ul><ul><li>Clustering and Virtualization </li></ul></ul>
  24. 24. PAR Process: Step 3 DB Saturation <ul><li>Multiply the workload from Step 2 across identical application servers. </li></ul><ul><ul><li>Typically want 90% CPU utilization and sub-5 second response times. </li></ul></ul><ul><li>Example Metrics </li></ul><ul><ul><li>Database CPU saturation close to X > 80% where X = CPU utilization + (1) Standard Deviation of the CPU Utilization </li></ul></ul><ul><ul><li>Memory Utilization </li></ul></ul><ul><ul><li>Database Shadow Processes </li></ul></ul><ul><ul><li>I/O operations per second </li></ul></ul><ul><li>Strategies </li></ul><ul><ul><li>Increase CPU speed and count </li></ul></ul><ul><ul><li>Optimize storage configuration </li></ul></ul>
  25. 25. PAR Process: Additional Steps <ul><li>Hypothesis and Proof </li></ul><ul><ul><li>Essential part in this process is determine theoretical performance. </li></ul></ul><ul><ul><li>Understanding of linear, sub-linear or super-linear performance. </li></ul></ul><ul><ul><li>Simulate to determine actual. </li></ul></ul><ul><li>PARs can be gathered for other peripherals such as Load-Balancers, Storage Sub-Systems, Memory, CPUs, etc… </li></ul>
  26. 26. User Experience: High Performance and Heavy Integration (Reference Architecture) <ul><li>Insert Visio Here </li></ul>
  27. 27. Investment and Relationships <ul><li>PerfEng Team (10 Team Members) </li></ul><ul><ul><li>Combined both teams as part of the merger and increased head count. </li></ul></ul><ul><li>Software Tools </li></ul><ul><ul><li>Mercury LoadRunner </li></ul></ul><ul><ul><li>Quest Product Suite </li></ul></ul><ul><ul><li>Homegrown Tools: Simulation, Log Parsing, Modeling and Sampling </li></ul></ul><ul><ul><li>Hotsos Oracle Profiler </li></ul></ul>
  28. 28. Investment and Relationships <ul><li>Performance Lab Sponsors </li></ul><ul><ul><li>Dell: Servers and Remote Lab </li></ul></ul><ul><ul><li>Sun: Servers, Storage and Remote Lab </li></ul></ul><ul><ul><li>Intel: Servers </li></ul></ul><ul><ul><li>Coradiant: TrueSight Device </li></ul></ul><ul><ul><li>Quest: All Software Products </li></ul></ul><ul><ul><li>NetApp: Storage </li></ul></ul>
  29. 29. Part 3: Process and Methodology
  30. 30. Process and Methodology
  31. 31. SPE Overview… Assess Performance Risk <ul><li>Assessing the performance risk at the outset of the project (During Requirements) </li></ul><ul><li>Identify, qualify and mitigate: Rapid Cognition </li></ul><ul><li>Factors affecting risk: http://lightwave.blackboard.com/Engineering/1899 </li></ul>Identify Critical Use Cases <ul><li>Identify use cases where risk of performance goals not met causes the system to fail or be less than successful . </li></ul><ul><li>Ranking of use cases based on workload variation, execution paths, processing considerations and utility. </li></ul>Select Key Performance Scenarios <ul><li>Most frequently executed scenarios, or those that are critical to the perceived performance of the system. </li></ul><ul><li>Each performance scenario corresponds to a workload characterization. </li></ul><ul><li>Define execution models, behavior models, cognition models, data models and processing models. </li></ul>Establish Performance Objectives <ul><li>Specify the quantitative criteria for evaluating the performance characteristics of the system under development. </li></ul><ul><li>Must specify objectives prior to any simulations or analysis. </li></ul>Construct Performance Models <ul><li>Modeling techniques for representing the software processing steps for the performance model. </li></ul><ul><li>Sequence Diagramming, Markovian Probability Models and Discrete Simulation Models </li></ul>Software Execution Model <ul><li>Determination of software resource utilization to appropriately measure effect of software as it scales in usage. </li></ul><ul><li>Identification of Performance Anti-Patterns targeted for refactoring. </li></ul><ul><li>Method-R Analytics and Problem Solving via Decision Tree and Pattern Recognition. </li></ul>System Execution Model <ul><li>Determination of system resource requirements utilized by the software under a given workload. </li></ul><ul><li>Used for sizing and capacity models. </li></ul><ul><li>Method-R Analytics and Problem Solving via Decision Tree and Pattern Recognition. </li></ul>
  32. 32. Method-R: Requirements of a Good Methodology (Milsap, Carey) <ul><li>Predictive Capacity: A method must enable the analyst to predict the impact of proposed remedy. </li></ul><ul><li>Reliability: A method must identify the correct root cause of the problem, no matter what the root cause may be. </li></ul><ul><li>Determinism: A method must guide the analyst through an unambiguous sequence of steps that always rely upon documented axioms, no experience or intuition. </li></ul>
  33. 33. Method-R: Requirements of a Good Methodology (Milsap, Carey) <ul><li>Finiteness: A method must have a well-defined terminating condition, such as proof of optimality. </li></ul><ul><li>Practicality: A method must be usable in any reasonable operating condition. It would be unacceptable for a performance improvement method to rely upon tools that exist in some other operating environment but not others. </li></ul>
  34. 34. Method-R: Response Time Performance Improvement <ul><li>Practical way of thinking. </li></ul><ul><li>Often asking someone to be practical is in itself impractical. </li></ul><ul><ul><li>Select the user actions that the business needs improved performance. </li></ul></ul><ul><ul><li>Collect properly scoped diagnostic data that will allow you to identify the causes of a response time consumer while it is performed sub-optimally. </li></ul></ul><ul><ul><li>Execute the candidate optimization activity that will have the greatest net payoff. </li></ul></ul><ul><ul><li>Suspend your improvement activities until something changes. </li></ul></ul>
  35. 35. Part 4: Performance Lab ‘07
  36. 36. What projects are coming out of the PerfEng Lab in ‘07 <ul><li>Blackboard Performance Sizing and Certification Program. </li></ul><ul><li>Special Projects </li></ul><ul><ul><li>Virtualization: Zen, LDOMs/Containers and VMWare </li></ul></ul><ul><ul><li>Scalent Management Suite </li></ul></ul><ul><ul><li>Monitoring and Management </li></ul></ul><ul><ul><li>User Experience and Incident Management: Coradiant </li></ul></ul><ul><ul><li>Storage Protocols: NFS, IP-SAN and FC-SAN </li></ul></ul>
  37. 37. What is the BPSC? <ul><li>The BPSC is a benchmarking program designed to showcase the enterprise architecture, performance and scalability of the Blackboard Application Suite. </li></ul><ul><li>The BPSC will help Blackboard customers make the appropriate purchasing decisions (hardware and software) to support their Blackboard implementation. </li></ul><ul><li>The BPSC is a joint effort by Blackboard and members of the Blackboard Technology Family. This includes ISVs such as Microsoft, Oracle and Quest, as well as OEMs such Dell, Sun and Coradiant. </li></ul>
  38. 38. Special Projects <ul><li>Virtualization </li></ul><ul><ul><li>VMWare, Zen and LDOMs </li></ul></ul><ul><li>Management </li></ul><ul><ul><li>Scalent and Quest </li></ul></ul><ul><li>Monitoring </li></ul><ul><ul><li>Coradiant and Quest </li></ul></ul><ul><li>Storage Protocols </li></ul><ul><ul><li>FC/SAN, IP/SAN and NFS </li></ul></ul><ul><li>Scale-Up and Scale-Out Databases </li></ul><ul><ul><li>Oracle RAC </li></ul></ul><ul><ul><li>64-bit SQL Server </li></ul></ul>
  39. 39. Questions?
  40. 40. Links/References <ul><li>Blackboard Academic Suite Hardware Sizing Guide (Behind the Blackboard) </li></ul><ul><li>Performance and Capacity Planning Guidelines for the Blackboard Academic Suite (Behind the Blackboard) </li></ul><ul><li>http:// www.perfeng.com </li></ul><ul><li>http://www.spec.org/sfs97r1/results/sfs97r1.html </li></ul><ul><li>http:// www.storageperformance.org </li></ul><ul><li>http:// www.coradiant.com </li></ul><ul><li>http:// www.quest.com </li></ul><ul><li>http:// www.bmc.com </li></ul><ul><li>Performance by Design : Computer Capacity Planning By Example ; Menasce, Daniel </li></ul><ul><li>Scaling for E-Business: Technologies, Models, Performance, and Capacity Planning ; Menasce, Daniel </li></ul><ul><li>Linux Performance Tuning and Capacity Planning; Fink, Jason </li></ul><ul><li>Network Administrators Survival Guide ; Deveriya, Anand </li></ul><ul><li>Capacity Planning for Internet Services ; Cockcroft, Adrian </li></ul><ul><li>http://www.blackboard.com/docs/r6/6_3/en_US/admin/bbas_performance_capacity.pdf </li></ul><ul><li>http:// msdn.microsoft.com/library/default.asp?url =/library/en-us/dnbda/html/bdadotnetarch081.asp </li></ul><ul><li>http:// developers.sun.com/solaris/articles/systemslowdowns.html </li></ul><ul><li>http://www.oracle.com/technology/deploy/performance/index.html </li></ul><ul><li>http:// tpc.org/tpc_app/default.asp (TPC-App) </li></ul><ul><li>http:// tpc.org/tpcw/default.asp (TPC-W) </li></ul><ul><li>http:// java.sun.com /docs/performance/ </li></ul><ul><li>http://support.microsoft.com/kb/224587 </li></ul><ul><li>http:// www.javaperformancetuning.com </li></ul><ul><li>http:// www.oraperf.com </li></ul><ul><li>http:// www.ixora.com.au </li></ul><ul><li>http:// www.hotsos.com </li></ul><ul><li>http://perl.apache.org/docs/1.0/guide/performance.html </li></ul><ul><li>Sherlog, Webalizer, WebTrends, Analog </li></ul><ul><li>http://dir.yahoo.com/Computers_and_Internet/Software/Internet/World_Wide_Web/Servers/Log_Analysis_Tools/ </li></ul><ul><li>http://www.serverwatch.com/tutorials/article.php/3518061 </li></ul><ul><li>http://www-106.ibm.com/developerworks/rational/library/4250.html </li></ul><ul><li>http:// www.keynote.com/downloads/articles/tradesecrets.pdf </li></ul><ul><li>Whalen, Edward. Oracle Database 10G: Linux Administration ISBN: 0-07-223053-3; </li></ul><ul><li>Milsap, Cary. Optimizing Oracle Performance ISBN: 0-596-00527-X </li></ul><ul><li>DeLuca, Steve. Microsoft SQL Server 2000 Performance Tuning Technical Reference ISBN: 0735612706 </li></ul><ul><li>McGehee, B. “SQL-Server Configuration Performance Checklist” http://sql-server-performance.com/sql_server_performance_audit5.asp </li></ul><ul><li>http://www.sql-server-performance.com/jc_sql_server_quantative_analysis1.asp </li></ul>
  41. 41. Merci

×