Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PAC 2019 virtual Alexander Podelko

The Forgotten Art of Performance Modeling

  • Login to see the comments

PAC 2019 virtual Alexander Podelko

  1. 1. The Forgotten Art of Performance Modeling By Alexander Podelko
  2. 2. AbouttheSpeaker • Alexander Podelko • Specializing in performance since 1997 • Currently Consulting Member of Technical Staff at Oracle (Stamford, CT) • Performance testing and optimization of Enterprise Performance Management (EPM) a.k.a. Hyperion products • Board director at Computer Measurement Group (CMG) – non- profit organization of performance and capacity professionals Disclaimer: The views expressed here are my personal views only and do not necessarily represent those of my current or previous employers. All brands and trademarksmentioned are the property of their owners.
  3. 3. Why do we need modeling and where are we? How much math do we need here? How may we approach the challenge?
  4. 4. Starting Points • Importance of performance • Preaching to the choir • Importance of modeling / large picture • A part of performance engineering • Provide a large picture • Validate results • Complement testing • Much cheaper (when a model is already built)
  5. 5. Modeling Cases • Always was: What-if? • Workload growth • Configuration change • Projecting performance test results • Now: Continuous performance engineering • Projecting continuous / component / service performance testing • Discussing “white box” modeling here • To be able to make changes in underlying architecture • Not touching data extrapolation: time series forecasting, trending, ML, etc.
  6. 6. Two OppositeViews • Authors describe in detail how you can model performance using sophisticated math • You feel that as soon as you comprehend that math, you won’t have any problem with modeling • Authors say that it is a black magic and you’d better stay away from it [or do it in a minimal way with simple trending] • While you probably won’t see that view in serious books, it is often can be seen in Internet discussions
  7. 7. 1966:Instrumentation • 1966 – SMF (System Management Facilities) released as part of OS/360 • Still in use • Big Data ? • Deep Diagnostics ? • IT Operations Analytics ? • Observability ?
  8. 8. 1977: Performance Analysis Tool • 1977 – BEST/1 was released by BGS Systems, capacity and performance management tool • the first commercial package for computer performance analysis to be based on analytic models. • BGS Systems was acquired by BMC Software in 1998
  9. 9. Two Main Challenges • Modeling works well for known resource limitations • You can’t model what you don’t know • Lack of performance-related metrics of hardware to use in modeling • Hardware specifications won’t tell you how fast your systems would work on this hardware
  10. 10. Tools - Analytical • BEST/1 [acquired by BMC, TrueSight Capacity Optimization now (??)] • TeamQuest [acquired by HelpSystems] • Metron Athene [acquired by SyncSort] • PDQ - open source by Dr. Neil Gunther • SPE· ED – by Dr. Connie U. Smith • JMT – Java Modelling Tools
  11. 11. Tools - Simulation • HyPerformix [acquired by CA ]] • Palladio • QPME (Queueing Petri net Modeling Environment) • Not IT-specific / Discrete Event Simulation • MathWorks SimEvents • OMNeT++ • Many other commercial and open source tools
  12. 12. Decline ofModeling Tools • No modeling tool in active development [according to my knowledge] • Existing tools doesn’t keep up with modern trends • Scaling out • Lower need for queueing models • Drastic increase in sophistication • Need to model very complex architectures • Configuration discovery • Large variations in systems, no generic approach
  13. 13. Resources • CPU, I/O, memory, and network • And software objects • Connection and thread pools, caches, etc. • Resource Utilization • Related to a particular configuration • Often generic policies like CPU below 70% • Relative values (in %) are not useful if configuration is not given • Commercial Off-the-Shelf (COTS) software • Virtual environments
  14. 14. Resources: Absolute Values • Absolute values • # of instructions, I/O per transaction • Seen mainly in modeling • MIPS in mainframe world • IOPS, MiB/s • Importance increases again with the trends of virtualization, cloud computing, and SOA • VMware: CPU usage in MHz • Microsoft: Megacycles • Amazon: EC2 Compute Units (ECU) • Oracle: OCPU
  15. 15. Why do we need modeling and where are we? How much math do we need here? How may we approach the challenge?
  16. 16. QueueingTheory • M/M/1 • Distribution of arrival/service /number of servers • Kendall’s notation • Analytical models / Simulation
  17. 17. Multiple Servers
  18. 18. QueueingPetri Nets
  19. 19. Operational Laws • Utilization Law • Utilization = Service Time x Throughput • Service Demand Law • Service Time = Utilization / Throughput • Forced Flow Law • Component Throughput = Visits x Throughput • Little’s Law • Number in system = Response Time x Throughput
  20. 20. Universal ScalabilityLaw • By Dr. Neil Gunther • Concurrency - ideal parallelism • Contention (α) – for shared resources • Coherency (β) – crosstalk delay • Extension of Amdahl’s Law (β=0)
  21. 21. Fundamental Books
  22. 22. More Flexible Approach Introductory Books
  23. 23. SometimesLinear Model Works • Linear model may often work for multi-processor machines • Considering the working range of CPU utilization • Most IT shops don't want more than 70-80% • Throughput and CPU utilization increase proportionally • Response times increase insignificantly
  24. 24. Example • Simple queuing model was built using TeamQuest • Specific workload executed on a four-way server • Eight different load levels • Step 1 represents 100-user workload, each next step adds 200 users, step 8 represents 1,500 users • Linear model would be good through step 6 • With CPU utilization 87%
  25. 25. CPU Utilization System (Test_Windows) CPU (Application) Utilization by Workload Percentages 0 10 20 30 40 50 60 70 80 90 100 Step: 1 Step: 2 Step: 3 Step: 4 Step: 5 Step: 6 Step: 7 Step: 8 Percentage Request Medium
  26. 26. Throughput Throughput 0 5 10 15 20 25 Step: 1 Step: 2 Step: 3 Step: 4 Step: 5 Step: 6 Step: 7 Step: 8 Completions/Second Request Medium
  27. 27. Response Time Response Time 0 2 4 6 8 10 12 14 Step: 1 Step: 2 Step: 3 Step: 4 Step: 5 Step: 6 Step: 7 Step: 8 Seconds Request Medium
  28. 28. Why do we need modeling and where are we? How much math do we need here? How may we approach the challenge?
  29. 29. Buildinga Model • Significantly increases the value of performance testing • One more way to validate tests and help to identify problems • Answers questions about sizing the system • Doesn't need to be a formal model • May be just simple observations as to how much resources are needed by each component for the specific workload
  30. 30. May BeasSimpleasThat • Software needs X units of hardware power per request • Processor has Y units of hardware power • The number of such processors N needed for processing Z requests as N=Z*X/Y • See the need in absolute values? • We ignore all non-linear effects here – to be verified
  31. 31. Principles • “All models are wrong, but some are useful” • by George Box • "Everything should be made as simple as possible, but not simpler" • by Albert Einstein
  32. 32. Today’sChallenges • System became very sophisticated • Chances to find an appropriate tool are limited • Some tools disappeared, other rather basic • Chances to find an appropriate “template” are limited • Depends on the degree of non-linearity, etc. • Probably at least a combination of existing approaches • Use ideas • Not much else available 
  33. 33. HyPerformix Approach • HyPerformix docs / papers by Richard Gimarc, Amy Spellmann • Three types of data (Type 1, 2, and 3) • T1 Single Business Function Trace • tier-to-tier workflow of each of the application’s business functions • T2 Single Business Function Load Test • system resources required to process each business function on each server • T3 Load Test • typical load test with a mixed workload • model validation • WIFR – Workload, Infrastructure, Flow, Resource
  34. 34. T1 - Architecture • HyPerformix: Flow data • Communications and flow between tiers • Visit counts to tiers • Message sizes for each communication • Architecture and data / processing flow • Configuration discovery • Already exists in some tools (APM, AIOps, etc) • Communication between tiers/component
  35. 35. T2 –Resources • Run independent tests for each business function (T2) • To see how much resources each function requires • Per tier / component • Steady load • Load shouldn't be too light • To be not distorted by noise • Load shouldn't be too heavy • To avoid non-linear effects
  36. 36. T3 - Realistic Workload • Realistic load test • Mix of business functions (of T2) • Way to validate model • Steady load • Not too low • Not too high
  37. 37. TeamQuest Approach • Steady production or test load • Not too light, not too heavy • Realistic mix (T3 in HyPerformix terms) • Get model parameters from it • Model change of load / config what-if • No easy way to change the mix
  38. 38. Palladio Approach
  39. 39. Positive Attitude • In modeling concentrate on what you can predict • Value of the model • Understanding what is outside your models • Not a reason not to do it at all
  40. 40. The Future ofPerformance Modeling Configuration Discovery Performance Testing Workload Analysis Modeling Production Monitoring Continuous Performance Testing – Modeling as quality gates [similar to Keptn / Pitometer] Performance / Capacity / Costs What-If Analysis
  41. 41. Summary • Modeling is an important part of performance engineering • Even more so when full-scale tests become more rare • You do not necessarily need sophisticated math • Minimal math and common sense often enough • Good performance engineers do have non-formalized models in their mind • It becomes too complex to keep it in mind • Let’s revive the art !
  42. 42. Questions? alex.podelko@oracle.com alexanderpodelko.com/blog @apodelko

×