Like this presentation? Why not share!

- Collaboration Importance In Agile S... by Veselin Georgiev 460 views
- Statistics & Decision Science for A... by Sanjaya K Saxena 817 views
- Stat dec theory by asriadam 154 views
- Office for National Statistics - A ... by AgileCymru 669 views
- Survival Data Analysis for Sekolah ... by Setio Pramono 3016 views
- Cycles of Developement - Pamela Lev... by Manu Melwin Joy 5703 views

452

-1

-1

Published on

Published in:
Technology

No Downloads

Total Views

452

On Slideshare

0

From Embeds

0

Number of Embeds

0

Shares

0

Downloads

23

Comments

0

Likes

1

No embeds

No notes for slide

- 1. Predictive Performance Testing Integrating Statistical Tests into Agile Development Lifecycles Tom Kleingarn Lead, Performance Engineering Digital River http://www.linkedin.com/in/tomkleingarn http://www.perftom.com ©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice
- 2. Agenda > Introduction > Performance engineering > Agile > Outputs from LoadRunner > Basic statistics > Advanced statistics > Summary > Practical application
- 3. About Me > Tom Kleingarn > Lead, Performance Engineering - Digital River > 4 years in performance engineering > Tested over 100 systems/applications > 100’s of performance tests > Tools > LoadRunner > JMeter > Webmetrics, Keynote, Gomez > ‘R’ and Excel > Quality Center > QuickTest Professional
- 4. > Leading provider of global e-commerce solutions > Builds and manages online businesses for software and game publishers, consumer electronics manufacturers, distributors, online retailers and affiliates. > Comprehensive platform offers > > > > > > > > Site development and hosting Order management Fraud management Export control Tax management Physical and digital product fulfillment Multi-lingual customer service Advanced reporting and strategic marketing
- 5. Performance Engineering > The process of experimental design, test execution, and results analysis, utilized to validate system performance as part of the Software Development Lifecycle (SDLC). > Performance requirements – measureable targets of speed, reliability, and/or capacity used in performance validation. > Latency < 10ms, measured at the 99th percentile > 99.95% uptime > Throughput of 1,000 requests per second
- 6. Performance Testing Cycle 1. Requirements Analysis 2. Create test plan 3. Create automated scripts 4. Define workload model 5. Execute scenarios 6. Analyze results > Rinse and repeat if… > Defects identified > Change in requirements > Setup or environment issues > Performance requirement not met Digital River Test Automation
- 7. Agile > A software development paradigm that emphasizes rapid process cycles, cross-functional teams, frequent examination of progress, and adaptability. Initial Plan Scrum Deploy
- 8. Agile Performance Engineering > Clear and constant communication > Involvement in initial requirements and design phase > Identify key business processes before they are built > Coordinate with analysts and development to build key business processes first > Integrate load generation requirements into project schedule > Test immediately with v1.0 > Schedule tests to auto-start, run independently > Identify invalid test results before deep analysis
- 9. LoadRunner Results > Measures of central tendency > Average = ∑(all samples)/(sample size) = > Median = 50th percentile > Mode – highest frequency, the value that occurred the most > Measures of variability > Min, max > Standard Deviation = > 90th percentile
- 10. LoadRunner Results 90% 50% 50% 10%
- 11. Basic Statistics – Sample vs. Population > Performance requirement: average latency < 3 seconds > What if you ran 50 rounds? 100 rounds?
- 12. Basic Statistics – Sample vs. Population > Sample – set of values, subset of population > Population – all potentially observable values > Measurements > Statistic – the estimated value from a collection of samples > Parameter – the “true” value you are attempting to estimate Not a representative sample!
- 13. Basic Statistics – Sample vs. Population > Sampling distribution – the probability distribution of a given statistic based on a random sample of size n > Dependent on the underlying population > How do you know the system under test met the performance requirement?
- 14. Basic Statistics – Normal Distribution > With larger samples, data tend to cluster around the mean
- 15. Basic Statistics – Normal Distribution Sir Francis Galton’s “Bean Machine”
- 16. Confidence Intervals > The probability that an interval made up of two endpoints will contain the true mean parameter μ > 95% confidence interval: > … where 1.96 is a score from the normal distribution associated with 95% probability:
- 17. Confidence Intervals > In repeated rounds of testing, a confidence interval will contain the true mean parameter with a certain probability: True Average
- 18. Confidence Intervals in Excel Statistic Value 95% Value 99% Formula Average 3.40 3.40 Standard Deviation 1.45 1.45 Sample size 500 500 Confidence Level 0.95 0.99 Significance Level 0.05 0.01 0.0127 0.167 =CONFIDENCE(Sig. Level, Std Dev, Sample Size) Lower Bound 3.273 3.233 =Average - Margin of Error Upper Bound 3.527 3.567 =Average + Margin of Error Margin of Error =1-(Confidence Level) > 95% confidence - true average latency 3.273 to 3.527 seconds > 99% confidence - true average latency 3.233 to 3.567 seconds > Our range is wider at 99% compared to 95%, 0.334 sec vs. 0.254 sec
- 19. The T-test > Test that your sample mean is greater than/less than a certain value > Performance requirement: Mean latency < 3 seconds > Null hypothesis: Mean latency >= 3 seconds > Alternative hypothesis: Mean latency is < 3 seconds Add pic
- 20. T-test – Raw Data from LoadRunner n = 500
- 21. T-test in ‘R’ > ‘R’ for statistical analysis > http://www.r-project.org/ Load test data from a file: > datafile <- read.table("C:Datatest.data", header = FALSE, col.names= c("latency")) Attach the dataframe: > attach(datafile) Create a “vector” from the dataframe: > latency <- datafile$latency
- 22. T.Test in ‘R’ > t.test(latency, alternative="less", mu=3, tails=1) One Sample t-test data: latency t = -2.9968, df = 499, p-value = 0.001432 alternative hypothesis: true mean is less than 3 > There is a 0.14% probability that the true average latency of the system is greater than 3 seconds. In this case we would reject the null hypothesis. > There is a 99.86% probability that the true average latency is less than 3 seconds
- 23. T-test – Number of Samples Required > power.t.test(sd=sd, sig.level=0.05, power=0.90, delta=mean(latency)*0.01, type="one.sample") One-sample t test power calculation n = 215.5319 delta = 0.03241267 sd = 0.1461401 sig.level = 0.05 power = 0.9 alternative = two.sided > We need at least 216 samples > Our sample size is 500, we have enough samples to proceed
- 24. Test for Normality > Test that the data is “normal” > Clustered around a central value, no outliers > Roughly fits the normal distribution > shapiro.test(latency) Shapiro-Wilk normality test data: latency p-value = 0.8943 > Our sample distribution is approximately normal > p-value < 0.05 indicates the distribution is not normal
- 25. Review > Sample vs. Population > Normal distribution > Confidence intervals > T-test > Sample size > Test for normality > Practical application > Performance requirements > Compare two code builds > Compare system infrastructure changes
- 26. Case Study > Engaged in a new web service project > Average latency < 25ms > Applied statistical analysis > System did not meet requirement > Identified problem transaction > Development fix applied > Additional test, requirement met > Prevented a failure in production
- 27. Implementation in Agile Projects > Involvement in early design stages > Identify performance requirements > Build key business processes first > Calculate required sample size > Apply statistical analysis > Run fewer tests with greater confidence in your results > Prevent performance defects from entering production > Prevent SLA violations in production

No public clipboards found for this slide

×
### Save the most important slides with Clipping

Clipping is a handy way to collect and organize the most important slides from a presentation. You can keep your great finds in clipboards organized around topics.

Be the first to comment