Manuel Pichler, profile picture
Uploaded byManuel Pichler
303 views

12 05 ipc_se_continuous_performance_tests

The document discusses continuous performance testing, highlighting its importance for identifying bottlenecks and ensuring optimizations remain effective over time. It covers best practices, tools like JMeter for performance testing, and the integration of automation tools like Ant for creating test environments. The authors, experienced professionals in software quality, stress the significance of continuous evaluation in maintaining high performance in web applications.

Embed presentation

Continuous Performance Tests IPC Spring 2012 Kore Nordmann (@koredn) Manuel Pichler (@manuelp) June 6, 2012 Continuous Performance Tests 1 / 25
About us Degree in computer sience More than 10 years of Qafoo passion for software quality professional PHP Open source enthusiasts Contributing to various FLOSS projects Continuous Performance Tests 2 / 25
About us Degree in computer sience More than 10 years of Qafoo passion for software quality professional PHP Open source enthusiasts Contributing to various FLOSS projects Continuous Performance Tests 2 / 25
About us Degree in computer sience More than 10 years of Qafoo passion for software quality professional PHP Open source enthusiasts Contributing to various FLOSS projects Continuous Performance Tests 2 / 25
About us Co-founders of Degree in computer sience More than 10 years of Qafoo passion for software quality professional PHP Open source enthusiasts Contributing to various FLOSS projects Continuous Performance Tests 2 / 25
About us Co-founders of Degree in computer sience More than 10 years of Qafoo passion for software quality professional PHP Open source enthusiasts We help people to create Contributing to various high quality web FLOSS projects applications. Continuous Performance Tests 2 / 25
About us Co-founders of Degree in computer sience More than 10 years of Qafoo passion for software quality professional PHP Open source enthusiasts We help people to create Contributing to various high quality web FLOSS projects applications. http://qafoo.com Continuous Performance Tests 2 / 25
Outline Motivation Conclusion Continuous Performance Tests 3 / 25
Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Tests 4 / 25
Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Tests 4 / 25
Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Tests 4 / 25
Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Tests 4 / 25
Often used tools Often misused tools sieve ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Continuous Performance Tests 5 / 25
Often used tools Often misused tools sieve ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Continuous Performance Tests 5 / 25
Often used tools Often misused tools sieve ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Continuous Performance Tests 5 / 25
Often used tools Often misused tools sieve ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Continuous Performance Tests 5 / 25
The problem is more complex Your task: Create a new webshop Assume it’s march The deadline is October this year, right before Christmas Continuous Performance Tests 6 / 25
The problem is more complex Your task: Create a new webshop Assume it’s march The deadline is October this year, right before Christmas Continuous Performance Tests 6 / 25
The problem is more complex Your task: Create a new webshop Assume it’s march The deadline is October this year, right before Christmas Continuous Performance Tests 6 / 25
A webshop Awesome Shop 0 articles 0.00 € Smartphone Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 1337,-- € 5 items in stock Comments Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor. Continuous Performance Tests 7 / 25
A webshop 2 articles 42.32 € Continuous Performance Tests 7 / 25
A webshop Smartphone Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 1337,-- € 5 items in stock Continuous Performance Tests 7 / 25
A webshop Comments Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor. Continuous Performance Tests 7 / 25
A webshop 1337,-- € 5 items in stock Continuous Performance Tests 7 / 25
A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25
A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25
A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25
A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25
A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25
JMeter 101 Thread Group Controller Config Element Timer Sampler Continuous Performance Tests 9 / 25
Getting started Create a test plan What do users actually do on your site? Example: Random browser User registration Sign on Shopping with checkout Commenting products Continuous Performance Tests 10 / 25
Getting started Create a test plan What do users actually do on your site? Example: Random browser User registration Sign on Shopping with checkout Commenting products Continuous Performance Tests 10 / 25
Using JMeter Demo Continuous Performance Tests 11 / 25
Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25
Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25
Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25
Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25
Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25
Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25
Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25
Apache Ant example 26 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache − f i l e ” 27 depends= ” setup −apc − bytecode − f i l e −cache , − b u i l d ” 28 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and f i l e based cache . ” / > 29 30 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache −apc ” 31 depends= ” setup −apc − bytecode −apc −cache , − b u i l d ” 32 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and APC based cache . ” / > 211 < t a r g e t name= ” − remote −exec − p a r a l l e l ” > 212 < subant t a r g e t = ” $ { t a r g e t } ” i n h e r i t a l l = ” t r u e ” > 213 < f i l e s e t d i r = ” $ { p r o j e c t . d i r } ” i n c l u d e s = ” s e r v e r ∗ . xml ” / > 214 < / subant > 215 </ t a r g e t > 216 217 < t a r g e t name= ” − remote −exec ” > 218 < sshexec command= ” $ { command } ” 219 username= ” $ { ssh . username } ” 220 password= ” $ { ssh . password } ” 221 h o s t = ” $ { hostname } ” 222 t r u s t =” true ” /> 223 </ t a r g e t > Continuous Performance Tests 13 / 25
Apache Ant example 26 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache − f i l e ” 27 depends= ” setup −apc − bytecode − f i l e −cache , − b u i l d ” 28 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and f i l e based cache . ” / > 29 30 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache −apc ” 31 depends= ” setup −apc − bytecode −apc −cache , − b u i l d ” 32 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and APC based cache . ” / > 211 < t a r g e t name= ” − remote −exec − p a r a l l e l ” > 212 < subant t a r g e t = ” $ { t a r g e t } ” i n h e r i t a l l = ” t r u e ” > 213 < f i l e s e t d i r = ” $ { p r o j e c t . d i r } ” i n c l u d e s = ” s e r v e r ∗ . xml ” / > 214 < / subant > 215 </ t a r g e t > 216 217 < t a r g e t name= ” − remote −exec ” > 218 < sshexec command= ” $ { command } ” 219 username= ” $ { ssh . username } ” 220 password= ” $ { ssh . password } ” 221 h o s t = ” $ { hostname } ” 222 t r u s t =” true ” /> 223 </ t a r g e t > Continuous Performance Tests 13 / 25
Apache Ant example 26 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache − f i l e ” 27 depends= ” setup −apc − bytecode − f i l e −cache , − b u i l d ” 28 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and f i l e based cache . ” / > 29 30 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache −apc ” 31 depends= ” setup −apc − bytecode −apc −cache , − b u i l d ” 32 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and APC based cache . ” / > 211 < t a r g e t name= ” − remote −exec − p a r a l l e l ” > 212 < subant t a r g e t = ” $ { t a r g e t } ” i n h e r i t a l l = ” t r u e ” > 213 < f i l e s e t d i r = ” $ { p r o j e c t . d i r } ” i n c l u d e s = ” s e r v e r ∗ . xml ” / > 214 < / subant > 215 </ t a r g e t > 216 217 < t a r g e t name= ” − remote −exec ” > 218 < sshexec command= ” $ { command } ” 219 username= ” $ { ssh . username } ” 220 password= ” $ { ssh . password } ” 221 h o s t = ” $ { hostname } ” 222 t r u s t =” true ” /> 223 </ t a r g e t > Continuous Performance Tests 13 / 25
Apache Ant example 26 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache − f i l e ” 27 depends= ” setup −apc − bytecode − f i l e −cache , − b u i l d ” 28 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and f i l e based cache . ” / > 29 30 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache −apc ” 31 depends= ” setup −apc − bytecode −apc −cache , − b u i l d ” 32 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and APC based cache . ” / > 211 < t a r g e t name= ” − remote −exec − p a r a l l e l ” > 212 < subant t a r g e t = ” $ { t a r g e t } ” i n h e r i t a l l = ” t r u e ” > 213 < f i l e s e t d i r = ” $ { p r o j e c t . d i r } ” i n c l u d e s = ” s e r v e r ∗ . xml ” / > 214 < / subant > 215 </ t a r g e t > 216 217 < t a r g e t name= ” − remote −exec ” > 218 < sshexec command= ” $ { command } ” 219 username= ” $ { ssh . username } ” 220 password= ” $ { ssh . password } ” 221 h o s t = ” $ { hostname } ” 222 t r u s t =” true ” /> 223 </ t a r g e t > Continuous Performance Tests 13 / 25
Apache Ant example 26 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache − f i l e ” 27 depends= ” setup −apc − bytecode − f i l e −cache , − b u i l d ” 28 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and f i l e based cache . ” / > 29 30 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache −apc ” 31 depends= ” setup −apc − bytecode −apc −cache , − b u i l d ” 32 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and APC based cache . ” / > 211 < t a r g e t name= ” − remote −exec − p a r a l l e l ” > 212 < subant t a r g e t = ” $ { t a r g e t } ” i n h e r i t a l l = ” t r u e ” > 213 < f i l e s e t d i r = ” $ { p r o j e c t . d i r } ” i n c l u d e s = ” s e r v e r ∗ . xml ” / > 214 < / subant > 215 </ t a r g e t > 216 217 < t a r g e t name= ” − remote −exec ” > 218 < sshexec command= ” $ { command } ” 219 username= ” $ { ssh . username } ” 220 password= ” $ { ssh . password } ” 221 h o s t = ” $ { hostname } ” 222 t r u s t =” true ” /> 223 </ t a r g e t > Continuous Performance Tests 13 / 25
Apache Ant example 454 < t a r g e t name= ” − r e s t a r t − h o s t ” > 455 < a n t c a l l t a r g e t = ” − remote −exec ” > 456 <param name= ” command ” v a l u e = ” shutdown − r now ” / > 457 </ a n t c a l l > 458 459 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } t o s t o p . . . ” / > 460 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 461 < not > 462 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 463 < / not > 464 </ w a i t f o r > 465 466 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } i s up again . . . ” / > 467 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 468 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 469 </ w a i t f o r > 470 </ t a r g e t > Continuous Performance Tests 14 / 25
Apache Ant example 454 < t a r g e t name= ” − r e s t a r t − h o s t ” > 455 < a n t c a l l t a r g e t = ” − remote −exec ” > 456 <param name= ” command ” v a l u e = ” shutdown − r now ” / > 457 </ a n t c a l l > 458 459 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } t o s t o p . . . ” / > 460 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 461 < not > 462 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 463 < / not > 464 </ w a i t f o r > 465 466 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } i s up again . . . ” / > 467 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 468 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 469 </ w a i t f o r > 470 </ t a r g e t > Continuous Performance Tests 14 / 25
Apache Ant example 454 < t a r g e t name= ” − r e s t a r t − h o s t ” > 455 < a n t c a l l t a r g e t = ” − remote −exec ” > 456 <param name= ” command ” v a l u e = ” shutdown − r now ” / > 457 </ a n t c a l l > 458 459 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } t o s t o p . . . ” / > 460 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 461 < not > 462 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 463 < / not > 464 </ w a i t f o r > 465 466 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } i s up again . . . ” / > 467 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 468 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 469 </ w a i t f o r > 470 </ t a r g e t > Continuous Performance Tests 14 / 25
Apache Ant example 454 < t a r g e t name= ” − r e s t a r t − h o s t ” > 455 < a n t c a l l t a r g e t = ” − remote −exec ” > 456 <param name= ” command ” v a l u e = ” shutdown − r now ” / > 457 </ a n t c a l l > 458 459 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } t o s t o p . . . ” / > 460 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 461 < not > 462 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 463 < / not > 464 </ w a i t f o r > 465 466 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } i s up again . . . ” / > 467 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 468 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 469 </ w a i t f o r > 470 </ t a r g e t > Continuous Performance Tests 14 / 25
Apache Ant example 421 < t a r g e t name= ” j m e t e r ” depends= ” − s e t t i n g s − i n i t , − s t a r t − j m e t e r ” / > 422 423 < t a r g e t name= ” − s t a r t − j m e t e r ” > 424 < a n t c a l l t a r g e t = ” − s t a r t − j m e t e r − before −hook ” / > 425 426 < j m e t e r jmeterhome= ” $ { l o c a l . j m e t e r . home . d i r } ” 427 r e s u l t l o g =” ${ l o c a l . jmeter . log . f i l e } ” 428 t e s t p l a n = ” $ { l o c a l . j m e t e r . t e s t . d i r } / $ { j m e t e r . f i l e } ”> 429 430 < p r o p e r t y name= ” j m e t e r . data . d i r ” v a l u e = ” $ { l o c a l . p r o j e c t . data . d i r } ” / > 431 < p r o p e r t y name= ” j m e t e r . rampup . t i m e ” v a l u e = ” $ { j m e t e r . rampup . t i m e } ” / > 432 < p r o p e r t y name= ” j m e t e r . e x e c u t i o n . t i m e ” v a l u e = ” $ { j m e t e r . e x e c u t i o n . t i m e } ” / > 433 </ j m e t e r > 434 435 < a n t c a l l t a r g e t = ” − s t a r t − j m e t e r − a f t e r −hook ” / > 436 </ t a r g e t > Continuous Performance Tests 15 / 25
Apache Ant example 421 < t a r g e t name= ” j m e t e r ” depends= ” − s e t t i n g s − i n i t , − s t a r t − j m e t e r ” / > 422 423 < t a r g e t name= ” − s t a r t − j m e t e r ” > 424 < a n t c a l l t a r g e t = ” − s t a r t − j m e t e r − before −hook ” / > 425 426 < j m e t e r jmeterhome= ” $ { l o c a l . j m e t e r . home . d i r } ” 427 r e s u l t l o g =” ${ l o c a l . jmeter . log . f i l e } ” 428 t e s t p l a n = ” $ { l o c a l . j m e t e r . t e s t . d i r } / $ { j m e t e r . f i l e } ”> 429 430 < p r o p e r t y name= ” j m e t e r . data . d i r ” v a l u e = ” $ { l o c a l . p r o j e c t . data . d i r } ” / > 431 < p r o p e r t y name= ” j m e t e r . rampup . t i m e ” v a l u e = ” $ { j m e t e r . rampup . t i m e } ” / > 432 < p r o p e r t y name= ” j m e t e r . e x e c u t i o n . t i m e ” v a l u e = ” $ { j m e t e r . e x e c u t i o n . t i m e } ” / > 433 </ j m e t e r > 434 435 < a n t c a l l t a r g e t = ” − s t a r t − j m e t e r − a f t e r −hook ” / > 436 </ t a r g e t > Continuous Performance Tests 15 / 25
Apache Ant example 351 < t a r g e t name= ” −backup − data − from − h o s t ” > 352 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . error . log } ’ ” 353 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− e r r o r . l o g ” 354 t r u s t =” true ” /> 355 356 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . access . l o g } ’ ” 357 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− access . l o g ” 358 t r u s t =” true ” /> 359 360 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . php . e r r o r . log } ’ ” 361 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− p h p e r r o r s . l o g ” 362 t r u s t =” true ” /> 363 364 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . b a s e d i r } / s c r i p t s / load . log ’ ” 365 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− l o a d . l o g ” 366 t r u s t =” true ” /> 367 368 </ t a r g e t > Continuous Performance Tests 16 / 25
Apache Ant example 351 < t a r g e t name= ” −backup − data − from − h o s t ” > 352 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . error . log } ’ ” 353 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− e r r o r . l o g ” 354 t r u s t =” true ” /> 355 356 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . access . l o g } ’ ” 357 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− access . l o g ” 358 t r u s t =” true ” /> 359 360 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . php . e r r o r . log } ’ ” 361 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− p h p e r r o r s . l o g ” 362 t r u s t =” true ” /> 363 364 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . b a s e d i r } / s c r i p t s / load . log ’ ” 365 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− l o a d . l o g ” 366 t r u s t =” true ” /> 367 368 </ t a r g e t > Continuous Performance Tests 16 / 25
Apache Ant example 351 < t a r g e t name= ” −backup − data − from − h o s t ” > 352 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . error . log } ’ ” 353 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− e r r o r . l o g ” 354 t r u s t =” true ” /> 355 356 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . access . l o g } ’ ” 357 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− access . l o g ” 358 t r u s t =” true ” /> 359 360 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . php . e r r o r . log } ’ ” 361 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− p h p e r r o r s . l o g ” 362 t r u s t =” true ” /> 363 364 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . b a s e d i r } / s c r i p t s / load . log ’ ” 365 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− l o a d . l o g ” 366 t r u s t =” true ” /> 367 368 </ t a r g e t > Continuous Performance Tests 16 / 25
Apache Ant example 351 < t a r g e t name= ” −backup − data − from − h o s t ” > 352 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . error . log } ’ ” 353 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− e r r o r . l o g ” 354 t r u s t =” true ” /> 355 356 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . access . l o g } ’ ” 357 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− access . l o g ” 358 t r u s t =” true ” /> 359 360 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . php . e r r o r . log } ’ ” 361 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− p h p e r r o r s . l o g ” 362 t r u s t =” true ” /> 363 364 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . b a s e d i r } / s c r i p t s / load . log ’ ” 365 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− l o a d . l o g ” 366 t r u s t =” true ” /> 367 368 </ t a r g e t > Continuous Performance Tests 16 / 25
Apache Ant example 351 < t a r g e t name= ” −backup − data − from − h o s t ” > 352 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . error . log } ’ ” 353 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− e r r o r . l o g ” 354 t r u s t =” true ” /> 355 356 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . access . l o g } ’ ” 357 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− access . l o g ” 358 t r u s t =” true ” /> 359 360 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . php . e r r o r . log } ’ ” 361 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− p h p e r r o r s . l o g ” 362 t r u s t =” true ” /> 363 364 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . b a s e d i r } / s c r i p t s / load . log ’ ” 365 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− l o a d . l o g ” 366 t r u s t =” true ” /> 367 368 </ t a r g e t > Continuous Performance Tests 16 / 25
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
Extended system metrics Response time Requests per second 75 300 Seconds R/s 50 200 25 100 00:00 05:00 09:00 13:00 17:00 21:00 01:00 00:00 05:00 09:00 13:00 17:00 21:00 01:00 Minimum Average Maximum Average CPU usage Memory usage 100 100 Percent Percent 75 75 50 50 25 25 05:00 09:00 13:00 17:00 21:00 01:00 05:00 05:00 09:00 13:00 17:00 21:00 01:00 05:00 System User Used Memory Shared Cached Continuous Performance Tests 18 / 25
Continuous Performance Jenkins Plugin available Sonar JMeter Plugin available Continuous Performance Tests 19 / 25
Continuous Performance Jenkins Plugin available Sonar JMeter Plugin available Continuous Performance Tests 19 / 25
Continuous Performance Jenkins Plugin available Sonar JMeter Plugin available Continuous Performance Tests 19 / 25
Continuous Performance Jenkins Plugin available Sonar JMeter Plugin available Continuous Performance Tests 19 / 25
Continuous testing with Jenkins Demo Continuous Performance Tests 20 / 25
Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25
Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25
Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25
Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25
Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25
Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
Outline Motivation Conclusion Continuous Performance Tests 23 / 25
Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Tests 24 / 25
Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Tests 24 / 25
Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Tests 24 / 25
Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Tests 24 / 25
Thanks for listening Rate this talk https://joind.in/6662 More about us: http://qafoo.com Continuous Performance Tests 25 / 25

More Related Content

PDF
Selenium DeTox for Achieving the Right Testing Pyramid
byNaresh Jain
 
PPTX
The Test Pyramid
byThiago Ghisi
 
PPTX
Test Automation Pyramid
byT. Alexander Lystad
 
PDF
Improving the Quality of Incoming Code
byNaresh Jain
 
PDF
Acceptance Testing for Continuous Delivery by Dave Farley at #AgileIndia2019
byAgile India
 
PDF
Inverting Test Pyramid - A First Hand Experience Report
byNaresh Jain
 
PDF
The Three Pillars Approach to an Agile Testing Strategy
byTechWell
 
PDF
Building and Scaling High Performing Technology Organizations by Jez Humble a...
byAgile India
 
Selenium DeTox for Achieving the Right Testing Pyramid
byNaresh Jain
 
The Test Pyramid
byThiago Ghisi
 
Test Automation Pyramid
byT. Alexander Lystad
 
Improving the Quality of Incoming Code
byNaresh Jain
 
Acceptance Testing for Continuous Delivery by Dave Farley at #AgileIndia2019
byAgile India
 
Inverting Test Pyramid - A First Hand Experience Report
byNaresh Jain
 
The Three Pillars Approach to an Agile Testing Strategy
byTechWell
 
Building and Scaling High Performing Technology Organizations by Jez Humble a...
byAgile India
 

What's hot

PDF
Continuous delivery its not about the technology, its about the people.
byTomas Riha
 
PDF
A Day in the Life of a Test Architect
byTechWell
 
PDF
Continuous delivery its not about the technology, its about the people. @pipe...
byTomas Riha
 
PPT
Software Testing with Agile Requirements Practices
byA B M Moniruzzaman
 
PDF
A DevOps Primer: Whole Team Approaches for Better Software Quality
byTechWell
 
PDF
Moving to Continuous Delivery Without Breaking Your Code
byXebiaLabs
 
PPTX
Test pyramid agile pune conference
bySachin Natu
 
PDF
Are Your Continuous Tests Too Fragile for Agile?
byErika Barron
 
PDF
Scaling Test first for the Enterprise
byQASymphony
 
PDF
How testers add value to the organization appium conf
byCorina Pip
 
KEY
Testing and DevOps Culture: Lessons Learned
byLB Denker
 
PPT
Testing in Agile Projects
bysriks7
 
PPTX
DevOps Summit 2015 Presentation: Continuous Testing At the Speed of DevOps
bySailaja Tennati
 
PPTX
Agile tour ncr test360_degree - agile testing on steroids
byVipul Gupta
 
PPTX
Relieveing the Testing Bottle Neck - Webinar
byCprime
 
PPTX
Going extreme-with-extreme-programming
byMichael Green
 
PPTX
Essentials for Building Five-Star Alexa Voice Experiences
byApplause
 
PPTX
Continuous Testing - The Final Frontier
byJaco Greyling
 
PPTX
New model
byTEST Huddle
 
PPTX
How to get Automated Testing "Done"
byTEST Huddle
 
Continuous delivery its not about the technology, its about the people.
byTomas Riha
 
A Day in the Life of a Test Architect
byTechWell
 
Continuous delivery its not about the technology, its about the people. @pipe...
byTomas Riha
 
Software Testing with Agile Requirements Practices
byA B M Moniruzzaman
 
A DevOps Primer: Whole Team Approaches for Better Software Quality
byTechWell
 
Moving to Continuous Delivery Without Breaking Your Code
byXebiaLabs
 
Test pyramid agile pune conference
bySachin Natu
 
Are Your Continuous Tests Too Fragile for Agile?
byErika Barron
 
Scaling Test first for the Enterprise
byQASymphony
 
How testers add value to the organization appium conf
byCorina Pip
 
Testing and DevOps Culture: Lessons Learned
byLB Denker
 
Testing in Agile Projects
bysriks7
 
DevOps Summit 2015 Presentation: Continuous Testing At the Speed of DevOps
bySailaja Tennati
 
Agile tour ncr test360_degree - agile testing on steroids
byVipul Gupta
 
Relieveing the Testing Bottle Neck - Webinar
byCprime
 
Going extreme-with-extreme-programming
byMichael Green
 
Essentials for Building Five-Star Alexa Voice Experiences
byApplause
 
Continuous Testing - The Final Frontier
byJaco Greyling
 
New model
byTEST Huddle
 
How to get Automated Testing "Done"
byTEST Huddle
 

Viewers also liked

PDF
khososyat namayandeggan
byEng. QaSeMy
 
PDF
12 05 ipc_se_testen_state_of_the_art
byManuel Pichler
 
PDF
Cn5 formacao solos_vito_solos
byVitoriano Pereira
 
PPTX
Como subir un documento a slideshare
bylaura6440
 
PDF
160526 auditories rd 56 2016 granollers mercat_icaen
byEmpresa i Emprenedoria Granollers
 
DOCX
Resume of Jaun Abbas
byJaun Abbas
 
PDF
Richard simmons flyer
byToddy Wobbema
 
PDF
Bobkó Gergely - CV
byGergely Bobk
 
PDF
160526 auditories rd 56 2016 granollers mercat_enginyers industrials catalunya
byEmpresa i Emprenedoria Granollers
 
PDF
161109 jornada eficiència icaen
byEmpresa i Emprenedoria Granollers
 
DOCX
Project center in trichy @ieee 2016 17 titles for java and dotnet
byTripleN Infotech
 
PDF
PLE: aprenentatge personal
byIgnasi Alcalde
 
PDF
Date concerto chaplin
byagimus
 
PPTX
Colombo PerfTest Meetup - April 2016
byChamal Asela Perera
 
PDF
Proceso Conceptualizacion
byjohannaAC
 
PPTX
Rewriting The Revenue Rules: From Mobile-First To Mobile-Only Mobile Shopping...
bySOASTA
 
PPTX
Keerah gracia
byKeerah Gracia
 
khososyat namayandeggan
byEng. QaSeMy
 
12 05 ipc_se_testen_state_of_the_art
byManuel Pichler
 
Cn5 formacao solos_vito_solos
byVitoriano Pereira
 
Como subir un documento a slideshare
bylaura6440
 
160526 auditories rd 56 2016 granollers mercat_icaen
byEmpresa i Emprenedoria Granollers
 
Resume of Jaun Abbas
byJaun Abbas
 
Richard simmons flyer
byToddy Wobbema
 
Bobkó Gergely - CV
byGergely Bobk
 
160526 auditories rd 56 2016 granollers mercat_enginyers industrials catalunya
byEmpresa i Emprenedoria Granollers
 
161109 jornada eficiència icaen
byEmpresa i Emprenedoria Granollers
 
Project center in trichy @ieee 2016 17 titles for java and dotnet
byTripleN Infotech
 
PLE: aprenentatge personal
byIgnasi Alcalde
 
Date concerto chaplin
byagimus
 
Colombo PerfTest Meetup - April 2016
byChamal Asela Perera
 
Proceso Conceptualizacion
byjohannaAC
 
Rewriting The Revenue Rules: From Mobile-First To Mobile-Only Mobile Shopping...
bySOASTA
 
Keerah gracia
byKeerah Gracia
 

Similar to 12 05 ipc_se_continuous_performance_tests

PDF
Testing for continuous delivery with visual studio 2012
byCristiano Caetano
 
PDF
SmartBear Microsoft Datasheet,
bySmartBear
 
PDF
Effective Strategies for Distributed Testing
byAnand Bagmar
 
PDF
Supersize me: Making Drupal go large
byTom Phethean
 
PPTX
Eric Ries Lean Startup Presentation For Web 2.0 Expo April 1 2009 A Disciplin...
byEric Ries
 
PPTX
Distributed agile testing_for_enterprises
byThoughtWorks Studios
 
PDF
From dev to ops and beyond - getting it done
byEdorian
 
PDF
Continuous Delivery Overview
byWill Iverson
 
PPT
Scaling Web Apps P Falcone
byjedt
 
PDF
Iqnite keynote
byScott Barber
 
PDF
High Performance Software Engineering Teams
byLars Thorup
 
PDF
QA is dead long live the new QA - Agile Dev and QA Conference Israel
byYuval Yeret
 
PDF
Continuous Integration testing based on Selenium and Hudson
byZbyszek Mockun
 
PDF
Open Source tools in Continuous Integration environment (case study for agil...
bysuwalki24.pl
 
PPTX
Leveraging HP Performance Center
byMartin Spier
 
PPT
Smartere test og udvikling med virtualiserede miljøer, Mark Garcia, IBM US
byIBM Danmark
 
PPTX
What they don't teach you in CS departments
byWillSullivan
 
PPTX
An Introduction to Software Performance Engineering
byCorrelsense
 
PDF
Keynote: Next Generation Testing
byLoh Chuan Ho Ervin
 
PDF
Continuous Performance Testing_ A Comprehensive Guide for Developers.pdf
byflufftailshop
 
Testing for continuous delivery with visual studio 2012
byCristiano Caetano
 
SmartBear Microsoft Datasheet,
bySmartBear
 
Effective Strategies for Distributed Testing
byAnand Bagmar
 
Supersize me: Making Drupal go large
byTom Phethean
 
Eric Ries Lean Startup Presentation For Web 2.0 Expo April 1 2009 A Disciplin...
byEric Ries
 
Distributed agile testing_for_enterprises
byThoughtWorks Studios
 
From dev to ops and beyond - getting it done
byEdorian
 
Continuous Delivery Overview
byWill Iverson
 
Scaling Web Apps P Falcone
byjedt
 
Iqnite keynote
byScott Barber
 
High Performance Software Engineering Teams
byLars Thorup
 
QA is dead long live the new QA - Agile Dev and QA Conference Israel
byYuval Yeret
 
Continuous Integration testing based on Selenium and Hudson
byZbyszek Mockun
 
Open Source tools in Continuous Integration environment (case study for agil...
bysuwalki24.pl
 
Leveraging HP Performance Center
byMartin Spier
 
Smartere test og udvikling med virtualiserede miljøer, Mark Garcia, IBM US
byIBM Danmark
 
What they don't teach you in CS departments
byWillSullivan
 
An Introduction to Software Performance Engineering
byCorrelsense
 
Keynote: Next Generation Testing
byLoh Chuan Ho Ervin
 
Continuous Performance Testing_ A Comprehensive Guide for Developers.pdf
byflufftailshop
 

Recently uploaded

PDF
Artificial Intelligence and Barbarism - Conceptual Map
byalfadix
 
PDF
Navigating the Spectrum of Advanced AI – Agentic, Autonomous, and Autopoietic...
byScott M. Graffius
 
PDF
Video Infrastructure_ Streaming Architecture and Delivery Systems.pdf
byTenbyte Limited
 
PPTX
Why 2026 Could Be a Turning Point for Decentralized Exchanges.pptx
bycalliemorgan193
 
PDF
Top Benefits of Using KVM VPS Hosting for Growing Businesses
byEthernet Servers
 
PPTX
Salesforce Spring 26 Release Key .pptx
byMehediHasan939830
 
DOCX
Top Websites To ⭐Buy ⭐Old ⭐Gmail ⭐Accounts (PVA & Bulk) (5).docx
byBuy Old Gmail Accounts
 
PDF
AI Driven & AI Native Development AI native development
byZainKarim7
 
PDF
MAD (1).pdf Mobile application develipment
byprathiT1
 
PPTX
Large Language Model. LLM Audit Artificial Intelligence
byMANASCHOUDHARY22
 
PDF
Regenerative Agriculture Finance : Environmental impact
byrose mason
 
PDF
Digital Marketing Trends in 2026: AI, Data, Content & Future Growth
byConacent Consulting
 
PDF
Benefits of Using the IAC 500 Sensor for Ambient Conditions
byCS Instruments
 
PDF
A Brief Introduction About Christopher Elwell Woburn
byChristopher Elwell Woburn
 
PDF
Unit 1.2 Components of a Computer System.pdf
bySanieBautista
 
PDF
Techbrains Baku 2025 by GoUP - all speaker session
byHusseinMalikMammadli
 
PPTX
Introduction to Computer Network Concepts.pptx
byAnacrissa Soriano
 
PDF
IAI-Unit1-notes useful.............................
bydinesh620610
 
PPTX
Compare and contrast types of attacks.pptx
bysyednaqihassan14
 
PPTX
Corporate AI Training to AI Enable a Company Workforce
byStélio Inácio
 
Artificial Intelligence and Barbarism - Conceptual Map
byalfadix
 
Navigating the Spectrum of Advanced AI – Agentic, Autonomous, and Autopoietic...
byScott M. Graffius
 
Video Infrastructure_ Streaming Architecture and Delivery Systems.pdf
byTenbyte Limited
 
Why 2026 Could Be a Turning Point for Decentralized Exchanges.pptx
bycalliemorgan193
 
Top Benefits of Using KVM VPS Hosting for Growing Businesses
byEthernet Servers
 
Salesforce Spring 26 Release Key .pptx
byMehediHasan939830
 
Top Websites To ⭐Buy ⭐Old ⭐Gmail ⭐Accounts (PVA & Bulk) (5).docx
byBuy Old Gmail Accounts
 
AI Driven & AI Native Development AI native development
byZainKarim7
 
MAD (1).pdf Mobile application develipment
byprathiT1
 
Large Language Model. LLM Audit Artificial Intelligence
byMANASCHOUDHARY22
 
Regenerative Agriculture Finance : Environmental impact
byrose mason
 
Digital Marketing Trends in 2026: AI, Data, Content & Future Growth
byConacent Consulting
 
Benefits of Using the IAC 500 Sensor for Ambient Conditions
byCS Instruments
 
A Brief Introduction About Christopher Elwell Woburn
byChristopher Elwell Woburn
 
Unit 1.2 Components of a Computer System.pdf
bySanieBautista
 
Techbrains Baku 2025 by GoUP - all speaker session
byHusseinMalikMammadli
 
Introduction to Computer Network Concepts.pptx
byAnacrissa Soriano
 
IAI-Unit1-notes useful.............................
bydinesh620610
 
Compare and contrast types of attacks.pptx
bysyednaqihassan14
 
Corporate AI Training to AI Enable a Company Workforce
byStélio Inácio
 

12 05 ipc_se_continuous_performance_tests

  • 1.
    Continuous Performance Tests IPC Spring 2012 Kore Nordmann (@koredn) Manuel Pichler (@manuelp) June 6, 2012 Continuous Performance Tests 1 / 25
  • 2.
    About us Degree in computer sience More than 10 years of Qafoo passion for software quality professional PHP Open source enthusiasts Contributing to various FLOSS projects Continuous Performance Tests 2 / 25
  • 3.
    About us Degree in computer sience More than 10 years of Qafoo passion for software quality professional PHP Open source enthusiasts Contributing to various FLOSS projects Continuous Performance Tests 2 / 25
  • 4.
    About us Degree in computer sience More than 10 years of Qafoo passion for software quality professional PHP Open source enthusiasts Contributing to various FLOSS projects Continuous Performance Tests 2 / 25
  • 5.
    About us Co-founders of Degree in computer sience More than 10 years of Qafoo passion for software quality professional PHP Open source enthusiasts Contributing to various FLOSS projects Continuous Performance Tests 2 / 25
  • 6.
    About us Co-founders of Degree in computer sience More than 10 years of Qafoo passion for software quality professional PHP Open source enthusiasts We help people to create Contributing to various high quality web FLOSS projects applications. Continuous Performance Tests 2 / 25
  • 7.
    About us Co-founders of Degree in computer sience More than 10 years of Qafoo passion for software quality professional PHP Open source enthusiasts We help people to create Contributing to various high quality web FLOSS projects applications. http://qafoo.com Continuous Performance Tests 2 / 25
  • 8.
    Outline Motivation Conclusion Continuous Performance Tests 3 / 25
  • 9.
    Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Tests 4 / 25
  • 10.
    Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Tests 4 / 25
  • 11.
    Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Tests 4 / 25
  • 12.
    Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Tests 4 / 25
  • 13.
    Often used tools Often misused tools sieve ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Continuous Performance Tests 5 / 25
  • 14.
    Often used tools Often misused tools sieve ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Continuous Performance Tests 5 / 25
  • 15.
    Often used tools Often misused tools sieve ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Continuous Performance Tests 5 / 25
  • 16.
    Often used tools Often misused tools sieve ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Continuous Performance Tests 5 / 25
  • 17.
    The problem ismore complex Your task: Create a new webshop Assume it’s march The deadline is October this year, right before Christmas Continuous Performance Tests 6 / 25
  • 18.
    The problem ismore complex Your task: Create a new webshop Assume it’s march The deadline is October this year, right before Christmas Continuous Performance Tests 6 / 25
  • 19.
    The problem ismore complex Your task: Create a new webshop Assume it’s march The deadline is October this year, right before Christmas Continuous Performance Tests 6 / 25
  • 20.
    A webshop Awesome Shop 0 articles 0.00 € Smartphone Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 1337,-- € 5 items in stock Comments Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor. Continuous Performance Tests 7 / 25
  • 21.
    A webshop 2 articles 42.32 € Continuous Performance Tests 7 / 25
  • 22.
    A webshop Smartphone Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 1337,-- € 5 items in stock Continuous Performance Tests 7 / 25
  • 23.
    A webshop Comments Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor. Continuous Performance Tests 7 / 25
  • 24.
    A webshop 1337,-- € 5 items in stock Continuous Performance Tests 7 / 25
  • 25.
    A real toolfor performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25
  • 26.
    A real toolfor performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25
  • 27.
    A real toolfor performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25
  • 28.
    A real toolfor performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25
  • 29.
    A real toolfor performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25
  • 30.
    JMeter 101 Thread Group Controller Config Element Timer Sampler Continuous Performance Tests 9 / 25
  • 31.
    Getting started Create a test plan What do users actually do on your site? Example: Random browser User registration Sign on Shopping with checkout Commenting products Continuous Performance Tests 10 / 25
  • 32.
    Getting started Create a test plan What do users actually do on your site? Example: Random browser User registration Sign on Shopping with checkout Commenting products Continuous Performance Tests 10 / 25
  • 33.
    Using JMeter Demo Continuous Performance Tests 11 / 25
  • 34.
    Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25
  • 35.
    Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25
  • 36.
    Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25
  • 37.
    Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25
  • 38.
    Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25
  • 39.
    Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25
  • 40.
    Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25
  • 41.
    Apache Ant example 26 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache − f i l e ” 27 depends= ” setup −apc − bytecode − f i l e −cache , − b u i l d ” 28 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and f i l e based cache . ” / > 29 30 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache −apc ” 31 depends= ” setup −apc − bytecode −apc −cache , − b u i l d ” 32 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and APC based cache . ” / > 211 < t a r g e t name= ” − remote −exec − p a r a l l e l ” > 212 < subant t a r g e t = ” $ { t a r g e t } ” i n h e r i t a l l = ” t r u e ” > 213 < f i l e s e t d i r = ” $ { p r o j e c t . d i r } ” i n c l u d e s = ” s e r v e r ∗ . xml ” / > 214 < / subant > 215 </ t a r g e t > 216 217 < t a r g e t name= ” − remote −exec ” > 218 < sshexec command= ” $ { command } ” 219 username= ” $ { ssh . username } ” 220 password= ” $ { ssh . password } ” 221 h o s t = ” $ { hostname } ” 222 t r u s t =” true ” /> 223 </ t a r g e t > Continuous Performance Tests 13 / 25
  • 42.
    Apache Ant example 26 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache − f i l e ” 27 depends= ” setup −apc − bytecode − f i l e −cache , − b u i l d ” 28 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and f i l e based cache . ” / > 29 30 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache −apc ” 31 depends= ” setup −apc − bytecode −apc −cache , − b u i l d ” 32 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and APC based cache . ” / > 211 < t a r g e t name= ” − remote −exec − p a r a l l e l ” > 212 < subant t a r g e t = ” $ { t a r g e t } ” i n h e r i t a l l = ” t r u e ” > 213 < f i l e s e t d i r = ” $ { p r o j e c t . d i r } ” i n c l u d e s = ” s e r v e r ∗ . xml ” / > 214 < / subant > 215 </ t a r g e t > 216 217 < t a r g e t name= ” − remote −exec ” > 218 < sshexec command= ” $ { command } ” 219 username= ” $ { ssh . username } ” 220 password= ” $ { ssh . password } ” 221 h o s t = ” $ { hostname } ” 222 t r u s t =” true ” /> 223 </ t a r g e t > Continuous Performance Tests 13 / 25
  • 43.
    Apache Ant example 26 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache − f i l e ” 27 depends= ” setup −apc − bytecode − f i l e −cache , − b u i l d ” 28 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and f i l e based cache . ” / > 29 30 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache −apc ” 31 depends= ” setup −apc − bytecode −apc −cache , − b u i l d ” 32 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and APC based cache . ” / > 211 < t a r g e t name= ” − remote −exec − p a r a l l e l ” > 212 < subant t a r g e t = ” $ { t a r g e t } ” i n h e r i t a l l = ” t r u e ” > 213 < f i l e s e t d i r = ” $ { p r o j e c t . d i r } ” i n c l u d e s = ” s e r v e r ∗ . xml ” / > 214 < / subant > 215 </ t a r g e t > 216 217 < t a r g e t name= ” − remote −exec ” > 218 < sshexec command= ” $ { command } ” 219 username= ” $ { ssh . username } ” 220 password= ” $ { ssh . password } ” 221 h o s t = ” $ { hostname } ” 222 t r u s t =” true ” /> 223 </ t a r g e t > Continuous Performance Tests 13 / 25
  • 44.
    Apache Ant example 26 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache − f i l e ” 27 depends= ” setup −apc − bytecode − f i l e −cache , − b u i l d ” 28 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and f i l e based cache . ” / > 29 30 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache −apc ” 31 depends= ” setup −apc − bytecode −apc −cache , − b u i l d ” 32 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and APC based cache . ” / > 211 < t a r g e t name= ” − remote −exec − p a r a l l e l ” > 212 < subant t a r g e t = ” $ { t a r g e t } ” i n h e r i t a l l = ” t r u e ” > 213 < f i l e s e t d i r = ” $ { p r o j e c t . d i r } ” i n c l u d e s = ” s e r v e r ∗ . xml ” / > 214 < / subant > 215 </ t a r g e t > 216 217 < t a r g e t name= ” − remote −exec ” > 218 < sshexec command= ” $ { command } ” 219 username= ” $ { ssh . username } ” 220 password= ” $ { ssh . password } ” 221 h o s t = ” $ { hostname } ” 222 t r u s t =” true ” /> 223 </ t a r g e t > Continuous Performance Tests 13 / 25
  • 45.
    Apache Ant example 26 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache − f i l e ” 27 depends= ” setup −apc − bytecode − f i l e −cache , − b u i l d ” 28 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and f i l e based cache . ” / > 29 30 < t a r g e t name= ” b u i l d −apc − bytecode −cache − user −cache −apc ” 31 depends= ” setup −apc − bytecode −apc −cache , − b u i l d ” 32 d e s c r i p t i o n = ” −> Run w i t h APC opcode cache and APC based cache . ” / > 211 < t a r g e t name= ” − remote −exec − p a r a l l e l ” > 212 < subant t a r g e t = ” $ { t a r g e t } ” i n h e r i t a l l = ” t r u e ” > 213 < f i l e s e t d i r = ” $ { p r o j e c t . d i r } ” i n c l u d e s = ” s e r v e r ∗ . xml ” / > 214 < / subant > 215 </ t a r g e t > 216 217 < t a r g e t name= ” − remote −exec ” > 218 < sshexec command= ” $ { command } ” 219 username= ” $ { ssh . username } ” 220 password= ” $ { ssh . password } ” 221 h o s t = ” $ { hostname } ” 222 t r u s t =” true ” /> 223 </ t a r g e t > Continuous Performance Tests 13 / 25
  • 46.
    Apache Ant example 454 < t a r g e t name= ” − r e s t a r t − h o s t ” > 455 < a n t c a l l t a r g e t = ” − remote −exec ” > 456 <param name= ” command ” v a l u e = ” shutdown − r now ” / > 457 </ a n t c a l l > 458 459 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } t o s t o p . . . ” / > 460 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 461 < not > 462 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 463 < / not > 464 </ w a i t f o r > 465 466 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } i s up again . . . ” / > 467 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 468 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 469 </ w a i t f o r > 470 </ t a r g e t > Continuous Performance Tests 14 / 25
  • 47.
    Apache Ant example 454 < t a r g e t name= ” − r e s t a r t − h o s t ” > 455 < a n t c a l l t a r g e t = ” − remote −exec ” > 456 <param name= ” command ” v a l u e = ” shutdown − r now ” / > 457 </ a n t c a l l > 458 459 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } t o s t o p . . . ” / > 460 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 461 < not > 462 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 463 < / not > 464 </ w a i t f o r > 465 466 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } i s up again . . . ” / > 467 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 468 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 469 </ w a i t f o r > 470 </ t a r g e t > Continuous Performance Tests 14 / 25
  • 48.
    Apache Ant example 454 < t a r g e t name= ” − r e s t a r t − h o s t ” > 455 < a n t c a l l t a r g e t = ” − remote −exec ” > 456 <param name= ” command ” v a l u e = ” shutdown − r now ” / > 457 </ a n t c a l l > 458 459 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } t o s t o p . . . ” / > 460 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 461 < not > 462 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 463 < / not > 464 </ w a i t f o r > 465 466 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } i s up again . . . ” / > 467 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 468 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 469 </ w a i t f o r > 470 </ t a r g e t > Continuous Performance Tests 14 / 25
  • 49.
    Apache Ant example 454 < t a r g e t name= ” − r e s t a r t − h o s t ” > 455 < a n t c a l l t a r g e t = ” − remote −exec ” > 456 <param name= ” command ” v a l u e = ” shutdown − r now ” / > 457 </ a n t c a l l > 458 459 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } t o s t o p . . . ” / > 460 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 461 < not > 462 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 463 < / not > 464 </ w a i t f o r > 465 466 <echo taskname= ” w a i t f o r ” message= ” Wait f o r $ { hostname } i s up again . . . ” / > 467 < w a i t f o r maxwait= ” 5 ” m a x w a i t u n i t = ” minute ” checkevery= ” 100 ” > 468 < h t t p u r l = ” h t t p : / / $ { hostname } ” / > 469 </ w a i t f o r > 470 </ t a r g e t > Continuous Performance Tests 14 / 25
  • 50.
    Apache Ant example 421 < t a r g e t name= ” j m e t e r ” depends= ” − s e t t i n g s − i n i t , − s t a r t − j m e t e r ” / > 422 423 < t a r g e t name= ” − s t a r t − j m e t e r ” > 424 < a n t c a l l t a r g e t = ” − s t a r t − j m e t e r − before −hook ” / > 425 426 < j m e t e r jmeterhome= ” $ { l o c a l . j m e t e r . home . d i r } ” 427 r e s u l t l o g =” ${ l o c a l . jmeter . log . f i l e } ” 428 t e s t p l a n = ” $ { l o c a l . j m e t e r . t e s t . d i r } / $ { j m e t e r . f i l e } ”> 429 430 < p r o p e r t y name= ” j m e t e r . data . d i r ” v a l u e = ” $ { l o c a l . p r o j e c t . data . d i r } ” / > 431 < p r o p e r t y name= ” j m e t e r . rampup . t i m e ” v a l u e = ” $ { j m e t e r . rampup . t i m e } ” / > 432 < p r o p e r t y name= ” j m e t e r . e x e c u t i o n . t i m e ” v a l u e = ” $ { j m e t e r . e x e c u t i o n . t i m e } ” / > 433 </ j m e t e r > 434 435 < a n t c a l l t a r g e t = ” − s t a r t − j m e t e r − a f t e r −hook ” / > 436 </ t a r g e t > Continuous Performance Tests 15 / 25
  • 51.
    Apache Ant example 421 < t a r g e t name= ” j m e t e r ” depends= ” − s e t t i n g s − i n i t , − s t a r t − j m e t e r ” / > 422 423 < t a r g e t name= ” − s t a r t − j m e t e r ” > 424 < a n t c a l l t a r g e t = ” − s t a r t − j m e t e r − before −hook ” / > 425 426 < j m e t e r jmeterhome= ” $ { l o c a l . j m e t e r . home . d i r } ” 427 r e s u l t l o g =” ${ l o c a l . jmeter . log . f i l e } ” 428 t e s t p l a n = ” $ { l o c a l . j m e t e r . t e s t . d i r } / $ { j m e t e r . f i l e } ”> 429 430 < p r o p e r t y name= ” j m e t e r . data . d i r ” v a l u e = ” $ { l o c a l . p r o j e c t . data . d i r } ” / > 431 < p r o p e r t y name= ” j m e t e r . rampup . t i m e ” v a l u e = ” $ { j m e t e r . rampup . t i m e } ” / > 432 < p r o p e r t y name= ” j m e t e r . e x e c u t i o n . t i m e ” v a l u e = ” $ { j m e t e r . e x e c u t i o n . t i m e } ” / > 433 </ j m e t e r > 434 435 < a n t c a l l t a r g e t = ” − s t a r t − j m e t e r − a f t e r −hook ” / > 436 </ t a r g e t > Continuous Performance Tests 15 / 25
  • 52.
    Apache Ant example 351 < t a r g e t name= ” −backup − data − from − h o s t ” > 352 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . error . log } ’ ” 353 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− e r r o r . l o g ” 354 t r u s t =” true ” /> 355 356 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . access . l o g } ’ ” 357 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− access . l o g ” 358 t r u s t =” true ” /> 359 360 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . php . e r r o r . log } ’ ” 361 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− p h p e r r o r s . l o g ” 362 t r u s t =” true ” /> 363 364 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . b a s e d i r } / s c r i p t s / load . log ’ ” 365 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− l o a d . l o g ” 366 t r u s t =” true ” /> 367 368 </ t a r g e t > Continuous Performance Tests 16 / 25
  • 53.
    Apache Ant example 351 < t a r g e t name= ” −backup − data − from − h o s t ” > 352 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . error . log } ’ ” 353 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− e r r o r . l o g ” 354 t r u s t =” true ” /> 355 356 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . access . l o g } ’ ” 357 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− access . l o g ” 358 t r u s t =” true ” /> 359 360 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . php . e r r o r . log } ’ ” 361 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− p h p e r r o r s . l o g ” 362 t r u s t =” true ” /> 363 364 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . b a s e d i r } / s c r i p t s / load . log ’ ” 365 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− l o a d . l o g ” 366 t r u s t =” true ” /> 367 368 </ t a r g e t > Continuous Performance Tests 16 / 25
  • 54.
    Apache Ant example 351 < t a r g e t name= ” −backup − data − from − h o s t ” > 352 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . error . log } ’ ” 353 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− e r r o r . l o g ” 354 t r u s t =” true ” /> 355 356 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . access . l o g } ’ ” 357 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− access . l o g ” 358 t r u s t =” true ” /> 359 360 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . php . e r r o r . log } ’ ” 361 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− p h p e r r o r s . l o g ” 362 t r u s t =” true ” /> 363 364 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . b a s e d i r } / s c r i p t s / load . log ’ ” 365 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− l o a d . l o g ” 366 t r u s t =” true ” /> 367 368 </ t a r g e t > Continuous Performance Tests 16 / 25
  • 55.
    Apache Ant example 351 < t a r g e t name= ” −backup − data − from − h o s t ” > 352 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . error . log } ’ ” 353 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− e r r o r . l o g ” 354 t r u s t =” true ” /> 355 356 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . access . l o g } ’ ” 357 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− access . l o g ” 358 t r u s t =” true ” /> 359 360 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . php . e r r o r . log } ’ ” 361 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− p h p e r r o r s . l o g ” 362 t r u s t =” true ” /> 363 364 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . b a s e d i r } / s c r i p t s / load . log ’ ” 365 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− l o a d . l o g ” 366 t r u s t =” true ” /> 367 368 </ t a r g e t > Continuous Performance Tests 16 / 25
  • 56.
    Apache Ant example 351 < t a r g e t name= ” −backup − data − from − h o s t ” > 352 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . error . log } ’ ” 353 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− e r r o r . l o g ” 354 t r u s t =” true ” /> 355 356 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . webserver . access . l o g } ’ ” 357 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− access . l o g ” 358 t r u s t =” true ” /> 359 360 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . php . e r r o r . log } ’ ” 361 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− p h p e r r o r s . l o g ” 362 t r u s t =” true ” /> 363 364 < scp r e m o t e f i l e = ” $ { ssh . username } : $ { ssh . password }@${ hostname } : ’ $ { remote . b a s e d i r } / s c r i p t s / load . log ’ ” 365 l o c a l t o f i l e = ” $ { l o c a l . b u i l d d i r } / $ { hostname }− l o a d . l o g ” 366 t r u s t =” true ” /> 367 368 </ t a r g e t > Continuous Performance Tests 16 / 25
  • 57.
    Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
  • 58.
    Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
  • 59.
    Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
  • 60.
    Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
  • 61.
    Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
  • 62.
    Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
  • 63.
    Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
  • 64.
    Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25
  • 65.
    Extended system metrics Response time Requests per second 75 300 Seconds R/s 50 200 25 100 00:00 05:00 09:00 13:00 17:00 21:00 01:00 00:00 05:00 09:00 13:00 17:00 21:00 01:00 Minimum Average Maximum Average CPU usage Memory usage 100 100 Percent Percent 75 75 50 50 25 25 05:00 09:00 13:00 17:00 21:00 01:00 05:00 05:00 09:00 13:00 17:00 21:00 01:00 05:00 System User Used Memory Shared Cached Continuous Performance Tests 18 / 25
  • 66.
    Continuous Performance Jenkins Plugin available Sonar JMeter Plugin available Continuous Performance Tests 19 / 25
  • 67.
    Continuous Performance Jenkins Plugin available Sonar JMeter Plugin available Continuous Performance Tests 19 / 25
  • 68.
    Continuous Performance Jenkins Plugin available Sonar JMeter Plugin available Continuous Performance Tests 19 / 25
  • 69.
    Continuous Performance Jenkins Plugin available Sonar JMeter Plugin available Continuous Performance Tests 19 / 25
  • 70.
    Continuous testing withJenkins Demo Continuous Performance Tests 20 / 25
  • 71.
    Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25
  • 72.
    Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25
  • 73.
    Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25
  • 74.
    Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25
  • 75.
    Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25
  • 76.
    Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25
  • 77.
    Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
  • 78.
    Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
  • 79.
    Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
  • 80.
    Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
  • 81.
    Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
  • 82.
    Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
  • 83.
    Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
  • 84.
    Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 ∗ 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So . . . 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25
  • 85.
    Outline Motivation Conclusion Continuous Performance Tests 23 / 25
  • 86.
    Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Tests 24 / 25
  • 87.
    Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Tests 24 / 25
  • 88.
    Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Tests 24 / 25
  • 89.
    Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Tests 24 / 25
  • 90.
    Thanks for listening Rate this talk https://joind.in/6662 More about us: http://qafoo.com Continuous Performance Tests 25 / 25