Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Variety of automated tests
Similar to Variety of automated tests (20)
More from Артём Курапов
Recently uploaded
Recently uploaded (20)
Variety of automated tests
- 1. Variety of automated tests Artjom Kurapov
- 2. Kurapov.ee @tot_ra 10 years in IT / PHP lvl 65 elf/hunter
- 4. Developer QA engineer Fear of sudden death
- 7. Objectivity - state or quality of being true even outside of a subject’s individual biases, interpretations, feelings, and imaginings.
- 8. Client – I dropped a phone Test that it works
- 10. OVERVIEW MarFn Fowler hHp://marFnfowler.com/arFcles/microservice-‐tesFng/ WHITE GRAY BLACK SUBJECTIVE IMPORTANCE GITHUB LINKS TEST FAMILY
- 11. PHP testing tools & authors SebasFan Bergmann @s_bergmann atoum/atoum mlively/Phake ribozz/PhpSpock marco-‐fiset/TesFfy.php SimpleTest inclusive-‐design/AChecker Ciaran McNulty @ciaranmcnulty KonstanFn Kudryashov @everzet Michael Bodnarchuk @davert
- 12. LEVEL 1 BLACK BOX
- 13. Scenario tests 1
- 14. Basic scenario tests • Create a feature list – Register as new user – Register with coupon – Login – Checkout • Timeouts between commands • Hard-‐coded data 1 CodecepFon/CodecepFon giorgiosironi/phpunit-‐selenium minkphp/MinkSelenium2Driver
- 15. UI testing tools • Browser controllers – Selenium, Sahi • Headless browsers – CasperJS, PhantomJS, GouHe • Custom stuff -‐ WaFr, TestComplete, iMacros 1 CodecepFon/CodecepFon giorgiosironi/phpunit-‐selenium minkphp/MinkSelenium2Driver
- 16. 1
- 17. Sum of complexity • Not informaJve if fails • Environment maintenance • New tech – dragging, canvas, svg, – touch, mulFtouch, 3d-‐touch 1 CodecepFon/CodecepFon giorgiosironi/phpunit-‐selenium minkphp/MinkSelenium2Driver
- 18. Instability & non-‐determinism • Async -‐ ajax, animaFons, upload • HTML-‐dependent • Hard-‐coded data & dependent tests • External websites • Drivers 1 CodecepFon/CodecepFon giorgiosironi/phpunit-‐selenium minkphp/MinkSelenium2Driver
- 19. AbstracFng QA framework • Syntax improvement • Page objects + Workflows • API for data reset / VM per process • RepeaFng funcFons – Login, Messages – Form saving, Layout mode 2 CodecepFon/CodecepFon giorgiosironi/phpunit-‐selenium minkphp/MinkSelenium2Driver
- 20. Behaviour tests • Human-‐readable specificaFon • Needs parsing • Business people-‐oriented Behat/Behat 3
- 21. 3
- 22. 3
- 23. 3
- 24. Screenshots / CSS regression Pixel-‐by-‐pixel comparison Similarity in % imagick::compareImages() 4 Huddle/PhantomCSS
- 25. LEVEL 2 GRAY BOX
- 26. Environment diagnosFcs • RAM / HDD / file permissions • Packages, extensions, libraries • Services / docker containers 1
- 27. 1
- 28. 1
- 29. IntegraFon external APIs • Availability -‐ FTP / REST API – server is down – no file / API 404 • CompaFbility – empty file – wrong json/XML schema 2 guzzle/guzzle CodecepFon/CodecepFon
- 30. 2
- 31. Frontend / UI Backend / DB API Single Page ApplicaFon
- 32. Own internal API • HTTP GET on all endpoints No “Stack trace”, “error”, “excepFon” in response • POST/PUT for data change – Mobile app scenarios – UDP + binary data – Image upload 3 guzzle/guzzle CodecepFon/CodecepFon
- 33. Why you need to test with real DB • DB specifics – dates / autoincrements • Stored procedures • TransacFons • Handling large data • ORM mapping 3
- 34. Internal API + separate DB • DB isolaFon for each test • Slow & SQL fixtures harder to maintain • AlternaFves – transacFon & rollback – Virtual machine – raw db file copy 4
- 35. 4
- 36. Security + fuzzing tests Bruteforce own API • InjecFons -‐ sql, xss • Argument types – bool, array, string, json • Binary/Images – Different types, sizes drodil/PHP-‐Fuzz-‐tester 5
- 37. 5
- 38. 5
- 39. 5
- 42. Performance tests - backend • Response Fme < 1s • DB queries < 15 • Memory load < 10MB • Is the app geong slower? 6 jokkedk/webgrind
- 43. Data integrity/sync tests • Check data in N services – files in Mysql, mongo, AWS • JSON schema, NULLs, Dates Example – translaFons export-‐import in UTF16 CSV 7
- 44. Application migration/upgrade tests • If you run delta scripts / migraFons (like phinx) • MigraFon script = unit tests • Data integrity auer run 8
- 45. Configuration matrix tests If you support mulFple environment combinaFons (CMS) • Browsers • PHP versions • Servers • OS Virtualize! – docker, vagrant 4*4*4*4 = 256 variations, but it can get worse 9
- 46. LEVEL 3 WHITE BOX
- 47. Unit tests • Limited to isolated funcFon or class • Data processing confirmaFon – math / logic / price calculators – regexp / parsers / validators – random() raFo sing() 1
- 48. Unit tests of private methods “you should not test privates, its a bad pracFce” • Abstract class • Public efficiency / cycles • Debug new ReflectionClass(‘Man’)-‐>getMethod(‘urinate’) -‐>setAccessible(true) urinate() 1
- 49. Unit tests calling private methods “(In phpspec) you cannot mock system under test. This is a bad pracFce” runkit::method_redefine urinate() 1 visitBathRoom()
- 50. 1
- 52. Unit tests calling other classes Test doubles: • Dummy – argument that’s not used (during test) • Stub – hard-‐coded value • Spy – did methods got called? • Fake – dynamic stubs • Mock – all-‐in-‐one padraic/mockery phpspec/prophecy Codeception/AspectMock sebastianbergmann/phpunit-mock-objects 2
- 53. Mocks ExpectaJons of coupling dynamics: • What objects/funcFons to be called • Order & Fming • Arguments • Emulates returns • ExcepFons 2 padraic/mockery phpspec/prophecy Codeception/AspectMock sebastianbergmann/phpunit-mock-objects
- 54. man home sleep() Mocks useBed() protect() warm() store() 2 padraic/mockery phpspec/prophecy Codeception/AspectMock sebastianbergmann/phpunit-mock-objects
- 55. Mocks – does it call something? 2 man hostel sleep() useBed() padraic/mockery phpspec/prophecy Codeception/AspectMock sebastianbergmann/phpunit-mock-objects
- 56. 2
- 57. + prophecy 2 Mocks with magic syntax
- 58. 2 Mocks chains
- 59. 2 Mock magic chains
- 60. 2 Mocks – backward thinking Code • Update() • Pdo prepare() • Statement execute() • Statement rowCount() Test • Statement execute() • Statement rowCount() • Pdo prepare() • Update()
- 61. Contracts man wife tests tests 3 sex() getNewKid()
- 62. Side 1 has expectations 3 man Perfect wife sex_anyTime() sex_noCondom() getNewKid_hasMyHair() sex() getNewKid()
- 63. Side 2 has different expectations abortKid() findNewDaddy() 3 Rich daddy wife sex_forPleasure() getNewKid_auer30() sex() getNewKid()
- 64. Contracts – a smarter interface • Input validaFon • Method call sequence • Internal state limitaFons • Output expectaFons empty array, false, Null, array, stdClass, Inherited enFty, id-‐based map, paginaFon object… 3
- 65. Contracts – mock matcher man wife 3 Contract
- 66. Contract tests ParFes have obligaFons • PrecondiFons – before running method • PostcondiFons • Invariants – limitaFons for the enFre class lisachenko/php-‐deal stuartherbert/ContractLib 3
- 67. 3
- 68. 3
- 69. In-‐memory filesystem tests • built-‐in streaming funcFon support – file_put_contents, fwrite • permissions & viewer • large file support • result structure asserFon • relaFve paths are not supported – chdir(), realpath() too mikey179/vfsStream 4
- 70. 4
- 71. In-memory database tests FuncFon call expected, not enFre stack. Fast, but narrow PHPUnit_Extensions_Database_TestCase -‐ too complex • Data sources: XML, CSV, mysql dump • DB virtualizaFon: PDO + SQLite See also: H2 Database, HyperSQL, Apache Derby sebasFanbergmann/dbunit liip/LiipFuncFonalTestBundle 5
- 72. 5
- 73. Component testing Component / library -‐ coherent, replaceable uFlity classes For example: -‐ Form builder (please don’t) -‐ Validator -‐ Logger -‐ Tree builder -‐ Image resizer -‐ File storage abstracFon -‐ CSV/XML generator/parser man wife cat sofa sleepOn() clean() destroy() 6
- 74. TEST DESIGN
- 75. Speed - order by types PHPUNIT.xml • Unit tests – General project unit tests (validators, DB adapter) – Unit tests per component (models) • IntegraFon tests – External – Internal API – Internal with DB isolaFon – Performance + Security – Data integrity • UI tests – Screenshoong Time
- 76. Modularity /controllers/arFcle.php /models/arFcle.php /tests/integraFon/arFcleControllerTest.php /tests/unit/arFcleModelTest.php /arFcle/controller/ /article/model/ /article/modelTests/ /article/controllerTests/ Divide your app into modules -‐ see what tests you lack where
- 77. Annotations jump between funcFon and test @see ArFcleController::postCreate @Fcket T-‐1000 + JIRA API (PHPUnit supports) @group producFon @group failed phpunit -‐-‐testsuite unit
- 78. More than code coverage Code coverage = you tried to test Delta coverage – what test adds MutaFons help to find • tests that lack enough asserts • fragile code henrikbjorn/PhpSpecCodeCoverageExtension
- 80. What else to test? • Frontend • Stress / Load • SQL procedures • Accessibility tests • GlobalizaFon
- 81. Track your mistakes • Monitoring tests failures • Coverage trend • Test quality Monitoring code health: • Complexity • Code standard compliance Block8/PHPCI Halleck45/PhpMetrics squizlabs/PHP_CodeSniffer
- 82. Awareness Clarity Tests Architecture
- 83. QUESTIONS?