Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How we tested our code "Google way"

4,417 views

Published on

Experience report about Playtika Infrastructure team

Published in: Software
  • Be the first to comment

How we tested our code "Google way"

  1. 1. HOW WE TESTED OUR CODE “Google way” Oleksiy Rezchykov @twincengray #xpdays 2
  2. 2. About me Software Test Engineer Last 7 years working with Java and Spring XP/Agile/Lean practitioner Lazy Pragmatic programmer @twincengray #xpdays 3
  3. 3. Where it all began @twincengray #xpdays 4
  4. 4. Test size concept My test is using mocks – is it still a Unit test? I want to test concurrency, my test is using thread sleeps – can I consider this as Functional test? My test which is using in-memory DB running for about a second – Is it an Integration test? @twincengray #xpdays 5
  5. 5. Test size concept Small tests Medium tests Large tests Enormous tests Time Goals (per method) Execute in less than 100 ms Execute in less than 1 sec Execute as quickly as possible Execute as quickly as possible Time Limits Enforced Kill small test targets after 1 minute Kill medium test targets after 5 minutes Kill large test targets after 15 minutes Kill enormous test targets after 1 hour Source: http://books.google.com.ua/books?id=vHlTOVTKHeUC&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q=small%20tests&f=false @twincengray #xpdays 6
  6. 6. Test size concept Resource Large Medium Small Network Services (Opens a Socket) Yes localhost only Mocked Database Yes Yes Mocked File System Access Yes Yes Mocked Access to User-Facing Yes Discouraged Mocked Systems Invoke Yes Discouraged No Syscalls Multiple Threads Yes Yes Discouraged Sleep Statements Yes Yes No System Properties Yes Yes No Source: http://books.google.com.ua/books?id=vHlTOVTKHeUC&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q=small%20tests&f=false @twincengray #xpdays 7
  7. 7. Test size concept Small tests lead to code quality. Medium and large tests lead to product quality. @twincengray #xpdays 8
  8. 8. Context Playtika facts Founded in 2010 Multiplatform social games developer Headquarters in Tel-Aviv Offices in US, Canada, Ukraine, Belarus, Romania and Argentina Industry facts Social Casino segment is growing faster than other Social Games segments Mobile platforms is the main focus for developers More about Playtika and Social Casino genre: http://www.slideshare.net/eladkushnir/final-ccsf-elad-kushnir @twincengray #xpdays 9
  9. 9. Context My team is called Infra (Infrastructure) We are providing infrastructure (logic, which is mostly not related to the game itself) for different games (Slotomania, Caesars Casino), platforms (Web, Mobile) and social networks (FB, Yahoo, e.t.c.) SOA Services Microservices clusters Spring Framework, SQL, NoSQL, e.t.c. @twincengray #xpdays 10
  10. 10. One day … @twincengray #xpdays 11
  11. 11. Starting point “Unit” “Functional” “Integration” Present for each service Yes No No Run on CI Yes No No Coverage Depends on legacy code %* Some feature acceptance cases “Smoke” suite, tricky cases Environment set-up needed No Yes Yes Duration Up to 35 sec** Up to 10 min n/a*** *- All new services was written using TDD so coverage was quite high ** - In some services we had to test stored procedures *** - Those tests were heavily depending on environment configuration @twincengray #xpdays 12
  12. 12. Small tests JUnit tests/Mockito tests Data driven unit/functional tests (framework) Short tests with Spring Context Even stored procedures tests using embedded MySQL @twincengray #xpdays 13
  13. 13. A little bit more about our services Service is a regular Spring/Spring MVC application Each service provides Library and Module (Plugin) for other services to communicate with it REST JSON and Hessian calls are used for communication Configuration Service is used to load/reload settings during start/runtime Separation between feature toggling and user A/B testing New services are developed using Spring Boot @twincengray #xpdays 14
  14. 14. “Functional” to Medium tests So called “Functional” tests Were not run on CI Required pre set up environment (DB’s) Service started in a standalone container External “fake” Configuration Service is used Duration - up to 5min @twincengray #xpdays 15
  15. 15. “Functional” to Medium tests Our improvements Generally we are trying not to test anything that does not belong to specific service DB migration to prepare DB’s state anywhere anytime Meta information about the environment needed No external processes - embedded container and Configuration No @DirtiesContext @twincengray #xpdays 16
  16. 16. Medium tests as we have them now Concurrency tests Cache tests Service Configuration testing Service API testing Service module/library testing Persistence layer testing with real life data Time - less than 2min @twincengray #xpdays 17
  17. 17. DEMO @twincengray #xpdays 18
  18. 18. “Integration” to Large tests So called “Integration tests” Integration tests used Docker to store “production snapshot” Was not run on CI Not any computer could handle such a large image Each small change required some efforts updating the image @twincengray #xpdays 19
  19. 19. “Integration” to Large tests Improvements Real Conf. Manager, Real persistence, Real services (except payments and stuff like that) Evolution – Docker, Ansible, Gradle OPS will do their stuff, we’ll do ours @twincengray #xpdays 20
  20. 20. Large tests as we have them now Regression End-2-End scenarios Acceptance tests for new features @twincengray #xpdays 21
  21. 21. DEMO @twincengray #xpdays 22
  22. 22. Build pipeline Small • Small tests run • Code analyzers • Artifact publish Medium • Medium tests for Game X • Medium tests for Game Y • This build is triggered by Git push Large • Large tests run • Manual trigger Release • Release artifact publish • Snapshot increment @twincengray #xpdays 23
  23. 23. QA Test plan creation (Large, medium tests), AC for the new features Manual testing (playing games basically) Automating their work @twincengray #xpdays 24
  24. 24. Production @twincengray #xpdays 25
  25. 25. Plans/Challenges Test run infrastructure Load tests for ALL pain points to be run on regular basis Back-up compatibility tests (Modules/Libraries vs Services) Make large tests run automatically Test results specified in artifact metadata Environment provisioning in real time using TC as trigger @twincengray #xpdays 26
  26. 26. Summary Testing is needed on all application levels Each level of testing has it’s own goal It us useful to group tests according to the run time and not only the test context Opensource your code in the organization If your code could be used elsewhere compose a library and write a HOWTO Wiki page Check Wiki before writing code Do not reverse your testing pyramid Improve, innovate, improvise @twincengray #xpdays 27
  27. 27. Thank you! orezchykov@playtika.com.ua oleksiy.rezchykov@gmail.com @twincengray We are hiring! @twincengray #xpdays 28

×