Continuous integration умеет запускать тесты. Для этого его и придумали. Однако, как всегда, есть нюансы, связанные с теми или иными видами тестов. На митапе я расскажу про особенности настройки CI для работы с нагрузочным, приёмочным, интеграционным, UI- и Smoke-тестированием, проверкой корректности установки и апгрейда. Поговорим о том, на какие грабли можно при этом наступить и как их обойти. Avito Automation Meetup (26.08.2017) https://avitotech.timepad.ru/event/542380/

1. Running tests in
Continuous Integration
—
Sergey Pak
Jetbrains

2. Test Automation
—
Why?

3. Test Automation
—
1) Save resources

4. Test Automation
—
1) Save resources
2) Faster feedback

5. Test Automation
—
1) Save resources
2) Faster feedback
3) Continuous testing

6. Continuous Testing
—
1) No manual test execution

7. Continuous Testing
—
1) No manual test execution
2) Tests in VCS

8. Continuous Testing
—
1) No manual test execution
2) Tests in VCS
3) Automated tests are a part of the application code

9. Basic Scenario
—
Prepare environment

10. Basic Scenario
—
Execute tests

11. Basic Scenario
—
Cleanup

12. Prepare environment
—
1. Build Agent

13. Prepare environment
—
1. Build Agent
2. Prefer containers (Docker) or VM template

14. Prepare environment
—
1. Build Agent
2. Prefer containers (Docker) or VM template
3. Configure environment before build

15. Prepare environment
—
1. Build Agent
2. Prefer containers (Docker) or VM template
3. Configure environment before build
4. Privileges for BuildAgent
1. Prefer roles for VM

16. Clean-up environment
—
1. Write in build directory

17. Clean-up environment
—
1. Write in build directory
2. Cleanup created files, rollback modifications
1. Prefer VCS functionality
2. Swabra

18. Clean-up environment
—
1. Write in build directory
2. Cleanup created files, rollback modifications
1. Prefer VCS functionality
2. Swabra
3. Kill created processes (Swabra)

19. Clean-up environment
—
1. Write in build directory
2. Cleanup created files, rollback modifications
1. Prefer VCS functionality
2. Swabra
3. Kill created processes (Swabra)
4. Investigation takes time

20. Tests Execution
—
• Frameworks (Selenium) -> Utility (Maven, MS Build)
• Tools (Testcomplete) -> Plugin/script
1. Both can be run in CI

21. Tests Execution
—
• Frameworks (Selenium) -> Utility (Maven, MS Build)
• Tools (Testcomplete) -> Plugin/script
1. Both can be run in CI
2. Store tests in XML

22. Tests Reporting
—
1. Standard runners

23. Tests Reporting
—
1. Standard runners
2. Report tabs

24. Tests Reporting
—
1. Standard runners
2. Report tabs

25. Tests Reporting
—
1. Standard runners
2. Report tabs
3. Custom format:
1. Transformation + Report
consumer (XML Report
Processing)
2. Service messages
echo "##teamcity[testStarted name=‘test.MyTest']"
echo "##teamcity[testStdOut name='test.MyTest' out='My Test output']"
echo "##teamcity[testFinished name='test.MyTest' duration='150']"
echo "##teamcity[testStarted name='test.MyFailedTest']"
echo "##teamcity[testFailed name='test.MyFailedTest' message=‘test failure'
details='MyFailedTest has failed']"
echo "##teamcity[testFinished name='test.MyFailedTest' duration='150']"

26. Run tests. Reuse code
—
1. Parameterization
2. Templates
3. Agent requirements
1. No gold agents*
* Except performance tests

27. Run tests. Reuse code
—
1. Parameterization
2. Templates
3. Agent requirements
1. No gold agents*
Demo
* Except performance tests

28. Performance tests
—
1. Gold agent + Gold build

29. Performance tests
—
1. Gold agent + Gold build
2. Failure condition by time
1. Specify exact test names/wildcard

30. Performance tests
—
1. Gold agent + Gold build
2. Failure condition by time
1. Specify exact test names
3. Statistics (per build/test/etc.) over the time

31. Performance tests
—
1. Gold agent + Gold build
2. Failure condition by time
1. Specify exact test names
3. Statistics (per build/test/etc.) over the time
Demo

32. Build dependencies
—
1. Snapshot dependencies
2. Build Chain

33. Build dependencies
—
1. Snapshot dependencies
2. Build Chain
3. Build promotion
1. Permissions
2. Tags

34. Build dependencies
—
1. Snapshot dependencies
2. Build Chain
3. Build promotion
1. Permissions
2. Tags
4. Artifact dependencies

35. Build dependencies
—
1. Snapshot dependencies
2. Build Chain
3. Build promotion
1. Permissions
2. Tags
4. Artifact dependencies
5. Schedule trigger (+ watched builds + tags)

36. Build dependencies
—
1. Snapshot dependencies
2. Build Chain
3. Build promotion
1. Permissions
2. Tags
4. Artifact dependencies
5. Schedule trigger (+ watched builds + tags)
6. Shared resources

37. Test Results
—
1. Composite build (2017.2)

38. Test Results
—
1. Composite build (2017.2)
2. Report tabs

39. Test Results
—
1. Composite build (2017.2)
2. Report tabs
3. Test history

40. Test Results
—
1. Composite build (2017.2)
2. Report tabs
3. Test history
4. Flacky tests

41. Test Results
—
1. Composite build (2017.2)
2. Report tabs
3. Test history
4. Flacky tests
5. Investigations

42. jetbrains.com
Thank you
for your attention
—