1. Киев 2017
Первый в Украине фестиваль тестирования
E2E testing: Challenges and
recommendation in test process
organization for cross-integration teams
Vlada Benyukh

2. Киев 2017
Первый в Украине фестиваль тестирования
Speaker info
Vlada Benyukh, Ukraine
QA Manager/QA Consultant at Ciklum, TCoE
•7 years in QA
•Manual QA Engineer, QA Manager, mentor
•Certified: SAFe, ICAgile, ISTQB
in/vlada-benyukh

3. Киев 2017
Agenda
1. E2E testing in cross-integration teams, What is it?
2. Test process and approach
3. Infrastructure management
4. Collaboration
5. Test team structure
6. Automation setup
7. Questions
E2E testing: Challenges and recommendation in test process organization for cross integration teams

4. Киев 2017
What is E2E testing?
Main goal is to identify system dependencies and to ensure
that the data integrity is maintained between various
applications/components correctly
E2E testing: Challenges and recommendation in test process organization for cross integration teams

5. Киев 2017
What is E2E testing in cross-
integration teams?
Expectation
E2E testing: Challenges and recommendation in test process organization for cross integration teams

6. Киев 2017
What is E2E testing in cross-
integration teams?
Process and approach
Infrastructure management
Communication and
collaboration
Most of the challenges in a E2E team relates
to cross-integration work:
E2E testing: Challenges and recommendation in test process organization for cross integration teams

7. Киев 2017
Test Process challenges
E2E testing: Challenges and recommendation in test process organization for cross integration teams

8. Киев 2017
Test Process challenges – Team
structure
E2E testing: Challenges and recommendation in test process organization for cross integration teams

9. Киев 2017
Why team structure was a
challenge?
 3 QA Automation
Specialists
 1 Automation QA Lead
E2E testing: Challenges and recommendation in test process organization for cross integration teams

10. Киев 2017
1. Understanding of the systems flows
2. Accesses to the systems
3. Requirements for end-to-end flows
4. Test Design for end-to-end flows
5. Test data for each system
6. Test environment readiness
Why team structure was a
challenge?
E2E testing: Challenges and recommendation in test process organization for cross integration teams

11. Киев 2017
Do not change team structure until challenges resolved…
Why team structure was a
challenge?
E2E testing: Challenges and recommendation in test process organization for cross integration teams

12. Киев 2017
Test Process challenges – Solution
1. Systems:
 7 separate systems which integrated into 1 CRM
 1 common aggregation level for all systems
 1 common integration level for all systems
E2E testing: Challenges and recommendation in test process organization for cross integration teams
2. Many development and test teams (in different
countries and time zones)

13. Киев 2017
Test Process challenges – Solution
Gather knowledge about the systems!
 Get access to the systems
 Understand systems specific and
functionality
 Get information about architecture
overview and technologies used
 Figure out scope and readiness dates
of the systems
E2E testing: Challenges and recommendation in test process organization for cross integration teams

14. Киев 2017
Test Process challenges – Solution
Gather knowledge about key representatives in
each system!
 System teams
 Integration development system teams
E2E testing: Challenges and recommendation in test process organization for cross integration teams

15. Киев 2017
Ask key representatives about
end-to-end flows
Gather, document and approve
acceptance criteria
Prioritize acceptance criteria
Requirements challenges – What
was the plan?
E2E testing: Challenges and recommendation in test process organization for cross integration teams

16. Киев 2017
No understanding and goal of
end-to-end team
No requirements for end-to-end
flows
No responsible person for end-
to-end team
Requirements challenges – What
was in reality?
E2E testing: Challenges and recommendation in test process organization for cross integration teams

17. Киев 2017
Requirements challenges
How to get requirements …
E2E testing: Challenges and recommendation in test process organization for cross integration teams

18. Киев 2017
Get system requirements
Explain the goal of end-to-end
testing
Review system requirements
Document, prioritize and
approve with business side
Requirements challenges – Solution
E2E testing: Challenges and recommendation in test process organization for cross integration teams

19. Киев 2017
Requirements challenges – Summary
E2E testing: Challenges and recommendation in test process organization for cross integration teams
1. General knowledge about system were gathered
2. Scope and priorities for end-to-end were identified
3. Detailed Requirements for end-to-end flows were
documented

20. Киев 2017
Test Design challenges
E2E testing: Challenges and recommendation in test process organization for cross integration teams

21. Киев 2017
• No technical skills to create test
documentation based on the best
practices
• No input test data for each
system
Test Design challenges - Problem
E2E testing: Challenges and recommendation in test process organization for cross integration teams

22. Киев 2017
Test Design challenges – Solution
• Explain the necessity of manual QA
specialists to senior management
• Involve manual QA Engineers from
system teams in test cases creation
• Motivate automation team to
collaborate with manual QA
Engineers
E2E testing: Challenges and recommendation in test process organization for cross integration teams

23. Киев 2017
How motivate automation team to create test
cases?
Test Design challenges – Solution
E2E testing: Challenges and recommendation in test process organization for cross integration teams

24. Киев 2017
 Explain the temporarily situation and
what can happen without end-to-end
team involvement
 Be involved and continue to work on
automation direction together
Test Design challenges – Solution
E2E testing: Challenges and recommendation in test process organization for cross integration teams

25. Киев 2017
 Ask involved Manual QA Engineers to
be responsible for test cases creation
 Define test case template which will
be used by manual and automation
QAs
 Structure test cases by functionality
and priorities
Test Design challenges – Solution
E2E testing: Challenges and recommendation in test process organization for cross integration teams

26. Киев 2017
Test Design challenges – Summary
E2E testing: Challenges and recommendation in test process organization for cross integration teams
1. Responsible team members for test cases creation
were defined
2. Structure and template were identified
3. Flows for end-to-end were covered by test cases

27. Киев 2017
Testing approach agreement across
teams
Agreements should be aligned before starting test
execution activities
E2E testing: Challenges and recommendation in test process organization for cross integration teams

28. Киев 2017
Testing approach agreement across
teams
Discuss process/approach with
teams
Involve all stakeholders in
process/approaches discussions
Align procedures across teams
Document all agreements
E2E testing: Challenges and recommendation in test process organization for cross integration teams

29. Киев 2017
Testing approach – Testing Order
Smoke
Acceptance critical flows
Regression
UAT
E2E testing: Challenges and recommendation in test process organization for cross integration teams

30. Киев 2017
Testing approach – Testing Order
Type of Test Scope Outcome
Smoke One End-to-end flow
for most critical
functionality
Blocker bugs
Acceptance critical
flows
End-to-end flows P1/P2/P3/P4
bugs
Regression P1/P2 P1/P2 test cases P1/P2 bugs
Regression P3/P4 P3/P4 test cases P3/P4 bugs
UAT User Acceptance
Testing
P1/P2 and
P3 bugs
E2E testing: Challenges and recommendation in test process organization for cross integration teams

31. Киев 2017
Testing approach – Defect
management
RAISE ON E2E
BOARD
DISCUSS ON
BUG TRIAGE
DUPLICATE
AND ASSIGN
TO A PROPER
TEAM
ANALYSE CASE
WHEN DEFECT
HAPPENED
E2E testing: Challenges and recommendation in test process organization for cross integration teams

32. Киев 2017
Test Execution challenges - Solution
• Ask Manual QA representatives focus their
attention on execution activities from manual
point of view
• Involve Automation QA representatives in
technical details analysis to find a workaround
for infrastructure challenges
!!! Infrastructure challenges will be covered on the next section!!!
E2E testing: Challenges and recommendation in test process organization for cross integration teams

33. Киев 2017
Infrastructure challenges
E2E testing: Challenges and recommendation in test process organization for cross integration teams

34. Киев 2017
Test environment requirements for
verification of end-to-end flows:
 All items should be processed from
start system to end system
automatically
Infrastructure challenges
E2E testing: Challenges and recommendation in test process organization for cross integration teams

35. Киев 2017
But what we had….
Infrastructure challenges
E2E testing: Challenges and recommendation in test process organization for cross integration teams

36. Киев 2017
Based on the system configurations, limitations and application
specific, items are integrated from start system to end
application into two different ways:
 Semi-automatically
 Non-automatically
Infrastructure challenges - Problem
E2E testing: Challenges and recommendation in test process organization for cross integration teams

37. Киев 2017
Automation QA team was involved to resolve next items:
 Data formats and mappings for items processing
 Access to file transfer folders which can help to
make items processed to end system
 Get permissions to monitor messages to diagnose
the problem
Infrastructure challenges - Solution
E2E testing: Challenges and recommendation in test process organization for cross integration teams

38. Киев 2017
Infrastructure challenges – semi-
automatic procedure
Creation
Status
Verification
Verification
in CRM
Automation QAsManual QAs
• Create items in start
system
• Verify that details
about created items
are stored properly
• Track the processing
status and monitor
issues
• Verify that item details are displayed in end system
E2E testing: Challenges and recommendation in test process organization for cross integration teams

39. Киев 2017
Infrastructure challenges – non-
automated procedure
Creation
Status
Verification
Verification
in CRM
• Create items in start
system
• Verify that details about
created items are stored
properly
• Generate data formats
which required for
processing
• Verify that item details are displayed in end system
Manual
Processing
• Process and track the
processing status and
monitor issues
Automation QAsManual QAs
E2E testing: Challenges and recommendation in test process organization for cross integration teams

40. Киев 2017
Infrastructure challenges –
Summary
E2E testing: Challenges and recommendation in test process organization for cross integration teams
1. End-to-end flows were executed successfully
2. End-to-end scenarios found bugs and discrepancies
in the systems functionality

41. Киев 2017
Collaboration challenges
E2E testing: Challenges and recommendation in test process organization for cross integration teams

42. Киев 2017
Collaboration challenges – Cross
teams
What is it if NO:
1. Frequency and scope for updates
2. Release readiness dates
3. Changes in the systems
4. Possible impact and risks for end-to-end
flows
1. Unexpected issues occur while testing
2. Possibly unexpected configuration issues
3. Dependencies are not tracked and mitigated
in time
E2E testing: Challenges and recommendation in test process organization for cross integration teams

43. Киев 2017
How to avoid issues?
Collaboration challenges – Cross
teams
E2E testing: Challenges and recommendation in test process organization for cross integration teams

44. Киев 2017
Regular cross-teams Sync Ups
Participants:
 Project management teams
representatives
 Integration Program Manager
 Team Tech Leads
 E2E QA Lead
Goals:
 Define scope for updates
 Get system updates dates
 Analyze and mitigate risks that might
affect E2E testing before testing starts
 Escalate critical problems
 Discuss other common problems and
questions
E2E testing: Challenges and recommendation in test process organization for cross integration teams

45. Киев 2017
Collaboration challenges – cross-
teams progress understanding
What is it if NO:
1. Scope and activities for end-to-end testing
2. Understanding of the progress end-to-end
testing
3. Risks and blockers for end-to-end testing
4. Release readiness dates
E2E testing: Challenges and recommendation in test process organization for cross integration teams

46. Киев 2017
How to avoid issues?
Collaboration challenges – cross-
teams progress understanding
E2E testing: Challenges and recommendation in test process organization for cross integration teams

47. Киев 2017
Regular E2E team Sync Up
Participants:
 Integration Program Manager
 E2E QA Lead
 E2E key QA specialists
Goals:
 Show the progress
 Escalate critical problems
 Discuss other common problems and
questions
E2E testing: Challenges and recommendation in test process organization for cross integration teams

48. Киев 2017
Example of Daily Progress Visibility
E2E testing: Challenges and recommendation in test process organization for cross integration teams

49. Киев 2017E2E testing: Challenges and recommendation in test process organization for cross integration teams
Example of Daily Progress Visibility

50. Киев 2017
Example of Daily Progress Visibility
E2E testing: Challenges and recommendation in test process organization for cross integration teams

51. Киев 2017
Collaboration challenges – Summary
E2E testing: Challenges and recommendation in test process organization for cross integration teams
1. Regular updates on system updates and readiness were
completely clear
2. Risks were identified and mitigated regularly
3. Critical problems were escalated in time
4. Test progress and project health were synchronized
between teams

52. Киев 2017E2E testing: Challenges and recommendation in test process organization for cross integration teams
Test Team structure

53. Киев 2017
QA Lead
- QA Process and
Approaches
implementation
- Cross team ongoing
QA work planning and
monitoring
- Regular QA reporting
- People management
between teams
- Support in building
and aligning QA
automation approach
Manual QAs
-Main knowledge
keepers in the systems
- Responsible for test
cases preparation and
execution semi-
automated procedure
- Escalate issues
- Define dependencies
and quality risks. Inform
about them in time
Automation QAs
- Support in test cases
creation
- Responsible for
incomplete test
procedure processing
testing
- Responsible for test
automation
implementation
- Escalate issues
- Define dependencies
and quality risks.Inform
about them in time
Test Team structure - Solution
E2E testing: Challenges and recommendation in test process organization for cross integration teams

54. Киев 2017
QA Lead
QA Manual QA AutomationQA Manager
QA Manual Lead
QA Manual Lead
Middle QA
Middle QA
Senior QA
Middle QA
Middle QA
Senior QA Senior QA
Automation team
Test Team structure - Solution
Team 1 Team 2
E2E testing: Challenges and recommendation in test process organization for cross integration teams

55. Киев 2017
Automation challenges
E2E testing: Challenges and recommendation in test process organization for cross integration teams

56. Киев 2017
Automation challenges – What was
it?
2. Complex and obsolete architecture
1. End-to-end flow is not fully automated
3. Many external and internal dependencies
E2E testing: Challenges and recommendation in test process organization for cross integration teams

57. Киев 2017
Automation challenges – What was
solution?
1. Get access to systems via API to create items
2. Generate required data formats and make to be processed
it automatically
3. Automate validation of items in common CRM
E2E testing: Challenges and recommendation in test process organization for cross integration teams

58. Киев 2017
Automation challenges – Order to
automate
Smoke Regression
Acceptance
flows
E2E testing: Challenges and recommendation in test process organization for cross integration teams

59. Киев 2017
Recommendations
1. Do not change team structure until all
important challenges were not clarified
2. Be proactive and look for the solutions
3. Do not try to resolve all issues by yourself
4. Collaborate with stakeholders and systems
team members
E2E testing: Challenges and recommendation in test process organization for cross integration teams

60. Киев 2017
Questions
E2E testing: Challenges and recommendation in test process organization for cross integration teams