Why would you like to become a "Consultant". Where does it begin? And what to do if you're selected to consult a big Bank abroad :) Successful Test Automation and Performance Testing Consultancy case study will demonstrate the whole process from finding the client through assessment, evaluation, and implementation to a happy client accepting the project.

1. Тема доклада
Тема доклада
Тема доклада
KYIV 2019
Mikhail Chub
TEST AUTOMATION CONSULTANCY
QA CONFERENCE #1 IN UKRAINE

2. 2
HELLO
Head of QA Competence, Infopulse Ukraine
Lecturer, KPI
MIKHAIL CHUB
16 years
in QA
5 years
in Education
ISTQB
ATM
Hobby mikhailchub mikhail.chubqamania

3. NEW OPPORTUNITIES
Stronger Boss – better loot
BOOST EXPERTISE
Work harder – get stronger
MOM WILL BE PROUD
Attaboy!
APPLY ALL
KNOWLEDGE
Right now!
WHY
CONSULTING

4. CONTENTS
4
START
ABOUT
PROJECT & CLIENT
Few words about project specifics,
client and the scope.

5. CONTENTS
5
“WARM” PHASE
CLOSURE
• Acceptance
• Maintenance
“HOT” PHASE
OPERATION
• Assessment
• Tools Evaluation
• Proof of Concept
• Training
• Pilot
• Strategy
“COLD” PHASE
INITIATION
• Opportunity
• Waiting
• Hiring

6. CONTENTS
6
END
AFTER THE PROJECT
WHAT’s NEXT
• New projects
• Build competence
• Adjusted templates
LESSONS LEARNED
SUMMARY
• What was good
• What was NOT so good

7. ABOUT
CLIENT & PROJECT
CONSULTANCY

8. ABOUT CLIENT
8
.
Bank
under transformation

9. ABOUT PROJECT
9
.
Bank
under transformation

10. ABOUT PROJECT
10
How to make it right
AUTOMATED
TESTING
Teach how to do it right
TRAINING
How to make it right
PERFORMANCE
TESTING
Implement few tests together with
the local team so they are able to
copy-paste progress further
PILOT
Evaluate and select the best
suitable tool
PROPER TOOLS
Define strategy for auto and perf
tests development
SYSTEMATIC
APPROACH

11. INITIATION
CONSULTANCY

12. OPPORTUNITY
12
FIND
RFP
Make sure your Sales
understand what to
look for and what to
propose
UNDERSTAND
DECOMPOSITION
Ask questions, recall your past projects,
identify high-level activities
ESTIMATE
PROJECT
PLAN
Estimate activities using
your expertise and
statistical methods
PROVE
CASE
STUDY
Collect proofs of
your expertise,
compile them to a
convincing slides
WIN
PRESENT
Present your approach
personally to the potential
client, be fair and
convincing

13. ESTIMATION
13
. . .

14. PROJECT PLAN
14
JAN FEB MAR APR MAY JUN JUL
ASSESSMENT1
TOOLS EVALUATION, POC2
AUTOTESTING TRAINING3
AUTOTESTING PILOT4
PERFTESTING TRAINING5
PERFTESTING PILOT6
MAINTENANCE7

15. OPPORTUNITY
15
FIND
RFP
Make sure your Sales
understand what to
look for and what to
propose
UNDERSTAND
DECOMPOSITION
Ask questions, recall your past projects,
identify high-level activities
ESTIMATE
PROJECT
PLAN
Estimate activities using
your expertise and
statistical methods
PROVE
CASE
STUDY
Collect proofs of
your expertise,
compile them to a
convincing slides
WIN
PRESENT
Present your approach
personally to the potential
client, be fair and
convincing

16. WAITING
16

17. HIRING
17
Explain complicated things
in simple words
ADVISORY SKILLS
Able to establish plans,
milestones, KPIs and follow
them
MANAGEMENT SKILLS
Making people happier
rather than writing good
autotests
COMMUNICATION SKILLS
TECHNICAL SKILLS
Solid hands-on experience

18. INTERVIEWING
18

19. INTERVIEWING
19

20. TEAM MEMBERS
20
AutoTesting Consultant
Does magic with automated tests.
Principal Consultant
Recruits team of superheroes and makes sure
that other superheroes remember about the
Great Goal.
PerfTesting Consultant
Uses science to hurt.

21. OPERATION
CONSULTANCY

22. ASSESSMENT
22
COMMUNICATION

23. ASSESSMENT
23
0201
03
400-500
ACTIVE USERS
Core Banking
INTEGRATIONS
Open Account, Top-up Mobile
MAIN USE CASE
Give me all your money
PURPOSE
WinForms
TECHNOLOGY
Each 3 months
RELEASE FREQUENCY
STRUCTURE

24. ASSESSMENT
24
AUTOMATED TESTS
Number of scripts implemented
MILESTONES ARE MET
PROJECT PHASES FINISHED
IN TIME
Priorities defined, Tools selected,
Scope agreed
TOOLS EVALUATION MATRIX
Detailed tools comparison
SCRIPTS IN GIT
Developed and stored
KPI
METHODOLOGY
Written and agreed

25. TOOLS EVALUATION
25
TEST COMPLETE RANOREX WIN APP DRIVER COLUMN ONE
Technology
COMPARISON CATEGORY
CI
IDE, Debug
Maintainability
Execution, Reporting
Support
TOTAL
AUTOTESTING

26. TOOLS EVALUATION
26
LOAD RUNNER LOAD UI WAPT LOAD IMPACT
Technology
CATEGORY
CI
IDE, Debug
Maintainability
Execution,
Reporting
Support
TOTAL
NEOLOADJMETER
PERFTESTING

27. TRAINING
PRACTICE
Gradually rolling to Pilot
implementation
THEORY
For PerfTesting mostly
TRAINING PROGRAM
Intensive: 8d – Auto, 6d - Perf
SELF-STUDY MATERIALS
Video, Books, Forums, Chats,
Articles, Other links

28. PILOT
28
AUTOTESTING
+ =>

29. PILOT
29
PERFTESTING

30. PILOT
30
PERFTESTING
SATURATION POINT
in Client Bank

31. CLOSURE
CONSULTANCY

32. ACCEPTANCE
• According to defined earlier:
– Milestones
– Deliverables
– KPIs
• Accept what is met
• Agree on what is NOT met
32

33. ACCEPTANCE
Phase Artifacts Milestones KPIs
1. Assessment 1.1. Communication Matrix + 1.1. Intro Workshop + 1. Milestones are met +
1.2. Project Priorities + 1.2. Meetings +
1.3. Priorities +
2. Evaluation 2.1. Tools Evaluation Matrix + 2.1. AutoTool selected, agreed +
2.1. >= 2 Test Scenarios are successfully automated for chosen
system +
2.2. Costs comparison + 2.2. PerfTool selected, agreed + 2.2. >= 2 PerfTest Scenarios are designed for chosen system +-
2.3. Time-to-market comparison +- 2.3. Project scope agreed +
3. Methodology 3.1. Methodology document + 3.1. Methodology drafted + 3. Milestones are met +-
3.2. Training Program (Auto) + 3.2. Methodology finalized +
3.3. Training Program (Perf) + 3.3. Methodology agreed X
4. Autotesting Training 4.1. Projects from students in VCS + 4.1. Customer: classroom prepared + 4. Automation exam marks >= 65% X
4.2. Autotesting training projects completed by students +
5. Autotesting PoC 5.1. Implemented PoC project in VCS + 5.1. Autotests Architecture designed + 5.1. Implemented 5-10 P1 autotests for chosen system +
5.2. Test Execution Reports + 5.2. Test Data generator developed +- 5.2. Able to run autotests from CI +
5.3. Autotests Adopted for CI + 5.3. Atomation Test Reports are understandable +
5.4. Autotests Adopted for Reporting +
6. Perftesting Training 6.1. Training presentation + 6.1. Perftest Homeworks done + 6. Perftest exam marks >= 65% +
6.2. Team Profiling doc +- 6.2. Perftest theory exam conducted +
6.3. Perftest training final marks are set +
7. Perftesting PoC 7.1. Test Scripts + 7.1. Perftest activities on practice + 7.1. 5-10 Perftests are implemented for chosen system +
7.2. Test Data + 7.2. List of future PerfTest candidates prepared + 7.2. Able to run perftests from CI +
7.3. Test Reports + 7.3. Test Reports are understandable +
7.4. Lessons Learnt +
33

34. MAINTENANCE
• Weekly calls
• to solve problems
34

35. WHAT’s NEXT
CONSULTANCY

36. BASELINE
36
Plum gummi bears. Apple
pie yummy cake with candy
SERVICE QUALITY
Sugar gummi bears. Apple
pie yummy cake with candy
TRANSPARENT RELATIONSHIP
Sugar plum gummi bears.
pie yummy cake with candy
TEST PROCESS REVIEW

37. ADDITIONAL OPPORTUNITIES AND SERVICES
37
BUILD COMPETENCE CENTER
Help to establish competence
MANUAL TESTING TRAININGS
Help to increase competence on the
Basic and Advanced level
THREE NEW OPPORTUNITIES
• Solutions Development
• DevOps Consultancy
• Additional Testing Services
TEACH TEACHERS
Implement methodology for further
competence growth

38. SUMMARY
CONSULTANCY

39. PROBLEMS
39
PROBLEMS
AND THEIR RESOLUTION
CONSULTANCY
• Shared
• Hardware isn’t close to PROD
ENVIRONMENT
• Only through terminal
• No VPN access
ACCESS
• Hard to allocate internally
• Hard to find externally
HIRING

40. PROBLEMS
40
PROBLEMS
AND THEIR RESOLUTION
CONSULTANCY
DOUBTSAUTHORITY
• Local team
• UI localization
• Test Cases
LANGUAGE

41. LESSONS LEARNED
41
TEMPLATES
Reuse saves time
PROJECT PLAN
Is a must
RESPECT –
IS EVERYTHING
Уважение клиента
ЛУЧШЕЕ –
ВРАГ ХОРОШЕГО
Иногда можно быть проще
БОЛЕЗНЬ
А НЕ СИМПТОМЫ
До автотестов еще
нужно дойти
CLASSIC TRAINING
WITH THEORY
Теория + Практика
лучше чем одна Практика

42. THANK YOU
QUESTIONS?