Automated Testing Best Practices and Tips. QA Automation and Test automation process flow. Continuous Delivery, Continuous integration and Test-driven development in cloud-based system. Automatic Deployment and Post-deployment verification. Agile development and quality assurance. Cloud-based telephony service.

1. How to Make Test
Automation
For Cloud-Based System
by Nick Babich

2. Telephony
Automation
Team
Alexey Georgy Alexandra
Nikita Alexey Igor
2

3. Contents
1. RingCentral product and system specifics.
2. Automation: What-When-Where-How and Why
3. Questions and Answers
3

4. 4
Classic PBX System

5. 5
Cloud Telephony System

6. RingCentral: Cloud Business Communications
6

7. RingCentral Service
Telephony is a cornerstone service for the RingCentral. It consist of
many different features and each new release brings more to
production.
7

8. 8
Quality
Cost
Delivery

9. Automation
What?
When?
Where?
How?
Why?
9

10. What?
Type of Automation 10

11. 11

12. Testing Types
● Functional Tests
● Acceptance Tests
● A/B Tests
● Unit Tests ● Stress Tests
● Security Tests
Business Facing
Technology
Facing
Support
Programming
CritiqueProduct
12

13. 13
Most Common Use-Case

14. How to Define Scope for Automation
● Automate the most critical cases (P0 cases, especially
one which are hard-to-execute manually).
● Automate the most frequently used scope (included in
regular regression cycles).
● Automation for the most automatable scope (easy to
automate and move to the execution).
Make a distinction between “ephemeral” and “long-standing”
cases 14

15. Test Automation Criteria
15

16. When?
General Automation Flow16

17. Product Release Cycle
17

18. Agile Development
18

19. Testing Process
Create new test scope
Define a scope for automation
& automate it
Execute automated
scope
Who: Scrum team QA
engineers.
Task: Create test design
for new features.
Deliverable: Prioritized
test scope, proposal for
automation
Who: Automation QA
engineers.
Task: Automate test
cases
Deliverable: Test scripts
ready for execution
Who: Scrum team QA or
DEV.
Task: Execute
automated test cases.
Deliverable: Execution
results. Understanding
product quality.
19

20. Test Automation Plan
Development
Team
Step 1: Starting
development new
telephony feature.
Development
Team
Step 2: Prioritized test
design for new
functionality
Development
Team
Step 3: Automation for
the most critical test
cases
Telephony
Automation
Team
Step 4: Continue
automation for 2-nd
and 3-rd category
Telephony
Automation
Team
Step 5: Delivering
automated cases to
the customers
20

21. Where?
Test Environment 21

22. 22
Test Environment and Production

23. Test Environment Update
ADB
Codebase
change or
Manual
request
Start
Automatic
Deployment
System
SVN
New
Build
23

24. Automatic Deployment System
3
2
1
Deploy → Verify (Post Deployment Verification) →
Execute Regular Scope
24

25. 25
Distributed Test Execution Scheme

26. How?
Test Execution Scheme26

27. Tools
• Framework and automated tests cases (scripts).
• Test Environment
• Dashboard for test execution results and reports.
• Bug tracking system
27

28. Technologies We Use
28

29. How It Works
29

30. Execution Scheme
Test
Environment
Deployment
Test Scope
Execution
Send
Execution
Results to
Dashboard
New Build
Available
30

31. Framework and Automated Test Cases
Business Layer
Test Scripts Layer
Application Part (exe)
Command line
application with a set
of parameters (What
we execute and Where)
Set of modules
required for test scripts
(WebDriver, Rest API,
Oracle ORA)
Automated test cases
(test scripts)
Powered by
31

32. Test Case Overview
32

33. Test design and Test script
33

34. Test script report
Powered by
34
• Start time (timestamp)
• Group (e.g. feature)
• Execution result (Passed / Failed / Skipped)
• Screenshots and attachments (if test failed during execution ).

35. Execution Results for the Scope
● Archive for all test reports
● Tested build (Branch and Build number)
● Backend log data and system traces
Powered by
35

36. Execution Report in Dashboard
• Configuration properties
• Backend build (from execution results)
• Execution Statistics (passed / failed / blocked)
36
Powered by

37. Powered by
Dashboard Sample
37

38. Why?
38

39. 39
Metaphor Behind Test Automation

40. 40
Cost: Manual vs. Automated Testing

41. 41
Benefits of Automated Testing
Global Continuous
Integration Team
6 Development Team
Regression Team
Total number of test cases: 5500
Total number of Automated: 2100
On a daily
basis
On a
weekly
basis
On a daily
basis
Automated

42. Test Automation Disadvantages
42
1. Test scripts base requires regular support and
actualization.
2. Troubleshooting for environment deployment and
execution results analysis.
3. It works only on long-term basis.

43. Important Moments
● Feedback from stakeholders
● Task Prioritization
● Popularization of automation
43

44. 44

45. Contact Information
nick@babich.biz
www.babich.biz
www.linkedin.com/in/nbabich
45
twitter.com/101babich