With the acceleration of software creation and delivery, test activities must align to the new tempo. Developers need immediate feedback to be efficient and correct defects as those are introduced. The path to achieving this vision is to build a reliable and scalable continuous test solution. All beginnings are hard. Having a well-defined plan outlining the approach for your organization to create test automation is key to ensure long term success. Join Diego Molina, Senior Software Engineer at Sauce Labs as he discusses: The importance of setting up the team correctly from the start Choosing the right Testing Framework for your organization Identifying the right scenarios and workflows to test Learning to avoid common pitfalls at the beginning of the transformation journey

1. 5 STEPS TO JUMP-START
YOUR TEST AUTOMATION
DIEGO MOLINA, SENIOR SOFTWARE ENGINEER
MAY 28, 2019

2. Who Am I?
• OSS & Selenium Enthusiast
• Love testing
• Selenium contributor & committer
• Co-creator and maintainer docker-selenium and Zalenium projects
• Sr. Software Engineer @
diemol diegofmolina
• Will be a the next Appium Conference in Bangalore, June 13-15
2

3. E
D
C
B
A
Team Setup
Testing
Framework
Test the
Right Thing
When to
Run Tests
Get People
to Join You

4. STORY TIME
Photo by Ehud Neuhaus on Unsplash

5. My Story
5
Testing was done
by end users
Wrote tests for every
single use case
Oh, I have too
many tests!
Built my own test
infrastructure
Hmm, I can add
more tests!
Oh, I have too
many tests!

6. Why Automate?
6

7. Why Automate?
7
Software Development has changed, we are faster now
Proprietary
Single Stack
On-Premise
Open Source
Best of Breed
Cloud-based
Tools
Process
Waterfall
Fast
Waterfall
Continuous
Integration
Continuous
Delivery
Speed of
Releases
On DemandDailyWeeklyMonthlyQuarterlyAnnually
Desktop/Packaged Apps
Thick-Client Apps
Web Apps
Mobile Apps
Desktop Web
Responsive Web
Apps

8. Photo by Mpho Mojapelo on Unsplash
TEAM SETUP

9. Testers are in a separate team
• Languages with an easy setup and a simple entry level, like Python or Ruby
9
Jumping into automation? What programming language should we use?
PYPL PopularitY of Programming Language

10. Testers are in a separate team
• Languages with an easy setup and a simple entry level, like Python or Ruby
10
Jumping into automation? What programming language should we use?
Top languages over time - GitHub Octoverse StackOverflow Survey 2019

11. Testers are in a separate team
• Aim for great communication with the Product Owner and the Dev team
• Get access the story|task|feature definitions to create tests
• Get business context from Product Owners to prioritise activities
• Get as much information as possible (tech stack, system architecture, etc…)
• Go step by step when implementing the first automated tests
• Be in synch with the Dev team
• Define a bug report flow
• Agree on a working pace
11

12. Testing Role is part of the team
• One or more team members have the role of guiding the team in testing
• App code and test code use the same programming language
• Leverages team collaboration
• Testing has the good practices used in the software development lifecycle
• Test automation is also a software project
• Test code is in the same repository where the App code is
• Simplifies flows to write tests for features that are not yet in production
12

13. Independently from the team setup
• Invest time looking for a test framework that helps the team move faster
• More about this on the next slides
• Test design and implementation are a team task
• Document how:
• Tests are structured
• Run, add and remove tests
• Pair program and do code reviews
• Treat test code as production code
• Continuously evaluate if the existing test cases deliver enough value
13

14. How the Team Setup should evolve
14
Continuous
Integration
Full adoption of Agile
Automated testing
dominates;
manual for debugging or
exploratory
Dev. and QA
collaborate closely
Fast Waterfall
Initial adoption of Agile
Automated testing begins
Dev. & QA start
communicate
Waterfall
Traditional sequential
design model
Manual testing
dominates
Dev. & QA completely
separate
Process
People
Tools
Continuous
Delivery
Fully automated
Development process
Automated testing core
to Dev. & Delivery
Manual for exploratory
Dev. and QA
functions merge

15. TESTING FRAMEWORK
Photo by Dayne Topkin on Unsplash

16. Why a testing framework?
Reduce maintenance costs
Be Faster
Be Reactive
16

17. 17
Assertions on Actions
Initialization and Cleanup
Data Modeling/Mocking
Configuration
Site Modeling Abstractions
Wrappers and Helpers
API Usage
Future ready features
Local and Cloud setups
Speed
Debugging features
Cross browser
Simulators-Emulators/Real Devices
Built in reporting or easy to plug in
What features should my testing framework have?

18. How do I build my testing framework?
By using Open Source
WebDriverIO Watir Selenide Nerodia
Allure GHP Reporter
WireMock Mockoon Json-Server
Mimesis Faker(Ruby, Python, JS) Mockaroo
(and many others)
18

19. How to choose an Open Source tool?
Programming language support
Check ★s and forks
Check activity (issues/PRs)
Help/Support channels
Documentation
Involve the whole team
Do a Proof of Concept
19

20. Photo by Matt Briney on Unsplash
TEST THE RIGHT THING

21. Should we automate everything?
21
https://xkcd.com/1319/

22. What is the Right Thing to test?
• Identify main application flows, the ones that must always work
• A clear understanding of the flows is needed, inputs/outputs must be clear.
• How bad is it if this feature/behaviour breaks?
• How much value does the test have? How big is the risk that mitigates?
• Analytics can help you to get more answers
22

23. 23
Is login working?
Is registration working?
Are errors shown for wrong passwords?
Are ads displayed?
Are emails being sent?
Can items be added to the basket?
Can users change their information?
Can users pay?
Can users add new payment methods?
Are product images displayed?
Is the page responsive?
Can users logout?
Example
Online Shopping Site

24. 24
Is login working?
Is registration working?
Are errors shown for wrong passwords?
Are ads displayed?
Are emails being sent?
Can items be added to the basket?
Can users change their information?
Can users pay?
Can users add new payment methods?
Are product images displayed?
Is the page responsive?
Can users logout?
Example
Online Shopping Site

25. How Analytics could help
25
Browsers

26. How Analytics could help
26
Browsers/Platforms

27. How Analytics could help
27
Browser Versions

28. How Analytics could help
28
Browser/Screen Resolution

29. How Analytics could help
29
Device Details / Platforms

30. How many tests are enough?
• To start, we identified 5 main flows. Automating them gives us:
• A lot of value because most of the users go through them
• Security because they lower the risk of breaking something important
• Based on Analytics, we will run them in:
• 3 different browsers (Chrome, Firefox, Safari)
• 2 different OS (Windows, OSX)
• 3 different screen resolutions
• 7 different browser versions (Chrome 71-74, Firefox 64-65, Safari 12)
• 120 + 60 + 15 = 195 tests
• Automate high value tests
30

31. Use good practices
• Atomic and focused tests
• Readable, faster and more stable
• Sauce Labs Continuous Testing Benchmark report
31
https://info.saucelabs.com/sauce-labs-continuous-testing-benchmark-report-M.html

32. Use good practices
• Focus on reusability and maintainability
• Avoid code duplication across tests and helper classes/methods
• Every test must be autonomous
• Tests can run in any order without depending on each other
• Write tests and code only for the current requirements
• Avoid complex designs that consider potential future use cases
• Overengineering is overkilling
• Get familiar with software design patterns
• They can benefit automation testing as well
• Base your work on testing plans and/or strategies, not on tools
32

33. Photo by Maximilian Weisbecker on Unsplash
WHEN (AND WHERE)
TO RUN TESTS

34. It all depends on the stage where we are
34
Continuous
Integration
Full adoption of Agile
Automated testing
dominates;
manual for debugging or
exploratory
Dev. and QA
collaborate closely
Fast Waterfall
Initial adoption of Agile
Automated testing begins
Dev. & QA start
communicate
Waterfall
Traditional sequential
design model
Manual testing
dominates
Dev. & QA completely
separate
Process
People
Tools
Continuous
Delivery
Fully automated
Development process
Automated testing core
to Dev. & Delivery
Manual for exploratory
Dev. and QA
functions merge

35. Early stages...
35
Process
People
Tools
When
Big pre-release session
Bug hunting sessions
Regular validation cycles
Where
Locally on desktops
or in the cloud
Shared testing
environments
Why
Not enough time to test
more often due to the
nature of manual testing
Waterfall
Traditional sequential
design model
Manual testing
dominates
Dev. & QA completely
separate

36. Early automation stage...
36
Process
People
Tools
One or more times
per day
Triggered manually or on
schedule based
Combined with manual
before releasing
Locally on desktops, own
Grid or in the cloud
Initial setup of a CI server
to run on schedule
Shared testing
environments
Fast Waterfall
Initial adoption of Agile
Automated testing begins
Dev. & QA start
communicate
When Where Why
While gaining confidence
in automation, results
need to be triaged and
use that to improve test
resilience and confidence.
Automation starts to get
relevant but it is not part
of the main pipeline yet.

37. Automation dominates...
37
Process
People
Tools
On every commit and
pull/merge request
After merging
Cloud
CI server integrated to
development flow
Managed testing
environment for SUT
Continuous
Integration
Full adoption of Agile
Automated testing
dominates;
manual for debugging or
exploratory
Dev. and QA
collaborate closely
When Where Why
Fast feedback through
automated testing
Automated testing is now
part of the Quality Gate
Cloud - infrastructure is
vital for success. A
malfunctioning
infrastructure slows down
the development process.

38. Automation in everything...
38
Process
People
Tools
When
On pre-commit, commit,
and merge/pull requests
After merging
After release
Cloud
CI server integrated to
development flow
Dedicated testing
environment for SUT
Production
Continuous
Delivery
Fully automated
Development process
Automated testing core
to Dev. & Delivery
Manual for exploratory
Dev. and QA
functions merge
When Where Why
Automated tests are fully
resilient and trustable.
Can be used in all
environments.
Cloud - infrastructure is
vital for success. A robust
infrastructure is key for
confidence.

39. Photo by Priscilla Du Preez on Unsplash
GET PEOPLE TO JOIN YOU

40. How do I motivate my coworkers?
You
Team/Department
Company
40
Getting your team, manager, department and company onboard

41. Motivate yourself
Risks
• Not moving in a constantly changing industry: testing
• Missing opportunities to improve testing of the product
• Getting overwhelmed by repetitive tasks
Actions
• Get a % of your time for training, with the purpose of spreading it internally
• Join online testing chats to get ideas and support
• Be the agent that triggers change in your team and organisation
41

42. Motivating my Team/Department
Risks
• Dev vs. QA brings confusion and low motivation
• Silos break communication, brings different expectations for testing
• Low trust in automation
Actions
• Test earlier, pair with devs to find better ways to test features
• Create a testing framework based on OSS
• Show progress through beautiful reports and dashboards
• Create simple guidelines on what to test and what to expect from testing
• Perform internal training sessions to help people improve
42

43. Motivating my Company
Risks
• Slow organisations hardly evolve at the same pace as the marketplace
• Testing can become a bottleneck, which could lead to unstable products
• User experience and reliability are hurt if testing is treated as an afterthought
Actions
• Connect business success metrics with testing
• How did a bug in production reduce revenue?
• PoC with testing as a business enabler
• Increase release frequency and speed
• Improve product reliability
43

44. Photo by Cathryn Lavery on Unsplash
RECAP

45. E
D
C
B
A
Team Setup
Testing
Framework
Test the
Right Thing
When to
Run Tests
Get People
to Join You

46. The Continuous
Testing Journey
The Continuous
Testing Journey
LEGACY
TESTING
CONTINUOUS
TESTING
● Executive Sponsor
● Org Structure
● Development
Methodology
● Methodology,
Framework &
Infrastructure
● Authoring Plan
● Skills Development
Plan
● Cloud execution &
parallelization
● Pipeline integration
● Adoption Plan
● Analytics
● Shift Left
● Shift Right
PROOF OF
AUTOMATION VALUE
● Release faster
● Lower risks
● Increase developer productivity
● Delight your customers

47. THANK YOU
Photo by Stefan Widua on Unsplash

48. Resources
48
Sauce Labs Continuous Testing Benchmark report
https://info.saucelabs.com/sauce-labs-continuous-testing-benchmark-report-M.html
The Inquiry Method for Test Planning
https://testing.googleblog.com/2016/06/the-inquiry-method-for-test-planning.html
Sauce Labs free trainings
https://training.saucelabs.com/
Selenium Channel
https://seleniumhq.herokuapp.com/
Appium Channel
https://gitter.im/appium/appium
Ministry of Testing Channel
https://www.ministryoftesting.com/slack_invite
Software Testers Channel
https://software-testers.herokuapp.com/

49. 5 STEPS TO JUMP-START YOUR TEST AUTOMATION
DIEGO MOLINA, SENIOR SOFTWARE ENGINEER
MAY 28, 2019
Q/A

50. 50
It is never too late
Diffusion of Innovation (developed by E.M. Rogers, 1962)
Innovators
2.5%
Early Adopters
13.5%
Early Majority
34%
Late Majority
34%
Laggards
16%