Automated testing of an application with many dependent services can be challenging. Achieving continuous deployment across these services can be even more so. Managing, coordinating, and scaling deployments of services can become overwhelming and error prone over time. Ensuring that you are able to detect important defects before customers do can be difficult. Executing only relevant tests after each code change rather than always running everything (boiling the ocean) can be a formidable task and might not scale well as the size of the app increases. Manoj Pahuja and Daniel Clayton suggest a remedy. What if simply adding an additional YAML file in your source code repository could eliminate a lot of these pain points and make managing all these jobs and tasks easier? What if defining properties in this file could select and run the most appropriate tests? What if this file could set up the infrastructure to run the app and tests auto-magically? Manoj and Daniel present ideas and practical solutions that work to scale deployments and automated test runs. Join them as they share strategies to determine which tests to run in response to each code change.

1. DT4
DevOps & Testing
6/8/2017 11:30:00 AM
DT4 Scaling Automated Tests: Choosing
an Appropriate Subset
Presented by:
Manoj Pahuja
GoDaddy
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888-­‐268-­‐8770 ·∙ 904-­‐278-­‐0524 - info@techwell.com - https://www.techwell.com/

2. Manoj Pahuja
GoDaddy
Test architect at GoDaddy Manoj Pahuja has several years' experience in test
planning, strategy, and automation for web, desktop, and mobile applications. He
has worked on building automation frameworks, created and managed test
management software, and is an active member of the open source community.
Manoj's experience writing automation in VBScript, Java, Ruby, and now NodeJS
has helped him deal with the challenges of automation. A strong believer in
learning while sharing, he co-organizes meetups in the San Francisco Bay Area.

3. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Copyright© 2015 GoDaddy Operating Company, LLC. All rights reserved.
Scaling Automated Tests:
Choosing an Appropriate
Subset
June 2017
Daniel Clayton and Manoj Pahuja

4. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Agenda
• Define Continuous Integration and Continuous
Deployment (CI/CD).
• How GoCentral adopted CI/CD.
• Identifying What to test.
• Life cycle of a Test.

5. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Define CI/CD
3
to a developer… automate all things!
to a tester… smaller deployments, reduce risk!
to a manager… release quickly, deliver value!
to a product owner… slice features small enough to prove hypothesis!
Release Quickly with Confidence
What does CI/CD mean…
Continuous Integration – frequently testing and integrating your changes with others
Continuous Deployment/Delivery – frequently delivering your changes to your customers

6. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
CI/CD with GoCentral
4
v6
Manual Testing
Manual Deployment
Stress/Uncertainty
0.029 mph
Deployments to Production
v7
Agile
Automation
High Confidence
6.3 mph
368
979
vNEXT
Microservices
Elastic/Resilient
Abstracted Pipelines
45 mph
3179
7,000+
26
2011 2014 2016 2017

7. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
CI/CD with GoCentral
5
build
deploy
feature-branch (PR)
s8d7fss.myapp.int.godaddy.com
testing:
➔ smoke
➔ bucket
➔ performance
➔ etc
appname: myapp
ssl: true
slack: ‘#myapp’
node_version: 6.10.2
start_command: node build/server.js
hosts:
- myapp.int
testing:
tests:
- shell: npm run ... smoke-tests
- parallel:
- npm run tests --bucket A
- npm run tests --bucket B
- ...
/cicd.yaml master
build
myapp.int.godaddy.com

8. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Testing Pyramid

9. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Buckets
Smoke
Regression
New
TUT - Failing tests
App Area

10. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Example of buckets and tags
bucket: "[dt_chrome_regression]
[mo-emu_chrome_regression]
[mo-android_chrome_regression_tut]
[mo-ios_safari_regression_tut]
[dt_firefox_regression] [dt_ie_regression] [footer]
[layout]"

11. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Change Detection in CI/CD
appname: myapp
...
change_detection:
BUTTON_A_CHANGED:
- lib/buttonA.js
BUTTON_B_CHANGED:
- lib/buttonB.js
testing:
tests:
- shell: npm run ... smoke-tests
- parallel:
- shell: npm run tests --bucket buttonA
if: BUTTON_A_CHANGED
- shell: npm run tests --bucket buttonB
if: BUTTON_B_CHANGED
- ...
/cicd.yaml
change_detection:
BUTTON_A_CHANGED:
- lib/buttonA/*.js
CORE_CHANGED:
- lib/core/*.js
- shell: npm run tests --bucket buttonA
if: BUTTON_A_CHANGED
- shell: npm run tests --bucket regression
if: CORE_CHANGED

12. Copyright© 2017 GoDaddy Inc. All Rights Reserved.

13. Copyright© 2017 GoDaddy Inc. All Rights Reserved.

14. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Customizable Test Runner
'multi="spec=- mocha-elk-reporter=-"ui-tester ./test/ui/specs
--config=./test/ui/config
--env ${NODE_ENV}
--server ${SERVER}
--tags ${TEST_TAGS}
--platform ${PLATFORM}
--browser ${TEST_BROWSER:-chrome}
--timeout ${TIMEOUT:-120000}
--tagsType and
--reporter mocha-multi
--project vnext-editor'

15. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Test Run Numbers

16. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Life Cycle of a Test

17. Copyright© 2017 GoDaddy Inc. All Rights Reserved.

18. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
TMNT - reliability calculators
TMNT-bots Bucket
Service
Profile
Service
mocha-
elk-repo
rter
Architecture

19. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
{
"uuid":"image-00005",
"originalTags":"[dt_chrome_regression_tut](reliability: 77.50,
totalRuns: 40) [mo-emu_chrome_regression_tut](reliability: 100.00,
totalRuns: 40) [mo-android_chrome_regression_tut](reliability: 0.00,
totalRuns: 40) [mo-ios_safari_regression_tut](reliability: 0.00,
totalRuns: 40) [image](totalRuns: 0) [about](totalRuns: 0)
[dt_firefox_regression_tut](reliability: 100.00, totalRuns: 40)
[dt_ie_regression_tut](reliability: 15.00, totalRuns: 40)",
"newTags":"[dt_chrome_regression_tut] [mo-emu_chrome_regression]
[mo-android_chrome_regression_tut] [mo-ios_safari_regression_tut]
[image] [about] [dt_firefox_regression] [dt_ie_regression_tut]"
}

20. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
{
"uuid":"image-00005",
"originalTags":"[dt_chrome_regression_tut](reliability: 77.50,
totalRuns: 40) [mo-emu_chrome_regression_tut](reliability: 100.00,
totalRuns: 40) [mo-android_chrome_regression_tut](reliability: 0.00,
totalRuns: 40) [mo-ios_safari_regression_tut](reliability: 0.00,
totalRuns: 40) [image](totalRuns: 0) [about](totalRuns: 0)
[dt_firefox_regression_tut](reliability: 100.00, totalRuns: 40)
[dt_ie_regression_tut](reliability: 15.00, totalRuns: 40)",
“newTags”

21. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Take Aways
• Adopt Continuous integration and continuous deployment.
• Following the Testing pyramid to scale
• Ability to selectively run tests.
• Adopt bucketing / categorization of tests.
• Enable Change detection.
• Measure and calculate reliability .

22. Copyright© 2017 GoDaddy Inc. All Rights Reserved.
Questions?
Connect with us:
Daniel Clayton: https://www.linkedin.com/in/azwebmaster
Manoj Pahuja: https://www.linkedin.com/in/manojpahuja