In this webinar you will learn how Walmart migrated from many independent, siloed teams doing manual testing to integrated teams that run over 50k automated tests daily using shared resources like Test Armada and Sauce Labs.

1. Pranav Verma
Sr. Engineering Manager
Developer Experience Tools
EMAIL:
pverma@walmartlabs.co
m
Testing Infrastructure at Scale
September 2018
E X P E R I E N C E T O O L S
CUSTO M ER
E X P E R I E N C E T O O L S
DEVELOPER

2. Agenda
• Test Armada Introduction
• Test Armada Scope
• Test Armada Infrastructure
• Continuous Integration
• Automation Cloud
• Data Insights
• Q&A

3. A Cultural Shift in Quality Automation
within Customer Experience
Engineering
2016

4. You have 6 months to
automate100% for your
P1/P2 test cases

5. Automate 2000+ Manual Test
Train 600 Associates
On-board 25+ number of Apps
Migrate FE to new Platform
THE DETAILS

6. An Organization Going Through a Transfor
DENIAL UPSET RESISTANCE ACCEPTANCE

7. The Before and After Transformation
2000+ Manual Test
100% P1/P2
Automated
Lack of accountability
around Quality
Shared ownership of
Quality
2 Deployments a
month
Daily deployments
Test limited to a
single browser
Multi-browser
combinations part of CI
process

8. THE
SOLUTION

9. End to end test platform for web, native, and
backend services with actionable reporting
capabilities

10. Scope Across the Company
TEAMS WE SUPPORT
85 projects in 2 years
200 projects by 2020
1M Development Hours Saved
16M Functional Tests Executed
149M Mocked API Calls
LOCATIONS WE SUPPORT NUMBERS AT SCALE
Bay Area
San Diego
Bangalore
Canada
Omaha Hoboken
Bentonville
UK
Mexico
Point of Sales RCT ASDA Growth
USGM Jet.com Pharmacy GIF Store Systems OmniChannel
Customer Promise Hayneedle
USGR
EXPO
Portland

11. Infrastructure at Scale
AUTOMATION
CLOUD
CONTINUOUS
INTEGRATION
DATA
INSIGHTS

12. How to run automation tests fast
at a massive scale reliably.
CONTINUOUS INTEGRATION
PROBLEM
Tests need to run against multiple OS/Browsers, which multiplies the numbers
of tests. Need a solution to run massive number of tests without compromising
execution time.
GOAL

13. SOLUTION
Shard
Master
Homepage
64TESTS
Item Page
65TESTS
Cart
228TESTS
64 TestsShard 1
64 TestsShard 2
64 TestsShard 3
65 TestsShard 1
65 TestsShard 2
228 TestsShard 1
228 TestsShard 2
228 TestsShard 3
228 TestsShard 4
Thread 2
Thread 1
Thread 3
Thread 4
Thread n

14. BENEFITS
CART PR Verify
(228 Tests on 2 Browsers)
6
HOURS
Average Serial
Test ExecutionTime
25
MINUTES
Average Parallel
Test ExecutionTime
to 93%
Savings
=

15. How to test Walmart websites and native apps on
all supported browsers and devices.
AUTOMATION CLOUD
PROBLEM
Need a solution to support automation reliably atWalmart scale, and also
allows team to configure as per their testing needs.
GOAL

16. SOLUTION
Browser Plugin
Device Plugin
TestAgent Plugin
Resource
Manager
Reconciliation Engine
Quota Management
Priority Management
ResourcesTest LabsPlugin Architecture
Internal
Cloud
External
Cloud
Internal
Cloud
External
Cloud
Internal
Cloud
External
Cloud

17. BENEFITS
1,000+
Unique combinations
250KAutomated Developer
Hours Per Day
30KEngineersWorking
Per Day

18. How to see health of your project in one place.
DATA INSIGHTS
PROBLEM
The application quality information is scattered between JIRA, CI,Test
frameworks and Automation clouds. Need a solution to visualize all
information in one place to make informed decisions.
GOAL

19. Data Insights Platform
Managed Services
SOLUTION
FunctionalTestAgent
PerfTestAgent
CI
…
DB
Functional Report Client Perf Report Backend Perf Report
Self-Service Portal
API
H
X
HYGIEIA
XRAY
CLIENT

20. BENEFITS
Messages per Second
1Million
SCALABLE PROCESSING
Report Retention
6Months
TRENDING REPORTS
Latency
1Minute
REAL TIME REPORTS

21. r
Open Test Armada Platform
Resource Manager Self ServiceData Insights
TESTARMADA FEATURES
PROGRAMMING LANGUAGE
Magellan
TEST RUNNER
TestArmadaTDK
TESTARMADA LIBRERIES
OUT OFTHE BOX SOLUTION
Any Language
PROGRAMMING LANGUAGE
Any Runner
TEST RUNNER
TestArmada Service
TESTARMADAAPI
Resource Manager Self ServiceData Insights
TESTARMADA FEATURES
CUSTOMIZED SOLUTION

22. Q&A

23. Contact us
Contact Us
https://www.testarmada.io