Claude Jones, Sr. Director of Engineering at Walmart Labs, provides an inside look of what it took to push from within and help one of the world’s largest companies adopt a culture of quality. His presentation covers identifying the right project, building the right team structure, removing the fear of change, providing value with the right metrics, making testing fun and reinforcing the benefits. This case study will challenge you to re-think your approach or empower you to stay on your current path of adopting a culture of quality.

1. THE PUSH FROM WITHIN:
TRANSFORMATION
A JOURNEY OF
Sr. Director of Engineering
Walmart Labs
Claude Jones

2. WALMART LABS EXPERIENCE TOOLS
MY PERSONAL
JOURNEY
Part 1 Part 2Adopting a Culture Quality
WHAT I DID
Maintaining a Culture of Quality
WHAT I’M DOING

4. THE SCALE OF WALMART
270M
WEEKLY
CUSTOMERS
11K+
STORES WORLD
WIDE
2.3M
ASSOCIATES
100M+
ITEMS SOLD
ONLINE

5. GROCERY
STORE SYSTEMS
ASSOCIATE
TECH STORES
GROWTH AND VOICE COMMERCE
INTERNATIONAL
MARKETS AND US SAMS
PERSONALIZATION
AND ADVERTISING
WALMART.COM
CE TECH AT JET.COM
Responsible for all tech that power our
omni-channel experiences

6. THE INVISIBLE WALL ANTI-PATTERN
Lack of accountability
between teams
Poor product quality
QA team is pressured to deliver under
tight time constraints

7. AT WALMART
SCALE
PRODUCTION
ISSUES HAVE A
MASSIVE
IMPACT
TIMEFINANCIAL CUSTOMER
How Do We Address The Problem?

8. PART 1: TRANSFORMATION JOURNEY
adopting a culture of quality
2016

9. You have 6 months to
automate 100% for your
P1/P2 test cases
THE EXECUTIVE PUSH
Automate 2000+ Manual Test
Train 600 Associates
Onboard 25+ Apps on our testing platform
Adopt Nodejs stack
THE DETAILS
THE MANDATE

10. ORGANIZATIONAL CHANGES
DECENTRALIZE QA1 MAKE EVERYONE A SOFTWARE ENGINEER2
DENIAL
RESISTANCE
ACCEPTANCE
THIS CREATED AN EMOTIONAL ROLLER COASTER ACROSS THE ORGANIZATION

11. THREE STEPS
THAT ENABLED
OUR SUCCESS
Identify the right team1
Identify the right tools2
Provide Training & support3

12. IDENTIFY THE RIGHT TEAM

13. IDENTIFY THE RIGHT TOOLS
FUNCTIONAL
TESTING
PERFORMANCE
TESTING
SYNTHETIC
TESTING
VISUAL
TESTING
ADA, ANALYTICS
& SECURITY

14. IDENTIFY THE RIGHT TOOLS

15. PROVIDE TRAINING & SUPPORT

16. 100% P1/P2 Automated on Test Armada
Daily deployments on the new FE stack
Shared ownership of Quality
Scalable test labs
WHAT WAS DELIVEREDWHAT WAS ASKED FOR
Automate 2000+ Manual Test
Train 600 Associates
On-board 25+ Apps on our testing platform
Adopt Nodejs stack
WHAT WE WERE ABLE TO ACCOMPLISH

17. TEST ARMADA
BY THE NUMBERS
30K
Test Cases Automated
5M
Hours Saved
25M
Test Executed API Mocked
360M
100 Projects Onboarded

18. OPERATIONAL EFFICIENCY AT SCALE
1,000+
Unique combinations
1,150
Available Browser/OS
combinations
Devices in a private
cloud
81
5MAutomated Developer
Hours Per Day
625KEngineers Working
Per Day

19. THERE WAS A PROBLEM
We were so focused on adoption that
we were missing the bigger picture…

20. PART 2: TRANSFORMATION JOURNEY
maintaining a culture of quality
2019

21. QUALITY
TIME
EXECUTIVE PUSH
CULTURAL TRANSITION
RESISTANT TO CHANGE
VALUE ADD
DECLINE IN USAGE
REALITY HIT HARD
2016 2018
HIGH
LOW

22. Our success of creating a culture of
quality had negative side effects!
WHAT WAS CAUSING THE DECLINE?

23. WHILE WE CELEBRATED OUR WINS…
0
20
40
60
80
100
120
23 24 25 27 28 30 31 32 33 34 36 37 38 40 41 42 43 44 45 46 47 48 49 50
Projects Created Cumulative Projects Created
KPI
• Reached 100 projects in the last 6 months
KPI
• Average 125k Per day
(compared to previous estimate of 50k per day)
SURGING ADOPTION INCREASING TEST EXECUTION

24. …WE LOST FOCUS ON THE BIGGER PICTURE
CREATED VS. RESOLVED CHART
• Peaked at 40 issues per week
• Average 20 issues per week
HIGH VOLUME OF SUPPORT CASES
CHALLENGES
CONCURRENT JOBS
• Usage more than doubled over past year
• Hitting max limit every day
HITTING OUR RESOURCE LIMITS
CHALLENGES

25. WE NEED TO GET BACK TO THE BASICS
DEVELOPER
EFFICIENCY
THIS PROVIDED A POOR DEVELOPER EXPERIENCE
PRODUCT
DELIVERY
PRODUCT
QUALITY

26. DEVELOPER
EFFICIENCY
& PRODUCT
DELIVERY
WHAT THEY WANTED
PR VERIFY
10 MINUTES
MASTER VERIFY
1 HOUR OR LESS
PROD VERIFY
1 HOUR OR LESS
WHAT THEY GOT INSTEAD
PR VERIFY
60 MINUTES
MASTER VERIFY
3 - 4 HOURS
PROD VERIFY
3 - 4 HOURS

27. WHAT ARE WE DOING ABOUT IT?
Goal Approach Detail
Streamline the utilization of
our hardware
Expand internal infrastructure Internal headless browser grid
Speed up test execution times Smart test case execution Modular test grouping and
execute long running test in
parallel
Test case development best
practices
Improve test automation design SauceLabs consulting to help
optimize test case code

28. PRODUCT
QUALITY
APPLICATION QUALITY
FUNCTIONAL SCORE
Automation Score
Escape Score
Test Efficiency Score
Code Coverage Score
PERFORMANCE SCORE
SYNTHETIC SCORE
VISUAL SCORE
ADA, ANALYTICS, & SECURITY
OTHER SCORING TYPES

29. WHAT ARE WE DOING ABOUT IT?

30. OUR PROMISE TO THE CUSTOMER
Holding the tools team accountable to a higher standard through measurable results
Reduce tooling support
incidents by 60%
DEVELOPER EFFICIENCY
Reduce cycle time to
product by 80%
PRODUCT DELIVERY
Reduce customer facing
issues by 50%
PRODUCT QUALITY

31. QUALITY
TIME
INDIVIDUAL PUSH
SENSE OF
ACCOUNTABILITY
INDIVIDUAL PUSH
SENSE OF
ACCOUNTABILITY
WE ARE HERE
HEALTHY QUALITY ECOSYSTEM
EXECUTIVE PUSH
CULTURAL TRANSITION
RESISTANT TO CHANGE
VALUE ADD
2016 2018 2019
HIGH
LOW
Helping the organization maintain a culture of quality.

32. TRANSFORMATIVE
JOURNEY
• Identify the right team
• Identify the right tools
• Provide Training & support
1 Adopting a Culture of Quality
• Focus on Developer Efficiency
• Focus on Product Delivery
• Focus on Product Quality
Maintaining a Culture of Quality2
CONTINUOUS DEPLOYMENTS
Speed
CONTINUOUS TESTING
Automation
CONTINUOUS MONITORING
Quality

33. THANK YOU
Sr. Director of Engineering
Walmart Labs
Claude Jones