The document discusses the implementation of continuous testing at Zenefits, highlighting its importance for maintaining product quality as the company scales. It outlines various testing strategies throughout the development and deployment pipeline, emphasizing the integration of both manual and automated testing to overcome challenges. Zenefits successfully managed to scale its customer base to over 10,000 without an in-house QA team by utilizing a combination of automation and manual testing.

1. Continuous Testing for CTOs:
How Zenefits Deploys 3x Day
Laks Srini
CTO & Co-Founder
Russ Smith
CTO & Co-Founder

2. #continuoustesting
Agenda
● Why do Continuous testing?
● Types of testing
● How do you integrate testing into dev flow?
● How did Zenefits approach this need for QA?

3. #continuoustesting
Challenges with Testing Today
● Manual testing is an important aspect of comprehensive testing, but it is
usually done in huge chunks and takes forever
● Automated testing is brittle, not comprehensive
● Most in-house QA teams have very limited bandwidth
https://www.capgemini.com/events/world-quality-report-2015-16-pre-order-form

4. #continuoustesting
Test Continuously (Shift Left)
Test as early & often as possible for Faster Shipping, Better Quality
● Unit tests
● UI tests (automated vs. manual)
Production monitoring + Rollbacks

5. #continuoustesting
How does continuous testing fit into the
continuous delivery pipeline?
Code
Development
Continuous
Integration
Staging Production
Continuous
Deployment

6. #continuoustesting
How does continuous testing fit into the
continuous delivery pipeline?
Code
Development
Continuous
Integration
Staging Production
Continuous
Deployment

7. #continuoustesting
Test Strategies in Development
Code
Development
Staging Production
Usability Testing
Functional Testing
Regression Testing
GUI Testing
Cross Browser Testing

8. #continuoustesting
Test Strategies in Staging
Code
Development
Staging Production
Exploratory
End to end
Usability
Functional
Regression
GUI
Cross Browser Testing
Database Testing

9. #continuoustesting
Test Strategies in Production
Code
Development
Staging Production
Sanity /
Smoke Tests
Automated Testing

10. #continuoustesting
How do you begin testing continuously?
● Identify how comprehensive are your
current testing processes?
● Incorporate smaller tests, throughout your
development pipeline
● Continuous Integration (CI) - automate
regression suites
● Automation alone can be brittle and flaky
Zenefits was able to scale their
customer base to 10,000+
without an in-house QA team
by supplementing their testing
automation suites with manual
testing.

11. #continuoustesting
Challenge:
Maintain product quality while growing exponentially.
There was no room for failure.

12. #continuoustesting
Why did Zenefits approach Continuous Testing?

13. #continuoustesting
Why did you choose Rainforest?

14. #continuoustesting
What is the Hub & Spoke model?
Unique challenges with this

15. #continuoustesting
Challenges in SOA?

16. #continuoustesting
Scalability

17. #continuoustesting
Focus on Product

18. #continuoustesting
Ah-ha! moments
How has continuous testing has improved Zenefits?
● Automate the execution of manual testing
● Firing off manual tests is just as easy as automated tests, part of the same
process
● Now we can confirm results of automated tests and test things in bulk
with a human.
● Overcome brittle/flaky automated tests with safety net
● Deployment happened when it was ready

19. Questions?
Laks Srini
CTO & Co-Founder
Russ Smith
CTO & Co-Founder

20. Thank you!