This document discusses the importance of infrastructure testing for QA in a DevOps environment. It outlines the differences between typical development and production infrastructures, and provides examples of what aspects of infrastructure QA should test, including database failure scenarios, application performance, server or service availability, logging and archiving, errors, resource usage, third party services, security, caching, and cross-functional requirements. It also discusses tools that can be used for infrastructure testing and debunks myths about the need for such testing.

1. QAing INFRASTRUCTURE
A QA's Role in DevOps World
by
Aroj P George
Harshad Wankhede

2. What is Infrastructure Testing?

3. Typical Development Environment - Single Server
Developer/QA Laptop
W
E
B
S
E
R
V
E
R
App
LI
Service
Mocked ThirdParty Services
DB

4. Production Infrastructure – Multi-Tier
Replica Set
L
O
A
D
C
D
N
B
A
L
A
N
C
E
R
App
App
App
L
O
A
D
B
A
L
A
N
C
E
R
Service
DB
Service
Service
Third Party Services
DB
DB

5. Application Ecosystem
Application
Libraries
Operational Processes
Network Services
System Services
Authentication and Authorization Rules
Virtualization
Storage Configuration
Network Configuration
Firewalls and VPN and Secure Access Rules

6. What if we don’t test the infrastructure?

8. What is DevOps?
It’s a mindset
Bridges gap between Developers and Operations

9. What to Test?

10. Database Failure Scenarios
Do we have proper error handling, logging and alerting in place for below
scenarios?
if the application is unable to read data
 Network issues
 DB Primary is down
Does the DB Primary failover if it goes down?
Does the application keep running
Do we have alerts for the below?
Replication has failed
DB backups are not happening
DB high resource utlilization
DB queries are very slow

11. Application Performance Issues
Do we have monitoring for high resource utilization
Are we tracking the response times for application and dependent
web services
Are we tracking errors generated in the logs?
Do we get alerts if a cron job fails to run or if it errors?
Ex: Newrelic, Nagios, Ganglia metrics collection, Executing
Performance Testing scripts regularly

12. Server or Services are Down
How does the system react to database failure
How does the system react to application failure
How does the system react to third party service failure
What happens if the load balancer is down
Ex: Chaos Monkey

13. Logging and Archiving
Have you configured appropriate logging?
Are the logs being properly indexed and rotated?
Do you maintain log history using tools like Splunk?
Are the DB backups being archived?

14. Application Errors
Do we show appropriate error pages for various errors
500, 503, 404 Pages
What happens if there is a sudden increase in the rate of errors?
Do we get alerted regarding the same.

15. Resource Consumption Issues
Have you configured alerting for CPU usage?
Have you configured alerting for Memory usage?
Have you configured alerts for Disk Utilization?
Warning if utilization reaches 80%
Critical if utilization reaches 95%

16. CPU Usage

17. Memory Usage

18. Network Usage

19. Third Party Service Failure
Have you configured alerting when third party services are down?

20. Security
Have you blocked access to admin configuration?
Have you configured firewall rules properly?
Penetration Testing

21. Caching
Have you validated application caching?
Ex: CacheWarp

22. Cross functional/Non Functional Requirements
Testing infrastructure is nothing but testing the Cross-functional
Requirements

23. Chef Recipe

24. Chef - Templatize Configuration

25. Chef Demo
Chef is a systems and cloud infrastructure automation framework
that makes it easy to deploy servers and applications
Infrastructure as Code
Knife is a Chef utility
Using the Knife SSH command to test the infrastructure

26. RSpec Tests
Demo

27. CacheWarp
Demo
https://rubygems.org/gems/cachewarp

28. QA Skills Required:
Good Understanding of the Infrastructure
Ability to foresee Issues
Command-line proficiency, Vim
Ability to debug and troubleshoot issues
Should be able to write automated tests (rspec)
Basic shell scripting knowledge

29. Myths about Infrastructure Testing
No need to test Infrastructure. If it works locally it will work on any
environment
Application and Infrastructure Testing strategies are similar
Application Testing is more important than Infrastructure Testing
There aren’t too many different things to test in infrastructure
There is no business value in testing Infrastructure

30. Tools
Chef + Knife
Ganglia
Splunk
GO
Nagios
Mingle
CacheWarp
Vagrant
Vim
Tmux
Putty

31. Questions