Migrating applications & platforms from VM-based infrastructure to Container-based infrastructure is a challenging task.
One of the key strategies is to make sure the quality attributes (software -ilities) is carefully tested during the migration.
In this presentation, we will share our solution to automating software -ilities testing, particularly for Operability and Reliability.
9. Definition
9
Reliability : the capability of the system to maintain its
service provision under defined conditions for defined
periods of time.
Operability : ability of the software to be easily operated
by a given user in a given environment.
(ISO 9126 Software Quality Characteristics)
12. User Story
12
1. As SRE I want to be notified by the monitoring / alert
system once there is an incident within 5 minutes.
2. As SRE, when I scale out the application, there should
be no error alert triggered by the monitoring system.
3. As QA I want to verify if certain percentage of request
shall be succeed when there is an incident.
13. Current Problem & Situation
13
It requires at least 10 days to complete operability and
reliability testing
• Manual execution of manifest configuration settings
• Manual checking of alert system / configuration
• Environment preparation
21. Reliability + Functional Test
21
QA
Functional
Test
Framework
API
Reliability
Test
Framework
dependency
trigger system failure functional test case
24. Results
24
Before
It requires 10 days to complete due to :
• Manual execution of manifest
configuration settings
• Manual checking of alert triggered
• Environment preparation
After
It only takes approximately 2 days
to finish all the test, since all of the
test setup and scenarios are
automated.
25. Summary
25
1. This test framework could reduce the lead time by giving
confidence for SRE team about their system configurations
2. Provide transparency between all stakeholders about
operational activities
3. Allowing QA / Test engineer to test on reliability
perspective.