Prashant Yadav from Amdocs has given us a lighting talk on "Quality Engineering in DevOps Enabling Business Growth with seamless Production" only at #ATAGTR2018. please refer our linkedin post for session details https://www.linkedin.com/pulse/lightning-talk-quality-engineering-devops-enabling-growth-alliance/

1. Quality Engineering in DevOps : Enabling
Business Growth with seamless Production
- Prashant Yadav

2.  Agenda
Traditional Testing Model- A Mere Bottleneck
Amdocs Quality Engineering- Shift Left in DevOps
Shift Left Enablers
MTV
Business Case Study
Q&A

3. Information Security Level 2 – Sensitive
© 2018 – Proprietary & Confidential Information of Amdocs3
Traditional Testing Model- A Mere Bottleneck
System
Test
Functional
Test
Regression
Test
Business
UAT
E2E Test
Integration
Test
GoLive
Compatibility
Test
Accessibility
Test
Security Test
Test Automation Test
Automation
Development Responsibility Testing Responsibility
Revenue
Assurance
Unit
Test
Testing Responsibility
Performance
Test

4. Information Security Level 2 – Sensitive
© 2018 – Proprietary & Confidential Information of Amdocs4
Amdocs Quality Engineering- Shift Left in DevOps
System
Test
Functional
Test
Regression
Test
Performance
Test
Business
UAT Supt
E2E Test
Integration
Test
GoLive
Compatibility
Test
Accessibility
Test
Security Test
Test Automation Test Automation
Automation Nuggets
Scrum
Performance Test
E2E Performance
Test
Compatibility Test
Accessibility Test
Security Test
Continuous Progression Automation
Continuous Regression Test
E2EPerformanceTest
BusinessUAT
E2ESanityTest
Consolidated Functional Test
(Scrum Test + Integration Test)
Unit
Test

5. Shift Left Enablers

6. Information Security Level 2 – Sensitive
© 2018 – Proprietary & Confidential Information of Amdocs
Consolidated Functional Test
(Scrum Test + Integration Test)
Shift Left Enablers
GoLive
Automation Nuggets
Scrum
Performance Test
Compatibility Test
Accessibility Test
Security Test
Continuous Progression Automation
Continuous Regression Test
E2EPerformanceTest
BusinessUATSupt
E2ETest
Development Responsibility Testing Core Responsibility
1
2
3
TA
Automation
One Team
Iterative Scoping & Backlog
Grooming

7. Information Security Level 2 – Sensitive
© 2018 – Proprietary & Confidential Information of Amdocs7
Test Architect Role • Acceptance by Account Stake-Holders
• 100 % MTV Driven Scope Break-Down
• Non Functional Considerations For Each MTVs
• MTV driven IA Sign-Off
• Proactive Participation in Iteration Planning
• Early Insight to Testing
• Scrum Testing Quality Control

8. Information Security Level 2 – Sensitive
© 2018 – Proprietary & Confidential Information of Amdocs8
Automation – Progression Automation
• Benefits
• Ability to run multiple execution cycles in CJ
helping us to reduce escaping defects to UAT
• Supporting Dormant CRs Testing before Go-Live
• On-going Regression Suite enhancement
• Challenges
• Multiple Environment Configuration Demands
additional effort for re-use
• Unable to Stabilized Suite before CJ start due to
time/people limitations

9. Information Security Level 2 – Sensitive
© 2018 – Proprietary & Confidential Information of Amdocs9
Automation – Robotic CI & Code Coverage Analysis
• Current Status
• Unattended successful execution of extended
sanity with nightly build (Ginger Integration
With Jenkins)
• 1st Successful Run of Regression Suite for Code
Coverage Analysis using Jacoco
• Approach & Benefits
• Continuous enhancing scope of CI build sanity to
ensure wide test coverage on nightly build.
• Availability of Working Build Every Morning (
Currently limited to ST Env)

10. Information Security Level 2 – Sensitive
© 2018 – Proprietary & Confidential Information of Amdocs10
One Team • Continuous Rotation of People Across Teams-
Enabling CFT Enhancement
• ITV Team Support in Portfolio Testing – Helped to
Accelerate Progression Automation
• Re-use of Automation- Continuous Enhancement in
Regression Suite

11. Information Security Level 2 – Sensitive
© 2018 – Proprietary & Confidential Information of Amdocs11
MTV – Breaking the bottleneck! SIT waits till the first PI which provides
Testable features on integrated environment
with Business value to the customerDevelopment System Testing SIT / E2E / UAT
SIT
Scope
Agile To UAT
Agile To ProductionBDD + TDD = MTV
PI 01
Dev
User Story
Feature
Epic CJ
PI 02
Dev
User Story
Feature
Epic CJ
PI 03
Dev
User Story
Feature
Epic CJ
PI 01
Dev
User Story
Feature
Epic
CJ
Scope
SIT/E2E
PI 02
Dev
User Story
Feature
Epic
CJ
Scope
SIT/E2E
PI 03
Dev
User Story
Feature
Epic
CJ
Scope
SIT/E2E

12. Information Security Level 2 – Sensitive
© 2018 – Proprietary & Confidential Information of Amdocs12
MTV Concept
Project PI 1
5-7 Weeks
Project PI 2
5-7 Weeks
Project PI 3
5-7 Weeks
Every PI starts from Scoping in
which MTVs are defined
Development & Testing are done continuously per MTV,
enabling production deployment of each MTV
Separately
MTV
( Minimal Testable Value)
An entity defined by the Test Architect during scoping, to describe the integrated scope element which the testers must
receive in order to certify scope which carries business value and supports E2E tests
MTV Example Data only mobile postpaid activation & usage for Residential non VPN subscribers
Continuous Testing
Testing activities are done in parallel at all test levels, certifying in each stage the scope of MTVs separately and moving
them up to the next stage of testing. The certification is done at MTV level, allowing the customer to take the final
decision if to approve the certified MTVs to production or wait for additional scope in the next delivered MTVs.

13. Customer Case Study

14. Information Security Level 2 – Sensitive
© 2018 – Proprietary & Confidential Information of Amdocs14
NAM Customer
Business Challenges
1. Continuous drops for progression MTVs
2. Manual intensive test execution
3. Resource optimization
AQE Solution
1. High levels of automation & reusability
2. More testing coverage during progression
3. Automated production sanity
4. Unattended non-production sanity and CI
5. Automated regression run for weekly SWP
~110 flows being
executed unattended
on daily/while CI
~60%
Regression
5 days 1 Day
Customer Journey
regression duration
Reduction in
Escaping
Defects to UAT
22%
8%
~1250 TCs executed
weekly via automation
in progression &
regression
40%
Of defects found
via automation
5-6
times
Regression execution
frequency
Once
Monthly ~15000-16000 TCs are
executed during production
sanity integrated
Progression Automation
MTV: Minimum Testable Value CI: Continuous Integration
~70%

15. Information Security Level 2 – Sensitive
© 2018 – Proprietary & Confidential Information of Amdocs15
DevOps: NAM Customer
• Testing driven MTV
• Performance Shift Left
• Feature level delivery
• EPIC Certification
• Robotic CI
• Automation
• Successful Customer Demos
• One Quality authority
• Minimal ENV
• Automation reuse
CI CD
• MTV/Feature delivered
weekly (3 to 7)
• Further reduction in
escaping defects
• Earliest production
stabilization
• MTV/Feature level
deployment
• Continuous improvement
• Automation reuse
• Delivery KPI Dashboard

16. Q & A