IT Project success depends on realizing value realization and customer success. In addition to stimulating test environment and test data, projects need to focus on geographically distributed and culturally diversified people who can simulate End-user scenarios, User Experience, and Customer Outcomes This presentation focuses on leveraging agile teams to simulate an end-user environment based on the team’s knowledge, understanding, and skills.

1. Delight Your Customers with Four Eyes of Quality
Dr. Kaali Dass, PMP, PhD
Sr.Technical Program
Manager
20th May 2021
Agile Quality Improvement Strategies

2. Agenda
What Is Quality?
Software Quality Challenges
Four Eyes of Quality Model
Q & A

3. Cisco Confidential 3
© 2013-2014 Cisco and/or its affiliates. All rights reserved.
What Is Quality?

4. What Is Quality: Product or Service
Source: asq.org
Ability to Satisfy
Stated or Implied
Needs
Free of
Deficiencies
Conformance to
requirements
- Philip Crosby
Fitness for use
- Joseph Juran
Functionality
Functionality
Portability
Portability
Meeting
Specifications
Meeting
Specifications
Customer
Satisfaction
Customer
Satisfaction
International
Regulations
International
Regulations
Compliance
Compliance
Conformance
Conformance
On-Schedule
On-Schedule
On-Budget
On-Budget
User Experience
User Experience
Security
Security
Accuracy
Accuracy
Federal / Local
Regulations
Federal / Local
Regulations
Performance
Performance
Availability
Availability
Accessibility
Accessibility

5. Quality is Everywhere
Compliance
Performance Reliability
Operability
UX
Localization
Functional
Security
Maintainability
Adapted from CISQ-TR-2012-01 by OMG & ISO/IEC 25010
Compatibility
Portability

6. Cost of Quality
Ref: PMBOK Guide 5th Edition, Adapted from Fig 8-5 Cost of Quality
Cost of
Conformance
Cost of
Conformance
Prevention Costs
(Training, Documentation, Tools, etc.)
Appraisal Costs
(Testing, Inspection, Compliance to
Regulations and Laws)
Cost of
Non-Conformance
Cost of
Non-Conformance
Internal Failure Costs
(Rework, Scrap)
External Failure Costs
(Liabilities, Warranty, Lost Business, Fines
and Penalties from regulators)

7. Cost of Failures
Ref: Computerweekly.com, ashireporter.org
In the US alone it is estimated to be
upwards of $75billion a year in re-
work costs and abandoned systems
In the US alone it is estimated to be
upwards of $75billion a year in re-
work costs and abandoned systems
Software bugs, or errors, cost the U.S.
economy an estimated $59.5 billion
annually
Software bugs, or errors, cost the U.S.
economy an estimated $59.5 billion
annually

8. Cisco Confidential 8
© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Software Quality Challenges

9. Security
There are two types of companies
Those that have been hacked, and those who don't know they have
been hacked John Chambers
Ex-Chairman, Cisco Systems,
Inc.
Data
Governance/Classifi
cation/Handling
Policy Compliance
Regulatory
Compliance
Network Security Encryption
Authentication and
Access Controls
Logging, Auditing
Vulnerability
Management
Malware Protection Patch Management Endpoint Protection
Mobile & Wireless
Security
Source: https://www.tcdi.com/cybersecurity/it-security-risk-assessment/

10. Regulations, Laws, Standards
HIPAA
(Health
Insurance
Portability and
Accountability
Act)
HITECH
Omnibus Rule
PCI-DSS
(The Payment
Card Industry
Data Security
Standard)
GDPR
(General Data
Protection
Regulation)
CCPA
(California
Consumer
Privacy Act)
AICPA
(American
Institute of
Certified Public
Accountants)
SOX
(Sarbanes-Oxley
Act)
GLBA
(Gramm-Leach-
Bliley Act)
FISMA
(Federal
Information
Security
Modernization
Act of 2014)
FedRAMP
(The Federal
Risk and
Authorization
Management
Program)
FERPA
(The Family
Educational
Rights and
Privacy Act of
1974)
COPPA
(Children’s
Online Privacy
Protection Rule)
HIPAA
(Health
Insurance
Portability and
Accountability
Act)
HITECH
Omnibus Rule
PCI-DSS
(The Payment
Card Industry
Data Security
Standard)
GDPR
(General Data
Protection
Regulation)
CCPA
(California
Consumer
Privacy Act)
AICPA
(American
Institute of
Certified Public
Accountants)
SOX
(Sarbanes-Oxley
Act)
GLBA
(Gramm-Leach-
Bliley Act)
FISMA
(Federal
Information
Security
Modernization
Act of 2014)
FedRAMP
(The Federal
Risk and
Authorization
Management
Program)
FERPA
(The Family
Educational
Rights and
Privacy Act of
1974)
COPPA
(Children’s
Online Privacy
Protection Rule)
Clinical Research Software (FDA), 21 CFR Part 11 Standard, Banking Industry, and More…

11. User Stories
User Stories
Global Customers /
Stakeholders
Epics
User Stories
Process Dependencies System Dependencies
Project Dependencies
Enterprise Release Management
Critical Business Systems/Regulatory
Agile Team 1
Agile Team 2
Agile Team 3
Enterprise IT Projects

12. Different Types of Testing
Integration Testing
Unit Testing
Language Testing
Progression / Regression Testing
Business Acceptance Testing
Reliability / Availability
Testing
System Testing (Functional, Non-
Functional, Integration)
User Experience Testing
Exploratory Testing
Smoke Testing Performance Testing
Penetration Testing
Need based

13. Software Development: Waterfall to Agile

14. Project Goals
Scope
On-Budget On-Time

15. Waterfall
Requirements Design Development Testing Implementation Support
Delivery Time: Many Months to Years

16. Agile
Product Owner + Scrum Master + Scrum Team
Product Owner + Scrum Master + Scrum Team
Plan and
Commit
Sprint(s)
Demo and
Deliver
Inspect
and Adapt
Incremental
Capability
Incremental
Capability
Continuous
Integration
Continuous
Integration Delivered in Weeks
Delivered in Weeks
Accept Changes
Accept Changes
Learn fast, Learn
often, Improve
Learn fast, Learn
often, Improve

17. Cisco Confidential 17
© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Quality is everyone's responsibility
Dr. W. Edwards Deming
Father of Quality
Who is Everyone?
What Roles every team member play in product testing?
How is the testing performed?
How the quality is measured?
Who is Everyone?
What Roles every team member play in product testing?
How is the testing performed?
How the quality is measured?

18. Vision
Create economic
opportunity for every
member of the global
workforce
Mission
Connect the world’s
professionals to make them
more productive and
successful
Culture
Transformation, Integrity,
Collaboration, Humor,
Results
Values
Members First
Relationship Matters
BeOpen
Honest, and Constructive
Inspire Excellence
Take Intelligent Risks
Act Like anOwner
Source: https://about.linkedin.com/ & https://careers.linkedin.com/culture-and-values
Organization Culture & Values are Key to Quality. Example: LinkedIn

19. Four Eyes Of Quality Model

20. Four Eyes of Quality
Product Vision &
Roadmap
Identify
Quality Goals
Integrate Product
Testing
From Requirements
to Release
Orchestrate Process,
Automate, and
Predict
Analyze and
Continually Improve

21. Product Vision &
Roadmap
Identify
Quality Goals
Integrate Product
Testing
From Requirements to
Release
Orchestrate Process,
Automate, and Predict
Analyze and Continually
Improve

22. Product Vision
 Interaction between potential customer
and potential product
 Describe of the target market
 Product features in terms of concepts and
designs
 Idea of what will constitute the
interaction between the user and the
product
Source: Towards a conceptual framework for product vision João Luís Guilherme Benassi, Daniel Capaldo Amaral and Lucelindo Dias Ferreira (2013)

23. Product Vision &
Roadmap
Identify
Quality Goals
Integrate Product
Testing
From Requirements to
Release
Orchestrate Process,
Automate, and Predict
Analyze and Continually
Improve

24. Characteristics
Functional
Suitability
Performance
Efficiency
Compatibility Usability Reliability Security Maintainability Portability
Quality Focus High NA NA High NA High Low NA
Customer
Product Goals / Project Goals
Cost Vs Benefit

25. Product Vision &
Roadmap
Identify
Quality Goals
Integrate Product
Testing
From Requirements to
Release
Orchestrate Process,
Automate, and Predict
Analyze and Continually
Improve

26. Requirements Analysis & Design Development & Testing Release Post Release Support
Build Quality Incrementally
Integrate and Improve
Business Needs to Technical Needs
Customer Experience to Business Needs

27. Automation and Quality Tools

28. Developer Eye
Who: Developer / QA
Four Eyes of Quality Model: Developer Eye
Functional Suitability – Requirement Traceability - Bidirectional
Usability (Design Thinking / HCI)
Security
Characteristics
Functional
Suitability
Performance
Efficiency
Compatibility Usability Reliability Security Maintainability Portability
Quality Focus High NA NA High NA High Low NA

29. Developer Eye
Who: Developer / QA
Unit Testing Functional Testing Integration Testing
Non-Functional
Testing
Test Coverage > 80%
Peer Review Metrics
Component, #lines, Issues by Category
Test Coverage > 80%
Peer Review Metrics
Component, #lines, Issues by Category
Dev
Dev Dev
Dev QA
QA Dev
Dev QA
QA Dev
Dev QA
QA
Testing in Dev
Testing in Dev
Testing in Stage
Testing in Stage
Four Eyes of Quality Model: Developer Eye

30. Performance / Reliability
Maintainability / Portability / Compatibility
Non-Functional and Operations Eye
Who: Arch / Dev / Specialists
Security
Four Eyes of Quality Model: Non-Functional and Operations
Characteristics
Functional
Suitability
Performance
Efficiency
Compatibility Usability Reliability Security Maintainability Portability
Quality Focus High NA NA High NA High Low NA

31. Maintenance
Portability /
Compatibility
Security
Performance /
Reliability
Dev
Dev Specialist
Specialist
Testing in Dev / Stage
Testing in Dev / Stage
Dedicated Environment
Dedicated Environment
Non-Functional and Operations Eye
Who: Arch / Dev / QA
Dev
Dev Specialist
Specialist
Dev
Dev Specialist
Specialist
Dev
Dev Specialist
Specialist
Four Eyes of Quality Model: Non-Functional and Operations

32. Four Eyes of Quality Model: Product Owner Eye
Functional Suitability
Usability / Persona Based Testing
Product Owner Eye
Who: Product Owner / Business Analyst / QA
Characteristics
Functional
Suitability
Performance
Efficiency
Compatibility Usability Reliability Security Maintainability Portability
Quality Focus High NA NA High NA High Low NA

33. Usability Functionality
Analyst
Analyst PO
PO QA
QA
Testing in Stage / UAT environment
Testing in Stage / UAT environment
Four Eyes of Quality Model: Product Owner Eye
Product Owner Eye
Who: Product Owner / Business Analyst / QA

34. Four Eyes of Quality Model: Customer Eye
Functional Suitability
Usability / Persona Based Testing
The Customer Eye
Who: Customer Focus Groups / Product Owner
Characteristics
Functional
Suitability
Performance
Efficiency
Compatibility Usability Reliability Security Maintainability Portability
Quality Focus High NA NA High NA High Low NA

35. Usability Functionality
Customers
Customers PO
PO Analyst
Analyst
Testing in Stage / UAT environment
Testing in Stage / UAT environment
Four Eyes of Quality Model: Customer Eye
The Customer Eye
Who: Customer Focus Groups / Product Owner

36. The Customer Eye
Who: Customer Focus Groups
/ Product Owner
Product Owner Eye
Who: Product Owner / Business Analyst / QA
Functional and Operations Eye
Who: Arch / Dev / QA
Developer Eye
Who: Developer / QA
Characteristics
Functional
Suitability
Performance
efficiency
Compatibility Usability Reliability Security Maintainability Portability
Quality Focus High NA NA High NA High Low NA
Four Eyes of Quality Model – A Scenario
Functional Suitability
Usability
Security

37. Metrics and Measurement
Nominal
Status
Example: Green, Yellow, Red
Status
Example: Green, Yellow, Red
Ordinal
Backlog Priorities
Rank the requirements in
sequence (1..10)
Backlog Priorities
Rank the requirements in
sequence (1..10)
Interval
Customer Satisfaction
Example: Scale of 1 – 5, 1 being
low and 5 being high
Customer Satisfaction
Example: Scale of 1 – 5, 1 being
low and 5 being high
Ratio
Actual Performance Numbers
Example: Response time 3
Seconds
Actual Performance Numbers
Example: Response time 3
Seconds
Choose SMART Metrics
Buy-in from stakeholders
Periodically Review & Revise

38. Cost of Quality in Agile Enterprise Release
Arch, Design
&
Development
Stage Pre Production Production Post Production
© Kaali Dass
Customer
Dissatisfaction
Defects Escaped from One Environment to Another

39. Cisco Confidential 39
© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Shift Left Quality – Objective and key results
Source: Metrics and Models in Software Quality Engineering, by Stephen H. Kan
Focus Results
Lower Defect
Lower Defect
Lower % of Defects
Lower % of Defects
Lower UAT / BAT Defects
Lower UAT / BAT Defects
Inspect Design and Code Early
Inspect Design and Code Early
Higher % of Reviews
Higher % of Reviews
Through Unit Testing
Through Unit Testing

40. Four Eyes of Quality
Source: Metrics and Models in Software Quality Engineering, by Stephen H. Kan
Customer Satisfaction
Customer Problems
Defects
The Customer Eye
Who: Customer Focus Groups
/ Product Owner
Product Owner Eye
Who: Product Owner /
Business Analyst / QA
Functional and Operations Eye
Who: Arch / Dev / QA
Developer Eye
Who: Developer / QA

41. Summary
1. In addition to stimulating test environment and test data, projects
need to focus on people who can simulate End user scenarios, User
Experience and CustomerOutcomes
2. Four-Eyes of Quality Model leverages Agile methodologies and
optimize unified resources, knowledge, and skills to achieve the
desired customer experience and outcomes

42. Quality is not an act, it is
a habit
-Aristotle

43. Questions?
https://www.linkedin.com/in/kaalidass/
dassconnect@gmail.com
For more information about this topic, please check this URL:
https://www.linkedin.com/pulse/delivering-quality-enterprise-
projects-four-eyes-dass-cism-pmp-phd/