The document discusses techniques for elevating quality assurance to program assurance by introducing independent measurements and audits to track program deliverables and improve execution, highlighting the need for controls to successfully deliver strategic and high-risk transformation programs, and providing examples of program assurance techniques including establishing controls, consolidating data across projects, and conducting reviews.

1. | ©2012, Cognizant
©2012, Cognizant
CBC Practices Collaboration:
Elizabeth Lerner, Partner – Program Management Consulting
Susan Schanta, Associate Director - Program & Quality Consulting
September 18, 2012
New Techniques to Elevate
Quality Assurance to Program Assurance

2. | ©2012, Cognizant1
Agenda
2 Introduction
3 Program Management
7 Program Assurance -
Extending QA Principles
10 Program Schedule Standards
15 Managing Environmental Limitations
11 BRD Ambiguity Reviews
14 Architectural Reviews
17 UAT Strategy
18 Quality Measures
20 Balanced Scorecard

3. | ©2012, Cognizant
Introduction
• An IT-enabled Transformation
should recognize that these are
typically business-driven, not
technology-led.
• A Program drives the conversion of
strategic intent into reality through
the co-ordination of multiple projects
focused on common objectives.
• Program Management stretches the
project management model through
a heightened level of cross-team
coordination, cost management and
quality controls to deliver a
predictable outcome.
• Controls are needed to deliver
strategic, mission critical, and high
risk programs of transformation
successfully.
• Introducing a quality management
model into the program early
reduces risk and increases flexibility
to respond to challenges, and create
efficiencies, preventing of do-over’s.
Program
Management
Program
Assurance
Trends
Analysis
Value
Management
Planning &
Strategy
Architecture
Project
Management
Implementation
Management
Process
Modeling
Applications
Information
Infrastructure
Planning
Gap
Analysis
Project
Monitoring
Resource
Management
Risk
Management
Strategy
Formulation
2

4. | ©2012, Cognizant
Program Management
GOVERN
Build and implement best-
practice project governance
and controls while minimizing
bureaucracy
SUPPORT
Help project teams by
providing hands-on PM
expertise. Coach and
mentor project teams and
become a center for project
knowledge and learning
ASSURE
Provide independent
challenge to projects. Assist
in the identification and
management of cross-project
dependencies to ensure
successful delivery
INFORM
Establish regular and
accurate project reporting
to provide ‘early warning’
management information
Program
Management
Office
Deliver
Business
Solutions
Successfully
On Time,
to Cost, and
Quality
Enhance
Organizational
Capability
Resource
Management,
Delivery, and
On Demand
Reporting
Driving the
Strategic
Agenda
A Project is an endeavor funded by finite resources with a finite duration and
a specific outcome. A Program is a group of projects aligned by a single
objective or outcome that can be better managed collectively than individually.
3

5. | ©2012, Cognizant
Program Management Dimensions
Review & Reporting
Risk & Issue Management
Program Assurance
Resource Management
Cost Management
Benefits Management
Communication &
Business Input
Program Standards
Vendor Management
Governance & Organization Ensure projects are properly set up for success and business need is clearly articulated. Implement
the overall, pragmatic, and consistent framework for managing the portfolio of client initiatives.
Consolidated view of projects. Maintain key management information from CFO level down. Report
key actions and decisions required
Planning & Tracking Implement consistent approach for planning and tracking projects. Provide consolidated view of the
portfolio roadmap
Implement portfolio-wide risk planning and management process to ensure the mechanism for dealing
with issues is speedy / robust
Implement an independent challenge and assurance function that helps identify and resolve key issues
of common concern and threats to benefits across the program
Build and maintain an overall resource plan to align the supply of resources and skills to the demand of
the project portfolio
Program finances are managed, tracked, and reported across the Project Portfolio to ensure the cost of
work remains in budget
Track and report the delivery of benefits arising from the portfolio and the value add to the
organization
Implement the discipline of engaging and managing business input to projects. Develop and implement
a communication plan
Set and maintain standard approach, tools, and techniques to support program delivery. Ensure
standards adherence
Ensure rigor in the procurement and management of subcontracted third party services and products.
Program Management provides real management value by striking a balance between challenging and supporting projects.
Program scope clearly articulated. Implement consistent, pragmatic framework to management a portfolio of projects
4

6. | ©2012, Cognizant
Program Assurance
Plan > Assure > Control
Purpose: Define quality standards in advance; review deliverables against standards; continuously
question whether delivery is in line with the overall goals.
Benefit: Ensures that what the program delivers will achieve the overall aims
Agree
Agree to expectations up from to
ensure that program deliverables
are fit for purpose.
Record the agreement in a
deliverable description record
including what will be delivered,
target date, format / template, and
to what quality standard.
Program Quality Director helps the
PM define the deliverable, apply
knowledge of best practices, and
ensure the end deliverable will
meet the program’s requirements.
Question
Throughout the lifecycle and at all
levels of the program, ask:
- Does this align with the
program’s vision?
- How will this help achieve the
program’s goals?
Test
When a deliverable is completed, it
must be assessed against the
quality standards set forth in the
deliverable description.
The Program Quality Director will
facilitate the process to ensure an
appropriate independent reviewer
has the opportunity to ‘test’ the
deliverable and sign-off.
5

7. | ©2012, Cognizant
Polling Question
Program Assurance includes audits for
 Project Schedules, Resource Management, Cost Management
 Requirements Traceability, Environment Definition, Construction
Readiness
 Code Turnover Requirements, System Test Process
 All of the Above
6

8. | ©2012, Cognizant
Program Assurance - Extending QA Principles
Program Assurance provides quality governance within the Program Management framework
by introducing independent measurements to track and report against deliverables.
Measurements are captured through frequent audits and risk assessments, making it possible
to improve program execution.
Quality Assurance
• Requirements Traceability
• Test Coverage
• Test Execution Pass / Fail Rates
• Defect Management
Program Assurance
• Program Standards
• Planning and Tracking
• Cost Management
• Resource Management
• Environmental Limitations
• Risk / Issue Management
• Quality Gates
 BRD Ambiguity Reviews
 Construction Readiness
 Architectural Reviews
• UAT Strategy
• Zero Uninspected Defects at Release
7

9. | ©2012, Cognizant
Techniques to Introduce and Maintain Controls
• Establish and maintain controls
• Ensure program management information is up-to-date and available
• Identify and manage business stakeholders
• Consolidate and analyze data across projects
• Produce regular and ad-hoc reports
• Track and realize program benefits
• Ensure all project plans are properly integrated
• Ensure plans remained aligned and dependencies and cross-project
issues and risks are effectively managed
• Ensure project plans are complete, feasible, and all options are considered
• Ensure current status and outlook is realistic
• Ensure all issues are surfaced and addressed
• Advising and supporting individuals (especially PMs) in difficult or
unfamiliar circumstances
• Support the business through effective change management
• Conduct reviews (e.g. risk or reviews, reviews)
• Facilitating planning workshops
• Leading / participating in taskforces teams to resolve issues
Process and data
• Hard techniques; the most
visible PMO deliverables
• Earns the right to do other
things - necessary but not
sufficient
• Set up right and get running
smoothly
Content and people
• How experience is leveraged to
add major value
• Earns the right to advise at a
senior level
• The key to personal
development through the
project office
• Less obvious, need to search
out
Controlling
Reporting
Integrating
Challenging
Coaching
Supporting
Controls are needed to deliver strategic, mission critical, and high risk programs of transformation successfully.
Effective control is required to minimize disruptions and ensure that the program will meet its aims within time and cost constraints.
A set of controls handle the different challenges of program management, underpinned by a robust governance framework.
8

10. | ©2012, Cognizant
Project Lifecycle - Fundamental Building Block
Approval
to proceed
Project
Management
Stages
Project
Management *
Deliverables
Program
Related
Processes
• Project Brief • Statement of
Requirements
• Business Case
• Project Charter
• Management
Plan
• Cost Plan
• Detailed
Schedules
• Risk Assessment
• Quality Plan
• Create Balanced
Scorecard
• Progress
Reports
• Cost Reports
• Change Requests
• Risk / Issue
Registers
• Implement Plan
• Quality
Assurance
• Assurance Audits
• Post
Implementation
Review (PIR)
• Benefits Dashboard
Pre-feasibility Feasibility
Management
Planning
Execution
and Delivery
Review and
Closure
Start-Up
Work prioritization and approvals process (portfolio management)
Resource scheduling and allocation process
Annual budgeting / forecasting process
* Technical deliverables excluded from diagram
Approval
to proceed
Approval
to proceed
A consistent project lifecycle is the fundamental building block for control
Monitoring and Controlling
9

11. | ©2012, Cognizant
Program Schedule Standards
How do you measure?
Consistent structure in project schedules results in
consistent work effort measurement in the Integrated
Project Plan.
• Consistent Structure for Tasks
 Across Phases - Requirements, Design, etc.
 By Team - Sub teams within departments
 By Artifact - SDLC documents
• Options
 Tracking artifacts in separate matrix
 Separate stakeholder detailed project plans for
day by day tracking of low level tasks
Make Program Level Decisions
Ensure a consistent measurement of
program progress
10

12. | ©2012, Cognizant
BRD Ambiguity Reviews
Ambiguity
Type
Search for What to Look for Comment Feedback
Term Appropriate Not quantifiable Ambiguous - Define appropriate
Term As required Not quantifiable Ambiguous - Define as required
Term Difficult Not quantifiable Ambiguous - Define difficult
Term e.g.
Versus
i.e.
Misuse of i.e. versus e.g.
i.e. means "that is"
I work most effectively at small coffee shops i.e.
Starbucks…
e.g. means "for example"
I like melons e.g. honeydew, watermelon,
cantaloupe, etc.
Should be i.e. which means for that is
Should be e.g. which means for example
Term Error Error message or error reason not stated… What is the error message?
Term Error Error condition stated but not what causes the error
condition to occur?
What is the error condition that causes this
scenario?
Term Flexible Not quantified Ambiguous - Define flexible
Term If
Applicable
Not quantified Ambiguous - What determines applicability?
Term ly Search for "ly" to find words that end in "ly" where
the words are not quantifiable because each person
can apply their own meaning.
Examples: frequently, accordingly, efficiently,
normally, similarly, commonly, etc.
Define _______
Examples:
Ambiguous - Define frequently
Ambiguous - Define efficiently
Term Maximum Not quantifiable Ambiguous - Define maximum
Term Minimum Not quantifiable Ambiguous - Define minimum
• Business Requirements define what will be delivered in the release.
• Ambiguity Reviews provide a measurement of whether scope documents are quantitative,
clear, correct, and complete. Ambiguity reviews help eliminate defects in later phases by
building quality into the scoping effort.
11

13. | ©2012, Cognizant
Sample Ambiguity Review
12

14. | ©2012, Cognizant
BRD Ambiguity Reviews
275
47
454
7
78
624
3
14
136
0
200
400
600
800
Ambiguous
Term
Clarification
Needed
Field/Namin
g
Consistency
Glossary
Grammar
and Wording
Incomplete
Requirement
Possible
Duplicate
Possible
Missing
Requirement
Unclear
Requirement
New Accounts BRD
Ambiguity Review by Category
141
240
54
27 64
274 79 45 67
50
0 100 200 300 400 500
Ambiguous Term
Field/Naming
Consistency
Incomplete
Requirement
New Accounts
Most Frequent Ambiguous Findings
(Ambiguous Adjective) All
Date/Time?
Define specific
Should Title be included?
SSN/TIN Field consistency
What are the valid values? Is this a limited list?
What are the valid values? Is this a limited list? What about Province
for Canada?
• Can a non-SME understand the
requirements?
• Are the requirements quantifiable?
• Can the design be measured for
requirements fulfillment - Yes or No?
• Does the reader need to have all
requirements available to understand the
context of the requirement?
13

15. | ©2012, Cognizant
Architectural Reviews
Traditional Approach
• Architecture Documents Created
• Architectural Review Board during Design Phase
 High Level Design
 Functional Specifications
 Detailed Technical Design
Enhanced Approach - Construction Phase Reviews
• Selective Criteria
 Highest Impact on Failure
 Highest Volume
 Most Complex Algorithm
• Format
 White board walkthrough
 Q&A Session
14

16. | ©2012, Cognizant
Managing Environmental Limitations
• Establish working list of environmental limitations based on existing systems
• Meet with key stakeholders to document limitations
• Facilitate meetings to address solutions to limitations
• Escalate budget requirements for licenses, resources, etc. to Project Manager
15

17. | ©2012, Cognizant
Polling Question
When does UAT planning occur in large
programs?
 Incrementally throughout the lifecycle of individual
projects
 Only at the end of the program
 Across all project lifecycle phases and at the end of the
program
 After Construction but before text execution
16

18. | ©2012, Cognizant
UAT Strategy
Shift User Acceptance Testing Earlier
Project Level: Early validation in System Test phase to confirm requirements fulfillment for deliverables such
as reports, screens, calculations and business rules. This also offers an opportunity for the business to refine
their workflow processes in response to the new or updated application.
Program Level UAT: Traditional E2E validation to validate the life of a transaction.
Event Calendar
A predefined schedule to help define,
manage, and report on UAT test
execution. The calendar supports the
operational process of test execution by
providing a shared understanding of the
schedule and expectation for business
user time commitments, scheduled
system activities such as environmental
updates and running batch jobs for
electronic file processing.
Run Books
Compilation of transactions that will be
executed during a specific UAT cycle in
the Event Calendar. Run Books assist
the business users in organizing their
test execution content and help control
the order of processing. Run Books also
provide a reference for the UATC and
business to ensure everyone
understands the goal of what will be
executed in a given time period.
17

19. | ©2012, Cognizant
Quality Measures
18

20. | ©2012, Cognizant
Quality Measures
67%
33%
Project 1
Pass Fail
45%
55%
Project 2
Pass Fail
82%
18%
Project 3
Pass Fail
19%
81%
Project 4
Pass Fail
34%
66%
Project 5
Pass Fail
89%
11%
Project 6
Pass Fail
19

21. | ©2012, Cognizant
Balanced Scorecard
The Balanced Scorecard is a
strategic performance
management tool to track progress
against plan.
• Point in time view
• Mixture of financial and non-
financial measures
• Tracks execution
• Monitors quality
• Highlights risks and issues
Getting Started
• Set expectations low
• Define pain points and
milestones tracked across the
program and / or project
• Determine available data inputs
• Create initial scorecard at the
project level and build into a
program level scorecard
• Use Excel to model appearance
• Automate when stable
20

22. | ©2012, Cognizant
Balanced Scorecard
Program Level
• % Work Complete
• Budget - Estimate vs. Actual
• Program Tollgates - Final Dates
• Project - Artifact Deliverables
• Top 3 Risks
• Top 3 Issues
Requirements
• Ambiguity Review Results
• BRD to FSD Requirements Mapping
• Requirements Assigned by Team
• Test Case Coverage to Requirements
• Program Completion of FSDs
Testing
• Test Execution % Complete
• Test Case Pass / Fail Rates
• Functional Modules Impact
• Automation % Coverage
• Performance Benchmark Results
Defect Management
• Defect Fix Rate Trend to Complete
• Defect Root Cause
• Defects by State
• Defect Trend by Severity
• Regression Defects
• % Regression Defects - Automation
21

23. | ©2012, Cognizant| ©2012, Cognizant
Questions
22

24. | ©2012, Cognizant23
Speakers Bios
Elizabeth Lerner, Partner - Program Management Consulting, Cognizant
Elizabeth (Liz) has spent 30 years managing large-scale change programs as a Strategic Global
Program Director. In this capacity, she has been leading, managing, and delivering mission-critical
results to top-tier global corporations. Liz brings with her a vast experience while working in
various industries, successfully leading and delivering strategic change programs, overseeing
initiatives of large-scale scope and complexity resulting in significant business value, top and
bottom line growth, global industry leadership, business expansion, acquisitions and divestitures,
operational efficiencies, and organizational restructuring. Liz has proven experience in strategy
consulting for Fortune 100 clients. She is recognized for delivering large-scale programs on time
and within budget and scope, thereby consistently meeting and exceeding customer expectations.
Liz can be contacted at elizabeth.lerner@cognizant.com.
Susan Schanta, Associate Director - Program & Quality Consulting, Cognizant
Susan has spent 20 years managing large-scale Program and Quality Assurance programs in new
ventures for global Fortune 500 companies. She brings with her vast experience spanning the
financial, healthcare, and retail domains. She has led several corporate initiatives in business
optimization, onshore-to-offshore transitions, and QA enterprises including SDLC management,
automation, and performance. Susan’s expertise in the industry-best practices and project
execution has helped companies achieve their goals. Her expertise goes into niche skills such as
implementing lifecycle disciplines through change management, sharply reducing budget variances,
improving performance of tiered applications, introducing new disciplines for UAT execution to
achieve improved quality and business workflow processes, and devising calibration methods to
measure these achievements.
Susan can be contacted at susan.schanta@cognizant.com.

25. | ©2012, Cognizant24
References
Project Management Institute
http://www.pmi.org/
Project Management: 6 Best Practices for Ensuring Software Implementation Success
http://www.cio.com/article/683570/Project_Management_6_Best_Practices_for_Ensuring_Software_
Implementation_Success
The Ambiguity Review Process
http://benderrbt.com/Ambiguityprocess.pdf
Eliminating Ambiguity from Your Requirements
http://www.modernanalyst.com/Resources/Articles/tabid/115/Default.aspx?ArticleType=ArticleView
&ArticleID=989
Building & Implementing a Balanced Scorecard: Nine Steps to Success
http://www.balancedscorecard.org/BSCResources/TheNineStepstoSuccess/tabid/58/Default.aspx
Balanced Scorecard & The Project Manager
http://www.ittoday.info/Articles/Balanced_Scorecard_and_Project_Manager.pdf