SlideShare a Scribd company logo
| ©2012, Cognizant
©2012, Cognizant
CBC Practices Collaboration:
Elizabeth Lerner, Partner – Program Management Consulting
Susan Schanta, Associate Director - Program & Quality Consulting
September 18, 2012
New Techniques to Elevate
Quality Assurance to Program Assurance
| ©2012, Cognizant1
Agenda
2 Introduction
3 Program Management
7 Program Assurance -
Extending QA Principles
10 Program Schedule Standards
15 Managing Environmental Limitations
11 BRD Ambiguity Reviews
14 Architectural Reviews
17 UAT Strategy
18 Quality Measures
20 Balanced Scorecard
| ©2012, Cognizant
Introduction
• An IT-enabled Transformation
should recognize that these are
typically business-driven, not
technology-led.
• A Program drives the conversion of
strategic intent into reality through
the co-ordination of multiple projects
focused on common objectives.
• Program Management stretches the
project management model through
a heightened level of cross-team
coordination, cost management and
quality controls to deliver a
predictable outcome.
• Controls are needed to deliver
strategic, mission critical, and high
risk programs of transformation
successfully.
• Introducing a quality management
model into the program early
reduces risk and increases flexibility
to respond to challenges, and create
efficiencies, preventing of do-over’s.
Program
Management
Program
Assurance
Trends
Analysis
Value
Management
Planning &
Strategy
Architecture
Project
Management
Implementation
Management
Process
Modeling
Applications
Information
Infrastructure
Planning
Gap
Analysis
Project
Monitoring
Resource
Management
Risk
Management
Strategy
Formulation
2
| ©2012, Cognizant
Program Management
GOVERN
Build and implement best-
practice project governance
and controls while minimizing
bureaucracy
SUPPORT
Help project teams by
providing hands-on PM
expertise. Coach and
mentor project teams and
become a center for project
knowledge and learning
ASSURE
Provide independent
challenge to projects. Assist
in the identification and
management of cross-project
dependencies to ensure
successful delivery
INFORM
Establish regular and
accurate project reporting
to provide ‘early warning’
management information
Program
Management
Office
Deliver
Business
Solutions
Successfully
On Time,
to Cost, and
Quality
Enhance
Organizational
Capability
Resource
Management,
Delivery, and
On Demand
Reporting
Driving the
Strategic
Agenda
A Project is an endeavor funded by finite resources with a finite duration and
a specific outcome. A Program is a group of projects aligned by a single
objective or outcome that can be better managed collectively than individually.
3
| ©2012, Cognizant
Program Management Dimensions
Review & Reporting
Risk & Issue Management
Program Assurance
Resource Management
Cost Management
Benefits Management
Communication &
Business Input
Program Standards
Vendor Management
Governance & Organization Ensure projects are properly set up for success and business need is clearly articulated. Implement
the overall, pragmatic, and consistent framework for managing the portfolio of client initiatives.
Consolidated view of projects. Maintain key management information from CFO level down. Report
key actions and decisions required
Planning & Tracking Implement consistent approach for planning and tracking projects. Provide consolidated view of the
portfolio roadmap
Implement portfolio-wide risk planning and management process to ensure the mechanism for dealing
with issues is speedy / robust
Implement an independent challenge and assurance function that helps identify and resolve key issues
of common concern and threats to benefits across the program
Build and maintain an overall resource plan to align the supply of resources and skills to the demand of
the project portfolio
Program finances are managed, tracked, and reported across the Project Portfolio to ensure the cost of
work remains in budget
Track and report the delivery of benefits arising from the portfolio and the value add to the
organization
Implement the discipline of engaging and managing business input to projects. Develop and implement
a communication plan
Set and maintain standard approach, tools, and techniques to support program delivery. Ensure
standards adherence
Ensure rigor in the procurement and management of subcontracted third party services and products.
Program Management provides real management value by striking a balance between challenging and supporting projects.
Program scope clearly articulated. Implement consistent, pragmatic framework to management a portfolio of projects
4
| ©2012, Cognizant
Program Assurance
Plan > Assure > Control
Purpose: Define quality standards in advance; review deliverables against standards; continuously
question whether delivery is in line with the overall goals.
Benefit: Ensures that what the program delivers will achieve the overall aims
Agree
Agree to expectations up from to
ensure that program deliverables
are fit for purpose.
Record the agreement in a
deliverable description record
including what will be delivered,
target date, format / template, and
to what quality standard.
Program Quality Director helps the
PM define the deliverable, apply
knowledge of best practices, and
ensure the end deliverable will
meet the program’s requirements.
Question
Throughout the lifecycle and at all
levels of the program, ask:
- Does this align with the
program’s vision?
- How will this help achieve the
program’s goals?
Test
When a deliverable is completed, it
must be assessed against the
quality standards set forth in the
deliverable description.
The Program Quality Director will
facilitate the process to ensure an
appropriate independent reviewer
has the opportunity to ‘test’ the
deliverable and sign-off.
5
| ©2012, Cognizant
Polling Question
Program Assurance includes audits for
 Project Schedules, Resource Management, Cost Management
 Requirements Traceability, Environment Definition, Construction
Readiness
 Code Turnover Requirements, System Test Process
 All of the Above
6
| ©2012, Cognizant
Program Assurance - Extending QA Principles
Program Assurance provides quality governance within the Program Management framework
by introducing independent measurements to track and report against deliverables.
Measurements are captured through frequent audits and risk assessments, making it possible
to improve program execution.
Quality Assurance
• Requirements Traceability
• Test Coverage
• Test Execution Pass / Fail Rates
• Defect Management
Program Assurance
• Program Standards
• Planning and Tracking
• Cost Management
• Resource Management
• Environmental Limitations
• Risk / Issue Management
• Quality Gates
 BRD Ambiguity Reviews
 Construction Readiness
 Architectural Reviews
• UAT Strategy
• Zero Uninspected Defects at Release
7
| ©2012, Cognizant
Techniques to Introduce and Maintain Controls
• Establish and maintain controls
• Ensure program management information is up-to-date and available
• Identify and manage business stakeholders
• Consolidate and analyze data across projects
• Produce regular and ad-hoc reports
• Track and realize program benefits
• Ensure all project plans are properly integrated
• Ensure plans remained aligned and dependencies and cross-project
issues and risks are effectively managed
• Ensure project plans are complete, feasible, and all options are considered
• Ensure current status and outlook is realistic
• Ensure all issues are surfaced and addressed
• Advising and supporting individuals (especially PMs) in difficult or
unfamiliar circumstances
• Support the business through effective change management
• Conduct reviews (e.g. risk or reviews, reviews)
• Facilitating planning workshops
• Leading / participating in taskforces teams to resolve issues
Process and data
• Hard techniques; the most
visible PMO deliverables
• Earns the right to do other
things - necessary but not
sufficient
• Set up right and get running
smoothly
Content and people
• How experience is leveraged to
add major value
• Earns the right to advise at a
senior level
• The key to personal
development through the
project office
• Less obvious, need to search
out
Controlling
Reporting
Integrating
Challenging
Coaching
Supporting
Controls are needed to deliver strategic, mission critical, and high risk programs of transformation successfully.
Effective control is required to minimize disruptions and ensure that the program will meet its aims within time and cost constraints.
A set of controls handle the different challenges of program management, underpinned by a robust governance framework.
8
| ©2012, Cognizant
Project Lifecycle - Fundamental Building Block
Approval
to proceed
Project
Management
Stages
Project
Management *
Deliverables
Program
Related
Processes
• Project Brief • Statement of
Requirements
• Business Case
• Project Charter
• Management
Plan
• Cost Plan
• Detailed
Schedules
• Risk Assessment
• Quality Plan
• Create Balanced
Scorecard
• Progress
Reports
• Cost Reports
• Change Requests
• Risk / Issue
Registers
• Implement Plan
• Quality
Assurance
• Assurance Audits
• Post
Implementation
Review (PIR)
• Benefits Dashboard
Pre-feasibility Feasibility
Management
Planning
Execution
and Delivery
Review and
Closure
Start-Up
Work prioritization and approvals process (portfolio management)
Resource scheduling and allocation process
Annual budgeting / forecasting process
* Technical deliverables excluded from diagram
Approval
to proceed
Approval
to proceed
A consistent project lifecycle is the fundamental building block for control
Monitoring and Controlling
9
| ©2012, Cognizant
Program Schedule Standards
How do you measure?
Consistent structure in project schedules results in
consistent work effort measurement in the Integrated
Project Plan.
• Consistent Structure for Tasks
 Across Phases - Requirements, Design, etc.
 By Team - Sub teams within departments
 By Artifact - SDLC documents
• Options
 Tracking artifacts in separate matrix
 Separate stakeholder detailed project plans for
day by day tracking of low level tasks
Make Program Level Decisions
Ensure a consistent measurement of
program progress
10
| ©2012, Cognizant
BRD Ambiguity Reviews
Ambiguity
Type
Search for What to Look for Comment Feedback
Term Appropriate Not quantifiable Ambiguous - Define appropriate
Term As required Not quantifiable Ambiguous - Define as required
Term Difficult Not quantifiable Ambiguous - Define difficult
Term e.g.
Versus
i.e.
Misuse of i.e. versus e.g.
i.e. means "that is"
I work most effectively at small coffee shops i.e.
Starbucks…
e.g. means "for example"
I like melons e.g. honeydew, watermelon,
cantaloupe, etc.
Should be i.e. which means for that is
Should be e.g. which means for example
Term Error Error message or error reason not stated… What is the error message?
Term Error Error condition stated but not what causes the error
condition to occur?
What is the error condition that causes this
scenario?
Term Flexible Not quantified Ambiguous - Define flexible
Term If
Applicable
Not quantified Ambiguous - What determines applicability?
Term ly Search for "ly" to find words that end in "ly" where
the words are not quantifiable because each person
can apply their own meaning.
Examples: frequently, accordingly, efficiently,
normally, similarly, commonly, etc.
Define _______
Examples:
Ambiguous - Define frequently
Ambiguous - Define efficiently
Term Maximum Not quantifiable Ambiguous - Define maximum
Term Minimum Not quantifiable Ambiguous - Define minimum
• Business Requirements define what will be delivered in the release.
• Ambiguity Reviews provide a measurement of whether scope documents are quantitative,
clear, correct, and complete. Ambiguity reviews help eliminate defects in later phases by
building quality into the scoping effort.
11
| ©2012, Cognizant
Sample Ambiguity Review
12
| ©2012, Cognizant
BRD Ambiguity Reviews
275
47
454
7
78
624
3
14
136
0
200
400
600
800
Ambiguous
Term
Clarification
Needed
Field/Namin
g
Consistency
Glossary
Grammar
and Wording
Incomplete
Requirement
Possible
Duplicate
Possible
Missing
Requirement
Unclear
Requirement
New Accounts BRD
Ambiguity Review by Category
141
240
54
27 64
274 79 45 67
50
0 100 200 300 400 500
Ambiguous Term
Field/Naming
Consistency
Incomplete
Requirement
New Accounts
Most Frequent Ambiguous Findings
(Ambiguous Adjective) All
Date/Time?
Define specific
Should Title be included?
SSN/TIN Field consistency
What are the valid values? Is this a limited list?
What are the valid values? Is this a limited list? What about Province
for Canada?
• Can a non-SME understand the
requirements?
• Are the requirements quantifiable?
• Can the design be measured for
requirements fulfillment - Yes or No?
• Does the reader need to have all
requirements available to understand the
context of the requirement?
13
| ©2012, Cognizant
Architectural Reviews
Traditional Approach
• Architecture Documents Created
• Architectural Review Board during Design Phase
 High Level Design
 Functional Specifications
 Detailed Technical Design
Enhanced Approach - Construction Phase Reviews
• Selective Criteria
 Highest Impact on Failure
 Highest Volume
 Most Complex Algorithm
• Format
 White board walkthrough
 Q&A Session
14
| ©2012, Cognizant
Managing Environmental Limitations
• Establish working list of environmental limitations based on existing systems
• Meet with key stakeholders to document limitations
• Facilitate meetings to address solutions to limitations
• Escalate budget requirements for licenses, resources, etc. to Project Manager
15
| ©2012, Cognizant
Polling Question
When does UAT planning occur in large
programs?
 Incrementally throughout the lifecycle of individual
projects
 Only at the end of the program
 Across all project lifecycle phases and at the end of the
program
 After Construction but before text execution
16
| ©2012, Cognizant
UAT Strategy
Shift User Acceptance Testing Earlier
Project Level: Early validation in System Test phase to confirm requirements fulfillment for deliverables such
as reports, screens, calculations and business rules. This also offers an opportunity for the business to refine
their workflow processes in response to the new or updated application.
Program Level UAT: Traditional E2E validation to validate the life of a transaction.
Event Calendar
A predefined schedule to help define,
manage, and report on UAT test
execution. The calendar supports the
operational process of test execution by
providing a shared understanding of the
schedule and expectation for business
user time commitments, scheduled
system activities such as environmental
updates and running batch jobs for
electronic file processing.
Run Books
Compilation of transactions that will be
executed during a specific UAT cycle in
the Event Calendar. Run Books assist
the business users in organizing their
test execution content and help control
the order of processing. Run Books also
provide a reference for the UATC and
business to ensure everyone
understands the goal of what will be
executed in a given time period.
17
| ©2012, Cognizant
Quality Measures
18
| ©2012, Cognizant
Quality Measures
67%
33%
Project 1
Pass Fail
45%
55%
Project 2
Pass Fail
82%
18%
Project 3
Pass Fail
19%
81%
Project 4
Pass Fail
34%
66%
Project 5
Pass Fail
89%
11%
Project 6
Pass Fail
19
| ©2012, Cognizant
Balanced Scorecard
The Balanced Scorecard is a
strategic performance
management tool to track progress
against plan.
• Point in time view
• Mixture of financial and non-
financial measures
• Tracks execution
• Monitors quality
• Highlights risks and issues
Getting Started
• Set expectations low
• Define pain points and
milestones tracked across the
program and / or project
• Determine available data inputs
• Create initial scorecard at the
project level and build into a
program level scorecard
• Use Excel to model appearance
• Automate when stable
20
| ©2012, Cognizant
Balanced Scorecard
Program Level
• % Work Complete
• Budget - Estimate vs. Actual
• Program Tollgates - Final Dates
• Project - Artifact Deliverables
• Top 3 Risks
• Top 3 Issues
Requirements
• Ambiguity Review Results
• BRD to FSD Requirements Mapping
• Requirements Assigned by Team
• Test Case Coverage to Requirements
• Program Completion of FSDs
Testing
• Test Execution % Complete
• Test Case Pass / Fail Rates
• Functional Modules Impact
• Automation % Coverage
• Performance Benchmark Results
Defect Management
• Defect Fix Rate Trend to Complete
• Defect Root Cause
• Defects by State
• Defect Trend by Severity
• Regression Defects
• % Regression Defects - Automation
21
| ©2012, Cognizant| ©2012, Cognizant
Questions
22
| ©2012, Cognizant23
Speakers Bios
Elizabeth Lerner, Partner - Program Management Consulting, Cognizant
Elizabeth (Liz) has spent 30 years managing large-scale change programs as a Strategic Global
Program Director. In this capacity, she has been leading, managing, and delivering mission-critical
results to top-tier global corporations. Liz brings with her a vast experience while working in
various industries, successfully leading and delivering strategic change programs, overseeing
initiatives of large-scale scope and complexity resulting in significant business value, top and
bottom line growth, global industry leadership, business expansion, acquisitions and divestitures,
operational efficiencies, and organizational restructuring. Liz has proven experience in strategy
consulting for Fortune 100 clients. She is recognized for delivering large-scale programs on time
and within budget and scope, thereby consistently meeting and exceeding customer expectations.
Liz can be contacted at elizabeth.lerner@cognizant.com.
Susan Schanta, Associate Director - Program & Quality Consulting, Cognizant
Susan has spent 20 years managing large-scale Program and Quality Assurance programs in new
ventures for global Fortune 500 companies. She brings with her vast experience spanning the
financial, healthcare, and retail domains. She has led several corporate initiatives in business
optimization, onshore-to-offshore transitions, and QA enterprises including SDLC management,
automation, and performance. Susan’s expertise in the industry-best practices and project
execution has helped companies achieve their goals. Her expertise goes into niche skills such as
implementing lifecycle disciplines through change management, sharply reducing budget variances,
improving performance of tiered applications, introducing new disciplines for UAT execution to
achieve improved quality and business workflow processes, and devising calibration methods to
measure these achievements.
Susan can be contacted at susan.schanta@cognizant.com.
| ©2012, Cognizant24
References
Project Management Institute
http://www.pmi.org/
Project Management: 6 Best Practices for Ensuring Software Implementation Success
http://www.cio.com/article/683570/Project_Management_6_Best_Practices_for_Ensuring_Software_
Implementation_Success
The Ambiguity Review Process
http://benderrbt.com/Ambiguityprocess.pdf
Eliminating Ambiguity from Your Requirements
http://www.modernanalyst.com/Resources/Articles/tabid/115/Default.aspx?ArticleType=ArticleView
&ArticleID=989
Building & Implementing a Balanced Scorecard: Nine Steps to Success
http://www.balancedscorecard.org/BSCResources/TheNineStepstoSuccess/tabid/58/Default.aspx
Balanced Scorecard & The Project Manager
http://www.ittoday.info/Articles/Balanced_Scorecard_and_Project_Manager.pdf

More Related Content

What's hot

The Practice of planning and the PMO, John Barben
The Practice of planning and the PMO, John BarbenThe Practice of planning and the PMO, John Barben
The Practice of planning and the PMO, John Barben
Association for Project Management
 
What You Need to Know About PMOs, 26 September 2017
What You Need to Know About PMOs, 26 September 2017What You Need to Know About PMOs, 26 September 2017
What You Need to Know About PMOs, 26 September 2017
Association for Project Management
 
ITFT - Project planning
ITFT  -    Project planningITFT  -    Project planning
ITFT - Project planning
Shruti Kunwar
 
PMI Central Indiana Chapter - The Agile PMO - Michael Nir - Slide deck
PMI Central Indiana Chapter - The Agile PMO - Michael Nir - Slide deckPMI Central Indiana Chapter - The Agile PMO - Michael Nir - Slide deck
PMI Central Indiana Chapter - The Agile PMO - Michael Nir - Slide deck
Michael Nir Business Agility Speaker and Coach
 
Establishing a-pmo-in-the-nhs-15-03-26
Establishing a-pmo-in-the-nhs-15-03-26Establishing a-pmo-in-the-nhs-15-03-26
Establishing a-pmo-in-the-nhs-15-03-26
Bestoutcome
 
PRINCE2 Training Melbourne
PRINCE2 Training MelbournePRINCE2 Training Melbourne
PRINCE2 Training Melbourne
Salim Karim
 
Manage complex projects to success using CMMI, Lean and Scrum
Manage complex projects to success using CMMI, Lean and ScrumManage complex projects to success using CMMI, Lean and Scrum
Manage complex projects to success using CMMI, Lean and Scrum
Systematic
 
PMI Michigan capital - The Agile PMO - Michael Nir - Slide deck
PMI Michigan capital - The Agile PMO - Michael Nir - Slide deckPMI Michigan capital - The Agile PMO - Michael Nir - Slide deck
PMI Michigan capital - The Agile PMO - Michael Nir - Slide deck
Michael Nir Business Agility Speaker and Coach
 
Agile and Scrum 101 –PMI Central Indiana Chapter - Michael Nir - Slide deck
Agile and Scrum 101 –PMI Central Indiana Chapter -  Michael Nir - Slide deckAgile and Scrum 101 –PMI Central Indiana Chapter -  Michael Nir - Slide deck
Agile and Scrum 101 –PMI Central Indiana Chapter - Michael Nir - Slide deck
Michael Nir Business Agility Speaker and Coach
 
PM Tool Meetup
PM Tool MeetupPM Tool Meetup
PM Tool Meetup
Paul McCormack
 
Programme Management in the transport sector - a TfL case study webinar, 19th...
Programme Management in the transport sector - a TfL case study webinar, 19th...Programme Management in the transport sector - a TfL case study webinar, 19th...
Programme Management in the transport sector - a TfL case study webinar, 19th...
Association for Project Management
 
Delivering Business Value through the PMO
Delivering Business Value through the PMODelivering Business Value through the PMO
Delivering Business Value through the PMO
Marisa Silva, the Lucky PM
 
Project Management Toolkit
Project Management ToolkitProject Management Toolkit
Project Management Toolkit
Brian Klempke
 
The PMO Value - Emma-Ruth Arnaz-Pemberton | Project Challenge March 2018
The PMO Value - Emma-Ruth Arnaz-Pemberton | Project Challenge March 2018The PMO Value - Emma-Ruth Arnaz-Pemberton | Project Challenge March 2018
The PMO Value - Emma-Ruth Arnaz-Pemberton | Project Challenge March 2018
Wellingtone
 
Reporting Within a PMO
Reporting Within a PMOReporting Within a PMO
Reporting Within a PMO
Boxalln
 
Project Portfolio Management 2015
Project Portfolio Management 2015Project Portfolio Management 2015
Project Portfolio Management 2015
Peter Schellinck
 
Transforming Project Delivery in MoD Defence Equipment and Support webinar
Transforming Project Delivery in MoD Defence Equipment and Support webinarTransforming Project Delivery in MoD Defence Equipment and Support webinar
Transforming Project Delivery in MoD Defence Equipment and Support webinar
Association for Project Management
 
Enterprise PMO Dependency and Risk Management
Enterprise PMO Dependency and Risk ManagementEnterprise PMO Dependency and Risk Management
Enterprise PMO Dependency and Risk ManagementLenny Shields
 

What's hot (20)

The Practice of planning and the PMO, John Barben
The Practice of planning and the PMO, John BarbenThe Practice of planning and the PMO, John Barben
The Practice of planning and the PMO, John Barben
 
What You Need to Know About PMOs, 26 September 2017
What You Need to Know About PMOs, 26 September 2017What You Need to Know About PMOs, 26 September 2017
What You Need to Know About PMOs, 26 September 2017
 
ITFT - Project planning
ITFT  -    Project planningITFT  -    Project planning
ITFT - Project planning
 
PMI Central Indiana Chapter - The Agile PMO - Michael Nir - Slide deck
PMI Central Indiana Chapter - The Agile PMO - Michael Nir - Slide deckPMI Central Indiana Chapter - The Agile PMO - Michael Nir - Slide deck
PMI Central Indiana Chapter - The Agile PMO - Michael Nir - Slide deck
 
Prince2 quick guide
Prince2 quick guidePrince2 quick guide
Prince2 quick guide
 
Establishing a-pmo-in-the-nhs-15-03-26
Establishing a-pmo-in-the-nhs-15-03-26Establishing a-pmo-in-the-nhs-15-03-26
Establishing a-pmo-in-the-nhs-15-03-26
 
PRINCE2 Training Melbourne
PRINCE2 Training MelbournePRINCE2 Training Melbourne
PRINCE2 Training Melbourne
 
Manage complex projects to success using CMMI, Lean and Scrum
Manage complex projects to success using CMMI, Lean and ScrumManage complex projects to success using CMMI, Lean and Scrum
Manage complex projects to success using CMMI, Lean and Scrum
 
PMI Michigan capital - The Agile PMO - Michael Nir - Slide deck
PMI Michigan capital - The Agile PMO - Michael Nir - Slide deckPMI Michigan capital - The Agile PMO - Michael Nir - Slide deck
PMI Michigan capital - The Agile PMO - Michael Nir - Slide deck
 
Agile and Scrum 101 –PMI Central Indiana Chapter - Michael Nir - Slide deck
Agile and Scrum 101 –PMI Central Indiana Chapter -  Michael Nir - Slide deckAgile and Scrum 101 –PMI Central Indiana Chapter -  Michael Nir - Slide deck
Agile and Scrum 101 –PMI Central Indiana Chapter - Michael Nir - Slide deck
 
PM Tool Meetup
PM Tool MeetupPM Tool Meetup
PM Tool Meetup
 
Programme Management in the transport sector - a TfL case study webinar, 19th...
Programme Management in the transport sector - a TfL case study webinar, 19th...Programme Management in the transport sector - a TfL case study webinar, 19th...
Programme Management in the transport sector - a TfL case study webinar, 19th...
 
Delivering Business Value through the PMO
Delivering Business Value through the PMODelivering Business Value through the PMO
Delivering Business Value through the PMO
 
Apm body of knowledge 6th edition
Apm body of knowledge 6th editionApm body of knowledge 6th edition
Apm body of knowledge 6th edition
 
Project Management Toolkit
Project Management ToolkitProject Management Toolkit
Project Management Toolkit
 
The PMO Value - Emma-Ruth Arnaz-Pemberton | Project Challenge March 2018
The PMO Value - Emma-Ruth Arnaz-Pemberton | Project Challenge March 2018The PMO Value - Emma-Ruth Arnaz-Pemberton | Project Challenge March 2018
The PMO Value - Emma-Ruth Arnaz-Pemberton | Project Challenge March 2018
 
Reporting Within a PMO
Reporting Within a PMOReporting Within a PMO
Reporting Within a PMO
 
Project Portfolio Management 2015
Project Portfolio Management 2015Project Portfolio Management 2015
Project Portfolio Management 2015
 
Transforming Project Delivery in MoD Defence Equipment and Support webinar
Transforming Project Delivery in MoD Defence Equipment and Support webinarTransforming Project Delivery in MoD Defence Equipment and Support webinar
Transforming Project Delivery in MoD Defence Equipment and Support webinar
 
Enterprise PMO Dependency and Risk Management
Enterprise PMO Dependency and Risk ManagementEnterprise PMO Dependency and Risk Management
Enterprise PMO Dependency and Risk Management
 

Similar to New Techniques to Elevate QA to Program Assurance

SSCG Project Management Services
SSCG Project Management ServicesSSCG Project Management Services
SSCG Project Management Services
SSCG Consulting
 
Program management - Fundamentals
Program management   - FundamentalsProgram management   - Fundamentals
Program management - Fundamentals
Julen Mohanty
 
Managing the Consulting Engagement
Managing the Consulting EngagementManaging the Consulting Engagement
Managing the Consulting EngagementSaumya Ganguly
 
Sharath's Resume-Project Manager-V5 30th May'15
Sharath's Resume-Project Manager-V5 30th May'15Sharath's Resume-Project Manager-V5 30th May'15
Sharath's Resume-Project Manager-V5 30th May'15Sharath Sri Karan
 
Program management scope management
Program management   scope managementProgram management   scope management
Program management scope management
Julen Mohanty
 
x-introduction-to-project-management-110720031057-phpapp01 (1).pdf
x-introduction-to-project-management-110720031057-phpapp01 (1).pdfx-introduction-to-project-management-110720031057-phpapp01 (1).pdf
x-introduction-to-project-management-110720031057-phpapp01 (1).pdf
EidTahir
 
Introduction to project management
Introduction to project managementIntroduction to project management
Introduction to project management
Mohamad Ashrafi
 
PMO (Project Management Office)
PMO (Project Management Office)PMO (Project Management Office)
PMO (Project Management Office)
Dilawar Abbas
 
Presentation1.pptx
Presentation1.pptxPresentation1.pptx
Presentation1.pptx
PunePMPML
 
Project management [module 1]
Project management [module 1]Project management [module 1]
Project management [module 1]
OyetadeTobi
 
PMO Manual v2.0 June-18.pptx
PMO Manual v2.0 June-18.pptxPMO Manual v2.0 June-18.pptx
PMO Manual v2.0 June-18.pptx
Ashfaq Ahmad
 
Introduction to project management.ppt for
Introduction to project management.ppt  forIntroduction to project management.ppt  for
Introduction to project management.ppt for
ItcHcm1
 
Making of Stakeholders Delight
Making of Stakeholders Delight Making of Stakeholders Delight
Making of Stakeholders Delight
Duraideivamani S, MBA,PMP,SCPM,PRINCE2,CSM,ITIL
 
Making of Stakeholders Delight
Making of Stakeholders DelightMaking of Stakeholders Delight
Making of Stakeholders Delight
Duraideivamani S, MBA,PMP,SCPM,PRINCE2,CSM,ITIL
 
SSCG Project Management Services
SSCG Project Management ServicesSSCG Project Management Services
SSCG Project Management Services
Eugene Nizeyimana
 
Setting up a project framework phase 1 june 20162
Setting up a project framework phase 1 june 20162Setting up a project framework phase 1 june 20162
Setting up a project framework phase 1 june 20162
Project Management Solutions
 
project control.pptx
project control.pptxproject control.pptx
project control.pptx
AlkaPandya5
 
"Project Management Concepts - Ready Reckoner" By SN Panigrahi, A QUick Refre...
"Project Management Concepts - Ready Reckoner" By SN Panigrahi, A QUick Refre..."Project Management Concepts - Ready Reckoner" By SN Panigrahi, A QUick Refre...
"Project Management Concepts - Ready Reckoner" By SN Panigrahi, A QUick Refre...
SN Panigrahi, PMP
 

Similar to New Techniques to Elevate QA to Program Assurance (20)

SSCG Project Management Services
SSCG Project Management ServicesSSCG Project Management Services
SSCG Project Management Services
 
Program management - Fundamentals
Program management   - FundamentalsProgram management   - Fundamentals
Program management - Fundamentals
 
Managing the Consulting Engagement
Managing the Consulting EngagementManaging the Consulting Engagement
Managing the Consulting Engagement
 
Sharath's Resume-Project Manager-V5 30th May'15
Sharath's Resume-Project Manager-V5 30th May'15Sharath's Resume-Project Manager-V5 30th May'15
Sharath's Resume-Project Manager-V5 30th May'15
 
Program management scope management
Program management   scope managementProgram management   scope management
Program management scope management
 
x-introduction-to-project-management-110720031057-phpapp01 (1).pdf
x-introduction-to-project-management-110720031057-phpapp01 (1).pdfx-introduction-to-project-management-110720031057-phpapp01 (1).pdf
x-introduction-to-project-management-110720031057-phpapp01 (1).pdf
 
041006-Program Management PMI NB - PMI Logo
041006-Program Management PMI NB - PMI Logo041006-Program Management PMI NB - PMI Logo
041006-Program Management PMI NB - PMI Logo
 
Introduction to project management
Introduction to project managementIntroduction to project management
Introduction to project management
 
PMO (Project Management Office)
PMO (Project Management Office)PMO (Project Management Office)
PMO (Project Management Office)
 
Presentation1.pptx
Presentation1.pptxPresentation1.pptx
Presentation1.pptx
 
Project management [module 1]
Project management [module 1]Project management [module 1]
Project management [module 1]
 
PMO Manual v2.0 June-18.pptx
PMO Manual v2.0 June-18.pptxPMO Manual v2.0 June-18.pptx
PMO Manual v2.0 June-18.pptx
 
Introduction to project management.ppt for
Introduction to project management.ppt  forIntroduction to project management.ppt  for
Introduction to project management.ppt for
 
Making of Stakeholders Delight
Making of Stakeholders Delight Making of Stakeholders Delight
Making of Stakeholders Delight
 
Making of Stakeholders Delight
Making of Stakeholders DelightMaking of Stakeholders Delight
Making of Stakeholders Delight
 
SSCG Project Management Services
SSCG Project Management ServicesSSCG Project Management Services
SSCG Project Management Services
 
Setting up a project framework phase 1 june 20162
Setting up a project framework phase 1 june 20162Setting up a project framework phase 1 june 20162
Setting up a project framework phase 1 june 20162
 
project control.pptx
project control.pptxproject control.pptx
project control.pptx
 
"Project Management Concepts - Ready Reckoner" By SN Panigrahi, A QUick Refre...
"Project Management Concepts - Ready Reckoner" By SN Panigrahi, A QUick Refre..."Project Management Concepts - Ready Reckoner" By SN Panigrahi, A QUick Refre...
"Project Management Concepts - Ready Reckoner" By SN Panigrahi, A QUick Refre...
 
Governance
GovernanceGovernance
Governance
 

New Techniques to Elevate QA to Program Assurance

  • 1. | ©2012, Cognizant ©2012, Cognizant CBC Practices Collaboration: Elizabeth Lerner, Partner – Program Management Consulting Susan Schanta, Associate Director - Program & Quality Consulting September 18, 2012 New Techniques to Elevate Quality Assurance to Program Assurance
  • 2. | ©2012, Cognizant1 Agenda 2 Introduction 3 Program Management 7 Program Assurance - Extending QA Principles 10 Program Schedule Standards 15 Managing Environmental Limitations 11 BRD Ambiguity Reviews 14 Architectural Reviews 17 UAT Strategy 18 Quality Measures 20 Balanced Scorecard
  • 3. | ©2012, Cognizant Introduction • An IT-enabled Transformation should recognize that these are typically business-driven, not technology-led. • A Program drives the conversion of strategic intent into reality through the co-ordination of multiple projects focused on common objectives. • Program Management stretches the project management model through a heightened level of cross-team coordination, cost management and quality controls to deliver a predictable outcome. • Controls are needed to deliver strategic, mission critical, and high risk programs of transformation successfully. • Introducing a quality management model into the program early reduces risk and increases flexibility to respond to challenges, and create efficiencies, preventing of do-over’s. Program Management Program Assurance Trends Analysis Value Management Planning & Strategy Architecture Project Management Implementation Management Process Modeling Applications Information Infrastructure Planning Gap Analysis Project Monitoring Resource Management Risk Management Strategy Formulation 2
  • 4. | ©2012, Cognizant Program Management GOVERN Build and implement best- practice project governance and controls while minimizing bureaucracy SUPPORT Help project teams by providing hands-on PM expertise. Coach and mentor project teams and become a center for project knowledge and learning ASSURE Provide independent challenge to projects. Assist in the identification and management of cross-project dependencies to ensure successful delivery INFORM Establish regular and accurate project reporting to provide ‘early warning’ management information Program Management Office Deliver Business Solutions Successfully On Time, to Cost, and Quality Enhance Organizational Capability Resource Management, Delivery, and On Demand Reporting Driving the Strategic Agenda A Project is an endeavor funded by finite resources with a finite duration and a specific outcome. A Program is a group of projects aligned by a single objective or outcome that can be better managed collectively than individually. 3
  • 5. | ©2012, Cognizant Program Management Dimensions Review & Reporting Risk & Issue Management Program Assurance Resource Management Cost Management Benefits Management Communication & Business Input Program Standards Vendor Management Governance & Organization Ensure projects are properly set up for success and business need is clearly articulated. Implement the overall, pragmatic, and consistent framework for managing the portfolio of client initiatives. Consolidated view of projects. Maintain key management information from CFO level down. Report key actions and decisions required Planning & Tracking Implement consistent approach for planning and tracking projects. Provide consolidated view of the portfolio roadmap Implement portfolio-wide risk planning and management process to ensure the mechanism for dealing with issues is speedy / robust Implement an independent challenge and assurance function that helps identify and resolve key issues of common concern and threats to benefits across the program Build and maintain an overall resource plan to align the supply of resources and skills to the demand of the project portfolio Program finances are managed, tracked, and reported across the Project Portfolio to ensure the cost of work remains in budget Track and report the delivery of benefits arising from the portfolio and the value add to the organization Implement the discipline of engaging and managing business input to projects. Develop and implement a communication plan Set and maintain standard approach, tools, and techniques to support program delivery. Ensure standards adherence Ensure rigor in the procurement and management of subcontracted third party services and products. Program Management provides real management value by striking a balance between challenging and supporting projects. Program scope clearly articulated. Implement consistent, pragmatic framework to management a portfolio of projects 4
  • 6. | ©2012, Cognizant Program Assurance Plan > Assure > Control Purpose: Define quality standards in advance; review deliverables against standards; continuously question whether delivery is in line with the overall goals. Benefit: Ensures that what the program delivers will achieve the overall aims Agree Agree to expectations up from to ensure that program deliverables are fit for purpose. Record the agreement in a deliverable description record including what will be delivered, target date, format / template, and to what quality standard. Program Quality Director helps the PM define the deliverable, apply knowledge of best practices, and ensure the end deliverable will meet the program’s requirements. Question Throughout the lifecycle and at all levels of the program, ask: - Does this align with the program’s vision? - How will this help achieve the program’s goals? Test When a deliverable is completed, it must be assessed against the quality standards set forth in the deliverable description. The Program Quality Director will facilitate the process to ensure an appropriate independent reviewer has the opportunity to ‘test’ the deliverable and sign-off. 5
  • 7. | ©2012, Cognizant Polling Question Program Assurance includes audits for  Project Schedules, Resource Management, Cost Management  Requirements Traceability, Environment Definition, Construction Readiness  Code Turnover Requirements, System Test Process  All of the Above 6
  • 8. | ©2012, Cognizant Program Assurance - Extending QA Principles Program Assurance provides quality governance within the Program Management framework by introducing independent measurements to track and report against deliverables. Measurements are captured through frequent audits and risk assessments, making it possible to improve program execution. Quality Assurance • Requirements Traceability • Test Coverage • Test Execution Pass / Fail Rates • Defect Management Program Assurance • Program Standards • Planning and Tracking • Cost Management • Resource Management • Environmental Limitations • Risk / Issue Management • Quality Gates  BRD Ambiguity Reviews  Construction Readiness  Architectural Reviews • UAT Strategy • Zero Uninspected Defects at Release 7
  • 9. | ©2012, Cognizant Techniques to Introduce and Maintain Controls • Establish and maintain controls • Ensure program management information is up-to-date and available • Identify and manage business stakeholders • Consolidate and analyze data across projects • Produce regular and ad-hoc reports • Track and realize program benefits • Ensure all project plans are properly integrated • Ensure plans remained aligned and dependencies and cross-project issues and risks are effectively managed • Ensure project plans are complete, feasible, and all options are considered • Ensure current status and outlook is realistic • Ensure all issues are surfaced and addressed • Advising and supporting individuals (especially PMs) in difficult or unfamiliar circumstances • Support the business through effective change management • Conduct reviews (e.g. risk or reviews, reviews) • Facilitating planning workshops • Leading / participating in taskforces teams to resolve issues Process and data • Hard techniques; the most visible PMO deliverables • Earns the right to do other things - necessary but not sufficient • Set up right and get running smoothly Content and people • How experience is leveraged to add major value • Earns the right to advise at a senior level • The key to personal development through the project office • Less obvious, need to search out Controlling Reporting Integrating Challenging Coaching Supporting Controls are needed to deliver strategic, mission critical, and high risk programs of transformation successfully. Effective control is required to minimize disruptions and ensure that the program will meet its aims within time and cost constraints. A set of controls handle the different challenges of program management, underpinned by a robust governance framework. 8
  • 10. | ©2012, Cognizant Project Lifecycle - Fundamental Building Block Approval to proceed Project Management Stages Project Management * Deliverables Program Related Processes • Project Brief • Statement of Requirements • Business Case • Project Charter • Management Plan • Cost Plan • Detailed Schedules • Risk Assessment • Quality Plan • Create Balanced Scorecard • Progress Reports • Cost Reports • Change Requests • Risk / Issue Registers • Implement Plan • Quality Assurance • Assurance Audits • Post Implementation Review (PIR) • Benefits Dashboard Pre-feasibility Feasibility Management Planning Execution and Delivery Review and Closure Start-Up Work prioritization and approvals process (portfolio management) Resource scheduling and allocation process Annual budgeting / forecasting process * Technical deliverables excluded from diagram Approval to proceed Approval to proceed A consistent project lifecycle is the fundamental building block for control Monitoring and Controlling 9
  • 11. | ©2012, Cognizant Program Schedule Standards How do you measure? Consistent structure in project schedules results in consistent work effort measurement in the Integrated Project Plan. • Consistent Structure for Tasks  Across Phases - Requirements, Design, etc.  By Team - Sub teams within departments  By Artifact - SDLC documents • Options  Tracking artifacts in separate matrix  Separate stakeholder detailed project plans for day by day tracking of low level tasks Make Program Level Decisions Ensure a consistent measurement of program progress 10
  • 12. | ©2012, Cognizant BRD Ambiguity Reviews Ambiguity Type Search for What to Look for Comment Feedback Term Appropriate Not quantifiable Ambiguous - Define appropriate Term As required Not quantifiable Ambiguous - Define as required Term Difficult Not quantifiable Ambiguous - Define difficult Term e.g. Versus i.e. Misuse of i.e. versus e.g. i.e. means "that is" I work most effectively at small coffee shops i.e. Starbucks… e.g. means "for example" I like melons e.g. honeydew, watermelon, cantaloupe, etc. Should be i.e. which means for that is Should be e.g. which means for example Term Error Error message or error reason not stated… What is the error message? Term Error Error condition stated but not what causes the error condition to occur? What is the error condition that causes this scenario? Term Flexible Not quantified Ambiguous - Define flexible Term If Applicable Not quantified Ambiguous - What determines applicability? Term ly Search for "ly" to find words that end in "ly" where the words are not quantifiable because each person can apply their own meaning. Examples: frequently, accordingly, efficiently, normally, similarly, commonly, etc. Define _______ Examples: Ambiguous - Define frequently Ambiguous - Define efficiently Term Maximum Not quantifiable Ambiguous - Define maximum Term Minimum Not quantifiable Ambiguous - Define minimum • Business Requirements define what will be delivered in the release. • Ambiguity Reviews provide a measurement of whether scope documents are quantitative, clear, correct, and complete. Ambiguity reviews help eliminate defects in later phases by building quality into the scoping effort. 11
  • 13. | ©2012, Cognizant Sample Ambiguity Review 12
  • 14. | ©2012, Cognizant BRD Ambiguity Reviews 275 47 454 7 78 624 3 14 136 0 200 400 600 800 Ambiguous Term Clarification Needed Field/Namin g Consistency Glossary Grammar and Wording Incomplete Requirement Possible Duplicate Possible Missing Requirement Unclear Requirement New Accounts BRD Ambiguity Review by Category 141 240 54 27 64 274 79 45 67 50 0 100 200 300 400 500 Ambiguous Term Field/Naming Consistency Incomplete Requirement New Accounts Most Frequent Ambiguous Findings (Ambiguous Adjective) All Date/Time? Define specific Should Title be included? SSN/TIN Field consistency What are the valid values? Is this a limited list? What are the valid values? Is this a limited list? What about Province for Canada? • Can a non-SME understand the requirements? • Are the requirements quantifiable? • Can the design be measured for requirements fulfillment - Yes or No? • Does the reader need to have all requirements available to understand the context of the requirement? 13
  • 15. | ©2012, Cognizant Architectural Reviews Traditional Approach • Architecture Documents Created • Architectural Review Board during Design Phase  High Level Design  Functional Specifications  Detailed Technical Design Enhanced Approach - Construction Phase Reviews • Selective Criteria  Highest Impact on Failure  Highest Volume  Most Complex Algorithm • Format  White board walkthrough  Q&A Session 14
  • 16. | ©2012, Cognizant Managing Environmental Limitations • Establish working list of environmental limitations based on existing systems • Meet with key stakeholders to document limitations • Facilitate meetings to address solutions to limitations • Escalate budget requirements for licenses, resources, etc. to Project Manager 15
  • 17. | ©2012, Cognizant Polling Question When does UAT planning occur in large programs?  Incrementally throughout the lifecycle of individual projects  Only at the end of the program  Across all project lifecycle phases and at the end of the program  After Construction but before text execution 16
  • 18. | ©2012, Cognizant UAT Strategy Shift User Acceptance Testing Earlier Project Level: Early validation in System Test phase to confirm requirements fulfillment for deliverables such as reports, screens, calculations and business rules. This also offers an opportunity for the business to refine their workflow processes in response to the new or updated application. Program Level UAT: Traditional E2E validation to validate the life of a transaction. Event Calendar A predefined schedule to help define, manage, and report on UAT test execution. The calendar supports the operational process of test execution by providing a shared understanding of the schedule and expectation for business user time commitments, scheduled system activities such as environmental updates and running batch jobs for electronic file processing. Run Books Compilation of transactions that will be executed during a specific UAT cycle in the Event Calendar. Run Books assist the business users in organizing their test execution content and help control the order of processing. Run Books also provide a reference for the UATC and business to ensure everyone understands the goal of what will be executed in a given time period. 17
  • 20. | ©2012, Cognizant Quality Measures 67% 33% Project 1 Pass Fail 45% 55% Project 2 Pass Fail 82% 18% Project 3 Pass Fail 19% 81% Project 4 Pass Fail 34% 66% Project 5 Pass Fail 89% 11% Project 6 Pass Fail 19
  • 21. | ©2012, Cognizant Balanced Scorecard The Balanced Scorecard is a strategic performance management tool to track progress against plan. • Point in time view • Mixture of financial and non- financial measures • Tracks execution • Monitors quality • Highlights risks and issues Getting Started • Set expectations low • Define pain points and milestones tracked across the program and / or project • Determine available data inputs • Create initial scorecard at the project level and build into a program level scorecard • Use Excel to model appearance • Automate when stable 20
  • 22. | ©2012, Cognizant Balanced Scorecard Program Level • % Work Complete • Budget - Estimate vs. Actual • Program Tollgates - Final Dates • Project - Artifact Deliverables • Top 3 Risks • Top 3 Issues Requirements • Ambiguity Review Results • BRD to FSD Requirements Mapping • Requirements Assigned by Team • Test Case Coverage to Requirements • Program Completion of FSDs Testing • Test Execution % Complete • Test Case Pass / Fail Rates • Functional Modules Impact • Automation % Coverage • Performance Benchmark Results Defect Management • Defect Fix Rate Trend to Complete • Defect Root Cause • Defects by State • Defect Trend by Severity • Regression Defects • % Regression Defects - Automation 21
  • 23. | ©2012, Cognizant| ©2012, Cognizant Questions 22
  • 24. | ©2012, Cognizant23 Speakers Bios Elizabeth Lerner, Partner - Program Management Consulting, Cognizant Elizabeth (Liz) has spent 30 years managing large-scale change programs as a Strategic Global Program Director. In this capacity, she has been leading, managing, and delivering mission-critical results to top-tier global corporations. Liz brings with her a vast experience while working in various industries, successfully leading and delivering strategic change programs, overseeing initiatives of large-scale scope and complexity resulting in significant business value, top and bottom line growth, global industry leadership, business expansion, acquisitions and divestitures, operational efficiencies, and organizational restructuring. Liz has proven experience in strategy consulting for Fortune 100 clients. She is recognized for delivering large-scale programs on time and within budget and scope, thereby consistently meeting and exceeding customer expectations. Liz can be contacted at elizabeth.lerner@cognizant.com. Susan Schanta, Associate Director - Program & Quality Consulting, Cognizant Susan has spent 20 years managing large-scale Program and Quality Assurance programs in new ventures for global Fortune 500 companies. She brings with her vast experience spanning the financial, healthcare, and retail domains. She has led several corporate initiatives in business optimization, onshore-to-offshore transitions, and QA enterprises including SDLC management, automation, and performance. Susan’s expertise in the industry-best practices and project execution has helped companies achieve their goals. Her expertise goes into niche skills such as implementing lifecycle disciplines through change management, sharply reducing budget variances, improving performance of tiered applications, introducing new disciplines for UAT execution to achieve improved quality and business workflow processes, and devising calibration methods to measure these achievements. Susan can be contacted at susan.schanta@cognizant.com.
  • 25. | ©2012, Cognizant24 References Project Management Institute http://www.pmi.org/ Project Management: 6 Best Practices for Ensuring Software Implementation Success http://www.cio.com/article/683570/Project_Management_6_Best_Practices_for_Ensuring_Software_ Implementation_Success The Ambiguity Review Process http://benderrbt.com/Ambiguityprocess.pdf Eliminating Ambiguity from Your Requirements http://www.modernanalyst.com/Resources/Articles/tabid/115/Default.aspx?ArticleType=ArticleView &ArticleID=989 Building & Implementing a Balanced Scorecard: Nine Steps to Success http://www.balancedscorecard.org/BSCResources/TheNineStepstoSuccess/tabid/58/Default.aspx Balanced Scorecard & The Project Manager http://www.ittoday.info/Articles/Balanced_Scorecard_and_Project_Manager.pdf