Escorts in Nungambakkam Phone 8250092165 Enjoy 24/7 Escort Service Enjoy Your...
Getting Some Respect - How to Measure and Communicate Your EA Success
1. Getting Some Respect:
How to Measure and
Communicate Your EA Success
David Baker
Michael Janiszewski
Diamond Management and Technology Consultants
david.baker@diamondconsultants.com,
michael.janiszewski@diamondconsultants.com
October 26, 2007
2. Introductions
David Baker Mike Janiszewski
Partner and chief architect for Manager for Diamond
Diamond Management & Management & Technology
Technology Consultants Consultants
We both have extensive experience practicing Enterprise Architecture
as well as developing Enterprise Architecture organizations,
processes, and measurement techniques
Page 1
3. Introduction – Ice Breaker
Split up into groups and discuss your expectations for this
workshop
Before you begin the discussion, assign a person who will take notes and
share the highlights of the discussion with rest of the team
Upon completion of the assignment, share your group’s thoughts
Intro – 5 minutes
Exercise – 10 minutes
Debrief – 15 minutes
Page 2
4. Introduction – Ice Breaker
Take the next 10 minutes to discuss these three questions…
1. What do you hope to learn from this workshop?
2. Why are EA metrics important to your organization?
3. What emerging trends are shaping the future of EA measurement?
Page 3
6. By the end of this course, you should be able to…
Explain the importance and classification of EA metrics
Include appropriate resources in your EA metrics program
Analyze EA measurements to improve your EA program
Integrate EA metrics into your organization’s IT lifecycle
Identify training and management issues associated with launching
an EA program
Identify organizational problems that may limit your EA metrics
program
Design your own EA metrics deliverables and reporting templates
Page 5
7. Agenda
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 6
8. What Problem Are We Solving?
We know EA is valuable, but our stakeholders often do not
Business
“I just can’t see the benefit of our EA spend. Do we really
need this EA thing around?”
IT Managers
“We know doing EA is a best practice, but we can’t really
quantify how good we are at it. You [the business] can’t
measure it that way”
Technical Staff
“Our EA is so conceptual, we don’t know if we’re really doing
what you want us to do with it”
Page 7
9. What Problem Are We Solving?
The reality is…
EA offers significant benefits
Architecture and IT are cost organizations
EA groups don’t have significant budgets
We need to demonstrate the EA value associated with our business
impact
If we can’t quantify our value, we shouldn’t be in business
Metrics are a key part of marketing the value of EA
Page 8
10. What Problem Are We Solving?
Discussion: How do you market your EA
What do you measure?
EA operations?
EA compliance?
Business value?
How do you communicate those measures?
Page 9
11. What Problem Are We Solving?
We can market EA by focusing on results
IT
Business
Managers
wants to know
want to know
EA Metrics
Is my strategy being realized? Is my IT portfolio aligned with
Is my EA delivering business the business?
value? Is IT meeting expectations
efficiently?
What risks am I taking?
Technical
Staff
wants to know
Am I making efficient use of EA assets?
How well am I aligning with our EA?
What things should I NOT be developing?
Page 10
12. What Problem Are We Solving?
Why are EA metrics important?
Metrics quantify what we do in a useful way
Metrics allow stakeholders to make informed decisions
Manage change
Set targets and challenges
Manage risk
Celebrate what goes well
Understand and address what does not
The capability to measure, track, and evaluate drives continual
improvement
Page 11
13. What Problem Are We Solving?
EA metric capture and reporting is difficult for a variety of
reasons
Scope – The Enterprise
Dynamic - EA is a moving target
Skills - Lack of development within the discipline
Vague - Success and failure are sometimes difficult to define
Measurability - Actionable EA deliverables are not always
discrete/easily measurable
Priorities - Stakeholders value other disciplines to assess
performance (e.g., program management, development practices)
Page 12
14. Agenda
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 13
15. Context for Measuring EA
EA Operates within the context of the IT Lifecycle…
Business Release Project
IT Strategic Planning Business
Strategic Execution
Planning (Portfolio
(SDLC) Operations
Planning Mgmt)
Prioritize the allocation of
Develop projects that
IT resources to achieve
Use enterprise and business unit support businesses’
business strategy, in Run the business
direction and goals to drive IT plans annual and strategic
alignment with enterprise
plans
architecture
Portfolio 1 Project
Blueprints
Project
Enterprise Portfolio 2
Filter
Blueprints Blueprints Project
Project
Portfolio 3
Blueprints Project
Multi-Year Plan Budget Cycle Project Cycle Continuous
Page 14
16. Context for Measuring EA
…with EA involvement at every stage
Business Release Project
IT Strategic Planning Business
Strategic Execution
Planning (Portfolio
(SDLC) Operations
Planning Mgmt)
Strategic Architecture Application Portfolio EA Oversight
Innovation Management Architecture Reviews
Blueprinting Application Portfolio Architecture Issue
Standards Definition Rationalization Management
and Management Inventory Refresh Issue Management
Reference Architecture Annual Budgeting Cycle Corrective Action Plan
Impact Analysis Escalation Management
Risk Analysis Exception Management
QA of IT Financials Architecture Support
RFP Process Support Short Term Issue
Content Resolution
Vendor Support Long Term Engagement
Assessment and Needs Assessment
Selection
Page 15
17. Context for Measuring EA
EA activities during strategic planning
Business
IT Strategic
Strategic
Planning
Planning
Objective Define Enterprise Architecture guidelines
Innovation
Associated Architecture Planning
Processes Standards Definition and Management
Reference Architecture
Domain architects responsible for Architecture Planning
Participants Enterprise architects responsible for other Strategy processes
Business owners and SMEs
Things to Guidelines (tech standards, blueprints, reference architecture)
measure used in other Arch. functions, (e.g. Project Execution)
Page 16
18. Context for Measuring EA
EA activities during release planning
Business Release
IT Strategic Planning
Strategic
Planning (Portfolio
Planning Mgmt)
Review the application portfolio
Objective Facilitate financial review of the application portfolio
Review content of RFPs, RFIs; assist in vendor evaluation
Application Inventory Impact Analysis Content
Associated Management Risk Analysis Vendor Support
Processes Inventory Refresh QA of IT Financials Assessment and
Selection
Domain architects responsible for these portfolio activities
Participants
Enterprise architects provide oversight and enterprise view
Updated application Dependencies RFP, RFI
Things to inventory Tech, bus. & delivery RFP/proposal
measure Inventory assessments risks assessment/scoring
Updated costs
Page 17
19. Context for Measuring EA
EA activities during project execution and operations
Business Release
IT Strategic Project Business
Strategic Planning Execution
Planning (Portfolio (SDLC) Operations
Planning Mgmt)
Monitor adherence to standards, blueprints, reference architectures
Objective Manage architecture-related issue resolution/escalation
Provide support for requests for architecture assistance
Issue Management Issue Management Short Term Issue
Associated and Gate Reviews Corrective Action Plan Resolution
Processes Vendor Assessment Escalation Mgmt Long Term Support
Oversight Exception Mgmt Needs Assessment
Domain architects responsible for these delivery activities
Participants
Enterprise architects provide oversight
CAPs, issues, exceptions, deferrals Resolution of CAP
Things to generated during reviews Project support
measure
Page 18
20. Context for Measuring EA
Discussion: How many of these processes do you have in
place?
How mature is your EA practice?
Is there a relationship between the maturity of your EA practice and
your ability to measure it’s impact?
Page 19
21. Agenda
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 20
22. Team Exercise 1
Brainstorm EA metrics
Brainstorm EA metrics by stakeholders
Break into teams, teams will brainstorm metrics in different topic areas
Before you begin the discussion, assign a person who will take notes and
share the highlights of the discussion with rest of the team
Upon completion of the assignment, debrief rest of the team about list of EA
metrics required by various stakeholders
Intro – 5 minutes
Exercise – 15 minutes
Debrief – 15 minutes
Page 21
23. Team Exercise 1
Brainstorm EA metrics
Topics for discussion
IT
Business
Managers
wants to know
want to know
EA Metrics
Topic 1 - Brainstorm a list of EA Topic 2 - Brainstorm a list of
metrics necessary for Business EA metrics necessary for IT
Managers
Technical
Staff
wants to know
Topic 3 - Brainstorm a list of EA
metrics required by Technical Staff
Page 22
24. Agenda
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 23
25. Agenda
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 24
26. EA Metrics – What to Measure
EA not only takes place at different points in the Lifecycle, it also
takes place at different levels of detail
Scope:
Enterprise The Enterprise
Business Business Business Business
One (or more) Domain Domain Domain Domain
lines of (line of (line of (line of (line of
business business) business) business) business)
One (or more)
individual
projects
Page 25
27. EA Metrics – What to Measure
It is helpful to group EA metrics into a small number of
categories
Measures benefits delivered as result of architecture
Value Creation
processes
IT/Business Measures how aligned a technical project is with respect to
Alignment the underlying business strategic requirements
Measures architectural risks inherent in project activities
Risk Management
(e.g. new platform risks, sunsetting, upgrades)
Operational Measures compliance accuracy with respect to enterprise
Effectiveness architecture
Measures effectiveness of architecture processes such as
governance
Page 26
28. EA Metrics – What to Measure
Measuring EA is different from program management
Enterprise Architecture Program Management
Cross-LOB and cross-program Cross-project and project
measures measures
On the path to the future state On-time, on-budget
Alignment focused Execution focused
Blueprint earned value Project earned value
Business operations System operations
e.g. compliance with SLAs e.g. system uptime
Page 27
29. EA Metrics – What to Measure
Supporting that difference requires a linked architecture…
Business Architecture
Business Business
Strategy Operations
Desired Business Capabilities
Blueprint &
Solution Architecture
Service Model
Roadmap
Information Application Interface
Model Model Model
Infrastructure Model
Technology Architecture
Data App Development Execution Operations Network Security
Models Models Models Models Models Models Models
Page 28
30. EA Metrics – What to Measure
…with sufficient detail to allow the relationships to be measured
Business Architecture
Mission
Linked Organization
Vision
Stakeholders Supported
business Goals
Business Functions business
metrics Objectives
Metrics functions
Business Services
Am I building
Solution Architecture the right
Reuse things
Data / Application / Common Services things?
that are of
interest to the
enterprise
Master
Applications Interfaces
Data
Infrastructure Model
Page 29
31. EA Metrics – What to Measure
Value creation metrics
Metric Definition Calculation
Metric that provides quantitative view of IT initiatives must have assigned
advancement towards a blueprint. Metric business capabilities
Blueprint Value
is created by tracing back from initiative - Calculate project advancement based
Measurement > capability -> objective -> corresponding upon blueprint's earned value
performance indicator (yearly target) measurements
Estimated architecture complexity* for
each initiative
Comparison of the top 10 architecturally Obtain top 10 architecturally complex
Architecture complex projects versus the top 10 high projects
Coverage business priority projects Obtain top 10 projects with the highest
Also applies to Linkage metrics category business priority
Count the number of projects that
intersect
% of Innovation
# of ideas explored that ended up in
Ideas Captures the success rate of ideas
production divided by the total # of ideas
Converted into explored by architecture explored
Production
* Discussed during “project typing” in “Metrics Process – How to Use Metrics”
Page 30
32. EA Metrics – What to Measure
Value creation metrics (continued)
Metric Analysis
The business should ensure that IT is making continual progress on delivery of the
defined business capabilities
Blueprint Value
IT should be able to explain any acceleration / deceleration in delivery
Measurement
The business should be able to clearly see how infrastructure projects
(investments) support desired business capabilities
There is no right or wrong answer here, but it is valuable to show how many of the
Architecture
business priorities depend upon solutions to architecturally complex problems. This
Coverage demonstration is valuable to the EA community.
# of Innovation
A high percentage is good, but does too high mean there’s not enough innovation
Ideas Converted (no risk being taken)?
into Production
Page 31
33. EA Metrics – What to Measure
Additional value creation metrics (continued)
% of Blueprints Defined
# of Shared Technology Services Identified
# of Shared Technology Services Delivered
# of Reference Implementations Developed
Savings from reuse of Standard Patterns
TCO Savings from Application Retirement
Page 32
34. EA Metrics – What to Measure
IT/Business alignment metrics
Metric Definition Calculation
Spend amount Cost of all IT projects associated with IT initiatives must have assigned
per business delivery of a business goal. Can be at the business goals
goal enterprise or LOB level. Sum the initiative costs for each goal
Number of
projects IT initiatives must have assigned
involved with Can be at the enterprise and LOB level business goals
each of the Count the projects under each initiative
business goals
Identifies the traceability between IT initiatives must be traceable back to
Change
business strategy, objectives and business capabilities and capabilities
Agenda strategic technology investments. traceable back to objectives and goals
Page 33
35. EA Metrics – What to Measure
IT/Business alignment metrics (continued)
Metric Analysis
Spend amount Look for cases where the spend is not commensurate with the business priority
per business High priority business goals with low spend (good thing)
goal Low priority business goals with high spend (bad thing)
Number of
projects Helps identify high risk business goals
involved with High number of projects may reflect complex implementation and therefore
each of the high risk
business goals
Can be used to conduct impact analysis of project delivery on advancing the
business strategy (and vice versa)
Change Agenda What is the business impact of NOT spending on an IT project?
What is the IT impact of changing a business strategy?
Page 34
36. EA Metrics – What to Measure
Risk management metrics
Metric Definition Calculation
Calculate percentage of lifecycle risks
Risks associated with project
attributed to each piece of the
Technology architectural component maturity.
technology lifecycle (e.g. emerging
Lifecycle Risk Measured in relation to product
technology, mature outside your
technology lifecycle
company, obsolete technology)
An architecture risk is a technical Accumulate risks as they are identified.
characteristic of a solution that has a Classify both their likelihood (high,
Architecture potential negative impact on that medium, low) and severity (severe,
risk count solution’s chances of a success (e.g. moderate, low)
vendor risk, new technology risk, security
risk, supportability risk, . . . ) Segment by architecture domain
Calculate ratio of high probability and
Severe severe risks to total architectural risks
Measures the number of architecture
Architecture risks across technical domains per project.
Risk Ratio
Segment by architectural domain
Page 35
37. EA Metrics – What to Measure
Risk management metrics (continued)
Metric Analysis
Provides insight into technology lifecycle root causes for projects and articulates
where the enterprise is with respect to technology early adopter, late adopter, etc.
Technology
This information can be used to ensure that projects use the right strategy for
Lifecycle Risk implementing the EA vision (e.g. due to high importance of integration efforts, such
projects should use DB2 (mature) as opposed to IDMS (sunsetting))
Architecture Can help identify common architecture elements that are at risk.
risk count Can help identify weaknesses in architecture skills (architecture domain disciplines)
Severe Provides insight into the overall severity of architectural risks encountered by
Architecture projects. Segmentation by architectural domain allows a starting point to take
Risk Ratio action to remedy at-risk projects (and at-risk reference architectures)
Page 36
38. EA Metrics – What to Measure
Operational effectiveness metrics - compliance
Metric Definition Calculation
Percentage of project using / reusing % of projects using standard products
Compliance standards, services, reference
architectures % reuse of standard services, patterns
Calculate number of projects that
EA Compliance Measures the percentage of projects receive governance signoff on first try
Efficiency with architecture signoff on first try divided by total number of reviewed
projects
Measures the percentage of projects Calculate number of reviewed projects
EA Compliance fully compliant with standards by found to be fully compliant with EA
Ratio business domain and/or technology standards divided by total number of
architecture domain reviewed projects
Measures the frequency of
Exception/ Calculate the total number of
exceptions/deferrals in information,
Deferrals exceptions/deferrals per architecture
application, technical and business
frequency domain
architecture as well as standards
Number of reference material revision
Reference Calculate total number of suggestions
suggestions submitted by business
Recommendation against a given reference material (e.g.
domain and/or technology architecture
Revision Count business domain blueprint)
domain
Page 37
39. EA Metrics – What to Measure
Operational effectiveness metrics - compliance (continued)
Metric Analysis
A low percentage could indicate an incomplete set of guidelines, or poorly
Compliance communicated guidelines, or just the wrong set of guidelines
A high percentage shoud be correlated with better program management metrics
EA Compliance Provides insight into accuracy of projects' delivery in compliance with your
Efficiency company’s EA standards
EA Compliance Allows the business to understand the level of EA compliance within its project
Ratio portfolio. Provides confidence level in delivered architecture.
Too many exceptions and deferrals may indicate an unachievable architecture
Exception/ vision or possibly an unachievable project schedule
Deferral
frequency Exceptions and deferrals are and important and valuable tool for IT governance,
yet deferrals should be tracked to ultimate remediation
Reference
Provides insight into use of reference materials. Also provides insight into
Recommendation
reference material needed by the enterprise.
Revision Count
Page 38
40. EA Metrics – What to Measure
Additional operational effectiveness (compliance) metrics
Number of exceptions (by level)
Reference Recommendation Revision Approval Ratio
Exception/Deferrals frequency by Blueprints across domains
% of Projects Rejected
% of Applications that are in Obsolescence Stage
Page 39
41. EA Metrics – What to Measure
Operational effectiveness metrics - governance
Metric Definition Calculation
Oversight Measures the number of architecture Calculate the total number of oversights
Frequency oversights conducted per week Find the average over a week
Measures the amount of time / level of Calculate the number of days and
Oversight
difficulty to complete the architecture sessions required to complete signoff
Efficiency
reviews for a project per project per gate (also average)
Issues,
exceptions, Totals and averages for the various Track, count, and average totals for
deferrals, dispositions of architecture issues each
corrective action accumulated during reviews Provide totals per project, per gate
plans
Week-to-week comparison of key
A complete view of architecture health metrics (issues, CAPs, exceptions,
Trending includes the total counts as well as their deferrals)
week-over-week trend
Indicate week-over-week trend
Page 40
42. EA Metrics – What to Measure
Operational effectiveness metrics – governance (continued)
Metric Analysis
Oversight
Ensures governance processes are occurring according to targets
Frequency
Oversight Ensures that governance processes remain lightweight and do not impact project
Efficiency delivery schedules
Issues,
exceptions, Low numbers indicate maturity level of project compliance
deferrals, Compare these historical numbers with problems during development and
corrective action operations to identify value of architecture compliance
plans
Trend combined with phase can warn of impending problems
Trending
Upward trend can signal the need to deploy senior architects
Page 41
43. EA Metrics – What to Measure
Additional operational effectiveness (governance) metrics
Issue Closure Efficiency
Oversight Capacity Sufficiency
Page 42
44. Agenda
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 43
45. Organizational Impacts
Discussion: Architecture Organization
Do you identify different types of architects?
Or is everyone becoming an architect?
How are your architects organized?
Highly distributed?
Centralized?
Virtual?
Who develops standards and reference architectures?
If you do blueprints, who does them?
What role does architecture play in project delivery?
Page 44
46. Organizational Impacts
Enterprise architecture is delivered via several different types of
architects
Type Role
Oversee the blueprinting process to translate enterprise and business unit
strategies into the technology roadmap
Enterprise Architect
Ensure interoperability across domains
Ensure compliance with architectural blueprints, roadmaps, and standards
Translate business unit strategy into functional domain (such as Sales) blueprints
Provide deep business and technology knowledge
Domain Architect
Guide project architects to ensure project compliance with enterprise and domain
architectural blueprints
Provide architectural expertise to lead the solution delivery projects
Project Architect Responsible for overall system architecture and integration with other domains
and systems
“Internal” architecture
Requirements analysis, detailed software design
Systems Engineer
Process planning and control
Verification, validation
Software Developer Code, unit testing
Page 45
47. Organizational Impacts
Architects are social, community oriented creatures, found in
groups
CEO
Operations CIO
Business Unit Business Unit Business Unit Enterprise Solutions
Infrastructure
A B C Architecture Delivery
Business Project Infra
Architect Architect architect
Domain Domain Domain Application Systems
Architect Architect Architect Architect Engineer
Information Software
Architect Developers
Enterprise Architecture Virtual Community
Diagram assumes a large
organization, same principles
apply to smaller organizations
but structure will be different
Page 46
48. Organizational Impacts
The various architects engage at different levels to deliver
projects
Business Environment
Emerging Technologies (regulation, compliance,
market factors)
Enterprise Architecture Solutions Delivery
Business Strategy
Business Architects IT Strategy
Application Architecture Program Management Office
Information Architecture Blueprints Application Development Services
(Enterprise &
Infrastructure Architecture Domain) Quality Engineering Services
Services Services Services Services Services Services
Release Planning
Sample Application Domain
Domain Architects Domain Manager
Architecture
Community Project Manager
(updates, exceptions)
Project Architects Systems Engineers
Software Developers
Page 47
49. Organizational Impacts
EA requires the attention of a small group focused on
Architecture project management
The Enterprise Architecture Group should be a small, high powered
group of Enterprise Architects
One per architecture discipline
Domain architects may also be in this organization (or in the
business)
The Enterprise Architecture Group should also contain a small
number of project management resources
Oversees the operation of the virtual community and its
working groups
Manages and tracks oversight and issue management
This is the group that collects and reports the EA metrics
Page 48
50. Organizational Impacts
Transformation to / creation of a centralized enterprise
architecture group requires:
Identify the true architects
Be sure to look in the business groups as well
Software engineers / programmers don’t always make the bets
architects
Establish career path for architects
Application -> project -> domain -> enterprise
Assess architect skills to identify gaps
Define standard architecture processes so that standard metrics can
be measured and reported across the enterprise
Innovation, blueprinting, portfolio assessment, governance,
issue management, etc
Page 49
51. Organizational Impacts
Training and change management
Establish a small enterprise architecture group, initial focus on
establishing the virtual community and defining the architecture
processes (including metric definition)
Provide adequate training and learning opportunities to teach the
standard architecture processes to both architect and non-architect
resources
Implement these processes one domain at a time, gaining
experience as you go
Start with the operational metrics (governance and compliance)
Add risk management metrics as your processes mature
Then add the alignment and business value metrics
Page 50
52. Organizational Impacts
Training metrics
Metric Definition Calculation
% completion of Establish a curriculum
Architecture Measures progress towards creation
Training and delivery of architecture training Use curriculum project plan to estimate
Curriculum completion
Skills Assessments
Identification of sub-par skills among
# of Skills Gaps
the architects and specific plans to Count of skills gaps
addressed
address each
Plans to address each gap
% of Architects To understand the level of training # of architects trained divided by total
trained activity occurring in EA number of architects
Page 51
53. Organizational Impacts
Training metrics (continued)
Metric Analysis
x% completion of
Architecture A high percentage indicates that the Architecture training curriculum is complete
Training and available for use
Curriculum
# of Skills Gaps Ensure any identified skills gap is being addressed either via training or on job
addressed experience
% of Project
Low is bad, high is good
Architects trained
Page 52
54. Agenda
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 53
55. Agenda
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 54
56. Metrics Process – How to Use Metrics
Establish the reporting process
Establish the Architecture program management office
Establish and communicate targets
Track trends
Integrate metrics with
Business & IT Strategy planning (feedback loop)
Release Planning
Project Execution and Operations
Page 55
57. Metrics Process – How to Use Metrics
Remember the context within which EA takes place
Business Release Project
IT Strategic Planning Business
Strategic Execution
Planning (Portfolio
(SDLC) Operations
Planning Mgmt)
Strategic Architecture Application Portfolio EA Oversight
Innovation Management Architecture Reviews
Blueprinting Application Portfolio Architecture Issue
Standards Definition Rationalization Management
and Management Inventory Refresh Issue Management
Reference Architecture Annual Budgeting Cycle Corrective Action Plan
Impact Analysis Escalation Management
Risk Analysis Exception Management
QA of IT Financials Architecture Support
RFP Process Support Short Term Issue
Content Resolution
Vendor Support Long Term Engagement
Assessment and Needs Assessment
Selection
Page 56
58. Metrics Process – How to Use Metrics
Responsibility for collecting and using metrics varies by
lifecycle stage
Business Release Project
IT Strategic Planning Business
Strategic Execution
Planning (Portfolio
(SDLC) Operations
Planning Mgmt)
Strategic Architecture Application Portfolio EA Oversight
Enterprise Architects Management Domain Architects
Domain Architects Domain Architects (primary)
Enterprise Architects
Architecture Issue
Annual Budgeting Cycle Management
Domain Architects Domain Architects
Enterprise Architects
Architecture Support
RFP Process Support Domain Architects
Domain Architects
Page 57
59. Metrics Process – How to Use Metrics
Follow a goal -> objective -> metric framework to establish
targets
Strategic Business Architecture
This approach should be built into
MISSION the your business architecture
Generate domain specific
business metrics for
KEY DRIVERS & GUIDING PRINCIPLES
VISION measuring value and
alignment
For each objective, set
progress goals by establishing
GOAL GOAL GOAL GOAL
yearly targets
Approach can also be applied to
BUSINESS
OBJECTIVE
METRICS generating targets for the risk and
operational metric categories
CAPABILITIES
Page 58
60. Example capabilities form a financial management system
Capability Linked Goal
Maintain common processes centrally Improve financial performance
across the Department to support
centralized administration and
standardization
Support ad hoc data access across all Improve operating efficiency of financial
LOBs. This will provide a simplified, management and procurement functions
single source for report information in a
timely fashion
Generate performance and Consistently comply with federal,
accountability reports, within OMB accounting, and system standards
specified timeline
Capability for drill-down to transaction Improve financial performance
level information
Page 59
61. Metrics Process – How to Use Metrics
Using metrics in the Business & IT Strategy phases
Generate future state
alignment & value
Business Alignment Business Future State metrics
• Confirm business strategy Definition
and goals • Define key business
• Refine understanding of capabilities to enable the
requirements driven by business strategy
recently defined business • Define future state business
strategy vision and models
Apply current state Roadmap
• Develop and prioritize
alignment and value metrics projects and required
investments
• Define implementation plan
Apply current state risk & and analyze economics
• Define risk mitigation strategy
operational metrics
Technology Technology Future
Assessment State Definition
• Review current architecture • Define key technology
and plans capabilities to enable the
• Document current technology business strategy
environment and pain points • Define future state IT vision
• Confirm enterprise and models
Apply future state
architecture strategy and
initiatives
alignment, value and risk
metrics
Page 60
62. Metrics Process – How to Use Metrics
Using metrics in the release planning phase
Blueprint Initiate Release Budget & Risk Assessment Update Release Transition Solution Design
Phase Plan Plan Project Initiation
Roadmap
Initiative’s Apply risk
Budget
metrics
Initiative
Risks Approved &
Initial Updated Funded
Sequenced Initiative Release Project
Release Release
Initiatives Assessment Schedule Charter
Plan Plan
Initiative
Depend-
encies
Apply compliance
Apply alignment Initiative metrics
Resource
metrics Reqs
Initiative/
Domain - wide Initiative-focused Domain-wide Portfolio-focused Project
focused
Page 61
63. Metrics Process – How to Use Metrics
Metrics can be used to determine a project’s complexity rating
Low Medium High
Attribute Key Considerations
Technology & Are there significant exceptions to existing technology
Information standards and reference architectures?
No standard More than one
Are products to be sunset used?
Standards exception exception
Does the project proposal match overall architecture
Architecture planning? Is an implementation out of sequence with
Planning proposed roadmap? No standard
exception
Significant
deviations
Compliance Are applications to be sunset being significantly enhanced?
Is data redundancy being introduced?
Cross-Domain Are there significant impacts to applications outside of the
business domains?
Impacts Are there significant impacts to shared technology services?
No impact Significant
impacts
Do the impacts require new model of integration?
New Are new technologies going to be introduced into the
environment?
Technologies How mature is the new technology?
No new
technologies
Significant new
technologies
Service Level Are the service levels going to be significantly changed (e.g.
number of users, performances, availability levels)?
Changes No service Significant service
level changes level changes
Are we enabling new / modifying existing business
Business capabilities? No impact to Significant impact
Impact Are new user types being introduced? business to business
Are there compliance issues?
Page 62
64. Metrics Process – How to Use Metrics
Project typing drives the level of governance during the project
execution phase
Category Level of EA Metric Reporting Characteristics
Maximum: • Highest # of Gate reviews
Type 3 • “High” metrics indicators
Enterprise & Domain
Projects • Enterprise and Domain Architects
Architect Involvement
involved
• Moderate # of Gate Reviews
Moderate: • “Medium” metrics indicators
Type 2 Domain level business • Domain Architect involved,
Projects value, blueprint alignment, Enterprise Architect involvement as
EA compliance needed
• Fewest # of Gate Reviews
Basic:
Type 1 • “Low” metrics indicators
Operational metrics such as Issue,
All Projects • Driven by Project Architect with
Exception, Deferral Mgmt
oversight by Domain Architects
Page 63
65. Agenda
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 64
66. Deliverables and Templates (Case Examples)
Case #1 – Healthcare Payer
The Company
~$20bn multi-line healthcare insurance company employing over 27,000 people with over
$19bn in revenue managing over 10 million medical members in all 50 states
The Challenge
Lack of integration across business units presents fragmented view of constituents
Second highest cost structure amongst direct competitors
Slow and expensive product development inhibits ability to anticipate or lead in the market
Corporate vision and strategy in place without an execution plan
The EA program
Phase I: Establish a program level architecture governance & design board to ensure the
3-Year program adheres to architecture goals & principles
Phase II: Establish an architecture governance process & metrics/measurement model for
the entire I.T. organization
Page 65
67. Deliverables and Templates (Case Examples)
Sample Report: Value creation report
Purpose: To provide visibility into the alignment of the actual release state with original target
release goals
Description: The Value Creation Report illustrates the each domain’s alignment with functional
and technical business capabilities. This report takes into account the impact of
exceptions and deferrals on the intended release state of each domain and exposes
any gaps that have formed between target and actual end-states.
Frequency: Monthly/Quarterly
Audience: CTO, Business Owners
Empty circle - Not
addressed in release
Half circle - Core
functionality met, high
exception score
3/2 circle - Core
functionality met, low
exception score
Full Circle - On target with
goal
Page 66
68. Deliverables and Templates (Case Examples)
Sample Report: Architecture health status report
Purpose: To provide a detailed illustration of the architectural health of each domain.
Description: The architecture health status report is a synthesis of key metric data provided by
the architecture dashboard and textual highlights of key issues and milestones.
Frequency: Bi-weekly
Audience: CTO, Program Directors
Page 67
69. Deliverables and Templates (Case Examples)
Sample Report: EA governance operational effectiveness report
Purpose: To illustrate project compliance with blueprints and standards and to measure
reference material effectiveness
Description: The Enterprise Operational Effectiveness Report will measure project compliance
with domain architecture standards, enterprise blueprints, and EA standards. It will
also measure the usage and effectiveness of EAG reference material.
Frequency: Bi-weekly
Audience: CTO, EA Core Team, Domain Architects, Program Leads, project Architects
Page 68
70. Deliverables and Templates (Case Examples)
Process example: Weekly project architecture health reporting
EA Manager
Request Architecture
health status
overview
Enterprise
Architecture
PMO EA Admin
Consolidate all
Create health
TSMs health status
report overview
information
Domain admin
Submit
Determine project aggregate
Domain Lead health status health status
report
Project Lead
Project Team
Project Architect
1
Submit Weekly
project health
status report
Legend
Triggering End Org Role
Process Gate Package Process Break
Event Result
Page 69
71. Deliverables and Templates (Case Examples)
Case #2 – Pharmaceutical Company
The Company
A leading pharmaceutical company, with over 100,000 employees and
$19 billion in reported net income for 2006
The Challenge
Having launched an aggressive enterprise-wide modernization program,
the company was looking for a way to ensure the program delivered the
promised capabilities and economic improvements
The EA program
Improve enterprise architecture service delivery through new organization
and governance structures and strong alignment with business value
Provide the mechanisms to more effectively link architecture to concerns
of the business as well as to the updated software Delivery Model
Create meaningful and objective measures which can be used to assess
the success of architectural efforts in business terms
Page 70
73. Deliverables and Templates (Case Examples)
Sample Report – Business Unit Portfolio Architecture Health
Page 72
74. Deliverables and Templates (Case Examples)
Sample Report – EA Operational Effectiveness Dashboard
Page 73
75. Deliverables and Templates (Case Examples)
Sample Report – Business Alignment and Value Creation
Page 74
76. Agenda
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 75
77. Team Exercise 2
What will you do differently when you get back to the office?
Consider metrics that you can begin to implement in your
organization
This is an individual exercise using the worksheets in your handouts
Identify one or two metrics in each category that would benefit your
organization
Jot down some notes about what you might do to get a metrics project
underway
The instructors will be around to help you think about what you can do
Intro – 10 minutes
Exercise – 20 minutes
Page 76
78. Team Exercise 2
EA Metrics Worksheet (1 of 3)
Operational Effectiveness (Governance) What will do you do?
_________________________________________________
Oversight Frequency
_________________________________________________
Oversight Efficiency
Issues, Exceptions, Deferrals, CAPs _________________________________________________
Trending _________________________________________________
Issue Closure Efficiency
_________________________________________________
Oversight Capacity Sufficiency
__________________________________ _________________________________________________
IT/Business Alignment Metrics What will do you do?
_________________________________________________
Spend per Business Goal
_________________________________________________
Number of Projects per Business Goal
Change Agenda Metrics _________________________________________________
__________________________________ _________________________________________________
__________________________________
_________________________________________________
Page 77
79. Team Exercise 2
EA Metrics Worksheet (2 of 3)
Operational Effectiveness (Compliance) What will do you do?
_________________________________________________
Compliance
_________________________________________________
EA Compliance Efficiency
EA Compliance Ratio _________________________________________________
Exception/ Deferrals frequency _________________________________________________
Reference Recommendation Revision Count
_________________________________________________
Number of exceptions (by level)
Reference Revision Approval Ratio _________________________________________________
Exception/Deferrals frequency by Blueprints _________________________________________________
% of Projects Rejected
_________________________________________________
% of Applications in Obsolescence Stage
__________________________________ _________________________________________________
Risk Management Metrics What will do you do?
Technology Lifecycle Risk _________________________________________________
Architecture risk count _________________________________________________
Severe Architecture Risk Ratio
_________________________________________________
Page 78
80. Team Exercise 2
EA Metrics Worksheet (3 of 3)
Value Creation Metrics What will do you do?
_________________________________________________
Blueprint value
_________________________________________________
Architecture Coverage
% of Innovation Idea converted _________________________________________________
Shared Technology Services _________________________________________________
% of Blueprints Defined
_________________________________________________
# of Shared Technology Services Identified
# of Shared Technology Services Delivered _________________________________________________
# of Reference Implementations Developed _________________________________________________
Savings from reuse of Standard Patterns
_________________________________________________
TCO Savings from Application Retirement
__________________________________ _________________________________________________
Page 79
81. Agenda
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 80
82. Agenda
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 81
83. Metrics Frameworks
Federal Enterprise Architecture – Performance Reference Model
Source: “How to Use the Performance Reference Model”, Version 1
Page 82
84. Metrics Frameworks
The PRM structure is designed to clearly articulate the cause
and effect relationship between inputs, outputs and outcomes
Source: “How to Use the Performance Reference Model”, Version 1
Page 83
85. Metrics Frameworks
Key intersections of the PRM and other management processes
Budget Assess progress and inform subsequent budget decisions
GPRA Progress towards relevant PRM Measurement Indicators informs
evaluations to inform subsequent strategic plans
Program Progress towards Measurement Indicators can facilitate cross-agency
Assessment evaluation of PART programs within a BRM Sub-function. Higher-
Rating Tool (PART) scoring programs can share best practices with lower-scoring
programs with similar missions.
IT Capital Planning Progress towards Measurement Indicators inform updated Exhibit
& Investment 300s (business cases) and Agency Post-implementation Reviews
Control (CPIC)
Enterprise Progress toward indicators can determine changes needed in
Architecture Migration Plans
Source: “How to Use the Performance Reference Model”, Version 1
Page 84
86. Metrics Frameworks
Example: Using PRM measures to track business case (Exhibit
300) progress
Source: “How to Use the Performance Reference Model”, Version 1
Page 85
87. Metrics Frameworks
COBIT – The overall framework is a form of IT lifecycle
Source: “CobiT 4.0”, www.itasca.org
Page 86
89. Metrics Frameworks
COBIT – IT Metrics and indicators
Source: “CobiT 4.0”, www.itasca.org
Page 88
90. Agenda
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 89
91. Deliverables and Templates (Case Examples)
An integrated EA toolset strategy is critical to EA success
Enterprise Architecture Tools
Strategic Planning EA Governance Support Project Implementation
Tools Tools Tools
Strategic planning tools such Governance support tools Project Implementation tools
as Metrics/Dashboard tools such as scheduling tools enable the sharing of
enables visualization of ties allows the EA PMO to important knowledge across
between the business, operate and function the enterprise leading to
vision, goals, and objectives. effectively and efficiently. fewer redundancies and
This feedback will help Such tools can optimize increased visibility and
escalate visibility of projects performance of the EA PMO clarity. Issue/ exception/
to upper management deferral tracking and
isolating key issues and management, approval
common inadequacies tracking are some examples
Architecture Deliverables Support
Support tools such as repository tools help provide a centralized store that can help the enterprise gain access to,
learn and implement standards and blueprints. EA modeling tools can also help deliver information in a reusable
and consistent manner
Page 90
92. Deliverables and Templates (Case Examples)
Tools must operate across governance levels and easily scale to
address enterprise-wide number of projects
Architecture Strategic Planning EA Governance Project Implementation Tools
Deliverables Tools Support Tools
Support
LEVEL 3
EA Governance
Dashboard
EA Modeling
Tools
Level 2 & 3 EA
LEVEL 2 Issue, Exceptions &
Deferral
Tracking and
Document Reporting DB
Management
Tools
EA Repository
Tools Decision CAP
Log Mgmt
LEVEL 1
Level 1 EA
Issue, Exceptions &
Scheduling Deferral
Status Reporting
Tools Tracking and
Reporting DB
Page 91
93. Agenda
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 92
94. Wrap-up
Key success factors for an EA metrics program
Get your architects working in unison via common architecture
processes
Innovation, blueprinting, portfolio management, governance, issue
management
Engage your architects in all phases of the IT Lifecycle
Business & IT Strategy
Release Planning (Budgeting/Funding)
Project Execution
Operations
Use a virtual community to report EA progress
Business value, Alignment, Risk, Compliance/Governance
Establish an EA program management office to aggregate and
report EA progress and value
Page 93
95. Thank you and good luck with your metrics efforts!
Page 94