Getting Some Respect - How to Measure and Communicate Your EA Success

Getting Some Respect: How to Measure and Communicate Your EA Success David Baker Michael Janiszewski Diamond Management and Technology Consultants david.baker@diamondconsultants.com, michael.janiszewski@diamondconsultants.com October 26, 2007

Introductions David Baker Mike Janiszewski Partner and chief architect for Manager for Diamond Diamond Management & Management & Technology Technology Consultants Consultants We both have extensive experience practicing Enterprise Architecture as well as developing Enterprise Architecture organizations, processes, and measurement techniques Page 1

Introduction – Ice Breaker Split up into groups and discuss your expectations for this workshop  Before you begin the discussion, assign a person who will take notes and share the highlights of the discussion with rest of the team  Upon completion of the assignment, share your group’s thoughts Intro – 5 minutes Exercise – 10 minutes Debrief – 15 minutes Page 2

Introduction – Ice Breaker Take the next 10 minutes to discuss these three questions… 1. What do you hope to learn from this workshop? 2. Why are EA metrics important to your organization? 3. What emerging trends are shaping the future of EA measurement? Page 3

Share your group’s discussion Page 4

By the end of this course, you should be able to…  Explain the importance and classification of EA metrics  Include appropriate resources in your EA metrics program  Analyze EA measurements to improve your EA program  Integrate EA metrics into your organization’s IT lifecycle  Identify training and management issues associated with launching an EA program  Identify organizational problems that may limit your EA metrics program  Design your own EA metrics deliverables and reporting templates Page 5

Agenda 8:00 AM Introduction 8:30 AM What Problem Are We Solving? 9:00 AM Context for Measuring EA 9:30 AM Team Exercise 10:00AM Break 10:15 AM EA Metrics – What to Measure 11:00 AM Organizational Impacts 11:30 AM Lunch 12:00 PM Metrics Process – How to Use Metrics 12:30 PM Deliverables and Templates (Case Examples) 1:15 PM Team Exercise 1:45 PM Break 2:00 PM Metrics Frameworks 2:30 PM Using Tools 2:45 PM Wrap-up Page 6

What Problem Are We Solving? We know EA is valuable, but our stakeholders often do not  Business  “I just can’t see the benefit of our EA spend. Do we really need this EA thing around?”  IT Managers  “We know doing EA is a best practice, but we can’t really quantify how good we are at it. You [the business] can’t measure it that way”  Technical Staff  “Our EA is so conceptual, we don’t know if we’re really doing what you want us to do with it” Page 7

What Problem Are We Solving? The reality is…  EA offers significant benefits  Architecture and IT are cost organizations  EA groups don’t have significant budgets  We need to demonstrate the EA value associated with our business impact  If we can’t quantify our value, we shouldn’t be in business  Metrics are a key part of marketing the value of EA Page 8

What Problem Are We Solving? Discussion: How do you market your EA  What do you measure?  EA operations?  EA compliance?  Business value?  How do you communicate those measures? Page 9

What Problem Are We Solving? We can market EA by focusing on results IT Business Managers wants to know want to know EA Metrics Is my strategy being realized? Is my IT portfolio aligned with Is my EA delivering business the business? value? Is IT meeting expectations efficiently? What risks am I taking? Technical Staff wants to know Am I making efficient use of EA assets? How well am I aligning with our EA? What things should I NOT be developing? Page 10

What Problem Are We Solving? Why are EA metrics important?  Metrics quantify what we do in a useful way  Metrics allow stakeholders to make informed decisions  Manage change  Set targets and challenges  Manage risk  Celebrate what goes well  Understand and address what does not  The capability to measure, track, and evaluate drives continual improvement Page 11

What Problem Are We Solving? EA metric capture and reporting is difficult for a variety of reasons  Scope – The Enterprise  Dynamic - EA is a moving target  Skills - Lack of development within the discipline  Vague - Success and failure are sometimes difficult to define  Measurability - Actionable EA deliverables are not always discrete/easily measurable  Priorities - Stakeholders value other disciplines to assess performance (e.g., program management, development practices) Page 12

Context for Measuring EA EA Operates within the context of the IT Lifecycle… Business Release Project IT Strategic Planning Business Strategic Execution Planning (Portfolio (SDLC) Operations Planning Mgmt) Prioritize the allocation of Develop projects that IT resources to achieve Use enterprise and business unit support businesses’ business strategy, in Run the business direction and goals to drive IT plans annual and strategic alignment with enterprise plans architecture Portfolio 1 Project Blueprints Project Enterprise Portfolio 2 Filter Blueprints Blueprints Project Project Portfolio 3 Blueprints Project Multi-Year Plan Budget Cycle Project Cycle Continuous Page 14

Context for Measuring EA …with EA involvement at every stage Business Release Project IT Strategic Planning Business Strategic Execution Planning (Portfolio (SDLC) Operations Planning Mgmt)  Strategic Architecture  Application Portfolio  EA Oversight  Innovation Management  Architecture Reviews  Blueprinting  Application Portfolio  Architecture Issue  Standards Definition Rationalization Management and Management  Inventory Refresh  Issue Management  Reference Architecture  Annual Budgeting Cycle  Corrective Action Plan  Impact Analysis  Escalation Management  Risk Analysis  Exception Management  QA of IT Financials  Architecture Support  RFP Process Support  Short Term Issue  Content Resolution  Vendor Support  Long Term Engagement  Assessment and  Needs Assessment Selection Page 15

Context for Measuring EA EA activities during strategic planning Business IT Strategic Strategic Planning Planning Objective  Define Enterprise Architecture guidelines  Innovation Associated  Architecture Planning Processes  Standards Definition and Management  Reference Architecture  Domain architects responsible for Architecture Planning Participants  Enterprise architects responsible for other Strategy processes  Business owners and SMEs Things to  Guidelines (tech standards, blueprints, reference architecture) measure used in other Arch. functions, (e.g. Project Execution) Page 16

Context for Measuring EA EA activities during release planning Business Release IT Strategic Planning Strategic Planning (Portfolio Planning Mgmt)  Review the application portfolio Objective  Facilitate financial review of the application portfolio  Review content of RFPs, RFIs; assist in vendor evaluation  Application Inventory  Impact Analysis  Content Associated Management  Risk Analysis  Vendor Support Processes  Inventory Refresh  QA of IT Financials  Assessment and Selection  Domain architects responsible for these portfolio activities Participants  Enterprise architects provide oversight and enterprise view  Updated application  Dependencies  RFP, RFI Things to inventory  Tech, bus. & delivery  RFP/proposal measure  Inventory assessments risks assessment/scoring  Updated costs Page 17

Context for Measuring EA EA activities during project execution and operations Business Release IT Strategic Project Business Strategic Planning Execution Planning (Portfolio (SDLC) Operations Planning Mgmt)  Monitor adherence to standards, blueprints, reference architectures Objective  Manage architecture-related issue resolution/escalation  Provide support for requests for architecture assistance  Issue Management  Issue Management  Short Term Issue Associated and Gate Reviews  Corrective Action Plan Resolution Processes  Vendor Assessment  Escalation Mgmt  Long Term Support Oversight  Exception Mgmt  Needs Assessment  Domain architects responsible for these delivery activities Participants  Enterprise architects provide oversight  CAPs, issues, exceptions, deferrals  Resolution of CAP Things to generated during reviews  Project support measure Page 18

Context for Measuring EA Discussion: How many of these processes do you have in place?  How mature is your EA practice?  Is there a relationship between the maturity of your EA practice and your ability to measure it’s impact? Page 19

Team Exercise 1 Brainstorm EA metrics Brainstorm EA metrics by stakeholders  Break into teams, teams will brainstorm metrics in different topic areas  Before you begin the discussion, assign a person who will take notes and share the highlights of the discussion with rest of the team  Upon completion of the assignment, debrief rest of the team about list of EA metrics required by various stakeholders Intro – 5 minutes Exercise – 15 minutes Debrief – 15 minutes Page 21

Team Exercise 1 Brainstorm EA metrics Topics for discussion IT Business Managers wants to know want to know EA Metrics Topic 1 - Brainstorm a list of EA Topic 2 - Brainstorm a list of metrics necessary for Business EA metrics necessary for IT Managers Technical Staff wants to know Topic 3 - Brainstorm a list of EA metrics required by Technical Staff Page 22

EA Metrics – What to Measure EA not only takes place at different points in the Lifecycle, it also takes place at different levels of detail Scope: Enterprise The Enterprise Business Business Business Business One (or more) Domain Domain Domain Domain lines of (line of (line of (line of (line of business business) business) business) business) One (or more) individual projects Page 25

EA Metrics – What to Measure It is helpful to group EA metrics into a small number of categories  Measures benefits delivered as result of architecture Value Creation processes IT/Business  Measures how aligned a technical project is with respect to Alignment the underlying business strategic requirements  Measures architectural risks inherent in project activities Risk Management (e.g. new platform risks, sunsetting, upgrades) Operational  Measures compliance accuracy with respect to enterprise Effectiveness architecture  Measures effectiveness of architecture processes such as governance Page 26

EA Metrics – What to Measure Measuring EA is different from program management Enterprise Architecture Program Management  Cross-LOB and cross-program  Cross-project and project measures measures  On the path to the future state  On-time, on-budget  Alignment focused  Execution focused  Blueprint earned value  Project earned value  Business operations  System operations  e.g. compliance with SLAs  e.g. system uptime Page 27

EA Metrics – What to Measure Supporting that difference requires a linked architecture… Business Architecture Business Business Strategy Operations Desired Business Capabilities Blueprint & Solution Architecture Service Model Roadmap Information Application Interface Model Model Model Infrastructure Model Technology Architecture Data App Development Execution Operations Network Security Models Models Models Models Models Models Models Page 28

EA Metrics – What to Measure …with sufficient detail to allow the relationships to be measured Business Architecture Mission Linked Organization Vision Stakeholders Supported business Goals Business Functions business metrics Objectives Metrics functions Business Services Am I building Solution Architecture the right Reuse things Data / Application / Common Services things? that are of interest to the enterprise Master Applications Interfaces Data Infrastructure Model Page 29

EA Metrics – What to Measure Value creation metrics Metric Definition Calculation  Metric that provides quantitative view of  IT initiatives must have assigned advancement towards a blueprint. Metric business capabilities Blueprint Value is created by tracing back from initiative -  Calculate project advancement based Measurement > capability -> objective -> corresponding upon blueprint's earned value performance indicator (yearly target) measurements  Estimated architecture complexity* for each initiative  Comparison of the top 10 architecturally  Obtain top 10 architecturally complex Architecture complex projects versus the top 10 high projects Coverage business priority projects  Obtain top 10 projects with the highest  Also applies to Linkage metrics category business priority  Count the number of projects that intersect % of Innovation  # of ideas explored that ended up in Ideas  Captures the success rate of ideas production divided by the total # of ideas Converted into explored by architecture explored Production * Discussed during “project typing” in “Metrics Process – How to Use Metrics” Page 30

EA Metrics – What to Measure Value creation metrics (continued) Metric Analysis  The business should ensure that IT is making continual progress on delivery of the defined business capabilities Blueprint Value  IT should be able to explain any acceleration / deceleration in delivery Measurement  The business should be able to clearly see how infrastructure projects (investments) support desired business capabilities  There is no right or wrong answer here, but it is valuable to show how many of the Architecture business priorities depend upon solutions to architecturally complex problems. This Coverage demonstration is valuable to the EA community. # of Innovation  A high percentage is good, but does too high mean there’s not enough innovation Ideas Converted (no risk being taken)? into Production Page 31

EA Metrics – What to Measure Additional value creation metrics (continued)  % of Blueprints Defined  # of Shared Technology Services Identified  # of Shared Technology Services Delivered  # of Reference Implementations Developed  Savings from reuse of Standard Patterns  TCO Savings from Application Retirement Page 32

EA Metrics – What to Measure IT/Business alignment metrics Metric Definition Calculation Spend amount  Cost of all IT projects associated with  IT initiatives must have assigned per business delivery of a business goal. Can be at the business goals goal enterprise or LOB level.  Sum the initiative costs for each goal Number of projects  IT initiatives must have assigned involved with  Can be at the enterprise and LOB level business goals each of the  Count the projects under each initiative business goals  Identifies the traceability between  IT initiatives must be traceable back to Change business strategy, objectives and business capabilities and capabilities Agenda strategic technology investments. traceable back to objectives and goals Page 33

EA Metrics – What to Measure IT/Business alignment metrics (continued) Metric Analysis Spend amount  Look for cases where the spend is not commensurate with the business priority per business  High priority business goals with low spend (good thing) goal  Low priority business goals with high spend (bad thing) Number of projects  Helps identify high risk business goals involved with  High number of projects may reflect complex implementation and therefore each of the high risk business goals  Can be used to conduct impact analysis of project delivery on advancing the business strategy (and vice versa) Change Agenda  What is the business impact of NOT spending on an IT project?  What is the IT impact of changing a business strategy? Page 34

EA Metrics – What to Measure Risk management metrics Metric Definition Calculation  Calculate percentage of lifecycle risks  Risks associated with project attributed to each piece of the Technology architectural component maturity. technology lifecycle (e.g. emerging Lifecycle Risk Measured in relation to product technology, mature outside your technology lifecycle company, obsolete technology)  An architecture risk is a technical  Accumulate risks as they are identified. characteristic of a solution that has a Classify both their likelihood (high, Architecture potential negative impact on that medium, low) and severity (severe, risk count solution’s chances of a success (e.g. moderate, low) vendor risk, new technology risk, security risk, supportability risk, . . . )  Segment by architecture domain  Calculate ratio of high probability and Severe severe risks to total architectural risks  Measures the number of architecture Architecture risks across technical domains per project. Risk Ratio  Segment by architectural domain Page 35

EA Metrics – What to Measure Risk management metrics (continued) Metric Analysis  Provides insight into technology lifecycle root causes for projects and articulates where the enterprise is with respect to technology early adopter, late adopter, etc. Technology This information can be used to ensure that projects use the right strategy for Lifecycle Risk implementing the EA vision (e.g. due to high importance of integration efforts, such projects should use DB2 (mature) as opposed to IDMS (sunsetting)) Architecture  Can help identify common architecture elements that are at risk. risk count  Can help identify weaknesses in architecture skills (architecture domain disciplines) Severe  Provides insight into the overall severity of architectural risks encountered by Architecture projects. Segmentation by architectural domain allows a starting point to take Risk Ratio action to remedy at-risk projects (and at-risk reference architectures) Page 36

EA Metrics – What to Measure Operational effectiveness metrics - compliance Metric Definition Calculation  Percentage of project using / reusing  % of projects using standard products Compliance standards, services, reference architectures  % reuse of standard services, patterns  Calculate number of projects that EA Compliance  Measures the percentage of projects receive governance signoff on first try Efficiency with architecture signoff on first try divided by total number of reviewed projects  Measures the percentage of projects  Calculate number of reviewed projects EA Compliance fully compliant with standards by found to be fully compliant with EA Ratio business domain and/or technology standards divided by total number of architecture domain reviewed projects  Measures the frequency of Exception/  Calculate the total number of exceptions/deferrals in information, Deferrals exceptions/deferrals per architecture application, technical and business frequency domain architecture as well as standards  Number of reference material revision Reference  Calculate total number of suggestions suggestions submitted by business Recommendation against a given reference material (e.g. domain and/or technology architecture Revision Count business domain blueprint) domain Page 37

EA Metrics – What to Measure Operational effectiveness metrics - compliance (continued) Metric Analysis  A low percentage could indicate an incomplete set of guidelines, or poorly Compliance communicated guidelines, or just the wrong set of guidelines  A high percentage shoud be correlated with better program management metrics EA Compliance  Provides insight into accuracy of projects' delivery in compliance with your Efficiency company’s EA standards EA Compliance  Allows the business to understand the level of EA compliance within its project Ratio portfolio. Provides confidence level in delivered architecture.  Too many exceptions and deferrals may indicate an unachievable architecture Exception/ vision or possibly an unachievable project schedule Deferral frequency  Exceptions and deferrals are and important and valuable tool for IT governance, yet deferrals should be tracked to ultimate remediation Reference  Provides insight into use of reference materials. Also provides insight into Recommendation reference material needed by the enterprise. Revision Count Page 38

EA Metrics – What to Measure Additional operational effectiveness (compliance) metrics  Number of exceptions (by level)  Reference Recommendation Revision Approval Ratio  Exception/Deferrals frequency by Blueprints across domains  % of Projects Rejected  % of Applications that are in Obsolescence Stage Page 39

EA Metrics – What to Measure Operational effectiveness metrics - governance Metric Definition Calculation Oversight  Measures the number of architecture  Calculate the total number of oversights Frequency oversights conducted per week  Find the average over a week  Measures the amount of time / level of  Calculate the number of days and Oversight difficulty to complete the architecture sessions required to complete signoff Efficiency reviews for a project per project per gate (also average) Issues, exceptions,  Totals and averages for the various  Track, count, and average totals for deferrals, dispositions of architecture issues each corrective action accumulated during reviews  Provide totals per project, per gate plans  Week-to-week comparison of key  A complete view of architecture health metrics (issues, CAPs, exceptions, Trending includes the total counts as well as their deferrals) week-over-week trend  Indicate week-over-week trend Page 40

EA Metrics – What to Measure Operational effectiveness metrics – governance (continued) Metric Analysis Oversight  Ensures governance processes are occurring according to targets Frequency Oversight  Ensures that governance processes remain lightweight and do not impact project Efficiency delivery schedules Issues, exceptions,  Low numbers indicate maturity level of project compliance deferrals,  Compare these historical numbers with problems during development and corrective action operations to identify value of architecture compliance plans  Trend combined with phase can warn of impending problems Trending  Upward trend can signal the need to deploy senior architects Page 41

EA Metrics – What to Measure Additional operational effectiveness (governance) metrics  Issue Closure Efficiency  Oversight Capacity Sufficiency Page 42

Organizational Impacts Discussion: Architecture Organization  Do you identify different types of architects?  Or is everyone becoming an architect?  How are your architects organized?  Highly distributed?  Centralized?  Virtual?  Who develops standards and reference architectures?  If you do blueprints, who does them?  What role does architecture play in project delivery? Page 44

Organizational Impacts Enterprise architecture is delivered via several different types of architects Type Role  Oversee the blueprinting process to translate enterprise and business unit strategies into the technology roadmap Enterprise Architect  Ensure interoperability across domains  Ensure compliance with architectural blueprints, roadmaps, and standards  Translate business unit strategy into functional domain (such as Sales) blueprints  Provide deep business and technology knowledge Domain Architect  Guide project architects to ensure project compliance with enterprise and domain architectural blueprints  Provide architectural expertise to lead the solution delivery projects Project Architect  Responsible for overall system architecture and integration with other domains and systems  “Internal” architecture  Requirements analysis, detailed software design Systems Engineer  Process planning and control  Verification, validation Software Developer  Code, unit testing Page 45

Organizational Impacts Architects are social, community oriented creatures, found in groups CEO Operations CIO Business Unit Business Unit Business Unit Enterprise Solutions Infrastructure A B C Architecture Delivery Business Project Infra Architect Architect architect Domain Domain Domain Application Systems Architect Architect Architect Architect Engineer Information Software Architect Developers Enterprise Architecture Virtual Community Diagram assumes a large organization, same principles apply to smaller organizations but structure will be different Page 46

Organizational Impacts The various architects engage at different levels to deliver projects Business Environment Emerging Technologies (regulation, compliance, market factors) Enterprise Architecture Solutions Delivery Business Strategy Business Architects IT Strategy Application Architecture Program Management Office Information Architecture Blueprints Application Development Services (Enterprise & Infrastructure Architecture Domain) Quality Engineering Services Services Services Services Services Services Services Release Planning Sample Application Domain Domain Architects Domain Manager Architecture Community Project Manager (updates, exceptions) Project Architects Systems Engineers Software Developers Page 47

Organizational Impacts EA requires the attention of a small group focused on Architecture project management  The Enterprise Architecture Group should be a small, high powered group of Enterprise Architects  One per architecture discipline  Domain architects may also be in this organization (or in the business)  The Enterprise Architecture Group should also contain a small number of project management resources  Oversees the operation of the virtual community and its working groups  Manages and tracks oversight and issue management  This is the group that collects and reports the EA metrics Page 48

Organizational Impacts Transformation to / creation of a centralized enterprise architecture group requires:  Identify the true architects  Be sure to look in the business groups as well  Software engineers / programmers don’t always make the bets architects  Establish career path for architects  Application -> project -> domain -> enterprise  Assess architect skills to identify gaps  Define standard architecture processes so that standard metrics can be measured and reported across the enterprise  Innovation, blueprinting, portfolio assessment, governance, issue management, etc Page 49

Organizational Impacts Training and change management  Establish a small enterprise architecture group, initial focus on establishing the virtual community and defining the architecture processes (including metric definition)  Provide adequate training and learning opportunities to teach the standard architecture processes to both architect and non-architect resources  Implement these processes one domain at a time, gaining experience as you go  Start with the operational metrics (governance and compliance)  Add risk management metrics as your processes mature  Then add the alignment and business value metrics Page 50

Organizational Impacts Training metrics Metric Definition Calculation % completion of  Establish a curriculum Architecture  Measures progress towards creation Training and delivery of architecture training  Use curriculum project plan to estimate Curriculum completion  Skills Assessments  Identification of sub-par skills among # of Skills Gaps the architects and specific plans to  Count of skills gaps addressed address each  Plans to address each gap % of Architects  To understand the level of training  # of architects trained divided by total trained activity occurring in EA number of architects Page 51

Organizational Impacts Training metrics (continued) Metric Analysis x% completion of Architecture  A high percentage indicates that the Architecture training curriculum is complete Training and available for use Curriculum # of Skills Gaps  Ensure any identified skills gap is being addressed either via training or on job addressed experience % of Project  Low is bad, high is good Architects trained Page 52

Metrics Process – How to Use Metrics Establish the reporting process  Establish the Architecture program management office  Establish and communicate targets  Track trends  Integrate metrics with  Business & IT Strategy planning (feedback loop)  Release Planning  Project Execution and Operations Page 55

Metrics Process – How to Use Metrics Remember the context within which EA takes place Business Release Project IT Strategic Planning Business Strategic Execution Planning (Portfolio (SDLC) Operations Planning Mgmt)  Strategic Architecture  Application Portfolio  EA Oversight  Innovation Management  Architecture Reviews  Blueprinting  Application Portfolio  Architecture Issue  Standards Definition Rationalization Management and Management  Inventory Refresh  Issue Management  Reference Architecture  Annual Budgeting Cycle  Corrective Action Plan  Impact Analysis  Escalation Management  Risk Analysis  Exception Management  QA of IT Financials  Architecture Support  RFP Process Support  Short Term Issue  Content Resolution  Vendor Support  Long Term Engagement  Assessment and  Needs Assessment Selection Page 56

Metrics Process – How to Use Metrics Responsibility for collecting and using metrics varies by lifecycle stage Business Release Project IT Strategic Planning Business Strategic Execution Planning (Portfolio (SDLC) Operations Planning Mgmt)  Strategic Architecture  Application Portfolio  EA Oversight  Enterprise Architects Management  Domain Architects  Domain Architects  Domain Architects (primary)  Enterprise Architects  Architecture Issue  Annual Budgeting Cycle Management  Domain Architects  Domain Architects  Enterprise Architects  Architecture Support  RFP Process Support  Domain Architects  Domain Architects Page 57

Metrics Process – How to Use Metrics Follow a goal -> objective -> metric framework to establish targets Strategic Business Architecture  This approach should be built into MISSION the your business architecture  Generate domain specific business metrics for KEY DRIVERS & GUIDING PRINCIPLES VISION measuring value and alignment  For each objective, set progress goals by establishing GOAL GOAL GOAL GOAL yearly targets  Approach can also be applied to BUSINESS OBJECTIVE METRICS generating targets for the risk and operational metric categories CAPABILITIES Page 58

Example capabilities form a financial management system Capability Linked Goal  Maintain common processes centrally  Improve financial performance across the Department to support centralized administration and standardization  Support ad hoc data access across all  Improve operating efficiency of financial LOBs. This will provide a simplified, management and procurement functions single source for report information in a timely fashion  Generate performance and  Consistently comply with federal, accountability reports, within OMB accounting, and system standards specified timeline  Capability for drill-down to transaction  Improve financial performance level information Page 59

Metrics Process – How to Use Metrics Using metrics in the Business & IT Strategy phases Generate future state alignment & value Business Alignment Business Future State metrics • Confirm business strategy Definition and goals • Define key business • Refine understanding of capabilities to enable the requirements driven by business strategy recently defined business • Define future state business strategy vision and models Apply current state Roadmap • Develop and prioritize alignment and value metrics projects and required investments • Define implementation plan Apply current state risk & and analyze economics • Define risk mitigation strategy operational metrics Technology Technology Future Assessment State Definition • Review current architecture • Define key technology and plans capabilities to enable the • Document current technology business strategy environment and pain points • Define future state IT vision • Confirm enterprise and models Apply future state architecture strategy and initiatives alignment, value and risk metrics Page 60

Metrics Process – How to Use Metrics Using metrics in the release planning phase Blueprint Initiate Release Budget & Risk Assessment Update Release Transition Solution Design Phase Plan Plan Project Initiation Roadmap Initiative’s Apply risk Budget metrics Initiative Risks Approved & Initial Updated Funded Sequenced Initiative Release Project Release Release Initiatives Assessment Schedule Charter Plan Plan Initiative Depend- encies Apply compliance Apply alignment Initiative metrics Resource metrics Reqs Initiative/ Domain - wide Initiative-focused Domain-wide Portfolio-focused Project focused Page 61

Metrics Process – How to Use Metrics Metrics can be used to determine a project’s complexity rating Low Medium High Attribute Key Considerations Technology &  Are there significant exceptions to existing technology Information standards and reference architectures? No standard More than one  Are products to be sunset used? Standards exception exception  Does the project proposal match overall architecture Architecture planning? Is an implementation out of sequence with Planning proposed roadmap? No standard exception Significant deviations Compliance  Are applications to be sunset being significantly enhanced?  Is data redundancy being introduced? Cross-Domain  Are there significant impacts to applications outside of the business domains? Impacts  Are there significant impacts to shared technology services? No impact Significant impacts  Do the impacts require new model of integration? New  Are new technologies going to be introduced into the environment? Technologies  How mature is the new technology? No new technologies Significant new technologies Service Level  Are the service levels going to be significantly changed (e.g. number of users, performances, availability levels)? Changes No service Significant service level changes level changes  Are we enabling new / modifying existing business Business capabilities? No impact to Significant impact Impact  Are new user types being introduced? business to business  Are there compliance issues? Page 62

Metrics Process – How to Use Metrics Project typing drives the level of governance during the project execution phase Category Level of EA Metric Reporting Characteristics Maximum: • Highest # of Gate reviews Type 3 • “High” metrics indicators Enterprise & Domain Projects • Enterprise and Domain Architects Architect Involvement involved • Moderate # of Gate Reviews Moderate: • “Medium” metrics indicators Type 2 Domain level business • Domain Architect involved, Projects value, blueprint alignment, Enterprise Architect involvement as EA compliance needed • Fewest # of Gate Reviews Basic: Type 1 • “Low” metrics indicators Operational metrics such as Issue, All Projects • Driven by Project Architect with Exception, Deferral Mgmt oversight by Domain Architects Page 63

Deliverables and Templates (Case Examples) Case #1 – Healthcare Payer  The Company  ~$20bn multi-line healthcare insurance company employing over 27,000 people with over $19bn in revenue managing over 10 million medical members in all 50 states  The Challenge  Lack of integration across business units presents fragmented view of constituents  Second highest cost structure amongst direct competitors  Slow and expensive product development inhibits ability to anticipate or lead in the market  Corporate vision and strategy in place without an execution plan  The EA program  Phase I: Establish a program level architecture governance & design board to ensure the 3-Year program adheres to architecture goals & principles  Phase II: Establish an architecture governance process & metrics/measurement model for the entire I.T. organization Page 65

Deliverables and Templates (Case Examples) Sample Report: Value creation report Purpose: To provide visibility into the alignment of the actual release state with original target release goals Description: The Value Creation Report illustrates the each domain’s alignment with functional and technical business capabilities. This report takes into account the impact of exceptions and deferrals on the intended release state of each domain and exposes any gaps that have formed between target and actual end-states. Frequency: Monthly/Quarterly Audience: CTO, Business Owners Empty circle - Not addressed in release Half circle - Core functionality met, high exception score 3/2 circle - Core functionality met, low exception score Full Circle - On target with goal Page 66

Deliverables and Templates (Case Examples) Sample Report: Architecture health status report Purpose: To provide a detailed illustration of the architectural health of each domain. Description: The architecture health status report is a synthesis of key metric data provided by the architecture dashboard and textual highlights of key issues and milestones. Frequency: Bi-weekly Audience: CTO, Program Directors Page 67

Deliverables and Templates (Case Examples) Sample Report: EA governance operational effectiveness report Purpose: To illustrate project compliance with blueprints and standards and to measure reference material effectiveness Description: The Enterprise Operational Effectiveness Report will measure project compliance with domain architecture standards, enterprise blueprints, and EA standards. It will also measure the usage and effectiveness of EAG reference material. Frequency: Bi-weekly Audience: CTO, EA Core Team, Domain Architects, Program Leads, project Architects Page 68

Deliverables and Templates (Case Examples) Process example: Weekly project architecture health reporting EA Manager Request Architecture health status overview Enterprise Architecture PMO EA Admin Consolidate all Create health TSMs health status report overview information Domain admin Submit Determine project aggregate Domain Lead health status health status report Project Lead Project Team Project Architect 1 Submit Weekly project health status report Legend Triggering End Org Role Process Gate Package Process Break Event Result Page 69

Deliverables and Templates (Case Examples) Case #2 – Pharmaceutical Company  The Company  A leading pharmaceutical company, with over 100,000 employees and $19 billion in reported net income for 2006  The Challenge  Having launched an aggressive enterprise-wide modernization program, the company was looking for a way to ensure the program delivered the promised capabilities and economic improvements  The EA program  Improve enterprise architecture service delivery through new organization and governance structures and strong alignment with business value  Provide the mechanisms to more effectively link architecture to concerns of the business as well as to the updated software Delivery Model  Create meaningful and objective measures which can be used to assess the success of architectural efforts in business terms Page 70

Deliverables and Templates (Case Examples) Sample Report – Local Portfolio Architecture Health Page 71

Deliverables and Templates (Case Examples) Sample Report – Business Unit Portfolio Architecture Health Page 72

Deliverables and Templates (Case Examples) Sample Report – EA Operational Effectiveness Dashboard Page 73

Deliverables and Templates (Case Examples) Sample Report – Business Alignment and Value Creation Page 74

Team Exercise 2 What will you do differently when you get back to the office? Consider metrics that you can begin to implement in your organization  This is an individual exercise using the worksheets in your handouts  Identify one or two metrics in each category that would benefit your organization  Jot down some notes about what you might do to get a metrics project underway  The instructors will be around to help you think about what you can do Intro – 10 minutes Exercise – 20 minutes Page 76

Team Exercise 2 EA Metrics Worksheet (1 of 3) Operational Effectiveness (Governance) What will do you do? _________________________________________________  Oversight Frequency _________________________________________________  Oversight Efficiency  Issues, Exceptions, Deferrals, CAPs _________________________________________________  Trending _________________________________________________  Issue Closure Efficiency _________________________________________________  Oversight Capacity Sufficiency  __________________________________ _________________________________________________ IT/Business Alignment Metrics What will do you do? _________________________________________________  Spend per Business Goal _________________________________________________  Number of Projects per Business Goal  Change Agenda Metrics _________________________________________________  __________________________________ _________________________________________________  __________________________________ _________________________________________________ Page 77

Team Exercise 2 EA Metrics Worksheet (2 of 3) Operational Effectiveness (Compliance) What will do you do? _________________________________________________  Compliance _________________________________________________  EA Compliance Efficiency  EA Compliance Ratio _________________________________________________  Exception/ Deferrals frequency _________________________________________________  Reference Recommendation Revision Count _________________________________________________  Number of exceptions (by level)  Reference Revision Approval Ratio _________________________________________________  Exception/Deferrals frequency by Blueprints _________________________________________________  % of Projects Rejected _________________________________________________  % of Applications in Obsolescence Stage  __________________________________ _________________________________________________ Risk Management Metrics What will do you do?  Technology Lifecycle Risk _________________________________________________  Architecture risk count _________________________________________________  Severe Architecture Risk Ratio _________________________________________________ Page 78

Team Exercise 2 EA Metrics Worksheet (3 of 3) Value Creation Metrics What will do you do? _________________________________________________  Blueprint value _________________________________________________  Architecture Coverage  % of Innovation Idea converted _________________________________________________  Shared Technology Services _________________________________________________  % of Blueprints Defined _________________________________________________  # of Shared Technology Services Identified  # of Shared Technology Services Delivered _________________________________________________  # of Reference Implementations Developed _________________________________________________  Savings from reuse of Standard Patterns _________________________________________________  TCO Savings from Application Retirement  __________________________________ _________________________________________________ Page 79

Metrics Frameworks Federal Enterprise Architecture – Performance Reference Model Source: “How to Use the Performance Reference Model”, Version 1 Page 82

Metrics Frameworks The PRM structure is designed to clearly articulate the cause and effect relationship between inputs, outputs and outcomes Source: “How to Use the Performance Reference Model”, Version 1 Page 83

Metrics Frameworks Key intersections of the PRM and other management processes Budget Assess progress and inform subsequent budget decisions GPRA Progress towards relevant PRM Measurement Indicators informs evaluations to inform subsequent strategic plans Program Progress towards Measurement Indicators can facilitate cross-agency Assessment evaluation of PART programs within a BRM Sub-function. Higher- Rating Tool (PART) scoring programs can share best practices with lower-scoring programs with similar missions. IT Capital Planning Progress towards Measurement Indicators inform updated Exhibit & Investment 300s (business cases) and Agency Post-implementation Reviews Control (CPIC) Enterprise Progress toward indicators can determine changes needed in Architecture Migration Plans Source: “How to Use the Performance Reference Model”, Version 1 Page 84

Metrics Frameworks Example: Using PRM measures to track business case (Exhibit 300) progress Source: “How to Use the Performance Reference Model”, Version 1 Page 85

Metrics Frameworks COBIT – The overall framework is a form of IT lifecycle Source: “CobiT 4.0”, www.itasca.org Page 86

Metrics Frameworks COBIT – Interrelationship of components Source: “CobiT 4.0”, www.itasca.org Page 87

Metrics Frameworks COBIT – IT Metrics and indicators Source: “CobiT 4.0”, www.itasca.org Page 88

Deliverables and Templates (Case Examples) An integrated EA toolset strategy is critical to EA success Enterprise Architecture Tools Strategic Planning EA Governance Support Project Implementation Tools Tools Tools Strategic planning tools such Governance support tools Project Implementation tools as Metrics/Dashboard tools such as scheduling tools enable the sharing of enables visualization of ties allows the EA PMO to important knowledge across between the business, operate and function the enterprise leading to vision, goals, and objectives. effectively and efficiently. fewer redundancies and This feedback will help Such tools can optimize increased visibility and escalate visibility of projects performance of the EA PMO clarity. Issue/ exception/ to upper management deferral tracking and isolating key issues and management, approval common inadequacies tracking are some examples Architecture Deliverables Support Support tools such as repository tools help provide a centralized store that can help the enterprise gain access to, learn and implement standards and blueprints. EA modeling tools can also help deliver information in a reusable and consistent manner Page 90

Deliverables and Templates (Case Examples) Tools must operate across governance levels and easily scale to address enterprise-wide number of projects Architecture Strategic Planning EA Governance Project Implementation Tools Deliverables Tools Support Tools Support LEVEL 3 EA Governance Dashboard EA Modeling Tools Level 2 & 3 EA LEVEL 2 Issue, Exceptions & Deferral Tracking and Document Reporting DB Management Tools EA Repository Tools Decision CAP Log Mgmt LEVEL 1 Level 1 EA Issue, Exceptions & Scheduling Deferral Status Reporting Tools Tracking and Reporting DB Page 91

Wrap-up Key success factors for an EA metrics program  Get your architects working in unison via common architecture processes  Innovation, blueprinting, portfolio management, governance, issue management  Engage your architects in all phases of the IT Lifecycle  Business & IT Strategy  Release Planning (Budgeting/Funding)  Project Execution  Operations  Use a virtual community to report EA progress  Business value, Alignment, Risk, Compliance/Governance  Establish an EA program management office to aggregate and report EA progress and value Page 93

Thank you and good luck with your metrics efforts! Page 94

Getting Some Respect - How to Measure and Communicate Your EA Success

by David Baker on Nov 24, 2008

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 12

Statistics

Getting Some Respect - How to Measure and Communicate Your EA Success — Presentation Transcript

http://www.slideshare.net	9
http://si.blueprint.st	2
http://www.lmodules.com	1