IT Performance Management - Doug Hubbard


Published on

Information Technology Performance Management
Measuring IT's Contribution to Mission Results
A Case Study of the Applied Information Economics Methodology For an Infrastructure IT Investment

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

IT Performance Management - Doug Hubbard

  1. 1. Information Technology Performance Management Measuring IT's Contribution to Mission Results A Case Study of the Applied Information Economics Methodology For an Infrastructure IT Investment September 2001 Conducted by the IT Performance Management Subcommittee For the Capital Planning and IT Management Committee Of the Federal Chief Information Officers Council
  2. 2. For More Information Regarding the VA Information Security Program: Ruth Anderson Department of Veterans Affairs Office of Information and Technology Code 045C 810 Vermont Avenue, N.W. Washington, DC 20420 Regarding the Applied Information Economics Methodology: Doug Hubbard President Hubbard Decision Research 2 South 761 Parkview Glen Ellyn, IL 60137 630-858-2789 ii
  3. 3. Table of Contents Executive Summary………………………………………………………………………………1 Purpose of Pilot…………………………………………………………………………………...3 Description of Host Agency Business Needs…………………………………………………….4 Description of Agency IT Initiative………………………………………………………………5 Agency AIE Core Team………………………………………………………………………….7 Overview of the AIE Methodology………………………………………………………………9 Steps of AIE Performance Metrics Methodology (Description of process not results)…………12 Estimated vs. Actual Timeline……..……………………………………………………………14 Results…………………………………………………………………………………………...15 Host Agency Opinions about the AIE Methodology …………………………………………...25 Implementation of the AIE Recommendations…………………………………………………25 Additional Comments…………………………………………………………………… … …25 Appendix 1: VA Information Security Survey…………………………………………………27 Appendix 2: Summary of Minimum vs. Optional Investments………………………………...30 Appendix 3: Glossary…………………………………………………………………………..31 Appendix 4: Bibliography……………………………………………………………………...32 iii
  4. 4. Executive Summary Federal executive agencies face significant management and technical challenges when measuring the contribution of information technology investments to mission results as required by the Clinger-Cohen Act. There is a shortage of knowledge and examples of how to measure IT’s contribution to mission results for government agencies with complex missions such as providing for the health and welfare of the citizens of the United States. To close this knowledge gap and to improve Federal performance management practices, the Federal Chief Information Officers Council sponsored pilot demonstrations of two measurement methodologies, Applied Information Economics and Balanced Scorecard. Those pilots, which were completed in May 2001, proved that each methodology was applicable in the federal environment, provided the host agency with a useful performance measures, and provided lessons learned for other federal agencies to benefit. This report presents the findings from the Applied Information Economics pilot. The Department of Veterans Affairs (VA) volunteered to participate in this pilot with its Information Security Program (ISP), which is an approved new infrastructure initiative that will mitigate information security-related risks across the department. The risks include reducing the cost and frequency of viruses, unauthorized access, fraud and other types of losses. The total cost for ISP over five years will be approximately $114 million. Applied Information Economics (AIE) is an analytical methodology that applies proven and practical scientific and mathematical methods to the IT decision process. Although AIE is six years old and has been used commercially in many industries, this is the first major application of AIE in the Federal government. The creator of the AIE methodology claims that there are no intangibles such as “improved decision making,” which cannot be measured. One of the principles of AIE is that measurement is for improving future decisions not for justifying past decisions. The only exception being compulsory reporting to satisfy the need for responsible oversight. The AIE methodology determines what to measure by using a sophisticated cost-benefit analysis method that calculates the value of information for each variable using a formula familiar to decision analysts and statisticians for more than 50 years. The value of information depends upon two things, the certainty of an expert about a particular variable such as “the number of viruses per year” or “the cost of each investment,” and the value of the decision to be made. If the value of information for a particular variable is $200K for example, then an organization, as a general rule, should expect to spend no more than 20 percent to take a measurement to gain additional information to improve the decision to be made. A high information value indicates that addition measurements would improve decision making. There are two principal decisions that VA needs to make regarding the Information Security Program. One is which combination of its optional investments will reduce the 1
  5. 5. greatest losses at a reasonable cost. The second is what is the best rollout strategy for VA’s Public Key Infrastructure (PKI) investment that will optimize the value of PKI. The AIE method determined that VA would make the best investment decision by taking seven key measurements. Those measurements will also allow VA to determine the effectiveness of the Information Security Program over time. The AIE method also determined a prioritization approach that will allow the VA to implement PKI for the high-risk areas first and defer implementation for the low-risk areas. The AIE methodology also determined that: • the Information Security Program should reduce by 75% to 95% the expected losses for all security incidents through 2006 estimated somewhere between $1.1 billion and $2.4 billion. • one major optional investment (certain parts of Intrusion Detection) did not reduce losses and therefore should not be made. This is about a $30 million cost avoidance. Considering only the cost avoidance of the Intrusion Detection investment, this pilot had a value of $30 million. The cost of the pilot, including all contractor fees and travel expenses plus the time of the VA staff, was less than $100,000. Even excluding the value of other strategy improvements, the AIE methodology provided a 300:1 payback. The total cost of the pilot was less than 0.1 percent of the investment size of VA’s Information Security Program. 2
  6. 6. 1. Purpose of the Pilot The purpose of the pilot project was to compare two different methods for developing performance metrics for IT projects. Each of the methods was assigned a project within a Federal agency and observers from various other agencies commented on the process and the results. 1.1 Pilot Objectives The VA created a pilot team from its Information • Test applicability of methodology to measure Security Program (ISP), which employed the contribution of IT to mission results, AIE. The USDA's Food Acquisition Tracking • Provide a real government example and Entitlement System (FATES) applied the BSC. lessons learned The FATES team was a tri-agency partnership • Provide host agency with measures of Agriculture Marketing Service (AMS), Food and Nutrition Service (FNS) and Farm Service 1.2 Approach Agency) (FSA). In addition to the Core team members, the pilot team meetings were also open The Information Technology (IT) Capital to interested observers from the CIO Council. Planning Committee and the Sub-Committee on The observers participated minimally in the IT Performance Management of the Federal workshops, occasionally asking questions for Chief Information Officers Council sponsored clarity, but rarely providing input to the core two pilot programs to demonstrate different IT teams. measurement methodologies. This was done because many federal agencies have had difficulty in responding to the Clinger-Cohen Act of 1996 which requires Federal agencies to measure the contribution of IT investments to mission results. The objectives of these pilots were: 1. To test the applicability of two different methodologies to measure contribution of IT to mission results; 2. To provide examples of government organizations using the methodologies; 3. To present lessons that were learned to interested agencies; and 4. To provide the host agencies with IT measures The two methodologies chosen for this pilot project were Applied Information Economics (AIE) and the Balanced Scorecard (BSC). The host agencies in the pilots were the Department of Veterans Affairs (VA) and the Department of Agriculture (USDA). Both host agencies provided the core team and resources necessary to complete the pilot. 3
  7. 7. 2. Description of Host Agency Business Needs The Department of Veterans Affairs employs over 240,000 individuals to care for the needs of veterans - including medical, pensions and burial. Information systems security is necessary to support the goals of the VA both directly and indirectly. Security incidents affect the cost, quality and timeliness of virtually all areas of veterans care. 2.1 VA Mission Statement • Medical Education • Medical Research "To care for him who shall have borne • Compensation the battle, and for his widow and his orphan." • Pension –Abraham Lincoln • Vocational Rehabilitation and Counseling • Education The mission of the Department of Veterans • Housing Affairs is to Honor, Care and Compensate • Insurance Veterans in Recognition of Their Sacrifices for • Burial America. 2.4 Relevant Objectives VA’s responsibility is to serve America’s veterans and their families with dignity and The VA Strategic Plan specifies performance and compassion and be their principal advocate in cost management objectives for all ensuring that they receive medical care, benefits, administrations within the VA. Many of them social support, and lasting memorial promoting are adversely affected by security risks. the health, welfare and dignity of all veterans in recognition of their service to the United States. The VA has specific output-cost-reduction objectives such as cost per claim completed, cost 2.2 VA Size per loan, etc. The costs of security risks adversely affects all of these objectives since VA employs over 240,000 individuals – over 13 security incidents affect productivity in all lines percent of the total federal workforce. Almost 98 of business. percent of the staff are assigned to provide direct services to veterans and their families in VA field The productivity losses due to security incidents operations. may also affect any of the numerous output- related objectives. Many of the VA objectives The delivery of veteran services is accomplished call for increased output and all of them are at through 172 medical centers, 527 ambulatory and risk of security incidents. community-based outpatient clinics, 206 veterans centers, 57 regional offices, more than 24 2.5 Summary discharge centers, additional out-based locations, and 119 national cemeteries. VA exists to give meaning, purpose, and reality to America’s commitment to her veterans. The VA provides services and benefits through requirements, preferences, and expectations of facilities in all 50 states, the District of Columbia, veterans directly shape the services VA provides. Puerto Rico, the Virgin Islands, and the Philippines. 2.3 VA Business Lines VA provides services and benefits through the following business lines: • Medical Care 4
  8. 8. 3. Description of Agency IT Initiative The Information Security Program (ISP) is a five-year investment in the VA's security infrastructure. The total cost over five years will be approximately $114 million. It will address viruses, intrusion, fraud and other security risks through new systems, procedures and training. 3.1 Information Security Program and benefits payments, is vulnerable to (ISP) Overview inadvertent or deliberate misuse, fraudulent use, improper disclosure or destruction. Information security has been a burgeoning discipline in Federal IT circles for years, but Vulnerabilities as a result of inadequate controls recent highly visible security breeches at other and oversight include unauthorized access to VA Federal agencies have made information security systems, lack of systems monitoring, inadequate a matter of paramount urgency and importance. physical security, inadequate segregation of VA’s current information security program is duties, and no controls over changes to operating under close scrutiny by the U.S. General systems, and incomplete disaster Accounting Office (GAO) and VA’s own recovery/contingency planning development and Inspector General (IG), with both organizations testing. mandating that VA drastically improve information security throughout the Department. 3.3 Mission of the ISP Furthermore, VA’s information security program has been reported as a Departmental material The VA Information Security Program mission is weakness under the Federal Manager’s Financial to ensure the confidentiality, integrity, and Integrity Act. availability of VA information assets. The VA ISP covers all VA information assets, including For these reasons, the Acting Assistant Secretary hardware, software and data. It applies to for Information and Technology, the VA Chief Departmental information resources located at Information Officer (CIO), approved a protocol VA facilities and those maintained at non-VA for a Department-wide information security facilities. It encompasses all measures, including program in February 1999. Subsequently, a team technical, administrative, personnel, and physical was formed to develop program requirements, controls, necessary to protect information assets responsibilities, and initiatives. This team against unauthorized (accidental or intentional) worked in a group with security managers from disclosure, modification, destruction, and denial each VA Administration and Staff Office to of services. quickly develop a more robust information security program for VA. Speed in development The VA ISP involves all VA employees, vendors, of this program was a necessity to address contractors, volunteers, veterans, service Departmental material weaknesses and to satisfy organizations, members of the general public, and the requirements of various statutes, OMB anyone else with access to, or who uses VA Circulars, and Presidential Decision Directives. information systems. It applies to data sharing agreements and similar understandings with other 3.2 Background: VA Security Risks government agencies, commercial business partners, and the public. The ISP will secure VA’s information assets from known threats and vulnerabilities. Without The VA ISP supports VA’s mission by protecting risk management, these threats and vulnerabilities VA’s information assets. In addition, the VA ISP can have serious consequences for VA as an proactively implements statutory and regulatory organization, and for individual veterans who requirements and industry best practices, and entrust VA with their most private data. Sensitive adapts to technology infrastructure dynamics. information, e.g., financial transaction data, The information assets required to execute personal information in veteran's medical records departmental mission programs for health care, benefits delivery, national cemeteries, and 5
  9. 9. associated administrative support functions are C. Simplified Sign-On. Sign-on will simplify themselves mission-critical. and improve the sign on event that regularly confronts employees, contractors, and other 3.4 Investment Area Definitions representatives granted access to VA’s internal networks, systems, and applications. The ISP has seven investment areas. Each of the It will improve workforce productivity and investment areas has components that are strengthen access controls. considered minimum necessities. Some of the investments also have optional components that D. VA Public Key Infrastructure (VAPKI). may or may not be added depending on the A public key infrastructure is a combination findings of this study and the subsequent metrics of hardware, software, policies, and implemented. See Appendix 2 for detailed administrative procedures that provide a descriptions of minimum vs. optional framework to use public key cryptography to components. transfer data in a secure and confidential manner. Currently, PKI is the only A. IT Systems Certification and identification and encryption solution that Accreditation Program (ITSCAP). provides all four components of a secure Certification is a technical evaluation of an electronic transaction: strong authentication; information technology system to see how data integrity; confidentiality; and, non- well security requirements are met. repudiation. PKI is an economic and simple Accreditation is the official management way of implementing these security services authorization to process. This initiative also in a system or application. includes the development of a data sensitivity model. This initiative uses E. VA Computer Incident Response contractor support to meet its objectives. Capability (VA-CIRC). VA has The mission of ITSCAP is to assure that the established and maintains a computer security controls of each individual system security incident reporting and response or application yield adequate security; where capability to ensure that computer security adequate security is an approximate balance incidents are detected, reported, and of the cost of controls and the value of the corrected as quickly as possible, and with information assets in the system or minimal impact. Incident reporting and application. ITSCAP gives the public faith response is designed to: detect and respond that VA achieves a standard of due diligence to computer security incidents as they occur, for the security of the system or application assist in preventing future incidents from comparable to other private or government occurring through awareness, contain entities doing the same kind or kinds of necessary response mechanisms to deal with business. incidents, and, support security controls and procedures. B. Intrusion Detection. Intrusion detection identifies intruders breaking into information F. Antivirus. Antivirus will protect VA’s systems or legitimate users misusing system networks, systems, and applications from resources. This initiative also includes virus attacks. This will limit the costs related developing secure gateway configurations. to loss of business-critical information, This assures that VA’s networks, systems, workforce productivity, or interruption in the and applications are adequately monitored services provided to VA beneficiaries. for threats from persistent adversaries both internal and external. Intrusion detection G. Training/Education/Awareness/Message gives the public faith that VA achieves a Building (TEAM). TEAM includes media standard of due diligence for the security of development, conference planning, satellite the network or system or application broadcast development, brochures, posters, comparable to other private or government announcements, and the ISP public presence. entities doing the same kind or kinds of TEAM will help ensure that the VA business. workforce is empowered to make their individual contributions to information security excellence. 6
  10. 10. 7
  11. 11. 4. VA Core Team The core team consists of seven persons from the VA IT staff and one outside consultant. Participants were chosen by their knowledge of various components of the ISP. 4.1 Team Selections Department of Veterans Affairs since 1976. His VA The core team members were selected based on career began as a Medical Technologist in their knowledge of their respective ISP initiatives Laboratory Service of a Medical Center. Later, he and their availability for data-gathering served as Clinical Microbiologist and Laboratory requirements of the AIE process. ADP Application Coordinator. In 1989, Michael became part of the Veterans Health Administration’s The criterion for choosing the project coordinator (VHA) Information Security Program (Medical was that person be available in the future for Information Security Service) serving in sequence in additional AIE analysis. the positions of Program Analyst, Security Specialist (RISO), and Special Assistant to the Director in the The consultant is from Hubbard Decision VHA security program. In February of 2000, he Research, which specializes in the use of Applied joined the Technology Integration Service’s Information Economics. information security program as Computer Specialist, leveraging his experience into the national, VA-wide, arena. Michael was selected for 4.2 Team Members the Core Team based on his expertise as team leader of TEAM. Ruth Anderson Fred Blumenthal Ruth Anderson is a project manager with the Department of Veterans Affairs Information Fred Blumenthal joined the VA CIO Information Security Program. In this role, she develops and Security Program staff in July of 2000 to oversee coordinates VA national policy on security issues and administer activities of the VA Computer such as employee personal use of the Internet, Incident Response Capability (VA-CIRC) Program. account and password management, external The VA-CIRC is the mechanism used VA-wide to electronic connections, and public key detect, report, and correct computer security infrastructure. Ruth was selected for the Core incidents before they adversely affect the Team because she is the project manager for Department’s critical information resources and VA’s national computer incident reporting and infrastructure. With this responsibility, Fred was response capability (CIRC), and she manages the selected as a member of the Core Team. Prior to Department’s public key infrastructure (PKI). joining the VA CIO Information Security Program Ruth is a member of the Federal PKI Steering staff, Fred spent four years administering the VA Committee, the FedCIRC Partners Group, and Procurement of Computer Hardware and Software GSA’s Access Certificates for Electronic (PCHS) Program. The PCHS Program consists of Services (ACES) Customer Advisory Board. two large-scale, Department wide Indefinite Ruth has been with VA since 1987 and her entire Delivery/Indefinite Quantity (IDIQ) contracts for career with VA has been spent in the field of personal computer hardware and software worth up information technology. Ruth holds a Master of to $1.5 billion. Fred received his Bachelors degree Arts degree from the University of Maryland, and from the University of Maryland. a Bachelor of Arts degree from Frostburg State College. Fred Catoe Michael Arant Fred Catoe is currently a project manager with the Department of Veterans Affairs (VA) Information Michael S. Arant is a senior member of the Security Program. Prior to joining VA, his national Information Security Program Office in experience was in systems analysis and design with VA Central Office. He has been with the the Naval Air Development Center, Boeing 8
  12. 12. Helicopters and General Electric. Since joining Peter Flynn is the team leader for the AIE Pilot VA, Fred has conducted technology assessments Demonstration. As such, he coordinates and and was instrumental in the development of the schedules the activities of the Core Team. Peter national Master Veteran Record project. Fred’s began his 25 years of service as a veteran's benefits successes have continued with his appointment as counselor at a VA Regional Office. At VA Central team leader for the development of anti-virus Office (headquarters) his experience has varied from software. This is reflected in Fred’s selection to human resources management to policy the Core Team. Fred holds Master of development. More recently, Peter has been Engineering degrees in Management Information instrumental in implementing the VA Information Systems and Industrial Engineering from Penn Technology Review Program. Peter holds Master of State University and a Bachelor of Science Science in Administration and Bachelor of Science degree in Mathematics from the University of in Business Administration degrees. Maryland. Doug Hubbard Jim Edwards Doug Hubbard is the inventor of Applied Jim Edwards has 32 years experience in Information Economics (AIE). His methodology has information technology management at the earned him critical praise from The Gartner Group, Department of Veterans Affairs. He began work Giga Information Group and Forrester Research. He as a computer programmer, later graduating to is an internationally recognized expert in the field of roles in both business systems analysis/design IT value and is a popular speaker at numerous and operating systems support. In recent years he conferences. His published articles are in managed the rollout of a headquarters-wide Information Week, CIO Enterprise, and DBMS document imaging/workflow solution, led the Magazine. He was formerly with Coopers & competitive acquisition of a $1.5 billion contract Lybrand and has over 12 years experience in IT program for personal computer products, and management consulting including 5 years experience managed implementation of a messaging solution specifically in teaching organizations to use his AIE for interchange of veteran demographics between method. His other professional experience includes vertical business lines. Since January 1999, Jim managing large IT strategy and software projects in has been the Department's senior information insurance, manufacturing, nuclear power, banking security officer, responsible for management of a and pharmaceuticals. He has an MBA in security capital fund, issuance of Department- Management Information Systems from the wide policies and controls, administration of University of South Dakota. awareness and training programs, and overall security program evaluation. 4.3 Roles Defined Jeff Shearer Members of the Core Team provided the detailed background information on the ISP that enabled the Jeff Shearer is directing VA’s Information selection of the specific investment areas. Through Technology Systems Certification and the Team’s participation in the developmental series Accreditation Program, Intrusion Detection of workshops, the Consultant processed the Capability, and Simplified Sign-on Capability. information provided through each member’s unique Jeff comes to VA’s Information Security expertise and devised the initial estimates for each Program after 13 years as an auditor with VA’s investment area, the value of information and the Office of Inspector General. He is a Certified initial measurement priorities. Later, through a Information Systems Auditor, Unix Systems lengthy series of one-on-one sessions, the Consultant Certified Professional, and Certified Government worked with the Team members to fine-tune the Financial Manager. With this expertise, Jeff was intricately detailed volumes of information on each selected as a member of the Core Team. Jeff investment area for the application of the AIE graduated Cum Laude from Central Missouri methodology. As the following sections define the State University with a Bachelor of Science in AIE process and its application to the ISP the Team Business Administration degree. participated in all facets with the Consultant, leading to the final results. Peter Flynn 9
  13. 13. 5. Overview of AIE Methodology Applied Information Economics (AIE) is the practical application of scientific and mathematical methods to the IT and business decision process. 5.1 Philosophy problems, including the following: Several characteristics distinguish Applied Using mathematical models to improve Information Economics (AIE) from alternative cost/benefit analysis (CBA) for better decision-making methods. decisions at all levels of IT; Developing financially-based quality • Everything is measurable assurance measurements to insure that the • Risk can be measured in an actuarially sound implementation of IT decisions are effective; way and • The value of information can be computed in Developing a strategic plan for information an economically sound way systems based on identifying the best AIE is a unique methodology to rigorously apply opportunities for economic contribution by a specialized economic theory to the problems information systems confronting the executive in charge of the “IT portfolio.” Understanding the AIE methodology requires a significant if not profound change in one’s AIE is a synthesis of techniques from a variety of thinking. Principles and methods familiar to scientific and mathematical fields. The tools of those in the scientific and mathematical fields economics, financial theory, and statistics are all that are used in AIE are often foreign to those in major contributors to AIE. But in addition to the information technology field. Consequently, these more familiar fields, AIE includes Decision many people experience the paradigm shifts Theory - the formulation of decisions into a listed the box below when first encountering AIE. Economic Operations Paradigm shifts in AIE Research Applied Decision/ Game Information Theory Portfolio Economics • Everything is measurable Theory Actuarial Science • The purpose of measurements is to Software provide information to make better Metrics Options Theory future decisions not to justify past decisions mathematical framework - and Information Theory - the mathematical modeling of transmitting and receiving information. It is • Using range estimates for costs and important to emphasize, however, that even benefits to estimate the value of IT is though AIE is a theoretically well-founded set of better than using averages or best techniques, it is a very practical approach. Every guesses as estimates proper application of AIE keeps the bottom line squarely in mind. All output from the AIE • The value of information needed to make project is in support of specific practical business decisions can be computed objectives. The powerful techniques of AIE clarify, measure, • Uncertainty and risk can be quantified and provide optimal recommendations for a variety of situations. AIE applies across the • Scientific methods of measurement are enterprise to solve some of its most perplexing practical for IT investment 10
  14. 14. 5.2 Key Methods of AIE probabilities to them. It is almost never the case that we will need exact numbers before we can Some of the basic techniques that make AIE a make an economically rational decision. The powerful set of tools are “unit of measure” ranges of values assigned to variables in a definitions, calculation methods for the value of decision model can be used to determine a information, methods for modeling uncertainty in “probability distribution” of the net benefit of a estimates, and treating the IT investment as a particular IT investment. AIE uses the “Monte type of investment portfolio. These methods are Carlo” method - the generating of thousands of used also by financial services firms to create random scenarios on a computer (also used in financial products and they are used also by the statistics, actuarial science and game theory) - to insurance companies to calculate premiums. develop a graph of the likelihood of each possible net benefit. Since part of this graph will usually 5.2.1 "Unit Of Measure" Definitions show that there is some chance of losing the investment or not making the desired return, the Most IT investment arguments include some risk of the investment can be quantified and costs or benefits, which are treated as assessed against its expected return. “intangibles” or factors that cannot be measured. Some common examples include “Strategic 5.2.3 The Calculation Of The Economic Alignment,” “Customer Satisfaction” or Value Of Information “Employee Empowerment.” In most of these cases, the factors only seem to be immeasurable Contrary to popular belief, the value of because they are ambiguously defined. AIE information can be calculated as a dollar value. removes this type of ambiguity by focusing on Although the term “information” is often used in definitions that can be express in units of an ambiguous manner, an unambiguous unit of measure. measure has been defined which can be used in an economic value calculation. This For example, an argument for a new Intrusion mathematical procedure can be paraphrased as Detection System may claim that, among other follows: things, it reduces “data exposure”. Does this mean that legal exposure from unauthorized 1. Information Reduces Uncertainty distribution of personal data is reduced? If so, 2. Less Uncertainty Improves Decisions how frequently do situations arise that result in 3. Better Decisions Result In More Effective legal costs and what is the cost per incident? Actions Does reduced “data exposure” mean that the cost 4. Effective Actions Improve Profit or Mission of fixing corrupted data is reduced? Does it mean Results that there will be less fraud resulting in monetary losses? Does this mean all of the above? These four steps can be stated in unambiguous mathematical terms. The mathematical model for 5.2.2 Analyzing Uncertainty Systematically this has been around since the late 1940's. From this the “elusive” value of information can be All investments have a measurable amount of determined precisely. If you were going to make uncertainty or risk. In fact, rational investment a decision about implementing a new information decisions must always take both the risk and system, you would find that you are uncertain of return of a given project into account. The ability the cost and duration of the investment as well as to quantify the risk of a given IT investment, and the various benefits. If you had less uncertainty compare its risk/return with other non-IT about these quantities then you would be more investments, is one of the many things that set likely to make the right decision about whether to AIE apart. proceed with the investment. AIE quantifies uncertainties with ranges of A decision to proceed with a major IT investment values and probabilities. In reality, there is is uncertain because of uncertain costs, benefits, uncertainty about any number that we would learning curves, etc. The wrong decision will apply to just about any cost/benefit variable. result in lost opportunities if a good investment is Instead of choosing a single estimate such as an rejected or misallocated resources if a bad average, AIE focuses on determining the range of investment is accepted. If the decision-maker possible values for a given variable and ascribing 11
  15. 15. had more information (i.e., less uncertainty) about ongoing maintenance costs, for example, she would have a higher chance of making the right decision. Reducing uncertainty on more variables would bring an even higher chance that the right decision will be made. The wrong decision will cost money and the right decision will make (or save) money. The formula for this simply computes the likely economic advantage from having less uncertainty. 5.2.4 IT Investments As An Investment Portfolio AIE uses the methods of Modern Portfolio Theory (MPT) and treats the set of an organization’s IT investments as another type of investment portfolio. By using techniques from MPT, we can determine whether the uncertainties inherent in a given IT investment decision are acceptable given the risk/return position for the firm. MPT also isolates or identifies the contribution or impact of multiple investments separately and together. This allows AIE to find the optimum combination of investments. 12
  16. 16. 6. Steps of the AIE Performance Metrics Approach The AIE performance metrics approach consists of 4 phases. These phases focused on identifying metrics that have a high economic value compared to their costs. 6.1 Overview VA Core Team. These estimates were not typical point values but "probability The four phases of the AIE performance metrics distributions" that represent the uncertainty of approach were: the estimator. Phase 1: Compute measurement economics The process behind the initial estimates was Phase 2: Design measurement methods based on the fact that assessing uncertainty is a Phase 3: Implement measurement methods general skill that can be measured and refined. Phase 4: Collect and analyze results In other words, experts can measure whether they are systematically “underconfident”, 6.2 Phase 1: Compute Measurement “overconfident” or have other biases about their Economics estimates. Once this self assessment has been conducted they can learn several techniques for The objective of Phase 1 was to compute the achieving a measurable improvement in economic value of potential measurement estimating. alternatives. AIE makes measurement decisions based on the economic value of the information This initial “calibration” process was critical to from the proposed measurements. The major the accuracy of the estimates later received about tasks of Phase 1 included the following: the project. The methods used during this 1. Identify decisions “calibration” have been designed in the recent 2. Model decisions past by such well known academics as 3. Compute information values Dr. Shoemaker from the University of Chicago. 6.2.1 Identify Decisions Few individuals tend to be naturally good estimators. Most of us tend to either be biased The major decisions that still needed to be made toward over or under confidence about our regarding the ISP had to be identified so that estimates. (see Definitions box) metrics could be identified that specifically supported them. Definitions The specific types of decisions to be made would Overconfidence: The individual routinely puts obviously affect the decision model. Is the too small of an “uncertainty” on estimated decision about whether or not some investment quantities and they are wrong much more often should be made? Is the decision about choosing then they think. For example, when asked to an optimal "portfolio" of combinations of make estimates with a 90% confidence interval investments? Is the decision about choosing the much fewer than 90% of the true answers fall best of several implementation plans? The within the estimated ranges. decision could be any of these or others. Underconfidence: The individual routinely puts 6.2.2 Model Decisions too large of an “uncertainty” on estimated quantities and they are correct much more often The decision model was a spreadsheet model then they think. For example, when asked to that included all the relevant decision variables. make estimates with a 90% confidence interval The objective was to take a large complicated much more than 90% of the true answers fall decision with lots of variables and represent it in within the estimated ranges. an ordered, structured fashion that is as simple as possible to communicate. Once the spreadsheet was developed a set of initial estimates was Academic studies by Dr. Shoemaker and others provided based mostly on the knowledge of the proved that better estimates are attainable when 13
  17. 17. estimators have been trained to remove their 6.3 Phase 2: Design Measurement personal estimating biases. The contractor Methods conducted a workshop for the Core Team where the participants recorded a low and high bound The objective of Phase 2 was to determine that represented a 90% confidence interval of measurement methods that have the highest their personal knowledge abouta given set of information value compared to their costs. general knowledge questions. This involved the following major tasks: Since the original estimates were made with a • Identify alternative measurement methods 90% confidence, an average of 1 in 10 should be • Estimate costs of methods incorrect. By reviewing the participants’ • Choose methods based on cost vs. answers to these questions we can derive and information value illustrate their over or under confidence. By performing this process of answer and review several times, participants become “calibrated” 6.4 Phase 3: Implement Measurement to the level of their personal confidence that Methods corresponds to a 90% level of statistical confidence. Phase 3 implemented the measurement methods identified in Phase 2. The VA Core Team Calibrated Probability Assessments conducted surveys and gathered data from research, passive measurements and the other When asked to provide a subjective 90% measurement methods previously identified. confidence interval, most managers provide a range that only has about a 40%-50% This included the implementation of any chance of being right organizational procedures required for continuous and persistent execution of the Perceived 90% Confidence Interval measurement. The objective was not to create "one-time" measurements but on-going measurements that will be part of the culture and the continued decision-making process. 6.5 Phase 4: Collect & Present Results Phase 4 captured data and reported results Actual 90% Confidence Interval gathered from data in Phase 3. This was not the "end" of the measurement process since the measurement process is on-going. It is merely a The initial set of estimates (all ranges) snapshot of the measurements gathered so far represented the current level of uncertainty of and represents the nature and effectiveness of the the team about the quantities involved. This measurements implemented. provided the basis for the next step - computing information values. 6.2.3 Compute Information Values Once the initial ranges were determined we asked if there was any value to reducing uncertainty and, if so, where. All measurements that have a value result in the reduction of uncertainty of some quantity that affects some decision. The variables vary by how uncertain they are and by how much they impact the final decision. 14
  18. 18. 7. Estimated vs. Actual Timeline Overall, the project was on schedule. The completion of Phase 2 was about one month behind expectations. Phase 3 was also pushed back, but Phase 4 progressed appropriately. This report was presented on May 8, 2001. 7.1 Projected Timeline 7.3 Reasons for Delay Start – November 15, 2000 The Pilot started appropriately and progressed at Phase 1 – Compute Measurement Economics: the expected pace early on. As the Team got Identify decisions and reporting requirements; deeper into the Pilot it became evident that the Model decisions and quantify reporting trade- ISP was much more complicated than originally offs; and, Compute optimal information estimated. Oringinally, the ISP consisted of quantities – Finish before year-end. eight investment areas. The VA completed a risk assessment prior to the pilot. For Pilot Phase 2 – Design Measurement Methods: purposes, the ISP included seven investment Identify alternative measurement methods; areas which required seven areas of analysis as Estimate costs of methods; and, Choose methods opposed to the initial estimated of one. based on cost vs. information value – January 2001. Constraints on resources forced reduced availability of the AIE staff. Additionally the Phase 3 – Implement Measurement Procedures – Team had responsibilities that reduced their January 2001. availability. The Pilot worked around these problems to reach the best solutions. As a result, Phase 4 – Collect and Present Results: Deliver the timeline was stretched to accommodate all Report - January 16, 2001 considerations. 7.2 Actual Timeline 7.4 Work Hours Invested Start – Kick Off Meeting – November 28, 2000 The total time that the VA Core Team and the contractor dedicated to this pilot were : Phase 1 – Compute Measurement Economics. The workshops started on November 29, 2000 VA Core Team – 300 hours and ended on December 14, 2000. AIE Contractor – 350 hours Phase 2 – Design Measurement Methods. Started with the Mid Point Review on January 4, 2001. Included a regular series of individual meetings and/or conferences (in person or telephone). Completed in mid February. Phase 3 – Implement Measurement Procedures. Information gathering and analysis. Completed in April. Phase 4 – Collect and Present Results. Report was delivered on May 8, 2001 15
  19. 19. 8. Results The productivity impact of viruses, frequency of intrusions, losses due to fraud and the cost of certain ISP investments were determined to be the critical measurements. 8.1 Phase 1 Results Incident types • Viruses 8.1.1 Identify Decisions • Unauthorized internal logical access • Unauthorized external logical access The team felt that all the ISP initatives (see • Unauthorized physical access section 3) had certain necessary components that • Environmental events had to occur. These were called the minimum Viruses -- Software designed and distributed for investments since there is no decision problem in the purpose of causing damage to information regards to them and they simply must be systems assets, and that replicates and distributes implemented. However there are other automatically. components of each of the investment areas where the value is not certain. These are called Unauthorized Internal Logical Access -- Access the optional investments and the decisions to to VA information systems by unauthorized proceed with them depend on the results of future individuals, originating within VA's network measurements. perimeter. The ISP investment decision: Unauthorized External Logical Access -- Access to VA information systems by unauthorized What is the best combination of optional individuals, working outside VA's network investments from each of the ISP initiatives? perimeter, that bypass authentication mechanisms, exploit vulnerabilities in system services, eavesdrop, or monitor network activity. 8.1.2 The Decision Model Unauthorized Physical Access -- Access into a VA facility by unauthorized individuals that in The team modeled the benefits of the ISP and turn causes denial of computer services, each of its optional investments. First, a model corruption of data, or compromise of of the total security losses was made so that we confidentiality. could see what security losses might look like without the ISP. This was called the "Loss Environmental Events -- Events that are caused Model" and it was a baseline for all the ISP by circumstances out of the control of human investment initiatives. The Loss Model is a forces, such as flood and fire, which result in spreadsheet that estimates the cost of five basic denial of service to information security assets. types of security incidents: Major Components of the ISP Model Investments: Incidents: Losses: …reduce… …result in… • ITSCAP • Virus • Fraud • VACIRC • Unauthorized Access • Productivity • Simplified Sign-on - logical internal • Interference • VA PKI - logical external w/mission • Intrusion Detection - physical • Legal Liability • Antivirus • Environmental Events • TEAM 16
  20. 20. The frequency and severity of each of the For each of the 104 variables, the estimators security incidents was estimated. The severity of provided ranges that best reflected their current an incident is the total cost of one incident. The level of uncertainty for each of the quantities. total costs included the following loss types: For example, the average duration of the period Loss types of productivity loss due to a virus attack is not • Fraud known exactly. But the estimators felt confident • Productivity that the average duration must be between 4 and • Interference w/mission 12 hours. • Legal Liability Furthermore, they were willing to say that they The value of the ISP investments were the were 90% certain the actual value falls between reduction in security incidents resulting in fewer this upper and lower bound. Finally, they were losses. As expected, most of the initial quantities willing to say that the distribution of possible came from Calibrated Probability Assessments. results was a normal distribution. They could The calibration training showed that most of the also have chosen other types of distributions estimators were able to adequately represent their (lognormal, uniform, beta and binary). uncertainties with probability distributions. Most of their ranges were conservatively wide. The Each of the possible distribution types says other source of data was Standard Metrics. See something about the probability of various appendix 5 for the detailed spreadsheet model. outcomes. In the case of the duration of the productivity impact of a virus attack, the Initial Measurement Source Summary estimators choice of a normal distribution says that there is a small chance of the value being (see App. 5 for detailed spreadsheet) outside their range (10%) and that the probability Source of Measurement Number of was symetrical (the true value is just as likely to variables be above 8 hours as below). The code for a Calibrated Probability Assessments 104 normal distribution type is a 1. See the excerpt – probability distributions gathered from the spreadsheet below to see how this data from calibrated estimators was recorded for the model. Financial Standard (Cost of 1 Capital) Total 105 Excerpt from Spreadsheet : The Productivity Impact of a Virus Attack (From Rows 21-26 in Appendix 5) Variable Name Lower Formulas & Upper Distribution Bound Best Estimate Bound Type Productivity Average number of people affected 25,000 45,000 65,000 1 Percentage productivity loss 15% 38% 60% 3 Average duration of productivity loss 4 8 12 1 Cost per person $ 50,000 $75,000 $ 100,000 1 Total productivity loss per incident $ 4,867,788 Indicates the type (shape) of the probability distribution chosen for this variable. A (1) indicates a Normal distribution. These ranges were used to generate thousands of possible scenarios so that the probability of various investment results can be assessed. 17
  21. 21. 8.1.3 Preliminary Findings Since the pie chart was generated from a Monte Carlo simulation, it only shows an average of all The ranges were used to generate 50,000 random potential outcomes. It does not show the exact scenarios – each of which is a potential outcome distribution of what losses will be experienced. of the ISP investment. This is called a Monte We did determine, however, that the losses due to Carlo simulation. An average of the security unauthorized internal access are 99% probable to losses from all these scenarios was produced to be the largest loss category. assess the likely costs of various potential security losses. This analysis produced the The same Monte Carlo simulation was used to following distribution of security losses in the assess the expected impact of the entire ISP – VA: including both the minimum required investments as well as the optional investments. Relative Costs of Simulated Security Losses The Monte Carlo simulation clearly showed that security related losses are significant and that the ISP will significantly reduce them. The graph Other (12%) below shows the expected reduction in security incident losses over the next six years. Viruses (17%) The anticipated total costs due to security related This cost is mostly incidents over six years are very uncertain but related to productivity there is a 90% probability that the losses will be somewhere between $1.1 billion and $2.4 billion. Unauthorized Internal Logical Access (71%) The ISP is expected to reduce this by 75% to This cost is mostly 95%. This will most likely result in cost related to fraud avoidance of several hundred million dollars over the next six years. This easily justifies the expense of the ISP. Each of the security-related events was expected to result in different types of losses. The initial The decision, therefore, is not about whether to calibrated estimates indicated that the team proceed with the ISP. It is only about which of believed most of the cost of unauthorized internal the various optional investments should proceed. access was due to fraud losses while events like The proposed ISP investments will reduce the viruses resulted in productivity losses. But losses severity and frequency of incidents that cause due to fraud were estimated to be over $100 security-related losses. The contractor used this million per year – much more than expected information to assess the viability of each of the productivity losses from virus attacks. Therefore, optional investments of the ISP and determined unauthorized access is a much greater risk than which should be pursued and how those virus attacks. investments should be measured. The other category in the above pie chart includes environmental events, unauthorized external access and unauthorized physical access. But, clearly, for the most impact the ISP should (and does) focus on mitigating the security risks of viruses and unauthorized internal logical access. 18
  22. 22. The Potential Losses Due to Security Incidents and the Effect of the ISP The ISP will avoid 75% to 95% in the cost of security incidents. Over six years this may be on the order of $1 billion. Bottom Line: the ISP is easily justified – the question is how to implement it for best results. 90% Confidence Intervals for the Total Losses over 6 years with and without the ISP Estimated Losses Estimated Losses w/o w/ISP: ISP: $0 $1 $2 $3 Statistical Breakeven Analysis for The ellipses represent the optional components of All ISP Optional Investments each of the following investment areas (ITSCAP and Anti-virus do not have optional components) This chart is a "solution space" that compares the Simplified Sign-on cost (over 6 years) of an optional investment to its reduction of security losses. An investment VA CIRC must be above the breakeven line to be justified. The shape of the ellipse represents uncertainty VAPKI (90% confidence intervals) about the cost and reduction in security losses. TEAM Intrusion Detection This chart shows that, even though costs and 10% security impact is somewhat uncertain, the Total reduction in Security Losses 9% optional investment for Intrusion Detection is 90% confidence 8% clearly not justified and should not be pursued. range for 7% Likewise, VA CIRC and Simplified Sign-on are breakeven easily justified. VAPKI and TEAM are the only 6% (above the line investments that are uncertain in terms of their 5% is a good ability to breakeven in 6 years. investment, 4% 3% 2% below is not) } This is consistent with the value of information calculations (see the following section) which indicate that metrics should focus on what is 1% required to make the best decisions for those 0% uncertain components of the ISP. $0 $10 $20 $30 $40 $50 Investment size over 6 years ($ millions) 19
  23. 23. 8.1.4 Compute Information Values For example, even though fraud costs had the highest EVPI the team felt that it was unlikely Initial measurements were based entirely on that an extensive measurement effort in this area calibrated probability assessments by the Core would produce useful results. It was decided that Team. The following areas areas required any additional reduction in uncertainty would additional measurements: most likely come from a few additional interviews and phone calls with auditors and • Fraud others that may track fraud losses. Therefore, the • Optional investment costs measurement effort was small although the EVPI • Frequency of logical intrusions was high. • Cost of viruses and reduction of viruses due to investments 8.2 Phase 2 Results The information value calculations clearly The Value of Information Analysis (VIA) indicated that these four types of quantities easily indicated that further measurements were needed justify additional measurement effort as can be to reduce uncertainty on fraud losses, cost of seen in the Summary Results of Initial Value of logical intrusions, and productivity impact of Information Analysis table below. viruses and the implementation costs of VAPKI. See Summary Results table for more information. The information value of each of these was computed with the "EVPI" method. In general, For fraud, the team engaged in additional the Expected Value of Perfect Information research of any historically documented (EVPI) represents the maximum value of experiences including internal VA data as well as additional information even if that information is outside studies on cyberfraud. perfect. This gives a good idea of the maximum cost that should be spent for an additional The team also initiated a security survey of VA measurement. department IT staff to better estimate productivity losses and possible fraud losses due to security EVPI is calculated by estimating the effect on a incidents. decision if there were no uncertainty in a particular variable. For example, if we knew Finally, the team further analyzed the cost of exactly what the productivity impact of a virus implementing VAPKI since this cost was fairly impact would be, there should be a higher chance uncertain. of making the right anti-virus investment decision. The difference between the expected Measurement Maxims outcome of an investment with this perfect information and the outcome without this • You have more data than you think: Be information is the EVPI for that variable. resourceful in finding existing information As a rule of thumb, 2% to 20% of the EVPI of • You need less data than you think: Be each variable should be spent in further resourceful in making useful conclusions measurements. The table below shows from limited information recommended ways for expending effort in more measurements. In addition to the EVPI, the cost • Measurement is usually easier than you and feasibility of additional information think: the first measurement method you gathering are considered when identifying think of is probably "the hard way", think of measurement priorities. the more resourceful, simple method 20
  24. 24. Summary Results of Initial Value of Information Analysis (VIA) Expected Justified Value of Effort (cost of Set of Variables Perfect the effort Measurement Approach Information should be 2- 20% of EVPI) Fraud, property loss, legal $787,763 Several work- The team felt that uncertainty would be difficult to liabilities weeks reduce significantly regardless of effort expended. We then decided to at least attempt to make calls within the audit function of the VA and determine what information is available. Optional investment costs $286,162 A few work- This is limited to the costs of the VAPKI, TEAM weeks and Simpilfied Sign-on. The team members will proceed with more detailed design in those areas to make better cost estimates. Logical intrusions $241,790 A few work- This will be a passive measure from the minimum weeks investment of the Instrusion Detection initiative. This system will produce these results anyway but the results will now be analyzed specifically to support this metric. Total effect of viruses and $151,910 Two work- A post-incident survey will be implemented that reduction of viruses due weeks or less will focus on productivity losses from a virus to investments shortly after the virus occurs. Anti-virus software will report on the difference in virus occurances due to other initatives. All other variables Under None $1,000 The estimators simply considered the information 8.3 Phase 3 Results and provided a new (subjective) calibrated estimate. From calibration training, the The research of historical data help to estimators learned they could consider qualitative significantly modify the range for annual fraud data or incomplete quantitative data and modify losses. The internal data was not as much help as their subjectively evaluated uncertainty. In this external studies that went into much more detail. case, the information from the report caused the estimators to modify the range for annual fraud One external study in particular had a significant losses due to internal unauthorized access (see effect on the opinions of the estimators. A report Summary of Phase 3 Measurement Results table). from the office of Senator Fred Thompson "Government Waste, Fraud & Abuse at Federal The security survey got 11 responses out of about Agencies" claims to have found $220 Billion in 50 surveys distributed. The survey form federal government funds lost (see inset). The distributed is shown in Appendix 1. This is a estimators realize that not all of this is due to small sample size but it still managed to reduce unauthorized computer access but it still caused uncertainty since some of the initial ranges were them to increase their estimate of fraud losses already very wide. The survey focused on the due to this reason. But they also realize that the productivity impact of virus attacks but it also VA represents a large percentage of the federal asked questions about fraud losses. Although the government budget and a similar proportion of results were somewhat indefinite for some these losses may be due to the VA. variables, the ranges of other variables were modified as a result of the survey. No advanced statistical method was required to interpret the impact of the Thompson Report. 21
  25. 25. Excerpts from the Sen. Thompson Report ( Summary of Phase 3 Measurement Results WASHINGTON, DC - Senate Governmental Initial Adjusted Affairs Committee Chairman Fred Thompson (R- Range Range TN) today released an alarming compilation of Annual Fraud $10M to $80M to government waste detailing $220 billion in losses due to $100M $180M taxpayer losses. In 1998 alone, $35 billion in internal taxpayer dollars was lost due to government unauthorized waste, fraud, abuse and mismanagement. access Number of 1 to 4 2 to 4 It’s difficult to track exactly how much the pandemic virus federal government loses to waste, fraud, abuse attacks per year and mismanagement primarily because federal Average number 30k to 80k 25k to 60k agencies are not required to keep this information of people and most don’t. With input from the agencies’ affected by a Inspectors General, the Committee was able to virus uncover $35 billion in wasted 1998 taxpayer Percentage 12% to 80% 15% to 60% dollars and $220 billion overall. productivity loss due to virus The Committee’s compilation of government outbreak waste includes: • Defense Dept. maintained $11 billion in Percentage of 2% to 8% 2% to 15% inventory it didn’t need; veterans • Energy Dept. invested $10 billion in projects affected that they never completed; • Education Dept. paid out $3.3 billion in VAPKI initial $2.5M to $1.3M to student loans that students failed to repay; costs of VA- $4.5M $2M • Agriculture Dept. sent out $1.4 billion in wide roll-out food stamps to ineligible recipients. VAPKI annual $1.2M to $1.1M to costs of VA- $1.6M $1.3M wide roll-out Finally, the more detailed cost estimate for VAPKI resulted in a significant modification to ranges for VAPKI initial and ongoing costs. 8.4 Phase 4 Results Each of these new ranges replaced the wider 8.4.1 Key Strategies existing ranges in the model. This is how observations are used to improve an existing The analysis of the Phase 3 measurements model. Every observation results in less clearly pointed to the following strategies for the uncertainty, i.e. more narrow ranges, about each performance metrics of the ISP: of the quantities observed. The narrower ranges result in a model with less uncertainty. • Put a high priority on implementing Anti- virus and the minimum component of See the following table (Summary of Phase 3 Intrusion Detection Measurement Results) for a summary of • Do not proceed with the defined “optional” modifications made to the ranges of specific component of Intrusion Detection (a savings variables. of at least $30M, a small % of which should be reallocated to pure metrics efforts) • Roll-out VAPKI on a security-priority basis based on passive feedback (see VAPKI roll- out criterion) 22
  26. 26. • Defer investment in optional TEAM not can be compared. This is the basis for investment until passive feedback can be measuring impact of TEAM initiatives on virus used to update its CBA outbreaks. • Implement measures for the Seven Key Metrics: Virus productivity impact – specifically: Metrics (below) • Number of VA personnel affected per virus Seven Key Metrics outbreak There are over 100 variables in the model, about 20 are unique and only seven are critical for • Duration of productivity impact per virus future metrics (i.e., they have the highest outbreak information value) • Fraud losses per year • Average annual cost per affected person • Intrusions per year • Pandemic virus events per year • Productivity loss during duration of outbreak • Number of VA personnel affected per virus outbreak Method : A random post-event survey of the • Duration of productivity impact per virus affected areas should assess each of these (only outbreak minor rewording of the current survey is needed). • Average annual cost per affected person The VIA indicates that a phone survey of 50 to • Productivity loss during duration of outbreak 80 respondents should be sufficient (this should be possible in two days just after the event). Anti-virus reports will also help to put ranges on 8.4.2 Detailed Metrics Information number affected and duration. Metric: Fraud losses per year Method : Continued analysis of reported frauds is 8.4.3 VAPKI Roll-out Criterion critical. Every step should be taken to encourage fraud reporting (emphasize in TEAM). The roll-out of VAPKI should occur in a Ultimately, diligent reporting and periodic audits particular order and it should only be are the best measure of fraud losses. implemented when a certain criterion is met. The main effect of VAPKI is to reduce unauthorized Metric: Intrusions per year intrusions and the main cost of unauthorized Method : Intrusion Detection should report intrusions is fraud. intrusions per year by VA area so that the Putting a priority on rolling-out VAPKI where following can be compared: fraud is the highest ensures the maximum impact of VAPKI. Wherever possible, VAPKI roll-out • Groups that have been trained under TEAM should be prioritized by "Annual fraud losses per vs. groups that have not person". The cost of implementing VAPKI is on a per-person basis therefore implementing on an • Groups that have rolled out VAPKI vs. annual fraud per person basis would be optimal. groups that have not The following formula should be used to test if a particular group of individuals should have VAPKI: • Groups that have implemented a simplified sign-on solution vs. groups that have not VAPKI Roll-out Criterion This is the basis for measuring impact of these 1.1% x (group fraud costs/yr) > VAPKI cost/person initiatives on intrusions per year: "Reduction in (number of people in group) Logical Unauthorized Intrusions". The quantity 1.1% is the expected reduction in Metric: Pandemic virus events per year fraud costs per person where VAPKI is Method : Anti-virus should report virus implemented. In other words, if the annual fraud outbreaks by VA area so that groups that have costs per person were $500 then the VAPKI costs been trained under TEAM vs. groups that have per person must be less than $5.50 per person to 23
  27. 27. justify rolling it out. Any group that does not meet this rule should be low priority for VAPKI or not receive it at all. 8.4.5 Additional Recommendations According to the estimates average fraud costs The contractor also made recommendations in the per person per year may be between $330 to following investment areas : $750. Differences among groups within the VA will vary by even more than this. In some groups Investment Area : Simplified Sign-on fraud costs per person per year may be thousands of dollars. In such high risk groups the costs of When specific solutions in Simplified Sign-on implementing VAPKI is easily justified. are formulated they should be given a full However, many groups will not be able to justify risk/return analysis. Currently, no specific the per person license fee for VAPKI by this rule investments are identified for the "optional" piece and therefore, should not have VAPKI. of Simplified Sign-on. A variety of bio-metric technologies and other solutions must still be 8.4.4 Six-month Review assessed. When particular plans are defined then they should be given a separate risk/return A review of all reports generated by VA CIRC, analysis with the AIE approach. This will ensure Anti-virus, and Intrusion Detection should occur that the investment is economically justified. The twice a year. Each of the Key Seven Metrics only reason a risk/return analysis cannot be done should be updated with all information available. at this time is because no specific investment has The spreadsheet model should be updated to been defined in this area. Without a specific reflect these findings. Key decisions on project scope and purpose identified, it will not continued roll-outs will be affected be possible to do the proper risk/return analysis of the investment. A method will be used to assess the total number of viruses and intrusions based on cross- referencing reports from anti-virus, intrusion Investment Area : Training, Education, detection, and VA CIRC. The following formula Awareness and Message Building (TEAM) insert shows how to aggregate VACIRC and intrusion detection but, intrusion detection could As more accurate costs for specific optional be substituted by anti-virus and the method TEAM solutions are identified, they should go would be the same. through a risk/return analysis with the AIE approach. As with Simplified Sign-On, this will ensure that the investment is economically Formula for aggregating reported incidents justified. But, again, this cannot proceed until a specific project scope is identified. X=(A+B)/A*C-C Estimated total = A+B+C+X Investment Area : IT Systems Certification and Accreditation Program (ITSCAP) A = number of incidents reported by both VACIRC and intrusion detection The VA plans to develop an ITSCAP scoring model to assess the security of a system before it B = number of incidents reported by VACIRC is put into production. The current approach but not by intrusion detection consists of a checklist of attributes that will be assessed for each system under the assumption C = number of incidents reported by intrusion that the attributes say something about the detection but not by VACIRC security of the system. The checklist would create a score or report card which VA would use X = estimated number of incidents unreported to determine if the system is safe. The scoring model will be more accurate if it is based on a real statistical analysis that predicts security risks The first six-month review should take place in based upon system attributes and weights the November, 2001 and then every six months factors according to actual security risks instead thereafter. of an arbitrary scoring method. 24