From Jack:I owe much of my success in this effort to my lovely spouse, Patti, whoserved as my partner, friend, and colleague in this endeavor. She is anexcellent consultant, an outstanding facilitator, a tenacious researcher,and an outstanding writer. Thank you, Patti, for all you do. Also, thanksto Wayne Brantley, who made this book a reality.From Patti:As always, much love and thanks go to Jack. You invest in others muchmore than you get in return. What a contribution you make! Thank youfor your inspiration and the fun you bring to my life.From Wayne:Jack, you are truly a guru. I appreciate the knowledge and insight thatboth you and Patti have shared with me. I want to thank my family fortheir love, support, and belief in me. I also want to thank Nathan Biskwho has allowed me to expand my horizons as a training professional.Jack and Patti PhillipsBirmingham, AlabamaWayne BrantleyTampa, Florida
ContentsPreface xiThe Need for This Book xiA Guide to Project ROI xiiCredibility Is Key xiiiAudience xiiiTarget Areas for Projects xivThe Difference xivFlow of the Book xvAcknowledgments xviiAbout the Authors xix1 Project Management Issues and Challenges 1What Is Project Management? 1Why Projects Fail 4Project Leadership: Getting Results 8Project Management Issues 9Final Thoughts 142 The Project Management Lifecycle 15The Project Management Steps 15Project Management Solutions 23Project Management Maturity Model 26Final Thoughts 293 ROI Methodology Basics 31Types of Data 31The Initial Analysis 34The ROI Process Model 36Operating Standards and Philosophy 47Implementing and Sustaining the Process 48vii
viii CONTENTSBeneﬁts of This Approach 49Final Thoughts 514 Achieving Business Alignment with the Project 53Importance of Business Alignment 53Determining the Potential Payoff 54Determining Business Needs 59Determining Performance Needs 64Determining Learning Needs 65Determining Preference Needs 67Case Study: Southeast Corridor Bank 68Developing Objectives for Projects 72Final Thoughts 765 Measuring Reaction and Learning 79The Importance of Reaction 79The Importance of Learning 81Sources of Data 84Topics for Reaction Measures 86Topics for Learning Measures 87Data Collection Timing 88The Challenges and Beneﬁts of Measuring Learning 88Data Collection Methods 90Data Use 93Final Thoughts 946 Measuring Application and Implementation 95The Importance of Application and Implementation 95Challenges 97Measurement Issues 99Data Collection Methods 101Barriers to Application 104Application Data Use 105Final Thoughts 1067 Measuring Business Impact 107Project versus Project Management 107The Importance of Business Impact 108Collecting Effective Impact Measures 110Business Performance Data Monitoring 117Data Collection Methods 118
Contents ixMeasuring the Hard to Measure 127Final Thoughts 1288 Isolation of Project Impact 131The Importance of This Issue 131Preliminary Issues 134Isolation Methods 136Final Thoughts 1509 Converting Data to Money 153The Importance of Converting Data to Money 153Key Steps in Converting Data to Money 156Standard Monetary Values 157When Standard Values Are Not Available 164Technique Selection and Finalizing Value 171Final Thoughts 17510 Measuring the Intangibles 177The Importance of Intangibles 177Measuring and Analyzing Intangibles 180Confronting Intangibles 186Final Thoughts 19811 Monitoring Project Costs and Calculating ROI 201The Importance of Costs and ROI 201Fundamental Cost Issues 203Speciﬁc Costs to Include 206Cost Classiﬁcations 208The ROI Calculation 209Other ROI Measures 214Final Thoughts 21512 Forecasting Value, Including ROI 217The Importance of Forecasting 217The Timing of Forecasting 219Pre-Project ROI Forecasting 221Forecasting with a Pilot Program 231Forecasting ROI with Reaction Data 232Forecasting ROI with Learning Data 237Forecasting ROI with Application Data 237Forecasting Guidelines 238Final Thoughts 241
x CONTENTS13 Reporting Results 243The Importance of Communicating Results 243Principles of Communicating Results 245The Process for Communicating Results 247The Need for Communication 248The Communication Plan 249The Audience for Communications 250Information Development: The Impact Study 252Media Selection 254Reactions to Communication 261Final Thoughts 26114 Implementing and Sustaining ROI 263The Importance of Sustaining the Use of ROI 263Implementing the Process: Overcoming Resistance 264Assessing the Climate 266Developing Roles and Responsibilities 266Establishing Goals and Plans 269Revising or Developing Policies and Guidelines 270Preparing the Project Team 272Initiating ROI Studies 273Preparing the Clients and Executives 275Removing Obstacles 275Monitoring Progress 278Final Thoughts 278Endnotes 279Index 283
PrefaceIf you are reading this book, chances are you are interested in projectmanagement, or you currently have or offer project management ser-vices. Possibly your alma mater is Hard Knocks U and you already knoweverything, or you have grown tired of starting over and over from failedprojects. Whether you are a beginner with project management or are aseasoned pro, you need to understand how to justify the value of projectsto executives. This book will cover the basics of successful project man-agement and will show how to evaluate projects and project managementusing the time-tested Return on Investment (ROI) methodology.THE NEED FOR THIS BOOKIn recent years, we have witnessed change in organizational accountabil-ity, especially toward investment in projects and project management(PM). Project sponsors and those who have responsibility for project suc-cess have always been concerned about the value of their initiatives.Today this concern translates into ﬁnancial impact—the actual mone-tary contribution from a project. Although monetary value is a criticalconcern, it is the comparison of this value with the project costs thatcaptures stakeholders’ attention—and translates into ROI.‘‘Show me the ROI’’ is the familiar response from individuals asked toinvest (or continue to invest) in major projects. At times, this response isappropriate. At other times, it may be misguided; measures not subject tomonetary conversion are also important, if not critical, to most projects.However, excluding the ROI from a success proﬁle is unacceptable in thisage of the ‘‘show me’’ generation. The ROI is often required before a projectis approved. Sometimes, it is needed as the project is being designed anddeveloped. Other times, it is needed after project implementation.This issue is compounded by concern that most projects today failto live up to expectations. A systematic process is needed that canxi
xii PREFACEidentify barriers to and enablers of success and can drive organizationalimprovements.The challenge lies in doing it—developing the measures of value,including monetary value, when they are needed and presenting them ina way so that stakeholders can use them• Before the project is initiated• During design and development, to plan for maximum value• During implementation, so that maximum value can be attained• During post-analysis, to assess the delivered value against theanticipated valueThis book is a guide that addresses all four scenarios.A GUIDE TO PROJECT ROIThis new book is a basic guide for anyone involved in implementing majorprojects—human capital programs, technology implementations, systemsintegration, new processes, Six Sigma, product design, new policies, andprocedures, or any other type of project where signiﬁcant expendituresof time and money are at stake. Strategies to assist in forecasting thevalue of the project in advance and in collecting data during and afterproject implementation are presented. This book uses a results-basedapproach to project evaluation, focusing on a variety of measures that arecategorized into six data types:1. Reaction and Perceived Value2. Learning and Conﬁdence3. Application and Implementation4. Impact and Consequences5. Return on Investment6. IntangiblesThis book offers a step-by-step guide to identifying, collecting, analyz-ing, and reporting all six types of data in a consistent manner that leadsto credible results.In addition, the book shows how to measure the impact and ROI of avariety of project management tools, such as:• Project Management Training• A Project Management Methodology
Preface xiii• Systems and Software• The Project Management Ofﬁce• Dedicated PM ResourcesCREDIBILITY IS KEYThis unique book focuses on building a credible process—one that willgenerate a balanced set of data that are believable, realistic, and accurate,particularly from the viewpoint of sponsors and key stakeholders. Morespeciﬁcally, the methodology presented in this book approaches credibilityhead-on through the use of:• Balanced categories of data• A logical, systematic process• Guiding principles, a conservative set of standards• A proven methodology based on thousands of applications• An emphasis on implementing the methodology within an organiza-tion to ensure that the process is sustained• A procedure accepted by sponsors, clients, and others who fundprojectsThe book explores the challenges of measuring the hard to measureand placing monetary values on the hard to value. It is a reference thatclariﬁes much of the mystery surrounding the allocation of monetaryvalues. Building on a tremendous amount of experience, application,practice, and research, the book draws on the work of many individualsand organizations, particularly those who have attained the ultimatelevels of accountability using the ROI methodology. Developed in aneasy-to-read format and fortiﬁed with examples and tips, this is anindispensable guide for audiences who seek to understand more aboutbottom-line accountability.AUDIENCEThe primary audience for this book is project managers concerned withthe valuation of their projects. Project managers are strongly committedto their projects and must show value in terms that project sponsors need.This book will provide all the information to accomplish this.A second audience is executives, administrators, and leaders who fundand support projects and project management. These executives need to
xiv PREFACEsee the value of projects and the value of project management. Impactand ROI are their two most important measures, and this book will showhow to connect projects to them.This book is also intended for professionals, analysts, and practitionerswho are responsible for evaluating the success of a project. It shows howthe various types of data are collected, processed, analyzed, and reported.This book serves as a guide to do this.Finally, another audience includes consultants, researchers, and pro-fessors who are dedicated to unraveling the value mystery, trying tounderstand more about the difﬁcult and demanding challenges of devel-oping measures and values for a variety of target areas. This book willmake an important contribution to the literature.TARGET AREAS FOR PROJECTSProject Management ROI: A Step-by-Step Guide for Measuring the Impactand ROI for Projects is geared toward a variety of functional areas inorganizations where projects are managed. These areas include (but arenot limited to) projects in:• Human resources, human capital• Learning and development, performance improvement• Technology, IT systems• Meetings, events, and conferences• Sales, marketing• Public relations, community affairs, government relations• Quality, Six Sigma• Operations, methods, engineering• Research and development, innovation• Finance, compliance• Logistics, distribution, supply chain• Public policy projects• Social projects• Charitable projects• Project management toolsTHE DIFFERENCEWhile other books may attempt to address the accountability of projects,this new book presents a methodical approach that can be replicated
Preface xvthroughout an organization, enabling comparisons of results from oneproject to another. The process described in this book is the mostdocumented method in the world, and its implementation has beenphenomenal, with over 4,000 organizations currently using it to measuresuccess routinely. Over 3,000 individuals have become a Certiﬁed ROIProfessional (CRP) through the ROI Institute. While many books tackleaccountability in a certain function or process, this book shows a methodthat works across all types of projects, ranging from leadership develop-ment to the implementation of new technology and from new educationalprograms to public policy initiatives.FLOW OF THE BOOKProject Management ROI: A Step-by-Step Guide for Measuring the Impactand ROI for Projects presents a methodology for determining the ROI on aproject, referred to as the ROI methodology. After identifying and explor-ing the factors that have created interest in this level of accountability,the book focuses on the process, showing how the ROI is developed, step-by-step, with each chapter devoted to each major element. In addition,two other chapters highlight matters that are critical to the overall pro-cess. One discusses the up-front analysis necessary to deﬁne the speciﬁcneed for the project, and the other focuses on forecasting the value beforethe project is developed and implemented. The remainder of the bookdetails the strategies and actions needed to sustain the methodology.
AcknowledgmentsNo book is the work of the authors alone. Many individuals, groups, andorganizations shaped the development of this book. We owe particularthanks to the hundreds of clients with whom we have had the pleasureto work in the past two decades. They have helped to develop, mold, andreﬁne this methodology. Their contributions are evident.Thanks to John Wiley for their support of this book. Special thanks toBob Argentieri, Executive Editor, for clearly seeing the need for this bookin the project management community.Many thanks go to Linda Arnall at the ROI Institute, who invariablycame through when we needed her for this assignment. Linda approachedthis project with a vengeance, and this is a much better book with herinput and effort.We would also like to thank our families. In spite of our ‘‘absence,’’ youcontinued to cheer us on. We love you for that and much more!xvii
About the AuthorsJack J. Phillips, Ph.D., a world-renowned expert on accountability,measurement, and evaluation, is chairman of the ROI Institute, aresearch, consulting, and workshop provider. He provides consultingservices for Fortune 500 companies and organizations in forty coun-tries. In addition, he conducts workshops for major conference providersthroughout the world. Phillips is also the author or editor of more thanforty books—ﬁfteen about measurement and evaluation—and more than150 articles.Phillips has received several awards for his books and work. TheSociety for Human Resource Management presented him an award forone of his books and honored a Phillips ROI study with its highest awardfor creativity. The American Society for Training and Development gavehim its highest award, Distinguished Contribution to Workplace Learningand Development for his work on ROI. On three occasions, Meeting Newsnamed him one of the 25 Most Inﬂuential People in the Meetings andEvents Industry, based on his work on ROI. His work has been featuredin the Wall Street Journal, BusinessWeek, and Fortune magazine. He hasbeen interviewed by several television programs, including CNN.His expertise in ROI measurement and evaluation is based on morethan twenty-seven years of corporate experience in ﬁve industries (aero-space, textiles, metals, construction materials, and banking). Phillipshas served as training and development manager at two Fortune 500ﬁrms, senior HR ofﬁcer for two ﬁrms, president of a regional federalsavings bank, and management professor at a major state university.Phillips and his wife, Dr. Patricia P. Phillips, recently served asauthors and series editors for the Measurement and Evaluation Seriespublished by Pfeiffer (2008), which includes a six-book series on theROI Methodology and a companion book of 14 best-practice case studies.Other books recently authored by Phillips include ROI for Technol-ogy Projects: Measuring and Delivering Value (Butterworth-Heinemann,xix
xx ABOUT THE AUTHORS2008); Return on Investment in Meetings and Events: Tools and Tech-niques to Measure the Success of all Types of Meetings and Events(Butterworth-Heinemann, 2008); Show Me the Money: How to Deter-mine ROI in People, Projects, and Programs (Berrett-Koehler, 2007);The Value of Learning (Pfeiffer, 2007); How to Build a SuccessfulConsulting Practice (McGraw-Hill, 2006); Investing in Your Company’sHuman Capital: Strategies to Avoid Spending Too Much or Too Little(Amacom, 2005); Proving the Value of HR: How and Why to MeasureROI (SHRM, 2005); The Leadership Scorecard (Elsevier Butterworth-Heinemann, 2004); Managing Talent Retention (Pfeiffer, 2009); Returnon Investment in Training and Performance Improvement Programs, 2nded. (Elsevier Butterworth-Heinemann, 2003); The Project ManagementScorecard, (Elsevier Butterworth-Heinemann, 2002); Beyond LearningObjectives (ASTD, 2008); The Human Resources Scorecard: Measur-ing the Return on Investment (Elsevier Butterworth-Heinemann, 2001);Measuring for Success (ASRD, 2010) and The Consultant’s Scorecard(McGraw-Hill, 2000). Phillips served as series editor for ASTD’s In Actioncasebook series, an ambitious publishing project featuring 30 titles. Hecurrently serves as series editor for Elsevier Butterworth-Heinemann’sImproving Human Performance series.Phillips has won awards for his work, research, and publicationsfrom the Society for Human Resources Management, ASTD, and otherorganizations.Patricia P. Phillips, Ph.D., is president and CEO of the ROI Insti-tute, a leading source of ROI competency building, implementationsupport, networking, and research. She assists organizations with theimplementation of the ROI methodology in countries around the worldincluding South Africa, Singapore, Japan, New Zealand, Australia, Italy,Turkey, France, Germany, Canada, and the United States.Phillips’s academic accomplishments include a doctoral degree in inter-national development and a master’s degree of arts in public and privatemanagement. She is certiﬁed in ROI evaluation and has been awardedthe Certiﬁed Performance Technologist designation by the InternationalSociety for Performance Improvement (ISPI).Phillips’s publications include The Bottomline on ROI, which won the2003 ISPI Award of Excellence; The Human Resources Scorecard: Mea-suring Return on Investment; and several of ASTD’s In Action casebooks,Measuring Return on Investment; Measuring ROI in the Public Sector, andRetaining Your Best Employees. She is published in a variety of journals,
About the Authors xxiserves as Professor of Practice teaching evaluation, survey design, andqualitative research at The University of Southern Mississippi, andspeaks on ROI at a variety of conferences.Wayne Brantley, MS Ed, PMP, ITIL, CRP, CPLP is the SeniorDirector of Professional Education for Villanova University Online.Wayne has taught and consulted on the topics of project manage-ment, quality management, leadership, curriculum development, Inter-net course development, and return on investment to Fortune 500companies around the world. With over twenty-ﬁve years of experiencewith the Air Force as a project manager for AF technology training andcurriculum development programs, Wayne has developed numerous AFand corporate training, classroom, and multimedia programs.Wayne has spoken at numerous conferences for organizations such asthe Project Management Institute (PMI®), the International Society forPerformance Improvement (ISPI), and the American Society of Trainingand Development (ASTD). Wayne is certiﬁed by the Project ManagementInstitute as a Project Management Professional (PMP), by EXIN asInformation Technology Infrastructure Library (ITIL) Foundation, byASTD as a Certiﬁed Professional in Learning and Performance (CPLP),and by the ROI Institute as a Certiﬁed Return on Investment Professional(CRP).Wayne is currently a continuing education faculty member forVillanova University Online and The Florida Institute of Technology.
2 PROJECT MANAGEMENT ISSUES AND CHALLENGESbody of knowledge, known as the project management body of knowledge,identiﬁes this profession’s accepted standards for PM.1Although PMI has an established body of knowledge, it can be arguedthat there is more than one way to ‘‘skin a cat’’ (apologies to pet loverseverywhere). Many parallels and similarities to the methodology usedby the PMI are in existence. At the end of the day, these differentmethodologies all say the same thing, albeit in different ways.The processes and issues shared in this book were not inspired bya climb up a Tibetan mountain in search of nirvana, or by a HarvardUniversity dissertation. They came from one of the most recognizablelearning institutions in the world—‘‘The School of Hard Knocks.’’ Thisbook is a tribute to the pain, agony, and turmoil that many projectmanagers have experienced.Many of you have assumed or been granted the title of ‘‘AccidentalProject Manager.’’ As most work is accomplished incrementally throughprojects, every time an ad campaign is launched, a system implementationis planned, a quality initiative is implemented, or a new drug is developed,a project is completed. The eternal weekend ‘‘honey-do’’ agenda evenqualiﬁes as a project list. The argument can be made that most work isaccomplished using the skill of project management.Let’s ﬁrst understand the meaning of the term project. By deﬁnition,it is a temporary and unique effort. The alternative is operational man-agement, which is comprised of the repetitive daily jobs that are notprojects. Projects have a lifecycle that is identiﬁed by the PMI as shownin Table 1.1.Project costs can range from a few dollars to several billion dollars.Project time can take several years in duration. Projects will requireTable 1.1 Project Management LifecycleProcess What OccursInitiation Project selection—go/no-go decisionPlanning Identifying all the work that is to be done, scheduling,budgeting, and much moreExecution Doing the work—building the productMonitoring andControlEvaluating and reporting project statusClosing Handoff of project and wrap up
What Is Project Management? 3as few as one part-time person to thousands of project team membersworking together. Organizations perform projects ad hoc, while othershave matured, deﬁned processes in place. Some organizations see projectmanagers as gloriﬁed administrative assistants, while others see themas omniscient wizards (it is possible that this is a view held singularly bythe project manager).As far as skill sets go, organizational development specialists consis-tently see project management as a top skill set required of employees. Aquick review of professional and managerial job descriptions will revealthat project management is a much in-demand skill. Some organizationsvalue the project management skill so much that they employ certiﬁedproject managers, known as Project Management Professionals (PMPs).Some organizations have even made project management a career path.Certiﬁcation Magazine, an IT magazine, has repeatedly rated the PMPas a top-ten certiﬁcation.2 This is impressive when you consider that thePMP is not an IT-speciﬁc skill. Throughout industry and government, thePMP is an in-demand certiﬁcation, and even the U.S. federal governmenthas mandated that IT project managers be certiﬁed.The interest in certiﬁcation is based on its perceived value. In arecent conference about certiﬁcation, the featured presenter made anintriguing declaration that today, certiﬁcations are more valuable than adegree. Taken at face value, what does this statement mean? Degrees arevaluable, but can sometimes be too generic and not speciﬁc enough to aparticular job or a position. With many certiﬁcations, a work experiencecomponent is almost always combined with a challenging examinationbased on a body of knowledge. Certiﬁcation predicts with some levelof certainty that someone can perform those duties associated with theperformer’s level of certiﬁcation.Becoming a successful project manager is easier with training andinformation available. PMI is the internationally recognized body ofgovernance for project managers around the world. As of 2011, PMI hasover 300,000 members, and there are over 400,000 certiﬁed PMPs. PMImaintains the standard for project management known as the Guideto the Project Management Body of Knowledge (PMBOK). The PMBOKidentiﬁes the processes, knowledge, and skills that a project managershould have. The PMBOK is a standard, not a textbook, and hence thereader will ﬁnd it informative rather than instructional. This leaves manyreaders frustrated and in need of further guidance.
4 PROJECT MANAGEMENT ISSUES AND CHALLENGESThe PMBOK lists nine knowledge areas that a project manager shouldidentify:• Integration Management—Ties all the other processes together.Includes the coordination, consolidation, and integrative processesnecessary to successfully execute a project.• Scope Management—Includes all the efforts to articulate andidentify the features and functions of a product. Includes the identi-ﬁcation of requirements, deliverables, tasks, and activities requiredto produce the product of the project.• Time Management—Includes all the processes required to estimateeffort duration and sequence in order to complete the project in atimely manner.• Cost Management—All costs are planned for and estimated inthis process. This process also includes project cost baselines andprocesses to control project budgets.• Quality Management—Processes required to achieve project deliv-erables that meet quality objectives for an organization.• Human Resource Management—Process required to organize theproject team.• Communications Management—The process of creating, collecting,and disseminating project information.• Risk Management—Includes planning for how an organization willconduct risk management. This includes analyzing, prioritizing, andresponding to risks.• Procurement Management—The processes needed to either pur-chase or acquire the needed products or services from outside theproject team.The various skills and knowledge throughout these nine areas mayrequire familiarity, while others demand mastery. It depends on yourfunction in project management. Some of these skills are referred toas soft skills, while others are the hard or technical skills of projectmanagement.WHY PROJECTS FAILThe ﬁrst Chaos Report published by The Standish Group identiﬁed somesobering data on IT projects in the United States:3• Over $250 billion is spent annually in the United States on ITapplication development projects
Why Projects Fail 5• There are approximately 175,000 projects annually• 31.1 percent of projects are cancelled before completion• 52.7 percent of projects will cost over 189 percent of budgetWhat do these numbers mean and what can be done about them today?As reported by the Chaos Report, this means that there would be $81billion in cancelled projects and a cost of an additional $59 billion overthe original estimates. While the Standish Group has updated this reportand has shown project performance improvements, organizations need torealize that similar results are occurring in their organization.Unfortunately, too many projects fail. They fail for a variety of reasons,and understanding those reasons helps us to have success with our ownprojects. The methodology introduced in this book is a way to measurethe success of a project throughout its lifecycle. When things are notworking as well as they should, data are available to make necessaryadjustments. In essence, this methodology focuses on results of theproject, ensuring that the project delivers the appropriate value forthe client. The methodology helps ensure that projects will not fail in thefuture. First, let’s explore some of those reasons for failure.The ROI Institute has been helping thousands of individuals showthe value of their various projects. Many project failures can be foundin the analysis of the results. A failure may mean that the projectproduced little or no results, delivering a negative ROI. Sometimes theyproduced less-than-desired results, disappointing the client. Perhaps theresults were acceptable, but there was signiﬁcant room for improvement.In any case, the project did not live up to its expectations. We haveanalyzed hundreds of projects and have listed the key reasons thatprojects fail.Lack of Business AlignmentUnfortunately, too many projects are ‘‘fuzzy’’ when it comes to the align-ment with speciﬁc business measures. This seems a little odd when weconsider that most projects start with a business initiative. While thatmay be the case, the alignment is sometimes very nonspeciﬁc. Examplesof projects might be the implementation of a business development con-ference, a new payroll system, or a launch of a ‘‘green’’ company. Thespeciﬁc business measures may not be clear in these situations, andwithout this clear connection to the business, their success in terms ofdriving business value may be limited or nonexistent. Therefore, one of
6 PROJECT MANAGEMENT ISSUES AND CHALLENGESthe ﬁrst steps is to ensure that the project is connected to the business,driving speciﬁc business measures.Inappropriate Project SolutionSome projects are designed to implement a particular solution. It mayinvolve the purchase of new software, the implementation of a newquality system, or processes taken from books, for example, The OpenBook Company. These prepackaged solutions may not be the methods toaddress the target project. That is, the solution itself will not drive thebusiness measures that are desired to change.Project Participants Are Not EngagedThe project team must be fully engaged. The project team consists ofthe individuals who must make the project work. They must clearlyunderstand the need and reason for the project. Lack of explanation orlack of persuasion can create an adverse reaction to the project, dooming itto failure. It is important to ensure that expectations are clearly outlinedand engagement occurs early.Lack of Focus on Business ResultsStakeholders sometimes lose sight of the ultimate objective. It is notclear as to why they are involved in the project as the project evolves.Business measures should be translated into impact objectives that arecommunicated to all stakeholders. Success should be routinely monitoredagainst those objectives. This provides focus throughout the project,ensuring that business alignment is always achieved.Failure to Prepare the Environment for the ProjectUsually, projects are implemented in a work system. Implementationoften involves change, and change must be accepted and supported inthat work unit. An important part of a project is to understand theenvironment. Any impediments to the success of the project or barriersto project implementation must be addressed early and often. Ideally,part of project planning would be the identiﬁcation of inhibiting factors
Why Projects Fail 7to the success of the project, and tackling those issues before they becomebarriers to success.Lack of Accountability within the ProjectToo often, project participants and other stakeholders do not feel thatsuccess is their responsibility. If no one accepts accountability, then no oneis accountable, and the project will fail as a consequence. Ideally, everyperson involved must understand his or her responsibility, clearly deﬁnedwith expectations and very speciﬁc objectives. It should be apparentto the project manager and other stakeholders that the project teammembers are meeting their goals, standards, and expectations. Withoutthat commitment, the project could easily drift and ultimately fail.Problems with Data CollectionSome projects fail because the appropriate data could not be collected, wasnot available, or there was no speciﬁc responsibility to collect the data.Data collection will always be an issue. At the impact level, business datashould be readily available in the system and records of an organization.Additional data collection is needed, some at the project level, and some atthe department or work unit level where the project is being implemented.Data collection must be carefully planned. If a particular data set is notreadily available, it should be developed from the speciﬁcations of thosewho must collect it and present it to the project management team.Failure to Isolate the Effects of the ProjectOften, there are factors apart from the project solution involved in drivingthe success of a project, whether it is new technology, a new ad campaign,or a quality initiative. The success of that technology, campaign, orinitiative is a huge driver. There are also external factors that couldinﬂuence success of the business measures linked to the project, apartfrom those internal to the project team itself. In addition, there is theeffect of the project management solutions, such as project managementtraining, project management tools, project management ofﬁce, or otherproject management solutions. An important challenge is to be able tosort out what has caused the results, isolating the success to individual
8 PROJECT MANAGEMENT ISSUES AND CHALLENGESfactors. This provides the sponsors who fund the project or provide projectmanagement solutions a clear understanding of the factors that causedthe success. The good news is that this step can be achieved crediblyin any project setting. The disappointing news is that this frequently isnot tackled appropriately in most projects, and leaves the success of theproject in doubt.Lack of Involvement with Key ManagersOutside the project team, there are other managers who support theproject and make it successful. Sometimes these are the managers of theparticipants involved in the project. At other times, they are the managersin the area where the project is being implemented. In either case, theirsupport and reinforcement is essential for the project’s success. Thesemanagers must be identiﬁed early, and steps must be taken to ensurethat they live up to their roles and responsibilities, and provide theproper reinforcement and support needed to make the project successful.Otherwise, it could be a failure.PROJECT LEADERSHIP: GETTING RESULTSWhat makes a project successful? When the areas described earlierare addressed throughout the lifecycle of the project, success is almostguaranteed. To achieve success is to avoid the pitfalls that cause failure,understanding those issues and making sure that they are working withthe project instead of against it. In essence, the ROI Methodology drivesthese kinds of results, because it provides a process and collects data toensure that failure is prevented.Strong leadership is necessary for projects to work. Leaders mustensure that projects are designed to achieve results. Table 1.2 showsthe twelve actions that must be taken to provide effective, results-basedproject management, which is critical to delivering results at the ulti-mate level, ROI. However, only one of the items involves data collectionand evaluation (number 11). The remaining leadership areas representsteps and processes that must be addressed throughout a project’s cycle.We developed these actions after observing, studying, conducting, andreviewing thousands of ROI studies. At the ROI Institute, we know whatkeeps projects working and what makes them successful. Following thesetwelve leadership roles can ensure project success.
Project Management Issues 9Table 1.2 Project Leadership for Results1. Allocate appropriate resources for projects.2. Assign responsibilities for projects.3. Link projects to speciﬁc business needs.4. Address performance issues involving the key stakeholders in the projectidentifying the behavior that must change.5. Understand what individuals must know how to make projects successful,addressing the speciﬁc learning needs.6. Develop objectives for the projects at multiple levels including reaction,learning, application, impact, and yes, ROI.7. Create expectations for success of the projects with all stakeholders involved,detailing what their role and responsibilities are for the project’s success.8. Address the barriers to the successful project early in the project so that thebarriers can be removed, minimized, or diminished.9. Establish the level of evaluation need for each project at the beginning sothat participants will understand the focus.10. Develop partnerships with key administrators and managers who can makethe project successful. For many, this is the manager or the person who isthe principle participant in the project.11. Ensure that measures are taken and the evaluation is complete withcollection and analysis of a variety of types of data.12. Communicate project results to the appropriate stakeholders as often asnecessary to focus on process improvement.PROJECT MANAGEMENT ISSUESSeveral issues need to be explored to understand the status and challengesof project management. These issues create opportunities for projectmanagers to make their projects more successful and prevent futurefailures.Identifying Detailed Requirements—Development of a DetailedWork Breakdown StructureA famous adage goes ‘‘Garbage in equals garbage out.’’ Projects musthave better-deﬁned requirements in order to decrease the rate of projectfailure. The project manager must deal with and meet the issues athand—the requirements already prepared to execute the project. These
10 PROJECT MANAGEMENT ISSUES AND CHALLENGESrequirements are sometimes incomplete, if not altogether incoherent.A growing discipline that is used to complement the PM role is theBusiness Analyst (BA). This position is becoming necessary to developrequirements that meet the goals of the project and the company.Equitable requirements for the project cannot be developed if the timeis not devoted to good planning. ‘‘Scope creep,’’ deﬁned as the exponentialexpansion of project features and functionality, can occur as the resultof less-than-credible requirements. Without detailed requirements, therewill be extra work, more resources used, and frustrations. This can alsoresult in increased cost and time for the project. The upfront effort thatmust be spent in the planning stage is needed to identify and developgood requirements. Here are some steps to enhance the requirements-gathering process.1. Assign the right people to gather the requirements.2. Improve the continuity between project initiation and planning.3. Draw a picture, if possible.4. Continue through until a detailed Work Breakdown Structure(WBS) is developed.5. Gain the approval of all stakeholders.Requirements tell you what is being developed. If the project plan isthe compass, requirements are the ‘‘true north.’’ Take the effort to masterthe skills to develop the requirements that will allow the development ofa detailed WBS that can meet the schedule and allocated budget.Project PlanningProjects are frequently poorly planned. Project managers overwhelminglyagree that their companies do not give due diligence to planning, theprocess that occurs after initiation. What often happens out of the purviewof the project manager is a commitment to do something—develop aproduct, service, or process. While the high-level planning process mayhave been completed, this may have resulted in a false sense of conﬁdencethat a plan or approach is in place to execute the project. Just a ‘‘few minordetails’’ are omitted that should only need a few hours to plan—and manysomewhat na¨ıve individuals (often executives) believe this to be true. ThePMBOK identiﬁes 20 processes that may be involved in the planning.They are listed in Table 1.3.
Project Management Issues 11Table 1.3 PMBOK Processes• Develop project management plan• Collect requirements• Deﬁne scope• Create WBS• Deﬁne activities• Sequence activities• Estimate activity resources• Estimate activity durations• Develop schedule• Estimate costs• Determine budgets• Plan quality• Develop human resource plan• Plan communications• Plan risk management• Identify risks• Perform qualitative risk analysis• Perform quantitative risk analysis• Plan risk responses• Plan procurementsIt seems simple—‘‘Only a couple of hours, and all of this shouldbe done and placed in a convenient Project Management Plan’’—whenperhaps several weeks would be a more appropriate timeframe. This mayseem daunting and overly detailed, but remember that project managersget paid to manage the project management plan and know what ishappening in each part of the plan. The plan is the compass and key toproject success.Exhaustive Risk ManagementA risk is an uncertain event that can result in a positive or negativeoutcome. It is amazing how we know that things will go wrong on ourprojects, yet we often manage to the fact that we can handle these asthey occur. The one member who you can count on being on your team is‘‘Murphy,’’ whose law is: what can go wrong will go wrong.As previously stated, poor planning will result in poor risk manage-ment. Think about how you have seen risk management performed in
12 PROJECT MANAGEMENT ISSUES AND CHALLENGESother organizations. This process is often not executed properly. Risk hasa set of processes that must also be given its due respect.• Risk Management Planning—identifying how you should performrisk management at the organization and project level. Risk man-agement is scalable to the project.• Risk Identiﬁcation—an exhaustive exploration of all the possiblerisks that you could have on a project. At this point, none are ruledout or ranked.• Qualitative Risk Analysis—probability and impacts are determinedto analyze the likelihood that the risk could occur.• Quantitative Risk Analysis—a more mathematical approach to fur-ther analyzing risk probabilities and impacts.• Risk Response Planning—this is when plans are developed to deter-mine how the risks could be handled.• Risk Monitoring and Control—the continued observance of riskmanagement processes. Determine if contingency plans were effec-tive and if the risks will reoccur.These processes must be performed repetitively throughout the lifecy-cle of the project. Doing them once will result in limited risk identiﬁcation.As the project plan is iteratively built, risk management should be itera-tively performed. Risk management plans should be updated throughoutthe development of the project plan and the project. The processes are notdifﬁcult, but they are time consuming.Development of the Project TeamThink about the evolution of becoming a project manager, from the ﬁrststep of employment, to job knowledge, to job proﬁciency, then becomingexpert enough to be assigned the title of Project Manager. This hasoccurred because the project manager became proﬁcient at the workof the project. The previous role was rooted in the comfort zone; thenew role removes us from that comfort zone. Project managers arenotorious for having highly developed technical skills and a notable lackof interpersonal skills. This lack of interpersonal skills is sometimes thereason for many project failures. This can be changed with a career pathfor project managers where they can be developed instead of appointed.
Project Management Issues 13With all that project managers need to learn, leadership skills maybe the most difﬁcult to understand. People are the most complex systemin the world. When personalities, backgrounds, education, cultures, andgender are factored, this challenge should be expected. This all adds up toa very complicated formula that requires a comprehensive understandingof human motivation. In addition to understanding people as individuals,group dynamics must be used. Effort, skills, and patience are the bestrecommendations to create a cohesive, high-performing team from agroup of individuals with diverse educational and interests backgrounds.The Tuckman model identiﬁes team development as ‘‘forming, storming,norming, and performing.’’4 This is a tidy model that identiﬁes what canbe expected as the team develops. It requires time to develop the skills tofacilitate these issues and become a great leader.An Organizational Culture for PMA proper culture must be developed. Securing ‘‘C’’ level support forinitiative project or project management is half of the battle. For example,the support of Jack Welch for Six Sigma was impressive. Through the SixSigma approach, GE was able to identify billions of dollars of savings, andWelch has stated that ‘‘Six Sigma must be a part of the corporate DNAfor GE.’’5 This is a deﬁnitive example of taking a stand and providingsupport. The support of a C level ofﬁcer will spur the momentum necessaryto achieve success of the initiative.After C level support, the other half of this battle is that the orga-nization as a whole needs to embrace this philosophy or methodology.Implementing a string of bureaucratic processes and challenging toolswill alienate the organization from embracing the concept. Start withcommunication and education about the implementation of the new pro-cesses and tools. Make it user-friendly and a part of the implementationplan. Show value and how it will make projects better.Initiate a grassroots effort by training key personnel. Offer the appro-priate amount of training that different segments of the workforce need.Begin with awareness training—maybe a four-hour overview of the con-cept to provide a 50,000 foot view. Then provide a basic level of training toestablish a common language and understanding of processes, and makeadvanced training and tools training available to those who will take onlarger roles. This training and implementation will prepare a workforce
14 PROJECT MANAGEMENT ISSUES AND CHALLENGESwith a common goal and a common vision, creating a culture of projectmanagement.FINAL THOUGHTSThis initial chapter reviews the fundamentals of project management andidentiﬁes project management issues. Understanding why projects failand what makes a project successful are the ﬁrst steps to understandingthe ROI of projects and project management. Understanding what isexpected from a project manager is also identiﬁed. The next chapterfocuses on the project management lifecycle and project managementsolutions.
16 THE PROJECT MANAGEMENT LIFECYCLEidentiﬁed at this level. The farther you move into the project planningprocess, the more detailed the effort becomes.Project PlanningThe planning phase begins after projects are initiated. In this phase, theproject management plan is developed. The areas identiﬁed in initiationare built upon in the planning process. The project manager is paid tomanage the project plan, an important component that serves as theroadmap for executing the project. Planning is an iterative process thatis repeated as greater detail is provided throughout the life of the project.Planning should include input from all stakeholders to ensure that theproject plan is comprehensive. The following are valuable tools used toplan projects:Work Breakdown StructureThe work breakdown structure (WBS) is the cornerstone of an effectiveproject management plan. It identiﬁes all the tasks and activities thatmust be accomplished for a project to succeed. Essential to a good WBSare detailed requirements that identify the features and functionalityof the product. The WBS can then be deconstructed, starting with theproduct and continuing to the subdeliverables and necessary tasks andactivities.SchedulingSchedules provide the framework for accomplishing the tasks and activ-ities of the WBS. While Gantt charts (bar charts that illustrate projectschedules) are prevalent, there is tremendous value in the scheduleor network diagram where the sequence of the tasks and activatesare identiﬁed. Dependencies between the various tasks and activitiesmust be understood to ensure the project tasks are accomplished at theappropriate time in order to complete the project.Time EstimatingTime estimates need to be as realistic as possible, and should beapproached as if submitting a bid on every project, external and inter-nal. Time estimating is a best-guess using the available information,and establishing the right climate for estimates is essential to ensure
The Project Management Steps 17accuracy. A culture should not be promoted where any estimate submit-ted will automatically be cut by 10 percent. The submitted estimatesmust be ﬁrm and monitored, and the project must be controlled to thebaselines established by the estimates.Sources of estimates are numerous. Start with those who are mostfamiliar with the project: subject matter experts, technical experts, thosewho will actually perform the work, or their managers. The impor-tance of having the most reliable source gather the estimates cannot beoveremphasized.A time estimation technique that is rarely used but deserves consider-ation is the Program Evaluation and Review Technique (PERT). This is aweighted mathematical estimating model that considers three durationvariables; optimistic, most likely, and pessimistic.BudgetingAssigning a project budget is difﬁcult and involves ‘‘best-guess’’ estimationmethods. All costs need to be factored into the budget estimate, includingdirect and indirect costs. The budget will generate a cost-performancebaseline called the performance measurement baseline. This baseline isoften referred to as the S-curve because of the shape it assumes, and willvary depending on when the project budget is spent.The project can be tracked to the performance measurement baselineas it progresses. The performance measurement baseline identiﬁes whatis ‘‘planned to be spent’’ at any point in the project; this is the plannedvalue (PV). The ‘‘percentage of this planned work that you complete’’ isthe earned value (EV). (Earned value is often used in government projectsand is enjoying widespread adoption throughout the commercial sectoras well.). The money that is ‘‘actually spent’’ on this work is the actualcosts (AC). These three values will allow you to identify project progressand use these values to forecast project performance.Human Resource PlanningHuman resource planning is another key consideration in project plan-ning. A signiﬁcant challenge often faced by project managers is the lackof formal authority over the people working on their projects, becausethose people typically report to someone else.Motivating and providing leadership is crucial for project managers. Aproject manager should be familiar with a number of management and
18 THE PROJECT MANAGEMENT LIFECYCLEleadership techniques in order to maximize team performance. Manage-ment and leadership books are available to help with PM developmentskills in this area. Understanding the political landscape of the organi-zation is essential to maneuvering around the land mines that lie beforethe project team.Risk ManagementRisk management plays a critical role in project planning. There arefactors or actions waiting to impede projects, and risk managementinitiatives will anticipate obstacles and put plans into place to overcomethem. Potential risks must be identiﬁed and prioritized, and strategiesdeveloped to minimize or avoid the risk. Project managers must realizethat time must be allotted for risk management planning to maximizethe ROI on projects.Quality ManagementQuality management asks two primary questions: ‘‘How good is goodenough?’’ and ‘‘Good enough according to whom?’’ Quality managementhas evolved over the years. Total quality management (TQM) is a qualitymanagement methodology that has been popular since the 1980s. Todaya similar, but more robust methodology called Six Sigma and LeanSix Sigma are in demand. These valuable theories and techniques areworth exploring. When the requirements of a project are identiﬁed,corresponding quality metrics need to be integrated into them so qualitycan be veriﬁed as the project develops.Procurement ManagementProcurement management is a specialized profession. Contracts andtheir legal structure affect the project, and project managers need tounderstand them.There are several different types of contracts, and the type of contractused can change a project’s ROI. Fixed-price contracts shift risk to theseller of services and commit the seller to the price they bid on. Costplus incentive fee–type contracts shift the risk to the buyer. The buyer iscommitted to what can be considered a blank check.Off-shoring has become a concern for many project managers, as issuesoften arise from cultural, language, and time differences. Contracting canbe used to mitigate or avoid certain risks. The risk that an organization
The Project Management Steps 19may not be able to perform a service or capability can be mitigatedthrough the use of contracts that outsource these risks.The Management PlanA project management plan addresses all the actions necessary to manageand complete a project. The plan deﬁnes how the project is executed,monitored, controlled, and closed. The content of the document variesdepending on the project. Table 2.1 presents a sample template for aproject management plan.Project ExecutionProject execution is how the work in the project plan is accomplished.The project manager will allocate much effort here to develop the projectteam. Teams experience a developmental evolution of their own. Timeand effort must be applied to ensure that they perform at a high level andthat the project’s mission is accomplished. Here are some of the tools andknowledge areas project managers need to execute projects.Team DevelopmentA popular saying in project management is ‘‘No project ever failed becauseof a bad Gantt chart.’’ Projects succeed because of people. ResearchTable 2.1 Project Management Plan Contents• Project scope management plan• Schedule management plan• Cost management plan• Quality management plan• Process management plan• Stafﬁng management plan• Communication management plan• Risk management plan• Procurement management plan• Milestones• Resource calendar• Schedule baseline• Cost baseline• Quality baseline• Risk register
20 THE PROJECT MANAGEMENT LIFECYCLEshows that human interactions, also known as the soft skills of projectmanagement, are often neglected in the professional development ofproject managers. The challenge is to utilize resources from many dif-ferent organizational departments to accomplish any project. Developingthis group of resources and turning them into a team requires leadershipfrom the project manager.Leadership DevelopmentDeveloping the leadership skills of a project manager incorporates manymethodologies and concepts. In order to be a successful project manager,understanding people and personality traits are critical. Negotiation andfacilitation skills are also necessary for the project manager.Quality AssuranceThe project management planning process identiﬁed that the projectquality standards should have been identiﬁed with the customer and keystakeholders. Understanding how to assure that the quality standardswill be achieved must be established. Quality assurance is done through-out the project development. The product should be evaluated to makesure that quality standards are being achieved.Distributing Project InformationA project manager can spend up to 90 percent of their time communicatinginformation on a project. They are gathering data, generating reports,and providing data. This can include status meetings, report generation,and project tracking information. Project management tools such as MSProject, Primavera, MS Word, and Visio can help a project managerreport project status.ProcurementsOutsourcing to subcontractors and contractors and off-shoring may be nec-essary, but can create numerous challenges for the project. Procurementscan also affect the proﬁtability and performance of the project.Identifying the requirements for the work to be done is critical todetermine what services are needed. There are many contract structuresthat a project manager can use to improve the ROI of the project.
The Project Management Steps 21Project Monitoring and ControllingAs the project is executed, monitoring and control activities are contin-uously accomplished to ensure that the project is on track. Correctiveand preventive actions are implemented to ensure that the project stayson plan. Here are some of the tools and knowledge used to monitor andcontrol projects.Change ControlChange control is advocated rather than change prevention on projects.Change can run rampant on projects, resulting in a phenomenon knownas ‘‘scope creep.’’ Scope creep occurs when the project/product featureschange and grow as the project develops. This phenomenon occurs oftenat the whim of the customer and sometimes the developer.An effective change control process starts with implementing formaldocuments that must ﬁrst be submitted before any changes to the scopeof the project are made. These documents should then be submitted toa change control board. The change control board should determine theimpact to the technical speciﬁcations, the budget, and the schedule. Thechange is then approved or disapproved based on these alterations.Earned Value ManagementEarned value is a useful tool to assess project budget and scheduleperformance. In order for an earned value management system to beimplemented, the project must have a detailed work breakdown structure,schedule, good duration and cost estimates. This will allow for a three-dimensional view of work planned, percent of work completed, and theactual cost for work completed.Earned value is daunting for many and is often avoided. EV shouldbe embraced as a performance management tool that will assist theproject team in assessing project status. It will help to develop budgetand duration estimating proﬁciency, allowing for better estimating andvariance control on future projects.Monitoring and Controlling RisksA risk management plan is useful to ensure that the project avoidscertain pitfalls. Contingency planning and implementation has savedmany projects and increased their ROI.
22 THE PROJECT MANAGEMENT LIFECYCLERisks are monitored continuously throughout the project, as they canchange the impact and/or probability assessed. Risks may have occurred,be ready to occur, or not have occurred at all, but are more costly if theyoccur toward the end of the project. Maintaining risk management plansis imperative to the success of the project. It is important to recognize thatrisk management is not only performed at the beginning of the project,but throughout the life of the project.Project ClosingProjects must be formally closed out. Following are three tools or processesused to close projects.Contract ClosureContracts must be closed out in order to complete the project. Contractscan be between your organization and a vendor, while others are betweenyour organization and your customer. There are many legal ramiﬁcationsinvolved in a contract closeout, and it is imperative to protect the organi-zation’s interest in this process. A contracting specialist or legal counselcan provide guidance in the contract closing process and its ramiﬁcations.Lessons LearnedLessons learned provide the opportunity to establish best practices foryour organization. Many organizations resort to ﬁnger-pointing whenthings don’t go so well. It is natural for employees to go into survivalmode when this occurs, and can create a condition where project problemsare hidden. This culture can be avoided by creating and encouraging anopen environment where problems can be openly addressed and analyzedfor solutions. This will allow the prevention or reduction of these types ofevents occurring in the future.Project CloseoutRecording and archiving the project documentation is invaluable asorganizational maturity in project management is developed. The projectaccomplishments will be a reference for other project managers as theyundertake new projects of their own.Archived project plans become the foundation for developing an orga-nization’s maturity in project management. Lessons learned from pastprojects are contained in these project plans, and future project baselines
Project Management Solutions 23can be established based on project performance. Project variance andfuture estimating improvements can be implemented.Obviously, the concept and formality of project management can bequite detailed. Implementing a project can be a challenge in any orga-nization. Processes, reporting, training, and new software introductionwill be implemented, and money will be spent. Showing senior leader-ship the return on their investment will support the project managementinitiative.PROJECT MANAGEMENT SOLUTIONSSpeciﬁc actions must be implemented to develop project managementmaturity. Not only should organizations say that they are implementingproject management, they must embrace it. The entire organizationmust commit to project management excellence. The following tools andtechniques will maximize the ROI for an organization’s investment inproject management.Senior Level SponsorshipFor any management methodology to be embraced, the initiative mustbe endorsed and sponsored by a senior level executive. The saying that‘‘it starts at the top’’ is imperative to maximize the effectiveness ofproject management. Many obstacles can be removed with senior levelendorsement, and it has the effect of gaining buy-in from other execu-tives to support the program. As a result, funding for various projectmanagement resources will be greatly enhanced.Senior level support can vary. It can include attending meetings, anddirecting policies and procedures in project management, but the primaryrole of senior level sponsors is review and approval. The support that theyprovide the project manager is critical to the success of any project in anorganization.The Project Management OfﬁceThe project management ofﬁce (PMO) should be the focal point for anorganization’s project management initiative. The PMO should be alignedunder one of the following options:1. Under the Chief Information Ofﬁcer (CIO) or the department wherethe majority of projects are generated, or
24 THE PROJECT MANAGEMENT LIFECYCLE2. Under the Chief Operations Ofﬁcer (COO) so that there is noparticular departmental function alignment. This is the preferredalignment, and will allow the PMO to support all the projects in theorganization without demonstrating departmental bias.3. Under the Chief Executive Ofﬁcer (CEO) for an organization with afew large, critical projectsThe PMO should have the following responsibilities, at minimum:• Deﬁne the methodology and establish best practices• Develop and identify the tools and templates to be used• Oversee the training for PMProject managers assigned to the PMO can be utilized as internal con-sultants to the various departments and their project management efforts.ROI data should be collected, analyzed, and maintained by the PMO.Dedicated Project Management ResourcesResources can be applied to projects on either a full-time or part-timebasis. Assigned project resources can report under authority of the PMOor they can maintain their functional alignment. This will require man-agement to use effective resource utilization so that they are not over- orunderused.Project Management MethodologyThe Project Management Institute (PMI®) has the internationallyaccepted methodology for project management. This organization hasbeen evolving the methodology for project management professionalssince 1969. A testament to their global reach is the number oforganizations that have endorsed their methodology and certiﬁcation,the Project Management Professional (PMP).The project management methodology that is selected by the organiza-tion must be standardized, and processes must be deﬁned, documented,and communicated throughout the organization. Project documentationmust be archived in order to develop and mature project managementprocesses in an organization. Archiving lessons learned and best prac-tices can be accomplished from this effort. We will explore this aspectmore as we discuss how an organization can develop project managementmaturity.
Project Management Solutions 25TrainingProject management is the most in-demand area of training since the mid-1990s, a testimonial to the demand for project management knowledgeand skills. Books can be as simple as Project Management for Dummies(For Dummies, 2010)1, or as involved as encyclopedia-thick textbooks.Training comes in all shapes and sizes, from one-day seminars to morethan thirty days of training during a four-year period. Online projectmanagement courses have proliferated since 2000, taking on the form oftalking textbooks to live virtual classrooms. The cost of these courses canexceed $10,000.A complete training proﬁle should include:• Sponsor training for senior management• Project management fundamentals training• Extensive and specialized project management training• Advanced project management training• Program and portfolio management training• Tools and software trainingTraining will enhance the probability of success for the project man-agement initiative. Training changes a culture that fears change, ordeveloping your workforce’s competency in new methods or tools. Throughtraining, employees will develop conﬁdence in their abilities to utilize thenew knowledge and skills.Systems and SoftwareAn investment in systems and software to execute project managementwill be required. The capability to communicate and share informationis constantly and rapidly evolving. Organizations can collect, store, andaccess data immediately from anywhere, and many monitoring and con-trol tools exist that can help an organization plan, execute, monitor,and report project performance. These tools must be selected prudently,and personnel must be trained to enhance the success of the projectmanagement initiative.Measures of Project Management PerformanceThe project plan is at the very core of project management. Without a planin place, there is no direction or guidance to meet the objectives set for the
26 THE PROJECT MANAGEMENT LIFECYCLE2 4(–)← →(+)$58M6 8 10 12 14 16 18Figure 2.1 Project variance.initiative. Project management performance is based on the comparisonof actual project performance against the planned project performance.A challenge for project managers is that they rarely know what deﬁnesan acceptable variance. It is best to acknowledge that projects willnot be completed as planned. For example, a large IT project has beenbudgeted at $58 million, and its expected duration is eighteen months (seeFigure 2.1). In this particular case, it would be unrealistic to expect thatthe project will be complete at eighteen months and meet the $58 millionbudget. Would the project be considered a failure if it was completed innineteen months for $61 million? It all depends on the acceptable variance.Performance should be measured within the acceptable variance. It iscritical to know the range for an acceptable variance in order to effectivelymanage a project.Variance analysis, earned value management, beneﬁts cost analysis,and returns on investment are measures that can be used to determineproject performance.PROJECT MANAGEMENT MATURITY MODELMaturity models have become prevalent in last twenty years. The Soft-ware Engineering Institute (SEI) and Carnegie-Mellon developed thecapability maturity model (CMM) that is used to assess the maturityof software development processes. The model identiﬁes ﬁve levels ofprocess maturity for an organization:1. Initial (chaotic, ad hoc, heroic)—the starting point for use of a newprocess.
Project Management Maturity Model 272. Repeatable (project management, process discipline)—the processis used repeatedly.3. Deﬁned (institutionalized)—the process is deﬁned/conﬁrmed as astandard business process.4. Managed (quantiﬁed)—process management and measurementtakes place.5. Optimizing (process improvement)—process management includesdeliberate process optimization/improvement.Several maturity models have recently been developed that are similarto the CMM model used to assess the maturity of project management.The Project Management Institute led the way with their model calledOrganizational Project Management Maturity Model, or OPM3®.2Organizations need to understand that in order to realize a largerROI for their project management initiatives, they must pursue higherlevels of maturity in models like OPM3. An organization must utilize thebest tools and techniques available to realize their beneﬁts. These sametools and techniques must be used in order to achieve higher levels ofmaturity.InitialAnalyze your organization’s project management practices. Table 2.2identiﬁes the tasks and activities that a project manager performs onprojects. Use this to determine what is being done at each of the projectmanagement lifecycle processes.RepeatableAssess the project management practices and methodology that yourorganization utilizes as they implement project management. Are thetools used on all projects or across the organization? Identify the tools ortechniques used to:• Select a project• Plan a project• Develop the project• Monitor and control the project• Close the project
28 THE PROJECT MANAGEMENT LIFECYCLETable 2.2 Tasks and Activities of the Project Management Lifecycle• Initiating◦ Conduct project selection and the feasibility of new products or services◦ Identify key stakeholders and gain buy-in and requirements◦ Deﬁne the scope of the project◦ Develop the project charter◦ Identify and document high-level risks, assumptions, and constraints• Planning◦ Identify key project team members and deﬁne roles and responsibilities◦ Create the WBS and develop cost, schedule, resource, quality, andprocurement plans◦ Identify project risks to deﬁne risk strategies and the risk plan◦ Obtain project plan approval and kick-off◦ Deﬁne and record detailed project requirements, constraints, andassumptions◦ Develop the change management plan• Executing◦ Manage resources◦ Execute the tasks◦ Align stakeholders and team members◦ Improve team performance◦ Implement a quality management plan◦ Implement change management◦ Obtain project resources in accordance with procurement plan◦ Minimize risks• Monitoring and Control◦ Measure project performance◦ Verify and manage changes◦ Monitor all risks◦ Ensure deliverables conform to quality standards• Closing◦ Formalize and obtain ﬁnal acceptance◦ Identify, document, and communicate lessons learned◦ Archive project records◦ Obtain ﬁnancial, legal, and administrative closure◦ Release all project resources◦ Create and distribute ﬁnal project report◦ Measure customer satisfaction
Final Thoughts 29DeﬁnedThe following questions will be helpful in assessing whether your organi-zation has properly deﬁned practices for project management.• Have policies and procedures using the tools and techniques identi-ﬁed in Level 2 been developed?• Is there training and information in regard to how to use these toolsand techniques?ManagedThe use of measurement tools for project performance must be utilized todevelop baseline data for project performance. Variance management isused to quantify project performance. Organizations must deﬁne what anacceptable variance is for a project. The variance is how project successwill be determined.Conﬁguration management is necessary to ensure that the expec-tations of stakeholders have been met. Stakeholder satisfaction is achallenge because it cannot be measured in time or cost units. Qualitativedata is valuable as stakeholder satisfaction is identiﬁed.OptimizedBest practices must be identiﬁed in order for an organization to contin-uously improve project management. The ROI impact study will be theultimate evaluation tool to compile the various qualitative and quantita-tive data necessary to ensure that the best project management tools andtechniques are used in order to maximum ROI.FINAL THOUGHTSThis chapter provided an overview to the project management process.Many tools are available for use throughout the project managementlifecycle. Project management can provide a guided methodology throughthe project. Many project managers take an ad hoc approach to projectmanagement, and too many organizations do not use the tools andprocesses introduced in this chapter. Learning these processes in greaterdetail is necessary for organizations and individuals that work on projects.This effort will require a substantial investment that will improve an
30 THE PROJECT MANAGEMENT LIFECYCLEorganization’s project performance. The ROI approach in this book isused to ensure that the right tools and processes are used.Finally, this chapter identiﬁes a ‘‘maturing’’ process that an organiza-tion goes through as it implements project management. Understandinghow an organization matures from an ad hoc approach to project man-agement to becoming a best-in-class project management organization isquite a journey. When properly implemented, an organization can realizethe full value of its investment into project management.
32 ROI METHODOLOGY BASICSLevel Measurement Focus Typical Measures0: Inputs and Indicators Types of projectsNumber of projectsNumber of peopleHours of involvementCost of projectsInputs into the project,including indicatorsrepresenting project’s scope1: Reaction and PerceivedValueRelevance ImportanceUsefulness AppropriatenessFairness MotivationalReaction to the project,including perceived valueof the project2: Learning and Confidence SkillsKnowledgeCapacityCompetenciesConfidencesContactsLearning to use the projectcontent, materials, andsystem, including theconfidence to use whatwas learned3: Application andImplementationUse of project content,materials ,and system inthe work environmentincluding progress withimplementationExtent of useTask completionFrequency of useActions completedSuccess with useBarriers to useEnablers to use4: Impact and Consequences The consequences of theuse of the project content,materials, and systemexpressed as businessimpact measuresProductivityRevenueQualityTimeEfficiencyCustomer satisfactionEmployee engagement5: ROI Comparison of project’smonetary benefits fromproject to project costsBenefits/costs ratio (BCR)RO I(%)Payback periodFigure 3.1 Types and levels of data.Reaction and Perceived Value (Level 1) marks the beginning of theproject’s value stream. Reaction data capture the degree to which theparticipants involved in the project, including the stakeholders, reactfavorably or unfavorably. The key is to capture the measures that reﬂectthe content of the project, focusing on issues such as usefulness, relevance,importance, and appropriateness. Data at this level provide the ﬁrst signthat project success may be achievable. These data also present project
Types of Data 33leaders with information they need to make adjustments to help ensurepositive results.The next level is Learning and Conﬁdence (Level 2). For everyprocess or project there is a learning component. For some—such asprojects for new technology, new systems, new competencies, and newprocesses—this component is substantial. For others, such as a new policyor new procedure, learning may be a small part of the process but is stillnecessary to ensure successful execution. In either case, measurementof learning is essential to success. Measures at this level focus on skills,knowledge, capacity, competencies, conﬁdence, and networking contacts.Application and Implementation (Level 3) measures the extent towhich the project is properly applied and implemented. Effective imple-mentation is a must if bottom-line value is the goal. This is one of themost important data categories, and most breakdowns occur at this level.Research has consistently shown that in almost half of all projects, par-ticipants and users are not doing what they should to make it successful.At this level, data collection involves measures—such as the extent ofuse of information, task completion, frequency of use of skills, successwith use, and actions completed—as well as barriers and enablers tosuccessful application. This level provides a picture of how well the sys-tem supports the successful transfer of knowledge and skills for projectimplementation.Level 4, Impact and Consequences, is important for understanding thebusiness consequences of the project. Here, data are collected that attractthe attention of the sponsor and other executives. This level shows theoutput, productivity, revenue, quality, time, cost, efﬁciencies, and levelof customer satisfaction connected with the project. For some, this levelreﬂects the ultimate reason the project exists: to show the impact withinthe organization on various groups and systems. Without this level ofdata, they assert, there is no success. When this level of measurementis achieved, it is necessary to isolate the effects of the project on thespeciﬁc measures. Without this extra step, alignment with the businesscannot occur.The ROI (Level 5) is calculated next. This shows the monetary beneﬁtsof the impact measures compared with the cost of the project. This valueis typically stated in terms of either a beneﬁts/costs ratio, the ROI as apercentage, or the payback period. This level of measurement requirestwo important steps: ﬁrst, the impact data (Level 4) must be converted tomonetary values; second, the cost of the project must be captured.
34 ROI METHODOLOGY BASICSAlong with the ﬁve levels of results and the initial level of activity(Level 0), there is a sixth type of data—not a sixth level—developedthrough this methodology. This consists of the intangible beneﬁts—thosebeneﬁts that are not converted to money but nonetheless constituteimportant measures of success.THE INITIAL ANALYSISResearch suggests that the number 1 reason for project failure is theproject’s lack of alignment with the business. The ﬁrst opportunity toobtain business alignment is in the initial analysis. Several steps aretaken to make sure that the project is absolutely necessary. As shownin Figure 3.2, this is the beginning of the complete, sequential modelrepresenting the ROI methodology. The ﬁrst step in this analysis exam-ines the potential payoff of solving a problem or taking advantage of anThe ROI MethodologyStart Here End HerePayoffneeds5 ROI objectives ROIBusinessneeds 4 Impact objectives ImpactJob performanceneeds3 Application objectives ApplicationInitialAnalysisMeasurementandEvaluationLearningneeds21Learning objectives LearningPreferenceneeds1Reaction objectivesReactionBusiness Alignmentand ForecastingThe ROI Process ModelPROJECT54321Figure 3.2 Business Alignment model.
The Initial Analysis 35opportunity. Is this a problem worth solving, or is the project worthyof implementation? For some situations the answer is obvious: Yes, theproject is worthy because of its critical nature, its relevance to the issueat hand, or its effectiveness in tackling a major problem affecting theorganization. A serious customer service problem, for example, is oneworth pursuing.The next step is to ensure that the project is connected to one or morebusiness measures. The measures that must improve as a reﬂection ofthe overall success of the project are deﬁned. Sometimes the measure isobvious; at other times it is not.Next, the job performance needs are examined with the question ‘‘Whatmust change on the job to inﬂuence the business measures previouslydeﬁned?’’ This step aligns the project with the business and may involvea series of analytical tools to solve the problem, analyze the cause ofthe problem, and ensure that the project is connected with businessimprovement in some way. This appears to be quite complex, but it isa simple approach. A series of questions helps: What is keeping thebusiness measure from being where it needs to be? If it is a problem, whatis its cause? If it is an opportunity, what is hindering it from moving inthe right direction? This step is critical because it provides the link to theproject solution.Once job performance needs have been determined, the learning needis examined by asking: What speciﬁc skills, knowledge, or perceptionsmust change or improve so that job performance can change? Everysolution involves a learning component, and this step deﬁnes what theparticipants or users must know to make the project successful. Theneeded knowledge may be as simple as understanding a policy, or be ascomplicated as learning many new competencies.The ﬁnal step is identifying the structure of the solution. How bestcan the information be presented to ensure that the necessary knowledgewill be acquired and job performance will change to solve the businessproblem? This level of analysis involves issues surrounding the scope,timing, structure, method, and budget for project implementation anddelivery.Collectively, these levels clearly deﬁne the issues that led to initiationof the project. When these preliminary steps are completed, the projectcan be positioned to achieve its intended results.Understanding the need for a project is critical to positioning thatproject for success. Positioning a project requires the development of
36 ROI METHODOLOGY BASICSclear, speciﬁc objectives that are communicated to all stakeholders. Objec-tives should be developed for each level of need and should deﬁne success,answering the question ‘‘How will we know the need has been met?’’ Ifthe criteria of success are not communicated early and often, processparticipants will go through the motions, with little change resulting.Developing detailed objectives with clear measures of success will posi-tion the project to achieve its ultimate objective. More detail on the initialanalysis and objective are contained in the next chapter.Before a project is launched, forecasting the outcomes is importantto ensure that adjustments can be made or alternative solutions can beinvestigated. This forecast can be simple, relying on the individuals closestto the situation, or it can be a more detailed analysis of the situationand expected outcome. Recently, forecasting has become a critical toolfor project sponsors who need evidence that the project will be successfulbefore they are willing to plunge into a funding stream for it. Because ofits importance, forecasting is the focus of Chapter 12.THE ROI PROCESS MODELThe next challenge for many project leaders is to collect a variety of dataalong a chain of impact that shows the project’s value. Figure 3.3 displaysthe sequential steps that lead to data categorized by the ﬁve levels ofresults.1This ﬁgure shows the ROI methodology, a step-by-step processbeginning with the objectives and concluding with reporting of data. Themodel assumes that proper analysis is conducted to deﬁne need beforethe steps are taken.Planning the EvaluationThe ﬁrst phase of the ROI methodology is evaluation planning. This phaseinvolves several procedures, including understanding the purpose of theevaluation, determining the feasibility of the planned approach, planningdata collection and analysis, and outlining the details of the project.Evaluation PurposeEvaluations are conducted for a variety of reasons:• To improve the quality of projects and outcomes• To determine whether a project has accomplished its objectives
38 ROI METHODOLOGY BASICS• To identify strengths and weaknesses in the process• To enable the cost-beneﬁt analysis• To assist in the development of marketing projects in the future• To determine whether the project was the appropriate solution• To establish priorities for project fundingThe purposes of the evaluation should be considered prior to develop-ing the evaluation plan because the purposes will often determine thescope of the evaluation, the types of instruments used, and the type ofdata collected. As with any project, understanding the purpose of theevaluation will give it focus, and will also help gain support from others.FeasibilityAn important consideration in planning the ROI impact study is thedetermination of the levels at which the project will be evaluated. Someevaluations will stop at Level 3, where a detailed report will determinethe extent to which participants are using what they have learned. Otherswill be evaluated to Level 4, Impact, where the consequences of on-the-jobapplications are monitored and measures directly linked to the project areexamined. If the ROI calculation is needed, the evaluation will proceedto Level 5. To reach this level of measurement, two additional stepsare required: The Level 4 impact data must be converted to monetaryvalues, and the costs of the project must be captured so that the ROIcan be developed. Evaluation at Level 5 is intended for projects that areexpensive, high-proﬁle, and have a direct link to business needs.ObjectivesThe initial analysis, which deﬁnes the needs along the ﬁve levels, alsodeﬁnes the objectives at these levels. Projects need very clear direction,and the objectives provide this clarity. Objectives, when deﬁned precisely,provide the participants and other stakeholders with the direction theyneed to make the project successful. The objectives are deﬁned along thesame ﬁve levels as the needs assessment:• Reaction objectives• Learning objectives• Application and implementation objectives• Impact objectives• ROI objectives
The ROI Process Model 39These speciﬁc objectives take the mystery out of what this projectshould achieve.On occasion, the initial analysis may stop with Level 2 objectives,excluding the application and impact objectives that are needed to directthe higher levels of evaluation. If application and impact objectives arenot available, they must be developed using input from such groups asjob incumbents, analysts, project developers, subject matter experts, andon-the-job team leaders.Three simple planning documents are developed next: the data collec-tion plan, the ROI analysis plan, and the project plan. These documentsshould be completed during evaluation planning and before the evalu-ation project is implemented—ideally, before the project is designed ordeveloped. Appropriate up-front attention will save time later, when dataare actually collected.Data Collection PlanFigure 3.4 shows a completed data collection plan for a project to imple-ment a telecommuting system in an insurance company.This document provides a place for the major elements and issuesregarding data collection. Broad objectives are appropriate for planning.Speciﬁc, detailed objectives are developed later, before the project isdesigned. Entries in the Measures column deﬁne the speciﬁc measure;entries in the Method/Instruments column describe the technique used tocollect the data; in the Sources column, the source of the data is identiﬁed;and the Timing column indicates when the data are collected.ROI Analysis PlanFigure 3.5 shows a completed ROI analysis plan for the telecommutingproject. This planning document captures information on key items thatare necessary to develop the actual ROI calculation. In the ﬁrst column,signiﬁcant data items are listed. Although these are usually Level 4impact data, in some cases this column includes Level 3 items. Theseitems will be used in the ROI analysis.The method employed to isolate the project’s effects is listed next toeach data item in the second column. The method of converting data tomonetary values is included in the third column. The cost categories thatwill be captured for the project are outlined in the next column. Normally,the cost categories will be consistent from one project to another. Theintangible beneﬁts expected from the project are outlined in the ﬁfth
The ROI Process Model 43column. This list is generated from discussions about the project withsponsors and subject matter experts. Communication targets are outlinedin the sixth column. Finally, other issues or events that might inﬂuenceproject implementation and its outputs are highlighted in the last column.Typical items include the capability of participants, the degree of accessto data sources, and unique data analysis issues.The ROI analysis plan, when combined with the data collection plan,provides detailed information for calculating the ROI, illustrating howthe evaluation will develop from beginning to end.Project Evaluation PlanThe ﬁnal plan developed for the evaluation planning phase is a projectevaluation plan, as shown in Figure 3.6. A project plan consists of adescription of the project and the timeline of the project evaluation, fromthe planning of the study through the ﬁnal communication of the results.This plan becomes an operational tool to keep the project evaluationon track.Collectively, the three planning documents provide the direction nec-essary for the ROI impact study. Most of the decisions regarding theprocess are made as these planning tools are developed. The remainder ofthe project becomes a methodical, systematic process of implementing theplan. This is a crucial step in the ROI methodology, in which valuable timeallocated to this process in the beginning will save precious time later.Project Evaluation PlanF M A M J JDecision to conduct ROI studyEvaluation planning completeInstruments designedData collectedData tabulation/Preliminary summaryAnalysis conducteReport writtenReport printedResults communicatedImprovements initiatedImplementation completeFigure 3.6 Project evaluation plan.
44 ROI METHODOLOGY BASICSCollecting DataData collection is central to the ROI methodology. Both hard data (rep-resenting output, quality, cost, and time) and soft data (including jobsatisfaction and customer satisfaction) are collected. Data are collectedusing a variety of methods, including• Surveys• Questionnaires• Tests• Observations• Interviews• Focus groups• Action plans• Performance contracts• Business performance monitoringThe important challenge in data collection is to select the method ormethods appropriate for the setting and the speciﬁc project, within thetime and budget constraints of the organization. Data collection methodsare covered in more detail in Chapters 5 through 7.Isolating the Effects of the ProjectAn often overlooked issue in evaluations is the process of isolating theeffects of the project. In this step, speciﬁc strategies are explored thatdetermine the amount of output performance directly related to theproject. This step is essential because many factors will inﬂuence perfor-mance data. The speciﬁc strategies of this step pinpoint the amount ofimprovement directly related to the project, resulting in increased accu-racy and credibility of ROI calculations. The following techniques havebeen used by organizations to tackle this important issue:• Control groups• Trend line analysis• Forecasting models• Participant estimates• Managers’ estimates• Senior management estimates• Experts’ input• Customer input
The ROI Process Model 45Collectively, these techniques provide a comprehensive set of tools tohandle the important and critical issue of isolating the effects of projects.Chapter 8 is devoted to this important step in the ROI methodology.Converting Data to Monetary ValuesTo calculate the return on investment, Level 4 impact data are convertedto monetary values and compared with project costs. This requires thata value be placed on each unit of data connected with the project.Many techniques are available to convert data to monetary values. Thespeciﬁc technique selected depends on the type of data and the situation.The techniques include• Output data• Cost of quality• Time savings converted to participants’ wage and employee beneﬁts• Historical costs• Internal and external experts• External databases• Participant estimates• Manager estimates• Soft measures mathematically linked to other measuresThis step in the ROI model is important and absolutely necessary indetermining the monetary beneﬁts of a project. The process is challenging,particularly with soft data, but can be methodically accomplished usingone or more of these strategies. Because of its importance, this step in theROI methodology is described in detail in Chapter 9.Identifying Intangible BeneﬁtsIn addition to tangible, monetary beneﬁts, intangible beneﬁts—those notconverted to money—are identiﬁed for most projects. Intangible beneﬁtsinclude items such as:• Increased employee engagement• Increased brand awareness• Improved networking• Improved customer service
46 ROI METHODOLOGY BASICS• Fewer complaints• Reduced conﬂictDuring data analysis, every attempt is made to convert all data tomonetary values. All hard data—such as output, quality, and time—areconverted to monetary values. The conversion of soft data is attempted foreach data item. However, if the process used for conversion is too subjec-tive or inaccurate, and the resulting values lose credibility in the process,then the data are listed as an intangible beneﬁt with the appropriateexplanation. For some projects, intangible, nonmonetary beneﬁts areextremely valuable, and often carry as much inﬂuence as the hard dataitems. Chapter 10 is devoted to the nonmonetary, intangible beneﬁts.Tabulating Project CostsAn important part of the ROI equation is the calculation of project costs.Tabulating the costs involves monitoring or developing all the relatedcosts of the project targeted for the ROI calculation. Among the costcomponents to be included are• Initial analysis costs• Cost to design and develop the project• Cost of all project materials• Costs for the project team• Cost of the facilities for the project• Travel, lodging, and meal costs for the participants and teammembers• Participants’ salaries (including employee beneﬁts)• Administrative and overhead costs, allocated in some convenientway• Evaluation costsThe conservative approach is to include all these costs so that the totalis fully loaded. Chapter 11 includes this step in the ROI methodology.Calculating the Return on InvestmentThe return on investment is calculated using the program beneﬁts andcosts. The beneﬁts/costs ratio (BCR) is calculated as the project beneﬁts
Operating Standards and Philosophy 47divided by the project costs. In formula form,BCR =Project BeneﬁtsProject CostsThe return on investment is based on the net beneﬁts divided by projectcosts. The net beneﬁts are calculated as the project beneﬁts minus theproject costs. In formula form, the ROI becomesROI (%) =Net Project BeneﬁtsProject Costs× 100This is the same basic formula used in evaluating other investments, inwhich the ROI is traditionally reported as earnings divided by investment.Chapter 11 provides more detail.Reporting ResultsThe ﬁnal step in the ROI process model is reporting, a critical step thatis often deﬁcient in the degree of attention and planning required toensure its success. The reporting step involves developing appropriateinformation in impact studies and other brief reports. At the heart ofthis step are the different techniques used to communicate to a widevariety of target audiences. In most ROI studies, several audiences areinterested in and need the information. Careful planning to match thecommunication method with the audience is essential to ensure that themessage is understood and that appropriate actions follow. Chapter 13 isdevoted to this critical step in the ROI process.OPERATING STANDARDS AND PHILOSOPHYTo ensure consistency and replication of impact studies, operating stan-dards must be developed and applied as the process model is used todevelop ROI studies. The results of the study must stand alone and mustnot vary with the individual who is conducting the study. The operatingstandards detail how each step and issue of the process will be handled.Table 3.1 shows the twelve guiding principles that form the basis for theoperating standards.The guiding principles serve not only to consistently address each step,but also to provide a much needed conservative approach to the analysis.A conservative approach may lower the actual ROI calculation, but it willalso build credibility with the target audience.
48 ROI METHODOLOGY BASICSTable 3.1 Twelve Guiding Principles of ROI1. When conducting a higher-level evaluation, collect data at lower levels.2. When planning a higher-level evaluation, the previous level of evaluation isnot required to be comprehensive.3. When collecting and analyzing data, use only the most credible sources.4. When analyzing data, select the most conservative alternative for calcula-tions.5. Use at least one method to isolate the effects of a project.6. If no improvement data are available for a population or from a speciﬁcsource, assume that little or no improvement has occurred.7. Adjust estimates of improvement for potential errors of estimation.8. Avoid use of extreme data items and unsupported claims when calculatingROI.9. Use only the ﬁrst year of annual beneﬁts in ROI analysis of short-termsolutions.10. Fully load all costs of a solution, project, or program when analyzing ROI.11. Intangible measures are deﬁned as measures that are purposely not convertedto monetary values.12. Communicate the results of ROI methodology to all key stakeholders.IMPLEMENTING AND SUSTAINING THE PROCESSA variety of environmental issues and events will inﬂuence the successfulimplementation of the ROI methodology. These issues must be addressedearly to ensure the success of the ROI process. Speciﬁc topics or actionsinclude• A policy statement concerning results-based projects• Procedures and guidelines for different elements and techniques ofthe evaluation process• Formal meetings to develop staff skills with the ROI process• Strategies to improve management commitment to and support forthe ROI process• Mechanisms to provide technical support for questionnaire design,data analysis, and evaluation strategy• Speciﬁc techniques to place more attention on resultsThe ROI process can fail or succeed based on these implementationissues. Chapter 14 is devoted to this important topic.
Beneﬁts of This Approach 49In addition to implementing and sustaining ROI use, the process mustundergo periodic review. An annual review is recommended to determinethe extent to which the process is adding value.BENEFITS OF THIS APPROACHNow for the good news: The methodology presented in this book has beenused consistently and routinely by thousands of organizations in the pastdecade. Much has been learned about the success of this methodology andwhat it can bring to the organizations using it.Aligning with BusinessThe ROI methodology ensures project alignment with the business,enforced in three steps. First, even before the project is initiated, themethodology ensures that alignment is achieved up front, at the time theproject is validated as the appropriate solution. Second, by requiring spe-ciﬁc, clearly deﬁned objectives at the impact level, the project focuses onbusiness impact over its course, in essence driving the business measureby its design, delivery, and implementation. Third, in the follow-up data,when the business measures may have changed or improved, a method isused to isolate the effects of the project on that data, consequently prov-ing the connection to that business measure, i.e., showing the amountof improvement directly connected to the project and ensuring there isbusiness alignment.Validating the Value PropositionIn reality, most projects are undertaken to deliver value. As described inthis chapter, the deﬁnition of value may on occasion be unclear, or may notbe what a project’s various sponsors, organizers, and stakeholders desire.Consequently, there are often value shifts. Once the values are ﬁnallydetermined, the value proposition is detailed. The ROI methodology willforecast the value in advance, and if the value has been delivered, itveriﬁes the value proposition agreed to by the appropriate parties.Improving ProcessesThis is a process improvement tool by design and by practice. It collectsdata to evaluate how things are—or are not—working. When things
50 ROI METHODOLOGY BASICSare not where they should be—as when projects are not proceeding aseffectively as expected—data are available to indicate what must bechanged to make the project more effective. When things are workingwell, data are available to show what else could be done to make thembetter. Thus, this is a process improvement system designed to providefeedback to make changes. As a project is conducted, the results arecollected and feedback is provided to the various stakeholders for speciﬁcactions for improvement. These changes drive the project to better results,which are then measured while the process continues. This continuousfeedback cycle is critical to process improvement and is inherent in theROI methodology approach.Enhancing the Image; Building RespectProject managers are criticized for being unable to deliver what isexpected. For this, their image suffers. The ROI methodology is oneway to help build the respect a function or profession needs.The ROI methodology can make a difference in any function whereprojects are managed. This methodology shows a connection to the bottomline and shows the value delivered to stakeholders. It removes issuesabout value and a supposed lack of contribution to the organization.Consequently, this methodology is an important part of the process ofchanging the image of the function of the organization and buildingneeded respect.Improving SupportSecuring support for projects is critical, particularly at the middle man-ager level. Many projects enjoy the support of the top-level managers whoallocated the resources to make the projects viable. Unfortunately, somemiddle-level managers may not support certain projects because theydo not see the value the projects deliver in terms the managers appre-ciate and understand. Having a methodology that shows how a projectis connected to the manager’s business goals and objectives can changethis support level. When middle managers understand that a project ishelping them meet speciﬁc performance indicators or departmental goals,they will usually support the process, or will at least resist it less. In thisway, the ROI methodology may actually improve manager support.
Final Thoughts 51Justifying or Enhancing BudgetsSome organizations have used the ROI methodology to support proposedproject budgets. Because the methodology shows the monetary valueexpected or achieved with speciﬁc projects, the data can often be leveragedinto budget requests. When a particular function is budgeted, the amountbudgeted is often in direct proportion to the value that the function adds.If little or no credible data support the contribution, the budgets are oftentrimmed—or at least not enhanced.Building a Partnership with Key ExecutivesProject managers partner with operating executives and key managersin the organization. Unfortunately, some managers may not want to bepartners. They may not want to waste time and effort on a relationshipthat does not help them succeed. They want to partner only with groupsand individuals who can add value and help them in meaningful ways.Showing the projects’ results will enhance the likelihood of building thesepartnerships, with the results providing the initial impetus for makingthe partnerships work.FINAL THOUGHTSThis chapter presents the overall approach to measuring ROI. It presentsthe different elements and steps in the ROI methodology, the standards,and the different concepts necessary to understand how ROI works, butwithout a great deal of detail. This chapter brings the methodology intofocus. Before one can accept the approach, the steps and the detail have tobe shown. This detail will be presented in the rest of the book. Chapter 4provides more detail on project alignment.
54 ACHIEVING BUSINESS ALIGNMENT WITH THE PROJECTThis standardized approach adds credibility and allows for consistentapplication so that the analysis can be replicated. A disciplined approachmaintains process efﬁciency as various tools and templates are developedand used. This initial phase of project development calls for focus andthoroughness, with little allowance for major shortcuts.Not every project should be subjected to the type of comprehensiveanalysis described in this chapter. Some needs are obvious and requirelittle analysis other than that necessary to develop the project. Additionalanalysis may be needed to conﬁrm that the project answers the perceivedneed and perhaps to ﬁne-tune the project for future application. Theamount of analysis required often depends on the expected opportunityto be gained if the project is appropriate or the negative consequencesanticipated if the project is inappropriate.When analysis is proposed, individuals may react with concern or resis-tance. Some are concerned about the potential for ‘‘paralysis by analysis,’’where requests and directives lead only to additional analyses. Thesereactions can pose a problem for an organization because analysis isnecessary to ensure that the project is appropriate. Unfortunately, anal-ysis is often misunderstood—conjuring up images of complex problems,confusing models, and a deluge of data along with complicated statisticaltechniques to ensure that all bases are covered. In reality, analysis neednot be so complicated. Simple techniques can uncover the cause of aproblem or the need for a particular project.The remainder of the chapter delves into the components of analysisthat are necessary for a solid alignment between a project and thebusiness. First, however, reviewing the model introduced in Chapter 3may be helpful. It is presented here as Figure 4.1.DETERMINING THE POTENTIAL PAYOFFThe ﬁrst step in up-front analysis is to determine the potential payoffof solving a problem or seizing an opportunity. This step begins withanswers to a few crucial questions: Is this project worth doing? Is itfeasible? What is the likelihood of a positive ROI?For projects addressing signiﬁcant problems or opportunities withhigh potential rewards, the answers are obvious. The questions may takelonger to answer for lower-proﬁle projects or those for which the expectedpayoff is less apparent. In any case, these are legitimate questions, andthe analysis can be as simple or as comprehensive as required.
Determining the Potential Payoff 55The ROI MethodologyStart Here End HerePayoffneeds5 ROI objectives ROIBusinessneeds 4 Impact objectives ImpactJob performanceneeds3 Application objectives ApplicationInitialAnalysisMeasurementandEvaluationLearningneeds21Learning objectives LearningPreferenceneeds1Reaction objectivesReactionBusiness Alignmentand ForecastingThe ROI Process ModelPROJECT54321Figure 4.1 Business Alignment model.Essentially, a project will pay off in proﬁt increases or in cost savings.Proﬁt increases are generated by projects that drive revenue, e.g., thatimprove sales, drive market share, introduce new products, open newmarkets, enhance customer service, or increase customer loyalty. Otherrevenue-generating measures include increasing membership, increasingdonations, obtaining grants, and generating tuition from new and return-ing students—all of which, after subtracting the cost of doing business,should leave a signiﬁcant proﬁt.However, most projects drive cost savings. Cost savings can comethrough cost reduction or cost avoidance. Improved quality, reduced cycletime, lowered downtime, reduced complaints, limited employee turnover,and minimized delays are all examples of cost savings.Cost avoidance projects are implemented to reduce risks, avoid prob-lems, or prevent unwanted events. Some professionals may view costavoidance as an inappropriate measure to use to determine monetary
56 ACHIEVING BUSINESS ALIGNMENT WITH THE PROJECTbeneﬁts and calculate ROI. However, if the assumptions prove correct, anavoided cost, e.g., compliance ﬁnes, can be more rewarding than reducingan actual cost. Preventing a problem is more cost-effective than waitingfor the problem to occur and then having to focus on solving it.Determining the potential payoff is the ﬁrst step in the needs analysisprocess. This step is closely related to the next one, determining thebusiness need, since the potential payoff is often based on a considerationof the business. The payoff depends on two factors: the monetary valuederived from the business measure’s improvement and the approximatecost of the project. Identifying these monetary values in detail usuallyyields a more credible forecast of what can be expected from the chosensolution. However, this step may be omitted in situations where theproblem (business need) must be resolved regardless of the cost, or if itbecomes obvious that this is a high-payoff activity.The target level of detail may also hinge on the need to secure projectfunding. If the potential funding source does not recognize the value ofthe project compared with the potential costs, more detail may be neededto provide a convincing case for funding.Knowledge of the actual payoff is not necessary if widespread agree-ment exists that the payoff from the project will be high, or if the problemin question must be resolved regardless of cost. For example, if the pro-posed project involves a safety concern, a regulatory compliance issue, ora competitive matter, a detailed analysis is not needed.Obvious versus Not-So-Obvious PayoffThe potential payoff is obvious for some projects and not so obvious forothers. Opportunities with obvious payoffs may include• Operating costs 47 percent higher than industry average• Customer satisfaction rating of 3.89 on a 10-point scale• A cost to the city of $75,000 annually for each homeless person• Noncompliance ﬁnes totaling $1.2 million, up 82 percent from lastyear• Turnover of critical talent 35 percent above benchmark ﬁgure• System downtime is twice the average of last year’s results• Very low market share in a market with few players• Safety record is among the worst in the industry• Excessive product returns: 30 percent higher than previous year
Determining the Potential Payoff 57• Excessive absenteeism in call centers: 12.3 percent, compared to 5.4percent industry average• Sexual harassment complaints per 1,000 employees are the highestin the industryEach item appears to reﬂect a serious problem that needs to beaddressed by executives, administrators, or politicians.For other projects, the issues are sometimes unclear and may arisefrom political motives or bias. These potential opportunities are associatedwith payoffs that may not be so obvious. Such opportunities may include• Become a technology leader• Become a ‘‘green’’ company• Improve leadership competencies for all managers• Improve branding for all products• Create a great place to work• Organize a business development conference• Establish a project management ofﬁce• Provide job training for unemployed workers• Implement lean Six Sigma• Train all team leaders on crucial conversations• Provide training on sexual harassment awareness for all associates• Develop an ‘‘open book’’ company• Implement the same workout process that GE has used• Implement a CRM system• Convert to cloud computing• Implement a transformation program involving all employees• Implement a career advancement program• Create a wellness and ﬁtness centerWith each of these opportunities, there is a need for more speciﬁc detailregarding the measure. For example, if the opportunity is to become a‘‘green’’ company, one might ask: What is a green company? How willwe know when we’re green? How is green deﬁned? Projects with not-so-obvious payoffs require greater analysis than those with clearly deﬁnedoutcomes.The potential payoff establishes the fundamental reason for pursuingnew or enhanced projects. But the payoff—whether obvious or not—is notthe only reason for moving forward with a project. The cost of a problem
58 ACHIEVING BUSINESS ALIGNMENT WITH THE PROJECTis another factor. If the cost is excessive, it should be addressed. If not,then a decision must be made as to whether the problem is worth solving.The Cost of a ProblemSometimes projects are undertaken to solve a problem. Problems areexpensive and their solution can result in high returns, especially whenthe solution is inexpensive. To determine the cost of the problem, itspotential consequences must be examined and converted to monetaryvalues. Problems may encompass time, quality, productivity, and teamor customer issues. All of these factors must be converted to monetaryvalues if the cost of the problem is to be determined. Inventory shortagesare often directly associated with the cost of the inventory as well aswith the cost of carrying the inventory. Time can easily be translated intomoney by calculating the fully loaded cost of an individual’s time spent onunproductive tasks. Calculating the time for completing a project, task, orcycle involves measures that can be converted to money. Errors, mistakes,waste, delays, and bottlenecks can often be converted to money because oftheir consequences. Productivity problems and inefﬁciencies, equipmentdamage, and equipment underuse are other items whose conversion tomonetary value is straightforward.In examining costs, considering all the costs and their implications iscrucial. For example, the full cost of an accident includes not only thecost of lost workdays and medical expenses, but their effects on insurancepremiums, the time required for investigations, damage to equipment,and the time spent by all involved employees addressing the accident. Thecost of a customer complaint includes not only the cost of the time spentresolving the complaint, but also the value of the item or service that hasto be adjusted because of the complaint. The costliest consequence of acustomer complaint is the price to the company of lost future business andgoodwill from the complaining customer and from potential customerswho learn of the complaint.Placing a monetary value on a problem helps in determining if theproblem’s resolutions are economically feasible. The same applies toopportunities.The Value of an OpportunitySometimes projects are undertaken to pursue an opportunity. Just asthe cost of a problem can be easily tabulated in most situations, the
Determining Business Needs 59value of an opportunity can also be calculated. Examples of opportu-nities include implementing a new process, exploring new technology,increasing research and development efforts, and upgrading the work-force to create a more competitive environment. In these situations aproblem may not exist, but an opportunity to get ahead of the competitionor to prevent a problem’s occurrence by taking immediate action does.Assigning a proper value to this opportunity requires considering whatmay happen if the project is not pursued or acknowledging the windfallthat might be realized if the opportunity is seized. The value is determinedby following the different possible scenarios to convert speciﬁc businessimpact measures to money. The difﬁculty in this process is conducting acredible analysis. Forecasting the value of an opportunity entails manyassumptions compared with calculating the value of a known outcome.To Forecast or Not to Forecast?The need to seek and assign value to opportunities leads to an importantdecision: to forecast or not to forecast ROI. If the stakes are high andsupport for the project is not in place, a detailed forecast may be the onlyway to gain the needed support and funding for the project or to informthe choice between multiple potential projects. In developing the forecast,the rigor of the analysis is an issue. In some cases, an informal forecast issufﬁcient, given certain assumptions about alternative outcome scenarios.In other cases, a detailed forecast is needed that uses data collected froma variety of experts, previous studies from another project, or perhapsmore sophisticated analysis. Chapter 12 provides techniques useful fordeveloping forecasts.When the potential payoff, including its ﬁnancial value, has beendetermined, the next step is to clarify the business needs.DETERMINING BUSINESS NEEDSDetermining the business needs requires the identiﬁcation of speciﬁcmeasures so that the business situation can be clearly assessed. The con-cept of business needs refers to gains in productivity, quality, efﬁciency,time, and cost. This is true for the private sector as well as in government,nonproﬁt, and academic organizations.A business need is represented by a business measure. Any process,item, or perception can be measured, and such measurement is critical
60 ACHIEVING BUSINESS ALIGNMENT WITH THE PROJECTto this level of analysis. If the project focuses on solving a problem,preventing a problem, or seizing an opportunity, the measures are usuallyidentiﬁable. The important point is that the measures are present in thesystem, ready to be captured for this level of analysis. The challenge is todeﬁne the measures and to ﬁnd them economically and swiftly.Hard Data MeasuresTo focus on the desired measures, distinguishing between hard data andsoft data may be helpful. Hard data are primary measures of improvementpresented in the form of rational, undisputed facts that are usuallygathered within functional areas throughout an organization. These arethe most desirable type of data because they are easy to quantify andare easily converted to monetary values. The fundamental criteria forgauging the effectiveness of an organization are hard data items such asrevenue, productivity, and proﬁtability, as well as measures that quantifysuch processes as cost control and quality assurance.Hard data are objective and credible measures of an organization’s per-formance. Hard data can usually be grouped in four categories, as shownin Table 4.1. These categories—output, quality, costs, and time—aretypical performance measures in any organization.Hard data from a particular project involve improvements in the outputof the work unit, section, department, division, or entire organization.Every organization, regardless of the type, must have basic measuresof output, such as number of patients treated, students graduated, tonsproduced, or packages shipped. Since these values are monitored, changescan easily be measured by comparing ‘‘before’’ and ‘‘after’’ outputs.Quality is a very important hard data category. If quality is a majorpriority for the organization, processes are likely in place to measure andmonitor quality. The rising prominence of quality improvement processes(such as Total Quality Management, Continuous Quality Improvement,and Six Sigma) has contributed to the tremendous recent successesin pinpointing the proper quality measures—and assigning monetaryvalues to them.Cost is another important hard data category. Many projects aredesigned to lower, control, or eliminate the cost of a speciﬁc process oractivity. Achieving cost targets has an immediate effect on the bottomline. Some organizations focus narrowly on cost reduction. For example,consider Wal-Mart, whose tagline is ‘‘Always low prices. Always.’’ All
Determining Business Needs 61Table 4.1 Examples of Hard DataOutput Quality Costs TimeUnits produced Failure rates Shelter costs Cycle timeTons manufactured Dropout rates Treatment costs EquipmentdowntimeProduct returnsItems assembled Scrap Budget variances OvertimeMoney collected Waste Unit costs On-time shipmentsItems sold Rejects Cost by account Time to projectcompletionNew accountsgeneratedError rates Variable costsAccidents Processing timeForms processed Rework Fixed costsLoans approved Shortages Overhead cost Supervisory timeInventory turnover Product defects Operating costs Time to proﬁciencyPatients served Deviation fromstandardLearning timeApplicationsprocessedProduct failures Accident costs Adherence toschedulesStudentsgraduatedInventoryadjustmentsProgram costs Repair timeTasks completed Sales expense EfﬁciencyOutput per hour Incidents Work stoppagesProductivity CompliancediscrepanciesOrder responseWork backlog Agency ﬁnes Late reportingShipments Lost-time daysProject completionslevels of the organization are dedicated to lowering costs on processes andproducts and passing the savings along to customers.Time is a critical measure in any organization. Some organizationsgauge their performance almost exclusively in relation to time. Whenasked what business FedEx is in, company executives say, ‘‘We engineertime.’’
62 ACHIEVING BUSINESS ALIGNMENT WITH THE PROJECTSoft Data MeasuresSoft data are probably the most familiar measures of an organiza-tion’s effectiveness, yet their collection can present a challenge. Valuesrepresenting attitude, motivation, and satisfaction are examples of softdata. Soft data are more difﬁcult to gather and analyze, and therefore,they are used when hard data are not available or to supplement harddata. Soft data are also more difﬁcult to convert to monetary values, a pro-cess requiring subjective methods. They are less objective as performancemeasurements and are usually behavior related, yet organizations placegreat emphasis on them. Improvements in these measures representimportant business needs, but many organizations omit them from theROI equation because they are soft values. However, they can contributeto economic value to the same extent as hard data measures. The key isnot to focus too much on the hard versus soft data distinction. A betterapproach is to consider data as tangible or intangible. Table 4.2 showscommon examples of soft data by category.Tangible versus Intangible Beneﬁts: A Better ApproachA challenge with regard to soft versus hard data is converting softmeasures to monetary values. The key to this problem is to rememberthat, ultimately, all roads lead to hard data. Although creativity may becategorized as a form of soft data, a creative workplace can develop newproducts or new patents, which leads to greater revenue—clearly a harddata measure. Although it is possible to convert the measures listed inTable 4.2 to monetary amounts, it is often more realistic and practical toleave them in nonmonetary form. This decision is based on considerationsof credibility and the cost of the conversion. According to the standardsof the ROI methodology, an intangible measure is deﬁned as a measurethat is intentionally not converted to money. If a soft data measure canbe converted to a monetary amount credibly using minimal resources, itis considered tangible, reported as a monetary value, and incorporated inthe ROI calculation. If a data item cannot be converted to money crediblywith minimal resources, it is listed as an intangible measure. Therefore,in deﬁning business needs, the key difference between measures is notwhether they represent hard or soft data, but whether they are tangibleor intangible. In either case, they are important contributions toward thedesired payoff and important business impact data.
Determining Business Needs 63Table 4.2 Examples of Soft DataWork HabitsExcessive breaksTardinessVisits to the dispensaryViolations of safety rulesCommunication breakdownsWork Climate/SatisfactionGrievancesDiscrimination chargesEmployee complaintsJob satisfactionOrganization commitmentEmployee engagementEmployee loyaltyIntent to leaveStressInitiative/InnovationCreativityInnovationNew ideasSuggestionsNew products and servicesTrademarksCopyrights and patentsProcess improvementsPartnerships/alliancesCustomer ServiceCustomer complaintsCustomer satisfactionCustomer dissatisfactionCustomer impressionsCustomer loyaltyCustomer retentionLost customersEmployee Development/AdvancementPromotionsCapabilityIntellectual capitalRequests for transferPerformance appraisal ratingsReadinessNetworkingImageBrand awarenessReputationLeadershipSocial responsibilityEnvironmental friendlinessSocial consciousnessDiversityExternal awardsBusiness Data SourcesThe sources of business data, whether tangible or intangible, are diverse.The data come from routine reporting systems in the organization. Inmany situations, these items have led to the need for the project. A vastarray of documents, systems, databases, and reports can be used to selectthe speciﬁc measure or measures to be monitored throughout the project.Impact data sources include quality reports, service records, suggestionsystems, and employee engagement data.Some project planners and project team members assume that corpo-rate data sources are scarce because the data are not readily available tothem. However, data can usually be located by investing a small amount
64 ACHIEVING BUSINESS ALIGNMENT WITH THE PROJECTof time. Rarely do new data collection systems or processes need to bedeveloped in order to identify data representing the business needs of anorganization.In searching for the proper measures to connect to the project and toidentify business needs, it is helpful to consider all possible measures thatcould be inﬂuenced. Sometimes, collateral measures move in harmonywith the project. For example, efforts to improve safety may also improveproductivity and increase job satisfaction. Weighing adverse impacts oncertain measures may also help. For example, when cycle times arereduced, quality may suffer; when sales increase, customer satisfactionmay deteriorate. Finally, project team members must anticipate unin-tended consequences and capture them as other data items that might beconnected to or inﬂuenced by the project.In the process of settling on the precise business measures for theproject, it is useful to examine various ‘‘what if’’ scenarios. If the orga-nization does nothing, the potential consequences of inaction should bemade clear. The following questions may help in understanding theconsequences of inaction:• Will the situation deteriorate?• Will operational problems surface?• Will budgets be affected?• Will we lose inﬂuence or support?Answers to these questions can help the organization settle on a preciseset of measures and can provide a hint of the extent to which the measuresmay change as a result of the project.DETERMINING PERFORMANCE NEEDSThe next step in the needs analysis is to understand what led to thebusiness need. If the proposed project addresses a problem, this stepfocuses on the cause of the problem. If the project makes use of anopportunity, this step focuses on what is inhibiting the organization fromtaking advantage of that opportunity. This is the performance needs.Analysis TechniquesUncovering the causes of the problem or the inhibitors to success requiresa variety of analytical techniques. These techniques—such as problem
Determining Learning Needs 65Table 4.3 Analysis TechniquesStatistical Process ControlBrainstormingProblem AnalysisCause-and-Effect DiagramForce-ﬁeld AnalysisMind MappingAfﬁnity DiagramsSimulationsDiagnostic InstrumentsFocus GroupsProbing InterviewsJob Satisfaction SurveysEngagement SurveysExit InterviewsExit SurveysNominal Group Techniqueanalysis, nominal group technique, force-ﬁeld analysis, and just plainbrainstorming—are used to clarify job performance needs. Table 4.3 listsa few of the analysis techniques. The technique that is used will dependon the organizational setting, the apparent depth of the problem, and thebudget allocated to such analysis. Multiple techniques can be used sinceperformance may be lacking for a number of reasons.A Sensible ApproachAnalysis takes time and adds to a project’s cost. Examining records,researching databases, and observing individuals can provide importantdata, but a more cost-effective approach might include employing internaland/or external experts to help analyze the problem. Performance needscan vary considerably and may include ineffective behavior, a dysfunc-tional work climate, inadequate systems, a disconnected process ﬂow,improper procedures, a nonsupportive culture, outdated technology, anda nonaccommodating environment, to name a few. When needs vary andwith many techniques to choose from, the opportunity exists for overanal-ysis and excessive costs. Consequently, a sensible approach is called for.DETERMINING LEARNING NEEDSThe solution to performance needs uncovered in the previous step oftenrequires a learning component—such as participants and team memberslearning how to perform a task differently, or learning how to use aprocess or system. In some cases learning is the principal solution, asin competency or capability development, major technology change, andsystem installations. For other projects, learning is a minor aspect of the
66 ACHIEVING BUSINESS ALIGNMENT WITH THE PROJECTsolution and may involve simply understanding the process, procedure,or policy. For example, in the implementation of a new ethics policy foran organization, the learning component requires understanding how thepolicy works as well as the participants’ role in the policy. In short, alearning solution is not always needed, but all solutions have a learningcomponent.A variety of approaches are available for measuring speciﬁc learningneeds. Often, multiple tasks and jobs are involved in a project and shouldbe addressed separately. Sometimes the least effective way to identifythe skills and knowledge that are needed is to ask the participantsinvolved in implementing the project. They may not be clear on what isneeded and may not know enough to provide adequate input. One of themost useful ways to determine learning needs is to ask the individualswho understand the process. They can best determine what skills andknowledge are necessary to address the performance issues that havebeen identiﬁed. This may be the appropriate time to ﬁnd out the extentto which the knowledge and skills already exist.Job and task analysis is effective when a new job is created or whenan existing job description changes signiﬁcantly. As jobs are redesignedand the new tasks must be identiﬁed, this type of analysis offers asystematic way of detailing the job and task. Essentially, a job analysis isthe collection and evaluation of work-related information. A task analysisidentiﬁes the speciﬁc knowledge, skills, tools, and conditions necessary tothe performance of a particular job.Observation of current practices and procedures in an organizationmay be necessary as the project is implemented. This can often indicatethe level of capability and help to identify the correct procedures. Obser-vations can be used to examine work ﬂow and interpersonal interactions,including those between management and team members. Observersmay be previous employees, third-party participant observers, or mysteryshoppers.Sometimes, the demonstration of knowledge surrounding a certaintask, process, or procedure provides evidence of what capabilities existand what is lacking. Such demonstration can be as simple as a skill prac-tice or role play, or as complex as an extensive mechanical or electronicsimulation. The point is to use this as a way of determining if employ-ees know how to perform a particular process. Through demonstration,speciﬁc learning needs can evolve.Testing as a learning needs assessment process is not used as fre-quently as other methods, but it can be very useful. Employees are tested
Determining Preference Needs 67to reveal what they know about a particular situation. This informationhelps to guide learning issues.In implementing projects in organizations where there is an existingmanager or team leader, input from the management team may be usedto assess the current situation and to indicate the knowledge and skillsrequired by the new situation. This input can be elicited through surveys,interviews, or focus groups. It can be a rich source of information aboutwhat the users of the project, if it is implemented, will need to know tomake it successful.Where learning is a minor component, learning needs are simple.Determining learning needs can be very time-consuming for majorprojects where new procedures, technologies, and processes must bedeveloped. As in developing job performance needs, it is important notto spend excessive time analyzing learning needs but rather to collect asmuch data as possible with minimal resources.DETERMINING PREFERENCE NEEDSThe ﬁnal level of needs analysis determines the preferences that drive theproject requirements. Essentially, individuals prefer certain processes,schedules, or activities for the structure of the project. These preferencesdeﬁne how the particular project will be implemented. If the project is asolution to a problem, this step deﬁnes how the solution will be installed.If the project makes use of an opportunity, this step outlines how theopportunity will be addressed, taking into consideration the preferencesof those involved in the project.Preference needs typically deﬁne the parameters of the project in termsof scope, timing, budget, stafﬁng, location, technology, deliverables, andthe degree of disruption allowed. Preference needs are developed fromthe input of several stakeholders rather than from one individual. Forexample, participants in the project (those who must make it work) mayhave a particular preference, but the preference could exhaust resources,time, and budgets. The immediate manager’s input may help minimizethe amount of disruption and maximize resources. The funds that can beallocated are also a constraining resource.The urgency of project implementation may introduce a constraintin the preferences. Those who support or own the project often imposepreferences on the project in terms of timing, budget, and the use oftechnology. Because preferences correspond to a Level 1 need, the project
68 ACHIEVING BUSINESS ALIGNMENT WITH THE PROJECTstructure and solution will relate directly to the reaction objectives andto the initial reaction to the project.In determining the preference needs, there can never be too muchdetail. Projects often go astray and fail to reach their full potential becauseof misunderstandings and differences in expectations surrounding theproject. Preference needs should be addressed before the project begins.Pertinent issues are often outlined in the project proposal or planningdocumentation.CASE STUDY: SOUTHEAST CORRIDOR BANKPayoff NeedsAt this point, following a case study through the different levels ofneeds may be helpful. The following discussion explores the analysisat Level 5, determining payoff needs. Southeast Corridor Bank (SCB)operated branches in four states. (SCB has since been acquired by RegionsBank, one of the nation’s top ten banks.) Like many other fast-growingorganizations, SCB faced merger and integration problems, includingexcessive employee turnover. A project manager was assigned the task ofreducing the voluntary turnover.SCB’s annual employee turnover was 57 percent, compared with anindustry average of 26 percent. The ﬁrst step in addressing the problemwas answering these questions:• Is this a problem worth solving?• Is there a potential payoff to solving the problem?To the senior vice president of human resources, the answers wereclear. After reviewing several published studies about the cost ofturnover—including one from a ﬁnancial institution—he concluded thatthe cost of employee turnover ranged between 110 and 225 percent ofannual pay. At the current rate, employee turnover was costing the bankmore than $6 million per year. Lowering the rate to the industry averagewould save the bank at least $3 million annually. Although the structureand cost of the solution had not been determined at this point, it becameclear that this problem was worth solving. Unless the solution appearedto be very expensive, solving the problem would have a tremendousimpact. This was the only analysis that was needed at this level.
Case Study: Southeast Corridor Bank 69Business NeedsThe speciﬁc measure in question was voluntary turnover: the num-ber of employees leaving voluntarily divided by the average number ofemployees, expressed as a percentage. Clearly deﬁning the measure wasimportant. Still, with improvement in any one measure, other measuresshould also improve, depending on the speciﬁc solution. For example,stafﬁng levels, job satisfaction, customer service, sales revenue, andother measures could change. These considerations are detailed in thecontext of determining the solution.Performance NeedsTo identify the job performance needs, the cause of the problem hadto be determined. When the cause was determined, a solution could bedeveloped.The nominal group technique was selected as the analysis methodbecause it allowed unbiased input to be collected efﬁciently and accuratelyacross the organization. Focus groups were planned consisting of twelveemployees from each region, for a total of six groups representing all theregions. In addition, two focus groups were planned for the clerical staff inthe corporate headquarters. This approach provided approximately a 10percent sample, which was considered sufﬁcient to pinpoint the problem.The focus group participants who represented areas in which turnoverwas highest described why their colleagues were leaving, not why theythemselves would leave. Data were collected from individuals using acarefully structured format—during two-hour meetings at each location,with third-party facilitators—and were integrated and weighted so thatthe most important reasons were clearly identiﬁed. This process had theadvantages of low cost and high reliability, as well as a low degree ofbias. Only two days of external facilitator time were needed to collect andsummarize the data for review.Following are the ten major reasons given for turnover in the bankbranches:1. Lack of opportunity for advancement2. Lack of opportunity to learn new skills and gain new productknowledge3. Pay level not adequate
70 ACHIEVING BUSINESS ALIGNMENT WITH THE PROJECT4. Not enough responsibility and empowerment5. Lack of recognition and appreciation of work6. Lack of teamwork7. Lack of preparation for customer service problems8. Unfair and nonsupportive supervisor9. Too much stress at peak times10. Not enough ﬂexibility in work schedulesA similar list was developed for the administrative staff. However, theremainder of this case study will focus on the efforts to reduce turnover inthe branch network. Branch turnover was the most critical issue, becauseof its high rate and the large number of employees involved, and thefocus group results provided a clear pattern of speciﬁc needs. Recognizingthat not all causes of the turnover could be addressed immediately, thebank’s management concentrated on the top ﬁve reasons and considereda variety of options.The SolutionThe project manager determined that a skill-based pay system wouldaddress the top ﬁve reasons for employee turnover. The project wasdesigned to expand the scope of the jobs, with increases in pay awardedfor the acquisition of skills and a clear path provided for advancementand improvement. Jobs were redesigned from narrowly focused dutiesto an expanded role with a new title: Every teller became a bankingrepresentative I, II, or III.A branch employee would be designated a banking representative Iif he or she could perform one or two simple tasks, such as process-ing deposits and cashing checks. As an employee at this level took onadditional responsibilities and learned to perform different functions, heor she would be eligible for a promotion to banking representative II.A representative who could perform all the basic functions of the branch,including processing consumer loan applications, would be promoted tobanking representative III.Training opportunities were available to help employees developthe needed skills, and structured on-the-job training was provided bythe branch managers, assistant managers, and supervisors. Self-studyinformation was also available. The performance of multiple tasks wasintroduced to broaden responsibilities and enable employees to provide
Case Study: Southeast Corridor Bank 71excellent customer service. Pay increases were used to recognize skillacquisition, demonstrated accomplishment, and increased responsibility.Although the skill-based system had obvious advantages from theemployee’s perspective, the bank also beneﬁted. Not only was turnoverexpected to decline, but required stafﬁng levels were expected to decreasein the larger branches. In theory, if all employees in a branch couldperform all the necessary duties, fewer employees would be needed.Previously, certain critical jobs required minimum stafﬁng levels, andemployees in those positions were not always available for other duties.In addition, the bank anticipated improved customer service. The newapproach would prevent customers from having to wait in long lines forspecialized services. For example, in the typical bank branch, long linesfor special functions—such as opening a checking account, closing out acertiﬁcate of deposit, or accepting a consumer loan application—were notunusual under the old setup, whereas routine activities such as payingbills and receiving deposits often required little or no waiting. With eachemployee now performing all the tasks, shorter waiting lines could beexpected.To support this new arrangement, the marketing department featuredthe concept in its publicity about products and services. Included withchecking account statements was a promotional piece stating, ‘‘In ourbranches, there are no tellers.’’ This document described the new pro-cess and announced that every employee could now perform all branchfunctions and consequently provide faster service.Learning NeedsAt Level 2, learning needs fell into two categories. First, for each learn-ing project, both skill acquisition and knowledge development needswere identiﬁed. Learning measurements included self-assessment, test-ing, and demonstrations, among others, and were connected to eachspeciﬁc project.Second, it was necessary for employees to learn how the new projectworked. As the project was introduced in meetings with employees, asimple measurement of learning was necessary to capture employeeunderstanding of the following issues:• How the project is being pursued• What employees must do to succeed in the project
72 ACHIEVING BUSINESS ALIGNMENT WITH THE PROJECT• How promotion decisions are made• The timing of various stages of the projectThese major learning needs were identiﬁed and connected speciﬁcallywith the solution being implemented.Preference NeedsAs the project was rolled out and the solution was developed, the pref-erence needs were deﬁned. The project had to be rolled out as soonas possible so that its effects could be translated into lower employeeturnover. All the training projects must be in place and available toemployees. The amount of time employees must spend away from theirjobs for training was an issue, as was the managers’ control over the tim-ing of promotions. This process must move swiftly, or it would result indisappointment to employees who were eager to be trained and promoted.At the same time, the stafﬁng and workload concerns had to be balancedso that the appropriate amount of time was devoted to training and skillbuilding. More speciﬁcally, with the project’s announcement, the desiredemployee reaction was deﬁned. Project leaders wanted employees to viewthe project as very challenging, motivational, rewarding, and fair and asa solid investment in their futures. These needs were easily translatedinto the solution design.DEVELOPING OBJECTIVES FOR PROJECTSProjects are driven by objectives. These objectives will position the projectfor success if they represent the needs of the business and includeclearly deﬁned measures of achievement. A project may be aimed atimplementing a solution that addresses a particular dilemma, problem, oropportunity. In other situations, the initial project is designed to developa range of feasible solutions, with one speciﬁc solution selected prior toimplementation. Regardless of the project, multiple levels of objectives arenecessary. These levels follow the ﬁve-level data categorization schemeand deﬁne precisely what will occur as a project is implemented. Theycorrespond to the levels of evaluation and the levels of needs presentedin Figure 4.1.
Developing Objectives for Projects 73Reaction ObjectivesFor a project to be successful, the stakeholders immediately involvedin the process must react favorably—or at least not negatively—to theproject. Ideally, those directly involved should be satisﬁed with the projectand see the value in it. This feedback must be obtained routinely duringthe project in order to make adjustments, keep the project on track, andredesign certain aspects as necessary. Unfortunately, for many projects,speciﬁc objectives at this level are not developed, nor are data collectionmechanisms put in place to allow channels for feedback.Developing reaction objectives should be straightforward and relativelyeasy. The objectives reﬂect the degree of immediate as well as long-termsatisfaction and explore issues important to the success of the project.They also form the basis for evaluating the chain of impact, and theyemphasize planned action, when this is feasible and needed. Typicalissues addressed in the development of reaction objectives are relevance,usefulness, importance, appropriateness, rewards, and motivation.Learning ObjectivesEvery project involves at least one learning objective, and most involvemore. With projects entailing major change, the learning component isquite important. In situations narrower in scope, such as the implementa-tion of a new policy, the learning component is minor but still necessary.To ensure that the various stakeholders have learned what they need toknow to make the project successful, learning objectives are developed.The following are examples of learning objectives.After the launch of the project, participants should be able to:• Identify the six features of the new ethics policy.• Demonstrate the use of each software routine within the standardtime.• Score 75 or better on the new-product quiz.• Explain the value of diversity in a work group.• Successfully complete the leadership simulation.• Know how to apply for housing assistance.Objectives are critical to the measurement of learning because theycommunicate the expected outcomes from the learning component and
74 ACHIEVING BUSINESS ALIGNMENT WITH THE PROJECTdeﬁne the competency or level of performance necessary to make theproject successful. They provide a focus to allow participants to clearlyidentify what it is they must learn and do—sometimes with precision.Application ObjectivesThe application and implementation objectives clearly deﬁne what isexpected of the project and often the target level of performance. Appli-cation objectives are similar to learning objectives but relate to actualperformance. They provide speciﬁc milestones indicating when one partor all of the process has been implemented. Typical application objectivesare as follows:When the project is implemented . . .• At least 99.1 percent of software users will be following the correctsequences after three weeks of use.• Within one year, 10 percent of employees will submit documentedsuggestions for cutting costs.• Ninety-ﬁve percent of high-potential employees will complete indi-vidual development plans within two years.• Forty percent of the city’s homeless population will apply for specialhousing within one year of project launch.• Eighty percent of employees will use one or more of the three costcontainment features of the health care plan.• Fifty percent of conference attendees follow up with at least onecontact from the conference.• The average 360-degree leadership assessment score will improvefrom 3.4 to 4.1 on a 5-point scale.• Employees will routinely use problem-solving skills when faced witha quality problem.• Sexual harassment activity will cease within three months after thezero-tolerance policy is implemented.• By November, pharmaceutical sales reps will communicate adverseeffects of a speciﬁc prescription drug to all physicians in theirterritories.• Managers will initiate three workout projects within ﬁfteen days.• Sales and customer service representatives will use all ﬁve interac-tion skills with at least half the customers within the next month.
Developing Objectives for Projects 75Application objectives are critical because they describe the expectedoutcomes in the intermediate area—between the learning of new tasksand procedures and the delivery of the impact of this learning. Appli-cation and implementation objectives describe how things should be orthe desired state of the workplace once the project solution has beenimplemented. They provide a basis for evaluating on-the-job changes andperformance.Impact ObjectivesEvery project should drive one or more business impact measures. Impactobjectives indicate key business measures that should improve as theapplication and implementation objectives are achieved. The followingare typical impact objectives.When the project is completed . . .• System downtime should be reduced from three hours per month tono more than two hours per month in six months.• Incidents should decrease by 20 percent within the next calendaryear.• The average number of product defects should decrease from 214 to153 per month in the Midwest region.• The customer satisfaction index should rise by 2 percent during thenext calendar year.• Sales expenses for all titles at Proof Publishing Company shoulddecrease by 10 percent in the fourth quarter.• The average number of new accounts opened at Great Western Bankshould increase from 300 to 350 per month in six months.• The shelter costs per homeless person should be below $70,000 intwo years.• There should be an across-the-board reduction in overtime for front-of-house managers at Tasty Time restaurants in the third quarterof this year.• Employee complaints should be reduced from an average of threeper month to an average of one per month at Guarantee Insuranceheadquarters.• The companywide employee engagement index should rise by onepoint during the next calendar year.
76 ACHIEVING BUSINESS ALIGNMENT WITH THE PROJECT• There should be a 10 percent increase in Pharmaceuticals Inc. brandawareness among physicians during the next two years.• The dropout rate for high school students in the Barett Countysystem should decrease by 5 percent within three years.Impact objectives are critical to measuring business performancebecause they deﬁne the ultimate expected outcome from the project.They describe the business unit performance that should result fromthe project. Above all, impact objectives emphasize achievement of thebottom-line results that key client groups expect and demand.ROI ObjectivesThe ﬁfth level of objectives for projects represents the acceptable returnon investment, the monetary impact. Objectives at this level deﬁne theexpected payoff from investing in the project. An ROI objective is typicallyexpressed as an acceptable ROI percentage, which is expressed as annualmonetary beneﬁts minus cost, divided by the actual cost, and multiplied byone hundred. A 0 percent ROI indicates a breakeven project. A 50 percentROI indicates recapture of the project cost and an additional 50 percent‘‘earnings’’ (50 cents for every dollar invested).For some projects, such as the purchase of a new company, a newbuilding, or major equipment, the ROI objective is large relative to theROI of other expenditures. However, the calculation is the same for both.For many organizations, the ROI objective for a project is set slightlyhigher than the ROI expected from other ‘‘routine investments’’ becauseof the relative newness of applying the ROI concept to the types ofprojects described in this book. For example, if the expected ROI from thepurchase of a new company is 20 percent, the ROI from a new advertisingproject might be around 25 percent. The important point is that the ROIobjective should be established up front and in discussions with the projectsponsor. Excluding the ROI objective leaves stakeholders questioning theeconomic success of a project. If a project reaps a 25 percent ROI, is thatsuccessful? Not if the objective was a 50 percent ROI.FINAL THOUGHTSThis chapter outlines the beginning point in the ROI methodology, show-ing how a project can be structured from the outset, with detailed needs
Final Thoughts 77identiﬁed, ultimately leading to project objectives at ﬁve levels. This levelof detail ensures that the project is aligned with business needs andremains results-focused throughout the process. Without this analysis,the project runs the risk of failing to deliver the value that it should, ornot being in alignment with one or more business objectives. The outputsof the analysis are objectives, which provide a focus for project designers,developers, and implementers, as well as participants and users whomust make the project successful. Issues surrounding data collectionare discussed in the next three chapters. Collecting reaction data andperceived value data will be covered ﬁrst.
80 MEASURING REACTION AND LEARNINGCustomer SatisfactionReaction and perceived value are customer satisfaction measures for theproject. Without sustained, favorable reactions, the project may not suc-ceed. The individuals who have a direct role in the project are immediatelyaffected by it and often have to change processes or procedures or makeother job adjustments in response to the project’s initiation. Participantfeedback on preferences is critical to making adjustments and changes inthe project as it unfolds.The feedback of project supporters is also important because thisgroup will be in a position to inﬂuence the project’s continuation anddevelopment.The sponsors, who approve budgets, allocate resources, and ultimatelylive with the project’s success or failure, must be completely satisﬁedwith the project—and their overall satisfaction must be veriﬁed earlyand often.Immediate AdjustmentsA project can quickly go astray. A project can be mismatched to thesolution from the beginning, so getting feedback early in the process isnecessary to allow for immediate adjustments. This can help preventmisunderstandings, miscommunications, and, more important, misap-propriations. Collecting and using reaction data promptly can enable animproperly designed project to be altered before more serious problemsarise.Predictive CapabilityReaction data can be used to predict the success of a project usinganalytical techniques. Project participants are asked to estimate theeffectiveness of the project’s application and, in some cases, the resultingvalue of that application. The reaction data thus become a forecast(forecasting is described in detail in Chapter 12). Figure 5.1 shows thecorrelation between reactive feedback and application data. Countlessstudies have been conducted to verify this correlation.In this analysis, the reaction measures are taken as the project isintroduced, and the success of the implementation is later judged using
The Importance of Learning 81ReactionApplicationFigure 5.1 Correlations between reaction and application.the same scale, e.g., a 1 to 5 rating. When signiﬁcant positive correlationsare present, reaction measures can have predictive capability. Somereaction measures shown to have predictive capability are• The project is relevant to my work.• The project is necessary.• The project is important to my success.• The project is important to the success of this organization.• I intend to make this project successful.• I would recommend this project.Measures such as these consistently lead to strong positive correlationsand consequently represent more powerful feedback than typical mea-sures of satisfaction with the project. Some organizations collect these orsimilar reaction measures for every project or program initiated.THE IMPORTANCE OF LEARNINGSeveral key principles illustrate the importance of measuring learningduring the course of a project. Each of these in itself is sufﬁcient to justifythe measurement of learning; collectively, they provide an indication ofthe full range of beneﬁts that result from measuring the changes in skills,knowledge, and other qualities that occur during a project.
82 MEASURING REACTION AND LEARNINGThe Importance of Intellectual CapitalIntellectual capital has become an important concept as organizationshave progressed from agricultural to industrial to knowledge-based sys-tems. Intellectual capital is what the organization knows, and it canbe classiﬁed in a variety of ways for measurement purposes. Figure 5.2illustrates one categorization of intellectual capital, showing intellectualcapital as a combination of human capital, renewable capital, structuralcapital, and relationship capital.1As projects are implemented, they focuson increasing one or more of these major elements of intellectual capital.For some organizations, intellectual capital translates directly into suc-cess, in the form of rewards by the stock market. Up to 80 percent of themarket value of some high-technology ﬁrms is attributed to intellectualcapital. This demonstrates the value of measuring learning in projectsaimed at improving intellectual capital.The Learning OrganizationIn the past two decades, organizations have experienced a rapidtransformation of competitive global markets as a result of economicchanges. Organizations must learn new ways to serve customers and useHumancapitalRenewablecapitalStructuralcapitalIntellectualcapitalRelationshipcapitalIndividual capital:Personal expertiseand experience,and the ability totransform it intonew, sharedknowledgeNew knowledge:Created,transferred, andcommunicated bymany peopleIntellectualproperties:Patents, licenses,etc.Marketableinnovations:Products, services,and technologyWork processes:Institutionalizedknowledge in theform of procedures,policies, processtechnologies, etc.Documentation:Databases, records,and knowledgedocuments ofvarious formsNetworks:Resources forinformation andinfluenceCustomers:Particularly thosemost innovative intheir industriesFigure 5.2 Categories of intellectual capital.
The Importance of Learning 83innovations and technology to enhance their efﬁciency, to restructure,reorganize, and execute their functions globally. In response to this needfor a change in strategy, the concept of the learning organization evolved.This concept requires organizations to use learning proactively in anintegrated way and to support growth for individuals, teams, and entireorganizations.With the new focus on creating learning organizations—wherecountless activities and processes are in place to promote continuouslearning—measurement has become an even more important issue.The Compliance IssueOrganizations face an increasing number of regulations with which theymust routinely comply. These regulations involve all aspects of businessand are considered essential by governing bodies to protect customers,investors, and the environment. Employees must have a certain amount ofknowledge about the regulations to maintain compliance. Consequently,an organization must measure the extent of employee learning andunderstanding with regard to regulations to ensure that compliance isnot a problem.Some projects are compliance driven. For example, one large bankingorganization had to implement a major project to ensure that itsemployees were all familiar with money laundering regulations. Thisproject was precipitated by the bank’s’ continuing failure to comply withthe regulations. The problem appeared to be a lack of knowledge ofthe rules. When projects such as this are initiated, learning must bemeasured.The Use and Development of CompetenciesA competency model describes a particular combination of knowledge,skills, and characteristics necessary to perform a role in an organiza-tion. The use of competencies and competency models has dramaticallyincreased in recent years. In the struggle for a competitive advantage,many organizations have focused on people as the key to success. Com-petency models are used to ensure that employees do the right things,clarifying and articulating what is required for effective performance.Competency models help organizations align behavior and skills with thestrategic direction of the company.
84 MEASURING REACTION AND LEARNINGThe Role of Learning in ProjectsWhen projects involve new equipment, processes, and technology, thehuman factor is critical to project success. Whether an organizationis restructuring or adding new systems, employees must learn how towork in the new environment, and this requires the development of newknowledge and skills. Simple tasks and procedures do not automaticallycome with new processes. Instead, complex environments, procedures,and tools must be used in an intelligent way to reap the desired beneﬁtsfor the organization. Employees must learn in different ways—not just ina formal classroom environment, but through technology-based learningand on-the-job practice. Project leaders and managers serve as coaches ormentors in some projects. In a few cases, learning coaches or on-the-jobtrainers are used with projects to ensure that learning is transferred tothe job and is implemented as planned.Project team members don’t always fully understand what they mustdo. Although the chain of impact can be broken at any level, a commonplace for such a break is at Level 2, learning and conﬁdence. Employeessimply may not know what to do or how to do it properly. When theapplication and implementation does not go smoothly, project managerscan determine if a learning deﬁciency is the problem, and if so, theymay be able to eliminate it. In other words, learning measurement isnecessary to contribute to leaders’ understanding of why employees are,or are not, performing the way they should.SOURCES OF DATAPossible sources of reaction and learning data concerning the success ofa project can be grouped into distinct categories.Project ParticipantsThe most widely used data source for project evaluation is the partici-pants, those directly involved in the project. These ‘‘users’’ must take theskills and knowledge they acquired in the project or process and applythem on the job. They also may be asked to explain the potential impactof that application. Participants are a rich source of data for almost everyaspect of a project. They are by far the most credible source of reactionand perceived value data.
Sources of Data 85Participant ManagersAnother key source of data is the individuals who directly lead the par-ticipants. They have a vested interest in the project and are often in aposition to observe the participants as they attempt to apply the knowl-edge and skills acquired in the project. Consequently, they can reporton the successes associated with the project as well as the difﬁcultiesand problems. Although managerial input is usually most valuable asa source of reaction and perceived value data, it is also useful for otherlevels of evaluation.Other EmployeesWhen entire teams are involved in the implementation of the project, allemployees can provide useful information about the perceived changesprompted by the project. Input from this group is pertinent only toissues directly related to their work. Although data from this groupcan be helpful and instructive, it is sometimes not elicited because ofthe potential for introducing inaccuracies to the feedback process. Datacollection should be restricted to those team members capable of providingmeaningful insight into the value of the project.Internal or External CustomersThe individuals who serve as internal customers of the project are anothersource of reaction, perceived value, and other types of data. In somesituations, internal or external customers provide input on perceivedchanges linked to the project. This source of data is more appropriate forprojects directly affecting the customers. They report on how the projecthas inﬂuenced (or will inﬂuence) their work or the service they receive.Although this group may be somewhat limited in their knowledge of thescope of a project, their perceptions can be a source of valuable data thatmay indicate a direction for change in the project.Project Leaders and Team MembersThe project leader and project team may also provide input on the successof the project. This input may be based on on-the-job observations duringthe course of the project and after its completion. Data from this source
86 MEASURING REACTION AND LEARNINGhave limited utility because project leaders often have a vested interestin the outcome of the evaluation, and thus may lack objectivity.Sponsors and Senior ManagersOne of the most useful data sources is the sponsor or client group, usually asenior management team. The perception of the sponsor, whether an indi-vidual or a group, is critical to project success. Sponsors can provide inputon all types of issues and are usually available and willing to offer feed-back. This is a preferred source for reaction data, since these data usuallyindicate what is necessary to make adjustments and to measure success.TOPICS FOR REACTION MEASURESWhen capturing reaction data, it is important to focus on the content ofthe project. Too often, feedback data reﬂect aesthetic issues that may notbe relevant to the substance of the project.Many topics are critical targets for feedback. Feedback is needed inconnection with almost every major issue, step, or process to make surethings are advancing properly. Stakeholders will provide reaction inputas to the appropriateness of the project planning schedule and objectives,and the progress made with the planning tools. If the project is perceivedas irrelevant or unnecessary to the participants, more than likely itwill not succeed in the workplace. Support for the project—includingresources—represents an important area of feedback.Table 5.1 distinguishes content and non-content issues explored in areaction questionnaire.Table 5.1 Issues for ReactionReadiness Service-Oriented Project ManagerTimely Importance ValuableRelevance Motivational PowerfulIntent to Use Content Leading EdgeUseful Rewarding Project LeadershipTeam Climate Project Execution Just for MeNecessary New Information EfﬁcientRecommend to Others Facilities/Environment Easy/DifﬁcultAppropriate Practical
Topics for Learning Measures 87Project participants must be assured that the project has the nec-essary commitment. Issues important to project management and theorganization sponsoring the project include project leadership, stafﬁnglevels, coordination, and communication. Also, it is important to collectfeedback on how well the project team is working to address such issuesas motivation and cooperation.Finally, the perceived value of the project is often a critical parameter.Major funding decisions are made based on perceived value when strongerevidence of value is unavailable.TOPICS FOR LEARNING MEASURESMeasuring learning focuses on knowledge, skills, and attitudes as wellas the individual’s conﬁdence in applying or implementing the project orprocess as desired. Typical measures collected at this level are:• Skills• Knowledge• Awareness• Understanding• Contacts• Attitudes• Capacity• Readiness• ConﬁdenceObviously, the more detailed the knowledge area, the greater thenumber of objectives. The concept of knowledge is quite general andoften includes the assimilation of facts, ﬁgures, and ideas. Instead ofknowledge, terms such as awareness, understanding, and informationmay be used to denote speciﬁc categories. Sometimes, perceptions orattitudes are changed based on what a participant has learned. Forexample, participants’ perceptions about a diverse work group are oftenchanged with the implementation of a major diversity program. In somecases, the issue is developing a reservoir of knowledge and related skillstoward improving capability, capacity, or readiness. Networking is oftenpart of a project, and developing contacts who may be valuable lateris important. This may occur within or external to an organization.For example, within the organization, a project may include differentfunctional parts of the organization, and an expected outcome from alearning perspective is knowing who to contact at particular times inthe future. For projects that involve different organizations, such as amarketing event, new contacts that result from the event can be importantand ultimately pay off in terms of efﬁciency and/or revenue growth.
88 MEASURING REACTION AND LEARNINGDATA COLLECTION TIMINGThe timing of data collection centers on particular events connected withthe project. As discussed previously, capturing reaction and learning dataduring the early stages of implementation is critical. Ideally, this feedbackvalidates the decision to go forward with the project and conﬁrms thatalignment with business needs exists. Problems with the initial feedbackmean that adjustments can be made early in its implementation. Inpractice, however, many organizations omit this early feedback, waitinguntil signiﬁcant parts of the project have been implemented, at whichpoint feedback may be more meaningful.For longer projects, concerns related to the timing of feedback mayrequire data collection at multiple points in time. Measures can be takenat the beginning of the project and then at routine intervals once theproject is under way.THE CHALLENGES AND BENEFITS OF MEASURINGLEARNINGMeasuring learning involves major challenges that may inhibit a compre-hensive approach to the process. However, besides being an essential partof the ROI methodology, this measurement provides many other beneﬁtsthat help ensure a project’s success.ChallengesThe greatest challenge in measuring learning is to maintain objectivitywithout crossing ethical or legal lines while keeping costs low. A commonmethod of measuring learning is testing, but this approach generatesadditional challenges.The ﬁrst challenge is the ‘‘enjoyment’’ factor. Few people enjoy beingtested. Many are offended by it and feel that their professional prowessis being questioned. Some people are intimidated by tests, which bringback memories of their third-grade math teacher, red pen in hand.Another challenge with tests is the legal and ethical repercussionsof basing decisions involving employee status on test scores. Therefore,organizations use other techniques to measure learning, such as sur-veys, questionnaires, role plays, and simulations. The challenge with
The Challenges and Beneﬁts of Measuring Learning 89these methods, however, is the potential for inaccurate measures andthe ﬁnancial burden they impose. Consequently, there is a constanttrade-off between additional resources and the accuracy of the learningmeasurement process.The Beneﬁts of Measuring LearningThe beneﬁts of measuring learning are reﬂected in the reasons for learn-ing measurement described earlier. First, measurement at this levelchecks the progress of the project against the objectives. Objectives arecritical to a project, and they should be established at each level. Funda-mentally, the measurement of learning reveals the extent of knowledgeand skill acquisition in relation to the project. This is a critical element,particularly for knowledge-based projects, new technology applications,and projects designed to build competencies.In addition to assessing improvement in skills or knowledge, mea-surement at this level provides feedback to the individuals deliveringthe skills or knowledge so that adjustments can be made. If project par-ticipants are not learning, there may be a problem with the way thelearning is being delivered. A learning measure can identify strengthsand weaknesses in the method of project presentation or instruction andmay point out ﬂaws in the design or delivery. Thus, measuring learningcan pinpoint mismatches among various aspects of a project and therebylead to changes or improvement.Another important beneﬁt is that, in many cases, learning measuresenhance participant performance. Veriﬁcation and feedback concerningthe knowledge and skills acquired can encourage participants to improvein certain areas. When employees excel, feedback motivates them toenhance their performance even further. In short, positive feedbackon learning builds conﬁdence and the desire to continue improving.Without such measurement, participants will never know their potentialto perform.Finally, measuring learning helps to maintain accountability. Becauseprojects are aimed at making organizations better—whether a learningorganization is being built, competencies are being improved, or systemsand processes are being enhanced—learning is an important part of anyproject and its measurement is vital in conﬁrming that improvement hasin fact occurred.
90 MEASURING REACTION AND LEARNINGDATA COLLECTION METHODSA variety of methods can be used to collect reaction and learning data.Instruments range from simple surveys to comprehensive interviews. Theappropriate instrument depends on the type of data needed (quantitativevs. qualitative), the convenience of the method to potential respondents,the culture of the organization, and the cost of a particular instrument.Questionnaires and SurveysThe questionnaire or survey is the most common method of collectingreaction and learning data. Questionnaires and surveys come in all sizes,ranging from short forms to detailed, multiple-page instruments. Theycan be used to obtain subjective data about participants’ reactions andlearning, as well as to document responses for future use in a projectedROI analysis. Proper design of questionnaires and surveys is importantto ensure versatility.Several basic types of questions are used. The dichotomous question(yes/no) and the numerical scale, e.g., 1 to 5, are typical reaction mea-surement formats. Essentially, the individual is indicating the extent ofhis or her agreement with a particular statement or is giving an opinionof varying conviction on an issue. Surveys are a type of questionnairebut focus on attitudinal elements. Surveys have many applications in themeasurement of reaction and learning for projects designed to improvework. Depending on the purpose of the evaluation, the questionnaire orsurvey may contain one or more question types. The key is to select thequestion or statement that is most appropriate for the information sought.However, open-ended questions can sometimes be used, particularly inasking about speciﬁc problem areas. Checklists, multiple-choice ques-tions, and ranking scales are more appropriate for measuring learningand application, which are described in later chapters.For most reaction and learning evaluations, questionnaires and sur-veys are used. When a follow-up evaluation is planned, a wide range ofissues will be covered in a detailed questionnaire. Asking for too muchdetail in either the reaction questionnaire or the follow-up questionnairecan reduce the response rate. The objective, therefore, of questionnaireand survey design and administration is to maximize response. The fol-lowing actions can help ensure adequate response rates for questionnairesand surveys:
Data Collection Methods 911. Provide advance communication.2. Communicate the purpose.3. Identify who will see the results.4. Describe the data integration process.5. Let the target audience know that they are part of a sample.6. Add emotional appeal.7. Design for simplicity.8. Make it look professional and attractive.9. Use the local manager support.10. Build on earlier data.11. Pilot test the questionnaire.12. Recognize the expertise of participants.13. Consider the use of incentives.14. Have an executive sign the introductory letter.15. Send a copy of the results to the participants.16. Report the use of results.17. Provide an update to create pressure to respond.18. Present previous responses.19. Introduce the questionnaire during the program.20. Use follow-up reminders.21. Consider a captive audience.22. Consider the appropriate medium for easy response.23. Estimate the necessary time to complete the questionnaire.24. Show the timing of the planned steps.25. Personalize the process.26. Collect data anonymously or conﬁdentially.InterviewsInterviews, although not used as frequently as questionnaires to capturereaction and learning data, may be conducted by the project team, theclient team, or a third party to secure data that are difﬁcult to obtainthrough written responses. Interviews can uncover success stories thatmay help to communicate early achievements of the project. Respondentsmay be reluctant to describe their experiences using a questionnaire butmay volunteer the information to a skillful interviewer using probingtechniques. The interview is versatile and is appropriate for solicitingreaction and learning data as well as application and implementationdata. A major disadvantage of the interview is that it consumes time,
92 MEASURING REACTION AND LEARNINGwhich increases the cost of data collection. It also requires interviewerpreparation to ensure that the process is consistent.Focus GroupsFocus groups are particularly useful when in-depth feedback is needed.The focus group format involves a small-group discussion conducted byan experienced facilitator. It is designed to solicit qualitative judgmentson a selected topic or issue. All group members are required to provideinput, with individual input building on group input.Compared with questionnaires, surveys, and interviews, the focusgroup approach has several advantages. The basic premise behind theuse of focus groups is that when quality judgments are subjective, severalindividual judgments are better than one. The group process, where par-ticipants often motivate one another, is an effective method for generatingand clarifying ideas and hypotheses. It is inexpensive and can be quicklyplanned and conducted. Its ﬂexibility allows exploration of a project’sunexpected outcomes or applications.Performance TestsPerformance testing allows the participants and users to exhibit the skills(and occasionally knowledge or attitudes) that have been learned in aproject. A skill can be manual, verbal, or analytical, or a combinationof the three. Performance testing is used frequently in task-relatedprojects; here the participants are allowed to demonstrate what theyhave learned and to show how they would use the skill on the job.In other situations, performance testing may involve skill practice orrole-playing, e.g., participants are asked to demonstrate discussion orproblem-solving skills that they have acquired.In a particular situation, computer systems engineers were participat-ing in a system reengineering project. As part of the project, participantswere given the assignment of designing and testing a basic system. Aproject team manager observed participants as they checked out thesystem; then the manager carefully completed the same design and com-pared his results with those of the participants. These comparisons andthe performance of the designs provided an evaluation of the project andrepresented an adequate reﬂection of the skills learned in the project.
Data Use 93Technology and Task SimulationsAnother technique for measuring learning is simulation. This methodinvolves the construction and application of a procedure or task thatsimulates or models the work involved in the project or program. Thesimulation is designed to represent, as closely as possible, the actualjob situation. Participants try out the simulated activity and their per-formance is evaluated based on how well they accomplish the task.Simulations offer several advantages. They permit a job or part of a job tobe reproduced in a manner almost identical to the real setting. Throughcareful planning and design, the simulation can have all the centralcharacteristics of the real situation. Even complex jobs, such as that ofthe manager, can be simulated adequately.Although the initial development can be expensive, simulations can becost-effective in the long run, particularly for large projects or situationswhere a project may be repeated. Another advantage of using simulationsis safety. Safety considerations for many jobs require participants tobe trained in simulated conditions. For example, emergency medicaltechnicians risk injury and even death if they do not learn the neededtechniques prior to encountering a real-life emergency.Although a variety of simulation techniques are used to evaluate learn-ing during a project, two of the most common techniques are technologyand task simulation. A technology simulation uses a combination of elec-tronic and mechanical devices to reproduce real-life situations. Thesesimulations are used in conjunction with programs to develop operationaland diagnostic skills. Expensive examples are simulated ‘‘patients’’ ora simulation of a nuclear power plant operator. Other, less expensivedevices have been developed to simulate equipment operation.A task simulation involves a participant’s performance in a simulatedtask. For example, a customer service associate must demonstrate thetask of creating a new account. This task simulation serves as theevaluation.DATA USEUnfortunately, reaction and learning data are often collected and thendisregarded. Too many project evaluators use the information to feedtheir egos and then allow it to quietly disappear into their ﬁles, forgettingthe original purpose behind its collection. In an effective evaluation, the
94 MEASURING REACTION AND LEARNINGinformation collected must be used to make adjustments or verify success;otherwise, the exercise is a waste of time.Because this input is the principal measure supplied by key stakehold-ers, it provides an indication of their reaction to, and satisfaction with,the project. More important, these data provide evidence relating to thepotential success of the project. Reaction data should be used to:• Identify the strengths and weaknesses of the project and makeadjustments.• Evaluate project team members.• Evaluate the quality and content of planned improvements.• Develop norms and standards.• Link with follow-up data.• Market future projects based on the positive reaction.Learning data are used to:• Provide individual feedback to build conﬁdence.• Validate that learning has been acquired.• Improve the project.• Evaluate project leaders/facilitators.• Build a database for project comparisons.FINAL THOUGHTSThis chapter discusses data collection at the ﬁrst two levels ofevaluation—reaction and learning. Measuring reaction and learning is acomponent of every study and is a critical factor in a project’s success. Thedata are collected using a variety of techniques, although surveys andquestionnaires are most often used because of their cost-effectiveness andconvenience. The data are important in allowing immediate adjustmentsto be made on the project.Reaction data are important, but importance to executives increases asthe evaluation moves up the chain of impact. Data collection at the secondlevel, learning, is usually more important. The next chapter focuses onLevel 3, measuring application and implementation.
96 MEASURING APPLICATION AND IMPLEMENTATIONROI (Level 5). Thus, information concerning application and implementa-tion (Level 3) is more valuable to the client than are reaction and learningdata. This is not to discount the importance of the ﬁrst two levels, but toemphasize the importance of moving up the chain of impact. Measuringthe extent to which a project is implemented provides critical data aboutits success, and about factors that can contribute to greater success asthe project is fully implemented.The Level 1 and Level 2 measures occur during a project’s early stages,when more attention and direct focus are placed on the participants’ directinvolvement in the project. Measuring application and implementationoccurs after the project has been implemented and captures the successof moving the project forward through participants’ on-the-job use ofknowledge about the project. Essentially, this measure reﬂects the degreeto which the project is implemented by those who are charged with itssuccess. This is the ﬁrst step in transitioning to a new state, behavior, orprocess. Understanding the success of the transition requires measuringapplication and implementation.Project FocusBecause many projects and programs focus directly on implementationand application or new behaviors and processes, a project sponsor oftenspeaks in these terms and has concerns about these measures of success.The sponsor of a major project designed to transform an organization willbe greatly concerned with implementation and application, and will wantto know the extent to which key stakeholders adjust to and implementthe desired new behaviors, processes, and procedures.Problems and OpportunitiesIf the chain of impact breaks at this level, little or no correspondingimpact data and consequences will be available. Without impact thereis no ROI. This breakdown most often occurs because participants inthe project encounter barriers, inhibitors, and obstacles (covered later)that deter implementation. A dilemma arises when reactions to theproject are favorable and participants learn what is intended, but theyfail to overcome the barriers and don’t manage to accomplish what’snecessary.
Challenges 97When a project goes astray, the ﬁrst question usually asked is, ‘‘Whathappened?’’ More importantly, when a project appears to add no value,the ﬁrst question ought to be, ‘‘What can we do to change its direction?’’In either scenario, it is important to identify the barriers to success,the problems in implementation, and the obstacles to application. AtLevel 3, measuring implementation and application, these problems areaddressed, identiﬁed, and examined. In many cases, the stakeholdersdirectly involved in the process can provide important recommendationsfor making changes or using a different approach in the future.When a project is successful, the obvious question is, ‘‘How can werepeat this or even improve it in the future?’’ The answer to this questionis also found at Level 3. Identifying the factors that contribute directlyto the success of the project is critical. Those same items can be used toreplicate the process and produce enhanced results in the future. Whenkey stakeholders identify those issues, they make the project successfuland provide an important case history of what is necessary for success.Reward EffectivenessMeasuring application and implementation allows the sponsor and projectmanager to reward those who do the best job of applying the processesand implementing the project. Measures taken at this level provide clearevidence of success and achievement, and provide an excellent basis forperformance reviews. Rewards often have a reinforcing value, helping tokeep employees on track and communicating a strong message for futureimprovement.CHALLENGESCollecting application and implementation data brings into focus somekey challenges that must be addressed for success at this level. Thesechallenges often inhibit an otherwise successful evaluation.Linking with LearningApplication data should be linked closely with the learning data dis-cussed in the previous chapter. Essentially, project leaders need to knowwhat has been accomplished, what has been done differently, and whatactivities have been implemented, based on what the individuals learned
98 MEASURING APPLICATION AND IMPLEMENTATIONto do. This level measures the extent to which participants accuratelytook what they learned and applied it to their jobs.Building Data Collection into the ProjectApplication data are collected after the project’s implementation. Becauseof the time lag between project implementation and data collection, itis difﬁcult to secure a high quality and quantity of data. Consequently,one of the most effective ways to ensure that data are collected is tobuild data collection into the project from the beginning. Data collectiontools positioned as application tools must be built in as part of the imple-mentation. By analogy, consider that many software applications containoverlay software that shows a user performance proﬁle. Essentially, thesoftware tracks the user invisibly, capturing the steps, pace, and difﬁcul-ties encountered while using the software. When the process is complete,a credible data set has been captured, simply because project leadersbuilt it into the process at the beginning.Ensuring a Sufﬁcient Amount of DataWhether collecting data by questionnaire or through action plans, inter-views, or focus groups, poor response rates are a problem in mostorganizations. Having individuals participate in the data collection pro-cess is a challenge. To ensure that adequate amounts of high-quality dataare available, a serious effort is needed to improve response rates.Because many projects are planned on the basis of ROI methodology, itis expected that project managers will have collected impact data, mone-tary values, and the project’s actual ROI. This need to ‘‘show the money’’sometimes results in less emphasis being placed on measuring applicationand implementation. In many cases, it may be omitted or slighted in theanalysis. But it is through focused effort on process and behavior changethat business impact will occur. Therefore, emphasis must be placed oncollecting data that deal with application and implementation. As withGeneral Electric’s workout processes, although added value may havebeen the goal, attention was placed on changing processes, procedures,and tasks, and on removing barriers. Doing things differently can resultin substantial beneﬁts, but knowing the degree to which things are donedifferently is essential to guaranteeing those beneﬁts.
Measurement Issues 99MEASUREMENT ISSUESWhen measuring the application and implementation of projects andprograms, several key issues should be addressed, which are largelysimilar to those encountered when measuring reaction and learning.(A few issues may differ slightly because of the later timeframe forcollecting this type of data.)MethodsA variety of methods are available when collecting data at Level 3. Theseinclude traditional survey and questionnaire methods, and methods basedon observation, interviews, and focus groups. Other powerful methodsinclude action planning, in which individuals plan their parts of theimplementation, and follow-up sessions. Data collection methods aredescribed in more detail later in this chapter.ObjectivesAs with the other levels, the starting point for data collection is theobjectives set for project application and implementation. Without clearobjectives, collecting data would be difﬁcult. Objectives deﬁne what activ-ity is expected. (Chapter 3 discusses the basic principles for developingthese objectives.)Areas of CoverageTo a certain extent, the areas of coverage for this process parallel the areasidentiﬁed in Chapter 5. The later timeframe for data collection changesthe measurement to a post-project measure rather than a predictivemeasure. The key point is that this level focuses on activity or action, noton the ability to act (Level 2) and not on the consequences (Level 4). Thesheer number of activities to measure can be mind-boggling. Table 6.1shows examples of coverage areas for application, which will vary fromproject to project.Data SourcesThe sources of data mirror those identiﬁed in Chapter 5. Essentially,all key stakeholders are potential sources of data. Perhaps the most
100 MEASURING APPLICATION AND IMPLEMENTATIONTable 6.1 Examples of Coverage Areas for ApplicationAction Explanation ExampleEliminate Stopping a particular task oractivityEliminate the formal follow-upmeeting, and replace it with avirtual meetingMaintain Keeping the same level ofactivity for a particular processContinue to monitor theprocess with the sameschedule previously usedCreate Designing or implementing anew procedure, process, oractivityCreate a procedure forresolving the differencesbetween two divisionsUse Using a particular process,procedure, skill, or activityUse the new skill in situationsfor which it was designedPerform Carrying out a particular task,process, or procedureConduct a post-audit review atthe end of each activityParticipate Becoming involved in variousactivities, projects, orprogramsSubmit a suggestion forreducing costsEnroll Signing up for a particularprocess, program, or projectEnroll in a careeradvancement programRespond Reacting to groups,individuals, or systemsRespond to customer inquirieswithin 15 minutesNetwork Facilitating relationships withothers who are involved in orhave been affected by theprojectContinue networking withcontacts on (at minimum) aquarterly basisIncrease Increasing a particular activityor actionIncrease the frequency of useof a particular skillDecrease Decreasing a particularactivity or actionDecrease the number of timesa particular process must becheckedimportant sources of data are the users of the solutions, those directlyinvolved in the application and implementation of the project or program.Good sources may also be the project team or team leaders charged withthe implementation. In some cases, the source may be the organizationalrecords or system.TimingThe timing of data collection can vary signiﬁcantly. Because this is afollow-up after the project launch, the key issue is determining the best
Data Collection Methods 101time for a post-implementation evaluation. The challenge is to analyze thenature and scope of the application and implementation, and to determinethe earliest time that a trend and pattern will evolve. This occurs when theapplication of skills becomes routine and the implementation is makingsigniﬁcant progress. This is a judgment call. Going in as early as possibleis important so that potential adjustments can still be made. At the sametime, leaders must wait long enough so that behavior changes are allowedto occur and so that the implementation can be observed and measured.In projects spanning a considerable length of time, several measuresmay be taken at three- to six-month intervals. Using effective measuresat well-timed intervals will provide successive input on implementationprogress, and clearly show the extent of improvement.Convenience and constraints also inﬂuence the timing of data collec-tion. If the participants are conveniently meeting to observe a milestone orspecial event, this would be an excellent opportunity to collect data. Some-times, constraints are placed on data collection. Consider, for example,the time constraint that sponsors may impose. If they are anxious to havethe data to make project decisions, they may order the data collectionmoved to an earlier time than ideal.ResponsibilitiesMeasuring application and implementation may involve the responsibilityand work of others. With data collection occurring later than in Levels 1and 2, an important issue may surface in terms of who is responsiblefor this follow-up. Many possibilities exist, ranging from project staff andsponsors to an external, independent consultant. This matter should beaddressed in the planning stages so that no misunderstanding arisesas to the distribution of responsibilities. More important, those whoare responsible should fully understand the nature and scope of theiraccountabilities and what is needed to collect the data.DATA COLLECTION METHODSSome of the techniques previously mentioned that are available to collectapplication and implementation data are easy to administer and providequality data. Other techniques are more robust, providing greater detailabout success but raising more challenges in administration.
102 MEASURING APPLICATION AND IMPLEMENTATIONUsing Questionnaires to Measure Applicationand ImplementationQuestionnaires have become a mainstream data collection tool for mea-suring application and implementation because of their ﬂexibility, lowcost, and ease of administration. The discussion in Chapter 5 about ques-tionnaires designed to measure reaction and learning applies equallyto questionnaires developed to measure application and implementa-tion. One of the most difﬁcult tasks is determining the speciﬁc issuesto address in a follow-up questionnaire. Figure 6.1 presents contentitems necessary for capturing application, implementation, and impactinformation (Level 3 and Level 4 data).Using Interviews, Focus Groups, and ObservationInterviews and focus groups can be used during implementation or ona follow-up basis to collect data on implementation and application.Progress with objectivesAction plan implementationRelevance/importancePerception of valueUse of materialsKnowledge/skill enhancementSkills usedChanges with work/actionsImprovement/accomplishmentsDefinition of measureProvision of changeUnit valueBasisTotal impactList of other factorsImprovement linked with projectConfidence estimateLinkage with output measuresOther benefitsBarriersEnablersManagement supportOther solutionsRecommendations for other audiences/participantsSuggestions for improvementOther commentsOptional unless business impactand ROI analysis are pursuedFigure 6.1 Questionnaire content checklist.
Data Collection Methods 103However, the steps needed to design and administer these instrumentsapply to Levels 1 and 2 and will not be presented here. Other resourcescover this area quite well.For this level of data collection, observing participants on the job andrecording any changes in behavior and speciﬁc actions taken is an oftenused method. While observation is also used in collecting learning data, afundamental difference is that participants do not necessarily know theyare being observed when observation is used to collect application data.Participant observation is often used in sales and sales support projects.The observer may be a member of the project staff, the participant’smanager, a member of a peer group, or an external resource such asa mystery shopper. The most common observer, and probably the mostpractical one, is a member of the project staff. Technology also lendsitself as a tool to assist with observations. Recorders, video cameras, andcomputers play an important role in capturing application data.Using Action PlansIn some cases, follow-up assignments can be used to develop implemen-tation and application data. A typical follow-up assignment requires theparticipant to meet a goal or complete a task or project by a set date.A summary of the results of the completed assignment provides furtherevidence of the project’s success.The action plan is the most common type of follow-up assignment.With this approach, participants are required to develop action plansas part of the project. Action plans contain the detailed steps necessaryto accomplish speciﬁc objectives related to the project. The process isone of the most effective ways to enhance project support and buildthe sense of ownership needed for successful project application andimplementation.The action plan is typically prepared on a printed form that showswhat is to be done by whom, and by what date the objectives should beaccomplished. The action plan approach is a straightforward, easy-to-usemethod for determining how participants will change their behaviors onthe job and achieve success with project implementation. The approachproduces data that answers questions such as:• What on-the-job improvements have been realized since the projectwas implemented?
104 MEASURING APPLICATION AND IMPLEMENTATIONTable 6.2 Action Planning Checklist❑ Communicate the action plan requirement early❑ Describe the action planning process at the beginning of the project❑ Teach the action planning process❑ Allow time to develop the plan❑ Secure the project leader’s approval of the action plan❑ Require participants to assign a monetary value to each improvement∗❑ Ask participants to isolate the effects of the project∗❑ Ask participants to provide a conﬁdence estimate, when appropriate∗❑ Require action plans to be presented to the group (when possible)❑ Explain the follow-up mechanism❑ Collect action plans at the predetermined follow-up time❑ Summarize the data∗Optional for impact analysis• Are the improvements linked to the project?• What may have prevented participants from accomplishing speciﬁcaction items?Collectively, these data can be used to assess the success of projectimplementation and to make decisions regarding modiﬁcation.The action plan process can be an integral part of project implemen-tation and is not necessarily considered an add-on or optional activity.To gain maximum effectiveness from the evaluation data collected fromaction plans, attempt to implement the steps listed in Table 6.2.Conducting Follow-up SessionsA ﬁnal way to collect application and implementation data is the follow-upsession. A follow-up session is an intended regrouping of the project teamto assess the project status while continuing to move the project forward.Follow-up sessions provide the project team with a forum for discussingthe effectiveness of the project, and they are also ideal for discussingapplication barriers and developing strategies to overcome them.BARRIERS TO APPLICATIONOne of the important reasons for collecting application and implemen-tation data is to uncover barriers and enablers. Although both groups
Application Data Use 105are important, barriers can kill a project. The barriers must be identiﬁedand actions must be taken to minimize, remove, or go around them.Barriers are a serious problem that exists in every project. When theycan be removed or minimized, the project can be implemented. When bar-riers are identiﬁed, they become important reference points for changeand improvement. Typical barriers that will stiﬂe the success of projectsinclude:• My immediate manager does not support the project.• The culture in our work group does not support the project.• We have no opportunity to use the project skills, knowledge, orinformation.• We have no time to implement the project.• Technology was not available for the project.• Our systems and processes did not support the project.• Resources are not available to implement the project.• My job changed and this no longer applies.• The project is not appropriate for our work unit.• We didn’t see a need to implement the project.• Another project got in the way.• Our funding ran out.The important point is to identify any barriers and to use the data inmeaningful ways to make the barriers less of a problem.APPLICATION DATA USEData become meaningless if they are not used properly. As we moveup the chain of impact, the data become more valuable in the minds ofsponsors, key executives, and others who have a strong interest in theproject. Although data can be used in dozens of ways, the following arethe principal uses for data after they are collected:• To report and review results with various stakeholders• To adjust project design and implementation• To identify and remove barriers• To identify and enhance enablers• To recognize individuals who have contributed to project success
106 MEASURING APPLICATION AND IMPLEMENTATION• To reinforce in current and future project participants the value ofdesired actions• To improve management support for projects• To market future projectsFINAL THOUGHTSMeasuring application and implementation is critical in determining thesuccess of a project or program. This essential measure not only deter-mines the success achieved, but also identiﬁes areas where improvementis needed and where success can be replicated in the future. This chapterpresents a variety of techniques to collect application data, ranging fromobservation to use of questionnaires and action plans. The method chosenmust match the scope of the project. Understanding success with applica-tion is important in providing evidence that business needs should be met,but it is only through measurement at Level 4, impact and consequences,that a direct link between the project and business impact can be made.
108 MEASURING BUSINESS IMPACTsolution may inﬂuence the budget and time allocated for the project, aswell as the quality of the project’s success. These measures may be uniqueto the project management solution. Most of this chapter focuses on howthe project is valued, which is often the most critical issue. A portionof the project’s success is then allocated to the project managementsolution, which is contained following one or more techniques in the nextchapter.THE IMPORTANCE OF BUSINESS IMPACTSeveral rationales support the collection of business impact data relatedto a project.Higher-Level DataFollowing the assumption that higher-level data create more value forproject sponsors, business impact measures offer more valuable data.Impact data are the consequence of the application and implementation ofa project. They represent the bottom-line measures positively inﬂuencedwhen a project is successful. For some stakeholders, these are the mostvaluable data.The chain of impact can be broken at Level 4, and this happens in manyprojects. If the project does not drive business impact data—or drivestoo little data when converted to monetary values—to create a positiveROI, then the corresponding results may be less than satisfactory. Inextreme cases, the project can meet with success at the lower levels butfail at Level 4. Participants may react positively to the project; may learnsuccessfully to implement the project; and at Level 3 they may followthe correct implementation steps or use the skills needed to implementthe project. However, when the business impact measure (which isanticipated to be inﬂuenced by the project) does not change, the projectdoes not add value. What could cause this? There are two possibilities.First, the business alignment for the project may not have been completedproperly during the initial analysis, which would keep it from beingthe right solution. Although the project may have been implemented, ithas driven activity and not results. The second possibility is that otherfactors are driving the business measure. Although the project could beconnected to the measure, other inﬂuences may be affecting the businessmeasure in a direction opposite that desired by project planners. So it
The Importance of Business Impact 109may appear at ﬁrst glance that the project has no value, but in realityit could. This brings into focus the importance of isolating the effects ofa project. The business data may be disappointing, but they would beeven more disappointing without the project. The important process ofisolating the effects of the project is presented in Chapter 8.A Business Driver for ProjectsFor most projects, business impact data represent the initial driversfor the project. The problem of deteriorating (or less than expected)performance or the opportunity for improvement of a business measureusually leads to a project. If the business needs deﬁned by businessmeasures are the drivers for a project, then the key measure for evaluatingthe project is the business measure. The extent to which measures havechanged is the principal determinant of project success.‘‘The Money’’ for SponsorsFrom the perspective of the sponsor, business impact data reﬂect keypayoff measures. These are the measures often desired by the sponsorand the ones that the sponsor wants to see changed or improved. Theyoften represent hard, indisputable facts that reﬂect performance thatis critical to the business and operating unit level of the organization.Business impact leads to ‘‘the money’’—to the actual return on investmentin the project. Without credible business impact data linked directly tothe project, it would be difﬁcult, if not impossible, to establish a crediblemonetary value for the project. This makes this level of data collectionone of the most critical.Easy to MeasureOne unique feature of business impact data is that they are often easyto measure. Hard and soft data measures at this level often reﬂect keymeasures that are found in plentiful numbers throughout an organization.It is not unusual for an organization to have hundreds or even thousandsof measures reﬂecting speciﬁc business impact items. The challengeis to connect the objectives of the project to the appropriate businessmeasures. This is more easily accomplished at the beginning of theproject.
110 MEASURING BUSINESS IMPACTCOLLECTING EFFECTIVE IMPACT MEASURESData CategoriesChapter 4 deﬁned four data categories (hard, soft, tangible, and intan-gible). In addition to being classiﬁed as hard or soft and tangible orintangible, data can be categorized at several different levels, as shown inFigure 7.1. The ﬁgure illustrates that some data are considered strategicand are linked to the corporate level of an organization. Other data aremore operational, and are linked to the business unit level. Still othersare considered more tactical in nature and scope, and are used at theoperating level of an organization.Examples of data categorized at the strategic level include ﬁnancial,people-oriented, and internal versus external data. At the business unitlevel, classiﬁcations—such as output, quality, time, cost, job satisfaction,and customer satisfaction—are critical categories. At the tactical level,the categories are more plentiful and include: productivity, efﬁciency, costcontrol, quality, time, attitudes, and individual and team performance.The importance is not in the classiﬁcation of data itself but in the aware-ness of the vast array of data available. Regardless of their categories,these data are consequence measures (Level 4) of project success. Thechallenge is to ﬁnd the data items connected directly to the project.Metric FundamentalsWhen determining the type of measures to use, reviewing metric fun-damentals can be helpful. The ﬁrst important issue is identifying whatmakes an effective measure. Table 7.1 shows some of the criteria ofan effective measure. These are issues that should be explored whenexamining any type of measure.These criteria serve as a screening checklist as measures are con-sidered, developed, and ultimately added to the list of possibilities. Inaddition to meeting criteria, the factual basis of the measure should bestressed. In essence, the measure should be subjected to a fact-basedanalysis, a level of analysis never before applied to decisions about manyprojects, even when these decisions have involved huge sums of money.Distinguishing between the various ‘‘types’’ of facts is beneﬁcial. As shownbelow, the basis for facts ranges from commonsense to what employees‘‘say,’’ to actual data.
112 MEASURING BUSINESS IMPACTTable 7.1 Criteria for Effective MeasuresCriteria: EffectiveMeasures Are . . . Deﬁnition: The Extent to Which a Measure . . .Important Connects to strategically important business objectivesrather than to what is easy to measureComplete Adequately tracks the entire phenomenon rather thanonly part of the phenomenonTimely Tracks at the right time rather than being held to anarbitrary dateVisible Is visible, public, openly known, and tracked by thoseaffected by it, rather than being collected privately formanagement’s eyes onlyControllable Tracks outcomes created by those affected by it who havea clear line of sight from the measure to resultsCost-effective Is efﬁcient to track using existing data or data that areeasy to monitor without requiring new proceduresInterpretable Creates data that are easy to make sense of and thattranslate into employee actionSimplicity Simple to understand from each stakeholder’s perspectiveSpeciﬁc Is clearly deﬁned so that people quickly understand andrelate to the measureCollectible Can be collected with no more effort than is proportionalto the usefulness that resultsTeam-based Will have value in the judgment of a team of individuals,not in the judgment of just one individualCredible Provides information that is valid and credible in theeyes of management(Sources: Adapted from Kerr, Steve, ‘‘On the Folly of Rewarding A, While Hopingfor B,’’ Academy of Management Journal, vol. 18 (1995): 769–783; and Andrew Mayo,Measuring Human Capital. London: The Institute of Chartered Accountants, June2003.)• No facts. Commonsense tells us that employees will be more pro-ductive if they have a stake in the proﬁts of a company.• Unreliable facts. Employees say they are more likely to stay with acompany if they are offered proﬁt sharing.• Irrelevant facts. We have benchmarked three world-class companieswith variable pay plans: a bank, a hotel chain, and a defensecontractor. All reported good results.• Fact-based. Employee turnover in call centers is reducing opera-tional costs.1
Collecting Effective Impact Measures 113ScorecardsIn recent years, interest has increased in developing documents thatreﬂect appropriate measures in an organization. Scorecards like thoseused in sporting events provide a variety of measures for top executives. Intheir landmark book The Balanced Scorecard, Robert Kaplan and DavidNorton explore the concept of the scorecard for use by organizations.2Kaplan and Norton suggest that data can be organized in the fourcategories of process, operational, ﬁnancial, and growth.What exactly is a scorecard? The American Heritage Dictionary deﬁnesa scorecard from two perspectives:1. A printed program or card enabling a spectator to identify playersand record the progress of a game or competition2. A small card used to record one’s own performance in sportsScorecards are varied in type, ranging from Kaplan and Norton’sbalanced scorecard to the scored set in the president’s managementagenda that uses a trafﬁc-light grading system (green for success, yellowfor mixed results, red for unsatisfactory). Top executives place greatemphasis on scorecards, regardless of type. In some organizations, thescorecard concept has ﬁltered down to various functional business units,and each unit of the business has been required to develop a scorecard.A growing number of executives in different functions have developedscorecards to reﬂect their segments of the business.The scorecard approach is appealing because it provides a quick com-parison of key business impact measures and examines the status of theorganization. As a management tool, scorecards can be important in shap-ing and improving or maintaining the performance of the organizationthrough the implementation of preventive projects. Scorecard measuresoften link to particular projects. In many situations, it was a scorecarddeﬁciency measure that initially prompted the project.Identifying Speciﬁc Measures Linked to ProjectsAn important issue that often surfaces when considering ROI applica-tions is the understanding of speciﬁc measures that are often driven byspeciﬁc projects. Although no standard answers are available, Table 7.2represents a summary of typical payoff measures for speciﬁc types ofprojects. The measures are quite broad for some projects. For example,
116 MEASURING BUSINESS IMPACTTable 7.2 (Continued)ROI ApplicationsProject Key Impact MeasurementsTalent management Productivity/output, quality, efﬁciency, cost/timesavings, employee satisfaction, engagementTechnical training Productivity, sales, quality, time, costs, customerservice, turnover, absenteeism, job satisfactionTechnologyimplementationCycle times, error rates, productivity, efﬁciency,customer satisfaction, job satisfactionWellness/ﬁtness Turnover, medical costs, accidents, absenteeisma reward systems project can pay off in a variety of measures, as inimproved productivity, enhanced sales and revenues, improved quality,cycle-time reduction, and even direct cost savings. Essentially, the projectshould drive the measure that the reward is designed to inﬂuence. Inother projects, the inﬂuenced measures are quite narrow. For example,in labor-management cooperation projects, the payoffs are typically inreduced grievances, fewer work stoppages, and improved employee sat-isfaction. Orientation projects typically pay off in measures of earlyturnover (turnover in the ﬁrst ninety days of employment), initial jobperformance, and productivity. The measures that are inﬂuenced dependon the objectives and the design of the project.Table 7.2 also illustrates the immense number of applications of thismethodology and the even larger set of measures that can be drivenor inﬂuenced. In most of these situations, assigning monetary values tothese measures (as the beneﬁts of a given project are compared to thecosts) and developing the ROI become reasonable tasks.A word of caution: Presenting speciﬁc measures linked to a typicalproject may give the impression that these are the only measures inﬂu-enced. In practice, a given project can have many outcomes, and this canmake calculation of the ROI a difﬁcult process. The good news is that mostprojects are driving business measures. The monetary values are basedon what is being changed in the various business units, divisions, regions,and individual workplaces. These are the measures that matter to seniorexecutives. The difﬁculty often comes in ensuring that the connection tothe project exists. This is accomplished through a variety of techniquesto isolate the effects of the project on the particular business measures,as will be discussed in Chapter 8.
Business Performance Data Monitoring 117BUSINESS PERFORMANCE DATA MONITORINGData are available in every organization to measure business perfor-mance. Monitoring performance data enables management to measureperformance in terms of output, quality, costs, time, job satisfaction, cus-tomer satisfaction, and other measures. In determining the source of datain the evaluation, the ﬁrst consideration should be existing databases,reports, and scorecards. In most organizations, performance data will beavailable that are suitable for measuring improvement resulting froma project. If data are not available, additional record-keeping systemswill have to be developed for measurement and analysis. The question ofeconomics surfaces at this point. Is it economical to develop the record-keeping systems necessary to evaluate a project? If the costs will begreater than the expected beneﬁts, developing those systems is pointless.Identify Appropriate MeasuresExisting performance measures should be thoroughly researched to iden-tify those related to the proposed objectives of the project. Often, severalperformance measures are related to the same item. For example, theefﬁciency of a production unit can be measured in several ways:• The number of units produced per hour• The number of units produced on schedule• The percentage of equipment used• The percentage of equipment downtime• The labor cost per unit of production• The overtime required per unit of production• Total unit costEach of these in its own way measures the effectiveness or efﬁciency ofthe production unit. All related measures should be reviewed to determinethose most relevant to the project.Convert Current Measures to Usable OnesOccasionally, existing performance measures will become integrated withother data. Keeping existing performance measures isolated from unre-lated data may be difﬁcult. In these situations, all existing relatedmeasures should be extracted and retabulated to make them more
118 MEASURING BUSINESS IMPACTappropriate for comparison in the evaluation. At times, it may be nec-essary to develop conversion factors. For example, the average numberof new sales orders per month may be presented regularly in the per-formance measures for the sales department. In addition, the sales costsper sales representative may also be presented. However, in evaluatingthe project, the average cost per new sale is needed. The average num-ber of new sales orders and the average number of lost sales per salesrepresentative are required to develop the data necessary for comparison.Develop New MeasuresIn some cases, data needed to measure the effectiveness of a project arenot available, and new data are needed. The project staff must work withthe client organization to develop record-keeping systems, if economicallyfeasible. In one organization, delays of the sales staff in responding tocustomer requests were an issue. This issue was discovered from customerfeedback. The feedback data prompted a project to reduce the responsetime. To help ensure the success of the project, several measures wereplanned, including measuring the actual time to respond to a customerrequest. Initially, this measure was not available. As the project wasimplemented, new software was used to measure the time that elapsedin responding to customer requests.DATA COLLECTION METHODSFor many projects, business data are readily available to be monitored.However, at times, data won’t be easily accessible to the project teamor to the evaluator. Sometimes data are maintained at the individual,work unit, or department level and may not be known to anyone outsidethat area. Tracking down all those data sets may be too expensive andtime-consuming. When this is the case, other data collection methods maybe used to capture data sets and make them available for the evaluator.Three other options described in this book are the use of action plans,performance contracts, and questionnaires.Using Action Plans to Develop Business Impact DataAction plans can capture application and implementation data, as dis-cussed in Chapter 6. They can also be a useful tool for capturing business
Data Collection Methods 119impact data. For business impact data, the action plan is more focusedand credible than using a questionnaire. The basic design principles andthe issues involved in developing and administering action plans are thesame for business impact data as for application and implementationdata. However, a few issues are unique to business impact and ROI, andare presented here. The following steps are recommended when an actionplan is developed and implemented to capture business impact data andto convert the data to monetary values.Set Goals and TargetsAn action plan can be developed with a direct focus on business impactdata. Participants develop an overall objective for the plan, which isusually the primary objective of the project. In some cases, a project mayhave more than one objective, which requires additional action plans. Inaddition to the objective, the improvement measure and the current levelsof performance are identiﬁed. This information requires the participantto anticipate the application and implementation of the project and to setgoals for speciﬁc performances that can be realized.The action plan is completed during project implementation, oftenwith the input, assistance, and facilitation of the project team. Theevaluator or project leader actually approves the plan, indicating thatit meets the requirements of being Speciﬁc, Motivational, Achievable,Realistic, and Time-based (SMART). The plan can be developed in a one-to two-hour time frame and often begins with action steps related to theimplementation of the project. These action steps are Level 3 activitiesthat detail the application and implementation. All these steps buildsupport for and are linked to business impact measures.Deﬁne the Unit of MeasureThe next important issue is to deﬁne the actual unit of measure. In somecases, more than one measure may be used, which will subsequently becontained in additional action plans. The unit of measure is necessary tobreak down the process into the simplest steps so that its ultimate valuecan be determined. The unit may be output data—such as an additionalunit manufactured or package delivered—or it can be sales and marketingdata—such as additional sales revenue or a 1 percent increase in marketshare. In terms of quality, the unit can be one reject, one error, or onedefect. Time-based units are usually measured in minutes, hours, days,
120 MEASURING BUSINESS IMPACTor weeks. Other units are speciﬁc to their particular type of data, such asone grievance, one complaint, one absence, or one less person receivingwelfare payments. The important point is to break down impact data intothe simplest terms possible.Place a Monetary Value on Each ImprovementDuring project implementation, participants are asked to locate, calcu-late, or estimate the monetary value of each improvement outlined intheir plans. The unit value is determined using a variety of methods,including standard values, expert input, external databases, and esti-mates. The process used in arriving at the value is described in theinstructions for the action plan. When the actual improvement occurs,participants use these values to capture the annual monetary beneﬁts ofthe plan. For this step to be effective, it is helpful to understand the waysvalues can be assigned to the data (as discussed in Chapter 9).In the worst-case scenario, participants are asked to calculate thevalues themselves, although use of standard values and consultationwith an expert are better courses of action. When it is necessary forparticipants themselves to make the calculations, they must explain thebasis of them.Implement the Action PlanParticipants implement the action plan during project implementation,which often lasts for weeks or months following the launch of the project.The participants follow action plan steps, and the subsequent businessimpact results are achieved.Provide Speciﬁc ImprovementsAt the end of the speciﬁed follow-up period—usually three months, sixmonths, nine months, or one year—the participants indicate the speciﬁcimprovements made, usually expressed as a daily, weekly, or monthlyamount. This determines the actual amount of change observed, mea-sured, or recorded. Participants must understand the need for accuracy asdata are recorded. In most cases, only the changes are recorded, as theseamounts are needed to calculate the monetary value of the project. Inother cases, before-and-after data may be recorded, allowing the evaluatorto calculate the difference.
Data Collection Methods 121Isolate the Effects of the ProjectAlthough the action plan is initiated because of the project, the actualimprovements reported on the action plan may be inﬂuenced by otherfactors. Consequently, the project should not be given full credit for theimprovement. For example, an action plan to implement a new system ina division could only be given partial credit for a business improvementbecause other variables may have affected the impact measures. Althoughseveral ways are available to isolate the effects of a project, participantestimation is usually most appropriate in the action planning process.Consequently, participants are asked to estimate the percentage of theimprovement actually related to a particular project. This question can beasked on the action plan form or in a follow-up questionnaire. Sometimesit is beneﬁcial to precede this question with a request to identify theentire range of factors that could have inﬂuenced the results. This allowsparticipants to think through the relationships before actually allocatinga portion to this particular project.Provide a Conﬁdence Level for EstimatesThe process to isolate the amount of the improvement actually relatedto the project is not usually precise. Participants are asked to indicatetheir level of conﬁdence in their estimates. Using a scale of 0 to 100percent—where 0 indicates the values are completely false and 100percent indicates the values are absolutely certain—participants have away to express their uncertainty with their estimates.Collect Action Plans at Speciﬁed Time IntervalsBecause excellent high response rate is essential, several steps maybe necessary to ensure the action plans are completed and returned.Participants usually see the importance of the process and develop theirplans in detail at the beginning of the project. Some organizations sendfollow-up reminders by mail or e-mail; others phone participants to checktheir progress. Others offer assistance in developing the ﬁnal plan. Thesesteps may require additional resources, which must be weighed againstthe importance of having more data. Speciﬁc ways to improve responserates are discussed in Chapter 6.Summarize the Data and Calculate the ROIIf developed properly, each action plan should have annualized mone-tary values associated with improvements. Also, each individual should
122 MEASURING BUSINESS IMPACThave indicated the percentage of the improvement directly related to theproject. Finally, participants should have provided a conﬁdence percent-age to reﬂect their uncertainty with the process and the subjective natureof some of the data that may be provided.Because this process involves estimates it may appear to be inaccurate,although certain adjustments during analysis can make the process cred-ible and more accurate. These adjustments reﬂect the guiding principlesthat form the basis of the ROI methodology, as outlined in Table 3.1. Theadjustments are made in ﬁve steps as follows:Step 1: If participants provide no data, assume they had no improve-ment to report. (This is a very conservative approach.)Step 2: Check each value for realism, usability, and feasibility. Discardextreme values and omit them from analysis.Step 3: Because the improvement is annualized, assume the projecthad no improvement after the ﬁrst year (for short-termprojects). (Chapter 9 discusses projects that add value aftertwo and three years.)Step 4: Adjust the improvement calculated in Step 3, using the con-ﬁdence level multiplied by the conﬁdence percentage. Theconﬁdence level is actually a percentage of error suggestedby the participants. For example, a participant indicating80 percent conﬁdence with the process is reﬂecting a possibil-ity of 20 percent error. In a $10,000 estimate with an 80 percentconﬁdence factor, the participant is suggesting a value in therange of $8,000 to $12,000 (i.e., a range between 20 percentless and 20 percent more). To be conservative, use the lowernumber, and multiply the conﬁdence factor by the value of theimprovement.Step 5: Finally, adjust the new values by the percentage of theimprovement related directly to the project, using multipli-cation to isolate the effects of the project.Total the monetary values determined in these ﬁve steps to arrive atthe ﬁnal project beneﬁt. Because these values are already annualized, thetotal of these beneﬁts becomes the annual beneﬁts for the project. Placethis value in the numerator of the ROI formula to calculate the ROI. Theformula is:ROI =Beneﬁts − Costs × 100Costs
Data Collection Methods 123Using Performance Contracts to Measure Business ImpactAnother technique for collecting business impact data is the performancecontract. The performance contract is essentially a slight variation ofthe action plan. Based on the principle of mutual goal setting, a perfor-mance contract is a written agreement between a participant and theparticipant’s manager. The participant agrees to improve performance inan area of mutual concern related to the project. The agreement is inthe form of a goal to accomplish during the project or after the project’scompletion. The agreement details what is to be accomplished, at whattime, and with what results.Although the steps can vary according to the organization and thespeciﬁc kind of contract, a common sequence of events follows:1. The employee (participant) becomes involved in project implemen-tation.2. The participant and his or her immediate manager agree on ameasure or measures for improvement related to the project (What’sin it for me?).3. Speciﬁc, measurable goals for improvement are set, following theSMART requirements discussed earlier.4. In the early stages of the project, the contract is discussed and plansare developed to accomplish the goals.5. During project implementation, the participant works to meet thedeadline set for contract compliance.6. The participant reports the results of the effort to his or hermanager.7. The manager and participant document the results and forward acopy, with appropriate comments, to the project team.The process of selecting the area for improvement is similar to theprocess used in an action plan. The topic can cover one or more of thefollowing areas:• Routine performance related to the project, including speciﬁcimprovement in measures such as production, efﬁciency, and errorrates• Problem solving, focused on such problems as an unexpectedincrease in workplace accidents, a decrease in efﬁciency, or a lossof morale
124 MEASURING BUSINESS IMPACT• Innovative or creative applications arising from the project, whichcould include the initiation of improvements in work practices,methods, procedures, techniques, and processes• Personal development connected to the project, such as learningnew information and acquiring new skills to increase individualeffectivenessThe topic of the performance contract should be stated in terms of oneor more objectives that are• Written• Understandable by all involved• Challenging (requiring an unusual effort to achieve)• Achievable (something that can be accomplished)• Largely under the control of the participant• Measurable and datedThe performance contract objectives are accomplished by followingthe guidelines for action plans presented earlier, and the methods foranalyzing data and reporting progress are essentially the same as thoseused to analyze action plan data.Using Questionnaires to Collect Business Impact MeasuresAs described in the previous chapters, the questionnaire is one of themost versatile data collection tools and can be appropriate for Levels 1,2, 3, and 4 data. Essentially, the design principles and content issues arethe same as at other levels, except that questionnaires developed for abusiness impact evaluation will include additional questions to capturethose data speciﬁc to business impact.The use of questionnaires for impact data collection brings both goodnews and bad news. The good news is that questionnaires are easy toimplement and low in cost. Data analysis is very efﬁcient, and the timerequired to provide the data is often minimal, making questionnairesamong the least disruptive of data collection methods. The bad news isthat the data can be distorted and inaccurate, and are sometimes missing.The challenge is to take all the steps necessary to ensure that question-naires are complete, accurate, and clear, and that they are returned.Unfortunately, questionnaires are the weakest of methods of datacollection. Paradoxically, they are the most commonly used because of
Data Collection Methods 125their advantages. Of the ﬁrst 100 case studies published on the ROImethodology, roughly 50 percent used questionnaires as a method of datacollection. They are popular, convenient, low-cost, and have become away of life. The challenge is to improve them. The philosophy in theROI methodology is to take processes that represent the weakest methodand make them as credible as possible. Here the challenge is to makequestionnaires credible and useful by ensuring that they collect all thedata that is needed, that participants provide accurate and complete data,and that return rates are in the 70 to 80 percent range.The reason return rates must be high is explained in Guiding Principle6 of the ROI methodology outlined in Table 3.1: No data, no improve-ment. If an individual provides no improvement data, it is assumed thatthe person had no improvement to report. This is a very conservativeprinciple but necessary to bring the credibility needed. Consequently,using questionnaires will require effort, discipline, and personal atten-tion to ensure proper response rates. Chapter 5 presented suggestionsfor ensuring high response rates for Level 1 data collection. The sametechniques should be considered here. It is helpful to remember that thisis the least preferred method for collecting Level 4 data, and it is usedonly when other methods don’t work (i.e., when business performancedata cannot be easily monitored, when action plans are not feasible, orwhen performance contracting is not suitable).Selecting the Appropriate Method for Each LevelThe data collection methods presented in this and earlier chapters offer awide range of opportunities for collecting data in a variety of situations.Eight aspects of data collection should be considered when deciding onthe most appropriate method of collecting any type of data.Type of DataOne of the most important issues to consider when selecting the methodis the type of data to be collected. Some methods are more appropri-ate for business impact. Follow-up surveys, observations, interviews,focus groups, action planning, and performance contracting are bestsuited—sometimes exclusively—for application data. Performance mon-itoring, action planning, and questionnaires can easily capture businessimpact data.
126 MEASURING BUSINESS IMPACTInvestment of Participants’ TimeAnother important factor when selecting the data collection method isthe amount of time participants must spend with data collection andevaluation systems. Time requirements should always be minimized, andthe method should be positioned so that it is a value-added activity. Par-ticipants must understand that data collection is a valuable undertaking,and not an activity to be resisted. Sampling can be helpful in keepingtotal participant time to a minimum. Methods like performance moni-toring require no participant time, whereas others, such as conductinginterviews and focus groups, require a signiﬁcant investment in time.Investment of Managers’ TimeThe time that a project participant’s manager must allocate to datacollection is another issue in method selection. This time requirementshould always be minimized. Methods like performance contracting mayrequire signiﬁcant involvement from the manager before and after projectimplementation, whereas other methods, such as participants’ completionof a questionnaire, may not require any manager time.Cost of MethodCost is always a consideration when selecting the method. Some data col-lection methods are more expensive than others. For example, interviewsand observations are very expensive, whereas surveys, questionnaires,and performance monitoring are usually inexpensive.Disruption of Normal Work ActivitiesThe issue that generates perhaps the greatest concern among managersis the degree of work disruption that data collection will create. Routinework processes should be disrupted as little as possible. Data collectiontechniques like performance monitoring require very little time and causelittle distraction from normal activities. Questionnaires generally do notdisrupt the work environment and can often be completed in just a fewminutes, perhaps even after usual work hours. At the other extreme, tech-niques, such as the focus group and interview, may disrupt the work unit.Accuracy of MethodThe accuracy of the technique is another factor to consider when selectingthe method. Some data collection methods are more accurate than others.
Measuring the Hard to Measure 127For example, performance monitoring is usually very accurate, whereasquestionnaires are subject to distortion and may be unreliable. If on-the-job behavior must be captured, observation is clearly one of the mostaccurate methods. There is often a trade-off in the accuracy and costs ofa method.Utility of an Additional Method (Source or Time Frame)Because many different methods to collect data exist, using too manymethods is tempting. Multiple data collection methods add time and costto the evaluation, and may result in very little added value. Utility refersto the value added by each additional data collection method. When morethan one method is used, this question should always be addressed: Doesthe value obtained from the additional data warrant the extra time andexpense of the method? If the answer is no, the additional method shouldnot be implemented. The same issue must be addressed when consideringmultiple sources and time frames.Cultural Bias of Data Collection MethodThe culture or philosophy of the organization can dictate which datacollection methods are best to use. For example, if an organization oraudience is accustomed to using questionnaires, they will work wellwithin the culture of that organization. As another example, some orga-nizations will not use observation because their culture will not supportthe potential invasion of privacy.MEASURING THE HARD TO MEASUREThe focus of this chapter is on capturing the measures that are easy tocollect and easy to measure. These represent the classic deﬁnitions ofhard data and soft data—or, tangible data and intangible data. Muchattention today is focused on the very hard to measure—on some of theclassic soft items that are even softer than customer satisfaction and jobsatisfaction. Although this subject is discussed at length in Chapter 10,Measuring the Intangibles, a few comments are appropriate here.Everything Can Be MeasuredContrary to the thinking of some professionals, everything can bemeasured. Any item, issue, or phenomenon that is important to an
128 MEASURING BUSINESS IMPACTorganization can be measured. Even images, perceptions, and ideas in aperson’s mind can be measured. The thorny issue is usually in identifyingthe best way and the best resources to do the measuring. Although theimage of an organization in the community or the way that customersbecome aware of a brand can be measured accurately, doing so takestime and money.A case in point is the project launched by Nissan Motor Company inthe 1980s when it located its ﬁrst auto manufacturing plant in NorthAmerica. Nissan executives were very concerned about how a Japaneseautomaker would be regarded in a traditional Southern community. (Thiscame at a time when common attitudes toward Japanese automakerswere more hostile than today.) The project involved surveying in thecommunities that would host a Nissan plant. The results were impressive,and demonstrated that you can measure anything if you can deﬁne it andtake the time to measure it.Perceptions Are ImportantSome soft, or intangible, items are not based on perceptions, but othersare. For example, consider innovation. An important component of theinnovations in a company is image or perception. Also, some very clearmeasures determine how innovative the company is able to be in itsprocesses, products, and services, e.g., number of new patents, numberof new products. However, concepts like brand awareness are basedstrictly on perception, i.e., on what a person knows or perceives aboutan item, product, or service. In the past, perceptions were consideredirrelevant and not very valuable, but today many decisions are basedon perceptions. For example, consider that perceptions about servicequality from the customer’s viewpoint often drive tremendous changes.Employees’ perceptions of their employer often drive huge investmentsin projects to improve job satisfaction, organizational commitment, andengagement. Therefore, perceptions are very important and must be partof the measurement plan for the hard to measure.FINAL THOUGHTSBusiness impact data are critical to address an organization’s businessneeds. These data lead the evaluation to the ‘‘money.’’ Although perceivedas difﬁcult to ﬁnd, business impact data are readily available and very
Final Thoughts 129credible. After describing the types of data that reﬂect business impact,this chapter provides an overview of several data collection approachesthat can be used to capture business data. Some methods are gaininggreater acceptance for use in capturing impact data. Performance moni-toring, follow-up questionnaires, action plans, and performance contractsare used regularly to collect data for an impact analysis. This chapterfocuses on methods to collect data on project impact and consequences.Linking these consequences directly to the project requires the importantstep of isolating the effects of the project, a topic discussed in Chapter 8.
132 ISOLATION OF PROJECT IMPACTRealityIsolating the effects of projects on business measures has led to someimportant conclusions. First, other inﬂuences are almost always present.In almost every situation, multiple factors generate business results. Therest of the world does not stand still while a project is being implemented.Other processes are also operating to improve the same metrics targetedby the implemented project.Next, if the project effects are not isolated, no business link can beestablished. Without steps taken to document the project’s contribution,there is no proof that the project actually inﬂuenced the measures. Theevidence will show only that the project might have made a difference.Results have improved, but other factors may have inﬂuenced the data.Also, the outside factors and inﬂuences have their own protectiveowners. These owners will insist that it was their processes that madethe difference. Some of them will probably be certain that the resultsare due entirely to their efforts. They may present a compelling case tomanagement, stressing their achievements.Finally, isolating the effects of the project on impact data is a chal-lenging task. For complex projects in particular, the process is not easy,especially when strong-willed owners of other processes are involved. For-tunately, a variety of approaches are available to facilitate the procedure.MythsThe myths surrounding the isolation of project effects create confusionand frustration with the process. Some researchers, professionals, andconsultants go so far as to suggest that such isolation is not necessary.Here are the most common myths:1. Our project is complementary to other processes; therefore,we should not attempt to isolate the effects of the project.A project often complements other factors at work, all of whichtogether drive results. If the sponsor of a project needs to understandits relative contribution, the isolation process is the only way to doit. If accomplished properly, it will reveal how the complementaryfactors interact to drive improvements.2. Other project leaders do not address this issue. Some projectleaders do not grapple with the isolation problem because they wish
The Importance of This Issue 133to make a convincing case that all of the improvement is directlyrelated to their own processes.3. If we cannot use a research-based control group, we shouldnot attempt this procedure. Although an experimental researchdesign using randomly selected control and experimental groupsis the most reliable approach to identifying effects and causes, itis inapplicable to the majority of situations. Consequently, othermethods must be used to isolate the effects of a project. The chal-lenge is to ﬁnd a method that is effective and whose results arereproducible, even if it is not as credible as the group comparisonmethod.4. The stakeholders will understand the link to business impactmeasures; therefore, we do not need to attempt to isolate theeffects of the project. Unfortunately, stakeholders try to under-stand only what is presented to them. The absence of informationmakes it difﬁcult for them to understand the business links, partic-ularly when others are claiming full credit for the improvement.5. Estimates of improvement provide no value. It may be neces-sary to tackle the isolation process using estimates from those whounderstand the process best. Although this should be done only asa last alternative, it can provide value and credibility, particularlyonce the estimates have been adjusted for error in order to reducesubjectivity.6. Ignore the issue; maybe the others won’t think about it.Unfortunately, audiences are becoming more sophisticated on thistopic, and they are aware of the presence of multiple inﬂuences. Ifno attempt is made to isolate the effects of the project, the audiencewill assume that the other factors have had a major effect, perhapsthe only effect. A project’s credibility can deteriorate quickly. Onehas only to look at the recent business literature to understand whyisolating project effects is so important.These myths underscore the importance of tackling this issue. Theemphasis on isolation is not meant to suggest that a project is imple-mented independently and exclusively of other processes. Obviously,all groups should be working as a team to produce the desired results.However, when funding is parceled among different functions ororganizations—with different owners—there is always a struggle toshow, and often to understand, the connection between their activities
134 ISOLATION OF PROJECT IMPACTand the results. If you do not undertake this process, others will—leavingyour project with reduced budgets, resources, and respect.Project versus Project Management SolutionIsolating the effects of a particular factor allows the project manager toshow the value of a project management solution, as well as the differentfactors that caused the project success. Essentially, the project’s successmay be inﬂuenced by a variety of factors—some internal to the project,some external. In addition, there could be a variety of project manage-ment solutions implemented, such as project management training, asystematic project management methodology, systems and technology,or a project management ofﬁce. The effects of these solutions can beisolated from the other factors. One approach is to sort out the differentfactors that caused the project’s success and include in those factors theproject management solution. This shows the effect of that solution onthe project’s success. However, when a solution is implemented acrossmany projects, the issue may be focused on the impact of the projectmanagement solution across all projects. In this case, the project man-agement solution should have an impact on the time spent on the project(early delivery), budgets (less cost), and quality of the project (usually aqualitative measure). The techniques in this chapter will show how tosort out the effects of a project management solution on these three areas,which would be considered impact measures connected to the projectmanagement solution. Most of the presentation is focused on isolatingthe different factors that caused the project’s success. The easiest andmost credible approach is to consider the project management solutionone of those factors.PRELIMINARY ISSUESThe cause-and-effect relationship between a project and performance canbe confusing and difﬁcult to prove, but it can be demonstrated with anacceptable degree of accuracy. The challenge is to develop one or morespeciﬁc techniques to isolate the effects of the project early in the process,usually as part of an evaluation plan conducted before the project begins.Up-front attention ensures that appropriate techniques will be used withminimal cost and time commitments. Two important issues in isolatingthe effects of a project are covered next, followed by speciﬁc methods.
Preliminary Issues 135Chain of ImpactBefore presentation of the methods, examining the chain of impactimplicit in the different levels of evaluation will be helpful. Measur-able results from a project should be derived from the application of theproject (Level 3 data). Successful application of the project should stemfrom project participants learning to do something different, somethingnecessary to implement the project (Level 2 data). Successful learning willusually occur when project participants react favorably to the project’scontent and objectives (Level 1 data). Without this preliminary evidence,isolating the effects of a project is difﬁcult.To be sure, if there is an adverse reaction, no learning, or no application,it cannot be concluded that any business impact improvements werecaused by the project. From a practical standpoint, this requires datacollection at four levels for an ROI calculation (Guiding Principle 1 inTable 3.1). Although this requirement is a prerequisite to isolating theeffects of a project, it does not establish a direct connection, nor does itpinpoint the extent of the improvement caused by the project. It doesshow, however, that without improvements at previous levels, making aconnection between the ultimate outcome and the project is difﬁcult orimpossible.Identify Other Factors: A First StepAs a ﬁrst step in isolating a project’s impact on performance, all key factorsthat may have contributed to the performance improvement should beidentiﬁed. This step communicates to interested parties that other factorsmay have inﬂuenced the results, underscoring that the project is not thesole source of improvement. Consequently, the credit for improvement isshared among several possible variables and sources—an approach thatis likely to garner the respect of the client. Several potential sources areavailable for identifying major inﬂuencing variables:• The sponsor• Participants in the project• The project team• The immediate managers of participants• Subject matter experts• Other process owners
136 ISOLATION OF PROJECT IMPACT• Experts on external issues• Middle and top managementThe importance of identifying all of the factors is underscored byan example. A large Canadian bank has a sophisticated system foridentifying the reasons customers make product decisions. At the pointof sale, the purchaser records the reasons for the sale: Was it the price,the product design, the advertising, or the referral from a satisﬁedcustomer? This system, owned by the marketing department, is designedto isolate the factors underlying the success of various marketing projects.However, it omits factors outside marketing. In essence, it assumes that100 percent of the improvement in product sales can be attributed to somemarketing inﬂuence. It ignores the effect of the economy, competition,information technology, reward systems, learning and development, jobdesign, and other factors that could have had an important inﬂuence.Without identifying all the factors, the credibility of the analysis willsuffer. Thus, competing factions within that organization had to addresschanging the system so that other factors are considered in the analysis.Taking the time to focus on outside variables that may have inﬂuencedperformance adds accuracy and credibility to the process. Project teamleaders should go beyond this initial step and use one or more of thefollowing techniques to isolate the impact of the project.ISOLATION METHODSJust as there are many data collection methods available for collectingdata at different levels, a variety of methods are also available to isolatethe effects of a project.Control GroupsThe most accurate approach for isolating the impact of a project is anexperimental design with control groups. This approach involves theuse of an experimental group that goes through the implementation ofthe project and a control group that does not. The two groups shouldbe as similar in composition as possible and, if feasible, participantsfor each group should be randomly selected. When this is achievableand the groups are subjected to the same environmental inﬂuences, anydifference in performance between the two groups can be attributed to theproject.
Isolation Methods 137Control group(project is not implemented) MeasurementExperimental group(project is implemented)ProjectNo projectMeasurementFigure 8.1 Use of control groups.As illustrated in Figure 8.1, the control group and experimental groupdo not necessarily require pre-project measurements. Measurements canbe taken during the project and after the project has been implemented,with the difference in performance between the two groups indicating theamount of improvement that is directly related to the project.One caution should be observed: The use of control groups may cre-ate the impression that the project leaders are reproducing a laboratorysetting, which can cause a problem for some executives and administra-tors. To avoid this perception, some organizations conduct a pilot projectusing participants as the experimental group. A similarly constitutednonparticipant comparison group is selected but does not receive anycommunication about the project. The terms pilot project and comparisongroup are a little less threatening to executives than experimental groupand control group.The control group approach has some inherent challenges that canmake it difﬁcult to apply in practice. The ﬁrst major challenge is theselection of the groups. From a theoretical perspective, having identicalcontrol and experimental groups is next to impossible. Dozens of factorscan affect performance, some individual and others contextual. On apractical basis, it is best to select the four to six variables that willhave the greatest inﬂuence on performance. Essentially, this involves the80/20 rule or the Pareto principle. The 80/20 rule is aimed at selectingthe factors that might account for 80 percent of the difference. The Paretoprinciple requires working from the most important factor down to coverperhaps four or ﬁve issues that capture the vast majority of the factorshaving inﬂuence.Another major challenge is that the control group process is not suitedto many situations. For some types of projects, withholding the projectfrom one particular group while implementing it with another may notbe appropriate. This is particularly true where critical solutions areneeded immediately; management is typically not willing to withhold a
138 ISOLATION OF PROJECT IMPACTsolution from one area to see how it works in another. This limitationkeeps control group analyses from being implemented in many situations.However, in practice, many opportunities arise for a natural control groupto develop even in situations where a solution is implemented throughoutan organization. If it takes several months for the solution to encompasseveryone in the organization, enough time may be available for a parallelcomparison between the initial group and the last group to be affected. Inthese cases, ensuring that the groups are matched as closely as possible iscritical. Such naturally occurring control groups can often be identiﬁed inthe case of major enterprise-wide project implementations. The challengeis to address this possibility early enough to inﬂuence the implementationschedule to ensure that similar groups are used in the comparison.Another challenge is contamination, which can develop when partic-ipants involved in the project group (experimental group) communicatewith people in the control group. Sometimes, the reverse situation occurs,where members of the control group model the behavior of the projectgroup. In either case, the experiment becomes contaminated as the inﬂu-ence of the project is carried over to the control group. This hazard can beminimized by ensuring that the control and project groups are at differentlocations, are on different shifts, or occupy different ﬂoors of the samebuilding. When this is not possible, it should be explained to both groupsthat one group will be involved in the project now and the other will beinvolved at a later date. Appealing to participants’ sense of responsibilityand asking them not to share information with others may help preventthis problem.A closely related issue involves the passage of time. The longer acontrol versus experimental group comparison operates, the greater thelikelihood that other inﬂuences will affect the results; more variableswill enter into the situation, contaminating the results. On the otherend of the scale, enough time must pass to allow a clear pattern toemerge distinguishing the two groups. Thus, the timing of control groupcomparisons must strike a delicate balance between waiting long enoughfor performance differences to show, but not so long that the resultsbecome contaminated.Still another problem occurs when the different groups function underdifferent environmental inﬂuences. This is usually the case when groupsare at different locations. Sometimes the selection of the groups canprevent this problem from occurring. Another tactic is to use more groups
Isolation Methods 139than necessary and discard those groups that show some environmentaldifferences.A ﬁnal problem is that the use of control and experimental groupsmay appear too research oriented for most business organizations. Forexample, management may not want to take the time to experiment beforeproceeding with a project, in addition to the selective withholding problemdiscussed earlier. Because of these concerns, some project managers willnot entertain the idea of using control groups.Because the use of control groups is an effective approach for isolatingimpact, it should be considered when a major ROI impact study isplanned. In these situations, isolating the project impact with a highlevel of accuracy is essential, and the primary advantage of the controlgroup process is accuracy.Trend Line AnalysisAnother useful technique for approximating the impact of a project istrend line analysis. In this approach, a trend line is drawn to projectthe future, using previous performance as a base. When the project isfully implemented, actual performance is compared with the trend lineprojection. Any improvement in performance beyond what the trend linepredicted can be reasonably attributed to project implementation. Whilethis is not a precise process, it can provide a reasonable estimate ofthe project’s impact.Figure 8.2 shows a trend line analysis from the delivery ﬂeet of afood distribution company. The vertical axis reﬂects the level of fuelconsumption per day, per truck. The horizontal axis represents time.Data reﬂect conditions before and after a fuel savings scheduling projectwas implemented in October. As shown in the ﬁgure, an upward trendfor the data began prior to project implementation. However, the projectapparently had a dramatic effect on fuel consumption as the trend lineis much greater than the return. Project leaders may have been temptedto measure the improvement by comparing the one-year average for con-sumption prior to the project (55 percent) to the one-year average afterthe project (35 percent), which would yield a twenty-gallon difference.However, this approach understates the improvement because the mea-sure in question is moving in the wrong direction and the project turns itin the right direction.
140 ISOLATION OF PROJECT IMPACT8070605040302010•• ••••FuelConsumptionsGallonsPerDayPerTruckO N D J F M A M J J A S O N D J F M A M J J A S O7521AveragePre-Project 55Fuel SavingsSchedulingProjectAveragePost-Project 35• ••• • • •• ••••••••• ••Fuel Consumption Per TruckFigure 8.2 Trend line analysis for fuel savings project.A more accurate comparison is actual value after the project (the lasttwo months) versus the trend line value for the same period, a differenceof 54 (75 – 21). Using this measure increases the accuracy and credibilityof the process in terms of isolating the project’s impact.To use this technique, two conditions must be met:• It can be assumed that the trend that developed prior to the projectwould have continued if the project had not been implemented toalter it. In other words, had the project not been implemented, thistrend would have continued on the same path. The process owner(s)should be able to provide input to conﬁrm this assumption. If theassumption does not hold, trend line analysis cannot be used. If theassumption is a valid one, the second condition is considered.• No other new variables or inﬂuences entered the process duringproject implementation. The key word here is new; the understand-ing is that the trend has been established from the inﬂuencesalready in place, and no additional inﬂuences have entered theprocess beyond the project. If such is not the case, another methodwill have to be used. Otherwise, the trend line analysis presents areasonable estimate of the impact of this project.Pre-project data must be available in order for this technique to beused, and the data should show a reasonable degree of stability. If thevariance of the data is high, the stability of the trend line will be anissue. If the stability cannot be assessed from a direct plot of the data,more detailed statistical analyses can be used to determine if the data
Isolation Methods 141are stable enough to allow a projection. The trend line can be projecteddirectly from historical data using a simple routine that is available inmany calculators and software packages, such as Microsoft Excel.A primary disadvantage of the trend line approach is that it is notalways accurate. This approach assumes that the events that inﬂuencedthe performance variable prior to project implementation are still in place,except for the effects of the implementation (i.e., the trends establishedprior to the project will continue in the same relative direction). Also, itassumes that no new inﬂuences entered the situation during the courseof the project. This may not be the case.The primary advantage of this approach is that it is simple andinexpensive. If historical data are available, a trend line can quickly bedrawn and the differences estimated. While not exact, it does provide aquick general assessment of project impact.Forecasting MethodsA more analytical approach to trend line analysis is the use of forecastingmethods that predict a change in performance variables. This approachrepresents a mathematical interpretation of the trend line analysis whenother variables enter the situation at the time of implementation. Withthis approach, the output measure targeted by the project is forecastbased on the inﬂuence of variables that have changed during the imple-mentation or evaluation period for the project. The actual value of themeasure is compared with the forecast value, and the difference reﬂectsthe contribution of the project.A major disadvantage approach emerges when several variables enterthe process. The complexity multiplies, and the use of sophisticatedstatistical packages designed for multiple variable analyses is necessary.Even with this assistance, however, a good ﬁt of the data to the model maynot be possible. Unfortunately, some organizations have not developedmathematical relationships for output variables as a function of oneor more inputs, and without them the forecasting method is difﬁcultto use.EstimatesThe most common method of isolating the effects of a project is to useestimates from some group of individuals. Although this is the weakest
142 ISOLATION OF PROJECT IMPACTmethod, it is practical in many situations and it can enhance the cred-ibility of the analysis if adequate precautions are taken. The beginningpoint in using this method is ensuring that the estimates are providedby the most reliable source, which is often the participant—not a higher-level manager or executive removed from the process. The individualwho provides this information must understand the different factorsand, particularly, the inﬂuence of the project on those factors. Essen-tially, there are four categories of input. In addition to the participantsdirectly involved in the project, customers provide credible estimates inthe particular situations where they are involved. External experts maybe very helpful, and managers are another possible source. These are alldescribed next.Participants’ Estimate of ImpactAn easily implemented method of isolating the impact of a project is toobtain information directly from participants during project implemen-tation. The usefulness of this approach rests on the assumption thatparticipants are capable of determining or estimating how much of a per-formance improvement is related to the project implementation. Becausetheir actions have led to the improvement, participants may providehighly accurate data. Although an estimate, the value they supply islikely to carry considerable weight with management because they knowthat the participants are at the center of the change or improvement.The estimate is obtained by deﬁning the improvement and then askingparticipants the series of questions in Table 8.1.Participants who do not provide answers to the questions listed inTable 8.1 are excluded from the analysis. Erroneous, incomplete, andextreme information should also be discarded before the analysis. Toobtain a conservative estimate, the conﬁdence percentage can be factoredinto each of the values. The conﬁdence percentage is a reﬂection of theerror in the estimate. Thus, an 80 percent conﬁdence level equates to apotential error range of plus or minus 20 percent. In this approach, theestimate is multiplied by the level of conﬁdence using the lower side ofthe range.An example will help describe the situation. In an effort to increaserecycling in the community, three actions were taken. Recycling hadbeen available but because of the apathy of the community, the incon-venience with the location, and a lack of incentive to do it, the results
Isolation Methods 143Table 8.1 Questions for Participant EstimationWhat is the link between these factors and the improvement?What other factors have contributed to this improvement in performance?What percentage of this improvement can be attributed to the implementationof this project?How much conﬁdence do you have in this estimate, expressed as a percentage?(0% = no conﬁdence, 100% = complete conﬁdence)What other individuals or groups could provide a reliable estimate of thispercentage to determine the amount of improvement contributed by thisproject?were not acceptable. The community implemented three new approaches.One approach was to conduct awareness sessions in the schools, neigh-borhoods, community groups, and churches to make people aware of therecycling program and what it means to them and the environment.In addition, recycling was made more convenient so it was easier forresidents to conserve. Essentially, they could place three different con-tainers on the street and have them picked up. In addition, when citizensparticipated in recycling, a discount was provided to their regular wastemanagement bill. With these three services implemented, it was impor-tant to understand the effects of each of the processes. On a questionnaire,a sample of participants were asked to allocate the percentage that eachof these services led to their increased participation.As well, the participants were told the amount of increase in recyclingvolume (a fact), and they were asked to indicate if other factors couldhave caused this increase in addition to the three processes. Residentsmentioned only a few other processes. Table 8.2 shows one participant’sresponse. In the example, the participant allocates 60 percent of theimprovement to the awareness program and has a level of conﬁdence inthe estimate of 80 percent.The conﬁdence percentage is multiplied by the estimate to producea usable project value of 48 percent. This adjusted percentage is thenmultiplied by the actual amount of the improvement in recycling volume(post-project minus pre-project value) to isolate the portion attributed tothe project. For example, if volume increased by 50 percent, 24 percentwould be attributed to the awareness program. The adjusted improvementis now ready for conversion to monetary value and, ultimately, use in theROI calculation.
144 ISOLATION OF PROJECT IMPACTTable 8.2 Example of a Participant’s EstimationFact: Recycling volume has increased by 50 percentAdjusted % ofFactor That Percentage of ImprovementInﬂuenced Improvement Conﬁdence CausedImprovement Caused by Project Expressed as a % by ProjectGreen awareness 60% 80% 48%Convenience forparticipation15% 70% 10.5%Discounts forparticipating20% 80% 16%Other 5% 60% 3%Total 100%Although the reported contribution is an estimate, this approach isassociated with considerable accuracy and credibility. Five adjustmentsare effectively applied to the participant estimate to produce a conserva-tive value:1. Participants who do not provide usable data are assumed to haveobserved no improvements.2. Extreme data values and incomplete, unrealistic, or unsupportedclaims are omitted from the analysis, although they may be includedin the ‘‘other beneﬁts’’ category.3. For short-term projects, it is assumed that no beneﬁts are realizedfrom the project after the ﬁrst year of full implementation. For long-term projects, several years may pass after project implementationbefore beneﬁts are realized.4. The amount of improvement is adjusted by the portion directlyrelated to the project, expressed as a percentage.5. The improvement value is multiplied by the conﬁdence level,expressed as a percentage, to reduce the amount of the improvementin order to reﬂect the potential error.As an enhancement of this method, the level of management abovethe participants may be asked to review and approve the participantestimates.
Isolation Methods 145In using participants’ estimates to measure impact, several assump-tions are made:1. The project encompasses a variety of different activities, practices,and tasks all focused on improving the performance of one or morebusiness measures.2. One or more business measures were identiﬁed prior to the projectand have been monitored since the implementation process. Datamonitoring has revealed an improvement in the business measure.3. There is a need to associate the project with a speciﬁc amount ofperformance improvement and determine the monetary impact ofthe improvement. This information forms the basis for calculatingthe actual ROI.Given these assumptions, the participants can specify the resultslinked to the project and provide data necessary to develop the ROI. Thiscan be accomplished using a focus group, an interview, or a questionnaire.Manager’s Estimate of ImpactIn lieu of, or in addition to, participant estimates, the participants’manager may be asked to provide input concerning the project’s role inimproving performance. In some settings, the managers may be morefamiliar with the other factors inﬂuencing performance and thereforemay be better equipped to provide estimates of impact. The questions toask managers, after identifying the improvement ascribed to the project,are similar to those asked of the participants.Managers’ estimates should be analyzed in the same manner as theparticipant estimates, and they may also be adjusted by the conﬁdencepercentage. When participants’ and managers’ estimates have both beencollected, the decision of which estimate to use becomes an issue. If there isa compelling reason to believe that one estimate is more credible than theother, then that estimate should be used. The most conservative approachis to use the lowest value and include an appropriate explanation. Anotheroption is to recognize that each estimate source has a unique perspectiveand that an average of the two may be appropriate, with equal weightplaced on each input. It is recommended that input be obtained from bothparticipants and their managers.In some cases, upper management may provide an estimate of thepercentage of improvement attributable to a project. After considering
146 ISOLATION OF PROJECT IMPACTother factors that could contribute to the improvement—such as tech-nology, procedures, and process changes—they apply a subjective factorto represent the portion of the results that should be attributed to theproject. Despite its subjective nature, this input by upper management isusually accepted by the individuals who provide or approve funding forthe project. Sometimes, their comfort level with the processes used is themost important consideration.Customer’s Estimate of Project ImpactAn approach that is useful in some narrowly focused project situationsis to solicit input on the impact of a project directly from customers.Customers are asked why they chose a particular product or serviceor are asked to explain how their reaction to the product or servicehas been inﬂuenced by individuals or systems involved in the project.This technique often focuses directly on what the project is designed toimprove. For example, after the implementation of a customer serviceproject involving an electric utility, market research data showed thatthe level of customer dissatisfaction with response time was 5 percentlower when compared with the rate before the project implementation.Because response time was reduced by the project and no other factorwas found to contribute to the reduction, the 5 percent improvement incustomer satisfaction was attributed to the project.Routine customer surveys provide an excellent opportunity to col-lect input directly from customers concerning their reactions to new orimproved products, services, processes, or procedures. Pre- and post-project data can pinpoint the improvements spurred by a new project.Customer input should be elicited using current data collection meth-ods; the creation of new surveys or feedback mechanisms is to be avoided.This measurement process should not add to the data collection systemsin use. Customer input may constitute the most powerful and convincingdata if it is complete, accurate, and valid.Expert’s Estimate of ImpactExternal or internal experts can sometimes estimate the portion of resultsthat can be attributed to a project. With this technique, experts must becarefully selected based on their knowledge of the process, project, andsituation. For example, an expert in quality might be able to provide
Isolation Methods 147estimates of how much change in a quality measure can be attributed toa quality project and how much can be attributed to other factors.This approach has its drawbacks, however. It can yield inaccurate dataunless the project and the setting in which the estimate is made arequite similar to the project with which the expert is familiar. Also, thisapproach may lack credibility if the estimates come from external sourcesand do not involve those close to the process.This process has the advantage that its reliability is often a reﬂectionof the reputation of the expert or independent consultant. It is a quickand easy form of input from a reputable expert or consultant. Sometimestop management has more conﬁdence in such external experts than in itsown staff.Estimate Credibility: The Wisdom of CrowdsThe following story is a sample of the tremendous amount of researchshowing the power of input from average individuals. It is taken fromJames Surowicki’s best-selling book, The Wisdom of Crowds.One day in the fall of 1906, British scientist Francis Galton left hishome in the town of Plymouth and headed for a country fair. Galton was85 years old and was beginning to feel his age, but he was still brimmingwith the curiosity that had won him renown—and notoriety—for hiswork on statistics and the science of heredity. On that particular day,Galton’s curiosity turned to livestock.Galton’s destination was the annual West of England Fat Stock andPoultry Exhibition, a regional fair where the local farmers and towns-people gathered to appraise the quality of each other’s cattle, sheep,chickens, horses, and pigs. Wandering through rows of stalls examiningworkhorses and prize hogs may seem like a strange way for a scientistto spend an afternoon, but there was certain logic to it. Galton was aman obsessed with two things: the measurement of physical and mentalqualities and breeding. And what, after all, is a livestock show but a largeshowcase for the effects of good and bad breeding?Breeding mattered to Galton because he believed that only a very fewpeople had the characteristics necessary to keep societies healthy. He haddevoted much of his career to measuring those characteristics, in fact, inan effort to prove that the vast majority of people did not possess them.His experiments left him with little conﬁdence in the intelligence of theaverage person, ‘‘the stupidity and wrong-headedness of many men and
148 ISOLATION OF PROJECT IMPACTwomen being so great as to be scarcely credible.’’ Galton believed, ‘‘Only ifpower and control stayed in the hands of the select, well-bred few, coulda society remain healthy and strong.’’As he walked through the exhibition that day, Galton came across aweight judging competition. A fat ox had been selected and put on display,and many people were lining up to place wagers on what the weight ofthe ox would be after it was slaughtered and dressed. For sixpence, anindividual could buy a stamped and numbered ticket and ﬁll in his or hername, occupation, address, and estimate. The best guesses would earnprizes.Eight hundred people tried their luck. They were a diverse lot. Many ofthem were butchers and farmers, who were presumably expert at judgingthe weight of livestock, but there were also quite a few people who hadno insider knowledge of cattle. ‘‘Many non-experts competed,’’ Galtonwrote later in the scientiﬁc journal Nature. ‘‘The average competitor wasprobably as well ﬁtted for making a just estimate of the dressed weightof the ox, as an average voter is of judging the merits of most politicalissues on which he votes.’’Galton was interested in ﬁguring out what the ‘‘average voter’’ wascapable of because he wanted to prove that the average voter was capableof very little. So he turned the competition into an impromptu experiment.When the contest was over and the prizes had been awarded, Galtonborrowed the tickets from the organizers and ran a series of statisticaltests on them. Galton arranged the guesses (totaling 787—13 werediscarded because they were illegible) in order from highest to lowest andplotted them to see if they would form a bell curve. Then, among otherthings, he added up all of the contestants’ estimates and calculated themean. That number represented, you could say, the collective wisdom ofthe Plymouth crowd. If the crowd were viewed as a single person, thatwould be the person’s guess as to the ox’s weight.Galton had no doubt that the average guess of the group would be wayoff the mark. After all, mix a few very smart people with some mediocrepeople and a lot of dumb people, and it seems likely that you would end upwith a dumb answer. But Galton was wrong. The crowd had guessed thatthe slaughtered and dressed ox would weigh 1,197 pounds. In fact, afterit was slaughtered and dressed, the ox weighed 1,198 pounds. In otherwords, the crowd’s judgment was essentially perfect. The ‘‘experts’’ werenot even close. Perhaps breeding didn’t mean so much after all. Galtonwrote later: ‘‘The result seems more creditable to the trustworthiness
Isolation Methods 149of a democratic judgment than it might have been expected.’’ That wassomething of an understatement.What Francis Galton stumbled on that day in Plymouth was a simplebut powerful truth: Under the right circumstances, groups are remarkablyintelligent, and are often smarter than the smartest people in them.Groups do not need to be dominated by exceptionally intelligent peoplein order to be smart. Even if most of the people within a group arenot especially informed or rational, collectively they can reach a wisedecision.1Calculate the Impact of Other FactorsIt is sometimes possible, although not appropriate in all cases, to calculatethe impact of factors (other than the project) that account for part of theimprovement and then credit the project with the remaining part. Thatis, the project assumes credit for improvement that cannot be attributedto other factors.An example will help explain this approach. In a consumer lendingproject for a large bank, a signiﬁcant increase in consumer loan volumeoccurred after a project was implemented. Part of the increase in volumewas attributed to the project, and the remainder was due to the inﬂuenceof other factors in place during the same time period. Two additionalfactors were identiﬁed: (1) an increase in marketing and sales promotionand (2) falling interest rates.With regard to the ﬁrst factor, as marketing and sales promotionincreased, so did consumer loan volume. The contribution of this factorwas estimated using input from several internal experts in the marketingdepartment. As for the second factor, industry sources were used toestimate the relationship between consumer loan volume and interestrates. These two estimates together accounted for a modest percentage ofthe increase in consumer loan volume. The remaining improvement wasattributed to the project.This method is appropriate when the other factors can be easilyidentiﬁed and the appropriate mechanisms are in place to calculate theirimpact on the improvement. In some cases, estimating the impact ofoutside factors is just as difﬁcult as estimating the impact of the project,limiting this approach’s applicability. However, the results can be veryreliable if the procedure used to isolate the impact of other factors issound.
150 ISOLATION OF PROJECT IMPACTSelect the TechniqueWith all of these techniques available to isolate the impact of a project,selecting the most appropriate ones for a speciﬁc project can be difﬁcult.Some techniques are simple and inexpensive; others are time-consumingand costly. In choosing among them, the following factors should beconsidered:• Feasibility of the technique• Accuracy associated with the technique• Credibility of the technique with the target audience• Speciﬁc cost to implement the technique• Amount of disruption in normal work activities resulting from thetechnique’s implementation• Participant, staff, and management time required for the techniqueThe use of multiple techniques or multiple sources of data inputshould be considered since two sources are usually better than one.When multiple sources are used, a conservative method should be usedto combine the inputs. The reason is that a conservative approach buildsacceptance. The target audience should always be provided with anexplanation of the process and the subjective factors involved. Multiplesources allow an organization to experiment with different strategies andbuild conﬁdence in the use of a particular technique. For example, ifmanagement is concerned about the accuracy of participants’ estimates,the combination of a control group arrangement and participant estimatescould be useful for checking the accuracy of the estimation process.It is not unusual for the ROI of a project to be extremely large.Even when a portion of the improvement is allocated to other factors,the magnitude can still be impressive in many situations. The audienceshould understand that even though every effort has been made to isolatethe project’s impact, it remains an imprecise ﬁgure that is subject toerror. It represents the best estimate of the impact given the constraints,conditions, and resources available. Chances are it is more accuratethan other types of analysis regularly used in other functions within theorganization.FINAL THOUGHTSIsolating the effects of a project is an important step in answeringthe question of how much of the improvement in a business measure
Final Thoughts 151was caused by the project or speciﬁc project management solution. Thetechniques presented in this chapter are the most effective approachesavailable to answer this question and are used by some of the mostprogressive organizations. Too often results are reported and linked to aproject with no attempt to isolate the exact portion of the outcome associ-ated with the project. This leads to an invalid report trumpeting projectsuccess. If professionals wish to improve their images and are committedto meeting their responsibility to obtain results, the need for isolationmust be addressed early in the process for all major projects. Once thisimportant step is completed, the gathered data must be converted tomonetary values. The process for converting data to monetary values isdetailed in the next chapter.
154 CONVERTING DATA TO MONEYbeneﬁts of projects are pursued. Executives, sponsors, clients, adminis-trators, and other leaders in particular are concerned with the allocationof funds and want to see evidence of the contribution of a project in termsof monetary value. Any other outcome for these key stakeholders wouldbe unsatisfactory.Impact Is More UnderstandableFor some projects, the impact is more understandable when stated interms of monetary value. Consider, for example, the impact of a majorproject to improve the creativity of an organization’s employees andthereby enhance the innovation of the organization. Suppose this projectinvolved literally all employees and had an impact on all parts of theorganization. Across all departments, functions, units, and divisions,employees were being more creative, suggesting new ideas, taking on newchallenges, driving new products—in short, helping the organization ina wide variety of ways. The only way to understand the value of thistype of project is to convert the individual efforts and their consequencesto monetary values. Totaling the monetary values of all the innovationswould provide some sense of the value of the project.Consider the impact of a Lean Six Sigma project directed at all ofthe managers in an organization. As part of the project, the managerswere asked to select at least two measures of importance to them and toindicate what would need to change or improve for them to meet theirspeciﬁc goals. The measures could number in the dozens, if not hundreds.When the project’s impact was studied, a large number of improvementswere identiﬁed but were hard to quantify. Converting them to monetaryvalues allowed the improvements to be expressed in the same terms,enabling the outcomes to be more clearly reported.Converting to Monetary Values Is Similar to BudgetingProfessionals and administrators are familiar with budgets and areexpected to develop budgets for projects with an acceptable degree ofaccuracy. They are also comfortable with cost issues. When it comesto beneﬁts, however, many are not comfortable, even though some ofthe same techniques used in developing budgets are used to determine
The Importance of Converting Data to Money 155beneﬁts. Some of the beneﬁts of the project will take the form of costsavings or cost reductions, and this can make identiﬁcation of the costsor value easier for some projects. The monetary beneﬁt resulting from abudget is a natural extension of the budget.Monetary Value Is Vital to Organizational OperationsWith global competitiveness and the drive to improve the efﬁciencyof operations, awareness of the costs related to particular processesand activities is essential. In the 1990s this emphasis gave rise toactivity-based costing (ABC) and activity-based management. ABC isnot a replacement for traditional, general ledger accounting. Rather, itis a bridge between cost accumulations in the general ledger and theend users who must apply cost data in decision making. The problemlies in the typical cost statements, where the actual cost of a processor problem is not readily discernible. ABC converts inert cost data torelevant, actionable information. ABC has become increasingly usefulfor identifying improvement opportunities and measuring the beneﬁtsrealized from performance initiatives on an after-the-fact basis. Over 80percent of the studies conducted show projects beneﬁting the organizationthrough cost savings (cost reductions or cost avoidance). Consequently,understanding the cost of a problem and the payoff of the correspondingsolution is essential to proper management of the business.Monetary Values Are Necessary to Understand ProblemsIn any business, costs are essential to understanding the magnitude of aproblem. Consider, for example, the cost of employee turnover. Traditionalrecords and even those available through activity-based costing will notindicate the full value or cost of the problem. A variety of estimates andexpert inputs may be necessary to supplement cost statements to arriveat a deﬁnite value. The good news is that organizations have developeda number of standard procedures for identifying undesirable costs. Forexample, Wal-Mart has calculated the cost of a truck sitting idle at astore for one minute, waiting to be unloaded. When this cost is multipliedby the hundreds of deliveries per store and the result then multiplied by5,000 stores, the cost becomes huge.
156 CONVERTING DATA TO MONEYKEY STEPS IN CONVERTING DATA TO MONEYConverting data to monetary values involves ﬁve steps for each data item:1. Focus on a unit of measure. First, a unit of measure must bedeﬁned. For output data, the unit of measure is the item produced,e.g., one item assembled, service provided—one package shipped orsale completed. Time measures could include the time to completea project, cycle time, or customer response time, and the unit hereis usually expressed in terms of minutes, hours, or days. Quality isanother common measure, with a unit deﬁned as one error, reject,defect, or reworked item. Soft data measures vary, with a unit ofimprovement expressed in terms of absences, turnover, or a changein the customer satisfaction index. Speciﬁc examples of units ofmeasure are:• One student enrolled• One loan approved• One reworked item• One voluntary turnover• One hour of cycle time• One customer complaint• One less day of incarceration(Prison)• One patient served• One FTE employee• One grievance• One hour of downtime• One hour of employee time• One person removed from wel-fare2. Determine the value of each unit. Now comes the challenge:placing a value (V) on the unit identiﬁed in step 1. For measuresof production, quality, cost, and time, the process is relatively easy.Most organizations maintain records or reports that can pinpointthe cost of one unit of production or one defect. Soft data are moredifﬁcult to convert to money. For example, the monetary value of onecustomer complaint or a one-point change in an employee attitudemay be difﬁcult to determine. The techniques described in thischapter provide an array of approaches for making this conversion.When more than one value is available, the most credible or lowestvalue is generally used in the calculation.3. Calculate the change in performance data. The change inoutput data is calculated after the effects of the project have beenisolated from other inﬂuences. This change ( ) is the performanceimprovement that is directly attributable to the project, representedas the Level 4 impact measure. The value may represent the
Standard Monetary Values 157performance improvement for an individual, a team, a group ofparticipants, or several groups of participants.4. Determine the annual amount of change. The value is annu-alized to develop a value for the total change in the performance datafor one year ( P). Using annual ﬁgures is a standard approach fororganizations seeking to capture the beneﬁts of a particular project,even though the beneﬁts may not remain constant throughout theyear. For a short-term project, ﬁrst-year beneﬁts are used evenwhen the project produces beneﬁts beyond one year. This approachis considered conservative. More will be discussed about this later.5. Calculate the annual value of the improvement. The totalvalue of improvement is calculated by multiplying the annual per-formance change ( P) by the unit value (V) for the complete groupin question. For example, if one group of participants is involvedin the project being evaluated, the total value will include the totalimprovement for all participants providing data in the group. Thisvalue for annual project beneﬁts is then compared with the costs ofthe project to calculate the BCR, ROI, or payback period.A simple example will demonstrate these ﬁve steps. Suppose a largeretail store chain pilots an effort to replace all traditional lighting withenergy-efﬁcient LED bulbs. Prior to project implementation, the 90,000-square-foot store in which the pilot program occurs uses approximately14 kilowatt-hours (kWh) per square foot annually for a total of 1,260,000kWh per year. On average this is 105,000 kWh per month. After imple-menting the project, the store monitors electricity usage for six months,showing a new average monthly usage of 73,500 kWh. This is a decrease of31,500 kWh per month. By comparing this store’s usage to that of anotherstore with comparable characteristics, the 31,500 reduction is attributedto the change in bulbs. Given a monthly change in performance, theannual change is 378,000 kWh. The value of a kWh is approximately 10cents per kWh. The total annual savings to the store is $37,800. Table 9.1shows the example using the ﬁve steps.STANDARD MONETARY VALUESMost hard data items (output, quality, cost, and time) have standardvalues. A standard value is a monetary value assigned to a unit ofmeasurement that is accepted by key stakeholders. Standard values have
158 CONVERTING DATA TO MONEYTable 9.1 Converting Kilowatt-Hours to Monetary ValuesSetting: Retail store piloting replacement of traditional lighting to LED lighting.Step 1: Deﬁne the unit of measure. The unit of measure is deﬁned as one kWh.Step 2: Determine the value (V) of each unit. According to standard data (i.e.,the cost per kWh paid per month), the cost is 10 cents per kWh (V = 10 cents).Step 3: Calculate the change ( ) in performance data. Six months after theproject was completed, electricity usage decreased an average of 31,500 kWhper month. The isolation technique used was a control group (see Chapter 8).Step 4: Determine an annual amount of the change ( P). Using the six-monthaverage of 31,500 kWh per month yields an annual improvement of 378,000( P = 31,500 × 12 = 378,000).Step 5: Calculate the annual monetary value of the improvement. ( P × V) =378,000 × .10 = $37,800 cost savingsbeen developed because these are often the measures that matter to theorganization. They reﬂect problems, and their conversion to monetaryvalues shows their impact on the operational and ﬁnancial well-being ofthe organization.For the last two decades, quality implementations have typicallyfocused only on the cost of quality. Organizations have been obsessedwith placing a value on mistakes or the payoff from avoiding thesemistakes. This assigned value—the standard cost of an item—is oneof the critical outgrowths of the quality management movement. Inaddition, a variety of process improvement projects—such as reengineer-ing, reinventing the corporation, transformation, and continuous processimprovement—have included a component in which the cost of a par-ticular measure is determined. Finally, the development of a varietyof cost control, cost containment, and cost management systems—suchas activity-based costing—have forced organizations, departments, anddivisions to place costs on activities and, in some cases, relate those costsdirectly to the revenues or proﬁts of the organization.The following discussion describes how measure of output, quality, andtime can be converted to standard values.Converting Output Data to MoneyWhen a project results in a change in output, the value of the increasedoutput can usually be determined from the organization’s accounting
Standard Monetary Values 159or operating records. For organizations operating on a proﬁt basis, thisvalue is typically the marginal proﬁt contribution of an additional unitof production or service provided. An example is a team within a majorappliance manufacturing ﬁrm that was able to boost the production ofsmall refrigerators after a comprehensive work cell redesign project; theunit of improvement is the proﬁt margin associated with one refrigerator.For organizations that are performance driven rather than proﬁt driven,this value is usually reﬂected in the savings realized when an additionalunit of output is realized for the same input. For example, in the visasection of a government ofﬁce, one additional visa application may beprocessed at no additional cost; an increase in output translates into acost savings equal to the unit cost of processing a visa application.The formulas used to calculate this contribution depend on the type oforganization and the nature of its record keeping. Most organizations havestandard values readily available for performance monitoring and goalsetting. Managers often use marginal cost statements and sensitivityanalyses to pinpoint values associated with changes in output. If thedata are not available, the project team must initiate or coordinate thedevelopment of appropriate values.The beneﬁt of converting output data to money using standard valuesis that these calculations are already available for the most importantdata items. Perhaps no area has as much experience with standard valuesas the sales and marketing area. Table 9.2 shows a sampling of the salesand marketing measures that are routinely calculated and reported asstandard values.1Calculating the Cost of QualityQuality and the cost of quality are important issues in most manufac-turing and service organizations. Because many projects are designedto increase quality, the project team may have to place a value on theimprovement of certain quality measures. For some quality measures,the task is easy. For example, if quality is measured in terms of the defectrate, the value of the improvement is the cost to repair or replace theproduct. The most obvious cost of poor quality is the amount of scrap orwaste generated by mistakes. Defective products, spoiled raw materials,and discarded paperwork are all the result of poor quality. Scrap andwaste translate directly into a monetary value. In a production environ-ment, for example, the cost of a defective product is the total cost incurred
160 CONVERTING DATA TO MONEYTable 9.2 Examples of Standard Values from Sales and MarketingMetric Deﬁnition Conversion NotesSales The sale of the product orservice recorded in a varietyof different ways: byproduct, by time period, bycustomerThe data must be convertedto monetary value byapplying the proﬁt marginfor a particular salescategory.Proﬁt margin(%)Price−CostCost for the product,customer, and time periodFactored to convert sales tomonetary value-add to theorganization.Unit margin Unit price less unit cost This shows the value ofincremental sales.Channel margin Channel proﬁts as apercentage of channelselling priceThis would be used to showthe value of sales through aparticular marketingchannel.Retention rate The ratio of customersretained to the number ofcustomers at risk of leavingThe value is the saving ofthe money necessary toacquire a replacementcustomer.Churn rate Ratio of customers leavingto the number who are atrisk of leavingThe value is the saving ofthe money necessary toacquire a new customer.Customer proﬁt The difference between therevenues earned from andthe cost associated with thecustomer relationshipduring the speciﬁed periodThe monetary value addedis the proﬁt obtained fromcustomers. It all goestoward the bottom line.Customer valuelifetimeThe present value of thefuture cash ﬂows attributedto the customer relationshipBottom line; as customervalue increases, it addsdirectly to the proﬁts. Also,as a customer is added, theincremental value is thecustomer lifetime average.CannibalizationrateThe percentage of newproduct sales taken fromexisting product linesThis is to be minimized, as itrepresents an adverse effecton existing product, with thevalue added being the loss ofproﬁts due to the sales loss.Workload Hours required to serviceclients and prospectsThis includes the salaries,commissions, and beneﬁtsfrom the time the sales staffspend on the workloads.(continues)
Standard Monetary Values 161Table 9.2 (Continued)Metric Deﬁnition Conversion NotesInventories The total amount of productor brand available for salein a particular channelSince inventories are valuedat the cost of carrying theinventory, costs involvespace, handling, and thetime value of money.Insufﬁcient inventory is thecost of expediting the newinventory or the loss of salesbecause of the inventoryoutage.Market share Sales revenue as apercentage of total marketsalesActual sales are convertedto money through the proﬁtmargins. This is a measureof competitiveness.Loyalty The length of time thecustomer stays with theorganization, thewillingness to pay apremium, and thewillingness to searchThis is calculated as theadditional proﬁt from thesale or the proﬁt on thepremium.(Source: Adapted from Marketing Metrics: 50+ Metrics Every Executive Should Masterby Paul W. Farris, Neil T. Bendle, Phillip E. Pfeifer, and David J. Ribstein. (UpperSaddle River, NJ: Wharton School Publishing, 2006, pp. 46–47.)up to the point at which the mistake is identiﬁed, minus the salvagevalue. In the service environment, the cost of a defective service is thecost incurred up to the point at which the deﬁciency is identiﬁed, plus thecost to correct the problem, plus the cost to make the customer happy,plus the loss of customer loyalty.Employee mistakes and errors can be expensive. The costliest form ofrework occurs when a product or service is delivered to a customer andmust be returned for repair or correction. The cost of rework includes bothlabor and direct costs. In some organizations, rework costs can constituteas much as 35 percent of operating expenses.Quality costs can be grouped into six major categories:21. Internal failure represents costs associated with problems detectedprior to product shipment or service delivery. Typical such costs arereworking and retesting.
162 CONVERTING DATA TO MONEY2. Penalty costs are ﬁnes or penalties incurred as a result of unaccept-able quality.3. External failure refers to problems detected after product ship-ment or service delivery. Typical items here are technical support,complaint investigation, remedial upgrades, and ﬁxes.4. Appraisal costs are the expenses involved in determining the condi-tion of a particular product or service. Typical costs involve testingand related activities, such as product quality audits.5. Prevention costs involve efforts undertaken to prevent the provisionof unacceptable products or service quality. These efforts includeservice quality administration, inspections, process studies, andimprovements.6. Customer dissatisfaction is perhaps the costliest element of inad-equate quality. In some cases, serious mistakes result in lostbusiness. Customer dissatisfaction is difﬁcult to quantify, and arriv-ing at a monetary value may be impossible using direct methods.The judgment and expertise of sales, marketing, or quality man-agers are usually the best resources to draw upon in measuring theimpact of dissatisfaction. More and more quality experts are mea-suring customer and client dissatisfaction with the use of marketsurveys.3As with output data, the good news is that a tremendous number ofquality measures have been converted to standard values. Some of thesemeasures are• Defects• Rework• Processing errors• Date errors• Accidents• Grievances• Downtime—equipment• Downtime—system• Delay• Fines• Days sales uncollected• QueuesConverting Employee Time Using CompensationReducing the workforce or saving employee time is a common objectivefor projects. In a team environment, a project may enable the team tocomplete tasks in less time or with fewer people. A major project could
Standard Monetary Values 163lead to a reduction of several hundred employees. On an individual basis,a technology project may be designed to help professional, sales, andmanagerial employees save time in performing daily tasks. The value ofthe time saved is an important measure, and determining a monetaryvalue for it is relatively easy.The most obvious time savings stem from reduced labor costs forperforming a given amount of work. The monetary savings are found bymultiplying the hours saved by the labor cost per hour. For example, atime-saving process in one organization, participants estimated, saved anaverage of 74 minutes per day, worth $31.25 per day or $7,500 per year,based on the average salary plus beneﬁts for a typical participant.The average wage, with a percentage added for employee beneﬁts, willsufﬁce for most calculations. However, employee time may be worth more.For example, additional costs for maintaining an employee (ofﬁce space,furniture, telephones, utilities, computers, administrative support, andother overhead expenses) could be included in calculating the averagelabor cost. Thus, the wage rate used in the calculation can escalatequickly. In a large-scale employee reduction effort, calculating the costsof additional employees may be more appropriate for showing the value.However, for most projects, the conservative approach of using salaryplus employee beneﬁts is recommended.Beyond reducing the labor cost per hour, time savings can producebeneﬁts such as improved service, avoidance of penalties for late projects,and additional proﬁt opportunities. These values can be estimated usingother methods discussed in this chapter.A word of caution is needed concerning time savings. Savings arerealized only when the amount of time saved translates into a costreduction or a proﬁt contribution. Even if a project produces savingsin manager time, monetary value is not realized unless the managerputs the time saved to productive use. Having managers estimate thepercentage of time saved that is devoted to productive work may behelpful, if it is followed up with a request for examples of how theextra time was used. If a team-based project sparks a new process thateliminates several hours of work each day, the actual savings will bebased on the corresponding reduction in staff or overtime pay. Therefore,an important preliminary step in ﬁguring time savings is determiningwhether the expected savings will be genuine. FedEx is a primary exampleof assigning value to time.4
164 CONVERTING DATA TO MONEYFinding Standard ValuesStandard values are available for all types of data. Virtually every majordepartment will develop standard values that are monitored for thatarea. Typical functions in a major organization where standard valuesare tracked include• Finance and accounting• Production• Operations• Engineering• IT• Administration• Sales and marketing• Customer service and support• Procurement• Logistics• Compliance• Research and development• HRThanks to enterprise-wide systems software, standard values are com-monly integrated and made available for access by a variety of people. Insome cases, access may need to be addressed to ensure that the data canbe obtained by those who require them.WHEN STANDARD VALUES ARE NOT AVAILABLEWhen standard values are not available, several alternative strategies forconverting data to monetary worth are available. Some are appropriatefor a speciﬁc type of data or data category, while others may be used withvirtually any type of data. The challenge is to select the strategy that bestsuits the situation.Using Historical Costs from RecordsHistorical records often indicate the value of a measure and the cost (orvalue) of a unit of improvement. This strategy relies on identifying theappropriate records and tabulating the proper cost components for theitem in question.For example, suppose a large construction ﬁrm initiated a project toimprove safety. The project improved several safety-related performancemeasures, ranging from amounts spent in response to government ﬁnesto total worker’s compensation costs. From the company’s records for oneyear of data, the average cost for each safety measure was determined.
When Standard Values Are Not Available 165This value included the direct costs of medical payments, insurancepayments and premiums, investigation services, and lost-time paymentsto employees, as well as payments for legal expenses, ﬁnes, and otherdirect services. The amount of time used to investigate, resolve, andcorrect the issues was also factored in. This time involved not only thehealth and safety staff, but other personnel as well. In addition, the costsof lost productivity, disruption of services, morale, and dissatisfactionwere estimated to obtain a full cost. The corresponding costs for eachitem were then developed.This example suggests the challenges inherent in maintaining systemsand databases to enable the value for a particular data item to beidentiﬁed. It also raises several concerns about using historical costs as atechnique to convert data to money.TimeSorting through databases, cost statements, ﬁnancial records, and activ-ity reports takes a tremendous amount of time, time that may not beavailable for the project. It is important to keep this part of the processin perspective. This is only one step in the ROI methodology (convertingdata to monetary values) and only one measure among many that mayneed to be converted. Time needs to be conserved.AvailabilityIn some cases, data are not available to show all of the costs for aparticular item. In addition to the direct costs associated with a measure,an equal number of indirect or invisible costs may be present that cannotbe obtained easily.AccessCompounding the problems of time and availability is access. Mone-tary values may be needed from a system or record set that is undersomeone else’s control. In a typical implementation, the project leadermay not have full access to cost data. These are more sensitive thanother types of data and are often protected for a number of reasons,including competitive advantage. Therefore, access can be difﬁcult andsometimes is even prohibited unless an absolute need to know can bedemonstrated.
166 CONVERTING DATA TO MONEYAccuracyFinally, the need for accuracy in this analysis should not be overlooked.A measure provided in current records may appear to be based onaccurate data, but this may be an illusion. When data are calculated,estimations are involved, access to certain systems is denied, and differentassumptions are made (all of which can be compounded by differentdeﬁnitions of systems, data, and measures). Because of these limitations,the calculated values should be viewed as suspect unless means areavailable to ensure that they are accurate.Calculating monetary value using historical data should be done withcaution and only when these two conditions exist:• The sponsor has approved the use of additional time, effort, andmoney to develop a monetary value from the current records andreports.• The measure is simple and can be found by searching only a fewrecords.Otherwise, an alternative method is preferred.Using Input from Experts to Convert DataWhen it is necessary to convert data items for which historical costdata are not available, input from experts on the process might be aconsideration. Internal experts can provide the cost (or value) of oneunit of improvement. Individuals with knowledge of the situation andthe conﬁdence of management must be willing to provide estimates—aswell as the assumptions behind the estimates. Internal experts may befound in the department in which the data originated—sales, marketing,payroll, labor relations, or any number of other functions. Most expertshave their own methodologies for developing these values. So when theirinput is required, it is important to explain the full scope of what isneeded and to provide as many speciﬁcs as possible.If internal experts have a strong bias regarding the measure or arenot available, external experts are sought. External experts should beselected based on their experience with the unit of measure. Fortunately,many experts are available who work directly with important measures,
When Standard Values Are Not Available 167such as employee attitudes, customer satisfaction, turnover, absenteeism,and grievances. They are often willing to provide estimates of the cost (orvalue) of these intangibles.External experts—including consultants, professionals, or suppliersin a particular area—can also be found in obvious places. For example,the costs of accidents can be estimated by the worker’s compensationcarrier, or the cost of a grievance may be estimated by the labor attorneydefending the company in grievance transactions. The process of locatingan external expert is similar to the external database search, which isdescribed later.The credibility of the expert, whether internal or external, is a criticalissue if the monetary value placed on a measure is to be reliable. Fore-most among the factors behind an expert’s credibility is the individual’sexperience with the process or measure at hand. Ideally, he or she shouldwork with this measure routinely. Also, the person must be unbiased.Experts should be neutral in connection with the measure’s value andshould have no personal or professional interest in it.In addition, the credentials of external experts—published works,degrees, and other honors or awards—are important in validating theirexpertise. Many of these people are tapped often, and their track recordscan and should be checked. If their estimate has been validated in moredetailed studies and was found to be consistent, this can serve as aconﬁrmation of their qualiﬁcations in providing such data.Using Values from External DatabasesFor some measures, the use of cost (or value) estimates based on the workand research of others may be appropriate. This technique makes use ofexternal databases that contain studies and research projects focusing onthe cost of data items. Fortunately, many databases include cost studiesof data items related to projects, and most are accessible on the Internet.Data are available on the costs of turnover, absenteeism, grievances,accidents, and even customer satisfaction. The difﬁculty lies in ﬁndinga database with studies or research germane to the particular project.Ideally, the data should originate from a similar setting in the sameindustry, but that is not always possible. Sometimes, data on industriesor organizations in general are sufﬁcient, with adjustments possiblyrequired to suit the project at hand.
168 CONVERTING DATA TO MONEYLinking with Other MeasuresWhen standard values, records, experts, and external studies are notavailable, a feasible alternative might be to ﬁnd a relationship betweenthe measure in question and some other measure that can be easilyconverted to a monetary value. This involves identifying existing relation-ships that show a strong correlation between one measure and anotherwith a standard value.A classic relationship is the correlation between job satisfaction andemployee turnover. Suppose that in a project designed to improve jobsatisfaction, a value is needed to reﬂect changes in the job satisfactionindex. A predetermined relationship showing the correlation betweenincreases in job satisfaction and reductions in turnover can directly linkthe two measures. Using standard data or external studies, the costof turnover can easily be determined as described earlier. Therefore, achange in job satisfaction can be immediately converted to a monetaryvalue, or at least an approximate value. The conversion is not alwaysexact because of the potential for error and other factors, but the estimateis sufﬁcient for converting the data to monetary values.Finding a correlation between a customer satisfaction measure andanother measure that can easily be converted to a monetary value issometimes possible. A strong correlation often exists between customersatisfaction and revenue. Connecting these two variables allows themonetary value of customer satisfaction to be estimated.In some situations, a chain of relationships may establish a connectionbetween two or more variables. A measure that may be difﬁcult toconvert to a monetary value is linked to other measures that, in turn,are linked to measures to which values can be assigned. Ultimately,these measures are traced to a monetary value typically based on proﬁts.Figure 9.1 shows the model used by Sears.5The model connects jobattitudes (collected directly from the employees) to customer service,which is directly related to revenue growth. The rectangles in the ﬁgurerepresent survey information, and the ovals represent hard data. Theshaded measurements are collected and distributed in the form of Searstotal-performance indicators.As the model shows, a 5-point improvement in employee attitudesleads to a 1.3-point improvement in customer satisfaction. This, in turn,drives a 0.5 percent increase in revenue growth. If employee attitudes at
When Standard Values Are Not Available 169Returnon assetsOperatingmarginRevenuegrowthA Compelling Place to WorkAttitudeabout thejobAttitudeabout thecompanyEmployeebehaviorEmployeeretentionServiceHelpfulnessCustomerimpressionCustomerrecommendationsCustomerretentionA Compelling Place to ShopA Compelling Placeto Invest In5-unit increasein employeeattitude1.3-unit increase incustomer impression0.5% increasein revenue growthDrives DrivesMerchandiseValueFigure 9.1 Relationship between attitudes and proﬁts. (Copyright 1998. Presidentand Fellows of Harvard College. Used with permission.)a local store improved by 5 points and the previous rate of revenue growthwas 5 percent, the new rate of revenue growth would then be 5.5 percent.These links between measures, often called the service-proﬁt chain,offer a promising methodology for applying monetary values to hard-to-quantify measures.Using Estimates from ParticipantsIn some cases, participants in the project should estimate the value ofimprovement. This technique is appropriate when participants are capa-ble of providing estimates of the cost (or value) of the unit of measure thathas improved as a result of the project. With this approach, participantsshould be provided with clear instructions along with examples of thetype of information needed. The advantage of this approach is that theindividuals who are most closely connected to the improvement are oftenable to provide the most reliable estimates of its value. As with isolatingproject effects, when estimates are used to convert measures to monetaryvalues, adjustments are made to reduce the error in those estimates.
170 CONVERTING DATA TO MONEYUsing Estimates from the Management TeamIn some situations, participants in a project may be incapable of placinga value on the improvement. Their work may be so far removed from theultimate value of the process that they cannot provide reliable estimates.In these cases, the team leaders, supervisors, or managers of participantsmay be able to provide estimates. Thus, they may be asked to provide avalue for a unit of improvement linked to the project.In other situations, managers are asked to review and approve par-ticipants’ estimates and conﬁrm, adjust, or reject those values. Forexample, suppose a project involving customer service representativeswas designed to reduce customer complaints. The project did result ina reduction in complaints, but the value of a single customer complainthad to be identiﬁed to determine the value of the improvement. Althoughcustomer service representatives had knowledge of certain issues sur-rounding customer complaints, their scope was limited, so their managerswere asked to provide a value. These managers had a broader perspectiveof the impact of a customer complaint.Senior management can often provide estimates of the value of data.In this approach, senior managers concerned with the project are askedto place a value on the improvement based on their perception of itsworth. This approach is used when calculating the value is difﬁcult orwhen other sources of estimation are unavailable or unreliable.Using Project Staff EstimatesThe ﬁnal strategy for converting data to monetary values is using projectstaff estimates. Using all available information and experience, the staffmembers most familiar with the situation provide estimates of the value.For example, a particular project for an international oil company wasdesigned to reduce dispatcher absenteeism and address other perfor-mance problems. Unable to identify a value using the other strategies,the consulting staff estimated the cost of an absence to be $200. Thisvalue was then used in calculating the savings from the reduction inabsenteeism that followed the project implementation.Although the project staff may be qualiﬁed to provide accurateestimates, this approach is sometimes perceived as biased. It shouldtherefore be used only when other approaches are unavailable orinappropriate.
Technique Selection and Finalizing Value 171TECHNIQUE SELECTION AND FINALIZING VALUEWith so many techniques available, the challenge is selecting one ormore strategies appropriate for the situation and available resources.Developing a table or list of values or techniques for the situation may behelpful. The guidelines that follow may aid in selecting a technique andﬁnalizing the values.Choose a Technique Appropriate for the Type of DataSome strategies are designed speciﬁcally for hard data, whereas othersare more appropriate for soft data. Thus, the type of data often dictatesthe strategy. Standard values are developed for most hard data items, andcompany records and cost statements are used in the process. Soft dataoften involve the use of external databases, links with other measures,and estimates. Experts are used to convert each type of data to monetaryvalues.Move from Most Accurate to the Least AccurateThe techniques in this chapter are presented in order of accuracy. Stan-dard values are always most accurate and therefore the most credible.But, as mentioned earlier, they are not always readily available. Whenstandard values are not available, the following sequence of operationaltechniques should be tried:• Historical costs from company records• Internal and external experts• External databases• Links with other measures• EstimatesEach technique should be considered in turn based on its feasibility andapplicability to the situation. The technique associated with the highestaccuracy is always preferred if the situation allows.Consider Source AvailabilitySometimes the availability of a particular source of data determinesthe method selection. For example, experts may be readily accessible.
172 CONVERTING DATA TO MONEYSome standard values are easy to ﬁnd; others are more difﬁcult. In othersituations, the convenience of a technique is a major factor in the selection.The Internet, for example, has made external database searches moreconvenient.As with other processes, keeping the time investment for this phaseto a minimum is important so that the total effort directed to the ROIstudy does not become excessive. Some techniques can be implementedin much less time than others. Devoting too much time to the conversionprocess may dampen otherwise enthusiastic attitudes about the use ofthe methodology.Use the Source with the BroadestPerspective on the IssueAccording to Guiding Principle 3 in Table 3.1, the most credible datasource must be used. The individual providing estimates must be knowl-edgeable of the processes and the issues surrounding the valuation of thedata. For example, consider the estimation of the cost of a grievance in amanufacturing plant. Although a supervisor may have insight into whatcaused a particular grievance, he or she may have a limited perspective.A high-level manager may be able to grasp the overall impact of thegrievance and how it will affect other areas. Thus, a high-level managerwould be a more credible source in this situation.Use Multiple Techniques When FeasibleThe availability of more than one technique for obtaining values for thedata is often beneﬁcial. When appropriate, multiple sources should beused to provide a basis for comparison or for additional perspectives. Thedata must be integrated using a convenient decision rule, such as thelowest value. The conservative approach of using the lowest value waspresented as Guiding Principle 4 in Table 3.1, but this applies only whenthe sources have equal or similar credibility.Converting data to monetary values has its challenges. Once theparticular method has been selected and applied, several adjustments ortests are necessary to ensure the use of the most credible and appropriatevalue with the least amount of resources.
Technique Selection and Finalizing Value 173Apply the Credibility TestThe discussion of techniques in this chapter assumes that each data itemcollected and linked to a project can be converted to a monetary value.Highly subjective data, however, such as changes in employee attitudesor a reduction in the number of employee conﬂicts, are difﬁcult to convert.Although estimates can be developed using one or more strategies, suchestimates may lack credibility with the target audience, which can rendertheir use in analysis questionable.The issue of credibility in combination with resources is illustratedquite clearly in Figure 9.2. This is a logical way to decide whether toconvert data to monetary values or leave them intangible. Essentially,in the absence of standard values, many other ways are available tocapture the data or convert them to monetary values. However, thereis a question to be answered: Can it be done with minimum resources?Some of the techniques mentioned in this chapter—such as searchingrecords or maybe even searching the Internet—cannot be performed withminimal use of resources. However, an estimate obtained from a group orfrom a few individuals is available with minimal use of resources. Thenwe move to the next question: Will the executive who is interested inthe project buy into the monetary value assigned to the measure in twominutes? If so, then it is credible enough to be included in the analysis; ifnot, move it to the intangibles. The intangibles are also very importantand are covered in much more detail in the next chapter.Consider the Possibility of a Management AdjustmentIn organizations where soft data are common and values are derived usingimprecise methods, senior managers and administrators are sometimesoffered the opportunity to review and approve the data. Because of thesubjective nature of this process, management may factor (reduce) thedata to make the ﬁnal results more credible.Consider the Short-Term/Long-Term IssueWhen data are converted to monetary values, usually one year’s worth ofdata is included in the analysis—this is Guiding Principle 9 in Table 3.1,
174 CONVERTING DATA TO MONEYIs there astandard value?Is there amethod to getthere?Can we get therewith minimumresources?Add to numeratorMove tointangiblebenefitsYesCan we convince ourexecutive in twominutes that thevalue is credible?YesYesConvert dataand add tonumeratorNo NoYesMove tointangiblebenefitsNoMove tointangiblebenefitsNoFigure 9.2 Four-part test: to convert or not to convert?which states that for short-term solutions, only the ﬁrst year’s beneﬁtsare used. The issue of whether a project is short-term or long-termdepends on the time it takes to complete or implement the project. Ifone group participating in the project and working through the processtakes months to complete it, then it is probably not short-term. Someprojects literally take years to implement even for one particular group.In general, it is appropriate to consider a project short-term when oneindividual takes one month or less to learn what needs to be done to make
Final Thoughts 175the project successful. When the lag between project implementation andthe consequences is relatively brief, a short-term solution is appropriate.When a project is long-term, no time limit for data inclusion is used, butthe time value should be set before the project evaluation is undertaken.Input on the time value should be secured from all stakeholders, includingthe sponsor, champion, implementer, designer, and evaluator. After somediscussion, the estimates of the time factor should be very conservativeand perhaps reviewed by ﬁnance and accounting. When a project is along-term solution, forecasting will need to be used to estimate multipleyears of value. No sponsor will wait several years to see how a projectturns out.Consider an Adjustment for the Time Value of MoneySince investment in a project is made in one time period and the returnis realized at a later time, some organizations adjust project beneﬁts toreﬂect the time value of money using discounted-cash-ﬂow techniques.The actual monetary beneﬁts of the project are adjusted for the timeperiod. The amount of adjustment, however, is usually small comparedwith the typical beneﬁts of projects.Although this may not be an issue for every project, it should beconsidered for each project, and some standard discount rate should beused. Consider the following example of how this is calculated. Assumethat a project costs $100,000, and it is expected to take two years for thefull value of the estimate to be realized. In other words, this is a long-termsolution spanning two years. Using a discount rate of 6 percent, the costfor the project for the ﬁrst year would be $100,000 × 106 percent =$106,000. For the second year it is $106,000 × 106 percent, or $112,360.Thus, the project cost has been adjusted for a two-year value with a6 percent discount rate. This assumes that the project sponsor could haveinvested the money in some other project and obtained at least a 6 percentreturn on that investment.FINAL THOUGHTSShowing the real money requires just that—money. Business impactdata that have improved as a result of a project must be converted to
176 CONVERTING DATA TO MONEYmoney. Standard values make this process easier, but easy is not alwaysan option, and other techniques must sometimes be used. However,if a measure cannot be converted with minimum resources or withno assurance of credibility, the improvement in the measure should bereported as an intangible beneﬁt. After the data are converted to monetaryvalues, the next step is collecting the project costs and calculating theROI. These processes are covered in the next chapter.
178 MEASURING THE INTANGIBLESTable 10.1 Common Intangibles• Accountability• Alliances• Attention• Awards• Branding• Capacity• Clarity• Communication• Corporate social responsibility• Employee attitudes• Customer service (customer satisfaction)• Engagement• Human life• Image• Intellectual capital• Innovation and creativity• Job satisfaction• Leadership• Loyalty• Networking• Organizational commitment• Partnering• Poverty• Reputation• Stress• Team effectiveness• Timeliness• Sustainability• Work/life balanceorganizations are built on them. In every direction we look, intangi-bles are becoming not only increasingly important, but also critical toorganizations. Here’s a recap of why they have become so important.Intangibles Are the Invisible AdvantageWhen examining the success behind many well-known organizations,intangibles are often found. A highly innovative company continues todevelop new and improved products; a government agency reinventsitself; a company with highly involved and engaged employees attractsand keeps talent. An organization shares knowledge with employees,giving them a competitive advantage. Still another organization is able todevelop strategic partners and alliances. These intangibles do not oftenappear in cost statements and other record keeping, but they are there,and they make a huge difference.Trying to identify, measure, and react to intangibles may be difﬁcult,but the ability to do so exists. Intangibles transform the way organizationswork, the way employees are managed, the way products are designed,the way services are sold, and the way customers are treated. Theimplications are profound, and an organization’s strategy must be set upto deal with them. Although invisible, the presence of intangibles is feltand the results are concrete.
The Importance of Intangibles 179The Intangible EconomyThe intangible economy has evolved from basic changes that date to theIron Age, which evolved into the Agricultural Age. In the late nineteenthcentury and during the early twentieth century the world moved into theIndustrial Age. From the 1950s forward, the world has moved into theTechnology and Knowledge Age, and these moves translate into intan-gibles. During this time, a natural evolution of technology has occurred.During the Industrial Age, companies and individuals invested in tangi-ble assets like plants and equipment. In the Technology and KnowledgeAge, companies invest in intangible assets, like brands or systems. Thefuture holds more of the same, as intangibles continue to evolve into animportant part of the overall economic system.1More Intangibles Are Converted to TangiblesThe good news is that more data once regarded as intangible are nowbeing converted into monetary values. Because of this, classic intangiblesare now accepted as tangible measures, and their value is more easilyunderstood. Consider, for example, customer satisfaction. Just a decadeago, few organizations had a clue as to the monetary value of customersatisfaction. Now more ﬁrms have taken the extra step to link customersatisfaction directly to revenues, proﬁts, and other measures. Companiesare seeing the tremendous value that can be derived from intangibles.As this chapter will illustrate, more data are being accumulated to showmonetary values, moving some intangible measures into the tangiblecategory.Intangibles Drive ProjectsSome projects are implemented because of the intangibles. For example,the need to have greater collaboration, partnering, communication, team-work, or customer service will drive projects. In the public sector, theneed to reduce poverty, employ disadvantaged citizens, and save livesoften drives projects. From the outset, the intangibles are the importantdrivers and become the most important measures. Consequently, moreexecutives include a string of intangibles on their scorecards, key oper-ating reports, key performance indicators, dashboards, and other routine
180 MEASURING THE INTANGIBLESreporting systems. In some cases, the intangibles represent nearly half ofall measures that are monitored.The Intangible InvestmentThe Federal Reserve Bank of Philadelphia recently estimated that invest-ment in intangible assets amounts to at least $1 trillion.2 Only 15 percentof the value of a contemporary organization can be tied to such tangi-ble assets as buildings and equipment. Intangible assets have becomethe dominant investment in businesses. They are a growing force inthe economy, and measuring their values poses challenges to managersand investors. They can no longer be ignored. They must be properlyidentiﬁed, selected, measured, reported, and in some cases, converted tomonetary values.MEASURING AND ANALYZING INTANGIBLESIn some projects, intangibles are more important than monetary mea-sures. Consequently, these measures should be monitored and reportedas part of the project evaluation. In practice, every project, regardlessof its nature, scope, and content, will produce intangible measures. Thechallenge is to identify them effectively and report them appropriately.Measuring the IntangiblesFrom time to time it is necessary to explore the issue of measuring thedifﬁcult-to-measure. Responses to this exploration usually occur in theform of comments instead of questions. ‘‘You can’t measure it,’’ is a typicalresponse. This cannot be true, because anything can be measured. Whatthe frustrated observer suggests by the comment is that the intangibleis not something you can count, examine, or see in quantities, like itemsproduced on an assembly line. In reality, a quantitative value can beassigned to or developed for any intangible. If it exists, it can be measured.Consider human intelligence for example. Although human intelligenceis vastly complex and abstract with myriad facets and qualities, IQ scoresare assigned to most people and most people seem to accept them. Thesoftware engineering institute of Carnegie-Mellon University assignssoftware organizations a score of 1 to 5 to represent their maturity
Measuring and Analyzing Intangibles 181in software engineering. This score has enormous implications for theorganizations’ business development capabilities, yet the measure goespractically unchallenged.3Several approaches are available for measuring intangibles. Intangi-bles that can be counted include customer complaints, employee com-plaints, and conﬂicts. These can be recorded easily, and constitute one ofthe most acceptable types of measures. Unfortunately, many intangiblesare based on attitudes and perceptions that must be measured. The keyis in the development of the instrument of measure. The instruments areusually developed around scales of 3, 5, and even 10 to represent levels ofperception. The instruments to measure intangibles consist of three basicvarieties.The ﬁrst lists the intangible items and asks respondents to agree or dis-agree on a 5-point scale (where the midpoint represents a neutral opinion).Other instruments deﬁne various qualities of the intangible, such as itsreputation. A 5-point scale can easily be developed to describe degrees ofreputation, ranging from the worst rating—a horrible reputation—to thebest rating—an excellent reputation. Still other ratings are expressed asan assessment on a scale of 1 to 10, after respondents review a descriptionof the intangible.Another instrument to measure the intangible connects it, when pos-sible, to a measure that is easier to measure. As shown in Figure 10.1,most hard-to-measure items are linked to an easy-to-measure item. Inthe classic situation, a soft measure (typically the intangible) is con-nected to a hard measure (typically the tangible). Although this link canHard tomeasureEasy tomeasureSoft data(intangible)Hard data(tangible)Figure 10.1 The Link between hard-to-measure and easy-to-measure items.
182 MEASURING THE INTANGIBLESbe developed through logical deductions and conclusions, having someempirical evidence through a correlation analysis (as shown in the ﬁgure)and developing a signiﬁcant correlation between the items is the bestapproach. However, a detailed analysis would have to be conducted toensure that a causal relationship exists. In other words, just because acorrelation is apparent, does not mean that one caused the other. Con-sequently, additional analysis, other empirical evidence, and supportingdata could pinpoint the actual causal effect.A ﬁnal instrument for measuring the intangible is the development ofan index of different values. These could be a combination of both hardand soft data items that make up a particular index value. An index isa single score representing some complex factor that is constructed byaggregating the values of several different measures. Measures makingup the index are sometimes weighted based on their importance to theabstract factor being measured. Some index measures are based strictlyon hard data items. For example, the U.S. poverty level is based on afamily income amount equal to three times the money needed to feed afamily as determined by the U.S. Department of Agriculture, adjustedfor inﬂation using the consumer price index. Sometimes an index iscompletely intangible, such as the customer satisfaction index developedby the University of Michigan.Intangibles are often combined with a variety of tangibles to reﬂectthe performance of a business unit, function, or project. Intangibles arealso often associated with nonproﬁt, nongovernment, and public sectororganizations. Table 10.2 shows the performance measures reﬂectinggreatness at the Cleveland Orchestra. For the Cleveland Orchestra,intangibles include such items as comments from cab drivers; tangiblesinclude ticket sales. Collectively and regardless of how difﬁcult they areto obtain, these data sets reﬂect the overall performance of the orchestra.Converting to MoneyConverting the hard-to-measure to monetary values is challenging, tosay the least. Examples in this chapter show various attempts to convertthese hard-to-value measures to monetary values. When working withintangibles, the interest in the monetary contribution expands consider-ably. Three major groups have an interest in the monetary value. Firstare the sponsors who fund a particular project. They almost always seekmonetary values among the measures. Second, the public is involved
Measuring and Analyzing Intangibles 183Table 10.2 Measuring Greatness at the Cleveland OrchestraSuperior Performance Distinctive Impact Lasting Endurance– Emotional response ofaudience; increase innumber of standingovations– Wide technical range;can play any piece withexcellence, no matterhow difﬁcult—fromsoothing and familiarclassical pieces todifﬁcult and unfamiliarmodern pieces– Increased demand fortickets; demand formore complex,imaginative programsin Cleveland, NewYork, and Europe– Invited to SalzburgFestival (ﬁrst time in25 years), signifyingelite status among topEuropean orchestras– Cleveland’s style ofprogrammingincreasingly copied;becoming moreinﬂuential– A key point of civicpride; cab driverssay, ‘‘We’re reallyproud of ourorchestra.’’– Orchestra leadersincreasingly soughtfor leadership rolesand perspectives inelite industrygroups andgatherings– Excellencesustained acrossgenerations ofconductors—fromGeorge Szellthrough PierreBoulez, Christophvon Dohnanyi, andFranz Wilser-Most– Supporters donatetime and money;invest in long-termsuccess of orchestra;endowment triples(Source: Adapted from Good to Great and the Social Sector, Jim Collins, 2005.)in some way with many intangibles. Even private sector organizationsare trying to improve their image and reputation, and conﬁdence intheir organizations in the minds of the public. These days, the public isinterested in the ﬁnancial impacts of these organizations. They are nolonger willing to accept the notion that the intangibles are enough tofund projects, particularly if they are funded by tax dollars. Third, theindividuals who are actively involved with and support the project oftenneed, and sometimes demand, that the monetary value be developed.The approaches to convert to monetary values were detailed inChapter 9. The speciﬁc methods used in that chapter all representapproaches that may be used to convert the intangibles to monetaryvalues. Although these will not be repeated here, showing the pathmost commonly used to capture values for the intangibles is helpful.Figure 10.2 shows the typical path of converting intangibles to monetary
184 MEASURING THE INTANGIBLESValuing the Hard-to-ValueApproach ChallengeExisting data Find the right databaseExpert input Locating the “credible” expertStakeholder input Making the data credibleAnalysis of data ResourcesFigure 10.2 Converting to money.values, building on the methods in Chapter 9. The ﬁrst challenge is tolocate existing data or measure them in some way, making sure theyare accurate and reliable. Next, an expert may be able to place a mone-tary value on the item based on experience, knowledge, credentials, andtrack record. Stakeholders may provide their input, although it shouldbe factored for bias. Some stakeholders are biased in one way or theother—they want the value to be smaller or larger depending on theirparticular motives. These may have to be adjusted or thrown out alltogether. Finally, the data are converted using the conservative processesdescribed in Chapter 9, often adjusting for the error in the process.Unfortunately, no speciﬁc rule exists for converting each intangible tomonetary value. By deﬁnition, an intangible is a measure that is not con-verted to money. If the conversion cannot be accomplished with minimumresources and with credibility, it is left as an intangible.Identifying and Collecting IntangiblesIntangible measures can be taken from different sources and at differenttimes during the project lifecycle, as depicted in Figure 10.3. They canbe uncovered early in the process, during the needs assessment, andtheir collection can be planned for as part of the overall data collectionstrategy. For example, one technology project has several hard datameasures linked to it. Job stress, an intangible measure, is identiﬁedand monitored with no plans to convert it to a monetary value. Fromthe beginning, this measure is destined to be a nonmonetary, intangiblebeneﬁt reported along with the ROI results.
Measuring and Analyzing Intangibles 185Intangible Measures during the Project Life CycleIntangiblemeasuresInitialanalysis1 2ImplementationDatacollection3Dataanalysis4IntangiblemeasuresIntangiblemeasuresIntangiblemeasuresFigure 10.3 Identifying intangible measures during the project lifecycle.A second opportunity to identify intangible beneﬁts is in the planningprocess, when clients or sponsors of the project agree on an evaluationplan. Key stakeholders can usually identify the intangible measuresthey expect to be inﬂuenced by the project. For example, a changemanagement project in a large multinational company was conducted, andan ROI analysis was planned. Project leaders, participants, participants’managers, and experts identiﬁed potential intangible measures thatwere perceived to be inﬂuenced by the project, including collaboration,communication, and teamwork.A third opportunity to collect intangible measures presents itself dur-ing data collection. Although the measure may not be anticipated in theinitial project design, it may surface on a questionnaire, in an inter-view, or during a focus group. Questions are often asked about otherimprovements linked to a project, and participants usually provide sev-eral intangible measures for which no plans are available to assign avalue. For example, in the evaluation of a quality project, participantswere asked what speciﬁcally had improved about their work area andrelationships with customers as a result of the project. Participants pro-vided more than a dozen intangible measures that managers attributedto the project.The fourth opportunity to identify intangible measures is during dataanalysis and reporting, while attempting to convert data to monetaryvalues. If the conversion loses credibility, the measure should be reportedas an intangible beneﬁt. For example, in one sales improvement project,customer satisfaction was identiﬁed early in the process as a measure ofproject success. A conversion to monetary values was attempted, but it
186 MEASURING THE INTANGIBLESlacked accuracy and credibility. Consequently, customer satisfaction wasreported as an intangible beneﬁt.Analyzing IntangiblesFor each intangible measure identiﬁed, some evidence of its connection tothe project must be shown. However, in many cases, no speciﬁc analysisis planned beyond tabulation of responses. Early attempts to quantifyintangible data sometimes resulted in aborting the entire process, withno further data analysis being conducted. In some cases, isolating theeffects of the project may be undertaken using one or more of the methodsoutlined in Chapter 9. This step is necessary when project leaders needto know the speciﬁc amount of change in the intangible measure that islinked to the project. Intangible data often reﬂect improvement. However,neither the precise amount of improvement nor the amount of improve-ment directly related to a project is always identiﬁed. Because the valueof these data is not included in the ROI calculation, intangible measuresare not normally used to justify another project or to justify continuing anexisting project. A detailed analysis is not necessary. Intangible beneﬁtsare often viewed as additional evidence of the project’s success and arepresented as supportive qualitative data.CONFRONTING INTANGIBLESThere are so many intangibles that addressing them appropriately canbe difﬁcult. Many advances have been made in measuring intangibleseffectively and in converting them to money. Of course, when they areconverted to money, they are no longer intangible—they are tangible.This issue is not easy, but progress is being made and will continue to bemade. The following section covers ﬁve areas on which organizations arefocusing in measuring intangibles in the public and private sector. Thesemeasures include many of the examples listed in Table 10.1.Customer ServiceBecause of the importance of building and improving customer service,related measures are typically monitored to track project payoff. Sev-eral types of customer service projects have a direct inﬂuence on these
Confronting Intangibles 187measures. This metric makes our list because it is perceived as difﬁcultto measure and to convert to monetary value. However, in the last twodecades, much progress has been made in this area, and some of thesemeasures are routinely considered tangible because they are converted tomoney using the measures described in Chapter 9, where the techniqueof linking to other measures clearly illustrates the most common way inwhich customer service intangible measures are converted to money. Thistechnique follows the sequence shown in Figure 10.4. The ﬁrst step is tocreate awareness of a particular product, brand, or service. The next stepis to develop attitudes that deﬁne the beliefs, opinions, and intentionsregarding the product, service, or brand, and leads to usage, the ﬁnal stepthat conﬁrms the purchasing habits and loyalty of the customer.This important link is ingrained in most marketing and promotionprojects and processes and has led to a variety of measures that arebecoming standard in the industry. Table 10.3 shows customer intangi-bles and underscores the array of possibilities—all aimed at developingawareness, attitudes, and usage. Perhaps the most common intangible iscustomer satisfaction, which is generally measured on scales of 1 to 5,1 to 7, or 1 to 10 (although other scales are used, too).4 A tremendousamount of research has been accumulated about the value of satisﬁedcustomers and the loss connected with dissatisﬁed customers. Using elab-orate processes of decision tree analysis, probability theories, expectedvalue, and correlations, organizations have developed detailed monetaryvalues showing that movement in sales and proﬁts is connected to a vari-ety of measures. The most important measure is customer satisfaction.Awareness, Attitudes, and Usage: Typical QuestionsType Measures Typical QuestionsAwareness Awareness andknowledgeAttitudes Beliefs and intentionsUsage Purchasing habits andloyaltyHave you heard of Brand X?What brand comes to mindwhen you think “luxury car”?Is Brand X for me?On a scale of 1 to 5, is Brand Xfor young people?Did you use Brand X this week?What brand did you last buy?Figure 10.4 Customer service linkage.
188 MEASURING THE INTANGIBLESTable 10.3 Customer Service IntangiblesMetric Deﬁnition Issues PurposeAwareness Percentage of totalpopulation who areaware of a brandIs awarenessprompted orunprompted?Consideration ofwho has heard ofthe brandTop of mind First brand to beconsideredMay be subject tomost recentadvertising orexperienceSaliency of brandKnowledge Percentage ofpopulation whoknow product,have recollection ofits advertisingNot a formalmetric. Isknowledgeprompted orunprompted?Extent offamiliarity withproduct beyondname recognitionBeliefs Customers’/consumers’ view ofproduct, generallycaptured viasurvey responses,often throughratings on a scaleCustomers/consumers mayhold beliefs withvarying degrees ofconvictionPerception ofbrand by attributePurchasingintentionsProbability ofintention topurchaseTo estimateprobability ofpurchase,aggregate andanalyze ratings ofstated intentions(for example, toptwo boxes)Measurespre-shoppingdisposition topurchaseWillingnesstorecommendGenerallymeasured byratings on scale of1 to 5Nonlinear inimpactShows strength ofloyalty, potentialimpact on othersCustomersatisfactionGenerallymeasured on scaleof 1 to 5, in whichcustomers declaresatisfaction withbrand in general orwith speciﬁcattributesSubject to responsebias; capturesviews of currentcustomers, not lostcustomers;satisfaction isfunction ofexpectationsIndicateslikelihood ofrepurchase;reports ofdissatisfactionshow aspectsrequiringimprovement toenhance loyalty(continues)
Confronting Intangibles 189Table 10.3 (Continued)Metric Deﬁnition Issues PurposeWillingnessto searchPercentage ofcustomers willingto delay purchases,change stores, orreduce quantitiesto avoid switchingbrandsHard to capture Indicatesimportance ofdistributioncoverageLoyalty Measures includewillingness to paypremium, tosearch, to stay‘‘Loyalty’’ itself nota formal metric,but speciﬁc metricsdo measure aspectsof this dynamic.New productentries may alterloyalty levelsIndication of basefuture revenuestream(Sources: Adapted from Marketing Metrics: 50+ Metrics Every Executive ShouldMaster by Paul W. Farris, Neil T. Bendle, Phillip E. Pfeifer, and David J. Ribstein.Upper Saddle River, NJ: Wharton School Publishing, 2006, p. 16.)Within an organization, a variety of speciﬁc measures can be devel-oped, including customer response time, sensitivity to costs and pricingissues, and creativity with customer responses. Of particular importanceis the matter of timing. Providing prompt customer service is critical formost organizations. Therefore, organizations monitor the time requiredto respond to speciﬁc customer service requests or problems. Althoughreducing response times is a common objective, the measure is not usuallyconverted to a monetary value. Thus, customer response time is usuallyreported as an important intangible measure.Innovation and CreativityInnovation and creativity are related. Creative employees create innova-tive products, services, and solutions. In our knowledge- and technology-based economy, innovation and creativity are becoming important factorsin organizations’ success.InnovationInnovation is critical to most organizations. Just how important is innova-tion? Let’s put it in perspective. If it were not for the intellectual curiosity
190 MEASURING THE INTANGIBLESof employees—thinking things through, trying out new ideas, and takingwild guesses in all the R&D labs across the country—the United Stateswould have half the economy it has today. In a recent report on R&D, theAmerican Association for the Advancement for Science estimated that asmuch as 50 percent of U.S. economic growth in the half century since theFortune 500 came into existence is the result of advances in technology.5After a few years’ retrenchment and cost cutting, senior executives froma variety of industries now share the conviction that innovation—the abil-ity to deﬁne and create new products and services and quickly bring themto market—is an increasingly important source of competitive advantage.Executives are setting aggressive performance goals for their innova-tion and product development organizations, targeting 20 to 30 percentimprovements in such areas as time to market, development cost, productcost, and customer value.6A vast disconnect lies between hope and reality, however. A recentsurvey of 50 companies conducted by Booz Allen Hamilton shows thatcompanies are only marginally satisﬁed that their innovation organi-zations are delivering their maximum potential. Worse, executives saythat only half the improvement efforts they launch end up meetingexpectations. Several waves of improvement in innovation and productdevelopment have already substantially enhanced companies’ abilities todeliver differentiated, higher-quality products to markets faster and moreefﬁciently. However, the degree of success achieved has varied greatlyamong companies and among units within companies. The differencesin success stem from the difﬁculty in managing change in the com-plex processes and organizations associated with innovation and productdevelopment.Some companies have managed to assemble an integrated ‘‘innovationchain’’ that is truly global and allows them to outﬂank competitors thatinnovate using knowledge in a single cluster. They have been able toimplement a process for innovating that transcends local clusters andnational boundaries, becoming ‘‘meta-national innovators.’’ This strategyof using localized pockets of technology, market intelligence, and capabil-ities has provided a powerful new source of competitive advantage: morehigher-value innovation at lower cost.Innovation is both easy and difﬁcult to measure. Measuring the out-comes in areas such as new products and processes, improved productsand processes, copyrights, patents, inventions, and employee suggestionsis easy. Many companies track these items. They can be documented to
Confronting Intangibles 191reﬂect the innovative proﬁle of an organization. Unfortunately, comparingthese data with previous data or benchmarking with other organiza-tions is difﬁcult because these measures are typically unique to eachorganization.Perhaps the most obvious measure is tracking patents that are bothused internally and licensed for others’ use through a patent and licenseexchange. For example, IBM has been granted more patents than anyother company in the world—more than 25,000 U.S. patents. IBM’slicensing of patents and technology generates several billion dollars inproﬁts each year. IBM and Microsoft are at the top of the list, butmost organizations in the new economy monitor trademarks, patents,and copyrights as important measures of the innovative talent of theiremployees.It is helpful to remember that the registration of patents stems fromemployees’ inventive spirit. The good news is that employees do not haveto be highly degreed scientists or engineers to be inventive. Althoughinvention is often thought of in the context of technology, computing,materials, or energy, in fact it spans all disciplines and can thereforebe extracted from any technological realm for application to problems inany area.Through the years, inventors have been viewed as ‘‘nerds,’’ with muchof their inventiveness explained by their quirky personality makeup. Thisis because history is laced with well-known inventors endowed with aneccentric personality. In fact, inventors are usually ordinary people whopossess extraordinary imagination. Many modern organizations of wide-ranging focus are devoting resources to the encouragement of employeecreativity from which they will gain advantage over their competition.Organizations intent on sparking ingenuity will consider innovation,monitor it, and take action to enhance it.CreativityCreativity, often considered the precursor to innovation, encompassesthe creative experience, actions, and input of organizations. Measuringthe creative spirit of employees may prove more difﬁcult. An employeesuggestion system, a long-time measure of the creative processes ofthe organization, ﬂourishes today in many organizations and is easilymeasured. Employees are rewarded for their suggestions if they areapproved and implemented. Tracking the suggestion rates and comparing
192 MEASURING THE INTANGIBLESthem with other organizations is an important benchmarking item forcreative capability. Other measures that can be monitored are the numberof new ideas, comments, or complaints. Formal feedback systems oftencontain creative suggestions that can lead to improved processes.Some organizations measure the creative capabilities of employeesusing inventories and instruments that may be distributed at meetingsand training sessions. In other organizations, a range of statementsabout employee creativity is included in the annual employee feedbacksurvey. Using scaled ratings, employees either agree or disagree with thestatements. Comparing actual scores of groups of employees over timereﬂects the degree to which employees perceive improvement in creativityin the workplace. Having consistent and comparable measures is still achallenge. Other organizations may monitor the number, duration, andparticipation rate of creativity training projects. The last decade haswitnessed a proliferation of creativity tools, projects, and activity.Employee AttitudesEmployee SatisfactionAn important item monitored by most organizations is employee jobsatisfaction. Using feedback surveys, executives can monitor the degreeto which employees are satisﬁed with their employer’s policies, workenvironment, and supervision and leadership; with the work itself; andwith other factors. A composite rating may be developed to reﬂect anoverall satisfaction value or an index for the organization, division,department, or region.Whereas job satisfaction has always been an important factor inemployee relations, in recent years it has taken on a new dimensionbecause of the link between job satisfaction and other measures. Therelationship between job satisfaction and the attraction and retentionof employees is classic, in that ﬁrms with excellent job satisfactionratings have better success in attracting the most desirable employ-ees. Job satisfaction ratings high enough that companies can be titledamong the ‘‘Employers of Choice’’ or ‘‘Best Places to Work’’ have gaineda subtle but powerful recruiting tool. The recent heightened empha-sis on the relationship between job satisfaction and employee retentionowes to turnover and retention being such critical issues. These rela-tionships are now easily developed using human capital management
Confronting Intangibles 193systems featuring modules to calculate the correlation between turnoverrates and job satisfaction scores for various job groups, divisions, anddepartments.Job satisfaction has taken on new dimensions in connection withcustomer service. Dozens of applied research projects are beginning toshow a high correlation between job satisfaction scores and customersatisfaction scores. Intuitively, one understands that a more satisﬁedemployee is likely to provide more productive, friendly, and appropriatecustomer service. Likewise, a disgruntled employee will provide poor ser-vice. Research has established that job attitudes (job satisfaction) relateto customer impression (customer satisfaction), which relates to revenuegrowth (proﬁts). Therefore, the conclusion follows that if employee atti-tudes improve, revenues increase. These links, often referred to as aservice proﬁt chain, create a promising way to identify important rela-tionships between attitudes within an organization and the proﬁts theorganization earns.Organizational CommitmentIn recent years, organizational commitment (OC) measures have comple-mented or replaced job satisfaction measures. OC measures go beyondemployee satisfaction to include the extent to which the employees iden-tify with the organization’s goals, mission, philosophy, values, policies,and practices. The concept of involvement and commitment to the orga-nization is a key issue. OC more closely correlates with productivityand other performance improvement measures, whereas job satisfactionusually does not. OC is often measured in the same way as job satis-faction, using attitude surveys based on a 5-point or 7-point scale, andadministered directly to employees. As organizational commitment scores(taken as a standard index) improve, a corresponding improvement inproductivity should also be seen.Employee EngagementA different twist to the OC measure is one that reﬂects employee engage-ment. This involves the measures that indicate the extent to whichemployees are actively engaged in the organization. Consider the caseof a large British bank. With more than 115,000 employees, this bankconsidered it a strategic imperative to measure the effectiveness of itsinvestment in people and the impact of this investment on business
194 MEASURING THE INTANGIBLESperformance. Consequently, this bank built, validated, and introduceda human capital model that demonstrably links ‘‘people strategies’’ toperformance.This progressive bank moved beyond monitoring employee satisfactionand commitment to measuring whether employees actively improvedbusiness results. The bank did this using an employee engagement modelthat assesses employees’ likelihood of contributing to business proﬁts.The model linked separate elements of human resource (HR) informationin a consistent way, which it then linked to key business indicators. Theoutputs enabled senior executives to understand how to inﬂuence thebank’s results through its workforce.To test and validate its model, HR research and measurement teamreviewed the array of survey instruments used in HR activities. The HRteam decided to put the employee engagement model into practice in theprocessing and customer contact centers, where productivity measuresare very important as they relate to customer service. Using the amountof work processed as a throughput measure, the team found that produc-tivity increased in tandem with engagement levels. They were also ableto establish a correlation between increasing engagement and decreasingstaff turnover.Hundreds of organizations now use engagement data, reﬂecting theextent to which employees are engaged and how their engagement con-nects with productivity and turnover.LeadershipLeadership is perhaps the most difﬁcult measure to address. On thesurface, it would seem easy to measure the outcome because effectiveleadership leads to an effective organization. However, putting a mone-tary value on the consequences of new leadership behavior is not as easyas it appears. Leadership can (and usually does) determine the successor failure of an organization. Without appropriate leadership behaviorsthroughout an organization, resources can be misapplied or wasted, andopportunities can be missed. The news and literature are laced withexamples of failed leadership at the top, and accounts of mismanagedemployees, shareholders, investors, and the public. Some of these high-proﬁle failed leadership stories have been painful. At the same time,positive examples exist of leaders—for example, former General ElectricCEO Jack Welch—who have won extraordinary success at many levels
Confronting Intangibles 195of their organization over a sustained period. These leaders are oftendocumented in books, articles, and lists of admiration. They clearly makethe difference in their organizations. Obviously, the ultimate measure ofleadership is the overall success of the organization. Whenever overallmeasures of success have been achieved or surpassed, they are alwaysattributed to great leadership—perhaps rightfully so. However, attempt-ing to use overall success as the only measure of leadership is a cop-outin terms of accountability. Other measures must be in place to developsystem-wide monitoring of leaders and leadership in the organization.360◦FeedbackLeadership can be measured in many different ways, perhaps the mostcommon of which is known as 360◦feedback. Here, a prescribed set ofleadership behaviors desired in the organization is assessed by differentsources to provide a composite of overall leadership capability and behav-ior. The sources often consist of the immediate manager of the leaderbeing assessed, a colleague in the same area, the employees directlysupervised by the leader, internal or external customers, and the leader’sself-assessment. Combined, these assessments form a circle of inﬂuence(360◦). The measure is basically an observation of behavior captured in asurvey, often reported electronically. This 360◦feedback has been grow-ing rapidly in the United States, Europe, and Asia as an important wayto capture overall leadership behavior change. Because the consequencesof behavior change are usually measured as business impact, leadershipimprovement should be linked to the business in some way.Leadership InventoriesAnother way to measure leadership is to require the management team toparticipate in a variety of leadership inventories, assessing predeterminedleadership competency statements. The inventories reﬂect the extent towhich a particular leadership style, approach, or even success is in place.These inventories, although popular in the 1970s and 1980s, are todayoften being replaced by the 360◦feedback process.Leadership PerceptionIt is also useful to capture the quality of leadership from the perspectiveof employees. In some organizations, employees regularly rate the quality
196 MEASURING THE INTANGIBLESof their leadership. Top executives and middle managers are typicallythe subjects of this form of evaluation. The measure is usually taken inconjunction with the annual feedback survey, in the form of direct state-ments about the executive or immediate manager with which respondentsagree or disagree using a 5-point scale. This survey attempts to measurehow the followers in a particular situation perceive the quality, success,and appropriateness of leadership as exercised by their managers.Business ImpactThe outcomes of leadership development are clearly documented in manycase studies involving ROI analysis. Of the thousands of studies con-ducted annually, leadership development ROI studies are at the top ofthe list—not because conducting them is easier but because of the uncer-tainty and the unknown aspects of investing in leadership development.Most leadership development will have an impact in a particular leader’sarea. Leadership development produces new skills that are applied onthe job and produces improvements in the leader’s particular work unit.These can vary signiﬁcantly. The best way to evaluate a general leader-ship development project involving executives and leaders from a varietyof areas is with respect to the monetary impact. When particular mea-sures are improved, examining those measures individually makes littlesense. Examining the monetary value of each measure as a whole ismore worthwhile. The measures are converted to monetary values usingone of the methods discussed in Chapter 9. The monetary values of theimprovements from the ﬁrst year are combined into a total value, ulti-mately leading to an ROI calculation. Leadership development projectsaimed at improving leadership behavior and driving business improve-ment often yield high payoff, with ROI values that range from 500 percentto 1,000 percent.7 This is primarily because of the multiplicative effectas leaders are developed and changes of behavior inﬂuence importantmeasures in the leaders’ teams.Human LifeSeeing human life listed as an intangible may be surprising to some. It’snot intangible because of the difﬁculty in measuring it—obviously, bodiescan be counted. Human life is often considered to be intangible because
Confronting Intangibles 197of the difﬁculty in converting the value of life to money. Yet, this value isrequired in many projects and has been used in hundreds of studies. Theissue spans both the private sector and the public sector.Consider, for example, a recommendation by federal health ofﬁcialsthat 11- and 12-year-old girls be routinely vaccinated against the sex-ually transmitted human papilloma virus (HPV) that causes cervicalcancer. This recommendation has been endorsed by several federal healthagencies, and by insurance companies. One health insurance company,Wellpoint, Inc., announced its intention to cover the vaccine. What wouldlead Wellpoint to this decision? Insurance companies are willing to payfor the vaccine because it can reduce the number of claims, and ultimatelythe number of deaths, caused by cervical and vaginal cancers. Each yearin the United States alone, approximately 4,000 women die from cervicalcancer. The HPV vaccine Gardasil is expected to dramatically reduce thisnumber. The results are even more staggering when it is considered thatone or more types of HPV will infect more than 50 percent of sexuallyactive women in their lifetimes. On the logic of the situation, coveringthe vaccination pays off for insurance companies, but only if monetaryvalues are established for the cost of treating cancer (which should existin insurance company records), and for the cost of a death (which alsoshould exist in company records). Compensation limits are often deﬁnedby the amount of life insurance carried by the person who dies.The value of a life comes into question more clearly in public policyprojects, as when the government acts to try to prevent death. A cost-beneﬁt analysis can quickly show what is economically sound to attempt.An example is the valuation of a life according to the U.S. EnvironmentalProtection Agency, which places the value at $6.1 million. This estimatewas developed in 2000 to evaluate the beneﬁts of removing arsenic fromdrinking water. Since then, many government studies have placed avalue on a human life, and the results vary considerably, ranging from$1.5 to $5.8 million. The factors involved in placing a value on a humanlife include the person’s age, economic status, education, and potential.Consequently, not every life is of equal value.The private sector is very interested in this issue because of the risksassociated with everyday dangers in the safety and health sectors of theeconomy. Employers must be concerned about casualties that may resultfrom on-the-job accidents, and about their liability exposure. The valueof a human life is a constant concern in the arena of risk management.
198 MEASURING THE INTANGIBLESConsider, for example, a health care chain that is considering a newrisk management procedure that would reduce the likelihood of acts ofinfant abduction. Although the infant may not be killed during the actof abduction, assumptions must be made about such costs. To conductthis type of analysis, the probability of infant abduction without the riskmanagement procedure is compared to the probability of infant abductionwith the procedure in place, based on assumptions made by those whounderstand such processes well. The cost of the risk management proce-dure, which would be accounted for as an extra direct expense, is known.What must be identiﬁed to make the estimate complete is the cost of anabduction. Previous liability claims can be reviewed to estimate what itwould cost the hospital should an abduction occur, based on the valueof the human life. When this is known, it is a matter of using the ROImethodology to determine whether the risk management procedure iseconomically justiﬁed. To some, this would seem absurd, and no amountof money would be considered too great to prevent the abduction or eventhe death of an infant. However, organizations have limits on what theycan afford and are willing to pay.As with many intangibles, this one generates others. For example, lossof life not only generates pain and suffering for the family, but can alsolower morale and can even tarnish the image of an organization. Forexample, the energy company British Petroleum (BP) saw its stock pricenose-dive when unsafe conditions led to fatalities. BP’s safety record wasamong the worst in the industry. After an explosion on an oil platform,investors grew alarmed and began to sell shares. This obviously was animage problem that spooked investors. A damaged public image is anexpensive intangible that can generate other economic impacts as well.In summary, human life is considered an intangible primarily becauseof the perceived difﬁculty of placing a monetary value on life. However,a human life can be valued and human life is being valued routinely,making it more likely to be measured as a tangible in the future.FINAL THOUGHTSGet the picture? Intangible measures are crucial to reﬂecting the suc-cess of a project. Although they may not carry the weight of measuresexpressed in monetary terms, they are nevertheless an important part ofthe overall evaluation. Intangible measures should be identiﬁed, explored,
Final Thoughts 199examined, and monitored for changes linked to projects. Collectively, theyadd a unique dimension to the project report because most, if not all,projects involve intangible variables. Although ﬁve common intangiblemeasures are explored in some detail in this chapter, the coverage iswoefully incomplete. The range of intangibl e measures is practicallylimitless. Now the data have been organized and converted to monetaryvalues. Intangible measures have been identiﬁed. Next the costs must becaptured and the ROI calculated. This will be covered in the next chapter.
202 MONITORING PROJECT COSTS AND CALCULATING ROIand encourages the entire project team to spend wisely. And, of course,monitoring costs in an ongoing fashion is much easier, more accurate,and more efﬁcient than trying to reconstruct events to capture costsretrospectively. Developing accurate costs by category builds a databasefor understanding and predicting costs in the future.Monitoring project costs is an essential step in developing the ROIcalculation because it represents the denominator in the ROI formula.ROI has become a critical measure demanded by many stakeholders,including clients and senior executives. It is the ultimate level of evalua-tion, showing the actual payoff of the project, expressed as a percentageand based on the same formula as the evaluation for other types of capitalinvestment.A brief example will highlight the importance of costs and ROI. A newsuggestion system was implemented in a large electric utility. Thisnew plan provided cash awards for employees when they submitted asuggestion that was implemented and resulted in cost savings. Thisproject was undertaken to help lower the costs of this publicly ownedutility. As the project was rolled out, the project leaders captured reactionto ensure that the employees perceived the suggestion system as fair,equitable, motivating, and challenging. At Level 2, they measured learn-ing to make sure that the employees understood how to document theirsuggestions and how and when the awards were made. Application data(Level 3) were the actual submission of the awards, and the company hadthe goal of a 10 percent participation rate. Level 4 data corresponded tothe actual monetary value. In this case, $1.5 million was earned or savedover a two-year period.In most organizations, the evaluation would have stopped there. Theproject appeared to be a success, as the goals were met at each of the fourlevels. Bring on the champagne! However, the costs of the project for thesame two-year period totaled $2 million. Thus, the utility company spent$2 million to have $1.5 million returned. This is a negative ROI, and itwould not have been recognized if the ultimate measure, the ROI, hadnot been developed. (Incidentally, a negative ROI might be acceptableby some executives. After all, the intangibles for the utility showedincreased commitment, engagement, ownership, teamwork, cooperation,and communications. However, if the objective was a positive ROI, thissystem failed to achieve it, primarily because of excessive administrativecosts.)
Fundamental Cost Issues 203FUNDAMENTAL COST ISSUESThe ﬁrst step in monitoring costs is to deﬁne and address issues relatingto cost control. Several rules apply to tabulating costs. Consistency andstandardization are necessary. A few guidelines follow:• Monitor all costs, even if they are not needed.• Costs must be realistic and reasonable.• Costs will not be precise; estimates are okay.• Disclose all costs.Other key issues are detailed later in this section.Fully Loaded CostsWhen a conservative approach is used to calculate the ROI, costs shouldbe fully loaded, which is Guiding Principle 10 (see Chapter 3). With thisapproach, all costs (direct and indirect) that can be identiﬁed and linkedto a particular project are included. The philosophy is simple: For thedenominator, ‘‘when in doubt, put it in,’’ i.e., if there is any question as towhether a cost should be included, include it, even if the cost guidelinesfor the organization do not require it. When an ROI is calculated andreported to target audiences, the process should withstand even theclosest scrutiny to ensure its credibility. The only way to meet this test isto include all costs. Of course, from a realistic viewpoint, if the controlleror chief ﬁnancial ofﬁcer insists on not using certain costs, then leavingthem out or reporting them in an alternative way is best.Costs Reported without BeneﬁtsBecause costs can easily be collected, they are presented to managementin many ingenious ways, such as in terms of the total cost of the project,cost per day, and cost per participant. While these may be helpful forefﬁciency comparisons, presenting them without identifying the corre-sponding beneﬁts may be problematic. When most executives reviewproject costs, a logical question is raised: What beneﬁt was received fromthe project? This is a typical management reaction, particularly whencosts are perceived to be very high.Unfortunately, many organizations have fallen into this trap. Forexample, in one organization, all the costs associated with a major
204 MONITORING PROJECT COSTS AND CALCULATING ROItransformation project were tabulated and reported to the seniormanagement team, relaying the total investment in the project. From anexecutive perspective, the total ﬁgure exceeded the perceived value ofthe project, and the executive group’s immediate reaction was to requesta summary of (monetary and nonmonetary) beneﬁts derived from theoverall transformation. The conclusion was that few, if any, economicbeneﬁts were achieved from the project. Consequently, budgets forsimilar projects were drastically reduced in the future. While this maybe an extreme example, it shows the danger of presenting only half theequation. Because of this, some organizations have developed a policy ofnot communicating cost data unless the beneﬁts can be captured andpresented along with the costs, even if the beneﬁts are subjective andintangible. This helps maintain a balance between the two components.Develop and Use Cost GuidelinesWhen multiple projects are being evaluated, it may be helpful to detailthe philosophy and policy on costs in the form of guidelines for theevaluators or others who monitor and report costs. Cost guidelines detailspeciﬁcally which cost categories are included with projects and howthe data are captured, analyzed, and reported. Standards, unit costguiding principles, and generally accepted values are included in theguidelines. Cost guidelines can range from a one-page brief to a hundred-page document in a large, complex organization. The simpler approachis better. When fully developed, cost guidelines should be reviewed andapproved by the ﬁnance and accounting staff. The ﬁnal document servesas the guiding force in collecting, monitoring, and reporting costs. Whenthe ROI is calculated and reported, costs are included in summary ortable form, and the cost guidelines are usually referenced in a footnote orattached as an appendix.Sources of CostsIt is sometimes helpful to ﬁrst consider the sources of project costs. Fourmajor categories of sources are illustrated in Table 11.1. The chargesand expenses from the project team represent the major segment of costsand are transferred directly to the client for payment. These are oftenplaced in subcategories under fees and expenses. A second major costcategory relates to the vendors or suppliers who assist with the project.
Fundamental Cost Issues 205Table 11.1 Sources of Project CostsSource of Costs Cost Reporting IssuesProject team fees and expenses • Costs are usually accurate• Variable expenses are usuallyunderestimatedVendor/suppliers fees and expenses • Costs are usually accurate• Variable expenses are usuallyunderestimatedClient expenses, direct and indirect • Direct expenses are usually notfully loaded• Indirect expenses are rarelyincluded in costsEquipment, services, and other expenses • Sometimes understated• May lack accountabilityA variety of expenses, such as consulting or advisory fees, may fall inthis category. A third major cost category is those expenses borne bythe client organization—both direct and indirect. In many projects, thesecosts are not identiﬁed but nevertheless are part of the costs of the project.The ﬁnal cost category involves expenses not covered in the other threecategories. These include payments for equipment and services neededfor the project. Finance and accounting records should track and reﬂectthe costs from these different sources, and the process presented in thischapter can also help track these costs.Prorated versus Direct CostsUsually all costs related to a project are captured and expensed to thatproject. However, some costs are prorated over a longer period. Equipmentpurchases, software development and acquisitions, and the constructionof facilities are all signiﬁcant costs with a useful life that may extendbeyond the project. Consequently, a portion of these costs should beprorated to the project. Under a conservative approach, the expected lifeof the project is ﬁxed. Some organizations will assume a period of oneyear of operation for a simple project. Others may consider three to ﬁveyears appropriate. If a question is raised about the speciﬁc time periodto be used in this calculation, the ﬁnance and accounting staff should beconsulted, or appropriate guidelines should be developed and followed.
206 MONITORING PROJECT COSTS AND CALCULATING ROIEmployee Beneﬁts FactorEmployee time is valuable, and when time is required for a project, thecosts must be fully loaded, representing total compensation, includingemployee beneﬁts. This means that the employee beneﬁts factor shouldbe included. This number is usually well known in the organization andis used in other costing formulas. It represents the cost of all employeebeneﬁts expressed as a percentage of payroll. In some organizations, thisvalue is as high as 50 to 60 percent. In others, it may be as low as 25 to30 percent. The average in the United States is 38 percent.1SPECIFIC COSTS TO INCLUDETable 11.2 shows the recommended cost categories for a fully loaded, con-servative approach to estimating project costs. Consistency in capturingall these costs is essential, and standardization adds credibility. Eachcategory is described in this section.Table 11.2 Project Cost CategoriesCost Item Prorated ExpensedInitial analysis and assessmentDevelopment of projectsolution/contentAcquisition of project solutionImplementation and applicationSalaries/beneﬁts for project team timeSalaries/beneﬁts for coordination timeSalaries/beneﬁts for participant timeProject materialsHardware/softwareTravel/lodging/mealsUse of facilitiesCapital expendituresMaintenance and monitoringAdministrative support and overheadEvaluation and reporting
Speciﬁc Costs to Include 207Initial Analysis and AssessmentOne of the most underestimated items is the cost of conducting the initialanalysis and assessment that leads to the project. In a comprehensiveproject, this involves data collection, problem solving, assessment, andanalysis. In some projects, this cost is near zero because the project isconducted without an appropriate assessment. However, as more projectsponsors place attention on needs assessment and analysis in the future,this item will become a signiﬁcant cost.Development of SolutionsAlso signiﬁcant are the costs of designing and developing the projectsolution. These costs include time spent in both the design and devel-opment and the purchase of supplies, technology, and other materialsdirectly related to the solution. As with needs assessment costs, designand development costs are usually charged to the project. However, ifthe solution can be used in other projects, the major expenditures can beprorated.Acquisition CostsIn lieu of development costs, some project leaders use acquisition costsconnected to the purchasing of solutions from other sources to use directlyor in a modiﬁed format. The costs for these solutions include the purchaseprice, support materials, and licensing agreements. Some projects haveboth acquisition costs and solution development costs. Acquisition costscan be prorated if the acquired solutions can be used in other projects.Application and Implementation CostsThe largest cost segment in a project is associated with implementationand delivery. The time (salaries and beneﬁts), travel, and other expensesof those involved in the project in any way should be included. These costscan be estimated using average or midpoint salary values for correspond-ing job classiﬁcations. When a project is targeted for an ROI calculation,participants can provide their salaries directly in a conﬁdential manner.Project materials, such as ﬁeld journals, instructions, reference guides,case studies, surveys, and participant workbooks, should be included in
208 MONITORING PROJECT COSTS AND CALCULATING ROIthe implementation costs, along with license fees, user fees, and royaltypayments. Supporting hardware, software, CD-ROMs, and videos shouldalso be taken into account.The cost for the use of facilities needed for the project should beincluded. For external meetings, this is the direct charge for the confer-ence center, hotel, or motel. If the meetings are conducted in-house, theconference room represents a cost for the organization, and the cost shouldbe estimated and incorporated—even if it is uncommon to include facili-ties costs in other cost reporting. If a facility or building is constructed orpurchased for the project, it is included as a capital expenditure.Maintenance and MonitoringMaintenance and monitoring involve routine expenses necessary to main-tain and operate the project. These are ongoing expenses that allow thenew project solution to continue. They may involve staff members andadditional expenses, and they may be signiﬁcant for some projects.Support and OverheadThe cost of support and overhead includes the additional costs not directlycharged to the project—any project cost not considered in the abovecalculations. Typical items are the cost of administrative/clerical support,telecommunication expenses, ofﬁce expenses, salaries of client managers,and other ﬁxed costs. Usually, this is provided in the form of an estimateallocated in some convenient way.Evaluation and ReportingThe total evaluation cost completes the fully loaded costs. Activities underevaluation costs include developing the evaluation strategy, designinginstruments, collecting data, analyzing data, preparing a report, andcommunicating the results. Cost categories include time, materials, pur-chased instruments, surveys, and any consulting fees.COST CLASSIFICATIONSProject costs can be classiﬁed in two basic ways. One is with a descriptionof the expenditures, such as labor, materials, supplies, or travel. These are
The ROI Calculation 209expense account classiﬁcations, which are standard with most accountingsystems. The other way to classify costs is to use the categories inthe project steps, such as initial analysis and assessment, development,and implementation and application. An effective system monitors costsby account category according to the description of those accounts, butalso includes a method for accumulating costs in the process/functionalcategory. Many systems stop short of this second step. Although the ﬁrstgrouping adequately states the total project costs, it does not allow fora useful comparison with other projects to provide information on areaswhere costs might be excessive.THE ROI CALCULATIONThe term return on investment for projects and programs is occasionallymisused, sometimes intentionally. In this misuse, a very broad deﬁnitionfor ROI is given that includes any beneﬁt from the project. ROI becomesa vague concept in which even subjective data linked to a programare included. In this book, the return on investment is deﬁned moreprecisely and represents an actual value determined by comparing projectcosts to beneﬁts. The two most common measures are the beneﬁts/costsratio (BCR) and the ROI formula. Both are presented along with otherapproaches to calculate the return or payback.The formulas presented in this chapter use annualized values so thatthe ﬁrst-year impact of the investment can be calculated for short-termprojects. Using annualized values is becoming an accepted practice fordeveloping the ROI in many organizations. This approach is a conserva-tive way to develop the ROI, since many short-term projects have addedvalue in the second or third year. For long-term projects, longer timeframes should be used. For example, in an ROI analysis of a projectinvolving major software purchases, a ﬁve-year time frame was used.However, for short-term projects that take only a few weeks to imple-ment (such as a leadership development program), ﬁrst-year values areappropriate.In selecting the approach to measure ROI, the formula used andthe assumptions made in arriving at the decision to use this formulashould be communicated to the target audience. This helps preventmisunderstandings and confusion surrounding how the ROI value wasdeveloped. Although several approaches are described in this chapter,
210 MONITORING PROJECT COSTS AND CALCULATING ROItwo stand out as preferred methods: the beneﬁts/costs ratio and the basicROI formula. These two approaches are described next.Beneﬁts/Costs RatioOne of the original methods for evaluating projects was the beneﬁts/costsratio. This method compares the beneﬁts of the project with the costs,using a simple ratio. In formula form,BCR =Project BeneﬁtsProject CostsIn simple terms, the BCR compares the economic beneﬁts of the projectwith the costs of the project. A BCR of 1 means that the beneﬁts equal thecosts. A BCR of 2, usually written as 2:1, indicates that for each dollarspent on the project, two dollars were returned in beneﬁts.The following example illustrates the use of the BCR. A behavior mod-iﬁcation project designed for managers and supervisors was implementedat an electric and gas utility. In a follow-up evaluation, action planningand business performance monitoring were used to capture the beneﬁts.The ﬁrst-year payoff for the program was $1,077,750. The total, fullyloaded implementation costs were $215,500. Thus, the ratio wasBCR =$1,077,750$215,500= 5 : 1For every dollar invested in the project, $5 in beneﬁts were returned.ROI FormulaPerhaps the most appropriate formula for evaluating project investmentsis net program beneﬁts divided by costs. This is the traditional ﬁnancialROI and is directly related to the BCR. The ROI ratio is usually expressedas a percentage where the fractional values are multiplied by 100. Informula form,ROI(%) =Net project beneﬁtsProject costs× 100Net project beneﬁts are project beneﬁts minus costs. Subtract 1 fromthe BCR and multiply by 100 to get the ROI percentage. For example, a
The ROI Calculation 211BCR of 2.45 is the same as an ROI value of 145 percent (1.45 × 100%).This formula is essentially the same as the ROI for capital investments.For example, when a ﬁrm builds a new plant, the ROI is developedby dividing annual earnings by the investment. The annual earningsare comparable to net beneﬁts (annual beneﬁts minus the cost). Theinvestment is comparable to the fully loaded project costs.An ROI of 50 percent means that the costs were recovered and anadditional 50 percent of the costs were returned. A project ROI of 150percent indicates that the costs have been recovered and an additional1.5 times the costs are returned.An example illustrates the ROI calculation. Public- and private-sectorgroups concerned about literacy have developed a variety of projects toaddress the issue. Magnavox Electronics Systems Company was involvedin a literacy project that focused on language and math skills for entry-level electrical and mechanical assemblers. The results of the projectwere impressive. Productivity and quality alone yielded an annual valueof $321,600. The total, fully loaded costs for the project were just $38,233.Thus, the return on investment wasROI =$321,600 − $38,233$38,233× 100 = 741%For each dollar invested, Magnavox received $7.40 in return after thecosts of the consulting project were recovered.Investments in plants, equipment, subsidiaries, or other major itemsare not usually evaluated using the beneﬁts/costs method. Using the ROIformula to calculate the return on project investments essentially placesthese investments on a level playing ﬁeld with other investments whosevaluation uses the same formula and similar concepts. The ROI calcu-lation is easily understood by key management and ﬁnancial executiveswho regularly work with investments and their ROIs.Basis for Monetary BeneﬁtsProﬁts can be generated through increased sales or cost savings. Inpractice, there are more opportunities for cost savings than for proﬁts.Cost savings can be realized when improvements in productivity, quality,efﬁciency, cycle time, or actual cost reduction occur. In a review of almost500 studies, the vast majority of them were based on cost savings.Approximately 85 percent of the studies used a payoff based on cost
212 MONITORING PROJECT COSTS AND CALCULATING ROIsavings from output, quality, efﬁciency, time, or a variety of soft datameasures. The others used a payoff based on sales increases, where theearnings were derived from the proﬁt margin. Cost savings are importantfor nonproﬁts and public sector organizations, where opportunities forproﬁt are often unavailable. Most projects or programs are connecteddirectly to cost savings; ROIs can still be developed in such settings.The formula provided above should be used consistently throughoutan organization. Deviations from or misuse of the formula can createconfusion, not only among users but also among ﬁnance and accountingstaff. The chief ﬁnancial ofﬁcer (CFO) and the ﬁnance and accounting staffshould become partners in the implementation of the ROI methodology.The staff must use the same ﬁnancial terms as those used and expectedby the CFO. Without the support, involvement, and commitment of theseindividuals, the wide-scale use of ROI will be unlikely.Table 11.3 shows some ﬁnancial terms that are misused in the litera-ture. Terms such as return on intelligence (or information), abbreviated asROI, do nothing but confuse the CFO, who assumes that ROI refers to thereturn on investment described above. Sometimes return on expectations(ROE), return on anticipation (ROA), and return on client expectations(ROCE) are used, also confusing the CFO, who assumes the abbrevia-tions refer to return on equity, return on assets, and return on capitalemployed, respectively. The use of these terms in the payback calculationof a project will also confuse and perhaps lose the support of the ﬁnanceand accounting staff. Other terms such as return on people, return onresources, return on training, and return on web are often used withalmost no consistency in terms of ﬁnancial calculations. The bottom line:Don’t confuse the CFO. Consider this person an ally, and use the sameterminology, processes, and concepts when applying ﬁnancial returns forprojects.ROI TargetsSpeciﬁc expectations for ROI should be developed before an evaluationstudy is undertaken. Although no generally accepted standards exist, fourstrategies have been used to establish a minimum expected requirement,or hurdle rate, for the ROI of a project or program. The ﬁrst approachis to set the ROI using the same values used for investing in capitalexpenditures, such as equipment, facilities, and new companies. ForNorth America, Western Europe, and most of the Asian Paciﬁc area,
The ROI Calculation 213Table 11.3 Misused Financial TermsTerm Misuse CFO DeﬁnitionROI Return of information Return on investmentReturn of intelligenceROE Return on expectation Return on equityROA Return on anticipation Return on assetsROCE Return on client expectation Return on capital employedROP Return on people ?ROR Return on resources ?ROT Return on technology ?ROW Return on web ?ROM Return on marketing ?ROO Return on objectives ?ROQ Return on quality ?including Australia and New Zealand, the cost of capital is quite low, andthe internal hurdle rate for ROI is usually in the 15 to 20 percent range.Thus, using this strategy, organizations would set the expected ROI for aproject at the same value expected from other investments.A second strategy is to use an ROI minimum target value that is abovethe percentage expected for other types of investments. The rationale isthat the ROI process for projects and programs is still relatively new andoften involves subjective input, including estimations. Because of this, ahigher standard is required or suggested.A third strategy is to set the ROI value at a breakeven point. A0 percent ROI represents breakeven; this is equivalent to a BCR of 1.This approach is used when the goal is to recapture the cost of the projectonly. This is the ROI objective for many public sector organizations,where all of the value and beneﬁt from the program come through theintangible measures, which are not converted to monetary values. Thus,an organization will use a breakeven point for the ROI based on thereasoning that it is not attempting to make a proﬁt from a particularproject.A fourth, and often the recommended, strategy is to let the client orprogram sponsor set the minimum acceptable ROI value. In this scenario,the individual who initiates, approves, sponsors, or supports the projectestablishes the acceptable ROI. Almost every project has a major sponsor,
214 MONITORING PROJECT COSTS AND CALCULATING ROIand that person may be willing to specify an acceptable value. This linksthe expectations for ﬁnancial return directly to the expectations of thesponsor.OTHER ROI MEASURESIn addition to the traditional ROI formula, several other measures areoccasionally used under the general heading of return on investment.These measures are designed primarily for evaluating other ﬁnancialmeasures but sometimes work their way into project evaluations.Payback Period (Breakeven Analysis)The payback period is commonly used for evaluating capital expenditures.With this approach, the annual cash proceeds (savings) produced byan investment are compared against the original cash outlay for theinvestment to determine the multiple of cash proceeds that is equal tothe original investment. Measurement is usually in terms of years andmonths. For example, if the cost savings generated from a project areconstant each year, the payback period is determined by dividing theoriginal cash investment (including development costs, expenses, etc.)by the expected or actual annual savings. The net savings are found bysubtracting the project expenses.To illustrate this calculation, assume that the initial cost of a project is$100,000 and the project has a three-year useful life. Annual net savingsfrom the project are expected to be $40,000. Thus, the payback period isPayback period =Total investmentAnnual savings=$100,000$40,000= 2.5 yearsThe project will ‘‘pay back’’ the original investment in 2.5 years.The payback period method is simple to use but has the limitation ofignoring the time value of money. It has not enjoyed widespread use inthe evaluation of project investments.Discounted Cash FlowDiscounted cash ﬂow is a method of evaluating investment opportunitiesin which certain values are assigned to the timing of the proceeds from
Final Thoughts 215the investment. The assumption behind this approach is that a dollarearned today is more valuable than a dollar earned a year from now,based on the accrued interest possible from investing the dollar.There are several ways of using the discounted cash ﬂow concept toevaluate a project investment. The most common approach uses the netpresent value of an investment. The savings each year are comparedwith the outﬂow of cash required by the investment. The expected annualsavings are discounted based on a selected interest rate, and the outﬂowof cash is discounted by the same interest rate. If the present value ofthe savings exceeds the present value of the outlays, after the two havebeen discounted by the common interest rate, the investment is usuallyconsidered acceptable by management. The discounted cash ﬂow methodhas the advantage of ranking investments, but it requires calculationsthat can become difﬁcult.Internal Rate of ReturnThe internal rate of return (IRR) method determines the interest ratenecessary to make the present value of the cash ﬂow equal zero. Thisrepresents the maximum rate of interest that could be paid if all projectfunds were borrowed and the organization was required to break even onthe project. The IRR considers the time value of money and is unaffectedby the scale of the project. It can be used to rank alternatives and toaccept or reject decisions when a minimum rate of return is speciﬁed.A major weakness of the IRR method is that it assumes all returnsare reinvested at the same internal rate of return. This can make aninvestment alternative with a high rate of return look even better than itreally is and make a project with a low rate of return look even worse. Inpractice, the IRR is rarely used to evaluate project investments.FINAL THOUGHTSROI, the ﬁnal evaluation level, compares costs with beneﬁts. Costs areimportant and should be fully loaded in the ROI calculation. From apractical standpoint, some costs may be optional and depend on the orga-nization’s guidelines and philosophy. However, because of the scrutinyROI calculations typically receive, all costs should be included, even ifthis goes beyond the requirements of the organization’s policy. After the
216 MONITORING PROJECT COSTS AND CALCULATING ROIbeneﬁts are collected and converted to monetary values and the projectcosts are tabulated, the ROI calculation itself is easy. Plugging the valuesinto the appropriate formula is the ﬁnal step. This chapter presented thetwo basic approaches for calculating return: the ROI formula and the ben-eﬁts/costs ratio. Each has its advantages and disadvantages. Alternativesto the standard ROI determination were also brieﬂy discussed.Now that the process has been fully laid out, the next chapter detailshow to forecast the value of a project, including its ROI.
218 FORECASTING VALUE, INCLUDING ROIto purchase, to ensure that the monetary value of the process outcomesoutweigh the cost of equipment and implementation. While there maybe trade-offs in deploying a lower-proﬁle, lower-cost pilot, the pre-projectROI is still important, and may prompt some clients to stand ﬁrm untilan ROI forecast is produced.High Risks and UncertaintySponsors want to remove as much uncertainty as possible from the projectand act on the best data available. This concern sometimes pushes theproject to a forecast ROI, even before any resources are expended todesign and implement it. Some projects are high-risk opportunities orsolutions. In addition to being expensive, they may represent criticalinitiatives that can make or break an organization. Or the situation maybe one where failure would be disastrous, and where there is only onechance to get it right. In these cases, the decision maker must have thebest data possible, and the best data possible often include a forecast ROI.For example, one large restaurant chain developed an unfortunate rep-utation for racial insensitivity and discrimination. The fallout broughtmany lawsuits and caused a public relations nightmare. The companyundertook a major project to transform the organization—changing itsimage, attitudes, and actions. Because of the project’s high stakes and crit-ical nature, company executives requested a forecast before pursuing theproject. They needed to know not only whether this major program wouldbe worthwhile ﬁnancially, but also what speciﬁcally would change, andhow speciﬁcally the program would unfold. This required a comprehensiveforecast involving various levels of data, up to and including the ROI.Post-Project ComparisonAn important reason for forecasting ROI is to see how well the forecastholds up under the scrutiny of post-project analysis. Whenever a plan isin place to collect data on a project’s success, comparing actual results topre-project expectations is helpful. In an ideal world, a forecast ROI wouldhave a deﬁned relationship with the actual ROI—or at least one wouldlead to the other, after adjustments. The forecast is often an inexpensiveprocess because it involves estimates and assumptions. If the forecastbecomes a reliable predictor of the post-project analysis, then the forecast
The Timing of Forecasting 219ROI might substitute for the actual ROI. This could save money on theuse of post-project analysis.ComplianceMore than ever, organizations are requiring a forecast ROI before theyundertake major projects. For example, one organization requires anyproject with a budget exceeding $500,000 to have a forecast ROI beforeit grants project approval. Some units of government have enacted legis-lation that requires project forecasts. With increasing frequency, formalpolicy and legal structures are reasons to develop ROI forecasts.Collectively, these reasons are leading more organizations to developROI forecasts so their sponsors will have an estimate of projects’ expectedpayoff.THE TIMING OF FORECASTINGThe ROI can be developed at different times and with different levelsof data. Unfortunately, the ease, convenience, and costs involved incapturing a forecast ROI create trade-offs in accuracy and credibility. Asshown in Table 12.1, there are ﬁve distinct time intervals during a projectwhen the ROI can be developed. The relationship between the timing ofthe ROI and the factors of credibility, accuracy, cost, and difﬁculty is alsoshown in this table.• A pre-project forecast can be developed using estimates of the impactof the project. This approach lacks credibility and accuracy, but isthe least expensive and least difﬁcult to calculate. Because of theinterest in pre-project forecasting, this scenario is expanded.• Reaction data can be extended to develop an anticipated impact,including the ROI. In this case, participants anticipate the chain ofimpact as a project is implemented and drives speciﬁc business mea-sures. This is done after the project has begun. While accuracy andcredibility increase from the pre-project basis, this approach lacksthe credibility and accuracy desired in many situations. However, itis easily accomplished and is a low-cost option.• In projects where there is a substantial learning component, learn-ing data can be used to forecast the ROI. This approach is applicableonly when formal testing shows a relationship between test scores
Pre-Project ROI Forecasting 221and subsequent business performance. When this correlation isavailable (it is usually developed to validate the test), test datacan be used to forecast subsequent performance. The performancecan then be converted to monetary impact, and the ROI can bedeveloped. This has less potential as a forecasting tool.• When frequency of skills or knowledge use is critical, the applicationand implementation of those skills or knowledge can be convertedto a value using a concept called utility analysis. While this isparticularly helpful in situations where competencies are beingdeveloped and values are placed on improving competencies, it haslimited applications in most projects.• Finally, the ROI can be developed from business impact data con-verted directly to monetary values and compared to the cost of theprogram. This is not a forecast; but is a post-project evaluation—thebasis for other ROI calculations in this book. It is the preferredapproach, but because of the pressures outlined above, examiningROI calculations at other times and with other levels is sometimesnecessary.This chapter discusses in detail pre-project ROI forecasting and ROIforecasting based on reactions. In less detail, ROI forecasts developedfrom learning and application data are also discussed.PRE-PROJECT ROI FORECASTINGPerhaps one of the most useful ways to convince a sponsor that a projectis beneﬁcial is to forecast the ROI for the project. The process is similarto the post-project analysis, except that the extent of the impact must beestimated along with the project costs.Basic ModelFigure 12.1 shows the basic model for capturing the data necessaryfor a pre-project forecast, a modiﬁed version of the post-program ROIprocess model presented in Chapter 3. In the pre-project forecast, theproject outcomes are estimated, rather than being collected after projectimplementation. Data collection is kept simple, and relies on interviews,focus groups, or surveys of experts. Tapping into benchmarking studiesor locating previous studies may also be helpful.
222 FORECASTING VALUE, INCLUDING ROIEstimateprojectcostsCalculatereturn oninvestmentAnticipateintangiblebenefitsConvertdata tomonetaryvaluesEstimatechange inimpact dataEstimateapplicationEstimateamount oflearningAnticipatereaction tothe projectLevel 1 Level 2 Level 3 Level 4Figure 12.1 Pre-project forecasting model.Beginning at the reaction level, anticipated or estimated reactions arecaptured. Next, the anticipated learning that must occur is developed,followed by the estimated application and implementation data. Here,the estimates focus on what must be accomplished for the project tobe successful. Finally, the impact data are estimated by experts. Theseexperts may include subject matter experts, the supplier, or potentialparticipants in the project. In this model, the levels build on each other.Having data estimated at Levels 1, 2, and 3 enhances the quality of theestimated data at Level 4 (impact), which is needed for the analysis.The model shows that there is no need to isolate the effects of aproject as in the post-project model. The individual providing the datais asked the following question: ‘‘How much will the business impactmeasure change as a result of the project?’’ This question ties the changein the measure directly to the project; thus, isolation is not needed.This approach makes this process easier than the post-evaluation model,where isolating project impact is always required.Converting data to money is straightforward using a limited numberof techniques. Locating a standard value or ﬁnding an expert to makethe estimate is the logical choice. Analyzing records and databases areless likely alternatives at the forecasting stage. Securing estimates fromstakeholders is the technique of last resort.Estimating the project’s costs should be an easy step because costs caneasily be anticipated on the basis of previous or similar projects, factoringin reasonable assumptions about the project. To achieve a fully loadedcost proﬁle, include all cost categories.The anticipated intangibles are merely speculation in forecasting butcan be reliable indicators of which measures may be inﬂuenced in addition
Pre-Project ROI Forecasting 223to those included in the ROI calculation. At this point, it is assumed thatthese measures will not be converted to money.The formula used to calculate the ROI is the same as that used inthe post-analysis. The net monetary value from the data conversionis included as the numerator, and the estimated cost of the project isinserted as the denominator. The projected cost-beneﬁt analysis can bedeveloped along with the ROI. The speciﬁc steps to develop the forecastare detailed next.Basic Steps to Forecast ROIEighteen detailed steps are necessary to develop a credible pre-projectROI forecast using expert input:1. Understand the situation. Individuals providing input to the fore-cast and conducting the forecast must have a good understanding ofthe present situation. This is typically a requirement for selectingthe experts.2. Predict the present. The project is sometimes initiated because aparticular business impact measure is not doing well. However,such measures often lag the present situation; they may be basedon data that are several months old. Also, these measures arebased on dynamic inﬂuences that may change dramatically andquickly. It may be beneﬁcial to estimate where the measure is now,based on assumptions and current trends. Although this appearsto be a lot of work, it does not constitute a new responsibility formost of the experts, who are often concerned about the presentsituation. Market share data, for example, are often several monthsold. Trending market share data and examining other inﬂuencesdriving market share can help organizations understand the currentsituation.3. Observe warnings. Closely tied to predicting the present is makingsure that warning signs are observed. Red ﬂags signal that some-thing is going against the measure in question, causing it to go inan undesired direction or otherwise not move as it should. Theseoften raise concerns that lead to projects. These are early warningsthat things may get worse; they must be factored into the situationas forecasts are made.4. Describe the new process, project, or solution. The project mustbe completely and clearly described to the experts so they fully
224 FORECASTING VALUE, INCLUDING ROIunderstand the mechanics of what is to be implemented. Thedescription should include the project scope, the individualsinvolved, time factors, and whatever else is necessary to expressthe magnitude of the project and the proﬁle of the solution.5. Develop speciﬁc objectives. These objectives should mirror the lev-els of evaluation and should include reaction objectives, learningobjectives, application objectives, and impact objectives. Althoughthese may be difﬁcult to develop, they are developed as part of theup-front analysis described in Chapter 4. Objectives provide cleardirection toward the project’s end. The cascading levels representthe anticipated chain of impact that will occur as the project isimplemented.6. Estimate what participants will think about the project. In thisstep, the experts are trying to understand participants’ reaction:Will they support the project? How will they support it? Whatmay cause participants to become unsupportive? The response isimportant because a negative reaction can cause a project to fail.7. Estimate what the participants will learn. To some extent, everyproject will involve learning, and the experts will estimate whatlearning will occur. Using the learning objectives, the experts willdeﬁne what the participants will learn as they enter the project,identifying speciﬁc knowledge, skills, and information the partici-pants must acquire or enhance during the project.8. Estimate what participants should accomplish in the project. Build-ing on the application objectives, the experts will identify what willbe accomplished as the project is implemented successfully. Thisstep details speciﬁc actions, tasks, and processes that will be takenby the individuals. Steps 6, 7, and 8—based on reaction, learning,and application—provide important information that serves as thebasis for the next step, estimating improvement in business impactdata.9. Estimate the improvement in business impact data. This is a criticalstep in that the data generated are needed for the ﬁnancial forecast.The experts will provide the estimate—in either absolute numbersor percentages—of the monetary change in the business impactmeasure ( P). While accuracy is important, it is also important toremember that a forecast is no more than an estimate based on thebest data available at a given point. This is why the next step isincluded.
Pre-Project ROI Forecasting 22510. Apply the conﬁdence estimate. Because the estimate attained inthe previous step is not very accurate, an error adjustment isneeded. This is developed by deriving a conﬁdence estimate on thevalue identiﬁed in Step 9. The experts are asked to indicate theconﬁdence they have in the previous data. The conﬁdence level isexpressed as a percentage, with 0 indicating ‘‘no conﬁdence’’ and100 percent indicating ‘‘certainty.’’ This becomes a discount factorin the analysis.11. Convert the business impact data to monetary values. Using oneor more methods described in Chapter 9, the data are convertedto money. If the impact measure is a desired improvement such asproductivity, the value represents the gain obtained by having onemore unit of the measure. If it is a measure that the organizationis trying to reduce—like downtime, mistakes, or complaints—thevalue is the cost that the organization incurs as a result of oneincident. For example, the cost of unwanted employee turnover maybe 1.5 times annual pay. This value is noted with the letter V.12. Develop the estimated annual impact of each measure. The esti-mated annual impact is the ﬁrst-year improvement directly relatedto the project. In formula form, this is expressed as I = P × V× 12 (where I = annual change in monetary value, P = annualchange in performance of the measure, and V = the value of thatmeasure). If the measure is weekly or monthly, it must be convertedto an annual amount. For example, if three lost-time accidents willbe prevented each month, the time saved represents a total of 36.13. Factor additional years into the analysis for projects that will havea signiﬁcant useful life beyond the ﬁrst year. For these projects, thefactor should reﬂect the diminished beneﬁt of subsequent years. Theclient or sponsor of the project should provide some indication of theamount of the reduction and the values developed for the second,third, and successive years. It is important to be conservative byusing the smallest numbers possible.14. Estimate the fully loaded project costs. In this step, use all the costcategories described in Chapter 11, and denote the value as C whenincluding it in the ROI equation. Include all direct and indirect costsin the calculation.15. Calculate the forecast ROI. Using the total projected beneﬁts andthe estimated costs in the standard ROI formula, calculate the
226 FORECASTING VALUE, INCLUDING ROIforecast ROI as follows:ROI (%) =I − CC× 10016. Use sensitivity analysis to develop several potential ROI values withdifferent levels of improvement ( P). When more than one measureis changing, the analysis may take the form of a spreadsheet showingvarious output scenarios and the subsequent ROI forecasts. Thebreakeven point will be identiﬁed.17. Identify potential intangible beneﬁts. Anticipate intangible beneﬁtsusing input from those most knowledgeable about the situationon the basis of assumptions from their experience with similarprojects. Intangible beneﬁts are those beneﬁts not converted tomonetary values, but possessing value nonetheless.18. Communicate the ROI projection and anticipated intangibles withcaution. The target audience must clearly understand that theforecast is based on several assumptions (clearly deﬁned), and thatalthough the values are the best possible estimates, they mayinclude a degree of error.Following these eighteen steps will enable an individual to forecastthe ROI.Sources of Expert InputSeveral sources of expert input are available for estimating improvementin impact data when the project is implemented. Ideally, experiencewith similar projects in the organization will help form the basis of theestimates the experts make. The experts may include• Clients and/or sponsors• Members of project team• Prospective participants• Subject matter experts• External experts• Advocates (who can champion the project)• Finance and accounting staff• Analysts (if one is involved with the project)
Pre-Project ROI Forecasting 227• Executives and/or managers• CustomersCollectively, these sources provide an appropriate array of possibilitiesfor helping estimate the value of an improvement. Because errors maydevelop, ask for a conﬁdence measure when using estimates from anysource.Securing InputWith the experts clearly identiﬁed, three major steps must be addressedbefore developing the ROI. First, data must be collected from the individ-uals listed as experts. If the number of individuals is small (for example,one person from each of the expert groups involved), a short interviewmay sufﬁce. During interviews, it is critical to avoid bias and to ask clear,succinct questions that are not leading. Questions should be framed ina balanced way to capture what may occur as well as what may not.If groups are involved, using focus groups may be suitable. For largenumbers, surveys or questionnaires may be appropriate.When the groups are diverse and scattered, the Delphi techniquemay be appropriate. This technique, originally developed by the RandCorporation in the 1950s, has been used in forecasting and decisionmaking in a variety of disciplines. The Delphi technique was originallydevised to help experts achieve better forecasts than they might obtainthrough traditional group meetings by allowing access to the groupwithout in-person contact. Necessary features of a Delphi procedure areanonymity, continuous iteration, controlled feedback to participants, anda physical summary of responses. Anonymity is achieved by means ofa questionnaire that allows group members to express their opinionsand judgments privately. Between all iterations of the questionnaire thefacilitator informs the participants of the opinions of their anonymouscolleagues. Typically this feedback is presented as a simple statisticalsummary using a mean or median value. The facilitator takes the groupjudgment as the statistical average in the ﬁnal round.1In some cases, benchmarking data may be available and can be con-sidered as a source of input for this process. The success of previousstudies may provide input essential to the project as well. It may includean extensive search of databases using a variety of search engines. Theimportant point is to understand, as much as possible, what may occuras a result of the project.
228 FORECASTING VALUE, INCLUDING ROIConversion to MoneyThe measures forecast by the experts must be converted to monetaryvalues for one, two, three, or more years depending on the nature andscope of the project. Standard values are available for many of thesemeasures. Considering the importance of these measures, someone hasprobably placed monetary values on them. If not, experts are oftenavailable to convert the data to monetary values. Otherwise, existingrecords or databases may be appropriate sources. Another option is toask stakeholders—perhaps some of the experts listed above—to providethese values for the forecast. This step is the only means of showingthe money made from the project. Chapter 9 covered these techniques inmore detail.Estimate Project CostsProject cost estimates are based on the most reliable information avail-able, and include the typical categories outlined in Table 11.2. Theestimates can be based on previous projects. Although the costs areunknown, this task is often relatively easy to accomplish because of itssimilarity to budgeting, a process with usually routine procedures andpolicies in place. Dividing costs into categories representing the functionalprocesses of the project provides additional insight into project costs.Areas often not given enough attention include analysis, assessment,evaluation, and reporting. If these elements are not properly addressed,much of the value of the project may be missed. With these costs and mon-etary beneﬁts, the forecast can be made using the calculations presentedin Chapter 11.Case Study: Forecasting ROI for a Technology SolutionGlobal Financial Services (GFS) was in the process of implementingcontact management software to enable its sales relationship to trackroutine correspondence and communication with customers. A needsassessment and initial analysis determined the project was needed.The project would involve further detailing, selecting an appropriatesoftware package, and implementing the software with appropriate jobaids, support tools, and training. However, before pursuing the projectand purchasing the software, a forecast ROI was needed. Following the
230 FORECASTING VALUE, INCLUDING ROIsteps previously outlined, it was determined that four business impactmeasures would be inﬂuenced by implementation of this project:1. Increase in sales to existing customers2. Reduction in customer complaints caused by missed deadlines, lateresponses, and failure to complete transactions3. Reduction in response time for customer inquiries and requests4. Increase in the customer satisfaction composite survey indexSeveral individuals provided input in examining the potential problem.With comprehensive customer contact management software in place,relationship managers should beneﬁt from quick and effective customercommunication and have easy access to customer databases. The softwareshould also provide the functionality to develop calendars and to-do lists.Relationship managers should further beneﬁt from features such as built-in contact management, calendar sharing, and the fact that the softwareis Internet-ready. To determine the extent to which the four measureswould change, input was collected from six sources:1. Internal software developers with expertise in various softwareapplications provided input on expected changes in each of themeasures.2. Marketing analysts supplied information on sales cycles, customerneeds, and customer care issues.3. Relationship managers provided input on expected changes in thevariables if the software was used regularly.4. The analyst who conﬁrmed the initial need for the software providedsupplemental data.5. The sponsor provided input on what could be expected from theproject.6. The proposed vendor provided input based on previous experience.When input is based on estimates, the actual results will usuallydiffer signiﬁcantly. However, GFS was interested in a forecast based onanalysis that, although very limited, would be strengthened with the besteasily available expert opinion. Input was adjusted on the basis of theestimates and other information to assess its credibility. After discussing
Forecasting with a Pilot Program 231the availability of data and examining the techniques to convert it tomonetary values, the following conclusions were reached:• The increase in sales could easily be converted to a monetary valueas the average margin for sales increase is applied directly.• The cost of a customer complaint could be based on an internal valuecurrently in use, providing a generally accepted cost.• Customer response time was not tracked accurately, and the valueof this measure was not readily available, making it an intangiblebeneﬁt.• No generally accepted value for increasing customer satisfactionwas available, so customer satisfaction impact data would be listedas an intangible beneﬁt.The forecast ROI calculation was developed from combined input basedon the variety of estimates. The increase in sales was easily converted tomonetary values using the margin rates, and the reduction in customercomplaints was easily converted using the discounted value of a customercomplaint. The costs for the project could easily be estimated based oninput from those who brieﬂy examined the situation. The total costsincluded development costs, materials, software, equipment, facilitators,facilities, and lost time for learning activities, coordination, and evalua-tion. This fully loaded projected cost, compared to the beneﬁts, yielded arange of expected ROI values. Table 12.2 shows possible scenarios basedon payoffs of the two measures as assessed by six experts. The ROI valuesrange from a low of 12 percent to a high of 180 percent. The breakevenpoint could be developed with different scenarios. With these values inhand, the decision to move forward was easy: Even the worst-case sce-narios were positive and the best case was expected to yield more than 10times the ROI of the worst. As this example illustrates, the process mustbe simple, and must use the most credible resources available to quicklyarrive at estimates.FORECASTING WITH A PILOT PROGRAMBecause of inaccuracies inherent in a pre-project forecast, a betterapproach is to develop a small-scale pilot project with the ROI basedon post-program data. This involves the following steps:
232 FORECASTING VALUE, INCLUDING ROI1. As in the previous process, develop Level 1, 2, 3, and 4 objectives.2. Implement the project on a small-scale sample as a pilot project,excluding all the bells and whistles. (This keeps the project costslow without sacriﬁcing project integrity.)3. Fully implement the project with one or more of the groups who canbeneﬁt from the initiative.4. Develop the ROI using the ROI process model for post-projectanalysis as outlined in previous chapters.5. Based on the results of the pilot project, decide whether to implementthe project throughout the organization. Data can be developedusing all six of the measures outlined in this book: reaction, learning,application, impact, ROI, and intangibles.Evaluating a pilot project and withholding full implementation untilits results can be developed provides less risk than developing an ROIforecast. Wal-Mart uses this method to evaluate pilot programs beforeimplementing them throughout its chain of 4,000 U.S. stores. Using pilotgroups of eighteen to thirty stores called ﬂights, the decision to implementa project throughout the system is based on six types of post-programdata (reaction, learning, application, impact, ROI, and intangibles).FORECASTING ROI WITH REACTION DATAWhen a reaction evaluation includes the planned applications of a project,the data can ultimately be used in an ROI forecast. ROI information can bedeveloped with questions concerning how participants plan to implementthe project and what results they expect to achieve. For example, considera project proposed by a major pharmaceutical company. The ﬁrm wasconsidering installing high-speed DSL lines in the homes of each of itspharmaceutical sales representatives on the premise that this wouldsave the reps time that they could otherwise spend with their customers.However, reaction to the proposed project was not positive. The sales repssaid they do most of their online work at night when speed is not suchan issue, and even if they did save time, they would be unlikely to addanother call to their schedule, or even be able to spend more time withcustomers. Although the project’s goals had merit, from the standpoint offorecast monetary value, the project would not add value or improve theoriginal measure.
Forecasting ROI with Reaction Data 233Data CollectionAt the beginning of a project, participants are asked to state speciﬁcallyhow they plan to use the project and what results they expect to achieve.They are asked to convert their planned accomplishments into monetaryvalues and show the basis for developing the values. Participants canadjust their responses with a conﬁdence factor to make the data morecredible. Next, estimates are adjusted for conﬁdence level. When tabulat-ing data, participants multiply the conﬁdence levels by annual monetaryvalues. This produces a conservative estimate for use in data analysis.For example, if a participant estimated the monetary impact of the projectat $10,000 but was only 50 percent conﬁdent in his or her estimate, a$5,000 value would be used in the ROI forecast calculations.To develop a summary of the expected beneﬁts, discard any data thatare incomplete, unusable, extreme, or unrealistic. Then total individualdata items. Finally, as an optional exercise, adjust the total value again bya factor that reﬂects the unknowns in the process and the possibility thatparticipants will not achieve the results they anticipate. This adjustmentfactor can be determined by the project team. In one organization, thebeneﬁts are divided by 2 to develop a number to use in the calculation.Finally, calculate the forecast ROI using the net beneﬁts from the projectdivided by the project costs.Case Study: Forecasting ROI from Reaction DataThis process can best be described using an actual case. Global Engi-neering and Construction Company (GEC) designs and builds largecommercial projects like plants, paper mills, and municipal water sys-tems. Safety is always a critical matter at GEC and usually commandsmuch management attention. To improve safety performance, a safetyimprovement project was initiated for project engineers and constructionsuperintendents. The project solution involved policy changes, audits,and training. The project focused on safety leadership, safety planning,safety inspections, safety meetings, accident investigation, safety policiesand procedures, safety standards, and worker’s compensation. Safetyengineers and superintendents (participants) were expected to improvethe safety performance of their individual construction projects. A dozensafety performance measures used in the company were discussed andanalyzed at the beginning of the project. At that time, participants
234 FORECASTING VALUE, INCLUDING ROIcompleted a comprehensive feedback questionnaire that probed speciﬁcaction items planned as a result of the safety project and provided esti-mated monetary values of the planned actions. In addition, participantsexplained the basis for estimates and placed a conﬁdence level on theirestimates. Table 12.3 presents data provided by the participants. Onlynineteen of the twenty-ﬁve participants supplied data. (Experience hasshown that approximately 50 to 70 percent of participants will provideusable data on this series of questions.) The estimated cost of the project,including participants’ salaries for the time devoted to the project, was$358,900.The monetary values of the planned improvements were extremelyhigh, reﬂecting the participants’ optimism and enthusiasm at the begin-ning of an impressive project from which speciﬁc actions were planned.As a ﬁrst step in the analysis, extreme data items were omitted (oneof the guiding principles of the methodology). Data such as ‘‘millions,’’‘‘unlimited,’’ and ‘‘$4 million’’ were discarded, and each remaining valuewas multiplied by the conﬁdence value and totaled. This adjustment isone way of reducing highly subjective estimates. The resulting tabula-tions yielded a total improvement of $990,125 (rounded to $990,000). Theprojected ROI, which was based on the feedback questionnaire at thebeginning of the project, isROI =$990,000 − $358,900$358,900× 100 = 176%Although these projected values are subjective, the results were gen-erated by project participants who should be aware of what they couldaccomplish. A follow-up study would determine the true results deliveredby the group.Use of the DataCaution is required when using a forecast ROI: The calculations arehighly subjective and may not reﬂect the extent to which participantswill achieve results. A variety of inﬂuences in the work environmentand project setting can enhance or inhibit the attainment of performancegoals. Having high expectations at the beginning of a project is noguarantee that those expectations will be met. Project disappointmentsare documented regularly.
Forecasting ROI with Reaction Data 235Table 12.3 Level 1 Data for ROI Forecast CalculationsParticipant Estimated ConﬁdenceNo. Value Basis Level Adjusted1 $ 80,000 Reduction inlost-time accidents90% $ 72,0002 91,200 OSHA reportableinjuries80% 72,9603 55,000 Accident reduction 90% 49,5004 10,000 First-aidvisits/visits todoctor70% 7,0005 150,000 Reduction inlost-time injuries95% 142,5006 Millions Total accident cost 100% —7 74,800 Worker’scompensation80% 59,8408 7,500 OSHA citations 75% 5,6259 50,000 Reduction inaccidents75% 37,50010 36,000 Worker’scompensation80% 28,80011 150,000 Reduction in totalaccident costs90% 135,00012 22,000 OSHAﬁnes/citations70% 15,40013 140,000 Accident reductions 80% 112,00014 4 million Total cost of safety 95% —15 65,000 Total worker’scompensation50% 32,50016 Unlimited Accidents 100% —17 20,000 Visits to doctor 95% 19,00018 45,000 Injuries 90% 40,50019 200,000 Lost-time injuries 80% 160,000Total $ 990,125
236 FORECASTING VALUE, INCLUDING ROIAlthough the process is subjective and possibly unreliable, it does havesome usefulness.1. If the evaluation must stop at this point, this analysis providesmore insight into the value of the project than data from typicalreaction input, which report attitudes and feelings about a project.Sponsors and managers usually ﬁnd this information more usefulthan a report stating that ‘‘40 percent of project team participantsrated the project above average.’’2. These data can form a basis for comparing different projects of thesame type, e.g., safety projects. If one project forecast results in anROI of 300 percent and a similar project forecast results in a 30percent ROI, it would appear that one project may be more effective.The participants in the ﬁrst project have more conﬁdence in theplanned application of the project.3. Collecting these types of data focuses increased attention on projectoutcomes. Participants will understand that speciﬁc action isexpected, which produces results for the project. The data collectionhelps participants plan the implementation of what they havelearned. This issue becomes clear to participants as they anticipateresults and convert them to monetary values. Even if the forecastis ignored, the exercise is productive because of the importantmessage it sends to participants.4. The data can be used to secure support for a follow-up evaluation.A skeptical manager may challenge the data and this challengecan be converted into support for a follow-up to see whether theforecast holds true. The only way to know whether these results willmaterialize is to conduct a post-project evaluation.5. If a follow-up evaluation of the project is planned, the post-projectresults can be compared to the ROI forecast. Comparisons of forecastand follow-up data are helpful. If there is a deﬁned relationshipbetween the two, the less expensive forecast can be substituted forthe more expensive follow-up. Also, when a follow-up evaluationis planned, participants are usually more conservative with theirprojected estimates.The use of ROI forecasting with reaction data is increasing, andsome organizations have based many of their ROI forecast calculationson this type of data. For example, Wachovia Bank routinely develops
Forecasting ROI with Application Data 237ROI forecasts with reaction data. Although they may be subjective, thecalculations do add value, particularly if they are part of a comprehensiveevaluation system.FORECASTING ROI WITH LEARNING DATATesting for changes in skills and knowledge in a project or program is acommon method for measuring learning. In many situations, participantsare required to demonstrate their knowledge or acquired skills during aprogram, and their performance is expressed as a numeric value. Whenthis type of test is developed, it must be reliable and valid. Because a testshould reﬂect the content of the program, successful mastery of programcontent should be related to improved job performance. A relationshipbetween test scores and subsequent on-the-job performance should beevident. This relationship, expressed as a correlation coefﬁcient, is ameasure of validity for the test.This situation provides an opportunity for an ROI calculation withlearning data using valid test results. When a statistically signiﬁcantrelationship exists between test scores and on-the-job performance (out-put) and the performance can be converted to monetary values, it ispossible to use test scores to estimate the ROI during the project.This approach is best applied when signiﬁcant learning takes placeor when the program focuses almost entirely on developing learningsolutions. The absence of validated tests can create problems becausethe instruments cannot be used to forecast actual performance unlesstheir validity is ensured. Other resources provide more detail on how toconduct a forecast from learning data.2FORECASTING ROI WITH APPLICATION DATAAlthough not as credible as desired, a forecast can be made on the basis ofthe improved competencies or skills of the project implementation team.This process uses the concept of utility analysis, which is best describedin the experience of a large European bank that was seeking to developa leadership program for its executives. Bank managers identiﬁed thespeciﬁc competencies they wanted to develop. Before making the ¤8million investment in the program, the senior executive team wanted toknow the value it would add. The project team used utility analysis toconduct the forecast.
238 FORECASTING VALUE, INCLUDING ROIFirst, the team assessed the percentage of executives’ jobs covered inthe leadership competencies. To keep it simple, assume that this involved40 percent of their job content. This amount was derived from the sampleof the management team. Next, the average salary was determined—say,¤100,000, to keep it simple. Thus, the project could inﬂuence 40 percent of¤100,000, or ¤40,000. The managers assessed the team’s current level ofperformance of the competencies using a convenient scale. After reviewingthe competencies and the program’s objectives, the managers indicatedthat a 10 percent improvement could be achieved on these competenciesby implementing the leadership development program. Thus, the programhad a potential of improving the ¤40,000 portion of their salary by 10percent, or ¤4,000. (In essence, it would add ¤4,000.) Table 12.4 providesa summary of this process. This value is compared to the participant costto determine the forecast on an individual basis. If the cost of the programis ¤3,000, the ROI is 33 percent.Although this example is simple, it shows the concept of forecastingbased on improving competencies. It ignores what the managers orexecutives will accomplish with the competencies, so it is not as credibleas a Level 4 (impact) ROI. Nevertheless, it has value and is described inmore detail in other sources.3FORECASTING GUIDELINESWith the four different forecasting time frames outlined in this chapter,it may help to follow a few guidelines known to drive the forecastingTable 12.4 Forecasting Using Improved CompetenciesPercentage of managers’ jobs covered bycompetencies40%Average manager’s salary ¤100,000Monetary value of covered competencies(40% × ¤100,000)¤40,000Percentage of anticipated improvement incompetencies10%Added beneﬁt of improved competencies inmonetary terms (¤40,000 × 10%)¤4,000 per managerCost of program per participant ¤3,000 per managerROI 33%
Forecasting Guidelines 239possibilities within an organization. These guidelines are based on expe-rience in forecasting in a variety of projects and programs.41. If you must forecast, forecast frequently. Forecasting is an art anda science. Users can build comfort, experience, and history with theprocess by using it frequently.2. Make forecasting an essential part of the evaluation mix. Thischapter began with a list of essential reasons for forecasting. Theuse of forecasting is increasingly being demanded by many organi-zations. It can be an effective and useful tool when used properlyand in conjunction with other types of evaluation data. Some orga-nizations have targets for the use of forecasting, e.g., if a projectexceeds a certain cost, it will always require a pre-program forecast.Others will target a certain number of programs for a forecast basedon reaction data and use those data in the manner described. Itis important to plan for the forecast and let it be a part of theevaluation mix, using it regularly.3. Forecast different types of data. Although most of this chapterfocuses on how to develop a forecast ROI using the standard ROIformula, forecasting the value of the other types of data is impor-tant as well. A useable, helpful forecast will include predictionsaround reaction and perceived value, the extent of learning, andthe extent of application and implementation. These types of dataare very important in anticipating movements and shifts, basedon the project that is planned. It assists in developing the overallforecast and helps the project team understand the project’s totalanticipated impact.4. Secure input from those who know the process best. As forecastsare developed, it is essential to secure input from individuals whounderstand the dynamics of the workplace and the measures beinginﬂuenced by the project—go to the experts. This will increase notonly the accuracy of the forecast, but also the credibility of theresults. In other situations, it may be the analysts who are awareof the major inﬂuences in the workplace and the dynamics of thosechanges.5. Long-term forecasts will usually be inaccurate. Forecasting worksbetter when it covers a short time frame. Most short-term scenariosafford a better grasp of the inﬂuences that might drive the measures.In the long term, a variety of new inﬂuences, unforeseen now, could
240 FORECASTING VALUE, INCLUDING ROIenter the process and drastically change the impact measures. If along-term forecast is needed, it should be updated regularly.6. Expect forecasts to be biased. Forecasts will consist of data comingfrom those who have an interest in the issue. This is unavoidable.Some will want the forecast to be optimistic; others will have apessimistic view. Almost all input is biased in one way or another.Every attempt should be made to minimize the bias, adjust for thebias, or adjust for the uncertainty in the process. Still, the audienceshould recognize the forecast as a biased prediction.7. Serious forecasting is hard work. The value of forecasting oftendepends on the amount of effort put into the process. High-stakesprojects or programs need a serious approach, collecting all possibledata, examining different scenarios, and making the best predictionavailable. It is in these situations that mathematical tools can bemost valuable.8. Review the success of forecasting routinely. As forecasts are made, itis imperative to revisit the forecast with post-project data to checkits accuracy. This can aid in the continuous improvement of theprocesses. Sources could prove to be more or less credible, speciﬁcinputs may be more or less biased, certain analyses may be moreappropriate than others. It is important to constantly improve themethods and approaches for forecasting within the organization.9. The assumptions are the most serious error in forecasting. Of all thevariables that can enter the process, assumptions offer the greatestopportunity for error. It is important for the assumptions to beclearly understood and communicated. When multiple inputs aregiven, each forecaster should use the same set of assumptions, ifpossible.10. Utility is the most important characteristic of forecasting. The mostimportant use of forecasting is providing information and input forthe decision maker. Forecasting is a tool for those attempting tomake decisions about project implementation. It is not a processintended to maximize the output or minimize any particular vari-able. It is not a process undertaken to dramatically change theway a project is implemented. It is a process to provide data fordecisions.
Final Thoughts 241FINAL THOUGHTSThis chapter illustrates that ROI calculations can be developed at differ-ent times and at different evaluation levels, although most project leadersfocus only on impact data for ROI calculations. Although post-project dataare desired, impact data are not yet available in many situations. ROIforecasts developed before a project begins can be useful to the sponsorand are sometimes necessary before projects can be approved. Forecastsmade during project implementation can be useful to management andparticipants, and can focus participants’ attention on the economic impactof the project. However, using ROI estimates during the project may givea false sense of accuracy. As expected, pre-project ROI forecasts havethe least credibility and accuracy, yet have the advantage of being inex-pensive and relatively easy to develop. ROI calculations using impactdata are more credible and accurate than forecasts but are expensive anddifﬁcult to develop. The reality is that forecasting is an important part ofthe measurement mix. It should be pursued routinely and used regularlyin decision making. Whether a forecast ROI or a post-analysis ROI, theresults must be reported to stakeholders. The next chapter details howand when results are communicated and who they are communicated to.
244 REPORTING RESULTSCommunication Is Necessary to Make ImprovementsInformation is collected at different points during the process, and pro-viding feedback to involved groups enables them to take action and makeadjustments if needed. Thus, the quality and timeliness of communi-cation are critical to making improvements. Even after the project iscompleted, communication is necessary to make sure the target audi-ence fully understands the results achieved, and how the results may beenhanced in future projects or in the current project, if it is still opera-tional. Communication is the key to making important adjustments at allphases of the project.Communication Is Necessary to Explain the ContributionThe overall contribution of the project, as determined from the six majortypes of measures, is unclear at best. The different target audiences willeach need a thorough explanation of the results. The communicationstrategy—including techniques, media, and the overall process—willdetermine the extent to which each group understands the contribution.Communicating results, particularly in terms of business impact and ROI,can quickly overwhelm even the most sophisticated target audiences.Communication must be planned and implemented with the goal ofmaking sure the respective audiences understand the full contribution.Communication Is a Politically Sensitive IssueCommunication is one of those issues that can cause major problems.Because the results of a project may be closely linked to political issueswithin an organization, communicating the results can upset some indi-viduals while pleasing others. If certain individuals do not receive theinformation, or if it is delivered inconsistently between groups, problemscan quickly surface. Not only must the information be understood, butissues relating to fairness, quality, and political correctness make it cru-cial that the communication be constructed and delivered effectively toall key individuals.Different Audiences Need Different InformationWith so many potential target audiences requiring communication onthe success of a project, the communication must be individually tailored
Principles of Communicating Results 245to their needs. A varied audience has varied needs. Planning and effortare necessary to ensure that each audience receives all the informationit needs, in the proper format, at the proper time. A single report forpresentation to all audiences is inappropriate. The scope, the format,and even the content of the information will vary signiﬁcantly from onegroup to another. Thus, the target audience is the key to determining theappropriate method of communication.Communication is a critical need for the reasons just cited, althoughit is often overlooked or underfunded in projects. This chapter presentsa variety of techniques for accomplishing communication of all types forvarious target audiences.PRINCIPLES OF COMMUNICATING RESULTSThe skills one must possess to communicate results effectively are almostas sensitive and sophisticated as those necessary for obtaining results.The style of the communication is as important as the substance. Regard-less of the message, audience, or medium, a few general principles apply.Communication Must Be TimelyIn general, project results should be communicated as soon as theybecome known. From a practical standpoint, however, it is sometimesbest to delay the communication until a convenient time, such as thepublication of the next client newsletter or the next general managementmeeting. Several questions are relevant to the timing decision. Is theaudience ready for the results in view of other issues that may havedeveloped? Is the audience expecting results? When will the deliveryhave the maximum impact on the audience? Do circumstances dictate achange in the timing of the communication?Communication Should Be Targeted to Speciﬁc AudiencesAs stated earlier, communication is usually more effective if it is designedfor the speciﬁc group being addressed. The message should be tailored tothe interests, needs, and expectations of the target audience. The resultsof the project should reﬂect outcomes at all levels, including the sixlevels presented in this book. Some of the data are developed earlier inthe project and communicated during the implementation of the project.Other data are collected after project implementation and communicated
246 REPORTING RESULTSin a follow-up study. The results, in their broadest sense, may incorporateearly feedback in qualitative form all the way to ROI values expressed invarying quantitative terms.Media Should Be Carefully SelectedCertain media may be more appropriate for a particular group thanothers. Face-to-face meetings may be preferable to special bulletins.A memo distributed exclusively to top executives may be a more effectiveoutlet than the company newsletter. The proper format of communicationcan determine the effectiveness of the process.Communication Should Be Unbiased and Modest in ToneFor communication to be effective, fact must be separated from ﬁction andaccurate statements distinguished from opinions. Some audiences mayapproach the communication with skepticism, anticipating the presenceof biased opinions. Boastful statements can turn off recipients, and mostof the content will be lost. Observable phenomena and credible statementscarry much more weight than extreme or sensational claims. Althoughsuch claims may get an audience’s attention, they often detract from theimportance of the results.Communication Must Be ConsistentThe timing and content of the communication should be consistent withpast practices. A special presentation at an unusual time during thecourse of the project may provoke suspicion. Also, if a particular group,such as top management, regularly receives communication on outcomes,it should continue receiving communication even if the results are notpositive. Omitting unfavorable results leaves the impression that onlypositive results will be reported.Testimonials Are More Effective When They Comefrom Respected IndividualsOpinions are strongly inﬂuenced by other people, particularly those whoare respected and trusted. Testimonials about project results, whensolicited from individuals who are respected within the organization, can
The Process for Communicating Results 247inﬂuence the effectiveness of the message. This respect may be relatedto leadership ability, position, special skills, or knowledge. A testimonialfrom an individual who commands little respect and is regarded as asubstandard performer can have a negative impact on the message.The Audience’s Opinion of the Project Will Inﬂuencethe Communication StrategyOpinions are difﬁcult to change, and a negative opinion toward a projector project team may not change with the mere presentation of facts.However, the presentation of facts alone may strengthen the opinionsheld by those who already support the project. Presentation of the resultsreinforces their position and provides them with a defense in discussionswith others. A project team with a high level of credibility and respectmay have a relatively easy time communicating results. Low credibilitycan create problems when one is trying to be persuasive.These general principles are vital to the overall success of the com-munication effort. They should serve as a checklist for the project teamplanning the dissemination of project results.THE PROCESS FOR COMMUNICATING RESULTSThe communication of project results must be systematic, timely, andwell planned, and the process must include seven components in a precisesequence. The ﬁrst step is critical and consists of an analysis of the need tocommunicate the results from a project. Possibly, a lack of support for theproject was identiﬁed, or perhaps the need for adjusting or maintainingthe funding for the project was uncovered. Instilling conﬁdence or buildingcredibility for the project may be necessary. It is important ﬁrst of all tooutline the speciﬁc reasons for communicating the results.The second step focuses on the plan for communication. Planningshould include numerous agenda items to be addressed in all communi-cations about the project. Planning also covers the actual communication,detailing the speciﬁc types of data to be communicated, and when and towhich groups the communication will be presented.The third step involves selecting the target audiences for communi-cation. Audiences range from top management to past participants, andeach audience has its own special needs. All groups should be considered
248 REPORTING RESULTSin the communication strategy. An artfully crafted, targeted delivery maybe necessary to win the approval of a speciﬁc group.The fourth step is developing a report, the written material explainingproject results. This can encompass a wide variety of possibilities, froma brief summary of the results to a detailed research document on theevaluation effort. Usually, a complete report is developed, and selectedparts or summaries from the report are used for different media.Media selection is the ﬁfth step. Some groups respond more favorablyto certain methods of communication. A variety of approaches, both oraland written, are available to the project leaders.The content, in the form of detailed information, represents the sixthstep. The communication is delivered with the utmost care, conﬁdence,and professionalism.The last step, but certainly not the least signiﬁcant, is analyzingreactions to the communication. Positive reactions, negative feedback,and a lack of comments are all indicators of how well the informationwas received and understood. An informal analysis may be appropriatefor many situations. For an extensive and more involved communicationeffort, a formal, structured feedback process may be necessary. The natureof the reactions could trigger an adjustment to the subsequent communi-cation of results for the same project or provide input for adapting futureproject communications.The various steps are discussed further in the following sections.THE NEED FOR COMMUNICATIONBecause there may be various reasons for communicating results, a listshould be tailored to the organization and adjusted as necessary. Thereasons for communicating results depend on the speciﬁc project, thesetting, and the unique needs of each party. Some of the most commonreasons are• Securing approval for the project and the allocation of time andmoney• Gaining support for the project and its objectives• Securing agreement on the issues, solutions, and resources• Enhancing the credibility of the project leader• Reinforcing the processes used in the project• Driving action for improvement in the project
The Communication Plan 249• Preparing participants for the project• Optimizing results throughout the project and the quality of futurefeedback• Showing the complete results of the project• Underscoring the importance of measuring results• Explaining techniques used to measure results• Motivating participants to become involved in the project• Demonstrating accountability for expenditures• Marketing future projectsThere may be other reasons for communicating results, so the listshould be tailored to the needs of each organization.THE COMMUNICATION PLANAny activity must be carefully planned to achieve maximum results.This is a critical part of communicating the results of the project. Theactual planning of the communication is important to ensure that eachaudience receives the proper information at the right time and thatnecessary actions are taken. Several issues are crucial in planning thecommunication of results:• What will be communicated?• When will the data be communicated?• How will the information be communicated?• Where will the information be communicated?• Who will communicate the information?• Who is the target audience?• What are the speciﬁc actions required or desired?The communication plan is usually developed when the project isapproved. This plan details how speciﬁc information is to be developed andcommunicated to various groups and the expected actions. In addition,this plan details how the overall results will be communicated, the timeframe for communication, and the appropriate groups to receive theinformation. The project leader, key managers, and stakeholders need toagree on the degree of detail in the plan.An impact study can be used to present the results of a project. Thisis developed when a major project is completed and the overall, detailedresults are known. Among the major questions to be answered in an
250 REPORTING RESULTSimpact study are who should receive the results and in what form. Theimpact study is more specialized than the plan for the overall projectbecause it involves the ﬁnal results of the project.THE AUDIENCE FOR COMMUNICATIONSThe following questions should be asked about each potential audiencefor communication of project results:• Are they interested in the project?• Do they really want to receive the information?• Has a commitment been made to include them in the communica-tions?• Is the timing right for this audience?• Are they familiar with the project?• How do they prefer to have results communicated?• Do they know the project leader? The project team?• Are they likely to ﬁnd the results threatening?• Which medium will be most convincing to this group?For each target audience, three steps are necessary. To the greatestextent possible, the project leader should get to know and understand thetarget audience. Also, the project leader should ﬁnd out what informationis needed and why. Each group will have its own required amount ofinformation; some will want detailed information while others will prefera brief overview. Rely on the input from others to determine the audience’sneeds. Finally, the project leaders should take into account audience bias.Some audiences will immediately support the results, others may opposethem, and still others will be neutral. The staff should be empatheticand try to understand the basis for the differing views. Given thisunderstanding, communications can be tailored to each group. This iscritical when the potential exists for the audience to react negatively tothe results.The target audiences for information on project results are varied interms of job levels and responsibilities. Determining which groups willreceive a particular item of communication requires careful thought,because problems can arise when a group receives inappropriate informa-tion or is overlooked altogether. A sound basis for audience selection is toanalyze the reason for the communication, as discussed earlier. Table 13.1identiﬁes common target audiences and the basis for audience selection.
The Audience for Communications 251Table 13.1 Common Target AudiencesPrimary Target Audience Reason for CommunicationClient, top executives To secure approval for the projectImmediate managers, team leaders To gain support for the projectParticipants, team leaders To secure agreement with the issuesTop executives To enhance the credibility of the projectImmediate managers To reinforce the processesProject team To drive action for improvementTeam leaders To prepare participants for the projectParticipants To improve the results and quality offuture feedbackStakeholders To show the complete results of theprojectClient, project team To underscore the importance ofmeasuring resultsClient, project support staff To explain the techniques used tomeasure resultsTeam leaders To create the desire for a participant tobe involvedAll employees To demonstrate accountability forexpendituresProspective clients To market future projectsSeveral audiences stand out as critical. Perhaps the most important audi-ence is the client. This group (or individual) initiates the project, reviewsdata, usually selects the project leader, and weighs the ﬁnal assessmentof the effectiveness of the project. Another important target audience istop management. This group is responsible for allocating resources tothe project and needs information to help them justify expenditures andgauge the effectiveness of the efforts.Participants need feedback on the overall success of the effort. Someindividuals may not have been as successful as others in achieving thedesired results. Communicating the results creates additional pressureto implement the project effectively and improve results in the future.For those achieving excellent results, the communication will serve asreinforcement. Communication of results to project participants is oftenoverlooked, with the assumption that once the project is completed, theydo not need to be informed of its success.
252 REPORTING RESULTSCommunicating with the participants’ immediate managers is essen-tial. In many cases, these managers must encourage participants toimplement the project. Also, they are key in supporting and reinforc-ing the objectives of the project. An appropriate return on investmentstrengthens the commitment to projects and enhances the credibility ofthe project team.The project team must receive information about project results.Whether for small projects in which team members receive a projectupdate, or for larger projects where a complete team is involved, thosewho design, develop, facilitate, and implement the project require infor-mation on the project’s effectiveness. Evaluation data are necessary sothat adjustments can be made if the project is not as effective as it wasprojected to be.INFORMATION DEVELOPMENT: THE IMPACT STUDYThe type of formal evaluation report to be issued depends on the degreeof detail in the information presented to the various target audiences.Brief summaries of project results with appropriate charts may be suf-ﬁcient for some communication efforts. In other situations, particularlythose involving major projects requiring extensive funding, a detailedevaluation report is crucial. A complete and comprehensive impact studyreport is usually necessary. This report can then be used as the basis formore streamlined information aimed at speciﬁc audiences and using var-ious media. One possible format for an impact study report is presentedin here.Format of an Impact Study Report• General information• Background• Objectives of study• Methodology for impact study• Levels of evaluation• ROI process• Collecting data
Information Development: The Impact Study 253• Isolating the effects of the project• Converting data to monetary values• Data analysis issues• Costs• Results: General information• Response proﬁle• Success with objectives• Results: Reaction and perceived• Data sources• Data summary• Key issues• Results: Learning and conﬁdence• Data sources• Data summary• Key issues• Results: Application and implementation• Data sources• Data summary• Key issues• Results: Impact and consequences• General comments• Linkage with business measures• Key issues• Results: ROI and its meaning• Results: Intangible measures• Barriers and enablers• Barriers• Enablers• Conclusions and recommendations• Conclusions• Recommendations
254 REPORTING RESULTSWhile the impact study report is an effective, professional way topresent ROI data, several cautions are in order. Since this report doc-uments the success of a project involving a large group of employees,credit for the success must go completely to the participants and theirimmediate leaders. Their performance generated the success. Also, it isimportant to avoid boasting about results. Grand claims of overwhelmingsuccess can quickly turn off an audience and interfere with the deliveryof the desired message.The methodology should be clearly explained, along with the assump-tions made in the analysis. The reader should easily see how the valueswere developed and how speciﬁc steps were followed to make the processmore conservative, credible, and accurate. Detailed statistical analysesshould be placed in an appendix.MEDIA SELECTIONMany options are available for the dissemination of project results. Inaddition to the impact study report, commonly used media are meetings,interim and progress reports, organization publications, and case studies.Table 13.2 lists a variety of options.MeetingsIf used properly, meetings are fertile ground for the communication ofproject results. All organizations hold a variety of meetings, and somemay provide the proper context to convey project results. Along the chainof command, staff meetings are held to review progress, discuss currentproblems, and distribute information. These meetings can be an excellentTable 13.2 Options for Communicating ResultsDetailed Electronic MassReports Brief Reports Reporting PublicationsImpact study ExecutivesummaryWebsite AnnouncementsCase study(internal)Slide overview E-mail BulletinsCase study(external)One-pagesummaryBlog NewslettersMajor articles Brochure Video Brief articles
Media Selection 255forum for discussing the results achieved in a project that relates to thegroup’s activities. Project results can be sent to executives for use in astaff meeting, or a member of the project team can attend the meeting tomake the presentation.Regular meetings with management groups are a common practice.Typically, discussions will focus on items that might be of help to workunits. The discussion of a project and its results can be integrated intothe regular meeting format. A few organizations have initiated the useof periodic meetings for all key stakeholders, where the project leaderreviews progress and discusses next steps. A few highlights from interimproject results can be helpful in building interest, commitment, andsupport for the project.Interim and Progress ReportsA highly visible way to communicate results, although usually limited tolarge projects, is the use of interim and routine memos and reports. Pub-lished or disseminated by e-mail on a periodic basis, they are designedto inform management about the status of the project, to communi-cate interim results of the project, and to spur needed changes andimprovements.A secondary reason for the interim report is to enlist additional supportand commitment from the management group and to keep the projectintact. This report is produced by the project team and distributedto a select group of stakeholders in the organization. The report mayvary considerably in format and scope and may include a schedule ofplanned steps or activities, a brief summary of reaction evaluations, initialresults achieved from the project, and various spotlights recognizingteam members or participants. Other topics may also be appropriate.When produced in a professional manner, the interim report can boostmanagement support and commitment.Routine Communication ToolsTo reach a wide audience, the project leader can use internal, routinepublications. Whether a newsletter, magazine, newspaper, or electronicﬁle, these media usually reach all employees or stakeholders. The con-tent can have a signiﬁcant impact if communicated appropriately. The
256 REPORTING RESULTSscope should be limited to general-interest articles, announcements, andinterviews.Results communicated through these types of media must be importantenough to arouse general interest. For example, a story with the headline‘‘Safety Project Helps Produce One Million Hours without a Lost-TimeAccident’’ will catch the attention of many readers because it is likely theyparticipated in the project and can appreciate the relevance of the results.Reports on the accomplishments of a group of participants may notgenerate interest if the audience cannot relate to the accomplishments.For many projects, results are not achieved until weeks or even monthsafter the project is completed. Participants need reinforcement frommany sources. Communicating results to a general audience may lead toadditional pressure to continue the project or introduce similar ones inthe future.Stories about participants involved in a project and the results theyhave achieved can help create a favorable image. Employees are madeaware that the organization is investing time and money to improveperformance and prepare for the future. This type of story providesinformation about a project that employees otherwise may be unfamiliarwith, and it sometimes creates a desire in others to participate if giventhe opportunity.General-audience communication can bring recognition to projectparticipants, particularly those who excel in some aspect of the project.Public recognition of participants who deliver exceptional performancecan enhance their self-esteem and their drive to continue to excel. Aproject can generate many human interest stories. A rigorous projectwith difﬁcult challenges can provide the basis for an interesting story onparticipants who made the extra effort.E-Mail and Electronic MediaInternal and external Internet pages, companywide intranets, and e-mails are excellent vehicles for releasing results, promoting ideas, andinforming employees and other target groups of project results. E-mail, inparticular, provides a virtually instantaneous means of communicatingresults to and soliciting responses from large groups of people. For majorprojects, some organizations create blogs to present results and elicitreactions, feedback, and suggestions.
Media Selection 257Project Brochures and PamphletsA brochure might be appropriate for a project conducted on a continuingbasis or where the audience is large and continuously changing. Thebrochure should be attractive and present a complete description of theproject, with a major section devoted to results obtained with previousparticipants, if available. Measurable results and reactions from par-ticipants, or even direct quotes from individuals, can add spice to anotherwise dull brochure.Case StudiesCase studies represent an effective way to communicate the results of aproject. A typical case study describes the situation, provides appropriatebackground information (including the events that led to the project),presents the techniques and strategies used to develop the study, andhighlights the key issues in the project. Case studies tell an interest-ing story of how the project was implemented and the evaluation wasdeveloped, including the problems and concerns identiﬁed along the way.Routine Feedback on Project ProgressA primary reason for collecting reaction and learning data is to providefeedback so that adjustments can be made throughout the project. Formost projects, data are routinely collected and quickly communicated to avariety of groups. A feedback action plan designed to provide informationto several audiences using a variety of media may be an option. Thesefeedback sessions may point out speciﬁc actions that need to be taken.This process becomes complex and must be managed in a very proactivemanner. The following steps are recommended for providing feedback andmanaging the overall process. Many of the steps and concepts are basedon the recommendations of Peter Block in his successful book FlawlessConsulting.1• Communicate quickly. Whether the news is good news or bad, itshould be passed on to individuals involved in the project as soon aspossible. The recommended time for providing feedback is usually amatter of days and certainly no longer than a week or two after theresults become known.
258 REPORTING RESULTS• Simplify the data. Condense the data into an easily understand-able, concise presentation. This is not the appropriate situation fordetailed explanations and analysis.• Examine the role of the project team and the client in the feedbackprocess. The project leader is often the judge, jury, prosecutor,defendant, and/or witness. On the other hand, sometimes the clientﬁlls these roles. These respective functions must be examined interms of reactions to the data and the actions that are called for.• Use negative data in a constructive way. Some of the data will showthat things are not going so well, and the fault may rest with theproject leader or the client. In this case, the story basically changesfrom ‘‘Let’s look at the success we’ve achieved’’ to ‘‘Now we knowwhich areas to change.’’• Use positive data in a cautious way. Positive data can be misleading,and if they are communicated too enthusiastically, they may createexpectations that exceed what ﬁnally materializes. Positive datashould be presented in a cautious way—almost in a discountingmanner.• Choose the language of the meeting and the communication care-fully. The language used should be descriptive, focused, speciﬁc,short, and simple. Language that is too judgmental, macro, stereo-typical, lengthy, or complex should be avoided.• Ask the client for reactions to the data. After all, the client is thenumber one customer, and it is most important that the client bepleased with the project.• Ask the client for recommendations. The client may have some goodsuggestions for what needs to be changed to keep a project on track,or to put it back on track should it derail.• Use support and confrontation carefully. These two actions are notmutually exclusive. At times, support and confrontation are bothneeded for a particular group. The client may need support and yetbe confronted for lack of improvement or sponsorship. The projectteam may be confronted regarding the problem areas that havedeveloped, but may need support as well.• React to and act on the data. The different alternatives and possibil-ities should be weighed carefully to arrive at the adjustments thatwill be necessary.• Secure agreement from all key stakeholders. It is essential to ensurethat everyone is willing to make any changes that may be necessary.
Media Selection 259• Keep the feedback process short. Allowing the process to becomebogged down in long, drawn-out meetings or lengthy documents is abad idea. If this occurs, stakeholders will avoid the process insteadof being willing participants.Following these steps will help move the project forward and generateuseful feedback, often ensuring that adjustments are supported and canbe executed.Presentation of Results to Senior ManagementPerhaps one of the most challenging and stressful types of communicationis presenting an impact study to the senior management team, which alsoserves as the client for a project. The challenge is convincing this highlyskeptical and critical group that outstanding results have been achieved(assuming they have) in a very reasonable time frame, addressing thesalient points, and making sure the managers understand the process.Two potential reactions can create problems. First, if the results arevery impressive, making the managers accept the data may be difﬁcult.On the other extreme, if the data are negative, ensuring that managersdon’t overreact to the results and look for someone to blame is important.Several guidelines can help ensure that this process is planned andexecuted properly.Arrange a face-to-face meeting with senior team members to review theﬁrst one or two major impact studies. If they are unfamiliar with the ROImethodology, such a meeting is necessary to make sure they understandthe process. The good news is that they will probably attend the meetingbecause they have never seen ROI data developed for this type of project.The bad news is that it takes a lot of time, usually about an hour, forthis presentation. After the meeting with a couple of presentations, anexecutive summary may sufﬁce. At this point, the senior members willunderstand the process, so a shortened version may be appropriate. Oncethe target audience is familiar with the process, a brief version may bedeveloped, including a one- to two-page summary with charts and graphsshowing the six types of measures.The results should not be disseminated before the initial presentationor even during the session, but should be saved until the end of thesession. This will allow enough time to present the process and gatherreactions to it before the target audience sees the ROI calculation. Presentthe ROI methodology step by step, showing how the data were collected,
260 REPORTING RESULTSwhen they were collected, who provided them, how the effect of the projectwas isolated from other inﬂuences, and how data were converted to mon-etary values. The various assumptions, adjustments, and conservativeapproaches are presented along with the total cost of the project, sothat the target audience will begin to buy into the process of developingthe ROI.When the data are actually presented, the results are given one level ata time, starting with Level 1, moving through Level 5, and ending with theintangibles. This allows the audience to observe the reaction, learning,application and implementation, business impact, and ROI procedures.After some discussion of the meaning of the ROI, the intangible measuresare presented. Allocate time for each level as appropriate for the audience.This helps to defuse potential emotional reactions to a very positive ornegative ROI.Show the consequences of additional accuracy if this is an issue. Thetrade-off for more accuracy and validity often is more expense. Addressthis issue when necessary, agreeing to add more data if they are required.Collect concerns, reactions, and issues involving the process and makeadjustments accordingly for the next presentation.Collectively, these steps will help in the preparation and presentationof one of the most important meetings in the ROI process. Figure 13.1Presentation SequenceDescribe the program and explain why it is being evaluatedPresent the methodology process.Present the reaction and learning data.Present the application data.List the barriers and enablers to success.Address the business impact.Show the costs.Present the ROI.Show the intangibles.Review the credibility of the data.Summarize the conclusions.Present the recommendations.220.127.116.11.18.104.22.168.22.214.171.124.Purpose of the Meeting• Create awareness and understandingof ROI.• Build support for the ROI methodology.• Communicate results of study• Drive improvement from results.• Cultivate effective use of the ROImethodology.Meeting Ground Rules• Do not distribute the impact study untilthe end of the meeting• Be precise and to the point.• Avoid jargon and unfamiliar terms.• Spend less time on the lower levels ofevaluation data.• Present the data with a strategy in mind.Figure 13.1 Presenting the impact study to executive sponsors.
Final Thoughts 261shows the recommended approach to an important meeting with thesponsor.REACTIONS TO COMMUNICATIONThe best indicator of how effectively the results of a project have beencommunicated is the level of commitment and support from the managers,executives, and sponsors. The allocation of requested resources and voicedcommitment from top management are strong evidence of management’spositive perception of the results. In addition to this macro-level reaction,a few techniques can also be helpful in measuring the effectiveness of thecommunication effort.When results are communicated, the reactions of the target audi-ences can be monitored. These reactions may include nonverbal gestures,oral remarks, written comments, or indirect actions that reveal how thecommunication was received. Usually, when results are presented ina meeting, the presenter will have some indication of how they werereceived by the group. Usually, the interest and attitudes of the audi-ence can be quickly evaluated. Comments about the results—formal orinformal—should be noted and tabulated.Project team meetings are an excellent arena for discussing the reac-tion to communicated results. Comments can come from many sourcesdepending on the particular target audience. When major project resultsare communicated, a feedback questionnaire may be administered to theentire audience or a sample of the audience. The purpose of the ques-tionnaire is to determine the extent to which the audience understoodand/or believed the information presented. This is practical only whenthe effectiveness of the communication will have a signiﬁcant impact onfuture actions by the project team.FINAL THOUGHTSThe ﬁnal step in the ROI methodology, communication of results, is acrucial step in the overall evaluation process. If this step is not executedadequately, the full impact of the results will not be recognized, and thestudy may amount to a waste of time. The chapter began with generalprinciples and steps for communicating project results; these can serveas a guide for any signiﬁcant communication effort. The various targetaudiences were then discussed, with emphasis on the executive groupbecause of its importance. A suggested format for a detailed evaluation
262 REPORTING RESULTSreport was also provided. The chapter presented the most commonlyused media for communicating project results, including meetings, clientpublications, and electronic media.A ﬁnal issue regarding the ROI methodology will be discussed inthe next chapter: overcoming barriers to sustaining the use of themethodology.
264 IMPLEMENTING AND SUSTAINING ROIlegitimate barriers are the basis for resistance, minimizing or removingthem altogether is the challenge.Implementation Is KeyAs with any process, effective implementation is the key to its success.This occurs when the new technique, tool, or process is integrated intothe routine framework. Without effective implementation, even the bestprocess will fail. A process that is never removed from the shelf willnever be understood, supported, or improved. Clear-cut steps must bein place for designing a comprehensive implementation process that willovercome resistance.Consistency Is NeededConsistency is an important consideration as the ROI process is imple-mented. With consistency come accuracy and reliability. The only way tomake sure consistency is achieved is to follow clearly deﬁned processesand procedures each time the ROI methodology is used. Proper effectiveimplementation will ensure that this occurs.EfﬁciencyCost control and efﬁciency will be signiﬁcant considerations in any majorundertaking, and the ROI methodology is no exception. During imple-mentation, tasks must be completed efﬁciently and effectively. Doing sowill help ensure that process costs are kept to a minimum, that time isused economically, and that the process remains affordable.IMPLEMENTING THE PROCESS: OVERCOMINGRESISTANCEResistance shows up in varied ways: in the form of comments, remarks,actions, or behaviors. Table 14.1 lists representative comments thatindicate open resistance to the ROI process. Each comment signals anissue that must be resolved or addressed in some way. A few are basedon realistic barriers, whereas others are based on myths that mustbe dispelled. Sometimes, resistance to the process reﬂects underlyingconcerns. For example, the project managers involved may fear losing
Implementing the Process: Overcoming Resistance 265Table 14.1 Typical Objections to the Use of ROI MethodologyOpen Resistance1. It costs too much.2. It takes too much time.3. Who is asking for this?4. This is not in my job description.5. I did not have input on this.6. I do not understand this.7. What happens when the results are negative?8. How can we be consistent with this?9. The ROI looks too subjective.10. Our managers will not support this.11. ROI is too narrowly focused.12. This is not practical.control of their processes, and others may feel vulnerable to whateveraction may follow if the project is not successful. Still others may beconcerned about any process that brings change or requires the additionaleffort of learning.Project managers and team members may resist the ROI process andopenly make comments similar to those listed in Table 14.1. It maytake heavy persuasion and evidence of tangible beneﬁts to convince teammembers that it is in their best interest to make the project a success.Although most clients do want to see the results of the project, they mayhave concerns about the information they are asked to provide and aboutwhether their personal performance is being judged while the projectis undergoing evaluation. Participants may express the very same fearslisted in the table.The challenge is to implement the methodology systematically andconsistently so that it becomes normal business behavior and a routineand standard process built into projects. The implementation necessaryto overcome resistance covers a variety of areas. Figure 14.1 showsactions outlined in this chapter that are presented as building blocks toovercoming resistance. They are all necessary to build the proper baseor framework to dispel myths and remove or minimize barriers. Theremainder of this chapter presents speciﬁc strategies and techniquesdevoted to each building block identiﬁed in Figure 14.1. They apply
266 IMPLEMENTING AND SUSTAINING ROIInitiating ROI projectsPreparing clients and executivesRemoving obstaclesMonitoring progressPreparing project teamRevising policies and proceduresEstablishing goals and plansDeveloping roles and responsibilitiesAssessing climate for measuring ROIFigure 14.1 Building blocks to overcome resistance.equally to the project team and the client organization, and no attemptis made to separate the two in this presentation. In some situations, aparticular strategy would work best with the project team. In certaincases all strategies may be appropriate for both groups.ASSESSING THE CLIMATEAs a ﬁrst step toward implementation, some organizations assess thecurrent climate for achieving results. One way to do this is to developa survey to determine current perspectives of the project managementteam and other stakeholders (for an example go to www.roiinstitute.net).Another way is to conduct interviews with key stakeholders to determinetheir willingness to follow the project through to ROI. With an awarenessof the current status, the project leaders can plan for signiﬁcant changesand pinpoint particular issues that need support as the ROI process isimplemented.DEVELOPING ROLES AND RESPONSIBILITIESDeﬁning and detailing speciﬁc roles and responsibilities for differentgroups and individuals addresses many of the resistance factors andhelps pave a smooth path for implementation.
Developing Roles and Responsibilities 267Identifying a ChampionAs an early step in the process, one or more individual(s) should bedesignated as the internal leader or champion for the ROI methodology.As in most change efforts, someone must take responsibility for ensuringthat the process is implemented successfully. This leader serves as achampion for ROI and is usually the one who understands the processbest and sees vast potential for its contribution. More important, thisleader is willing to teach others and will work to sustain sponsorship.Developing the ROI LeaderThe ROI leader is usually a member of the project team who has theresponsibility for evaluation. For large organizations, the ROI leadermay be part of the support services for project management. This personholds a full-time position in larger project teams or a part-time positionin smaller teams. Client organizations may also have an ROI leader whopursues the ROI methodology from the client’s perspective. The typicaljob title for a full-time ROI leader is Manager of Measurement andEvaluation. Some organizations assign this responsibility to a team andempower it to lead the ROI effort.In preparation for this assignment, individuals usually receive specialtraining that builds speciﬁc skills and knowledge of the ROI process. Therole of the implementation leader is quite broad and serves a varietyof specialized duties. In some organizations, the implementation leadercan take on many roles, ranging from problem solver to communicator tocheerleader.Leading the ROI process is a difﬁcult and challenging assignmentthat requires unique skill. Fortunately, programs are available thatteach these skills. For example, one such program is designed to certifyindividuals who will be assuming leadership roles in the implementationof the ROI methodology. For more detail, see www.roiinstitute.net. Thiscertiﬁcation is built around ten speciﬁc skill sets linked to successfulROI implementation, focusing on the critical areas of data collection,isolating the effects of the project, converting data to monetary value,presenting evaluation data, and building capability. This process is quitecomprehensive but may be necessary to build the skills necessary fortaking on this challenging assignment.
268 IMPLEMENTING AND SUSTAINING ROIEstablishing a Task ForceMaking the ROI methodology work well may require the use of a taskforce. A task force usually comprises a group of individuals from differentparts of the project or client team who are willing to develop the ROImethodology and implement it in the organization. The selection of thetask force may involve volunteers, or participation may be mandatorydepending on speciﬁc job responsibilities. The task force should representthe cross section necessary for accomplishing stated goals. Task forceshave the additional advantage of bringing more people into the processand developing more ownership of and support for the ROI methodology.The task force must be large enough to cover the key areas but not solarge that it becomes too cumbersome to function. Six to twelve membersis a good size.Assigning ResponsibilitiesDetermining speciﬁc responsibilities is critical because confusion canarise when individuals are unclear about their speciﬁc assignmentsin the ROI process. Responsibilities apply to two areas. The ﬁrst isthe measurement and evaluation responsibility of the entire projectteam. Everyone involved in projects must have some responsibility formeasurement and evaluation. These responsibilities include providinginput on designing instruments, planning speciﬁc evaluations, analyzingdata, and interpreting the results. Typical responsibilities include:• Ensuring that the initial analysis for the project includes speciﬁcbusiness impact measures• Developing speciﬁc application and business impact objectives forthe project• Keeping participants focused on application and impact objectives• Communicating rationale and reasons for evaluation• Assisting in follow-up activities to capture application and businessimpact data• Providing assistance for data collection, data analysis, and reportingAlthough involving each member of the project team in all theseactivities may not be appropriate, each individual should have at leastone responsibility as part of his or her routine job duties. This assignmentof responsibility keeps the ROI methodology from being disjointed and
Establishing Goals and Plans 269separated during projects. More important, it brings accountability tothose directly involved in project implementation.Another issue involves technical support. Depending on the size of theproject team, establishing a group of technical experts to provide assis-tance with the ROI process may be helpful. Once the group is established,the project team must understand that the experts have been assignednot for the purpose of relieving the team of its evaluation responsibil-ities, but to supplement its ROI efforts with technical expertise. Thesetechnical experts are typically the individuals who participated in thecertiﬁcation and training process to build special skills. Responsibilitiesof the technical support group involve six key areas:1. Designing data collection instruments2. Providing assistance for developing an evaluation strategy3. Analyzing data, including specialized statistical analyses4. Interpreting results and making speciﬁc recommendations5. Developing an evaluation report or case study to communicateoverall results6. Providing technical support in all phases of the ROI methodologyThe assignment of responsibilities for evaluation requires attentionthroughout the evaluation process. Although the project team must beassigned speciﬁc responsibilities during an evaluation, requiring othersto serve in support functions to help with data collection is not unusual.These responsibilities are deﬁned when a particular evaluation strategyplan is developed and approved.ESTABLISHING GOALS AND PLANSEstablishing goals, targets, and objectives is critical to the implementa-tion, particularly when several projects are planned. The establishmentof goals can include detailed planning documents for the overall processand for individual ROI projects. The next sections discuss aspects of theestablishment of goals and plans.Setting Evaluation TargetsEstablishing speciﬁc targets for evaluation levels is an important wayto make progress with measurement and evaluation. As emphasized
270 IMPLEMENTING AND SUSTAINING ROITable 14.2 Evaluation Targets in a Large Organizationwith Many ProjectsLevel TargetLevel 1, Reaction 100%Level 2, Learning 80%Level 3, Application and Implementation 40%Level 4, Business Impact 25%Level 5, ROI 10%throughout this book, not every project should be evaluated to ROI.Knowing in advance to which level the project will be evaluated helps inplanning which measures will be needed and how detailed the evaluationmust be at each level. Table 14.2 presents examples of targets set forevaluation at each level. The setting of targets should be completedearly in the process with the full support of the entire project team. Ifpractical and feasible, the targets should also have the approval of keymanagers—particularly the senior management team.Developing a Plan for ImplementationAn important part of implementation is establishing a timetable for thecomplete implementation of the ROI process. This document becomes amaster plan for completion of the different elements presented earlier.Beginning with forming a team and concluding with meeting the targetspreviously described, this schedule is a project plan for transitioningfrom the present situation to the desired future situation. Items on theschedule include developing speciﬁc ROI projects, building staff skills,developing policy, and teaching managers the process. Figure 14.2 is anexample of an implementation plan. The more detailed the document, themore useful it becomes. The project plan is a living, long-range documentthat should be reviewed frequently and adjusted as necessary. Moreimportant, those engaged in work on the ROI methodology should alwaysbe familiar with the implementation plan.REVISING OR DEVELOPING POLICIES AND GUIDELINESAnother part of planning is revising or developing the organization’spolicy on project measurement and evaluation. The policy statementcontains information developed speciﬁcally for the measurement and
Revising or Developing Policies and Guidelines 271J F M A M J J A S SO ON ND J F M A AM J JTeam formedResponsibilities definedPolicy developedTargets setWorkshops developedROI Project (A)ROI Project (B)ROI Project (C)ROI Project (D)Project teams trainedManagers trainedSupport tools developedGuidelines developedFigure 14.2 Implementation plan for a large organization with many projects.evaluation process. It is developed with input from the project team andkey managers or stakeholders. Sometimes, policy issues are addressedduring internal workshops designed to build measurement and evaluationskills. The policy statement addresses critical matters that will inﬂuencethe effectiveness of the measurement and evaluation process. Thesemay include adopting the ﬁve-level framework presented in this book,requiring Level 3 and 4 objectives for some or all projects, and deﬁningresponsibilities for the project team.Policy statements are important because they provide guidance anddirection for the staff and others who work closely with the ROI method-ology. These individuals keep the process clearly focused, and enable thegroup to establish goals for evaluation. Policy statements also providean opportunity to communicate basic requirements and fundamentals ofperformance and accountability. More than anything else, they serve aslearning tools to teach others, especially when they are developed in acollaborative way. If policy statements are developed in isolation, staffand management will be denied the sense of their ownership, makingthem neither effective nor useful.
272 IMPLEMENTING AND SUSTAINING ROIGuidelines for measurement and evaluation are important for showinghow to use the tools and techniques, guide the design process, provideconsistency in the ROI process, ensure that appropriate methods areused, and place the proper emphasis on each of the areas. The guidelinesare more technical than policy statements and often include detailedprocedures showing how the process is undertaken and developed. Theyoften include speciﬁc forms, instruments, and tools necessary to facilitatethe process.PREPARING THE PROJECT TEAMProject team members may resist the ROI methodology. They often seeevaluation as an unnecessary intrusion into their responsibilities thatabsorbs precious time and stiﬂes creative freedom. The cartoon characterPogo perhaps characterized it best when he said, ‘‘We have met theenemy, and he is us.’’ Several issues must be addressed when preparingthe project team for ROI implementation.Involving the Project TeamFor each key issue or major decision involving ROI implementation, theproject team should be involved in the process. As policy statements areprepared and evaluation guidelines developed, team input is essential.Resistance is more difﬁcult if the team helped design and develop theROI process. Convene meetings, brainstorming sessions, and task forcesto involve the team in every phase of developing the framework andsupporting documents for ROI.Using ROI as a Learning ToolOne reason the project team may resist the ROI process is that theprojects’ effectiveness will be fully exposed, putting the reputation of theteam on the line. They may have a fear of failure. To overcome this, theROI methodology should be clearly positioned as a tool for learning, not atool for evaluating project team performance (at least not during the earlyyears of project implementation). Team members will not be interested indeveloping a process that may reﬂect unfavorably on their performance.Evaluators can learn as much from failures as from success. If theproject is not working, it is best to ﬁnd out quickly so that issues can
Initiating ROI Studies 273be understood ﬁrsthand, not from others. If a project is ineffective andnot producing the desired results, the failure will eventually be known toclients and the management group (if they are not aware of it already).A lack of results will make managers less supportive of immediate andfuture projects. If the projects’ weaknesses are identiﬁed and adjustmentsquickly made, not only can more effective projects be developed, but thecredibility of and respect for project implementation will be enhanced.Teaching the TeamThe project team and project evaluator usually have inadequate skills inmeasurement and evaluation, and will need to develop some expertise.Measurement and evaluation are not always a formal part of the team’sor evaluator’s job preparation. Consequently, the project team leadermust learn ROI methodology and its systematic steps, and the evaluatormust learn to develop an evaluation strategy and speciﬁc plan, to collectand analyze data from the evaluation, and to interpret results fromdata analysis. A one- to two-day workshop can help build the skillsand knowledge needed to understand the process and appreciate whatit can do for project success and for the client organization. Such ateach-the-team workshop can be a valuable tool in ensuring successfulimplementation of ROI methodology.INITIATING ROI STUDIESThe ﬁrst tangible evidence of the value of using the ROI methodology maybe seen at the initiation of the ﬁrst project for which an ROI calculationis planned. The next sections discuss aspects of identifying appropriateprojects and keeping them on track.Selecting the Initial ProjectIt is critical that appropriate projects be selected for ROI analysis. Onlycertain types of projects qualify for comprehensive, detailed analysis.Characteristic of projects that are suitable for analysis are those that(1) involve large groups of participants; (2) are expected to have a longlifecycle; (3) will be linked to major operational problems and opportu-nities upon completion; (4) are important to strategic objectives; (5) areexpensive; (6) are time-consuming; (7) have high visibility; and (8) have
274 IMPLEMENTING AND SUSTAINING ROIthe interest of management in performing their evaluation. Using theseor similar criteria, the project leader must select the appropriate projectsto consider for ROI evaluation. Ideally, sponsors should agree with orapprove the criteria.Developing the Planning DocumentsPerhaps the two most useful ROI documents are the data collectionplan and the ROI analysis plan. The data collection plan shows whatdata will be collected, the methods used, the sources, the timing, andthe assignment of responsibilities. The ROI analysis plan shows howspeciﬁc analyses will be conducted, including how to isolate the effects ofthe project and how to convert data to monetary values. Each evaluatorshould know how to develop these plans. These documents were discussedin detail in Chapter 3.Reporting ProgressAs the projects are developed and the ROI implementation gets underway, status meetings should be conducted to report progress and discusscritical issues with appropriate team members. These meetings keep theproject team focused on the critical issues, generate the best ideas foraddressing problems and barriers, and build a knowledge base for betterimplementation evaluation of future projects. Sometimes, these meetingsare facilitated by an external consultant, perhaps an expert in the ROIprocess. In other cases, the project leader may facilitate. In essence, themeetings serve three major purposes: reporting progress, learning, andplanning.Establishing Discussion GroupsBecause the ROI methodology is considered difﬁcult to understand andapply, establishing discussion groups to teach the process may be helpful.These groups can supplement formal workshops and other learning activ-ities and are often very ﬂexible in format. Groups are usually facilitatedby an external ROI consultant or the project leader. In each session, a newtopic is presented for a thorough discussion that should extend to how thetopic applies to the organization. The process can be adjusted for differenttopics as new group needs arise, driving the issues. Ideally, participants in
Removing Obstacles 275group discussions will have an opportunity to apply, explore, or researchthe topics between sessions. Group assignments such as reviewing a casestudy or reading an article are appropriate between sessions to furtherthe development of knowledge and skills associated with the process.PREPARING THE CLIENTS AND EXECUTIVESPerhaps no group is more important to the ROI process than the man-agement team that must allocate resources for the project and support itsimplementation. In addition, the management team often provides inputto and assistance for the ROI process. Preparing, training, and developingthe management team should be carefully planned and executed.One effective approach for preparing executives and managers for theROI process is to conduct a brieﬁng on ROI. Varying in duration from onehour to half a day, a practical brieﬁng such as this can provide criticalinformation and enhance support for ROI use. Managers leave thesebrieﬁngs with greater appreciation of the use of ROI and its potentialimpact on projects, and with a clearer understanding of their role in theROI process. More important, they often renew their commitment to reactto and use the data collected by the ROI methodology.A strong, dynamic relationship between the project team and keymanagers is essential for successful implementation of the ROI method-ology. A productive partnership is needed that requires each party tounderstand the concerns, problems, and opportunities of the other. Thedevelopment of such a beneﬁcial relationship is a long-term process thatmust be deliberately planned for and initiated by key project team mem-bers. The decision to commit resources and support to a project may bebased on the effectiveness of this relationship.REMOVING OBSTACLESAs the ROI methodology is implemented, there will inevitably be obstaclesto its progress. The obstacles are based on concerns discussed in thischapter, some of which may be valid, others of which may be based onunrealistic fears or misunderstandings.Dispelling MythsAs part of the implementation, attempts should be made to dispel themyths and remove or minimize the barriers or obstacles. Much of the
276 IMPLEMENTING AND SUSTAINING ROIcontroversy regarding ROI stems from misunderstandings about whatthe process can and cannot do and how it can or should be implementedin an organization. After years of experience with ROI, and having notedreactions during hundreds of projects and workshops, observers recognizemany misunderstandings about ROI. These misunderstandings are listedbelow as myths about the ROI methodology:• ROI is too complex for most users.• ROI is expensive and consumes too many critical resources.• If senior management does not require ROI, there is no need topursue it.• ROI is a passing fad.• ROI is only one type of data.• ROI is not future-oriented; it only reﬂects past performance.• ROI is rarely used by organizations.• The ROI methodology cannot be easily replicated.• ROI is not a credible process; it is too subjective.• ROI cannot be used with soft projects.• Isolating the inﬂuence of other factors is not always possible.• ROI is appropriate only for large organizations.• No standards exist for the ROI methodology.For more information on these myths, see www.roiinstitute.net.Delivering Bad NewsOne of the obstacles perhaps most difﬁcult to overcome is receivinginadequate, insufﬁcient, or disappointing news. Addressing a bad-newssituation is an issue for most project leaders and other stakeholdersinvolved in a project. Table 14.3 presents the guidelines to follow whenaddressing bad news. As the table makes clear, the time to think aboutbad news is early in the process, but without ever losing sight of the valueof the bad news. In essence, bad news means that things can change andneed to change and that the situation can improve. The team and othersneed to be convinced that good news can be found in a bad-news situation.Using the DataIt is unfortunately too often the case that projects are evaluated andsigniﬁcant data are collected, but nothing is done with the data. Failure to
Removing Obstacles 277Table 14.3 How to Address Bad NewsDelivering Bad News• Never fail to recognize the power to learn from and improve with a negativestudy.• Look for red ﬂags along the way.• Lower outcome expectations with key stakeholders along the way.• Look for data everywhere.• Never alter the standards.• Remain objective throughout the process.• Prepare the team for the bad news.• Consider different scenarios.• Find out what went wrong.• Adjust the story line to ‘‘Now we have data that show how to make thisprogram more successful.’’ In an odd way, this puts a positive spin on datathat are less than positive.• Drive improvement.use data is a tremendous obstacle because once the project has concluded,the team has a tendency to move on to the next project or issue and get onwith other priorities. Table 14.4 shows how the different levels of data canbe used to improve projects. It is critical that the data be used—the datawere essentially the justiﬁcation for undertaking the project evaluationin the ﬁrst place. Failure to use the data may mean that the entireevaluation was a waste. As the table illustrates, many reasons exist forcollecting the data and using them after collection. These can becomeTable 14.4 How Data Should Be UsedAppropriate Level of DataUse of Evaluation Data 1 2 3 4 5Adjust project designImprove implementationInﬂuence application and impactImprove management support for the projectImprove stakeholder satisfactionRecognize and reward participantsJustify or enhance budgetReduce costsMarket projects in the future
278 IMPLEMENTING AND SUSTAINING ROIaction items for the team to ensure that changes and adjustments aremade. Also, the client or sponsor must act to ensure that the uses of dataare appropriately addressed.MONITORING PROGRESSA ﬁnal element of the implementation process is monitoring the over-all progress made and communicating that progress. Although oftenoverlooked, an effective communication plan can help keep the imple-mentation on target and can let others know what the ROI methodologyis accomplishing for project leaders and the client.The initial schedule for implementation of ROI is based on key events ormilestones. Routine progress reports should be developed to communicatethe status of these events or milestones. Reports are usually developed atsix-month intervals but may be more frequent for short-term projects. Twotarget audiences—the project team and senior managers—are criticalfor progress reporting. All project team members should be kept informedof the progress, and senior managers should know the extent to whichROI is being implemented and how it is working within the organization.FINAL THOUGHTSEven the best model or process will die if it is not used and sustained.This chapter explored the implementation of the ROI process and waysto sustain its use. If not approached in a systematic, logical, and plannedway, the ROI process will not be an integral part of project evalua-tion, and project accountability will consequently suffer. This chapterpresented the different elements that must be considered and issuesthat must be addressed to ensure that implementation is smooth anduneventful. Smooth implementation is the most effective means of over-coming resistance to ROI. The result provides a complete integration ofROI as a mainstream component of major projects. Good luck with thismethodology.
280 ENDNOTESCHAPTER 51. Miller, W., ‘‘Building the Ultimate Resource: Today’s Competi-tive Edge Comes from Intellectual Capital,’’ Management Review(January 1999), 42–45.CHAPTER 71. Nalbantian, Haig R., and Richard A. Guzzo, Dave Kieffer, and JayDoherty. Play to Your Strengths: Managing Your Internal LaborMarkets for Lasting Competitive Advantage (New York: McGraw-Hill, 2004).2. Kaplan, Robert S., and David P. Norton. The Balanced Scorecard:Translating Strategy into Action (Boston: Harvard Business SchoolPress, 1996).CHAPTER 81. Surowicki, James, The Wisdom of Crowds: Why the Many AreSmarter Than the Few and How Collective Wisdom Shapes Business,Economics, Societies and Nations (New York: Doubleday, 2004).CHAPTER 91. Farris, Paul W., Neil T. Bendle, Phillip E. Pfeifer, and David J.Ribstein, Marketing Metrics: 50+ Metrics Every Executive ShouldMaster (Upper Saddle River, NJ: Wharton School Publishing, 2006).p. 46–47.2. Campanella, Jack, ed., Principles of Quality Costs, 3rd ed. (Milwau-kee: American Society for Quality, 1999).3. Rust, Roland T., Anthony J. Zahorik, and Timothy L. Keining-ham, Return on Quality: Measuring the Financial Impact of YourCompany’s Quest for Quality (Chicago: Probus, 1994).4. Hurd, Mark, and Lars Nyberg, The Value Factor: How GlobalLeaders Use Information for Growth and Competitive Advantage(New York: Bloomberg Press, 2004).5. Ulrich, Dave, ed., Delivering Results (Boston: Harvard BusinessSchool Press, 1998).
Endnotes 281CHAPTER 101. Boulton, Richard E. S., Barry D. Libert, and Steve M. Samek.Cracking the Value Code (New York: HarperBusiness, 2000).2. Frangos, Cassandra A. ‘‘Aligning Learning with Strategy,’’ ChiefLearning Ofﬁcer, March 2004, p. 26.3. Alden, Jay. ‘‘Measuring the ‘Unmeasurable,’’’ Performance Improve-ment, May/June 2006, p. 7.4. Farris, Paul W., Neil T. Bendle, Phillip E. Pfeifer, and David J.Ribstein, Marketing Metrics: 50+ Metrics Every Executive ShouldMaster (Upper Saddle River, NJ: Wharton School Publishing, 2006),p. 16.5. Brown, Stuart F. ‘‘Scientiﬁc Americans,’’ Fortune, September 20,2004, p. 175.6. Kandybihn, Alexander, and Martin Kihn. ‘‘Raising Your Return onInnovation Investment,’’ Strategy + Business, Issue 35, 2004.7. Phillips, Jack J., and Lynn Schmidt. The Leadership Scorecard(Woburn, MA: Butterworth-Heinemann, 2004).CHAPTER 111. ‘‘Annual Employee Beneﬁts Report,’’ Nation’s Business (January2006).CHAPTER 121. Armstrong, Scott J., Principles of Forecasting: A Handbook forResearchers and Practitioners (Boston, MA: Kluwer Academic Pub-lishers, 2001).2. Phillips, Jack J., The Consultant’s Scorecard, 2nd Edition (NewYork: McGraw-Hill, 2010).3. Phillips, Jack J., ROI in Training and Performance ImprovementPrograms (Woburn, MA: Butterworth-Heinemann, 2003).4. Bowers, David A., Forecasting for Control and Proﬁt (Menlo Park,CA: Crisp Publications, 1997).CHAPTER 131. Block, Peter, Flawless Consulting (San Diego, CA: Pfeiffer, 1981).