Wp vendor selection


Published on

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Wp vendor selection

  1. 1. Vendor Selection Decisions 4300 Wilson Boulevard, Suite 350 Ar l i n g t o n , V i r g i n i a 2 2 2 0 3 703-399-2100 www.decisionlens.com
  2. 2. Selecting the best vendor is a difficult challenge. The stakes can be high, jobs may be on the line, and, adding to this complexity, multiple stake- holders from various factions within the organi- zation may demand that their input and needs be addressed throughout the selection process.This white paper covers the Overview of Vendor Selection Most organizations approach vendor selection (also called “source selection”) bybest-practice approach for setting up an evaluation board representing the internal “customers.” The require-vendor selection decisions ments are defined, and the vendors are asked to respond through an RFP. They areand is meant for anyone then evaluated against a checklist of requirements by each evaluator, and sheets with checkboxes showing “meets requirements” or “does not meet requirements” areinvolved in a vendor produced. The result is long lists of checkboxes at a detailed level, with little insightselection, product selection, into whether the vendors are capable of delivering to the key criteria the organizationsource selection, acquisi- must meet.tion plan, procurement or Oftentimes evaluators aren’t able to define a real difference between vendors using a requirements checkbox evaluation. Another challenge is that a vendor may be ancontract planning function. expert at producing proposals (or hire professional proposal writers) and know how to address a requirements checklist so that the company looks good, regardless of how well each requirement is met. After collating all of the requirements checkboxes and coming down to an even match between vendors, the decision then becomes one of cost. This is a flawed process that results in the following questions: • How can we define our objectives and make the tough trade-offs to determine which vendor we should choose? • How can we elicit input from and drive consensus across the evaluation team in an equitable manner? • Can we focus the discussion so that we are more efficient in the evaluation process, while also elevating us from the hidden agendas and inherent biases that others are bringing to the table? • How do we tease out the key differences among the vendors to really set them apart, enabling us to make the “best value” decision? • How do we know if the vendor’s price is justified by its benefits? • How can we best defend the decision with senior management and with the ven- dors? The members of an organization must work together to assess its relative value— strengths and weaknesses—across all of the objectives that it’s trying to achieve. The members need to develop the evaluation criteria as a team, carefully defining the meaning of each; decide which capabilities are relatively more important than others in a quantified manner; and finally evaluate the vendors according to strengths and weaknesses against each of the criteria, collecting all of the relevant comments and ensuring that the inputs of the evaluation board are captured and aligned in consensus rather than as checklists of “meets” or “does not meet.”www.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 2
  3. 3. T h e D e c isi o n L en s Ven d or Se lec tion Proce ssDecision Lens has developed a best-in-class vendor selection process that is used by Fortune 500 companies and agencies acrossfederal and state governments. This process was developed through an exhaustive 18-month effort to evaluate procurement bestpractices across industries, and was then implemented as the core of the “Acquisition Center of Excellence” for the U.S. intelligencecommunity. It has since been vetted out through hundreds of vendor selections and has proved exceptionally effective at defendingagainst protests from vendors.The rigorous process treats the vendors fairly, and each stage is entirely transparent in terms of the evaluators’ priorities andanalysis of vendors’ performances. Form Source Develop Develop Review Conduct Make Justify & Selection Statement Proposal & Assess Sensitivity Best-Value Document Evaluation of Work Evaluation Proposals Analysis Selection Decision Board (SOW) Can the evaluation board be grouped one another as possible—mutuallyStep 1: Form vendor into specific functional areas? exclusive and completely exhaustive.selection evaluation board Yes, you can divide the evaluation boardThe evaluation board for the selection is Step 3: Develop proposal into Tier 1 and Tier 2 groups. Tier 1set up with representation from across groups are at the higher level, evaluating evaluationthe organization. Often there are two The list of needs is used to develop spe- the rollup to the top criteria. The Tierdifferent evaluation boards created—a cific selection criteria, which are placed 2 groups are more functionally focusedTechnical Assessment evaluation board, into a hierarchy illustrating all of the groups that do the feed-in evaluations ofconsisting of expert users and opera- criteria and their relationships to one the proposals with specific strengths andtional managers who will evaluate the another. Criteria may include items such weaknesses in their areas of expertise.vendors’ relative technical merits, and a as technical approach, project man-Business Assessment evaluation board, Step 2: Develop state- agement plan, quality of data, systemwhich will evaluate the vendors’ contrac-tual terms and conditions, past business ment of work (SOW) application support and maintenance, The evaluation board defines a list of corporate quality/experience and ease ofconduct, etc. needs that are as independent from use. Criteria also may be subdivided into contributing subcriteria. (See Figure 1.) Figure 1: Hierarchy of criteriawww.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 3
  4. 4. The group defines the relative impor- want the technical evaluation and busi-tance of the criteria through a set of pair- ness terms evaluation to be untainted bywise comparisons, asking not only if one cost. Once you have rated each vendorcriterion is more important than another, against the criteria, you are presentedbut by how much? with the specific value rating of each vendor towards your needs.The Decision Lens tools can be used tobuild the criteria, define them and then You can then do a cost analysis by look-use them to make the decision. The group ing at the relative benefit scores vs. thecreates a value model using all of the costs of the vendors. An independentquantified criteria. decision is made as to the value/cost Figure 2: Cost vs. value tradeoff tradeoff, and the final cost analysis If Vendor #1 provides 30% more valueHow is cost treated? should not be formulaic; rather, you have and only 20% more cost, you make aCost is usually treated as an independent to assess the value of the final configura- cost/benefit tradeoff decision. This is tion vs. the cost. a much more powerful approach thanvariable. You first evaluate the vendors simply selecting a vendor because itbased on the benefits. You offers the lowest cost.How is risk evaluated? criteria. In other words, one criterion of How are the criteria weighted?Risk should not be included as a sepa- “past performance” may not get a weight The hierarchy is broken down into arate criterion. You will be assessing risk that accurately reflects its influence in series of judgments (pair-wise compari-throughout each criterion in the model. relation to the strengths and weaknesses. sons) at each level. (See Figure 3.)The risks will be captured in the weak- Instead, we recommend that past per- For example, you are asked the questionnesses that you evaluate for each vendor, formance be evaluated throughout the “Would you give more value to an vendoras risk for the specific criterion. decision. When participants evaluate the for its technical approach than for its vendors on their strengths and weak- project management plan?” You wouldHow is past performance evaluated? nesses under each criterion, the past per- then judge the two criteria being com-You could develop a criterion for “past formance would be calculated as pared on a scale ranging from 1 (equalperformance,” but past performance part of that. importance) to 9 (extreme differenceshould be evaluated across all of the in importance). Figure 3: Pair-wise comparisonswww.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 4
  5. 5. How is the overall value model that will be used to assess the vendors developed?The result of the pair-wise comparison process is a set of ratio scale priorities showing the value of each of the criterion to thedecision makers. (See Figure 4.)Decision Lens derives the priorities from the group members and tracks the consistency of their logic and how they applied theircomparisons. This is not forcing them to agree—disagreement is okay—but the process is done explicitly so that all of the judg-ments across the evaluation board are represented. Figure 4: Priority graph What drives the Request for What if there are optional Proposal (RFP)? When does the RFP requirements? Are those included process begin? in the upfront hierarchy model? Don’t send out the RFP until you have If all of the vendors are going to be developed your selection criteria. The addressing the options in the model, then criteria will guide the RFP response. You optional requirements should be kept in.“We needed a decision should include the criteria and defini- You’ll want to note how important that process that would tions in the RFP itself, but not option is in your overall consideration receive buy-in from the weights. as well. all stakeholders … What if the RFP has already How are rating scales developed? transmission, been sent? Once you have defined the relative impor- distribution and IS.” The model can be structured to reflect tance of all of the criteria, you create a what went into the RFP. Or if you have ratings scale for each criterion. These Bill Tsolias specific potential issues not addressed are rulers that will be used to evaluate Manager, Energy Management Group, in the RFP, you can develop a model that exactly how well each vendor’s solution National Grid exactly matches the RFP and a second addressed the criteria (the strengths and model to ensure you have captured all of weaknesses of the vendor). your considerations. The ratings scales are detailed; an “excellent” rating has a specific amount of major strengths, just as “poor” has a specific amount of major weaknesses.www.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 5
  6. 6. R a t i n g W eigh t De s cri pt ion Proposal Ratings, Adjectives, and DefinitionsExceptional 1 Offeror’s proposal demonstrates an exceptional understanding of the goals and objectives of the acquisition. One or more major strengths exist. No major weaknesses exist. Strengths significantly outweigh the weaknesses. Expected to cause no disruption in schedule, increase in cost, or degradation in performance. Will require no organizational emphasis and monitoring to overcome difficulties.Very Good .8 Offeror’s proposal demonstrates a very good level of understanding of the goals and objectives of the acquisition. Strengths outbalance weaknesses that exist. Any weaknesses are easily correctable. Expected to cause minimal disruption of schedule, increase in cost or degradation of performance. Will require a low level of organizational emphasis and monitoring to overcome difficulties.Good .5 Offeror’s proposal demonstrates a good level of understanding of the goals and objectives of the acquisition. There may be strengths or weaknesses or both. Weaknesses are not offset by strengths, but the weaknesses do not significantly detract from the offeror’s response. Expected to cause minimal to moderate disruption in schedule, increase in cost, or degradation in performance. Will require low to medium level of organizational emphasis and monitoring to overcome difficulties.Marginal .2 Offeror’s proposal demonstrates a marginal level of understanding of the goals and objectives of the acquisition. Weaknesses have been found that outbalance any strengths that exist. Weaknesses will usually be difficult to correct. Expected to cause moderate to high disruption in schedule, increase in cost, or degradation in performance. Will require medium to high organizational emphasis and monitoring to overcome difficulties.Unacceptable 0 Offeror’s proposal demonstrates a poor understanding of the goals and objectives of the acquisition. No major strengths exist, and one or more major weaknesses exist. Weaknesses clearly surpass any strengths. Weaknesses are expected to be very difficult to correct or are not correctable. This feature is so poorly understood and demonstrated that it presents an extremely high risk to the success of the program. Expected to cause significant, serious disruption in schedule, increase in cost, or degradation in performance. Will require significant or constant, high level of organizational emphasis and monitoring to overcome difficulties. (See Figures 5 and 6.) Figure 5: Example of ratings definitions used in ratings scales Figure 6: Ratings scaleswww.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 6
  7. 7. Step 4: Review andassess proposalsA quick read of the proposals is firstcompleted. This is designed to helpevaluators calibrate the range of vendorproposal methods and approaches.The discussion that results is veryimportant, and the loss of these com-ments could be a fatal error. Withouta structured process for gatheringcomments and evaluating them againstthe criteria, the group will not cometo a logical decision.With Decision Lens, the commentsare organized right in the software(see Figure 7) or can be captured in Figure 7: Documenting comments using Decision Lensan Excel file.How are a vendor’s strengths and weaknesses evaluated?Identify the specific strengths and weaknesses of the vendor under each criterion.Avoid looking at it from a general “meets the requirements” point of view. There aremajor strengths and minor strengths, major weaknesses and minor weaknesses. Risks “Applying Decision Len’sshould be translated into weaknesses. innovative process forEach of the participants will write an individual evaluation report for each of thevendor’s proposals under each of the criteria, listing the major and minor weaknesses decision making enabledthat he or she has found in the proposal. He or she will evaluate the strengths and our Technical Evaluationweaknesses (major and minor); include a “Clarification Request” for information in Committees (TEC) to focusthe proposal that is inadequate for evaluation; and identify errors, minor omissions,misunderstandings and contradictory statements. on key business and technical drivers in eval-What is a Consensus Evaluation Report (CER)? uating vendor proposals.The evaluation reports written by the participants are then brought together into a The committees were ableConsensus Evaluation Report (CER). The strengths and weaknesses for each vendorunder each criterion are made explicit in the consensus document and are then used to quickly and effectivelyto rate the vendors. discern major strengthsThe CER does not necessarily indicate complete agreement but instead indicates the and weaknesses in ven-preferred choice of the group. dor proposals and consis- tently apply their logic inHow are the vendors rated?Because each rating scale specifically states the strengths and weaknesses required selecting the best vendor.”to meet each rating level on the ruler (i.e. “Excellent” or “Poor”), the group is generally Lenetta McCampbellvery aligned in their ratings. In other words, two members of the evaluation board Senior Director,should not be opposite from one another on the scale if the strengths and weaknesses On Board Systems, Amtrakare being applied against the scales correctly. (See Figure 8.)If there is a member of the group who gives radically different ratings than the overallgroup, this discrepancy should be addressed. If the group cannot come to agreementon the ratings, then the difference of opinion must be documented as a comment onlyand the group should move on.www.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 7
  8. 8. Figure 8: Ratings In January of 2006, Harvard Business and political capital. If senior manage-Step 5: Conduct sensitivity Review published an entire issue on deci- ment does not agree with the evaluation,analysis sion making. One of the key criticisms the team should seek the reason for theSensitivity analysis is a powerful diag- was that in most decision-making pro- disagreement (the criteria used, the pri-nostic tool. It enables the team to test cesses, there is no means to introduce ority given to the criteria, the rating ofthe “what-ifs.” For example, what if the ad-hoc objections or changes to see how the vendors, etc.). The reason can thenimportance of the technical approach is the decision would be affected. be addressed and evaluated for furtherincreased? Does that change the priority refinement. (See Figure 9.)of the vendors? At what point does one The Decision Lens process enables you tovendor pass another? introduce changes throughout the model, providing key insights while saving time Figure 9: “What-if” sensitivity analysiswww.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 8
  9. 9. Step 6: Make best-value selectionOnce all of the ratings are complete for each vendor across the criteria, you will have a specific, quantified measure showing thevalue of each vendor against the objectives of your organization. (See Figure 10.)You then may evaluate the relative “value” score against costs, and can determine which vendor best meets your needs. For example,if you are more cost sensitive, then you may not be willing to make a tradeoff of 30% more benefits from one vendor to another forjust 10% more cost. If cost is less of an issue, you may make this tradeoff.What if multiple vendors are to be chosen?Decision Lens optimization capabilities allow you to enter a total cost figure for each vendor. Define your available budget as abudget pool in the optimizer, and then optimize to get the highest value for cost. The calculation performed in Decision Lens isto maximize “benefit priority/cost.” Figure 10: Ratings scoresheet suited for this process and has beenStep 7: Justify and Conclusion proven through numerous “best-value”Document Decision The vendor selection process is indeed vendor selection decisions to be a trueThe power of this process is that each challenging, but when rigor is applied best-practice approach.stage is explicitly and rigorously defined through each step of the process withand quantified. Specific judgments from a sound methodology and the ability toeach evaluation board member are used analyze both qualitative and quantitativeto drive the value of the criteria. Explicit evaluation criteria together, you arrivestrengths and weaknesses and comments at a robust and defensible decision. It isare used to drive the vendor ratings. The critical that a major vendor selection pro-group works collaboratively, but consen- cess be able to bring together all of thesus is not forced at any stage; transpar- stakeholders into a common collabora-ency and explicit judgment are required. tion that generates buy-in, and that theirAs such, any future review or audit judgments, comments and evaluationhas all of the necessary information to points be captured throughout the pro-debrief senior management, vendors and/ cess as well. Decision Lens is ideallyor your customers.www.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 9