Okoli pawlowski2004


Published on

Published in: Education, Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Okoli pawlowski2004

  1. 1. Information & Management 42 (2004) 15–29 The Delphi method as a research tool: an example, design considerations and applications Chitu Okoli*, Suzanne D. Pawlowski a John Molson School of Business, Concordia University, GM 209-23, 1455 Boulevard de Maisonneuve Ouest, ´ ´ Montreal, Que., Canada H3G 1M8 b Department of Information Systems and Decision Sciences, Louisiana State University, Baton Rouge, LA, USA Accepted 8 November 2003 Available online 27 March 2004Abstract The Delphi method has proven a popular tool in information systems research for identifying and prioritizing issues formanagerial decision-making. However, many past studies have not adopted a systematic approach to conduct a Delphi study.This article provides rigorous guidelines for the process of selecting appropriate experts for the study and gives detailedprinciples for making design choices during the process that ensure a valid study. A detailed example of a study to identifykey factors affecting the diffusion of e-commerce in Sub-Saharan Africa illustrates the design choices that may be involved.We conclude with suggestions for theoretical applications.# 2004 Elsevier B.V. All rights reserved.Keywords: Delphi method; Group decision making; Research design; Strategic planning; Electronic commerce1. Introduction lines on how to conduct a rigorous Delphi study that identifies the most important issues of interest by The Delphi method has proven a popular tool in soliciting qualified experts. Second, we demonstrateinformation systems (IS) research [4,6,13,14,16,24, how to use a Delphi survey as a research tool to serve a25,35]. Citing ‘‘a lack of a definitive method for variety of different purposes in the theorizing process.conducting the research and a lack of statistical sup- Increasing the rigor will increase the confidence withport for the conclusions drawn,’’ Schmidt [34] pre- which researchers can use the results in subsequentsented a step-wise methodology for conducting such studies and managers can make decisions based onstudies. Building on the framework that Schmidt information gathered using these methods.developed, we offer two contributions towards Bricolage is a French term that means ‘‘to useincreasing the value of Delphi studies in investigating whatever resources and repertoire one has to performresearch questions. First, we fill in many details in the whatever task one faces’’ [40]. Characterizations ofcontext of Schmidt’s framework by providing guide- the research process as bricolage and the researcher as bricoleur [10] serve to remind us of the improvisation * Corresponding author. Tel.: þ1-514-848-2424x2967; and opportunism inherent in the research processfax: þ1-514-848-2824. and the need to put our research tools to multipleE-mail address: cokoli@jmsb.concordia.ca (C. Okoli). use. A third goal, then, is to encourage researchers to0378-7206/$ – see front matter # 2004 Elsevier B.V. All rights reserved.doi:10.1016/j.im.2003.11.002
  2. 2. 16 C. Okoli, S.D. Pawlowski / Information & Management 42 (2004) 15–29incorporate the Delphi method into their research method tailored to specific problem types and out-repertoire and to suggest some of the various ways come goals (see Linstone and Turoff for a descriptionthey could apply the method in the theorizing process. of the evolution of the method). One variant that has received widespread use is the ‘‘ranking-type’’ Delphi, used to develop group consensus about the relative2. Overview of the Delphi method importance of issues. Schmidt provides a detailed description of how to conduct this type of Delphi The Delphi method originated in a series of studies survey, including guidelines for data collection, datathat the RAND Corporation conducted in the 1950s. analysis (based on non-parametric statistical techni-The objective was to develop a technique to obtain the ques), and reporting of results.most reliable consensus of a group of experts [8]. Table 1 lists examples of studies that have used theWhile researchers have developed variations of the Delphi method in information systems research. Fore-method since its introduction, Linstone and Turoff casting and issue identification/prioritization represent[17] captured common characteristics in this descrip- one type of application of the method. The majority oftion: the Delphi efforts during the first decade were for pure forecasting, including both short- and long-range fore- Delphi may be characterized as a method for casts. Follow-up studies (e.g. [1,28]) have demon- structuring a group communication process so that strated the validity and long-range accuracy of the the process is effective in allowing a group of Delphi technique. While most forecasting studies use individuals, as a whole, to deal with a complex Delphi to surface a consensus opinion, others such as problem. To accomplish this ‘‘structured commu- the study by Kendall et al. [15] emphasize differences nication’’ there is provided: some feedback of of opinion in order to develop a set of alternative future individual contributions of information and knowl- scenarios. Concept/framework development repre- edge; some assessment of the group judgment or sents a second type of application of the Delphi view; some opportunity for individuals to revise method. These study designs typically involve a views; and some degree of anonymity for the two-step process beginning with identification/ela- individual responses. boration of a set of concepts followed by classifica- Delphi researchers employ this method primarily in tion/taxonomy development.cases where judgmental information is indispensable,and typically use a series of questionnaires inter-spersed with controlled opinion feedback [33]. A 3. Example research study design using thekey advantage of the approach is that it avoids direct Delphi methodconfrontation of the experts. Dalkey and Helmerobserved: Schmidt presented a guideline focusing on the major phases of the process and on analysis issues. [The controlled interaction] appears to be more However, the example we present in this paper focuses conducive to independent thought on the part of on perhaps the most important yet most neglected the experts and to aid them in the gradual formation aspect of the Delphi method—choosing appropriate of a considered opinion. Direct confrontation, on experts. This neglect is problematic, considering that the other hand, all too often induces the hasty most Delphi researchers characterize the technique as formulation of preconceived notions, an inclination a method for soliciting information from experts. We to close one’s mind to novel ideas, a tendency to based our guidelines primarily on those initially devel- defend a stand once taken, or, alternatively and oped by Delbecq et al. [9]. Setting these principles in sometimes alternately, a predisposition to be Schmidt’s framework, we provided a more complete swayed by persuasively stated opinions of others. guideline for a rigorous approach towards conducting Researchers have applied the Delphi method to a Delphi methods.wide variety of situations as a tool for expert problem Because of the detail of our instructions, it would besolving. They have also developed variations of the difficult to discuss them abstractly. Thus, rather than
  3. 3. C. Okoli, S.D. Pawlowski / Information & Management 42 (2004) 15–29 17Table 1Applications of the Delphi method in information systems researchApplication of the Delphi method Example studiesForecasting and issue Brancheau et al. [4,5]—Purpose: Identify the most critical issues facing IS executives in the coming identification/prioritization 3–5 years. Participants: Senior IS executives Czinkota and Ronkainen [7]—Purpose: Forecast changes in the international business environment over the next decade and the impact of these changes on corporate practices. Participants: Experts from policy, business and academic communities Hayne and Pollard [13]—Purpose: Identify the critical issues in IS in the coming 5 years perceived by Canadian IS executives and non-management IS personnel and compare to global study rankings. Participants: IS personnel Kendall et al. [15]—Purpose: Forecast the role of the systems analyst in the 21st century Lai and Chung [16]—Purpose: Identify a prioritized list of international data communications activities vital to multinational corporations in managing information exchanges for control and implementation of global business strategies. Participants: IS executives Viehland and Hughes [39]—Purpose: Compile a ranked list of 12 future scenarios related to the potential success of the Wireless Application Protocol (WAP). Participants: Industry and academic expertsConcept/framework development Bacon and Fitzgerald [2]—Purpose: Develop a framework of the main areas of the IS field. Participants: IS academics Holsapple and Joshi [14]—Purpose: Develop a descriptive framework of elemental knowledge manipulation activities. Participants: Researchers and practitioners in the knowledge management field Mulligan [24]—Purpose: Develop a capability-based typology of information technologies within the financial services industry. Participants: Members of 11 different organizations Nambisan et al. [25]—Purpose: Develop a conceptual taxonomy of organizational design actions— mechanisms to enhance technology users’ propensity to innovate in information technology. Participants: Practicing senior managers from diverse industries Schmidt et al. [35]—Purpose: Develop a ranked list of common risk factors for software projects as a foundation for theory building about IS project risk management. Participants: Three panels of experienced software project managers from Hong Kong, Finland and the United Statestrying to write generally for any Delphi investigation 3.1. Purpose of the research and research questionstopic, we have used a specific concrete researchproject (soliciting experts for critical success factors 3.1.1. Backgroundfor e-commerce in Sub-Saharan Africa (SSA)) as a Electronic commerce provides an important newparticular contextual illustration of how we would channel that connects the globe digitally for interna-apply our methodology. Although the guidelines are tional commercial transactions. However, when weset in a particular context, other Delphi researchers can examine the progress of electronic commerce in devel-readily adopt these principles for any other topic. We oping countries, we find a different story. As Petraz-illustrate basic steps in the approach as well as some of zini and Kibati [30] noted, ‘‘a closer look reveals greatthe choices and selection criteria that may be involved disparities between high- and low-income regions inat different points in the research design. For the terms of both internet hosts and users. More than 97%purposes of exposition, we present the description of all internet hosts are in developed countries that areof the design and rationale for design choices in the home to only 16% of the world’s population.’’ Numer-style of a research proposal. ous studies documenting the spread of the internet in
  4. 4. 18 C. Okoli, S.D. Pawlowski / Information Management 42 (2004) 15–29various parts of the world have highlighted the fact of non-governmental organizations (NGOs), as well asthat SSA is the region with the lowest level of eco- academics engaged in e-commerce diffusion researchnomic, technological, and internet development in the [19,38].world [27]. 3.2. Methodology3.1.2. Research questions and strategy (RQ) We wish to contribute to the growing body of work 3.2.1. Selection of the Delphi methodologyon the factors that affect the diffusion and expansion of Although we could conduct a traditional survey toinformation and communication technologies (ICTs), gather input from members of the major stakeholderparticularly the internet, into SSA (see Mbarika [18] groups concerning e-commerce infrastructure andfor a review of this). Specifically, the research study practices in SSA, we judged the Delphi method toinvestigates the following questions: be a stronger methodology for a rigorous query of experts and stakeholders. Table 2 compares and con-RQ1. What kinds of physical, economic, and socio- trasts the strengths and weaknesses of a Delphi study political infrastructure are necessary for the versus the traditional survey approach as a research establishment of viable e-commerce in SSA? strategy. In light of this comparison, we select theRQ2. What forms of e-commerce practices in SSA Delphi method for the following reasons: have the most potential for implementation in a period of 3 to 10 years for maximal economic 1. This study is an investigation of factors that would benefit? support e-commerce in SSA. This complex issueRQ3. What practicable solutions are available for requires knowledge from people who understand challenges in economic policy and manage- the different economic, social, and political issues rial strategy regarding viable and beneficial there. Thus, a Delphi study answers the study e-commerce in SSA? questions more appropriately. 2. A panel study most appropriately answers the We adopt a research program consisting of a three- research questions, rather than any individualstep strategy to investigate these questions. First, to expert’s responses. Delphi is an appropriate groupidentify factors that will answer the first two questions. method. Among other high-performing groupThis is largely subjective, as it depends on identifying decision analysis methods (such as nominal groupplausible factors. The second step involves quantita- technique and social judgment analysis [32]),tively testing the identified factors using a quasi- Delphi is desirable in that it does not require theexperimental design to verify if they are indeed experts to meet physically, which could bepertinent. This is more objective and should provide impractical for international experts.more confidence and solid directions for the third step: 3. Although there may be a relatively limited numberoffering practicable recommendations to address the of experts with knowledge about the researchthird research question. questions, the Delphi panel size requirements are modest, and it would be practical to solicit up to3.1.3. Scope of the initial study four panels from 10 to 18 members in size [29]. We will address the first step of the research strat- 4. The Delphi study is flexible in its design, andegy: subjectively, identifying pertinent factors related amenable to follow-up interviews. This permitsto e-commerce infrastructure and practices. Extant the collection of richer data leading to a deeperacademic literature provides some theoretical discus- understanding of the fundamental research ques-sion related to the factors of importance in determin- tions.ing e-commerce diffusion in SSA and their relative 5. We select the procedure for conducting Delphiimportance [12,20–23,37,41]. However, obtaining a studies outlined by Schmidt for the study becausemore comprehensive view necessitates perspectives it would serve the dual purpose of solicitingfrom all four major stakeholders in e-commerce diffu- opinions from experts and having them rank themsion: practitioners, government officials, and officials according to their importance.
  5. 5. C. Okoli, S.D. Pawlowski / Information Management 42 (2004) 15–29 19Table 2Comparison of traditional survey with Delphi methodEvaluation criteria Traditional survey Delphi studySummary of procedure The researchers design a questionnaire with All the questionnaire design issues of a survey also apply questions relevant to the issue of study. There are to a Delphi study. After the researchers design the numerous issues concerning validity of the questions questionnaire, they select an appropriate group of they must consider to develop a good survey. The experts who are qualified to answer the questions. The questionnaire can include questions that solicit researchers then administer the survey and analyze the quantitative or qualitative data, or both. The responses. Next, they design another survey based on the researchers decide on the population that the responses to the first one and readministers it, asking hypotheses apply to, and selects a random sample of respondents to revise their original responses and/or this population on whom to administer the survey. answer other questions based on group feedback from The respondents (who are a fraction of the selected the first survey. The researchers reiterate this process random sample due to non-response by some) fill out until the respondents reach a satisfactory degree of the survey and return it. The researchers then analyze consensus. The respondents are kept anonymous to each the usable responses to investigate the research other (though not to the researcher) throughout the questions. process.Representativeness Using statistical sampling techniques, the researchers The questions that a Delphi study investigates are those of of sample randomly select a sample that is representative of the high uncertainty and speculation. Thus, a general population of interest. population, or even a narrow subset of a general population, might not be sufficiently knowledgeable to answer the questions accurately. A Delphi study is a virtual panel of experts gathered to arrive at an answer to a difficult question. Thus, a Delphi study could be considered a type of virtual meeting or as a group decision technique, though it appears to be a complicated survey.Sample size for Because the goal is to generalize results to a larger The Delphi group size does not depend on statistical statistical power and population, the researchers need to select a sample power, but rather on group dynamics for arriving at significant findings size that is large enough to detect statistically consensus among experts. Thus, the literature significant effects in the population. Power analysis recommends 10–18 experts on a Delphi panel. is required to determine an appropriate sample size.Individual vs. The researchers average out individuals’ responses to Studies have consistently shown that for questions group response determine the average response for the sample, requiring expert judgment, the average of individual which they generalize to the relevant population. responses is inferior to the averages produced by group decision processes; research has explicitly shown that the Delphi method bears this out.Reliability and An important criterion for evaluating surveys is the Pretesting is also an important reliability assurance for response revision reliability of the measures. Researchers typically the Delphi method. However, test-retest reliability is not assure this by pretesting and by retesting to assure relevant, since researchers expect respondents to revise test-retest reliability. their responses.Construct validity Construct validity is assured by careful survey design In addition to what is required of a survey, the Delphi and by pretesting. method can employ further construct validation by asking experts to validate the researcher’s interpretation and categorization of the variables. The fact that Delphi is not anonymous (to the researcher) permits this validation step, unlike many surveys.Anonymity Respondents are almost always anonymous to Respondents are always anonymous to each other, but each other, and often anonymous to never anonymous to the researcher. This gives the the researcher. researchers more opportunity to follow up for clarifications and further qualitative data.Non-response issues Researchers need to investigate the possibility of Non-response is typically very low in Delphi surveys, non-response bias to ensure that the sample since most researchers have personally obtained remains representative of the population. assurances of participation.
  6. 6. 20 C. Okoli, S.D. Pawlowski / Information Management 42 (2004) 15–29Table 2 (Continued )Evaluation criteria Traditional survey Delphi studyAttrition effects For single surveys, attrition (participant drop-out) is Similar to non-response, attrition tends to be low in a non-issue. For multi-step repeated survey studies, Delphi studies, and the researchers usually can easily researchers should investigate attrition to assure that ascertain the cause by talking with the dropouts. it is random and non-systematic.Richness of data The richness of data depends on the form and In addition to the richness issues of traditional surveys, depth of the questions, and on the possibility of Delphi studies inherently provide richer data because of follow-up, such as interviews. Follow-up is often their multiple iterations and their response revision due limited when the researchers are unable to to feedback. Moreover, Delphi participants tend to be track respondents. open to follow-up interviews.3.2.2. Procedure for selecting experts Identifying experts: In alignment with the guide- Delbecq et al. provided detailed guidelines on how lines of Delbecq et al., the study will use a multiple-to solicit qualified experts for a nominal group tech- step iterative approach to identify the experts:nique study, making it clear that this procedure couldalso apply to a Delphi study. They described a rigorous Step 1. Prepare a knowledge resourceprocedure whose purpose was to ensure the identifica- nomination worksheet (KRNW). The purpose of thetion of relevant experts and gave them the opportunity Knowledge Resource Nomination Worksheet is toto participate in the study. A Delphi study does not help categorize the experts before identifying them,depend on a statistical sample that attempts to be in order to prevent overlooking any important class ofrepresentative of any population. It is a group decision experts. Table 3 displays an initial Worksheet for themechanism requiring qualified experts who have deep study. A research team of two academic researchersunderstanding of the issues. Therefore, one of the most and one practitioner, all of whom are familiar withcritical requirements is the selection of qualified issues concerning the Internet in SSA, will fill in theexperts. KRNW. We will identify the most appropriate Panel structure: We will divide experts into panels. disciplines, organizations, and literature that wouldTheir size and constitution depends on the nature of be most fruitful in identifying the world-class expertsthe research question and the dimensions along which on the Internet and e-commerce in SSA. Delbecq et al.the experts will probably vary. In this case, four emphasized that it is important not to write down anyrelevant categories of experts have important and specific names of experts at this stage. It is important tovaluable knowledge about e-commerce in SSA: aca- stay at a high level, first identifying classes of experts.demics, practitioners, government officials, and offi-cials of NGOs. These groups probably would have Step 2. Populating the KRNW with names.somewhat different perspectives. Since it is a goal to After the KRNW is completed, the following iterativeobtain a reasonable degree of consensus, it would be procedure will be used to populate the categories withbest to have panels that separate these groups. This actual names of potential experts for the Delphidesign also permits comparison of the perspectives of study. Each heading (disciplines, organizations, andthe different stakeholder groups. Following recom- literature) represents a different lens for identifyingmendations from Delphi literature, there will be 10 to and considering experts, and it is expected that there18 people in each panel. Within each panel, the goal is will be a very high degree of overlap of expert namesthat at least half the members actually work in SSA. between and within headings. However, this multiple-This structure will obtain a sufficient number of lens perspective is necessary to identify as manyperspectives from the ‘‘inside,’’ and we could perform experts as possible. For each category, we will firstanalyses to see if there are differences in perspectives go through our personal list of contacts to fit as manybetween respondents inside and outside. Fig. 1 out- names as we are able into the appropriate categories.lines the steps of our procedure for selecting experts. This is the baseline procedure. However, the personal
  7. 7. C. Okoli, S.D. Pawlowski / Information Management 42 (2004) 15–29 21 • Identify relevant disciplines or skills: academics, practitioners, Step 1: government officials, and officials of NGOs Prepare KRNW • Identify relevant organizations • Identify relevant academic and practitioner literature • Write in names of individuals in relevant disciplines or skills Step 2: • Write in names of individuals in relevant organizations Populate KRNW with names • Write in names of individuals from academic and practitioner literature Step 3: • Contact experts listed in KRNW Nominate additional experts • Ask contacts to nominate other experts Step 4: • Create four sub-lists, one for each discipline • Categorize experts according to appropriate list Rank experts • Rank experts within each list based on their qualifications • Invite experts for each panel, with the panels corresponding to each discipline Step 5: • Invite experts in the order of their ranking within their discipline sub- Invite experts list • Target size is 10-18 • Stop soliciting experts when each panel size is reached Fig. 1. Procedure for selecting experts in the example study.contacts of our research team, while extensive, are After beginning with personal contacts, we will golimited and biased to our personal networks. Thus, we through each heading in the KRNW and furtherwill follow the procedure detailed by Delbecq et al. to populate the lists according to the categories. Eachensure identification of the most qualified experts. heading has a different general strategy:Table 3Sample knowledge resource nomination worksheetDisciplines or skills Organizations Related literature Academic World Bank Academic: Journals list United Nations Economic Review of African Political Economy Practitioner Commission for Africa Journal of Management Information Internet Societies United Nations University Systems ITU sector members and associates Internet Societies in Africa European Journal of Information Systems Government official African governmental ministries of Journal of Global Information Management ITU Global Directory telecommunications Journal of Global Information Technology NGO official AFRIK-IT listserv Management Organizations list Electronic Journal of Information Systems in Developing Countries Practitioner: Communications of the ACM Africa Business Proceedings of ITU Telecom
  8. 8. 22 C. Okoli, S.D. Pawlowski / Information Management 42 (2004) 15–29 Disciplines or skills: Each category will require a for inclusion on the list. We will provide a briefdifferent approach for identifying experts: description of the Delphi study and explain that we have identified them as international experts on1. Academics: We will populate this list almost electronic commerce in SSA. Since this step does not entirely via a literature review of academic and solicit panelists for the final study, we will not invite practitioner journals under the ‘‘Related Litera- individuals to participate in the study. Rather, we ture’’ heading. will tell the experts and contacts that the researchers2. Practitioners: The first level of populating this list are currently gathering biographical information on will involve contacting the various national chap- experts. We will obtain as much biographical ters of the Internet Society in SSA. Currently, information as possible about their qualifications. We around 14 Sub-Saharan countries have chapters. will also track information about which contact refers The Internet Society chapters are expected to be a which other experts, to facilitate the follow-up needed place where the local ‘‘movers and shakers’’ of later. The first round of contacts aims primarily at Internet development congregate in their countries. extending the KRNW to ensure that it will include as3. Government: We will use the International Tele- many experts as can possibly be accessed. communication Union Global Directory to locate For the organizations and literature that the contacts contact persons in the national ministries of provide, we will follow the procedures detailed in Step telecommunications for each Sub-Saharan coun- 2 to identify specific names to place in the KRNW. try. Ministers (secretaries) of telecommunications Furthermore, we will need to obtain basic biographical are considered very qualified experts, being in information for every expert on the list in order to close contact with the most active and knowl- determine what qualifications they possess to make edgeable experts on the Internet in their countries. them experts. For example, the type of data recorded4. NGOs: Numerous NGOs focus on the development will include the number of papers published and of the Internet and ICTs in Africa. Two key places presentations made, the length of years of e-commerce to begin looking for lists (other than Web searches) practice in SSA, number of years of tenure in govern- are with the United Nations Economic Commission ment or NGO positions, etc. Adequate information on for Africa and the World Bank’s Information for each expert is needed in order to rank their expertise Development program. These are ideal central for the next step. At the end of this step, we expect to resources, both because they are active NGOs have a list of about 200 experts. themselves, they provide grants and other funding opportunities, and they sponsor conferences. Thus, Step 4. Ranking experts by qualifications. they are in touch with the various NGOs that At this step, we will compare the qualifications of approach them for resources and networking. In those on the large list of experts and rank them in addition, the ISWorld Developing Countries web- priority for invitation to the study. First, we will create site has an extensive list of relevant NGOs. the four sub-lists: practitioners, government officials, Organizations: The Web, e-mail, phone, fax, or NGO officials, and academics. Based on theirother pertinent methods will be the means of contact- qualifications, we will categorize experts into theing the identified organizations. The objective is to sub-lists. As experts take on multiple roles, we maycontact people in these organizations who are experts place an expert on more than one list. Next, eachthemselves, and who can provide additional contacts member of the research team will independently rankwithin and outside their own organizations. each sub-list, according to the person’s degree of Related literature: We will review the academic and qualification. In these independent rankings, thepractitioner literature to locate all articles concerning rankers will use ties. We will then come together tothe Internet or ICTs in SSA. reconcile the lists and create the four sub-lists as ranked by qualifications. Again, we will permit ties, Step 3. First-round contacts—nominations since we will invite multiple panelists. Finally, we willfor additional experts. At this point, we will contact invite the experts to participate in the study, stoppingthe identified experts and ask them to nominate others when we reach the required number.
  9. 9. C. Okoli, S.D. Pawlowski / Information Management 42 (2004) 15–29 Step 5. Inviting experts to the study. Based 3.2.3. Data collection and analysis methodon the rankings, we will create one panel for each ofthe four categories. Again, the target panel size is 18 Mechanism for administering the question-(10 minimum) with at least half the members having naires. The Delphi questionnaires will be adminis-worked within SSA. Choosing the maximum number tered using e-mail, fax, and the web. The panelists willprovides a buffer in case of attrition, even though be free to use whichever of these media was mostparticipant drop-out tends to be very low when convenient. The advantage of these ‘‘rapid’’ media isrespondents have verbally assured their participation that they speed up the turnaround time between(contrast this with Brancheau et al.). questionnaires. This is quite important using the We will contact each panelist and explain the Delphi method, which is notorious for the elapsedsubject of the study and the procedures required for time required for data collection. Delbecq et al.,it, including the commitment required. For this study, for example, estimated that the average Delphi studywe will ask panelists to commit to completing up to could take 45 days to 5 months. This assumes a scenariosix 15 min questionnaires and returning them within where the panelists are all in one country, and thethree days of receipt, for a total of one and a half hours researchers rely on the postal system to deliver andover a period of 1–3 months. We will impose a limit return the questionnaires. However, in this case, theof six questionnaires so as not to tax the participants, administration of the questionnaires is international.and yet give them an honest appraisal of their time Assuming that a panelist in SSA filled out and returnedcommitment. a questionnaire immediately (probably an overly opti- In this study, we will require participants to have mistic assumption), it would take about a month toaccess to e-mail, fax, or the Web for receiving and receive the completed questionnaire for analysis, beforereturning questionnaires. Normally, this might be a the next one could be sent out. Considering that theserious biasing factor. However, for a study employing researchers cannot send out the next questionnaire untilexperts on the use of e commerce, this is not an all the results for a panel are in, such a lag time would beunreasonable requirement. Following the recommen- unreasonably long.dation of Delbecq et al., the first questionnaire will be We will design three versions of each questionnaire:sent to each expert the same day they confirm their for e-mail, fax (which would be the most similar to adesire to participate. printed version), and the web. As the questionnaires For each sub-list, panelist solicitation will begin by will be designed carefully following the principles ofinviting the top nine experts who work within SSA survey design (see Dillman [11]), the results are not(half of our target of 18). Note that many of the experts expected to be significantly different from that bywill be based in North Africa and in South Africa, but mail.these experts will not be included in this target quota.We will invite the experts one at a time until they reach Administration procedure. Administration ofthe quota of nine. Next, we will invite the top nine the questionnaires will follow the procedure forexperts in the remaining sub-list, whether or not they ‘‘ranking-type’’ Delphi studies outlined by Schmidt.work in SSA. This will involve three general steps: (1) brainstorming The design of the five-step process will ensure the for important factors; (2) narrowing down the originalidentification and invitation of the most qualified list to the most important ones; and (3) ranking the listexperts available. of important factors. There are several incentives that may lead experts toparticipate in a Delphi study where they might decline General questionnaire design issues. Forto participate in other studies: (1) being chosen in a most of the design considerations, we will followdiverse but selective group; (2) the opportunity to learn the guidelines put forth by Delbecq et al. andfrom the consensus building; and (3) increasing their Dillman. First, considering that a Delphi study, withown visibility in their organization and outside. These its multiple steps and iterations, is considerably moreincentives can provide the strong inducements needed time-intensive for the respondents than a traditionalto attract busy experts. survey, one objective will be to ensure that no single
  10. 10. 24 C. Okoli, S.D. Pawlowski / Information Management 42 (2004) 15–29 • For this phase only, treat experts as individuals, not panels • Questionnaire 1: Ask experts to list relevant factors (not in any order) for Phase 1: infrastructure and expediency lists • Consolidate these two lists from all experts, regardless of panel Brainstorming • Remove exact duplicates, and unify terminology • Questionnaire 2: Send consolidated lists to experts for validation • Refine final version of consolidated lists • Henceforth treat experts as four distinct panels Phase 2: • Questionnaire 3: Send infrastructure and expediency lists to each expert Narrowing down • Each expert selects (not ranks) at least ten factors on each list • For each distinct panel, retain factors selected by over 50% of experts • Questionnaire 4: Ask experts to rank factors on each of their panel’s pared-down lists • Calculate mean rank for each item Phase 3: • Assess consensus for each list within each panel using Kendall’s W Ranking • Share feedback with each panelist and ask them to re-rank each list • Reiterate until panelists reach consensus or consensus plateaus • Final result is eight ranked lists, two for each panel Fig. 2. Delphi study administration process (adapted from [35]).questionnaire should take more than 30 min to to list at least six e-commerce applications, practices,complete. Second, considering the administration or features that practitioners could feasibly implementmechanism of simultaneously employing e-mail, fax, with beneficial effect within the next ten years in SSA.and Web versions of the survey, it is critical that the This question seeks to generate a second list of e-researchers carefully design the surveys to ensure that commerce practices, which we term the expediencythese three formats were equivalent. list. Unlike the first two questions, simply asking the Adapted from Schmidt et al., Fig. 2 outlines the experts their recommendations as items, then rankingprocess of administering the study. them will not appropriately address the third research question (RQ3). Recommendations are complex items3.2.4. Administration phase 1: brainstorming that result as a composite and synergistic conclusion from the findings of the other questions. Thus, to3.2.4.1. Questionnaire 1: Initial collection of factors. answer this question at this stage, a third DelphiWe will send the first questionnaire on the same day that question that is closely related to the other two willan expert agrees to serve on a Delphi panel using the be used: experts will be asked to offer a brief explana-expert’s preference of e-mail, fax, or Web. The initial tion (in two or three sentences for each factor) of thequestionnaire for a Delphi survey is very simple, since it importance of each factor they have listed for the firstconsists of an open-ended solicitation of ideas. The two questions. These explanations will serve the dualquestionnaire will ask three basic questions, each purpose of providing a qualitative empirical basis forcorresponding to one of the research questions. answering the third research question and helping us To address the first research question (RQ1), the to understand and reconcile the various experts’ fac-questionnaire will ask experts to list at least six tors. Moreover, the explanations will help to classifyimportant factors (see Schmidt) affecting the estab- the factors into categories and will provide clarifica-lishing and growth of business use of the internet in the tion for the next questionnaire, which renames andcountries of SSA. This question seeks to generate a list consolidates the factors.of infrastructure factors, which we refer to as the We will send this questionnaire to all the expertsinfrastructure list. To address the second research without considering their panel at this phase andquestion (RQ2), the questionnaire will ask experts analyze the results from all experts together. Various
  11. 11. C. Okoli, S.D. Pawlowski / Information Management 42 (2004) 15–29 25studies have found that in group decision making, their responses, the two lists and categorizations will beheterogeneous groups are more creative than homo- further refined.geneous ones. The study design purposefully groupsexperts based on their homogeneity. However, by 3.2.5. Phase 2: narrowing down factorsdividing the study into three phases, we will bypass The next two phases treat the experts as four distinctthis possibility of stifled creativity. The creative stage panels. In brief, panels will narrow down factors thatof the Delphi study occurs when soliciting factors reflect the perspectives of the constituent stakeholdersfrom participants, while the ranking and weighting (phase 2), and they will then facilitate consensusstages involve mainly judgmental opinion. This design (phase 3). In the second phase that narrows downdoes not use panels during the initial brainstorming the two lists of factors, our goal will be to understandstage, thus generating two lists of factors that represent the rating of importance of the factors based on thethe additive creativity of all the participants in the differing perspectives of various stakeholder groups.study, irrespective of their panels. Certain groups of experts might assess the problems In analyzing the responses from the first question- and opportunities for e-commerce in SSA somewhatnaire, we will first remove identical responses. At differently, and these differences might have impor-this time, we will record on the consolidated lists tant implications for government policy and manage-the number of panelists that initially suggests each rial action. Thus, the strategy is to have groups thatitem, and then group these factors conceptually into think similarly decide among themselves which fac-categories to make it easier for panelists to compre- tors are the most important, rather than trying tohend each list when returned for the next step. The reconcile significantly different perspectives.grouping will be simply for presentation purposesand not for analysis, and we will base the categor- Questionnaire 3: choosing most importantizations on our knowledge of the issues concerning factors. We will then present the complete consolidatede-commerce in SSA (such as political, cultural, and lists of items to each expert within each panel. The thirdeconomic issues). questionnaire will be randomly arranged to cancel out bias in the order of listing of the items. Each panelist3.2.4.2. Questionnaire 2: validation of categorized will be asked to select (not rank) at least 10 factors onlist of factors. Since the researchers, rather than the each list that they consider important to e-commerce inexperts, will perform the consolidation of the lists and SSA. When all of the panelists have returned theirgrouping into categories, before proceeding, we will responses, we will analyze each panel separately tosend a second questionnaire to validate the consolidated identify the factors selected by over 50% of the expertslists of factors. This questionnaire will list all the in the panel; we will retain these factors for that panel.consolidated factors obtained from the first This process will reduce the lists to a manageable size.questionnaire, grouped into categories. In addition to The target size for ranking will be no more than 20–23a brief, one-sentence explanation of each factor, an items.explanatory glossary will be included to define andexplain each factor, based on information provided by 3.2.6. Phase 3: ranking relevant factorsthe experts in the first questionnaire. Furthermore, we The goal of the final phase is to reach a consensus inwill give experts an exact copy of their responses to the the ranking of the relevant factors within each panel.first questionnaire. The second questionnaire will ask Studies have consistently found that it is more difficultexperts to (a) verify that we have correctly interpreted to reach consensus with Delphi groups than with onestheir responses and placed them in an appropriate that involve direct interaction between participants.category; and (b) verify and refine the categori- However, with a panel design it is less difficult tozations of the factors. According to Schmidt, attain consensus because the researchers deliberately‘‘without this step, there is no basis to claim that a select panel members for their homogeneity.valid, consolidated list has been produced.’’ At thistime, experts will be able to suggest additional items Questionnaire 4: ranking the chosen factors.that they might not have considered initially. Based on This phase of the procedure will involve each panel
  12. 12. 26 C. Okoli, S.D. Pawlowski / Information Management 42 (2004) 15–29separately ranking the factors on each of their distinct current level of consensus, based on the value of W (forpared-down lists. Each ranked list will reflect the example, ‘‘weak agreement’’); and (4) a paragraphpriority order for the specific panel. In this phase, summarizing the other panelists’ comments on whyeach expert will individually submit a rank ordering of they ranked that item as they did. Based on this, wethe items: one ordering for each of the two lists— will ask the panelists to revise their rankings for eachinfrastructure and expediency. The questionnaire also item, again asking them to explain their rankings andwill ask experts to submit comments explaining or revisions.justifying their rankings. We will reiterate this ranking process until we reach A review of the literature had not identified such an one of three stopping criteria: (1) W reaches a value ofexplanation of rankings in any Delphi study, although 0.7, indicating a satisfactory level of concordance. (2)Schmidt suggests soliciting helpful comments. Rohr- We reach the third iteration, which would be the sixthbaugh compared the Delphi methodology with a group questionnaire that a panelist received for this study,decision method based on social judgment analysis per our original promise. However, on the third itera-(SJA), which uses a formal graphical method to pre- tion, we will ask panelists if they were willing tosent the reasoning behind a panelist’s decisions to the continue iterating until they reached consensus. Ifother members of the panel. Although the results using enough panelists agree, we will continue the processSJA were not significantly more accurate, the groups until W rises to the desired level. (3) Followingreached a higher degree of consensus, since the mem- Schmidt’s suggestion, we will stop iterating if thebers understood one another’s reasoning. Based on mean rankings for two successive rounds is not sig-Rohrbaugh’s study, the panels should arrive at con- nificantly different. We could measure this differencesensus more quickly if provided with some sort of using the McNamar test, which ‘‘is typically used in afeedback about the panelists’ reasonings. repeated measures situation in which each subject’s When it comes to quantitatively determining the response is elicited twice (pre-post test)’’ [3].ranks of the items in the lists, Schmidt provided an At the end of this ranking phase, we will have eightexcellent and detailed guideline of principles to fol- ranked lists—two from each of the four panels—low, which we will use as the basis of our methodology representing the priorities that each of the panelshere. There are a number of different metrics for placed on various factors in affecting the practice ofmeasuring non-parametric rankings [36], but Ken- e-commerce in SSA. This rigorous process assuresdall’s W coefficient of concordance is widely recog- that the factors in the list are the most important, andnized as the best. The value of W ranges from 0 to 1, that the rankings are a valid indicator of the relativewith 0 indicating no consensus, and 1 indicating importance of the various factors. Based on theseperfect consensus between lists. Schmidt provided a results, we will be able to reassess our theoreticaltable for interpreting different values of W, with 0.7 observations from the literature and offer propositionsindicating strong agreement. After calculating the on expected relationships between factors in affectingconcordance within each panel, the W value suggests e-commerce in SSA.how to proceed in the ranking. A W value of 0.7 orgreater would indicate satisfactory agreement, and wewould consider the ranking phase completed. We 4. Conclusion: uses of the Delphi method forcould use mean rankings for each item to compute theory buildingthe final ranking for a completed panel. However, if W is less than 0.7, the ranking ques- The Delphi method is a versatile research tool thattionnaire must be resent to the members of that panel. researchers can employ at various points in theirEach reiteration would return the items for the panel, research. Use of the Delphi method for forecastinglisted in order of mean ranks. For each item, we will and issue identification/prioritization can be valuablegive the panelists the following information to help in the early stages, particularly in selecting the topicthem revise their rankings: (1) the mean rank of the and defining the research question(s). A major con-item for the panel; (2) the panelist’s ranking of the tribution of the MISRC/SIM studies of key managerialitem in the former round; (3) an indication of the IS concerns, published in MIS Quarterly, [5], for
  13. 13. C. Okoli, S.D. Pawlowski / Information Management 42 (2004) 15–29 27example, was to provide IS researchers with an under- have a wide range of experience, by inquiring aboutstanding of the most critical issues in IS management their experiences and opinions researchers signifi-as seen by practicing IS executives, thus contributing cantly extend the empirical observations upon whichto the relevance of IS research to practitioners. The their initial theory is based—thus strengthening theranked issues in these studies represented broad topic grounding of the theory and increasing the likelihoodareas (e.g., ‘‘Building a responsive IT infrastructure’’) that the resulting theory will hold across multiplefor future research. In Delphi studies where the ques- contexts and settings.tions posed to experts are narrower and more specific A third benefit to theory building derives fromin terms of subject, the resulting ranked lists can guide asking experts to justify their reasoning. This is anthe framing of specific questions. For example, the top optional feature of Delphi studies; the very first Delphiranked software project risk factor identified in the study conducted by Dalkey used it. Although notstudy by Schmidt et al.—‘‘Lack of top management many recent Delphi studies have taken advantage ofcommitment’’—could stimulate the formation of spe- this option, asking respondents to justify theircific research questions related to this particular issue. responses can be valuable aid to understanding theAnother risk factor identified in the Schmidt et al. study, causal relationships between factors, an understand-‘‘Conflict between user departments,’’ points to the ing that is necessary to build theory.body of theoretical work related to situations of conflict Fourth, a Delphi study can contribute to constructthat arise when groups seek to preserve their vested validity. Construct validity relies on a clear definitioninterests (e.g. [26,31]) as a useful lens for investigation of the construct. Delphi study designs, such as theof this issue. Table 4 summarizes some applications of example study, that ask participants to validate theirthe Delphi method in the research process. initial responses to make sure that that the researchers Researchers can use the Delphi method in a number understand the meanings of the list items submittedof ways related directly to theory building. First, the could contribute towards this goal. In addition, theranking of a Delphi study can be of value in the initial framing of construct definitions in alignment withstages of theory development—helping researchers to definitions in common use by practitioners also con-identify the variables of interest and generate proposi- tributes towards consistency in the understandings oftions. For example, the study described in the previous the construct by participants in future studies as wellsection would not only identify what factors experts as understandability by practitioners of the resultingperceive are important for e-commerce in Africa, but theory. One of the contributions of the Delphi study bywhich ones are viewed as more important than others. Holsapple and Joshi, for example, is that it provides aIn any study, researchers must select a parsimonious common language for discourse about knowledgelist of relevant variables. The experts’ rankings prior- management.itize the perceived effects of factors, and help the Although theory-building has not been the mainresearchers to select the factors with the strongest focus of much of the Delphi research, a carefullyeffects. designed study can not only be valuable for developing Second, additional advantages are possible up to the theory, but it can produce relevant theoretical research.extent of generalizability of resulting theory. Because Delphi studies, then, can contribute directly and imme-a Delphi study solicits information from experts who diately to both theory and practice. They build theory by the design and rigor of the study, whereas practi-Table 4 tioners will immediately have available to them lists ofApplications of the Delphi method in the research process prioritized critical factors, generated by experts, which they could apply to their individual situations. Identification of the research topic Specification of research question(s) The discussion in this section highlights the versa- Identification of a theoretical perspective for the research tility of the Delphi method as a research tool—a tool Selection of variables of interest/generation of propositions particularly well suited to new research areas and Preliminary identification of causal relationships exploratory studies. Through this discussion and Definition of constructs and creation of a common language detailed example of a Delphi study design, we hope for discourse to heighten awareness of the utility of the method for
  14. 14. 28 C. Okoli, S.D. Pawlowski / Information Management 42 (2004) 15–29different purposes the theory-building process. In [13] S. Hayne, C. Pollard, A comparative analysis of critical issues facing Canadian information systems personnel: a nationalconclusion, we encourage researchers to consider and global perspective, Information Management 38 (2),incorporating this tool in their personal repertoire of 2000, pp. 73–86.research methods so that it is available to them to use [14] P. Holsapple, K. Joshi, Knowledge manipulation activities:as needed to accomplish their research objectives. results of a Delphi study, Information Management 39 (6), 2002, pp. 477–490. [15] J.E. Kendall, K.E. Kendall, S. Smithson, I.O. Angell, SEER:Acknowledgements a divergent methodology applied to forecasting the future roles of the systems analyst, Human Systems Management 11 (3), 1992, pp. 123–135. We thank Victor Mbarika for his help in the e- [16] V. Lai, W. Chung, Managing international data communica-commerce study that we used as an illustrative frame- tions, Information Management 45 (3), 2002, pp. 89–93.work for this paper. We also thank Casey Cegielski for [17] H.A. Linstone, M. Turoff, The Delphi Method: Techniqueshis valuable references that gave us a literature base to and Applications, Addison-Wesley, London, 1975. [18] V.W.A. Mbarika, Africa’s Least Developed Countries’study the Delphi method. Teledensity Problems and Strategies, ME AGWECAMS ´ Publishers, Yaounde, Cameroon, 2001. [19] V.W.A. Mbarika, T.A. Byrd, P. McMullen, P. Musa,References Teledensity growth constraints and strategies for Africa’s LDCs: ‘‘Viagra’’ prescriptions or sustainable development [1] R.H. Ament, Comparison of Delphi forecasting studies in strategy? Journal of Global Information Technology Manage- 1964 and 1969, Futures March, 1970, p. 43. ment 5 (1), 2002, pp. 25–42. [2] D. Bacon, B. Fitzgerald, A systemic framework for the field [20] V.W.A. Mbarika, T.A. Byrd, J. Raymond, Growth of of information systems, Database for Advances in Informa- teledensity in least developed countries: need for a mitigated tion Systems 32 (2), 2001, pp. 46–67. euphoria, Journal of Global Information Management 10 (2), [3] N.H. Barry, Nonparametric statistics, http://faculty-staff.ou.e- 2002, pp. 14–27. du/B/Nancy.H.Barry-1/nonpar.html, 2002. [21] R. Montealegre, Implications of electronic commerce for [4] J.C. Brancheau, B.D. Janz, J.C. Wetherbe, Key issues in managers in less-developed countries, Information Technol- information systems management: 1994–95 SIM Delphi ogy for Development 7 (3), 1996, pp. 145–152. results, MIS Quarterly 20 (2), 1996, pp. 225–242. [22] R. Montealegre, Managing information technology in [5] J.C. Brancheau, J.C. Wetherbe, Key issues in information modernizing ‘‘against the odds’’: lessons from an organiza- systems management, MIS Quarterly 11 (1), 1987, pp. 23–45. tion in a less-developed country, Information Management [6] C.G. Cegielski, A Model of the Factors that Affect the 34, 1998, pp. 103–116. Integration of Emerging Information Technology Into Corpo- [23] R. Montealegre, Four visions of e-commerce in Latin rate Strategy, Unpublished Doctoral Dissertation, University America in the year 2010, Thunderbird International Business of Mississippi, 2001. Review 43 (6), 2001, pp. 717–735. [7] M.R. Czinkota, I.A. Ronkainen, International business and [24] P. Mulligan, Specification of a capability-based IT classifica- trade in the next decade: report from a Delphi study, Journal tion framework, Information Management 39 (8), 2002, pp. of International Business Studies 28 (4), 1997, pp. 827–844. 647–658. [8] N. Dalkey, O. Helmer, An experimental application of the [25] S. Nambisan, R. Agarwal, M. Tanniru, Organizational Delphi method to the use of experts, Management Science 9 mechanisms for enhancing user innovation in information (3), 1963, pp. 458–467. technology, MIS Quarterly 23 (3), 1999, pp. 365–395. [9] A.L. Delbecq, A.H. Van de Ven, D.H. Gustafson, Group [26] V.K. Narayanan, L. Fahey, The micro-politics of strategy Techniques for Program Planning: A Guide to Nominal Group formulation, Academy of Management Review 7, 1982, pp. and Delphi Processes, Scott, Foresman and Company, Glen- 25–34. view, Illinois, 1975. [27] M. Odedra, M. Lawrie, M. Bennet, S.E. Goodman, Sub-[10] N. Denzin, Y. Lincoln, Entering the field of qualitative Saharan Africa: A technological desert, Communications of research, in: N. Denzin, Y. Lincoln (Eds.), The Landscape of the ACM 36 (2), 1993, pp. 25–29. Qualitative Research, SAGE, Thousand Oaks, California, 1998, [28] R. Ono, D.J. Wedemeyer, Assessing the validity of the Delphi pp. 1–34. technique, Futures 26, 1994, pp. 289–305.[11] D.A. Dillman, Mail and Internet Surveys: The Tailored [29] S.J. Paliwoda, Predicting the future using Delphi, Manage- Design Method, Wiley, New York, 2000. ment Decision 21 (1), 1983, pp. 31–38.[12] A. Dutta, The physical infrastructure for electronic commerce [30] B. Petrazzini, M. Kibati, The Internet in developing countries, in developing nations: historical trends and the impact of Communications of the ACM 42 (6), 1999, pp. 31–36. privatization, International Journal of Electronic Commerce 2 [31] J. Pfeffer, Managing with Power: Politics and Influence in (1), 1997, pp. 61–83. Organizations, Harvard Business School Press, Boston, 1992.
  15. 15. C. Okoli, S.D. Pawlowski / Information Management 42 (2004) 15–29 29[32] J. Rohrbaugh, Improving the quality of group judgments: Chitu Okoli is an assistant professor in social judgment analysis and the Delphi technique, Organiza- the Department of Decision Sciences and tional Behavior and Human Performance 24, 1979, pp. 73–92. Management Information Systems of the[33] G. Rowe, G. Wright, F. Bolger, Delphi: a re-evaluation of John Molson School of Business at research and theory, Technological Forecasting and Social ´ Concordia University, Montreal, Canada. Change 39, 1991, pp. 235–251. He obtained a PhD in 2003 from the[34] R.C. Schmidt, Managing Delphi surveys using nonparametric Department of Information Systems and statistical techniques, Decision Sciences 28 (3), 1997, pp. Decision Sciences at Louisiana State 763–774. University, USA. He primarily researches[35] R.C. Schmidt, K. Lyytinen, M. Keil, P. Cule, Identifying applications of the Internet in developing software project risks: an international Delphi study, Journal countries, and strategically using the Internet for competitive of Management Information Systems 17 (4), 2001, pp. 5–36. advantage. In addition to the Delphi method, Chitu Okoli also uses[36] S. Siegel, N.J. Castellan Jr., Nonparametric Statistics for the quantitative surveys and stochastic simulation. Behavioral Sciences, McGraw-Hill, New York, 1988.[37] B. Travica, Diffusion of electronic commerce in developing Suzanne Pawlowski is an assistant countries: the case of Costa Rica, Journal of Global professor in the Department of Informa- Information Technology Management 5 (1), 2002, pp. 4–24. tion Systems and Decision Sciences at[38] UNECA, The process of developing national information and Louisiana State University. She holds a communications infrastructure (NICI) in Africa, http:// PhD degree in Computer Information www.uneca.org/adf99/nici.htm, 1999. Systems from Georgia State University[39] D. Viehland, J. Hughes, The future of the wireless application and MBA and BA degrees from the Uni- protocol, in: Proceedings of the Eighth Americas Conference versity of California, Berkeley. Research on Information Systems, Dallas, 2002, pp. 1883–1891. interests include enterprise systems and[40] K. Weick, Organizational redesign as improvisation, in: G. organizational learning, retention of IT Huber, W. Glick (Eds.), Organizational Change and Redesign: professionals, and collaborative visualization technologies. Ideas and Insights for Improving Performance, Oxford Scholarly publications include papers in ACM Computing University Press, New York, NY, 1993, pp. 346–379. Surveys, Communications of the ACM, IEEE Transactions on[41] P. Wolcott, L. Press, W. McHenry, S.E. Goodman, W. Foster, A Software Engineering, and Communications of AIS. Her previous framework for assessing the global diffusion of the Internet, work experience includes a 20-year career in IT at Lawrence Journal of the Association for Information Systems 2 (6), 2001, Livermore National Laboratory, including manager of application pp. 1–50. development.