Allowing for Uncertainty in Choice Experiments Dolores Garcia ∗ Departament dEconomia Aplicada Universitat de les Illes Balears E-07122 Palma de Mallorca, Spain e-mail: email@example.com David Hoyos Instituto de Economía Pública Edificio Central, 4ª. Planta Avda. Lehendakari Aguirre, 83 48015 Bilbao e-mail: firstname.lastname@example.org Pere Riera Departament d’Economia Aplicada Universitat Autònoma de Barcelona Edifici B, Campus UAB 08193 Bellaterra, Spain e-mail: email@example.com January, 2008 AbstractIn recent years choice modeling methods have gained popularity among the group ofvaluation techniques used to elicit the population’s preferences on non-market goods.Despite the large number of available applications, to our knowledge available studieshave not explicitly addressed the problem of WTP uncertainty. This paper is based onthe results of an exercise that combines a choice experiment and a contingent groupingformat. Follow-up questions were distributed in two sub-samples in order to captureeither the degree of certainty or the level of difficulty of the choice set, as perceived byrespondents. Also, the time spent to fulfill the tasks was accounted for, for each of thechoice sets faced by respondents. The main results are reported, comparing choice andcontingent grouping elicitation formats. Also, we investigate whether marginal valuesare affected by reported uncertainty. The exercise was applied in the Basque Countryand Navarra in order to value a set of key environmental attributes representative of thenatural area of Jaizkibel.JEL Classification: Q51∗ Corresponding author
1. Introduction The valuation of environmental goods has received a great deal of attention inthe last decades. Among the methods used for this purpose, stated preference (SP)techniques have become very popular, and numerous methodological advances havetaken place. One of the topics that has received attention is how uncertainty might affectestimated values. Thus, uncertainty might be present in the definition of the providedgood, the effects of a particular environmental policy, or even in the person’sknowledge about her true willingness to pay for the environmental change (Li andMattsson, 1995). Several studies have dealt with this topic in the CV literature (Berrenset al., 2002; Burton et al., 2003; Alberini et al., 2003; Vossler and Poe, 2005). The usualway of incorporating uncertainty into respondents’ answers is to allow them to state thedegree of certainty with which they answered. Depending on their answers, thenresponses are included or not in the estimation process that leads to the calculation ofmarginal values. This approach has been considered in this paper. A choice modeling frameworkhas been developed to value a set of environmental attributes characteristics of mountJaizkibel, a natural area on the Cantabrian Sea, near the French border (Bateman et al,2002). In particular, both a choice experiment and a contingent grouping exercise aredeveloped. Choice experiments have been widely applied for the valuation ofenvironmental amenities, and are probably the most popular among the group of choicemodeling methods. A contingent grouping approach was also used (Brey et al, 2005).Under the latter, individuals are asked to group the alternatives in sub-sets of thosewhich are better that the status quo or business-as-usual alternatives, and those whichare worse.
Follow-up questions were included to obtain information about the degree ofcertainty as perceived by respondents, or alternatively, about the difficulty whenresponding to the choice set scenario. We analyze the results of such questions, anddiscuss the extent to which every single respondent discriminates in their answers, howreported answers vary with the particular elicitation format used, and how marginalvalues are modified when uncertainty is accounted for. The structure of the paper is as follows. Next section briefly describes the siteobject of the valuation exercise, and the survey design. In section 3 the main results ofthe debriefing questions are presented, with three subsections: that devoted to thecertainty question, another one dedicated to the difficulty question, and a finalsubsection in which how times of responses vary under different scenarios are given.Section 4 first provides the results of a main effects conditional logit model, thendiscuss the new estimation outcomes when the result of the certainty question isinteracted with the cost attribute and included as explanatory variable. The final sectionincludes the main conclusions of the paper.2. Description of the exercise2.1. Description of the site and attributes The application was developed in order to value the relevant environmentalcharacteristics of mount Jaizkibel, a 2.400 hectares natural site that contains 15 zonesdeclared of high ecological interest by the European Union, situated in the municipalityof Pasaia (Basque Country). The environmental characteristics of this site might beaffected if the project to build a new port in the outside of the bay of Pasaia, under thehills of mount Jaizkibel, is undertaken. According to a recent study, the construction ofthe new seaport would provoke some critical impacts (Pozueta 2004), including cliffsdestruction, loss of vegetable cover and loss of seabed and local beaches, among other.
Detailed information about the environmental characteristics of Jaizkibel can be foundin Hoyos et al. (2008) In order to identify the key environmental attributes of mount Jaizkibel and thebusiness-as-usual levels of provision, interviews with experts and focus groups wereundertaken. It was concluded that the most important ones were landscape, flora,avifauna and seabed. In particular, landscape was defined as the percentage surfacefrom which today’s landscape could be seen in the future; flora was measured by thefuture level of protection of today’s population of armeria euskadiensis, an endemismof basque seacoast; avifauna, described in terms of the future level of protection oftoday’s population of lesser and peregrine falcon; and seabed, measured by the futurelevel of protection of today’s extension of red algae. Pictures were used to illustrate thedifferent levels of all four attributes (see figure 1). To decide upon the levels of the costattribute, focus groups together with a pilot survey including open-ended contingentvaluation questions were used. The payment consisted of an annual contribution to aFoundation exclusively dedicated to protecting mount Jaizkibel, that all Basque citizenswould be required to make. Table 1 summarizes the levels of all the attributes used. Table 1. Attributes and levels consideredAttribute LevelLandscape 40%* 60% 80% 100%Flora 50%* 70% 85% 100%Fauna 25%* 50% 75% 100%Seabed 50%* 70% 85% 100%Annual payment 5€ 10 € 15 € 20 € 30 € 50 € 100 € *BAU alternative levels
Combining all these attributes and levels, near two thousand differentcombinations were obtained (44x71). As it is usually done when the universe ofalternatives is very large, statistical design methods were used to simplify the choicesets construction (Louviere, Henser, and Swait, 2000). A main effects fractionalfactorial design with second order interactions reduced the number of alternatives to 96pairs of protection alternatives. The profiles were grouped in 24 blocks of four setscontaining two alternative protection programmes (programs A and B), plus thebusiness as usual option. Also, the “don’t know” option was included in order to avoidthe “yea saying” bias (Arrow et al. 1993). Then, each individual was shown 4 screens.Two of them asked for the most preferred alternative (choice), and the remaining tworequired their grouping as explained in the introductory section. The order in whichchoice and grouping were combined was randomly picked.2.2. Questionnaire and data collection A questionnaire was developed to simulate a market in which individuals wouldbe willing to choose among varying levels of protection of the attributes. Thequestionnaire was finally structured in three parts. The first part described the mainattributes and, the current situation of mount Jaizkibel and the potential future damagesto its environmental attributes. The second part (preference elicitation part) containedthe choice experiment and contingent grouping questions. An example of a cardincluding a choice set and the screen the interviewee saw is shown in figure A.1 in theannex. The last section collected some debriefing and socioeconomic questions,including those aimed at investigating about the degree of certainty in responses and theperceived level of difficulty when choosing or grouping alternatives. Thus, after each
set of alternatives had been shown, a follow-up question was added. A split sampleapproach was used. Half the sample was asked to state, in a scale from 1 to 7, how surethey felt when making their choice or groups. The other half was asked to state howdifficult the choice or grouping had been, again in a scale from 1 to 7. Interviews were conducted face-to-face at people’s homes, using laptopcomputers. Respondents could read the screens and listen to a recorded voice. Answerscould be typed in by themselves or dictated to the interviewer, at respondents’discretion. This way of administering the questionnaire allowed for the possibility ofcollecting information about the time individuals took to make their choices or groupingdecisions. The relevant population considered was the population from the BasqueAutonomous Community and Navarra in Spain as well as some French cities next to theSpanish border, accounting for 2.5 million people being at least 18 years old. The pilotwas conducted in October 2006, while the final survey was undertaken betweenNovember and December, 2006. A stratified random sample of 636 individuals wasselected from the relevant population. The strata used included age, gender and size ofthe town of residence, following official statistical information (EUSTAT). In eachlocation, the questionnaires were distributed using random survey routes.3. Descriptive statistics The sample accurately represented the population in terms of age, gender andincome levels. Even though the questionnaire included a number of questions asking forthe respondents’ view on the importance of the included attributes, their involvement inrecreational activities, and many other, here we mostly focus on the results of thefollow-up questions dealing with uncertainty and difficulty issues. For other outcomes,see Annex II of Hoyos et al. (2007).
3.1. Certainty in choices and grouping tasks The follow-up question asked the respondent how “sure” she felt with thechoice/grouping just provided, in a scale of 1 to 7, 7 representing the highest level ofsecurity. The means for the four follow-up questions resulted in values comprisedbetween 5.5 and 6, representing a medium-high level of confidence. The overall meanwas 5.75 (Table 2), and responses did not vary between the choice and the groupingelicitation formats. Dummy variables were created in order to capture whether the respondent hadfelt sure about the response (when answered 5, 6 or 7) or unsure (when answered 1,2, 3or 4). Table 2. Degree of certainty . Screens 1 to 4, and average. Mean (Std.Dev.) (values 1 to 7) % of unsure responses 1st screen 5.78 (1.57) 16.98% 2nd screen 5.63 (1.55) 22.28% 3rd screen 5.79 (1.44) 15.66% 4th screen 5.84 (1.44) 14.45% Average 5.76 (1.50) 18.30% When analyzing individual responses, it was found that a 54.61% of respondentsdid not change the value of certainty with the cards, that is, they picked the same valuefor all four choice sets. The remaining 45.39% did provide a different value dependingon the particular card just seen. Mean values when considering those who do and whodo not discriminate in their certainty answers are different. In average, those individuals
who do modify the degree of certainty in response show a smaller mean and a largerstandard deviation (table 3). Table 3. Degree of certainty according to individuals’ responses Mean (Std.Dev.) Respondent answered… (values 1 to 7) % of unsure responses Different scores 5.14 (1.62) 31’88% Same scores 6.26 (1.22) 6,02% When working on the experimental design, dominated profiles and profiles’ setsin which attributes’ levels did not vary tried to be eliminated or reduced (Huber andZwerina, 1996). However, a small number of choice sets remained in which the levelsof some attribute were the same. We describe these sets as easy. Although theyrepresent a small number, it results that the mean degree of certainty in responses ishigher (6), compared to difficult sets (5.75). Table 4. Degree of certainty depending on the difficulty of the choice set Mean (Std.Dev.) Profiles’ set (values 1 to 7) % of unsure responses Easy 6 (1.14) 17’64% Difficult 5.75 (1.51) 18,32%3.2. Difficulty of choices and grouping tasks As shown in the annex, some respondents faced follow-up questions after thechoice and grouping screens, asking for the degree of difficulty of the tasks of choosingor grouping the alternatives, and the answer again followed a closed format, in a scale
going from 1 to 7 (1 meaning the task had been very easy, 7 meaning it had been verydifficult). The main results are reported here. Table 5 summarizes the mean scores from the first to the fourth task. There waslittle variation in average means, results did not change either when grouping and choiceelicitation formats were compared. The average score resulted in 3.33, and so taskscould be considered as moderately easy. Answers were also recoded in a binary way: when the stated degree of difficultywas 1,2, 3 or 4, the task was considered not difficult, and difficult for scores above 4.The percentage of difficult was above 37% for the first choice set, and diminished as thesubsequent screens were shown, resulting in 31.68% for the last choice set. Table 5. Degree of difficulty. Screens 1 to 4, and average. Mean (Std.Dev.) (values 1 to 7) % of difficult responses 1st screen 3.41 (2.13) 37.12% 2nd screen 3.32 (2.08) 32.67% 3rd screen 3.39 (2.13) 32.17% 4th screen 3.27 (2.13) 31.68% Average 3.33 (2.12) 33.17% With respect to the certainty question, the percentage of individuals who did notkeep the same answer for all four choice sets was larger, achieving a 62.69%. Only theremaining 37.31% answered the same in all instances. The reported average degree ofdifficulty was Table 6. Degree of difficulty according to individuals’ responses Mean (Std.Dev.)
Respondent gave… (values 1 to 7) % of difficult responses Different scores 3.39 (1.93) 34.50% Same scores 3.19 (2.42) 32.30% Finally, it would be expected that easy choice sets, as described in the previoussubsection, would result easier for respondents. As table 7 shows, this was the case forchoice tasks; however, for contingent grouping tasks the average reported score resultedhigher for presumably easier tasks. Table 7. Degree of difficulty depending on the difficulty of the choice set Mean Profiles’ set (values 1 to 7) Choice experiment Contingent grouping Easy 3 3.53 Difficult 3.4 3.273.3. Times of responseAs explained in the previous section, we used a computer-aided questionnaire in whichresponses were automatically stored in a dataset, and in which the time spent inchoosing from or grouping the alternatives was stored.The average amount of time used to fulfill the task of choosing or grouping thealternatives was slightly above 27 seconds. Grouping the alternatives took an average of29’8 seconds, while for choices 24’8 seconds were required. The histogram in figure 1represents the percentage of responses for different time intervals.
Figure 1. Histogram for time intervals 45 40 35 30 25 20 15 10 5 0 <15 15-30 30-45 45-60 60-120 >2 seconds seconds seconds seconds seconds minuts % of cases In almost 40 per cent of the cases respondents answered in less than 15 seconds,and this proportion rises to more than 70 per cent for those under the 30 secondthreshold. More than 90 per cent of cases individuals invested less than 1 minute infulfilling the task of choosing or grouping. Also, the time of response clearly diminished as choice sets were presented. Forthe first choice of profiles shown, individuals took an average of 45 seconds to answer,while they invested about one third of such time for the last choice set (17 seconds). Thecorresponding standard deviations diminish as well. Table 8 summarizes some of thecommented results. Table 8. Time of response for choice/grouping screens 1 to 4, and average Mean (Std.Dev.) (figures in seconds) 1st screen 45.03 (51) 2nd screen 25.91 (32) 3rd screen 20.38 (21) 4th screen 17.09 (16)
Grouping 29.75 (39) Average 27.40 (34.27) Choices 24.84 (31) Combining the fact that respondents were progressively faster when providingtheir responses, and that they invested some more time in the grouping of alternatives,the times of response have been compared. Thus, average times of response have beenfound for grouping and choosing tasks, differentiating the cases in which individualsfirst faced the grouping screens and then the choice ones, or the other way around. Thefollowing table summarizes the results. Table 9. Time of response (seconds) depending on task and sequence of of tasks Mean (Standard deviation) Choice 1st, Grouping 2nd Grouping 1st, Choice 2nd Choice 32.26 (38) 16.79 (15) Grouping 20.58 (22) 39.22 (49) As expected, in all instances the first two tasks take more time than the secondones. The reduction of time is greater when the grouping screens appear first, withaverage times going from almost 40 seconds (for grouping) to around 17 (for choice).The shortest times correspond to the case in which choices appear in the second place(approximately 17 seconds), and the longest to the instance in which grouping screensgo first (39.22 seconds). Finally, times of response were compared taking into account whether theprofiles’ set was easy or not. There was a small difference of 1.31 seconds in favor of
difficult sets, which took less time. This somehow unexpected result is driven bydifferences in times for choice tasks: grouping tasks were performed quicker when theset of alternatives were “easy”, but with choice tasks it happened the other way around.4. Estimation results considering uncertainty In order to carry out the estimation of the values of the attributes at stake, thedatabase was depurated. In this section, only choice responses are considered. An 8 percent of interviewees declared not having understood the objective of the program topreserve mount Jaizkibel, and were not shown the choice sets. Similarly, protests to thevaluation questions took place for 171 individuals, representing the 26.8% of the totalsampled population. After all the aforementioned rejections, 415 observations out of theoriginal 637 remained valid for estimation purposes. Finally, since a split sampleprocedure was used to include follow-up questions, the number of valid observations inwhich certainty questions have been included diminishes to 304. As explained above, the attributes included in the choice sets were thepercentage values of landscape preservation, flora, avifauna and seabed. All of themwere treated as continuous variables. A conditional logit model was specified, includingthe attributes in the terms explained above, and considering all main effects. The modelincluding a constant for the business-as-usual alternative was also considered, but therestricted model was accepted according to the test of the likelihood ratio test. Table 10. Conditional logit model (model 1) Attribute Coefficient (t-ratios) Landscape 0.0246 (5.786) Flora 0.0160 (3.074) Avifauna 0.131 (3.622)
Seabed 0.0070 (1.813) Payment -0.0206 (-5.705) Log-likelihood function: -243.6517 Observations: 304 The results show that all variables resulted significant (for seabed, thecoefficient is significant at a 10% significance level), and the expected signs werefound. A second model specification including information on the reported certainty ofresponse was included, for the exact same observations. The reported scored wasrecoded into a binary variable capturing whether the choice had been made withcertainty or not, as explained above. Then, this variable was interacted with the costpayment. The objective was to see whether uncertainty could help explain choices, inthe first place, and see how the resulting marginal values would be affected. The resultsof the estimation appear in table 11. Table 11. Conditional logit model (model 2) considering uncertainty Attribute Coefficient (t-ratios) Landscape 0.0246 (5.796) Flora 0.0161 (3.087) Avifauna 0.0137 (3.774) Seabed 0.0072 (1.873) Payment -0.0185 (-5.034) (Payment)*(choice is uncertain) -0.0191 (-1.858) Log-likelihood function: -241.6445 Observations: 304
Again all attributes’ coefficients result significant, and so results the interactionincluding certainty of the choice made. The values corresponding to WTP for eachattribute, or implicit prices, could be calculated out of the coefficients in table 11. Theyare included in the following table. Table 12. Annual WTP for Jaizkibel attributes (in € of 2007/person-year) Model 1 Model 2 Attribute Marginal WTP Marginal WTP Certain Uncertain Landscape 1.194 1.330 0.654 Flora 0.777 0.870 0.428 Avifauna 0.636 0.741 0.364 Seabed 0.340 0.389 0.191 The values included in the table correspond to 1% marginal changes in theattribute’s preservation. The values corresponding to model 1 would be the resultingones when all choices are considered, and the information on the declared level ofcertainty in the response is ignored. When using the information on whether the choicewas a sure one or not, willingness to pay is affected by certainty. Thus, when choicesare certain, willingness to pay is larger for all four attributes. In other words, deletingthe “uncertain” responses from the dataset from which the models are estimated wouldlead to greater marginal WTPs.5. Conclusions This paper constitutes an attempt to include information on the degree ofcertainty of individuals, when making their choices, into the statistical analysis of dataresulting from a choice modeling exercise. The incipient results suggest that valuesmight result affected by feeling of confidence in the responses, and this poses thequestion of whether choices should be included when respondent do not feel sure about
them. Variation in resulting values might be of importance when calculating welfarechanges, especially at a more aggregate level. Other interesting issues have to do with the interpretation of uncertainty and howit might relate to other aspects of the exercise, including the level of difficulty of thetask at stake. The results show that respondents felt rather sure about their responses,and that they found the tasks easy enough not to be considered difficult (in average). Onthe other hand, the percentages of instances in which the tasks were fulfilled withuncertainty were considerably lower than the proportion of tasks considered difficult.While both types of responses could be related, the issue needs further investigation.The results were obtained from different sub-samples, too, so comparisons are notstraightforward. Likewise, the relationship between the times of response and the degreeof uncertainty in the response should be analyzed. The analysis here introduced had the drawback, however, that the final modelspecifications were made on a relatively low number of observations, for the sake ofcomparison, given that split samples were used for choice and contingent groupingformats, and also for certainty and difficulty questions. An immediate object of furtherresearch consists in extending the analysis by undertaking the joint estimation of choiceand grouping answers. Also, more accurate tests on the significance of the differences in resultingscores should be used to compare results dealing with uncertainty and difficulty. Also,the exploitation of the information provided by recording the times spent to respondingmight prove extremely useful to help understand the cognitive burden of alternativechoice modeling approaches such as contingent grouping and choice experiments.
AcknowledgementsThe authors acknowledge the financial support from the Department of Environment ofthe Basque Government, and from CICYT Project No. SEJ2004-00143/ECON andCEDEX project from the Spanish government.Bibiliography Alberini, A., K.Boyle and M.Welsh (2003) “Analysis of Contingent ValuationData with Multiple Bids and Response Options Allowing Respondents to ExpressUncertainty”, Journal of Environmental Economics and Management, vol. 45, pp. 40-62 Arrow K. et al. (1993). “Report of NOAA Panel on Contingent Valuation”. Bateman, I. J., Carson, R. T., Day, B., Hanemann, M., Hanley, N., Hett, T.,Jones-Lee, M., Loomes, G., Mourato, S., Özdemiroglu, E., Pearce, D. W., Sugden, R.,& Swanson, J. (2002). “Economic valuation with stated preference techniques: amanual” Cheltenham, UK: Edward Elgar. Berrens, R.P. et al. (2002) “Further Investigation of Voluntary Contribution inContingent Valuation: Fair Share, Time of Contribution, and Respondent Uncertainty”,Journal of Environmental Economics and Management, vol. 44, pp. 144-68. Brey R., O. Bergland O. and P. Riera (2005) “A contingent Grouping Approachfor Stated Preferences”, Working Paper #22/2005, Norwegian University of LifeSciences. Burton, A.C. et al. (2003) “An Experimental Investigation of Explanations forInconsistencies in Responses to Second Offers in Double Referenda”, Journal ofEnvironmental Economics and Management, vol. 46, pp. 472-89. Hausman J. and D. McFadden (1984) “Specification tests for the multinomiallogit model”, Econometrica, vol 52, pp. 1219-1240. Hoyos, D., P.Riera, M.C. Gallastegui, J. Fernández-Macho and D. Garcia (2008)“Choice modelling applied to assessing environmental impacts of transportinfrastructures: the case of Pasaia’s new seaport, Spain”. Manuscript Hoyos, D., P.Riera, M.C. Gallastegui, J. Fernández-Macho and D. Garcia (2007)“Informe final: Valoración Económica del entorno natural del monte Jaizkibel”. Finalreport for the Environmental Council of the Basque Government.
Huber, J., and K. Zwerina (1996). The importance in utility balance in efficientchoice designs. Journal of Marketing Research, 33, 307-317. Louviere J., D. A. Henser, and J. Swait (2000) “Stated choice methods: analysisand applications”. Cambridge University Press, Cambridge. Li, C. and L. Mattsson (1995) “Discrete Choice under Preference Uncertainty:An Improved Structural Model for Contingent Valuation”, Journal of EnvironmentalEconomics and Management, vol. 28, pp. 256-69. Pozueta J. (2004). “Estudio comparado de las alternativas de desarrollo delPuerto de Pasajes en relación con su grado de impacto medio-ambiental ysostenibilidad.” Departamento para el Desarrollo Sostenible. Diputación Foral deGipuzkoa. Vossler, C.A. and G.L. Poe (2005) “Analysis of Contingent Valuation Data withMultiple Bids and Response Options Allowing Respondents to Express Uncertainty: AComment”, Journal of Environmental Economics and Management, vol. 49, pp. 197-200.
Annex Figure A.1. Example of choice set in the valuation exercise
Figure A.2. Screen showing the questions about the degree of certainty in the response
Figure A.3. Screen showing the question about the degree of difficulty in the response