This presentation explores how the user interaction design of online choice environments such as booking sites can impact consumer choice behavior. The format of this presentation is a narrative; a powerful way to present study results in an engaging, inspiring and informative way. The work is based on two academic papers and has been presented before at an Advanced Analytics (ADAN) workshop of the Market Research Society in London, November 10, 2016
7. Animal Welfare Conference
Princess Astrid received a letter from the Animal Welfare Association.
They invited her as a keynote speaker to their Global conference…
8. … which would take place in the beautiful city of Istanbul, Turkey.
9. Astrid happily accepted the invitation. She got to work on it straight away. She booked
her own travel as a part of the adventure. She consulted a booking site to find a hotel.
10. Call to action
Hotel Chain, Type
& Style
Distance to city
center
Placement of 50 hotel on the results page - top to bottom
Review Score on
cleanliness, staff and
facilities; result in a mean
score and a label
Including room price per
night
The booking site contained several helpful features to help her search a hotel room.
50 rooms were listed top to bottom and Astrid had to scroll to see the lower entries. Also, the brand
of the hotel chain, the type and style of rooms available were listed along with their distance to the
city center, review scores, room price, and a call to action that suggested that Astrid had to act fast.
12. For example, usually her personal assistant would take care of all of her travel.
Astrid did not know about things like hotel brands, room styles and types, …
13. Also, Astrid did not understand the meaning of the distance to a city center,
because her security detail would take care of travel on site, …
14. Her domestic staff did her cleaning so Astrid did not understand room reviews and
ratings on such concepts as “cleanliness, staff and facilities” …
15. Astrid was seriously challenged to book her flight and hotel and she ended up what
many visitors do: click on one of the first hotels that appeared in her booking app.
16. When the day had come,
Astrid went on the flight to Istanbul …
17. And she ended up in a terrible hotel,
which made her cry her heart out.
19. We meet Oscar, revenue manager at the booking site that Astrid used …
20. Astrid’s bad room experience is
a side effect of the way the site, and Oscar, work.
21. Oscar knows that consumers tend to gravitate
to the top for their choices and leave the rest alone ….
22. … so Oscar adapted the fee structure
to charge the highest commission fees for the top entries.
23. By playing the game of placing hotels strategically in top positions and
charging fat commission fees for it, Oscar produced record-breaking revenues.
24. It went sometimes at the expense of visitors like Astrid,
who pay the price if margin is spent on booking fees instead of room quality.
27. Charlotte likes working at the booking site because she likes traveling herself. She
wants to do well to her fellow travelers, and give them access to great rooms and deals.
28. When Charlotte started working on the interaction design of the booking site, she was
surprised at the rudimentary design. She wondered where the support functions were.
29. Filter functions
on price and ratings
Sort functions on
price and rating
She decided to give the site some extra functionality to reduce the cognitive burden on
the consumer: sort and filter functions. The goal was to drive choices to entries towards
the bottom of the page and thereby achieve a better distribution of choices.
30. Charlotte explained her work to Oscar who listed attentively first,
but once he understood what the sort and filter functionalities could do…
31. … he got upset because it would disrupt his revenue model.
35. Meanwhile, Astrid had gotten the hang of traveling.
She was now a member of a ring of celebrities of NGOs and good causes.
36. She was invited back to Istanbul for the
next installment of the animal welfare conference.
37. She got to work on it straight away.
Astrid went back the booking site, but now armed with more knowledge and skill.
38. She was surprised to learn that the booking site now contained functions that she used
to set bottom limits for prices and ratings. After all, she was still a spoilt princess.
39. Because she really wanted to explore the city, she now also explored other amenities
such as the distance to the city center.
44. Of those having
the functions
available
67%
uses sort and/or filter
functions at least once
across the four tasks
47%
uses the filter
function at least once
42%
uses filter on price
27%
uses filter on rating
40%
uses the sort
function at least once
34%
uses Sort on price
11%
uses Sort on rating
33%
does not use the
functions, not even once
To see how the sort and filter functions were used, Charlotte created a plot. She learned that 67% of her visitor
used the sort or filter functions; 33% did not. It was disappointing. 47% uses a filter function at least once; 40%
used the sort function, so filtering is more popular than sorting. 42% filtered on price versus 27% on rating, so it is
more popular to filter or sort on price than on rating. Charlotte did not know if these numbers are high or low; at
least she had some benchmark numbers and time will tell.
45. 0% 2% 4% 6% 8% 10% 12%
50
47
44
41
38
35
32
29
26
23
20
17
14
11
8
5
2
Likelihood of choosing a room
Positionontheresultspage Ideal situation
Ideal situation
Ideally, the choices of rooms would be uniformly distributed across the search result page; at 50 entries, each
entry would have an equal 2% chance of being chosen.
46. 0% 2% 4% 6% 8% 10% 12%
50
47
44
41
38
35
32
29
26
23
20
17
14
11
8
5
2
Likelihood of choosing a room
Positionontheresultspage Situation before redesign
S&F not available
Yet, before the change and the addition of the sort and filter functions, there were many people like Astrid and
choices on the booking site are skewed towards the top, and surprisingly, to the bottom. At the bottom there is
also an option to choose none of the hotel rooms. The visitor had to scroll all the way down to find it.
47. 0% 2% 4% 6% 8% 10% 12%
50
47
44
41
38
35
32
29
26
23
20
17
14
11
8
5
2
Likelihood of choosing a room
Positionontheresultpage Sort & filter made available
S&F available
To her positive surprise, Charlotte observed that after she had added the sort and filter function, the distribution of
choices was flatter than when they were not available.
48. 0% 2% 4% 6% 8% 10% 12%
50
47
44
41
38
35
32
29
26
23
20
17
14
11
8
5
2
Likelihood of choosing a room
Positionontheresultspage If sort & filter are used
S&F used
The effect was strongest among those who actually used the sort and filter function:
the ideal flat distribution was approximated!
49. 0% 2% 4% 6% 8% 10% 12%
50
47
44
41
38
35
32
29
26
23
20
17
14
11
8
5
2
Likelihood of choosing a room
Positionontheresultspage If sort & filter are not used
S&F not used
The effect was offset by those who did not use the sort and filter functions,
and whose choices skewed even more strongly towards the top entries.
50. 0% 2% 4% 6% 8% 10% 12%
50
48
46
44
42
40
38
36
34
32
30
28
26
24
22
20
18
16
14
12
10
8
6
4
2
Likelihood of choosing the room
Positionontheresultpage All compared
S&F not available
S&F used
S&F not used
S&F available
In all, Charlotte had gotten the effect that she was looking for:
the distribution of choices was flatter after the redesign than before it,
and the result of those using the functions, were encouraging enough to stay on this path.
51. Top box
satisfaction
if sort and
filter are ...
Not available:
56%
Available:
62%
Also, Charlotte was happy to learn that visitors were happier with the task after the redesign than before it. After
the redesign, there was 62% top-box satisfaction, versus 56% before the redesign.
53. Average room
price
if filter is...
Not available
€123
Available
€116
Not used
€126
Used
€102
He noticed that the average price of the room booked was down from 123 Euros to 116 Euros
after the redesign and the introduction of the sort and filter functions.
This was due to those using the functions at an average room price of 102 Euros.
54. Use of the none
option, if filter
was
Not available:
12%
Not used:
12%
Used:
17%
Also, the site had started to suffer more from what we may call the empty basket syndrome:
visitors leaving the site or the choice task without making a choice, or here, using the none option.
It was up from 12% when the filter function was not available or used, to 17% if the filter was used.
56. It is in Astrid’s favor that room choices were much better distributed across the results page if the functions were added, which could
drive up the real estate value of the page. Also, satisfaction was up which may mean that visitors are more likely to come back to the
site. That’s two points for Astrid. However, the average room price booked is going down and the number of people not chosing a
room, goes up. Both result in a significant loss of revenue. That’s two points for Oscar.
Flatter distribution of choices
Higher task satisfaction
Lower room prices
Higher drop-out rates
57. And because we don’t know if Astrid and other people are more likely to come back to the booking site,
we discount the last point for Charlotte. Based on this test, we declare Oscar the winner,
much to the dismay of Charlotte and Astrid.
Flatter distribution of choices
Higher task satisfaction
Lower room prices
Higher drop-out rates
59. By bringing it to you as a story, I may have been more engaging, you may have
understood the results of the study better, you may remember it better and you may be
more likely to act upon it. But that’s only a part of the deal here.
Dude it’s just a story
60. Stories are usually about change. It is embedded in storytelling formats: once upon a
time; every day, one day, and then…, until ... The change has happened and a new
situation kicks in. Stories are a great way to engage people and to inspire the change.
CHANGE
AHEAD
61. Change involves stakeholders, and not all stakeholders may take the change lightly because of different vested
interests. Research-based stories are a good way to explore and share the implications for the stakeholders and
support their informed decision making, taking everyone’s interests into account.
Stakeholder
the market
Stakeholder,
protagonist
Stakeholder
antagonist
62. Also, purchase environment matters a lot and can change the results of a study. In our case, it was the addition of
two user interface functions, sort and filter. It is a challenge to implicit assume of conjoint analysis that the
purchase environment does not have an impact. It shows that we should conduct our choice exercises in a virtual
environment representative of the future environment to improve external validity.
63. This is what we do in our initiatives in which we play and experiment with e-commerce environments.
We believe it is the future of conjoint to replicate consumer behavior in these kinds of environments.
64. Experiment with us
Gerard Loosschilder, Paolo Cordella,
Jean-Pierre van der Rest and Zvi Schwartz
gerard.loosschilder@gmail.com
www.studiogerart.com
Editor's Notes
Once upon a time in a country far from here …
… there was a Princess by the name of Astrid. Astrid was preparing for a lifetime of governing as Queen Astrid, …
… but for now, she liked to spend her time traveling and seeing the world …
Also, she was highly engaged in good causes and animal welfare.
One day, Princess Astrid got a surprise that really took her on the road.
She received a letter from the Animal Welfare Association. They had discovered Astrid’s knack for animal welfare, so they invited her as a keynote speaker to their Global conference.
… which would take place in the beautiful city of Istanbul, Turkey.
Astrid happily accepted the invitation and she got to work on it straight away.
Her mind was set on booking her own travel, because she felt it would be great part of the adventure. She started by consulting a booking site to look for a hotel.
The booking site contained several helpful features to help her search a hotel room. There were 50 rooms listed top to bottom and Astrid had to scroll to see the lower entries. Also, the brand of the hotel chain, the type and style of rooms available were listed along with their distance to the city center, review scores of other travelers, a room price, and a call to action that suggested that Astrid had to act fast.
Astrid was distressed. As a princess, Astrid got easily stuck with all of these options….
For example, because usually her personal assistant would take care of all of her travel, so Astrid did not know about things like hotel brands, room styles and types, …
Also, Astrid did not understand the meaning of the distance to a city center, because her security detail would take care of travel on site, …
Her domestic staff did her cleaning so Astrid did not understand room reviews and ratings on such concepts as “hygiene” …
So Astrid was seriously challenged to book her flight and hotel and she ended up what many people do: click on one of the first hotels that appeared in her booking app.
When the day had come, Astrid went on the flight to Istanbul …
And she ended up in a terrible hotel, which made her cry her heart out.
Meanwhile, at the booking site …
We meet Oscar, revenue manager at the booking site that Astrid used …
Astrid’s bad room experience is a side effect of the way the site, and Oscar, work.
Oscar knows that consumers tend to gravitate to the top for their choices and leave the rest alone ….
… so Oscar adapted the fee structure of his the booking site to charge the highest commission fees for the top entries
By playing the game of placing hotels strategically in top positions and charging fat commission fees for it, Oscar produced record-breaking revenues.
Sometimes at the expense of consumers like Astrid, who pay the price if margin is spent on booking fees instead of room quality.
Cut to: Meet Charlotte, interaction designer.
Charlotte recently started at the booking site as an interaction designer
Charlotte likes working at the booking site because she likes traveling herself. She wants to do well to her fellow travelers, and give them access to great rooms and deals.
When Charlotte started working on the interaction design of the booking site, she was surprised at the rudimentary functionality and she wondered where the support functions were.
She decided to give the site some extra functionality to reduce the cognitive burden on the consumer: sort and filter functions. The goal was to drive choices to entries towards the bottom of the page and thereby achieve a better distribution of choices.
Charlotte explained her work to Oscar who listed attentively first, but once he understood what the sort and filter functionalities could do…
… he got upset because it would disrupt his revenue model.
They got into a fight …
… but then they decided to be professional about it and they engaged in a multivariate A/B test to see the effects of the changes
One year later, back to Astrid.
Meanwhile, Astrid had gotten the hang of traveling. She was now a member of a ring of celebrities of NGOs and good causes.
She was invited back to Istanbul for the next installment of the animal welfare conference.
She got to work on it straight away. Astrid went back the booking site, but now armed with more knowledge and skill.
She was pleasantly surprised to learn that the booking site had changed. It now contained sort and filter functions that she used to set bottom limits for prices and ratings. After all, she was still a spoilt princess.
Because she really wanted to explore the city, she now also explored other amenities such as the distance to the city center.
Along with the brand, type and style of hotel.
So that by the time Astrid went to Istanbul …
She knew that she would end up in a nice hotel, and she was more satisfied with her choices and the way she had gotten there.
Meanwhile, back in the office, the results are in.
To see how the sort and filter functions were used, Charlotte built a plot. She learned that 67% of her visitor used the sort or filter functions; 33% did not. Disappointing. 47% uses a filter function at least once; 40% used the sort function, so filtering is more popular than sorting. 42% filtered on price versus 27% on rating, so it is more popular to filter or sort on price than on rating. Charlotte did not know if these numbers are high or low; at least she had some benchmark numbers and time will tell.
Ideally, the choices of rooms would be uniformly distributed across the search result page; at 50 entries, each entry would have a 2% chance of being chosen.
Yet, before the change and the addition of the sort and filter functions, there were many people like Astrid and choices on the booking site are skewed towards the top, and surprisingly, to the bottom. At the bottom there is also an option to choose none of the hotel rooms. The visitor had to scroll all the way down to find it.
To her positive surprise, Charlotte observed that after she had added the sort and filter function, the distribution of choices was flatter than when they were not available.
The effect was strongest among those who actually used the sort and filter function: the ideal flat distribution was approximated!
The effect was offset by those who did not use the sort and filter functions, and whose choices skewed even more strongly towards the top entries.
In all, Charlotte had gotten the effect that she was looking for: the distribution of choices was flatter after the redesign than before it, and the result of those using the functions, were encouraging enough to stay on this path.
Also, Charlotte was happy to learn that visitors were happier with the task after the redesign than before it. After the redesign, there was 62% top-box satisfaction, versus 56% before the redesign.
But then Oscar requested his own cut of the data, and he did strike back.
He noticed that the average price of the room booked was down from 123 Euros to 116 Euros after the redesign and the introduction of the sort and filter functions. This was due to those using the functions at an average room price of 102 Euros.
Also, the site had started to suffer more from what is called an empty basket syndrome: visitors leaving the site or the choice task without making a choice, or here, using the none option. It was up from 12% when the filter function was not available or used, to 17% if the filter was used.
So who did win?
It is in Astrid’s favor that room choices were much better distributed across the results page if the functions were added, driving up the real estate value of the page as a whole. Also, satisfaction was up which may mean that visitors are more likely to come back to the site. That’s two points for Astrid.
However, we also find that the average room price is going down and the number of people not chosing a room, goes up at the redesign. That’s two points for Oscar, and a significant loss of revenue.
And because we don’t know if Astrid and other people are more likely to come back to the booking site, we discount the last point for Charlotte. Based on this test, we had to declare Oscar the winner, much to the dismay of Astrid.
So my dear colleagues, what’s in it for you? Well, first of all you may have noticed that I told you a story instead of taking you through an academic-style presentation.
By bringing it to you as a story, I may have been more engaging, you may have understood the results of the study better, you may remember it better and you may be more likely to act upon it. But that’s only a part of the deal here.
Stories are usually about change. It is embedded in storytelling formats: once upon a time; every day, one day, and then…, until ... The change has happened and a new situation kicks in. Stories are a great way to enage people in the outcome of the change ahead.
Change involves stakeholders, and not all stakeholders may take the change lightly because of different vested interests. Research-based stories are a good way to explore and share the implications for the stakeholders and support their informed decision making, taking all interest into account.
Also, we noticed the purchase environment matters a lot and can change the results of a study. In our case, it was the addition of two user interface functions, sort and filter. It is a challenge to conjoint analysis, in which we implicitly assume that the purchase environment does not have an impact. It shows that we should conduct our choice exercises in a virtual environment that is representative of the future environment, to improve external validity.
This is what we do in our initiatives in which we play and experiment with e-commerce environments. We believe it is the future of conjoint to replicate consumer behavior in these kinds of e-commerce environments.
I would like invite you to experiment with us and to understand how environments impact choice behavior. Feel free to contact me. Call me, drop me an email or visit me at StudioGerART.com.