Prototype Evaluation as a Strategic Tool

'The Connect gallery – using prototype evaluation as a strategic tool in the design
process of interactives.'

Introduction
This paper introduces the prototyping process and its benefits in the design process of
interactives. Through a case study from National Museums Scotland, it exemplifies how
going to the trouble of prototyping can really pay off in terms of not ending up with an
interactive that no one likes or understands.

Background to Connect
Connect is a permanent interactive science and technology gallery, situated on the ground
floor of the National Museum of Scotland. The overall theme of the gallery is Creativity,
Discovery and Innovation, inspired by the science and technology icons selected for the
gallery. These objects are key examples of innovation and creativity in their time and
provide inspiration for discovery and innovation in the future. One such example is Dolly
the Sheep, a key icon of twenty-first century scientific achievement.

Within the gallery there are five sub topics: genetics (‘Me2’), robotics (‘Robots’), Space
(‘Blast Off’), energy and power (‘Power Up’) and transport (‘Move It’). A series of
interactives are co-located with the relevant objects for each topic, and provide visitors
with opportunities to interact physically, emotionally and mentally through challenges,
puzzles, and physical activities. The objects serve as a focus for the interactives.

There are a total of 19 interactives in the gallery, of which about half are mechanical and
half are ICT interactives, as well as touch screen information stations around the gallery.
Seven of the ICT interactives and the information stations, all of which were specially
designed for the gallery, underwent prototyping during the design process.

Figure 1: Connect gallery floor plan showing the five sub topics

The Prototyping Process
First, the ICT developers produced the first prototypes. These mainly just contained the
‘core interaction’ of the interactives, designed to get the main messages and navigation
across. Most graphics were rough ‘placeholders’ in case substantial redesign was needed,
and paper storyboards alone were not acceptable.

The next step was to test the prototypes in the museum to see how visitors responded to
them. I’ll say a bit more about this step in a moment, but basically it consisted of two parts
– observation, followed by a short interview.

The responses from the testing were then analysed and fed back to the developers. This
consisted of a written report and a face-to-face meeting. The reports detailed problems
with the functionality of the prototypes and suggestions for improvements, and also
covered any content or design issues that needed attention.

Finally, the developers made relevant changes to the prototypes according to the
feedback. Then the process started again.

Figure 2: The Prototyping Process

Each interactive went through 3 phases of evaluation before final delivery. The second
round of prototypes incorporated recommendations from the earlier feedback, and most
sections incorporated final graphics or graphic styles so the museum could confirm it was
happy with that. The final version was slightly different in that the testing with visitors step
was replaced with robustness testing, where basically all available staff tested the
interactives for bugs, typos and other weaknesses.

We worked to a very tight schedule. We had one week per prototype interactive for the
testing, and a further week to gather the results, with the feedback meetings with the
developers taking place two weeks after submission of the relevant prototypes. As I said
before, 7 interactives and the information stations were tested and evaluated, which made
a total of 8 test subjects. We worked in three 4-week blocks, testing two prototype
interactives per week. The ICT developers were paid by the museum in stages which tied
into the delivery phases of the prototypes for evaluation.

Testing the Prototype Interactives

I want to say a little bit more about the step where we tested the prototypes with visitors.
Like I said, it consisted of two parts. After approaching visitors in the museum and inviting
them to take part in testing, we observed them trying out the prototype interactives to find
out what seemed to be working or not working. Specific things that the observers were
looking out for included whether visitors seemed to have any problems with the
instructions, understanding what they were meant to do, as well as difficulties with the
navigation.

The observation was followed by a short interview with the relevant visitor. If it was a group
of visitors being observed, then just one member of the group was interviewed. The
sample size for each prototype interactive was 20 groups/ interviewees. Questions that
were asked in the interviews included:
- What did you think the interactive was about?
- Did you find out anything new you didn’t know before?
- How would you describe this interactive to a friend? and
- What did you like most/ least about the interactive?
as well as practical things such as how easy or difficult they found the language to
understand, and their opinion on the length of the interactives.

1. Invite visitors to take part in testing

2. Observe visitors using prototype interactives

3. Short interview with visitors to get responses/ reactions on prototype interactives

Figure 3: Steps in testing the prototype interactives

Why Prototype?
So what are the benefits of prototyping? Well, it gives developers a chance to experiment
with new and different designs, and it formalises the process by which the museum feeds
back to the developers – feedback is sent a fixed number of times and in a written
document everyone agrees on.

More importantly, the museum can ensure that visitors
- understand the interactive (understanding)
- are able to operate the interactive (ergonomics), and
- enjoy the interactives (enjoyment).

Case Study: GM Foods
So, to end with I have an case study from our testing for the Connect gallery that
exemplifies how going to the trouble of prototyping can really pay off in terms of not ending
up with an interactive that no one likes or understands.

The ICT interactive I would like to present to you focuses on the GM food debate. It has a
17’’ touch screen, however, the prototype was operated by mouse. The interactive has an
introduction explaining GM, followed by a game-play based challenge, and then an
opportunity to vote on some topical GM issues at the end. The key messages the
interactive is intended to convey are the many different purposes of GM technology, and
that society must decide how best to use if for the future, i.e. a focus on the ‘pros and cons’
of GM and the surrounding debate.

The target audience for the interactive are schools and families with children age 11 and
above, as well as independent adults. I would like to focus on the middle part of the
interactive, the game-play based challenge1.

In the first prototype, visitors were given three scenarios to choose from – to create foods
that contain medicine, crops resistant to weedkiller, or foods that last longer on the
supermarket shelf. On choosing a scenario, visitors got some more information about that
scenario and were then invited to accept their challenge. On accepting their challenge,
visitors then had to choose a plant to modify. When individual plants were selected, three
characteristics popped up for each plant that visitors had to ‘accept’ or ‘decline’, based on
the information they had been given on the previous screen. Once visitors had identified
the correct plant for their challenge they could then modify it.

Feedback from the testing of the first prototype

So, did it work? Overall visitors seemed to like the interactive and enjoy the experience,
but that seemed to be mainly restricted to the colourful graphics, and perhaps the novelty
of being involved in the testing and getting a special preview. The understanding and
ergonomics showed a different picture.

1 Please note that the screenshots from the case study prototypes referred to in the original presentation at
the VSG Summer School were not available for inclusion in this paper.

As I already said, most of the visitors seemed to enjoy themselves, although half of them
found something they disliked. In terms of understanding of the key messages, only 55%
mentioned GM or genetic modification, and no one mentioned anything about choice or
debate. 40% found the language either quite difficult or very difficult, and overall 30% said
the interactive had taught them a lot.

The most significant result was in terms of ergonomics – 60% had a problem using the
interactive, including initial difficulties in understanding the challenge, to not really
understanding the interactive at all or why they were getting answers right or wrong.

Prototype 1 Prototype 2
Enjoyment 50% found something they
disliked
Understanding 55% mentioned GM or
genetic modification
0% mentioned choice or
debate
40% found the language
quite or very difficult
30% said it taught them a lot
Ergonomics 60% had a problem using
the interactive
Figure 4: Feedback from the testing of the first prototype

Prototype redesign

We decided to throw the baby out with the bathwater and completely revisit the game-play
of this interactive, concentrating more on ‘choice’ and ‘debate’.

The new prototype showed a card game after the introductory screens. It gives the visitor
10 pairs of picture cards they have to turn over two at a time and try to match. On correctly
matching a pair, a fact about GM food appears on screen relating to the matched picture.
The picture is then assigned to either a ‘Pro’ or ‘Contra’ column (unfortunately this screen
shot does not show the columns as no cards have been turned over yet).

When we tested this new prototype, we found a great improvement in both understanding
and ergonomics.

Only 45% found something they disliked, so slightly fewer than before. In terms of
understanding, however, the results differed significantly. All visitors mentioned GM or
genetic engineering, and half of them mentioned it was about the pros and cons of GM
food. Only 10% found the language quite difficult, which included foreign visitors whose
first language was not English. No one found the language very difficult. A total 80% said

they had learned something new. In terms of ergonomics, the results also differed
significantly – only 10% had any difficulties using the interactive.

Prototype 1 Prototype 2
Enjoyment 50% found something they 45% found something they
disliked disliked
Understanding 55% mentioned GM or 100% mentioned GM or
genetic modification genetic modification
0% mentioned choice or 50% mentioned it was about
debate pros and cons of GM foods
40% found the language 10% found the language
quite or very difficult quite difficult
30% said it taught them a lot 80% said they had learned
something new
Ergonomics 60% had a problem using 10% had a problem using
the interactive the interactive
Figure 5: Feedback from the testing of the second prototype

Conclusion
However, we still hadn’t got it quite right yet. Half of the visitors felt the game was too long.
And half of the visitors clicked on the cards in the game before the cards were ready to
turn over. We therefore recommended to the developers that the number of card pairs
should be reduced, and to make the cards react quicker.

The final version has 8 pairs of picture cards instead of 10, and you can also see the ‘pros
and cons’ columns in this screen shot, which we renamed ‘for and against’. The cards also
respond much faster, which I can’t show you here but if you’re ever in Edinburgh I
recommend you visit the Connect gallery and try it out for yourself. Thank you.

Jenni Fuchs
Visitor Studies Officer
National Museums Scotland
June 2007

Prototype Evaluation as a Strategic Tool

More Related Content

Similar to Prototype Evaluation as a Strategic Tool

Recently uploaded

Prototype Evaluation as a Strategic Tool