This document is a paper submitted by Stephen Norman for his MSc in User Experience Design program. The paper discusses and compares three user experience evaluation methods - ethnographic field studies, card sorting, and A/B testing. It presents these methods within a three dimensional framework of qualitative vs. quantitative, attitudinal vs. behavioral, and context of use. The paper then discusses how research goals should align with different stages of a product's development lifecycle and be informed by the appropriate evaluation methods.

1. Assignment Cover Sheet
MSc in User Experience Design
Student Name: Stephen Norman
Student Number: N00147768
Programme: MSc UX Design
Year of Programme: 2015/2016
Module Name: User Research and Usability
Assignment: Understanding Users and Tasks
Assignment Deadline: 08/01/2015
I declare that that this submission is my own work. Where I have read, consulted and
used the work of others I have acknowledged this in the text.
Signature: Stephen Norman Date: 08/01/2016

2. Understanding Users and Tasks
By Stephen Norman
Introduction
This paper is set out to discuss three user experience evaluation methods in detail. These are
Ethnographic Field Studies, Card Sorting, and A/B Testing. Before going in to detail on these
methods, the paper aims to present the reader with a full background of how these methods
are categorised within a three dimensional matrix. This will allow the reader to understand
why the author has decided to use these methods for different stages of the development
lifecycle. From this understanding the discussion will continue on to the types of goals set
during a project. How they change depending on the current stage of the project as well how
these goals are informed by the data gathering requirements. Unfortunately, this paper will
not discuss the eighty or so other research methods which are available and are just as equally
useful at collecting qualitative and qualitative user information. The ones selected are merely
a snippet, a view in to the user research field. It is hoped that the presentation of these
studies will give the reader a fuller and better understanding of the methods presented.
The Three Dimensions
Each research method can be categorised in to one of three dimensions, each with its own
style of measuring user information. The three plains which will be discussed are Attitudinal
vs. Behavioural, Qualitative vs. Quantitative, and Context of Use (See Fig. 1). These each vary
in their own importance, as each dimension will enable data collection in a specific way.
Figure 1. A landscape of User research methods.

3. Attitudinal vs. Behavioural dimensions
These two dimensions can be broken down to simply “what are the users thinking”;
attitudinal, versus “what are they doing”; behavioural. Attitudinal studies may be used in
exploratory research, such as surveys, interviews, and focus groups. These will give the
designers a better understanding of how their designs are perceived. This is not to say that
these types of studies cannot be used elsewhere in the lifecycle. In fact, post implementation
studies using these methods can contribute just a must valuable data and result in either
answering the project goals or not. It must be considered that what users think does not
always correlate to what they actually do. Behavioural studies are a more hands on approach
to research. These studies record the how the user behaves during a specific context, and
research is conducted based on their interactions. A very common and widely used
behavioural study is A/B Testing which collects quantitative user data; to be discussed later.
Other behavioural methods such as journey maps, and contextual inquires or field studies will
provide the same amount of both qualitative and quantitative information depending on their
use in a project.
Figure 2. Questions answered by research methods across the landscape.
Qualitative vs. Quantitative
Qualitative and quantitative methods can be explained in many ways. Rohrer (2014) discusses
that these methods differ in the way which the data is collected. While Davies (2014), states
that one approach is scientific; quantitative, and qualitative as a natural approach bringing
the world of context in to the fold. These are accurate in their own right. Quantitative is an
indirect and scientific approach to research collection. It uses large amounts scientific data
to answer goals. As mentioned above, A/B testing would be a quantitative study. Whereas
qualitative would be direct observations of the user such as usability studies, contextual

4. inquiries, card sorting and so forth. Qualitative types of research are based around smaller
numbers of users and are much better suited for answering the “why”. For example, a
quantitative study may provide a calculated percentage of users exiting a page within five
seconds, but not tell why they left. Whereas a qualitative study of the same page will tell you
what problems the user encountered, why, and their reason for leaving. These methods can
are chosen as per project goal requirements and will be discussed in depth later.
Context of Use
Context is a term described and defined differently depending on the person asked. For
example, Trivedi (2012) discusses context as a location, time of day, season, temperature. It
can also be described as the current state of the user, whether it’s physical, social or
emotional. Simply put “context is anything which has an effect on the human behaviour” –
Trivedi (2012). This qualitative data can yield significant rich information about a product,
and also unearth unforeseen issues. This is especially true around user artefacts. Artefacts
are described as any tool or object the user uses during the study. For example, it could be
that during the testing process the user picks up a pencil and jots down some information.
These are artefacts of the study and must be then considered in the research and design
process.
These dimensions are determined by the research goals of the project. Projects will use a
variety of these during its lifecycle. For example, an attitudinal quantitative study such as a
survey may be used at the initiation of project as exploratory research, then once the
development has reach a design phase a more qualitative approach both using behavioural
and context of use studies to gather the further details as to why.
The Selected Methods
Understanding the dimensions above will help determine the optimal method of conducting
research. As stated earlier there are many different types of research methods. For the
purpose of this paper, three have been chosen; Ethnography Field Studies, Card Sorting and
A/B Testing. Their characteristics and optimal range of use will be discussed and compared
below;
Ethnography Field Studies
Ethnography field studies are a qualitative research method developed by anthropologist
studying non-western cultures Jones (n.d.). It has become commonplace to be used in other
fields, such as human computer interaction. Ethnography is the study of how users interact
within their own environment while taking in all cultural and social factors. Boehm (2010)

5. argues that ethnography field studies should not be associated with the term contextual
inquiry. As contextual inquiries are more focused on the human-computer interface rather
than the social interactions. However, this is matter of opinion, and a contextual inquiry could
easily include ethnographic studies. Although it could be said that when a field study is
ethnographic that it correlates closer to the attitudinal dimension, where as a contextual field
study would be more behavioural. Again the overall purpose of this type of research is get
into the minds of the users, understand their perspectives and point of view while observing
their interactions with the system and each other.
Before any studies are granted, both parties must agree the terms of research. For instance,
materials may be sensitive or classified. In this case a non-disclosure agreement (NDA) is
signed. Only then can research begin, however it may limit the availability publishing the
study.
Another form of field study is covert research. However, this is very sensitive area as some
ethical codes strictly forbid this kind research. Jones (n.d.) discusses that where access is
granted, it is recommended that participants be informed and retrospective consent be
obtained. A major consideration is the protection of the individual. When required the study
should guarantee anonymity, and should be agreed well before the results are published.
Figure 3. An example of an Open Card Sort where the participant was asked to organize topics then classify each into groups
which they'd name.
Card Sorting
When compared to ethnography, card sorting shares similarities in the way that the exercise
is conducted. Card sorting could easily be incorporated in to an ethnography field study, as a
subset to the overall research. Ethnography field studies encompass a wide range studies,
such as interviews, and focus groups, and aims to understand the “why” about a user. Card

6. sorting is different, and is a specific exercise designed to organise the information architecture
and content of a website. Unlike the field studies, card sorting can be as qualitative as it can
be quantitative. If the research is conducted one on one, or in group settings the results are
both qualitative and quantitative. Card sorting remains as an attitudinal dimension because
the users are vocalising and thinking aloud of their actions, and because users don’t
necessarily do what they say at least fifteen users should be tested to ensure a good dataset.
There are two main ways of conducting a card sorting exercise; the first is an open card sort
(See Fig. 3), this card sort allows the user to organise the cards in to their own group which they
will then name. It is a good exercise to understand the user’s language. The other is closed
card sorting, the difference is that group categories are named, and it is up the user to classify
all content within these groupings. Card sorting is best done on either a one to one basis or
small groups. If studying the ethnography of users, then a group setting may provide more
qualitative data. Like ethnography field studies, card sorting may require prior consent and
confidentially from the user(s).
Figure 4. An example of A/B Testing using Google Experiments.
A/B Testing
A/B testing falls on the complete opposite field from card sorting. It is quantitative
behavioural testing solution which in most cases is conducted post implementation of a
project to assess specific key metrics. An A/B test is the process of testing one or more
variations against the original design, usually on a page for page basis however this can be
extended to test individual design elements as well. Benefits to A/B testing are that it is able
to accurately measure small performance differences, for instance if a card sorting exercise
was used to prioritise navigational elements, the A/B test could prove its effectiveness by
analysing the change against the original design. The downside to this type of testing is that
it doesn’t provide information as to why the user found it easier. To find the answer the
researchers could resort the card sorting qualitative data, or gather user feedback through
online surveys. Another benefit to A/B testing is that is cheap, or even free (See Fig.4) as Google
now offers Experiments though their analytics. Depending on the in-house skillset this has
almost eliminated the need for any third party testing platforms. A/B testing is quite narrowly
focused and ideally will only to answer one goal or key performance indicator (KPI) at a time.
Should A/B testing be required before final production Neilsen (2005) discusses that paper

7. prototyping may offer an alternate solution and yield more qualitative results. This type of
testing can eliminate bad designs and allow the time to polish good ones before any coding
has been commenced saving time and money.
Research requirements are based entirely on the goals created on the project. Regardless of
what the goal may be there is always a well suited research method designed to answer this.
The examples above have been selected for their differences and, but at the same time using
these together in a project would give the researcher better grounding to defend their
designs.
Setting Goals
Research goals are as varied as there are abilities to conduct research. One main point to
make when creating goals, is they are not static. Research goals will change throughout the
course of development with new goals coming from unforeseen results further along the
timeline. A very good example by Rohrer (2014) is setting up research methods to coincide
with the phase of the product development and its associated objectives (See Fig. 5). In the
beginning stages of development is the time to consider new ideas, and opportunities for
future development. Research during this phase is mostly exploratory but can be both
qualitative and quantitative nature. This research period will provide information of who is
using the product, and to what extent. During this early stage personas and scenarios are
created to help address the pain points. Sometimes this can lead to altering the goal or adding
new ones into the design process.
Figure 5. An example of product development phases and the types of research goals to coincide with each.

8. When the process moves in to the designing phase, the goals become more qualitative to
optimising the design. Examples of these would be card sorting, paper prototyping, and field
studies.
Once the development moves to its final stages, the designs need to be assessed and proven
against the original. Performance measurement and comparison goals are then set, with
research methods switching to more quantitative methods. This doesn’t mean that
quantitative methods are only methods to be used here. Some projects would require post
implementation laboratory studies or in other cases field studies to assess the actual impact
or benefits to the users. In some process flows such as Agile UX this is a highly iterative
process and after one design has been launched and assessed it then goes back to the drawing
board for the next version. In short research goals and project timeline are coupled, as one
phase of goals is met and the project moves forward so to do the research goals which are
then addressed by the unique set of research methods covering that particular dimension of
study.
Conclusion
This paper set out to discuss and compare three selected research methodologies. It did so
by first presenting the reader with dimensions of study in which all methods are categorised.
This categorisation was provided to allow a sufficient background information for each so that
it was understood how these methods are conducted and what type of information can be
expected. Three methods were covered; Ethnography Field Studies, Card Sorting, and A/B
Testing. These were discussed and compared to set the reader up for understanding their
relation to the goals set during a project lifecycle. The various types of goals were then
addressed especially around the nature of their change throughout the design process, and
also how these alterations can affect the types of research required.
In conclusion the information in this paper is only a small snippet of the full extent of user
research methodologies available. A core understanding was applied to all the dimensions
which led to a better explanation of three methods. More importantly it is hoped that the
information of the selected methodologies will grant a better understanding of their
placement within a project, and how they address particular goals.

9. References
Boehm, N. (2010). Ethnography in UX :: UXmatters. Uxmatters.com. Retrieved 7 January 2016, from
http://www.uxmatters.com/mt/archives/2010/06/ethnography-in-ux.php
Davies, M. B., & Hughes, N. (2014). Doing a successful research project: Using qualitative or quantitative
methods. Palgrave Macmillan.
Jones, M. Ethnographic And Field Study. Camtools.cam.ac.uk. Retrieved 3 January 2016, from
https://camtools.cam.ac.uk/wiki/site/e30faf26-bc0c-4533-acbc-cff4f9234e1b/ethnographic%20and%20field%20study.html
Nielsen, J. (2005). Putting A/B Testing in Its Place. Nngroup.com. Retrieved 6 January 2016, from
https://www.nngroup.com/articles/putting-ab-testing-in-its-place/
Rohrer, C. (2014). When to Use Which User-Experience Research Methods. Nngroup.com. Retrieved 7 January
2016, from https://www.nngroup.com/articles/which-ux-research-methods/
Trivedi, M. C., & Khanum, M. A. (2012). Role of context in usability evaluations: A review. arXiv preprint
arXiv:1204.2138.
Bibliography
Croft, P. (2014). Improving Your Information Architecture With Card Sorting: A Beginner's Guide Smashing
Magazine. Smashing Magazine. Retrieved 6 January 2016, from
https://www.smashingmagazine.com/2014/10/improving-information-architecture-card-sorting-beginners-
guide/
Fairhead, J., Kaur, R., Mookherjee, N., Staples, J., Degnen, C., & Marvin, G. (2011). Ethical Guidelines for good
research practice (4th ed.). Association of Social Anthropologists of the UK and the Commonwealth (ASA).
Retrieved from http://www.theasa.org/downloads/ASA%20ethics%20guidelines%202011.pdf
Jurca, G., Hellmann, T. D., & Maurer, F. (2014, July). Integrating agile and user-centered design: a systematic
mapping and review of evaluation and validation studies of agile-ux. In Agile Conference (AGILE), 2014 (pp. 24-
32). IEEE.
Nielsen, J. (2004). Card Sorting: How Many Users to Test. Nngroup.com. Retrieved 7 January 2016, from
https://www.nngroup.com/articles/card-sorting-how-many-users-to-test/
Phetteplace, E., & Etches, A. (2013). Know Thy Users. Reference & User Services Quarterly, 53(1), 13-17.
Retrieved from https://journals.ala.org/rusq/article/view/2882/2940
Usability.gov,. (2013). Card Sorting. Retrieved 3 January 2016, from http://www.usability.gov/how-to-and-
tools/methods/card-sorting.html