“Remote UX testing when everyone is remote” by Jonas Rauff Mortensen As a senior service designer at Nile in Edinburgh, Jonas helps businesses gain a competitive advantage by rethinking both why and how they produce, provide and capture value. He has extensive experience in leading design, innovation and research projects for large corporate and start-up clients such as FreeAgent, Tesco Bank, Exizent, Heineken, NatWest Group and The Body Shop. He comes from a strategic service design background with a focus on user-centred design, design innovation and systems thinking and has been raised in the Danish design culture with an academic background in industrial design and architectural engineering and one of the first alumni of the M.Sc. in Service Systems Design from Aalborg University Copenhagen. Jonas has a passion for understanding organisations and making sense the complexity of how they provide and produce value for themselves and their customers in order to identify the right problems he will design services and experiences for. This presentation is from the SDN_UK online event Transitioning to Remote User Research, 21st May 2020.

1. Service Design Network | UK Chapter | Transitioning to Remote Research
Remote UX testing when everyone is remote
Presenter: Jonas Rauff Mortensen
Date: 21st May 2020
T R A N S I T I O N I N G T O R E M O T E R E S E A R C H
R E M O T E U X T E S T I N G
W H E N E V E R Y O N E I S R E M O T E

2. H E L L O
Jonas Rauff Mortensen
Senior Service Designer
Nile, Edinburgh
jonas@nilehq.com
2 Nile | The Strategic Design Co.

3. WE ARE A STRATEG IC
DE SIGN COMPA NY
THINKER S, EDUCATORS AND MAKERS
3 Nile | The Strategic Design Co.

4. User research doesn’t have to stop
because many of us are remote working.
Now more than ever is the time to be talking with users and
customers to understand their new context before going live.
Here’s how Nile makes Remote UX Testing work
for our clients and research participants.
4 Nile | The Strategic Design Co.

5. BEFORE FO C US ING SOL ELY
ON REM OTE UX T ES T ING, L E T’ S R EFR ES H ON
THE BASICS OF UX TESTING
What do we do? Why do we do it? When do we do it?
We observe what target
users say and do while
using a product or
service, and record and
synthesise their responses
and reactions.
UX testing identifies
and validates new or
unexpected ways to
make a product or
service better for both
users and the business.
“Early enough”. It’s a
balancing act; we need to
have something concrete
to test, but can’t wait
until things are too late
to change.
5 Nile | The Strategic Design Co.

6. ESS ENTIALS FOR AL L MOD ERATE D UX TE STING
FOCUS ON WH AT YO U WANT TO LEARN
+
D ISC USSIO N GUIDE RECRUI TMENT SPEC
& CONSENT FORMS
😬
C LE A R OBJEC TIVES, &
THE R I GHT Q UESTIONS..
..FOR THE RIGHT
PARTICIPANTS..
+
“TE STABLE” EX PERI EN C E
..IN FRONT OF WH AT
WE WANT TO T EST
6 Nile | The Strategic Design Co.

7. Moderated Unmoderated
RemoteIn-person
Social
distancing
means these
aren’t
feasible
today
There are several ways to
conduct UX testing: in person
or remote, moderated or
unmoderated. However, social
distancing has eliminated half
of our options here.
WHAT ARE
YOUR OPTIONS?
WHEN IT COMES TO UX TESTING
7 Nile | The Strategic Design Co.

8. MODERATED OR U NMO DERATED UX T ESTI NG
WHI C H DO YO U NE ED?
Moderated
(Remote)
Surface-level insight
Can identify show stoppers,
bugs and issues, but only gives
surface level insight.
You can’t ask follow up
questions to research
participants, explore observed
behaviours, or probe their
context in detail.
Full depth of insight
Moderated testing can find all
usability improvements, not just
show stoppers, bugs and issues.
Done well, this goes beyond usability
to identify user needs, behaviours,
attitudes and reasoning behind
decisions. It’s also better for building
empathy with users and engagement
with project teams and stakeholders.
Unmoderated
(Remote)

9. REMOTE UX T ESTI NG
HAS M A NY A DVA NTAGES
WHAT A RE THE ADVANTAGES AND D I SADVANTAG E S O F R E M OTE TE STIN G?
Greater diversity of participants from a
wider geographical spread, and lowers
the barrier for hard to recruit audiences
Lower cost than in-person: lower
incentives, no venue hire, and no travel
required
Often faster: doesn’t depend on
research venue availability or require
complex logistics
Enables easier observation for project
teams and stakeholders
Can exclude those who don’t have
access to digital technology or are less
IT literate
Might have higher risk of drop-outs
without pre-engagement of
participants
The moderator has to work harder to
build trust and rapport with
participants
The observation experience can be less
engaging when observing alone
…A ND A FEW DOWNSIDES
TO M IT IGAT E AGAINST
9 Nile | The Strategic Design Co.

10. T H AT MEANS, REMOTE UX TESTING
I S NOT A ‘WORSE VERSION’
O F IN-PERSON UX TESTING AT ALL ..

11. ..T HAT’ S WHY, WE HERE AT NILE AL RE ADY
D O A BOUT HALF OF OUR RESEARCH
R EMOTELY BY DEFAULT (NOT JUST RIGHT NOW)

12. YO U SH OULD N OT
TRAN SI TION TO
REMOT E UX TESTI NG
YOU SHO UL D
ALWAY S CONSIDER REMOTE
UX T ESTI NG AS A N OP TION
(EVEN IF YOU CA N D O I N-PER SON )
12 Nile | The Strategic Design Co.

13. Confidential | Nile | The Strategic Design Company13
MAKING
REMOTE UX
TESTI NG A
SUCC ESS

14. Technology as the enabler
Building strong relationships
The value of live observation
1
2
3
Tried and tested technology that enables
running and observing remote UX testing is
crucial. But getting it right doesn’t guarantee
success - techniques need to change too.
To get the most from recruited research
participants, we take deliberate steps to build
additional rapport to guarantee participant
engagement with the testing.
Insight does not just come directly from
users. Live remote observers are a valuable
source of insight. We design for observer
experience throughout.
AT NILE WE USE THREE PRINCIPLES TO
MA KE R EM OTE UX TE STING A SUCCESS
We depend on
(not the answer)
And we focus closely on
..as well as making the most of
14 Nile | The Strategic Design Co.

15. TECHNOLOGY AS THE ENABLER1
WH AT’S IN OUR
REMOTE UX TOOLSET?
OU R TR IED AND TESTED TOOLSET
NOTE TAKING,
ANALYSIS AND INSIGHT
DEVELOPMENT
SHARABLE
PROTOTYPES
VIDEO CONFERENCING
& SCREEN SHARING
CARD SORTING
LIVE OBSERVATION
DISCUSSION
ACTIVE REMOTE DEBRIEFS
LIVE EXPERIMENTS
We are always looking at what’s
new. At the moment we are
kicking the tyres on:
COLLECTING CONSENT
SO WHAT?
Great technology still needs the right processes to support it. Let’s have
a look at our checklist limiting the risk of something going wrong.
15 Nile | The Strategic Design Co.

16. TECHNOLOGY AS THE ENABLER1
GET TING T HE T ECH RIGHT
& AVOIDI NG PROB LEMS
& OUR C HEC KLIST FOR..
1
2
3
4
5
Always, always, always run at least one pilot session
Having a plan is great, but you need to test it before
trying it with actual research participants.
Have backup options ready
Technology can fail. We always have a backup if
audio is acting up or the recording fails.
Brief the research participant well beforehand
It’s always great to set the right expectations and
also get participants to install and possibly test
applications.
Create tech workflows and connect systems
Streamline remote UX testing workflow by
connecting system e.g. Run and record in
Zoom, auto-transcript in Otter, and paste
directly into Dovetail.
Have someone ready to help observers
It’s not just the moderator or research
participant that can experience problems
with technology.
16 Nile | The Strategic Design Co.

17. BUILDING STRONG RELATIONSHIPS2
BUI LDI NG STR ONG
RE L ATI ON S HI PS WI TH
PARTICI PANTS
Separating a moderator and
a participant with a screen changes things.
Rapport is typically harder to build, and it’s much
harder to communicate using body language.
That’s why we intentionally invest time focusing
on participant engagement while moderating
remote UX testing. Because better engagement
means better insight.
GET BE T TER I NSIG HT BY
VS.

18. BUILDING STRONG RELATIONSHIPS2
GET TING THE B EST RESULT S WH EN
MOD ER AT I NG REMOTE UX TEST ING
We establish our
trustworthiness
They are not coming to your
research venue, so you need to
establish how you want to be
perceived - both with what you tell
them and how you and your
surroundings look.
We focus more time
learning about them
We invest more time up front in
getting to know the research
participant and refer back to it
during the session - sometimes we
even call them prior to the session to
warm them up.
We make silence
acceptable
We let participants know that silence
supports thinking and reflection time
up front. We design silence in to our
sessions to give space and time for
thought and reflection.
We lean in and
we are engaged
We show participants that we
are interested, but are aware
of not overdoing it to avoid
leading participants.
& OUR APPROAC H TO..

19. LIVE REMOTE OBSERV ERS
AS A SOURC E OF INSIGHT
Often having teams and stakeholders observing sessions and
discussing afterwards leads to new ways to look at a problem.
Right now, everyone is ‘remote’ - not just the research participant, but also
project teams and stakeholders observing the research, so we go even
further to make remote observation engaging, productive and valuable.
THE VALUE OF LIVE OBSERVATION3
INSIG HT DOE SN’ T JUST COME FR OM USERS
19 Nile | The Strategic Design Co.

20. ENGAG ING & PROD UCT IVE
REMOTE OB SERVATI ON
& OUR APPROAC H TO..THE VALUE OF LIVE OBSERVATION3
Observer
briefings
We brief all remote
observers before they join
their first session and give
active ways to contribute.
Live observer chat
& note-taking
During sessions we run a
live chat for all observers
that also gives the ability
to pass questions to the
moderator.
Active
remote debriefs
We facilitate active
remote debriefs after
each session and create
structure with Miro.
BEFORE DURING AFTER
20 Nile | The Strategic Design Co.

21. FOR AL L TH E D ETAI L S
FROM TODAY ’S PR ESENTATI ON
RE A D MOR E HE R E
21 Nile | The Strategic Design Co.

22. THAN K YOU!
AN Y Q UE ST IONS?
PS.
We love remote
UX testing, but we
still miss this place
22 Nile | The Strategic Design Co.

23. Confidential | Nile | Service, Business and Experience Design23
GET IN TOUCH, WE’D LOVE TO
HEAR FROM YOU
HQ 13-15 Circus Lane
Edinburgh, EH3 6SU
+44 (0) 131 220 5671
www.nilehq.com @NileSays