As museums and cultural institutions work to create more user-friendly products, we conduct research that sometimes ends up buried as we talk more about what was created. Whether it’s prototyping and testing along the way, or a summative evaluation of one project leading to changes for the next one, how do we learn from these findings? Desi Gonzalez, The Andy Warhol Museum, Pittsburgh Jenn Phillips-Bacher, Wellcome Collection, London Susan Wigodner, The Field Museum, Chicago

1. From Research to Action: Translating User Feedback into Digital Products
#MCN2017-T4
Desi Gonzalez, Jenn Phillips-Bacher, and Susan Wigodner
November 9, 2017

2. From Research to Action
#MCN2017-T4
2

3. Planning to Redesign
Fieldmuseum.org
#MCN2017-T4
Susan Wigodner
@swigs27
The Field Museum, Chicago
@FieldMuseum
MCN 201703

4. 4
https://digitalstories.wellcomecollection.org

5. 5
● Web & Digital Communications Director
● Project Manager
● Content Developer
● Project/Content Coordinator
● Social Media Manager
● Front-End Developer (currently vacant)
● UX/UI Designer (currently part-time)
● Media Producer
Web & Digital Communications
Marketing &
Public Relations
Information
Technology
Exhibitions
Institutional
Advancement
Science & Education
Website (Product)
Digital Content
Social Media
Microsites (On Hold)
+ Audience Insights and
Research Manager

6. Upfront User Research
6
Worked with Expose Your Museum from October 2016 - January 2017 on:
● Website pop-up surveys (20K respondents, 10 days)
● Analytics review
● Staff discussion groups (8 groups, 8-12 people each, 90 minutes long)
● User testing (12 sessions, 45 minutes long)
Shared research reports via All Staff email and Intranet in Spring 2017

7. Setting Priorities
7
Fieldmuseum.org Redesign
#MCN2017-T4

8. 8
Google Analytics
Jan - Dec 2016
Fieldmuseum.org Pageviews by Section

9. 9

10. 10

11. 11

12. 12
Survicate
Dec 2016
Fieldmuseum.org Pop-up Surveys

13. Setting Priorities
13
Rebranded site needs to launch Spring 2018
Phase 1: General Visitors (visit planning, programs and events, “About” section)
Phases 2+ (Priority Order, Schedule Not Yet Set)
● Supporters (donors and involvement opportunities)
● Revenue and Museum Promotion (press, events, traveling exhibitions)
● Educators
● Researchers & Scientists

14. Setting Goals
14
Fieldmuseum.org Redesign
#MCN2017-T4

15. What Are We Working Towards?
15
User Testing / Pop-Up Survey Takeaways Selected Goals
In testing, users:
● displayed an over-reliance on search
● took navigation language very literally
● often struggled to complete defined tasks
Develop user-focused site architecture and content
60+% of users classify reason for visiting as “Planning a
Visit”
In testing, users:
● requested clearer information about timing and
pricing for exhibitions and events
● struggled to find information about upcoming
exhibitions
Prioritize easy access to visit-related content
25% of users classify reason for visiting as “Personal
Interest” or “Browsing”
In testing, users:
● appeared interested in the collections, but
expressed a desire to have more generalist
access
Encourage discovery about science and natural history

16. Staff Discussion Groups
72 staff members from across all departments
16
Agreement On:
● Lack of brand and content consistency
● Challenge of varied and overlapping
audiences
● Need for more defined publishing
process
● Staff hear that people can’t find what
they need; rely heavily on search
● Concerns about accessibility
● Ticket options confuse visitors, online
purchase platform is difficult to use
Disagreement On:
● Priorities, particularly with how
homepage is used (visit information
versus highlighting science content)
● Who should be able to publish
website content
● When content can be retired
● Use and utility of media
...and on...

17. Guiding Principle Heard from Staff
Design Integrity “...no one has this unified view of how the Museum should be doing things”
“I have heard a lot of people say that there are places where it's very well done but you have to pick through the rest of the bodies to get
to those.”
Ease of Use “I'm on the phone a lot with clients and I really have to walk them to the exact page I want them to see and it's time consuming for them.”
“...oftentimes it is content that actually is on the site, but it's too many clicks away, or too many clicks deep for them to get to it.”
Consistency “I've noticed that [on] certain pages, someone will add content, and that same content will be on an entirely separate page, but then when
you go back to update it you may not think to update the second page that has that same content.”
“We have a lot of different people doing a lot of different things, and at some level, from my perspective, and I know this leads to chaos,
but that's what we ought to be celebrating with respect to the website. There ought to be a ton of content.”
Communication “We do need to have some high tiers on the site that are very professionally done, but also we need content. Up to the minute content is
going to maybe not be ideally perfect.”
“another [thing] I think that we struggle with ...is the way that we see [ourselves] organized, and our work, and maybe a slightly different
way that we would want to project and share that with partners.”
Accessibility and
Inclusivity
“...we've thought some about that, how to make this accessible to people who may need a wheelchair, may need some other help when
they come.”
“It feels like we're looking at it from our perspective of, "Here's everything we do," rather than looking at it from the outside in, going,
‘Who's the audience that's looking in?’"
How We Want to Work
17

18. Digital Stories:
Using Research to Iterate
#MCN2017-T4
Jenn Phillips-Bacher
@MrsAudiac
Wellcome Collection, London
@ExploreWellcome
MCN 2017018

19. 19
https://digitalstories.wellcomecollection.org
Wellcome Collection

20. 20
Digital Content
Digital Platform
Digital Production
Photography
Collections &
Research
Live Programme Exhibitions Publishing
Marketing &
Communications
The Product Team
● Full-Stack Developer
● 2 Front-End Developers
● UX Designer
● User Analyst
● Visual Designer
● Product Manager

21. 21

22. 22

23. 23
Digital Stories, Version 1: The Approach
Big design upfront
No user research
Agency and freelance
contractors
Summative evaluation at
end

24. Summative Evaluation Findings: Usability
24
Finding:
‘Find out more’ is
a distraction

25. Summative Evaluation Findings: Usability
25
Finding:
Source material
doesn’t enhance the
narrative

26. Summative Evaluation Findings: Usability
26
Finding:
Support wayfinding
Offer way to learn
about new stories

27. Summative Evaluation Findings: Analytics
27
69%
visit 1 or 2 chapters only
<5%
explore an entire story
in one session

28. Summative research is a
to-do list in waiting.

29. Digital Stories: The Return
29
What’s different this time?
Multidisciplinary team with permission to make product decisions
Continuous discovery
Small batch and large batch testing
Small numbers for initial concept testing and usability (guerrilla research)
Large numbers for data-driven experiments (AB testing, prototypes, remote surveys)
Rethink how we produce, publish and position Digital Stories

30. 30
A/B testing and
prototypes help us
decide

31. Chapter Navigation Prototype: Drop Down
31

32. Chapter Navigation Prototype: Horizontal Scroll
32
Action:
We built
this version

33. Sticky Navigation
33
Default state Sticky state

34. Sticky Navigation
34
Action:
Non-sticky
Retest on
widescreens

35. Email Signup Experiment
35
● Mobile visitors only
● Ran experiment for 3 weeks
● Used Mailchimp’s free account to collect
email
● More staff than public signed up
● Open rate: 62.5%-75%
● Click rate: 0%-29%
Action:
We parked
email

36. 36
Guerrilla testing and
interviews help us
understand and refine

37. Perception of Content and Usability
37
Action:
Changed titles
on next story
(The Outsiders)
“What’s an analog story?”
“I’m confused by the title’s meaning”
“I would like some sort of summary at the start
that told me what I could expect from being on
this page.”

38. 38
Make decisions better.
Make better decisions.

39. User Research at
The Warhol
#MCN2017-T4
Desi Gonzalez
@desigonz
The Andy Warhol Museum, Pittsburgh
@thewarholmuseum
MCN 2017039

40. 40

41. 41
● Manager of Digital Engagement
Digital Engagement
Marketing &
Communications
IT (shared by
four museums)
Learning & Public
Engagement
Strategic
Initiatives
Rights & Digital
Imaging

42. Out Loud
The Warhol’s Inclusive Audio Guide
#MCN2017-T4
MCN 2017042

43. 43
Out Loud
The Warhol’s Inclusive Audio Guide

44. User Research: Who
44
● 3 user-experts
● Advisory team
● 20 Accessibility Meetup members
● 3 user-experts
● Advisory team
● 20 Accessibility Meetup members

45. User Research: How
45
● Interviews
● Paper and interactive prototypes
● Usability tests

46. From Interviews to Action
46
Quote Design principle Design decision
“I want a little more independence” during a visit Provide a path with a choice
Split up audio content into smaller, modular stops and
allow visitors to dive deeper if they’d like to learn more
“I usually go to museums with other people; it’s a
social thing.” Build as a social experience
Develop audio content that sparks discussion
Provide headphone splitters for visitors to share
devices
“I expect a back button in the top left corner, the
navigation on the bottom, and [for audio or video]
a play/pause button in the center of the screen.”
Use industry standards for
interactions
Support VoiceOver functionality on iOS
Find inspiration in simple, everyday apps, such as
Apple’s Podcasts
Provide audio transcripts
“Because of my cane, I like things that I can do
with one hand.”
Keep physical and sound
interactions to a minimum
Lanyards for devices
Avoid playing audio automatically
“I like hearing about the history around a work of
art more than how it’s made.”
“I want to know the details—the medium, the
style, the process to make it.” Provide a mix of voices and tones
Include various staff voices and perspectives
Conversational, podcast-like tones

47. From Interviews to Action
47
Quote Design principle Design decision
“I want a little more independence” during a visit Provide a path with a choice
Split up audio content into smaller, modular stops and
allow visitors to dive deeper if they’d like to learn more
“I usually go to museums with other people; it’s a
social thing.” Build as a social experience
Develop audio content that sparks discussion
Provide headphone splitters for visitors to share
devices
“I expect a back button in the top left corner, the
navigation on the bottom, and [for audio or video]
a play/pause button in the center of the screen.”
Use industry standards for
interactions
Support VoiceOver functionality on iOS
Find inspiration in simple, everyday apps, such as
Apple’s Podcasts
Provide audio transcripts
“Because of my cane, I like things that I can do
with one hand.”
Keep physical and sound
interactions to a minimum
Lanyards for devices
Avoid playing audio automatically
“I like hearing about the history around a work of
art more than how it’s made.”
“I want to know the details—the medium, the
style, the process to make it.” Provide a mix of voices and tones
Include various staff voices and perspectives
Conversational, podcast-like tones

48. Delivering Modular Audio: Early Prototypes
48
Quartz app Prototype 2Prototype 1

49. Delivering Modular Audio: Final Design
49

50. Redesigning warhol.org
#MCN2017-T4
MCN 2017050

51. 51
Before
After

52. User Research: How
52
● Usertesting.com tests
● Personas + journey maps
● Usability testing

53. User Personas
53
TeacherTourist Access Researcher
Family Teen Young Professional

54. User Personas
54
Teen
Alex
A teen who identifies as gender non-binary
has learned about Warhol in art classes in
high school. Their teacher mentions the
opportunities for teens in class one day. They
are a junior who goes to public school on the
North Side. They don’t see themselves
reflected in their social circles and are looking
to connect to people and places that reflect
them. They are thinking about college, but
undecided on the way forward. Looking for
experiences to guide their thinking.

55. Journey Mapping
55

56. Usability Tests: Calendar
56
Observations Learning Design decision
Many used the homepage as a way to get an overview of events
and exhibitions at the museum
People want to search by audience type, but often this
segmentation is found in a “Programs”/“Learn” section of a site
Provide many different
ways into museum
events
Dynamically pull in calendar events on our more
static Programs pages
Include pathways for specific audiences to find
information tailored to them
Several users wondered what day of the week certain events were
so they could plan around their schedules
“I love that it says ‘Closing soon’ right at the top”
Give the basic info, and
make it clear
Design of event pages: highlights the important
logistic information
Dynamically change language for exhibition dates:
"Opens November 15, 2017" if two weeks before
an exhibition opens
Terms like “lectures” and “seminars” can be off-putting to art
newbies
Both tourist user profiles assumed “Film screenings” would be
under “Watch” on the Whitney’s website Language is everything
Carefully crafted nomenclature to guide our events
taxonomy; on calendar, surface events by type and
audience, but not by series brand
Several users noted the tension between wanting both a broad
overview of all events at the museum and needing to drill down to
a specific day or time span—they wanted both!
Users both want to
browse generally and
find specific information Toggle between 1, 7, and 30 day views
One tourist was mesmerized by the CMOA’s Iris van Herpen
image
A parent used images to decide if an exhibition would be
age-appropriate for her kids Images are powerful
Developed web photography guidelines that favor
strong images of people interacting with each other

57. Usability Tests: Calendar
57
Observations Learning Design decision
Many used the homepage as a way to get an overview of events
and exhibitions at the museum
People want to search by audience type, but often this
segmentation is found in a “Programs”/“Learn” section of a site
Provide diverse ways
into museum events
Dynamically pull in calendar events on our more
static Programs pages
Include pathways for specific audiences to find
information tailored to them
Several users wondered what day of the week certain events were
so they could plan around their schedules
“I love that it says ‘Closing soon’ right at the top”
Give the basic info, and
make it clear
Design of event pages: highlights the important
logistic information
Dynamically change language for exhibition dates:
"Opens November 15, 2017" if two weeks before
an exhibition opens
Terms like “lectures” and “seminars” can be off-putting to art
newbies
Both tourist user profiles assumed “Film screenings” would be
under “Watch” on the Whitney’s website Language is everything
Carefully crafted nomenclature to guide our events
taxonomy; on calendar, surface events by type and
audience, but not by series brand
Several users noted the tension between wanting both a broad
overview of all events at the museum and needing to drill down to
a specific day or time span—they wanted both!
Users both want to
browse generally and
find specific information Toggle between 1, 7, and 30 day views
One tourist was mesmerized by the CMOA’s Iris van Herpen
image
A parent used images to decide if an exhibition would be
age-appropriate for her kids Images are powerful
Developed web photography guidelines that favor
strong images of people interacting with each other

58. Research to Action
58

59. Conversation
and Questions
#MCN2017-T4
59
Desi Gonzalez
Manager of Digital Engagement
The Andy Warhol Museum
@desigonz
Jenn Phillips-Bacher
Product Manager, Digital Experience
Wellcome Collection
@MrsAudiac
Susan Wigodner
Web & Digital Project Manager
The Field Museum
@swigs27

60. Thank You!
MCN 2017060

61. 61
Links
Field Museum
● Fieldmuseum.org
Wellcome Collection
● Wellcomecollection.org
● Digital Stories: Mindcraft, The Collectors, Electric Sublime, The Outsiders
The Warhol
● Warhol.org
● Out Loud