Presentation for Museums and the Web 2018 (Vancouver)
April 19, 2018
In a world in which a family historian can type her grandfather's name into Ancestry.com to start building a family tree, and a journalist can Google to download public domain images, where do the collections searches, online tools, and APIs that museums and archives provide fit in? This paper outlines strategies for better serving people who are looking for the knowledge and expertise within your collections and staff. At the National Archives and Records Administration of the United States, we undertook a significant user experience (UX) research project to better understand the online experiences of professional researchers, family historians, and history enthusiasts. Research methods included audits of existing user data (e.g., Google Analytics, survey data) as well as new user interviews, usability testing, a survey, and a landscape analysis. Key findings include the fact that researchers struggle to complete their tasks using existing online tools; people researching family history are particularly unsatisfied and in need of better support; and all audiences require just-in-time help and appropriate orientation to archival research. A major challenge highlighted by this research is how to meet user expectations for item-level records while providing access to digitized records at massive scale.
Read full paper:
Workers Unloading Veteran's Bureau Records, 06/03/1936 https://catalog.archives.gov/id/7820631
Customer Satisfaction scores from Foresee survey data on all of Archives.gov, the Catalog, and History Hub for January 2017-November 2017.
Task accomplishment results from Foresee survey data on all of Archives.gov, the Catalog, and History Hub for January 2017-November 2017.
Archives.gov is NARA’s flagship website,serving more than 20 million users in 2017. It is the portal to information for researchers, veterans, educators, museum visitors, and more. The public can view statistics about NARA’s website usage, including most-used pages, on analytics.usa.gov thanks to the U.S. government’s Digital Analytics Program.
The National Archives Catalog (catalog.archives.gov) contains archival descriptions for over 95% of the holdings of the National Archives and over 39 million digitized copies of records. The site served 1.6 million users in 2017. Citizen archivists have added more than 1.2 million enhancements to the Catalog’s records in the form of user-contributed tags, transcriptions, and comments.
Satisfaction scores (via the Foresee survey) for the National Archives Catalog fell 16 points in 7 months. How can we help people find the needle they want in an ever-increasing haystack? A massive challenge for NARA is how to balance access to the scale of its holdings with usability and findability. As part of our strategic plan, we aim to digitize 500 million pages of records and make them available online to the public through the National Archives Catalog by 2024. A review of ForeSee data in 2016-2017 revealed that customer satisfaction scores for the Catalog were consistently declining, despite the fact that no user interface changes had been made to the site. The cause? During this period of time, we uploaded millions of new digital objects. Our hypothesis is that as the number of digitized items grows, customer satisfaction scores decreased because they had even more options to sift and sort through. The moral of the story? The more that is made available online, the more scaffolding and assistance we may need to provide to help people find that needle in the haystack.
History Hub (history.gov) is a crowdsourcing platform and online community that served 60,000 users in 2017. History Hub enables researchers to find expertise, share information, and work together. More than 700 research questions have been asked and answered on the platform since its launch in 2016. We are just beginning to understand the potential uses of History Hub as a platform, and this research project helped us better understand who uses the site (and who doesn’t), and what role it plays in supporting researchers. History Hub is intended to be a tool for many cultural institutions to use (not just NARA) and it is free and open to all
Several challenging goals set out in NARA’s strategic plan serve as the backdrop for this research project: By FY 2024, NARA will digitize 500 million pages of records and make them available online to the public through the National Archives Catalog. By FY 2025, NARA will provide digital, next-generation finding aids to 95 percent of the holdings described in the National Archives Catalog. By FY 2025, NARA will have 1 million records enhanced by citizen contributions to the National Archives Catalog. To achieve these goals, NARA must not only figure out how to provide access to a massive number of archival resources but also how to do it through user-centered digital products that scale.
See all digital user personas: https://www.archives.gov/digitalstrategy/personas
Audience role results from Survey Monkey survey, January 2018, with 1,847 respondents. This survey opportunity was provided to a smaller group of individuals who used the Catalog (or received its e-mail newsletter), Archives.gov/research or Archives.gov/citizen-archivist, History Hub, or the 1940 Census website.
in a self-service world
Museums and the Web - April 29, 2018
In the next 15 minutes
1. How and why the National Archives
is investing in UX research
2. Key findings that might apply to
3. Areas for further research and
About the National Archives
The National Archives (NARA) is
America’s record keeper. We hold
hundreds of miles of film and tape,
14 million still photographs, and
billions of letters, drawings, maps,
treaties, posters, and other
documents that tell the stories of
America’s history as a nation.
Workers unloading Veteran’s Bureau Records (June 1936)
● Digital products
developed over time
without a strong (or
● Low customer
● Lots of assumptions
A frustrated driver sits through a traffic jam in Herald Square (May 1973)
Satisfaction scores by audience
Public service gap
Most users are unable to accomplish their task
How might we we help more people
accomplish their tasks?
…(Yes and) how might we help people
better define research tasks so they can
Did you accomplish what
you wanted to
do today on the site?
Consistently declining customer satisfaction
Scores for the
Catalog fell 16
points in 7 months.
Learn more about how
you can get involved:
Why is this issue urgent now?
Our strategic plan dictates:
★ By FY 2024, NARA will digitize 500 million pages of records and make them
available online to the public through the National Archives Catalog.
★ By FY 2025, NARA will provide digital, next-generation finding aids to 95
percent of the holdings described in the National Archives Catalog.
★ By FY 2025, NARA will have 1 million records enhanced by citizen contributions
to the National Archives Catalog.
Why invest in UX?
❏ In the past, NARA designed and developed products without a user-centered
framework. These legacy products are now very difficult to change due to rigid
procurement and development processes.
❏ We must figure out not only how to provide access to a massive number of
archival resources but also how to do it in a way that is usable and useful at
❏ Our research is designed to help us use our limited resources most efficiently
and effectively to support successful user-centered products.
UX add to strategic plan!
Added “Build capabilities in design and user experience, to better understand user
To better understand today's online marketplace for researchers and how we can
best serve their needs.
1. Assess the current level of audience engagement
2. Understand NARA’s high-value audiences
3. Identify new, aspirational audiences
4. Review peer organizations for best practices, features, and trends
5. Recommend ways to improve user experience on current digital platforms
6. Identify new opportunities to engage audiences
Usability testing/ observation
9 remote users were given task-based
questions to carry out on Archives.gov,
Catalog, and History Hub, and asked to
explain their choices
Reviewed 12 websites to compare
features, functionality, and content to
understand the wider online research
Review of existing data
-Customer satisfaction survey
- Google Analytics data
- Search keyword metrics
- Social media analytics
-Textual analysis of Contact Us emails
19 interviews (30-60 min) with
genealogists, in-person researchers,
online researchers, electronic records
experts, and citizen archivists
New survey of researchers
Audiences vs. Tasks
“Online researcher” is
a diverse category.
We should optimize
digital services to
meet the needs of
specific user tasks
rather than broad
Online Researchers Expect
Online Records First
● Only 25% of online researchers have
ever visited a NARA research facility.
● 86% said online research (as opposed
to in-person research) is important or
● Most researchers self-identified as
World War I poster (1943-1945)
Not just a “learning curve”
Returning visitors are no more satisfied than new
➔ Suggests that no new knowledge is gained in
the first visit that makes subsequent visits easier
➔ Solution can’t just be about “better orientation”
or more research guides (although just-in-time
support would help).
➔ Need to fix fundamental UX problems so our
digital services are usable and useful.
Keyword topics and names of people provide relevant results.
People researching family history
are particularly unsatisfied
● At least 1/3 of our online researcher audience
● Only 24% of this audience was able to complete their
● Many visit Ancestry.com or similar sites first, where they
are accustomed to user-friendly features like person
name search and highlighted results
○ Popularity of DNA heritage tests is driving growth of this audience
● Most do not identify with the term “genealogist” or even
Even experienced researchers need help orienting to your
● scope and size
○ What is an archive?
○ What’s in YOUR archive?
● how is it structured
● what is digitized and what isn’t
Recommended reading: Whitelaw, M. “Generous Interfaces for Digital Cultural Collections”
Guidance (in a PPT file!) on a different
website is not helpful in the middle of a
How can we help people when they get
How might AI help us support research?
1. Item-level expectations - Our systems and processes are structured around
descriptions of records groups and series. While we have millions of digitized
items, we don’t have good metadata to support their discoverability (e.g., by topic,
by person, by what is viewable in an image). How might AI (artificial intelligence)
help bridge this gap?
2. Finding needles in the haystack - How might machine learning make search
results more relevant and get smarter over time based on use? (This is especially
interesting for thinking about the tsunami of email records coming soon.)
3. Providing in-line support - Can chatbots provide useful customer service for
common researcher requests? How can we re-purpose knowledgebases like
History Hub to scale up personalized support?
1. Consolidate products (distinction
between websites unclear to users)
2. Show digitized items first by default
3. Build streamlined interfaces to guide
users in specific research tasks
4. Orient audiences who are
underprepared for archival research,
including a new visual overview of
5. Offer in-line help and support
1. Transform our procurement and
development to more agile
methodologies with UX work baked in
to the process
2. Leverage AI and machine learning to
support discoverability and tailored
3. Expand and improve API (including
faster performance, better image
delivery, and more coherent data
How can we better leverage our API?
To kick the tires on our API and uncover limitations (performance, data structure,
documentation), we have developed 6 prototypes:
1. Treemap/overview of holdings, a “generous interface” that communicates scope
2. Facebook Messenger chatbot which encourages tagging of records
3. "When Am I?" game for engaging with records by time period (and adding tags)
4. Voice-activated Alexa skill which enables exploration of recipes
5. Drupal image selection and cropping tool (open source module)
6. Drupal CAPTCHA module that facilitates tagging records (open source module)
How might we massively scale up
crowdsourcing of metadata?
(Our API is read/write to include
transcriptions, and comments.)