Wellcome Collection is a free museum and library challenging how we think and feel about health. See how the Elasticsearch Service is used to aggregate descriptive data and provide unified search and discovery. See the video: https://www.elastic.co/elasticon/tour/2019/london/improving-search-at-wellcome-collection

1. 1
Jonathan Tweed
Technical Product Manager, Wellcome Trust
Improving search at
Wellcome Collection

2. 2
Wellcome exists to
improve health by
helping great ideas to
thrive

3. 3

4. 4

5. 5
A free museum &
library exploring
health, life and our
place in the world

6. 6

7. 7
Increasing reach, engagement
and diversity of audiences

8. 8

9. 9

10. 10

11. 11

12. 12
digitisation ≠ access and use

13. 13
NOT VIEWED
in six months
290k digitised items
VIEWED ONCE
in six months
>1000 TIMES
in six months
71% 7% 0.0002%
~40 million images

14. 14

15. 15
We have a discovery
problem
• Researchers don’t know what
they can search for
• Search is fragmented by how
things have been catalogued
• Found items are difficult to
understand in context
• 15% click through from search

16. 16
We needed a new
approach
• Existing third party systems
limited our ability to improve
• No better alternatives in the
library systems market
• Insufficient and inflexible
internal infrastructure
• Hired in-house digital team
• Decided to build on AWS

17. 17
Researching the researchers
User journey mapping

18. 18
Unified search and discovery
across collections

19. 19
Creating a single coherent experience
Aggregating multiple data sources
Library
catalog
Archive
catalog
Image
catalog
Digital
rights
Unified
search
Books, visual art,
ephemera, audio, film
Archive collections,
manuscripts
Digital images Availability, copyright,
licensing
One way to search
everything

20. 20
Building a data pipeline
Aggregating multiple data sources
Ingest Transform Aggregate Index Query
Sync with sources
and store local copy
Transform to
common data model
Find graphs of potential
matches
Merge matches and
index in Elasticsearch
Provide search API to
query Elasticsearch

21. 21
Data pipeline architecture
Aggregating multiple data sources

22. 22
Why we chose
Elasticsearch
• Support for full text search and
complex aggregations
• Availability of Kibana for
reporting and ad hoc analysis
• Extremely flexible indexing and
querying capabilities
• Potential to use it for other
things, like log aggregation

23. 23
Why we chose Elastic
Cloud
• We’re a small team, so we
prefer managed services
• Immediate access to latest
versions, on release day
• Access to all Elastic Stack
features, including graph
• Easy cluster management,
resizing and upgrades
• Confidence in Elastic and their
product knowledge

24. 24
Current status:
search is a journey

25. 25

26. 26
Iterating on search
AB testing
Front end can split traffic to different queries, to
test relevance improvements
Qualitative testing
Explicit feedback mechanism for users to
provide information on subjective quality
Metrics
We ingest search metrics into Elasticsearch,
via Segment, with reporting in Kibana and ad
hoc analysis in Jupyter notebooks

27. 27
Progress against
metrics
• 54% click through from search,
up from 15%
– This is with minimal query changes,
so further to go
• Mean clicks per session:
– Single page: 1.81
– Multi page: 7.62
• Complicated by multiple search
behaviours

28. 28
Multiple search behaviours
Search vs research

29. 29
Where we go from here
• Looks for ways to identify query
intent and add structure
• Consider different queries for
targeted search vs research
• Continue to AB test small, iterative
query changes
• Integrate data from machine
learning for colour and similarity

30. 30
Thank you
stacks.wellcomecollection.org
github.com/wellcometrust