Research into how people find and share expertise can be traced back to the 1960s, with early studies focusing on knowledge workers such as engineers and scientists and the information sources they consult. However, in recent years there has been a growing recognition that the effectiveness of expertise retrieval systems is highly dependent on a number of contextual factors, where the emphasis is on how people search for expertise in the context of a specific task. These studies have typically been performed in an enterprise context, where the aim is to utilize human knowledge within an organization as efficiently as possible. This talk presents results of an Innovate-UK funded project investigating the use of complex search strategies in the workplace, with the aim of producing requirements for the design of next generation search tools.

1. Dr.Tony Russell-Rose, FBCS CITP
UXLabs

2. April 2014: InnovateUK grant
Complex query formulation
Jan 2015: Smart Award
Proof of Market
3UXLabs - User Experience Research and Design - www.uxlabs.co.uk

3.  Assess the potential markets
Qualitative research involving
end users from target sectors
Catch-22
Target sector <-> market
research
Who uses complex queries?
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 4

4.  Usual suspects:
 Patent agents
 Lawyers
 Unknown quantities:
 Librarians?
 Power users:
 Media monitoring
 Healthcare (systematic evidence review)
 And…
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 5

5. UXLabs - User Experience Research and Design - www.uxlabs.co.uk 6

6.  Nested composite
Java AND (Design OR develop OR code OR Program) AND
("* Engineer" OR MTS OR "* Develop*" OR Scientist OR
technologist) AND (J2EE OR Struts OR Spring) AND
(Algorithm OR "Data Structure" OR PS OR Problem
Solving)
 Enumerations of related terms
("looking for" OR "in search of" OR "open to" OR "new
job" OR "actively pursuing" OR "pursuing new" OR
"searching for" OR "new opportunity" OR "new
opportunities" OR "available for" OR "in transition"
OR unemployed OR "immediately available" OR
"currently seeking" OR "seeking new" OR "seeking a
new" OR "interested in")
 Index field lookups
site:ca.linkedin.com "network engineer" "ccnp" "wan" "lan"
"vancouver" -intitle:"profiles" -inurl:"dir/ " -
inurl:job|jobs|jobs2
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 7

8.  Qualitative
 Interviews with stakeholders
▪ 2-3 from each sector
 Quantitative
 Online survey
 Baselines for comparison
▪ Patent search [Joho et al, 2010]
▪ Medical professional search [Geschwandtner et al, 2011]
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 9

9.  40 questions, 15 minutes
 5 sections
 Demographics:The background and professional
experience of the respondents
 Search tasks:The types of search task that
respondents perform in their work
 Query formulation:The approaches and techniques
used to construct queries
 Evaluation: How they assess and evaluate the results
of their search tasks
 Ideal search engine: Any other features additional to
those described above
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 10

10.  Distributed via social media + SurveyMonkey
 Qualifying question
 “Is your primary job function to recruit and hire
professionals for your organization or for clients?”
 09 June 2015 to 01 August 2015
 416 responses, 69 qualified, 64 usable
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 11

12.  Clients external (48%), internal (34%), both
(17%)
 patent searchers mostly internal (88%)
 Job titles: recruiter (15%) HR Manager (8%)
and HR Generalist (7%)
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 13
0
2
4
6
8
10
12
14
16
18-24 25-31 32-38 39-45 46-52 53-59 60-65 Over 66
Numberofparticipants
0
5
10
15
20
25
30
35
40
45
50
Male Female
Numberofparticipants

13. UXLabs - User Experience Research and Design - www.uxlabs.co.uk 14
0 5 10 15 20 25 30 35 40
Industry experience
Recruitment experience

14. UXLabs - User Experience Research and Design - www.uxlabs.co.uk 15
0 10 20 30 40 50 60
Psychometric assessment
Other
Employer/brand consulting
Candidate selection
Candidate search
Number of participants
Services provided
0 10 20 30 40 50 60
Other
Commercial database
Open web
Proprietary internal database
Social networks
Job boards
Number of participants
Data sources
0
5
10
15
20
25
Always Often Sometimes Rarely Never N/A
Numberofparticipants
Use of previous examples
0 5 10 15 20 25 30 35
Nobody
Colleagues in my workgroup
Colleagues across my organisation
Public forums
Other
Number of participants
Query sharing

15.  3 hours to complete a search task of 5 queries, with
each query taking around 5 minutes to formulate.
 iterative paradigm
▪ candidate search <-> candidate selection and evaluation
 Shorter tasks (3 hours vs. 12 hours)
 substantially longer than typical web search tasks
 Fewer queries (5 vs. 15)
 Same query formulation time (5 minutes)
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 16
Min Median Max
Search task completion time (hours) 0.06 3 30
Query formulation time (mins) 0.1 5 90
# queries submitted 1 5 50

16. UXLabs - User Experience Research and Design - www.uxlabs.co.uk 17
1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50
Case sensitivity
Query translation
Truncation
Wildcard
Misspellings
Field operators
Weighting
Proximity
Abbreviations
Query expansion
Synonyms
Boolean

17. UXLabs - User Experience Research and Design - www.uxlabs.co.uk 18
0 5 10 15 20 25 30 35 40
Manually, on paper
Text editor
Form based query builder
Visual query builder
Other
Number of participants

18.  Time to assess relevance same as patent
searchers (5 minutes).
 # results examined lower (30 vs. 100).
 satisficing strategy
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 19
Min Median Max
Ideal # of results 1 33 1000
Number of results examined 1 30 100000
Time to assess relevance 1 5 50

19.  Reflects Woudstra &Van den
Hooff (2008) selection criteria
 topic of knowledge (job function)
 physical proximity (location)
 perspective (industry
sector/career level)
 familiarity (previous contact with
recruiter)
 availability
 Different to medical searchers:
 Type of source, date range,
language, etc.
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 20
1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00
Previous contact with recruiter
Salary
Availability
Career level
Industry sector
Location
Job function

20.  Different to healthcare
professionals
 Woudstra &Van den
Hooff “source quality is
the most dominant factor
in the selection of human
information sources”
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 21
0 5 10 15 20 25 30 35 40
Other
Most recent item
Most trustworthy source
First item
Item which looks most relevant
Number of participants

21. UXLabs - User Experience Research and Design - www.uxlabs.co.uk 22
1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50
Exporting search queries (histories)
Storing search results with an expiry date
Using facets/guided filtering
Search history
Saving custom lists from search results
Alerting function
Combining search queries
Combining multiple search results
Organising search queries
Recency of retrieved results

23.  Lengthy search sessions, different notions of
relevance, different sources searched separately,
specific domain knowledge
 “The hardest part of creating a query is comprehending
new information and developing a mental model of the
ideal search result.”
 Some of the most complex queries of any
community
 ‘Boolean Blackbelts’
 Average time spent evaluating a typical result is
5 minutes (not 7 seconds!)
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 24

24.  Selection of suitable terms is an ongoing challenge:
 “The specific job is so new I cannot find terms used on
resumes to match”.
 Satisficing strategies
 “Generally speaking, it's a trade-off between time and
quality of results. [If] I can't identify the required information
in the available time... this is because the data is not present
in any of my go-to data sources, and working around that
limitation isn't the best use of the time”.
 The search tasks are inherently interactive, multiple
iterations of query formulation and results evaluation
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 25

25.  Unstudied community of information
professionals
 New set of real world ‘people search’ scenarios
 IR research assumes natural language queries
 Creation of new test collections and tasks?
 Publicly visible forum to compare and contrast
search strategies (Glen Cathey)
 Translate IR research into practice
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 26

26. Dr.Tony Russell-Rose, FBCS CITP
Director, UXLabs
 Web: http://uxlabs.co.uk
 Email: tgr@uxlabs.co.uk
 Blog: http://isquared.wordpress.com
 LinkedIn: http://uk.linkedin.com/in/tonyrussellrose
 Twitter: @tonygrr
27UXLabs - User Experience Research and Design - www.uxlabs.co.uk

27. UXLabs - User Experience Research and Design - www.uxlabs.co.uk 28