User requirements for complex search strategies

Dr.Tony Russell-Rose, FBCS CITP
UXLabs

April 2014: InnovateUK grant
Complex query formulation
Jan 2015: Smart Award
Proof of Market
3UXLabs - User Experience Research and Design - www.uxlabs.co.uk

 Assess the potential markets
Qualitative research involving
end users from target sectors
Catch-22
Target sector <-> market
research
Who uses complex queries?
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 4

 Usual suspects:
 Patent agents
 Lawyers
 Unknown quantities:
 Librarians?
 Power users:
 Media monitoring
 Healthcare (systematic evidence review)
 And…

 Nested composite
Java AND (Design OR develop OR code OR Program) AND
("* Engineer" OR MTS OR "* Develop*" OR Scientist OR
technologist) AND (J2EE OR Struts OR Spring) AND
(Algorithm OR "Data Structure" OR PS OR Problem
Solving)
 Enumerations of related terms
("looking for" OR "in search of" OR "open to" OR "new
job" OR "actively pursuing" OR "pursuing new" OR
"searching for" OR "new opportunity" OR "new
opportunities" OR "available for" OR "in transition"
OR unemployed OR "immediately available" OR
"currently seeking" OR "seeking new" OR "seeking a
new" OR "interested in")
 Index field lookups
site:ca.linkedin.com "network engineer" "ccnp" "wan" "lan"
"vancouver" -intitle:"profiles" -inurl:"dir/ " -
inurl:job|jobs|jobs2

 Qualitative
 Interviews with stakeholders
▪ 2-3 from each sector
 Quantitative
 Online survey
 Baselines for comparison
▪ Patent search [Joho et al, 2010]
▪ Medical professional search [Geschwandtner et al, 2011]

 40 questions, 15 minutes
 5 sections
 Demographics:The background and professional
experience of the respondents
 Search tasks:The types of search task that
respondents perform in their work
 Query formulation:The approaches and techniques
used to construct queries
 Evaluation: How they assess and evaluate the results
of their search tasks
 Ideal search engine: Any other features additional to
those described above

 Distributed via social media + SurveyMonkey
 Qualifying question
 “Is your primary job function to recruit and hire
professionals for your organization or for clients?”
 09 June 2015 to 01 August 2015
 416 responses, 69 qualified, 64 usable

 Clients external (48%), internal (34%), both
(17%)
 patent searchers mostly internal (88%)
 Job titles: recruiter (15%) HR Manager (8%)
and HR Generalist (7%)
0
2
4
6
8
10
12
14
16
18-24 25-31 32-38 39-45 46-52 53-59 60-65 Over 66
Numberofparticipants
0
5
10
15
20
25
30
35
40
45
50
Male Female

0 5 10 15 20 25 30 35 40
Industry experience
Recruitment experience

0 10 20 30 40 50 60
Psychometric assessment
Other
Employer/brand consulting
Candidate selection
Candidate search
Number of participants
Services provided
0 10 20 30 40 50 60
Other
Commercial database
Open web
Proprietary internal database
Social networks
Job boards
Data sources
0
5
10
15
20
25
Always Often Sometimes Rarely Never N/A
Use of previous examples
0 5 10 15 20 25 30 35
Nobody
Colleagues in my workgroup
Colleagues across my organisation
Public forums
Other
Query sharing

 3 hours to complete a search task of 5 queries, with
each query taking around 5 minutes to formulate.
 iterative paradigm
▪ candidate search <-> candidate selection and evaluation
 Shorter tasks (3 hours vs. 12 hours)
 substantially longer than typical web search tasks
 Fewer queries (5 vs. 15)
 Same query formulation time (5 minutes)
Min Median Max
Search task completion time (hours) 0.06 3 30
Query formulation time (mins) 0.1 5 90
# queries submitted 1 5 50

1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50
Case sensitivity
Query translation
Truncation
Wildcard
Misspellings
Field operators
Weighting
Proximity
Abbreviations
Query expansion
Synonyms
Boolean

0 5 10 15 20 25 30 35 40
Manually, on paper
Text editor
Form based query builder
Visual query builder
Other

 Time to assess relevance same as patent
searchers (5 minutes).
 # results examined lower (30 vs. 100).
 satisficing strategy
Min Median Max
Ideal # of results 1 33 1000
Number of results examined 1 30 100000
Time to assess relevance 1 5 50

 Reflects Woudstra &Van den
Hooff (2008) selection criteria
 topic of knowledge (job function)
 physical proximity (location)
 perspective (industry
sector/career level)
 familiarity (previous contact with
recruiter)
 availability
 Different to medical searchers:
 Type of source, date range,
language, etc.
1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00
Previous contact with recruiter
Salary
Availability
Career level
Industry sector
Location
Job function

 Different to healthcare
professionals
 Woudstra &Van den
Hooff “source quality is
the most dominant factor
in the selection of human
information sources”
0 5 10 15 20 25 30 35 40
Other
Most recent item
Most trustworthy source
First item
Item which looks most relevant

1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50
Exporting search queries (histories)
Storing search results with an expiry date
Using facets/guided filtering
Search history
Saving custom lists from search results
Alerting function
Combining search queries
Combining multiple search results
Organising search queries
Recency of retrieved results

 Lengthy search sessions, different notions of
relevance, different sources searched separately,
specific domain knowledge
 “The hardest part of creating a query is comprehending
new information and developing a mental model of the
ideal search result.”
 Some of the most complex queries of any
community
 ‘Boolean Blackbelts’
 Average time spent evaluating a typical result is
5 minutes (not 7 seconds!)

 Selection of suitable terms is an ongoing challenge:
 “The specific job is so new I cannot find terms used on
resumes to match”.
 Satisficing strategies
 “Generally speaking, it's a trade-off between time and
quality of results. [If] I can't identify the required information
in the available time... this is because the data is not present
in any of my go-to data sources, and working around that
limitation isn't the best use of the time”.
 The search tasks are inherently interactive, multiple
iterations of query formulation and results evaluation

 Unstudied community of information
professionals
 New set of real world ‘people search’ scenarios
 IR research assumes natural language queries
 Creation of new test collections and tasks?
 Publicly visible forum to compare and contrast
search strategies (Glen Cathey)
 Translate IR research into practice

Dr.Tony Russell-Rose, FBCS CITP
Director, UXLabs
 Web: http://uxlabs.co.uk
 Email: tgr@uxlabs.co.uk
 Blog: http://isquared.wordpress.com
 LinkedIn: http://uk.linkedin.com/in/tonyrussellrose
 Twitter: @tonygrr
27UXLabs - User Experience Research and Design - www.uxlabs.co.uk

User requirements for complex search strategies

More Related Content

Viewers also liked

Similar to User requirements for complex search strategies

More from Tony Russell-Rose

Recently uploaded

User requirements for complex search strategies