Successfully reported this slideshow.
Your SlideShare is downloading. ×

Enterprise search research - user satisfaction and search task performance

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 26 Ad

Enterprise search research - user satisfaction and search task performance

Download to read offline

No prior research has been identified that investigates the causal factors for workplace exploratory search task performance. The impact of user, task, and environmental factors on user satisfaction and task performance was investigated through a mixed methods study with 26 experienced information professionals using enterprise search in an oil and gas enterprise. Some participants found 75% of high-value items, others found none, with an average of 27%. No association was found between self-reported search expertise and task performance, with a tendency for many participants to overestimate their search expertise. Successful searchers may have more accurate mental models of both search systems and the information space. Organizations may not have effective exploratory search task performance feedback loops, a lack of learning. This may be caused by management bias towards technology, not capability, a lack of systems thinking. Furthermore, organizations may not “know” they “don’t know” their true level of search expertise, a lack of knowing. A metamodel is presented identifying the causal factors for workplace exploratory search task performance. Semi-structured qualitative interviews with search staff from the defence, pharmaceutical, and aerospace sectors indicates the potential transferability of the finding that organizations may not know their search expertise levels.

No prior research has been identified that investigates the causal factors for workplace exploratory search task performance. The impact of user, task, and environmental factors on user satisfaction and task performance was investigated through a mixed methods study with 26 experienced information professionals using enterprise search in an oil and gas enterprise. Some participants found 75% of high-value items, others found none, with an average of 27%. No association was found between self-reported search expertise and task performance, with a tendency for many participants to overestimate their search expertise. Successful searchers may have more accurate mental models of both search systems and the information space. Organizations may not have effective exploratory search task performance feedback loops, a lack of learning. This may be caused by management bias towards technology, not capability, a lack of systems thinking. Furthermore, organizations may not “know” they “don’t know” their true level of search expertise, a lack of knowing. A metamodel is presented identifying the causal factors for workplace exploratory search task performance. Semi-structured qualitative interviews with search staff from the defence, pharmaceutical, and aerospace sectors indicates the potential transferability of the finding that organizations may not know their search expertise levels.

Advertisement
Advertisement

More Related Content

Slideshows for you (18)

Similar to Enterprise search research - user satisfaction and search task performance (20)

Advertisement
Advertisement

Enterprise search research - user satisfaction and search task performance

  1. 1. Exploratory search task performance and user satisfaction in the enterprise Enterprise Search Europe, London 2015 Paul H. Cleverley
  2. 2. • Systems Thinking: A philosophical alterative to reductionism (pursuit of simple answers to complex issues). Fragmentation in order to make systems more manageable, risks losing sight of the big picture and consequences of actions taken (Senge 1990). – Single loop learning - operationalize actions – Double loop learning - question the norms (Argyris and Schon 1978) • Fallacy of centrality: Experts can overestimate the likelihood they would know about a phenomenon if it was taking place. The “fallacy of centrality” (Weick 1995), i.e. because I don’t know about this event, it must not be going on. • Pragmatism: Concerned with warranted assertions, plausible explanations that fit experiences (not concerned with what is ‘true’) Assumptions and Philosophies
  3. 3. • Two main types of search goal (Marchionini 2006) – Lookup/Known Item AND Exploratory search (Car Racing analogy) • Exploratory searching is important – Missed business opportunities: 14% annual revenue (Oracle 2012) – Poor search can miss evidence of fraud (Johnson, 2013) – Has caused fatalities in the health sector (Savulescu & Spriggs, 2002) • Search can be collaborative but is often an isolated activity • Few causal models for exploratory search task performance Background
  4. 4. Background – the car (the search engine)
  5. 5. Background – the car (the user interface)
  6. 6. Background – exploratory search UI’s (Cleverley and Burnett 2015) “.. with analogues you don’t know what terms to query on, because you don’t know what they are”
  7. 7. Background – Search Centre of Excellence
  8. 8. Background – Monitoring (search log analysis) Techniques such as Failed Search Analysis and Ranking tests such as Mean Reciprocal Rank (MRR) can be biased towards Lookup/known item search goals
  9. 9. Background – engine oil (the content)
  10. 10. Background – focus on the searcher (the driver)
  11. 11. • Search Literacy – “many believe a position has been reached that professional education and research are irrelevant to practice” (Wilson 2008), “We need more ..study of the unique needs and challenges of increasing information literacy skills in the workplace.” (Abram 2013) – Increasing search literacy may mitigate information overload (Bawden 2009) • User satisfaction – “need to study the relationships ..between various user, environment characteristics and satisfaction.” (Al-Maskari & Sanderson 2010) • Task performance – Where a ‘gold standard’ is used (TREC) studies do not measure how the searcher felt or how they (or organization) may have reacted to performance feedback, “we found..very few studies attempted to use some objective form of measuring the level of user’s search performance.” (Moore et al 2007) Background – focus on the searcher (driver)
  12. 12. • H1: There is a difference in user satisfaction (overload to non-overload) • H2: There is a difference is search task performance (overload to non-overload) • H3: There is an association between user satisfaction and search task performance • H4: There is an association between self reported search expertise and search task performance • Q1: What praxes and traits lead to search task success? • Q2: What are the underlying antecedents for search task performance? Research Questions
  13. 13. Methodology – Mixed Methods Use of the company enterprise search engine 1. Use search box only 2. 10 minutes per task 3. Find 10 most relevant items Two search tasks, Task1 providing a feeling of information overload (>500), Task2 a limited amount of possible relevant results (<100) Influence of subject matter knowledge mitigated by task design, choices made just on metadata Search Task 1—Gather recent gravity, magnetics reports for Peru (Task2 same but for Cyprus) 4 high value items were added by the researchers for each task Case study – oil and gas company Permission from management 26 experienced (>10 years) Information Management professionals supporting the exploration department
  14. 14. Methodology – Mixed Methods Participants provide User satisfaction for Task1 and Task2 using a Likert scale Participants provide 10 ‘most relevant’ items each for Task1 and Task2 Researchers identified how many of the high value items were found per participant which was used as the task performance measure The participants were unaware the researcher was able to view the search log in real time Task results (performance and patterns from the search log) were fed back to participants and management in interviews. Questionnaires collected more data
  15. 15. Methodology – Mixed Methods Qualitative: Thematically map interview data using an approach based on grounded theory Quantitative: Statistical inference tests: Kruskal-Wallis, Mann- Whitney U Wilcoxon Signed Rank Spearman Rank CC Task order, age, gender, native language, personality effects tested Conclusions Triangulate Convergence Coding Matrix (CCM) Share results with other organizations in interviews
  16. 16. RESULTS: H1 User Satisfaction (US) difference 54% satisfied for Task1 (Information overload), 65% Task2 (limited results) No statistical significant difference
  17. 17. RESULTS: H2 difference in task performance 18% high value items found for Task1 (Information overload), 36% Task2 (limited results). Difference is statistically significant. Overall performance considered poor.
  18. 18. RESULTS: H3 association US & task performance No association between task1 (overload) and user satisfaction. There is a statistically significant association between task 2 (limited) and user satisfaction
  19. 19. RESULTS – NEW THEORY PROPOSED A statistically significant association exists between user satisfaction ‘delta’ (between Task2-Task1) and search task performance. Mental models play a role. SE (USL-USO) Relative User Satisfaction Theory (RST) Where: SE=Search Expertise USL= User satisfaction (Limited results) USO= User satisfaction (Overload)
  20. 20. RESULTS: H4 self reported search expertise There is no statistically significant association between self reported search expertise and search task performance
  21. 21. RESULTS: Q1 Praxes and traits that led to success As well as knowledge of search query construction, Metacognition “thinking about thinking” is likely to be a key factor in search task performance 1. Absorbing task instructions 2. Recognising implications of only using plurals 3. Query discipline and remembering 4. Avoiding Boolean OR queries 5. Using wildcards correctly 6. Bruce force persistence 7. Creativity 8. Effective results synthesis 9. Adaptation (learning from results returned)
  22. 22. • In case study organization (surprise) – The searchers • “Unbelievable” [P19], “Interesting” [P6], “Very useful” [P21], “I obviously need to experiment more in the searches” [P19], “I will do things differently next time!” [P25]. – The organization: • General Manager for Global Exploration IM “It’s very surprising in 2015, that something so trivial [B2] is not handled as standard by all search engines.” • Search CoE Manager: “We are responsible for making the enterprise search engine work and that people can use it, not whether people are capable of knowing how to search.” RESULTS: Feedback Interviews
  23. 23. • In external organizations (Transferability of findings) – CEO Professional Society providing information “Interesting. I would probably rate myself as very good, but it would not surprise me if I turned out to be very poor!” [O1]. – Knowledge Manager Defence Sector organization, “Nobody takes a strategic overview of search other than making sure the IT service works” [O3] – Aerospace Search CoE Manager ”Very interesting for me to see, in the end, search in big enterprises is looking at the same type of challenges.” [O5]. – Interviews with six organizations (in aerospace, pharmaceuticals, defence and oil & gas sectors) indicate the expertise of searchers is not monitored or fed back. RESULTS: Feedback Interviews
  24. 24. Q2: Improve search: Do we need to learn to learn? TACIT EXPLICIT Individual Experiential Learning (Kolb 1984) Organizational Learning (Argyris & Schon 1978) INFORMAL FORMAL e.g. Enhance using User Interface Scaffolding e.g. “Bottom up” Integrating social networks more closely with enterprise search user interfaces e.g. “Top down” Health check experiments in high leverage/risk areas to assess search literacy.
  25. 25. • In information overload environments, there may be no association between user satisfaction and actual search task performance for exploratory search. • The research data in this sample suggests searchers may not be able to self assess their search competence accurately. • Based on the experience of participants, both they and the organization (management) were surprised at their low performance. In could be inferred that a lack of effective ‘learning’ loops has caused this. • This may have been caused by management bias towards technology, single loop learning and a lack of ‘systems thinking’ and double loop learning with respect to search capability in the enterprise. Conclusions
  26. 26. Thankyou for listening Cleverley, P.H., Burnett, S., Muir, L. (2015). Exploratory information searching in the enterprise: A study of user satisfaction and task performance. Journal of the association for information science and technology (JASIST) http://onlinelibrary.wiley.com/doi/10.1002/asi.23595/abstract Blog: www.paulhcleverley.com email: p.h.cleverley@rgu.ac.uk

×